Inconsistency in scale calculations

[mmetz]

In a course, I have an assignment where students calculate a mean composite score that requires them to reverse code a few items. I explain how to do this 'manually' by reversing the items and the computing a new variable, but I know they are now able to do this within the menu for calculating scale reliabilities. A student identified an issue where the values for each method are different - and they seem to be different by a constant. Attaching the documentation shared by the student here - any help figuring out what is leading to this difference would be great!

[reason180]

Hi. I would suggest that you post a jamovi (.omv) file--not an image file--that shows the inconsistency.

[Marielle]

The means calculated via a manual reverse coding and those via Reliability differ if some of the items don't use the full range. See attached example. My made-up items are each scored on a 5-point scale, but the actually used range for item C is [1; 4] and for D [2; 5]. If you recode those scores manually (or via a transform) the recoded ranges are [2; 5] and [1; 4] (a 1 becomes a 5, a 2 becomes a 4, etc).

However, if you reverse the items via Reliability, apparently the reversed C gets scores 4 to 1 and the reversed D gets scores 5 to 2. This results in a lower mean for ABC scores (with reversed C), and a higher mean for ABD scores (with reversed D).

[jonathon]

hey,

there's an even chance someone will chime in with an explanation in the next few days, but if that doesn't happen, ravi will take a look at this on wednesday.

cheers

[MAgojam]

there's an even chance someone will chime in with an explanation in the next few days, but if that doesn't happen, ravi will take a look at this on wednesday.

Hey @Marielle
Based on your analysis and the provided example, the issue is that the Reliability module doesn't use a standard inversion formula (like 6-x for a 1-5 scale), but instead relies on the actual minimum and maximum values present in the data.

Here's my response to the question:
The discrepancy you've noticed between manual calculation and the Reliability module is correct and depends on the inversion algorithm used in the module. Reliability doesn't use a fixed formula like "6 - score" (for 1-5 scales), but dynamically determines how to reverse scores based on the actual range of the data.
At this link you can take a look at the code: https://github.com/jamovi/jmv/blob/mast ... #L258-L281
In the code, the inversion is calculated as:

Code: Select all

minItem <- min(dataItem)
maxItem <- max(dataItem)
adjust <- minItem + maxItem
data[[item]] <- adjust - dataItem

This algorithm adapts to the actual range of the data.
In your example:

• Item C has range [1;4]: adjust = 1+4 = 5, so reversed C becomes (5-C)
• Item D has range [2;5]: adjust = 2+5 = 7, so reversed D becomes (7-D)

This approach maintains the same "distance" from the range boundaries. For example, if an item only uses values 2-5 of a possible 1-5 scale, the value 2 (which is distance 1 from the possible minimum) will become 5 (distance 1 from the possible maximum).

This perfectly explains why the composite means differ by 0.333 (or 1/3) when one of the three variables is reversed: if the actual range doesn't cover the entire possible scale, the inversion will produce values shifted by a constant.

The choice of this adaptive method makes sense because the module cannot know the intended scale in advance.
However, it might be useful to add an option allowing users to specify the complete theoretical range of the scale (e.g., 1-5, 1-7) to obtain an inversion based on that range rather than on the values actually observed.

Cheers,
Maurizio
https://www.jamovi.org/about.html

[reason180]

RE "The choice of this adaptive method makes sense because the module cannot know the intended scale in advance."

That's one way to look at it. But the way I see it, the only correct way to reverse-code is to base it on the theoretical, not the actual range of the data. For example, for -2 to 2 scale [Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree], the current reliability algorithm re-scores the value set,

0 ("Neutral")
-1 ("Disagree")
-2 ("Strongly Disagree") [reverse-score this]

to be

0 ("Neutral")
-1 ("Disagree")
0 ("Neutral")

which is of course incorrect.

So to guarantee correctness, I think the re-scoring should always be done outside the reliability module. Either that or, as was recommended. The module should be altered so that it requires the user to specify the theoretical ranges.

[mmetz]

Thank you all so much. It seems reverse scoring being done on actual, not theoretical, values seems to make sense.

My question then is - when is this the preferred approach? As an experimental social psychologist, I cannot imagine a time where that is the outcome I would want, so I'm curious if anyone knows!

[reason180]

Well, my suggestion was just the opposite: Always do the reverse scoring based on the theoretical not the actual range.

[mmetz]

Sorry @reason180, I meant it makes sense as an explanation for the discrepancy. I agree it does not make sense to me conceptually, hence my question!

[Ravi]

Hi everybody,

So first let me reply to the following:

That's one way to look at it. But the way I see it, the only correct way to reverse-code is to base it on the theoretical, not the actual range of the data. For example, for -2 to 2 scale [Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree], the current reliability algorithm re-scores the value set,

0 ("Neutral")
-1 ("Disagree")
-2 ("Strongly Disagree") [reverse-score this]

to be

0 ("Neutral")
-1 ("Disagree")
0 ("Neutral")

which is of course incorrect.

So to guarantee correctness, I think the re-scoring should always be done outside the reliability module. Either that or, as was recommended. The module should be altered so that it requires the user to specify the theoretical ranges.

So actually, because the adjustment score in the formula mentioned by @MAgojam is -2, this would turn into:

-2 ("Neutral")
-1 ("Disagree")
0 ("Strongly Disagree")

Which is a reversal of the scale based on the (limited) information that you have. For the reliability statistics (alpha and omega) this is enough, and you'll see that it will give you the same statistic whether you use your manually reversed items or the auto reversed items. So when would you use this: to easily calculate the reliability statitics.

However, the composite scores (mean and sum) provided in the reliability analysis are convenience functions that can only use the information that they have. For proper composite scores, I'd calculate it in the spreadsheet so that you have control over the theoretical constraints.