Reliability Analysis - Sum scores

Discuss statistics related things
Post Reply
hallgatoemese
Posts: 6
Joined: Tue Nov 17, 2020 7:33 pm

Reliability Analysis - Sum scores

Post by hallgatoemese »

Dear Jamovi community,

i am using Jamovi version 1.8.1.0., and i am excited to see that in the Reliability Analysis menu there is an option to SAVE sum scores and mean scores. I tried it out immeadiately, i have four items i analysed with PCA and they combine into a single component, and there are no reversed items. So i supposed that the sum score i can get from the menu (from Reliability analysis) should give me the exact same values i would obtain by simply adding the four values.

However, this is not the case, and i don't understand why.
Moreover, the sum score that I computed and the sum score that resulted from this menu does not even show correlation (only a very week one), while it shows a very high correlation with the PCA component score that i also saved. The reason i am writing this is that if i am right and there is a problem with the sum scores, the problem might extend to PCA component scores as well.

I am also curius how does the program compute sum scores when there are reversed items. Usually, when i compute it for myself, i use a very simple formula to reverse items:

the possible MINIMUM score on the scale + the possible MAXIMUM score on an scale - the ACTUAL score that was given.

However, in the reliability analysis window i can not provide these information, how does it know how to reverse scores?
or i am misunderstanding the aim of these scores, and they show something else?

Thank you!
Emese
User avatar
Ravi
Posts: 194
Joined: Sat Jan 28, 2017 11:18 am

Re: Reliability Analysis - Sum scores

Post by Ravi »

Hi Emese,

The sum score of reliability analysis should just give you the same as just adding the values within a row (see https://github.com/jamovi/jmv/blob/34b5bb3a7be259002e983277d49810b461c0e5eb/R/reliability.b.R#L193). When I calculate the sum score from within the reliability analysis I get the same scores as when I calculate them using a computed variable straight in the spreadsheet, so I'm curious to see why it is not working for you. Is it possible for you to attach the data over here (or send it to the following email: selker [dot] ravi [at] gmail [dot] com. replace the [dot] with . and [at] with @).

About the reverse scoring: it goes the exact way you expected it: https://github.com/jamovi/jmv/blob/34b5bb3a7be259002e983277d49810b461c0e5eb/R/reliability.b.R#L264-L272.

Cheers,
Ravi
User avatar
MAgojam
Posts: 421
Joined: Thu Jun 08, 2017 2:33 pm
Location: Parma (Italy)

Re: Reliability Analysis - Sum scores

Post by MAgojam »

Hi, @hallgatoemese.

I don't know if this can partially answer your questions, but I am attaching a screenshot, where I used the Iris database, present in the jamovi library, for a "Reliability Analysis" saving Total and Average of the scores in the sheet.
The results are correct, both with normal and reversed elements.
As can be seen from the screenshot, in addition to the 4 variables of interest, in the sheet there are a "Mean Score" and a "Sum Score" variable as the result of the selection of the 2 check-boxes for Save.
Three variables are in the "Normal Scaled Items" list-box and one (Sepal.Width) has been moved to the "Reverse Scaled Items" list-box.
For a "visual" comparison of the values in "Sum Score", I computed a new sum variable, considering, as you would have done for the inverted elements, MAX + MIN - ACTUAL score for the reversed variable (Sepal.Width), obtaining the same values as "Sum score".
Reliability Analysis.png
Reliability Analysis.png (100.21 KiB) Viewed 15318 times
Cheers,
Maurizio
hallgatoemese
Posts: 6
Joined: Tue Nov 17, 2020 7:33 pm

Re: Reliability Analysis - Sum scores

Post by hallgatoemese »

Thank you for your quick reply.

When i copied the relevant variables to a new dataset, it worked!
However, in the original dataset i still get the same. Maybe it has to do something with filters? I attached the file.
I deleted all the unnecessairy variables, only those remained that are used in the filter, and the target variables

TARGETVAR_0
TARGETVAR_1
TARGETVAR_2
TARGETVAR_3

(these are ordinal variables, although i did not set them as such. i didn't change that, i will try)
as you can see in the example, sum scores are not correctly calculated, and there are 66 missing values (which is another thing, but i don't get it).

Regarding reversed items:
i might be wrong, but my concern is the following. Let's say a scale is a likert scale 1-7. then a score 2 when reversed should be 1+7-2 = 6.

However, if i can not explicitly state that the minimum score is 1 and the maximum is 7, and nobody gave the scores 1, 2 or 7 (just scores 3-6), then the reversed score - based on the algorithm - will be 3+6-2 = 7. And thus results will not be comparable with standard values on the test.
I'm I missing something?

Emese
Attachments
Ravi_example.omv
(17.52 KiB) Downloaded 680 times
hallgatoemese
Posts: 6
Joined: Tue Nov 17, 2020 7:33 pm

Re: Reliability Analysis - Sum scores

Post by hallgatoemese »

ah, the example i gave regarding reversed items is wrong in that if nobody gave answers 1, 2 or 7, than - of course - there is no need to reverse the score 2, since nobody gave that score.
however, the problem persists if someone gives the score 4 in the example above.

likert 1 - 7
observed minimum 3, observed maximum 6
given score: 4

3 + 6 - 4 = 5

theorethical minimum 1, theorethical maximum 7

1+ 7 - 4 = 4
hallgatoemese
Posts: 6
Joined: Tue Nov 17, 2020 7:33 pm

Re: Reliability Analysis - Sum scores

Post by hallgatoemese »

I think i see a pattern in the previously attached file.
I am attaching a screenshot. it seems like it do has to do something with filters, and the problem seems to extend to PCA component scores as well (there is a very high correlation between the component scores and the wrong sum scores, but very low correlation between the component scores and the correct sum scores).
I hope it helps.

Emese
Attachments
jamovi_example.png
jamovi_example.png (25.99 KiB) Viewed 15269 times
User avatar
MAgojam
Posts: 421
Joined: Thu Jun 08, 2017 2:33 pm
Location: Parma (Italy)

Re: Reliability Analysis - Sum scores

Post by MAgojam »

Hi, Emese.

I took a look at the file you attached.
The problem occurs when you activate a filter and select the Save options, which are available for some analyzes.
The variables generated in the sheet are vectors that contain only the values for the cases that satisfy the filter conditions, but the values start from the first row of the sheet losing synchronization with the reference cases.
There are various filters in your file, which together exclude 71 cases from analysis.
Take a look at the variables "Sum Score - FROM RELIABILITY ANALYSIS" and "Score Component 1 - Saved from PCA" and you will notice that the values are present only from rows 1 to 373.
On rows 374 to 444, the cells are blank (as are the number of cases filtered), but they do not match only the gray cases for the filter.
I think we need an intervention from the Jamovi guys.

Cheers,
Maurizio
User avatar
jonathon
Posts: 2613
Joined: Fri Jan 27, 2017 10:04 am

Re: Reliability Analysis - Sum scores

Post by jonathon »

thanks mz,

we'll push out a fix shortly.

with thanks
Post Reply