Reliability analysis

Discuss statistics related things
Post Reply
vincep
Posts: 3
Joined: Sat May 06, 2017 11:37 am

Reliability analysis

Post by vincep »

Hi, I'm trying to get my head around the jamovi reliability analysis, and in particular using omega as a reliability measure.
My understanding of this is based on reading William Revelle:
http://www.personality-project.org/r/book/Chapter7.pdf
http://personality-project.org/r/psych/HowTo/factor.pdf

Here's my questions:
1. It seems like there are two omega values - omegaH and OmegaT. Which does jamovi report?

2. Is it possible to calculate omega reliabilities using polychoric correlations (ie the equivalent of setting poly=TRUE in the omega function in R). Is this necessary for working out reliabilities on data from Likert scales?

3. What is the equivalent R procedure (without using jmv) to produce a list of 'if item dropped' omega values?

4. The Revelle references explain omega calculations as involving an EFA plus some other steps. It seems like the default option for the psych omega function is to calculate an EFA with 3 factors. Is that also what jamovi does? If so, is that going to be ok for all datasets? Are there any situations where we would need to specifiy a different number of factors?

5. A slight bug: From the Reliability Analysis screen, if I clear all variables out of the 'items' selection box, this does not automatically clear items from the 'Reverse Scaled Items' box and causes an error.

Thanks for any clarity on these points.
User avatar
Ravi
Posts: 194
Joined: Sat Jan 28, 2017 11:18 am

Re: Reliability analysis

Post by Ravi »

Thanks for your questions. To calculate McDonald's omega we use the procedure described over here: http://journal.frontiersin.org/article/ ... 9/full#h12. In Appendix I you can see the R code used to calculate omega.

More specific answers:
1. jamovi reports omegaT

2. It's not possible yet to calculate omega using polychoric correlations in jamovi, I'll look into this a bit more. Generally, you can assume that Likert scale data behave fine for these analyses when there are 5 or more categories (See https://pdfs.semanticscholar.org/3833/3 ... 0524cb.pdf for a simulation study).

3. You can see the specific code we use over here: https://github.com/raviselker/Rjamovi/b ... .R#L43-L48. Basically, we calculate the omegaT without including the specific item.

4. We assume 1 factor. See the paper I linked to at the start of this post for more information.

5. Thanks for reporting this. Could you also report this on github (https://github.com/jamovi/jamovi/issues)? This way you'll get an update once this is fixed + we can see clearly which bugs we still need to fix.

Cheers
vincep
Posts: 3
Joined: Sat May 06, 2017 11:37 am

Re: Reliability analysis

Post by vincep »

Thanks so much for the quick reply and those links - very helpful!
I've added the github bug report you requested.

As a general suggestion - it would be really helpful to record the underlying R syntax for each procedure somewhere in the documentation. It's great that novices can quickly and easily use jamovi without too much expertise, but it would also be very nice to let curious users understand what is happening behind the scenes.
User avatar
Ravi
Posts: 194
Joined: Sat Jan 28, 2017 11:18 am

Re: Reliability analysis

Post by Ravi »

For now, the curious reader can already go to our github and check out the precise procedure that we used (https://github.com/jamovi/jmv/tree/master/R contains all our R scripts for jmv. However, it would be nice to make it easy for people to see which packages are used for a particular analysis, also to give credit to the maintainers of the packages we are using.
snfraser
Posts: 7
Joined: Thu May 25, 2017 7:48 pm

Re: Reliability analysis

Post by snfraser »

Ravi,

You can use the userfriendlyscience package in R to calculate Likert Omega and Alpha reliability.

I have a description, links, and example here: http://shawnsstats.blogspot.ca/2016/11/ ... omega.html

-shawn
Raymond89
Posts: 1
Joined: Tue Nov 14, 2017 12:18 pm

Re: Reliability analysis

Post by Raymond89 »

snfraser wrote:Ravi,

You can use the userfriendlyscience package in R to calculate Likert Omega and Alpha reliability.

I have a description, links, and example here: http://shawnsstats.blogspot.ca/2016/11/ ... omega.html

-shawn
Thanks for the link!
Post Reply