Weird behaviour of gamlj 2.5.3 GLM

Everything related to the development of modules in jamovi
User avatar
Arnaud Mortier
Posts: 32
Joined: Tue Apr 27, 2021 11:39 am

Weird behaviour of gamlj 2.5.3 GLM

Post by Arnaud Mortier »

Hi,

I'm experiencing a weird issue with GLM effect sizes in gamlj 2.5.3.

Say we have one factor (with balanced data) and one covariate.
I would expect eta squared to be SS_effect / SS total and partial eta squared to be SS_effect / (SS_effect + SS_residuals). None of this is true, except for the partial eta squared of the second effect, weirdly enough.

And yes, the values I get aren't the same depending on the order in which I put in the variables, be it the factor first or the covariate first, even though they don't go in the same box. So the second effect mentioned above can either be the effect of the factor or that of the covariate, depending on the order you put the variables in.

This behaviour looked somewhat like what you get with unbalanced data when you compare type I, II and III SS, but even with this approach I couldn't make sense of the results I get.

Any idea is more than welcome!
Thanks.
Attachments
testGLM.omv
Two GLMs giving the same results except for the effect sizes.
(12.58 KiB) Downloaded 620 times
User avatar
mcfanda@gmail.com
Posts: 457
Joined: Thu Mar 23, 2017 9:24 pm

Re: Weird behaviour of gamlj 2.5.3 GLM

Post by mcfanda@gmail.com »

Yes, you're right, it's a bug. The version 2.5.3 of gamlj relies on R package effectsize, which uses anova() when a lm model is passed, and R anova() uses type I SS. If you check the effect sizes selecting "Additional info" "on Effect size" in the "Options" panel, you see the correct effect sizes (there are computed based on the car::Anova() type 3 SS methods).

In previous versions there was no issue because the computation was done directly on the car::Anova() SS.

I'm pushing a new version that corrects the issue.
thanks for sharing
User avatar
Arnaud Mortier
Posts: 32
Joined: Tue Apr 27, 2021 11:39 am

Re: Weird behaviour of gamlj 2.5.3 GLM

Post by Arnaud Mortier »

Thank you very much, that makes sense!
I can see that the change has already taken effect, that's awesome.
One last thing I don't understand is that in the new version with the bug corrected, the eta squared of the entire model doesn't seem to be the sum of the eta squared of the effects, as I believe it should be. Am I missing something?
Thanks again for your quick reply!
Arnaud
User avatar
mcfanda@gmail.com
Posts: 457
Joined: Thu Mar 23, 2017 9:24 pm

Re: Weird behaviour of gamlj 2.5.3 GLM

Post by mcfanda@gmail.com »

Hi
This is a very interesting question which does not imply a software bug, but it requires to reflect on how to compute effect size indexes.

Let's start by saying that we know from theory that the eta-squared of the full model is not equal to the sum of the effects eta-squared. It is only so if the predictor terms are correlated exactly 0, such as in balanced anova. This is because SS_model (the SS of the predicted values) is not generally equal to the sum of the SS of the effects (Let's call the sum of effects SS as SS_effects). Intuitively, you can have a model that explains a lot of variance of the dependent variable, but none of the individual predictors explains much variance keeping constant the other predictor.

Now, eta-squared has been defined by Pearson as

Code: Select all

eta_squared=SS_model/(SS_model+SS_error)

(see Cohen, 1973). Other authors have simplified the formula as

Code: Select all

eta_squared=SS_effects/(SS_effects+SS_error)
, mainly because in balanced anova the two formulas yield exactly the same results. In general, the two results are not identical, even if the results are usually very, very close.

As regards the whole model, the model eta-squared is the R-squared. The R-squared is invariably always computed as.

Code: Select all

SS_model/(SS_model+SS_error)
. On this, there's never an issue, all software uses this formula.

When it comes to individual effects, however, many software and R packages use

Code: Select all

SS_x/(SS_effects+SS_error)
This is because it's easier to compute the eta-squared only knowing the SS of each individual effect. GAMLj followed this method, following R effectsise package.

Now, if SS_model=SS_effects, no issue arises, but when SS_model != SS_effects, the estimated eta-squared are only approximated. there's usually no substantial difference in practice, but they are not the exact eta-squared according to Cohen's definition.

Now that you made me think about the issue, I'm inclined to switch to a more accurate computation (using SS_model+SS_error as denominator). So in the next versions, we can have it implemented.

Again, a better computation will not make the individual eta's sum up to R-squared, but will make them more accurate.
User avatar
Arnaud Mortier
Posts: 32
Joined: Tue Apr 27, 2021 11:39 am

Re: Weird behaviour of gamlj 2.5.3 GLM

Post by Arnaud Mortier »

Hi,

Thank you very much for your answer, and sorry for the delay - I missed the notification.

It does make sense to me that with unbalanced data the whole reasoning depends on which type of SS one chooses, but I thought that with balanced data SS_model was by definition equal to SS_effects. Apparently there may be indeed some part of the theory that I'm missing, thanks for pointing this out.

However, it looks like Gamlj was updated back into a version that features the type-I SS bug again - or is it just me?

Best wishes,
Arnaud
User avatar
mcfanda@gmail.com
Posts: 457
Joined: Thu Mar 23, 2017 9:24 pm

Re: Weird behaviour of gamlj 2.5.3 GLM

Post by mcfanda@gmail.com »

Hi,
it should not! I've checked and I do not find a case in which Type III is not as expected. You're right in saying that in balanced designs (with categorical independent variables) SS_model is equal to SS_effects. It is in all my tests. Do you have an example where the software does not give you that?
User avatar
Arnaud Mortier
Posts: 32
Joined: Tue Apr 27, 2021 11:39 am

Re: Weird behaviour of gamlj 2.5.3 GLM

Post by Arnaud Mortier »

Hi,

For some technical reason (CIO too busy at the moment) since you fixed the package last time and my university upgraded jamovi to 2.2.2, I need to load the package from the library every time I use it - the computers that I use won't remember it. It did work fine for a few weeks and now it doesn't any more.

On my personal computer where the fixed version was still installed, I just proceeded to click Modules -> jamovi library -> Available -> Update (under gamlj)
I can tell that:
- the version is still numbered 2.6.1
- new kinds of effect sizes have appeared
- the bug with sums of squares is back.

As for the eta squared of the model not being equal to the sum of the eta squared of the effects, I haven't kept a file where the issue shows, and right now I can't seem to be able to replicate it, sorry about that. I'll post another answer if it comes back.

Thanks for all your help!
Best,
Arnaud
User avatar
mcfanda@gmail.com
Posts: 457
Joined: Thu Mar 23, 2017 9:24 pm

Re: Weird behaviour of gamlj 2.5.3 GLM

Post by mcfanda@gmail.com »

Hi, the latest version is 2.6.1 but the bug with SS is fixed. As for the eta squared of the model not being equal to the sum of the eta squared of the effects, that is fine, it is not a problem
User avatar
Arnaud Mortier
Posts: 32
Joined: Tue Apr 27, 2021 11:39 am

Re: Weird behaviour of gamlj 2.5.3 GLM

Post by Arnaud Mortier »

Hi,

All I can say is that on every computer I know with an access to the library directly from the software, 2.6.1 features the bug.

On one computer I can install gamlj only via sideloading, but the linux .jmo file from https://www.jamovi.org/library.html is version 2.4.5.

Thank you for your time.
Best wishes,
Arnaud
User avatar
Arnaud Mortier
Posts: 32
Joined: Tue Apr 27, 2021 11:39 am

Re: Weird behaviour of gamlj 2.5.3 GLM

Post by Arnaud Mortier »

Here is a screenshot where you can see the version name 2.6.1, the additional effect size epsilon squared that wasn't in Gamlj two weeks ago when the bug was still fixed, and the SS_model that is not equal to the SS_effects, which is a symptom of the bug.
gamlj.png
gamlj.png (68.69 KiB) Viewed 10300 times
Post Reply