Post hoc testing after two-way ANOVA

Discuss statistics related things
Post Reply
Bobafett
Posts: 76
Joined: Thu Jul 18, 2019 11:33 am

Post hoc testing after two-way ANOVA

Post by Bobafett »

Hiya,
Apologies if this has been posted previously, but I have a query about the number of post hoc tests calculated in a two-way ANOVA. jamovi appears to calculate all the possible combinations of condition pairings, rather than those that are required. For instance, in a simple (2*2) repeated measures design, whilst it is possible to calculate 6 paired post hoc tests, only 4 are required.

Let me give an example with a 2*2 factorial design (doesn't matter if it is between, repeated or mixed). I'm interested in the effect of a drug (ibuprofen vs. aspirin) and food (greasy fry-up vs. muesli) on hangover reduction. The interaction of these two IVs was significant and so I run post hocs.

I'm interested in four comparisons:
1. Ibuprofen & greasy food vs. Ibuprofen & muesli
2. Aspirin & greasy food vs. Aspirin & muesli
3. Greasy food & Ibuprofen vs. Greasy food & Aspirin
4. Muesli & Ibuprofen vs. Muesli & Aspirin

...however jamovi will also compare:
5. Greasy food & Aspirin vs. Muesli & Ibuprofen
6. Greasy food & Ibuprofen vs. Muesli & Aspirin

In my mind it makes no sense to run #5 & 6 - there is no common link between the combination of conditions i.e. different food type and different drugs are being compared. To me it's rather like comparing apples and oranges.

I could just pick and choose the 4 tests that I need from the post hoc table, but any corrections applied would be against all 6 tests and not the 4 I need.

This is just a long-winded way to ask is there a way to only run the post hoc tests that I need?

Cheers for any help.

p.s. great work on getting jamovi to run in a browser!
User avatar
jonathon
Posts: 2613
Joined: Fri Jan 27, 2017 10:04 am

Re: Post hoc testing after two-way ANOVA

Post by jonathon »

hey,

so the principle of post-hoc testing is that you have no principled reason for thinking one comparison more important than another. so i wonder if you don't want more of a 'planned comparisons' approach.

you could run the post-hocs without a correction, and then copy the 4 p-values into a correction procedure (but i'm not 100% sure this is allowed).

let's see if someone else can provide any insights.

jonathon
giu10571
Posts: 1
Joined: Sat Jan 29, 2022 5:29 pm

Re: Post hoc testing after two-way ANOVA

Post by giu10571 »

I think it is not a planned compared approach meaning I "manipulate" the procedure, but I plan to compare what is meaningful. There may be comparisons that in real life or relative to the research question have no sense or interest. Hence, the possibility to tell the software what comparisons will be required to answer the research question can be, in places, appropriate.
User avatar
reason180
Posts: 274
Joined: Mon Jul 24, 2017 4:56 pm

Re: Post hoc testing after two-way ANOVA

Post by reason180 »

Regarding "you could run the post-hocs without a correction, and then copy the 4 p-values into a correction procedure (but i'm not 100% sure this is allowed)" I too am unsure if it's "allowed."

However, a straightforward way to do these planned comparisons, with adjustment for familywise error, would be to conduct such comparisons completely outside the ANOVA framework. After running the ANOVA, run four separate t tests and then use R's p.adjust function to correct the resulting four p values:

Code: Select all

# This example code computes the Holm-corrected p values for a vector of 
# uncorrected p values. (Options besides "holm" are: 
# "hochberg", "hommel", "bonferroni", "BH", "BY", and "fdr".)

p.adjust(c(.027, .141, .018, .053), "holm")  

# output: [1] 0.081 0.141 0.072 0.106
(I seem to recall that Jonathon started writing a 'Many-t' module to save the tedium of having to to run many separate t tests [4 is OK but 10 would be tedious]. Did that project ever get finished?)
User avatar
jonathon
Posts: 2613
Joined: Fri Jan 27, 2017 10:04 am

Re: Post hoc testing after two-way ANOVA

Post by jonathon »

> started writing a 'Many-t' module to save the tedium of having to to run many separate t tests

no, but it's probably not a bad project for a beginner if anyone wanted to run with it.

jonathon
User avatar
Ravi
Posts: 194
Joined: Sat Jan 28, 2017 11:18 am

Re: Post hoc testing after two-way ANOVA

Post by Ravi »

Yeah, I started it a while ago but never finished it for some reason (See, https://github.com/raviselker/manytee). But in this module, all comparisons are done and you cannot just select the ones you are interested in.
User avatar
MAgojam
Posts: 421
Joined: Thu Jun 08, 2017 2:33 pm
Location: Parma (Italy)

Re: Post hoc testing after two-way ANOVA

Post by MAgojam »

Hey Ravi,
I took a look at your manytee module and it does well what it needs to do.
Why don't you bring it into the availability of jamovi's library?
I think it might be of interest to many.
TA, cheers
Maurizio
Post Reply