jamovi

Posted: **Mon Jun 30, 2025 12:09 pm**

Hi,
I did not find a possibility to check Bonferroni or Holm correction
for multiple comparisons ? How does it work here in Jamovi and how to spot the differences between groups
in output chi squre test result windows, in other words, how to find what is compared with what and where do
statistical differences occur ?

kind regards and thank you,
Andrzej

Posted: **Tue Jul 01, 2025 12:37 pm**

Hey @Andrzej_Andrew,
Thanks for your question about Bonferroni or Holm corrections for multiple comparisons in jamovi's chi-square test.

In jamovi, after a significant chi-square test, the standard and most informative way to understand where the statistical differences lie within your contingency table is by examining the Standardized Residuals and Deviance Residuals (both found under the "Cells" options in the output).
Here's why these are sufficient and why traditional multiple comparison corrections like Bonferroni or Holm are generally not applied in this specific context:

What they tell you: These residuals indicate how much the observed count in each cell deviates from what would be expected if there were no association between the variables.
Standardized Residuals are a form of adjusted Pearson residuals, which are approximately normally distributed, allowing you to identify cells with significant deviations (e.g., an absolute value greater than ~1.96 for a 5% significance level).
Deviance Residuals, calculated from a Poisson Generalised Linear Model (GLM) as implemented in the module, are particularly robust for assessing cell-wise contributions to the chi-square's significance.

Why corrections aren't needed: Unlike post-hoc tests for ANOVA (which compare multiple pairs of means and thus require correction to control the Type I error rate), the analysis of residuals in a chi-square test is a diagnostic tool for the overall table. You are not performing multiple independent pairwise comparisons between groups in the same way. Instead, you are assessing each cell's contribution to the observed deviation from independence.

In essence, the standardized and robust deviance residuals directly answer your question about "where statistical differences occur" by pinpointing the specific cells driving the chi-square's significance.

Kind regards,
Maurizio
https://www.jamovi.org/about.html

Posted: **Tue Jul 01, 2025 2:06 pm**

Thank you very much indeed Maurizio for your kind help.
Just to clarify things a little bit.
I have watched this:
https://www.youtube.com/watch?v=GnThiGZ-g5U
and the tutor is using adjusted residuals and simple Bonferroni correction.
This is why I asked my question. So is it OK to do it in a way like in YT video as well ?
I would be grateful for your kind comment.

kind regards,
Andrzej

Posted: **Tue Jul 01, 2025 10:23 pm**

Hey @Andrzej_Andrew,
Thank you for your follow-up and for sharing the video, it's a great point to clarify!

Yes, the approach shown in the video, using adjusted residuals combined with a Bonferroni correction, is absolutely valid and widely used, especially in educational contexts or when you want to interpret residuals inferentially.

Here’s how to reconcile both approaches:
In the video, the tutor treats each adjusted residual as a separate statistical test (similar to a z-test), asking whether each cell’s deviation is significant.
Since multiple cells are tested, a Bonferroni correction is applied to control for the family-wise error rate.
This method is statistically sound, especially when the goal is to make inferential claims about individual cells i.e., declaring that “this cell shows a statistically significant difference.”

In jamovi, the residuals (standardized and deviance) are presented without correction because they are primarily intended as diagnostics for which cells contribute most to the overall chi-square result, not as individual hypothesis tests.
This perspective is more exploratory/descriptive, and it aligns with the idea that the chi-square test is omnibus (testing the whole table).
That said, if you choose to interpret residuals inferentially, then applying a Bonferroni or Holm correction is entirely appropriate.

So which one is better?
Both are acceptable, it depends on your analytical intent:

Exploratory: Use standardized or deviance residuals directly (as jamovi presents them).

Inferential (confirmatory): Apply a multiple comparison correction like Bonferroni or Holm to the adjusted residuals’ p-values.

In fact, a growing number of statistical guides recommend combining both. Start with residuals for insight, and if you want to formally report significance for specific cells, adjust for multiple testing.

Cheers,
Maurizio
https://www.jamovi.org/about.html

Posted: **Wed Jul 02, 2025 1:55 am**

@Andrzej_Andrew

It is also common, after a chi square analysis, to want to know about proportions or ratios pertaining to subsets of the data. For example,
you might want to know, is A/(A+B) significantly different from 0.5: (i) for Malay, (ii) for Indian, (iii) for Chinese? The overall chi square analysis won't answer those questions. So one could do three additional chi square tests, each including only one ethnicity and only the A and B conditions. You could then apply a Hold correction to the three resulting p values, but jamovi won't do that correction (across analyses) for you. Instead you could use jamovi's Rj module. to run R code--for example: p.adjust(c(.027, .141, .018), "holm")

Posted: **Wed Jul 02, 2025 6:58 pm**

Thank you Both very much Maurizio/@MAgojam and @reason180.

Code: Select all

# Data setup

library(tidyverse)

residuals <- data.frame(
  Ethnicity = c("Malay", "Chinese", "Indian"),
  A = c(-1.0612, -2.0538, 2.9438),
  B = c(-0.47996, 0.047839, 0.40734),
  C = c(1.233, 1.8845, -2.9457)
)

# p-values
compute_p <- function(x) pchisq(x^2, df = 1, lower.tail = FALSE)
raw_p <- as.data.frame(lapply(residuals[ , 2:4], compute_p))
holm_df <- data.frame(matrix(p.adjust(unlist(raw_p), method = "holm"), nrow = 3, byrow = FALSE))
bonf_df <- data.frame(matrix(p.adjust(unlist(raw_p), method = "bonferroni"), nrow = 3, byrow = FALSE))
colnames(holm_df) <- colnames(raw_p)
colnames(bonf_df) <- colnames(raw_p)

# Formatting p-values with asterisks
add_stars <- function(p) {
  ifelse(p < 0.001, "***", ifelse(p < 0.01, "**", ifelse(p < 0.05, "*", "")))
}
format_with_stars <- function(p) sprintf("%.5f%s", p, add_stars(p))

raw_p_fmt <- data.frame(lapply(raw_p, format_with_stars), stringsAsFactors = FALSE)
holm_fmt <- data.frame(lapply(holm_df, format_with_stars), stringsAsFactors = FALSE)
bonf_fmt <- data.frame(lapply(bonf_df, format_with_stars), stringsAsFactors = FALSE)

# Combine everything
rownames_combined <- c(
  paste0(residuals$Ethnicity[1], "_adjusted_residuals"),
  paste0(residuals$Ethnicity[1], "_raw_p_values"),
  paste0(residuals$Ethnicity[1], "_p_values_Holm"),
  paste0(residuals$Ethnicity[1], "_p_values_Bonferroni"),
  paste0(residuals$Ethnicity[2], "_adjusted_residuals"),
  paste0(residuals$Ethnicity[2], "_raw_p_values"),
  paste0(residuals$Ethnicity[2], "_p_values_Holm"),
  paste0(residuals$Ethnicity[2], "_p_values_Bonferroni"),
  paste0(residuals$Ethnicity[3], "_adjusted_residuals"),
  paste0(residuals$Ethnicity[3], "_raw_p_values"),
  paste0(residuals$Ethnicity[3], "_p_values_Holm"),
  paste0(residuals$Ethnicity[3], "_p_values_Bonferroni")
)

final_matrix <- rbind(
  residuals[1, 2:4],
  raw_p_fmt[1, ],
  holm_fmt[1, ],
  bonf_fmt[1, ],
  residuals[2, 2:4],
  raw_p_fmt[2, ],
  holm_fmt[2, ],
  bonf_fmt[2, ],
  residuals[3, 2:4],
  raw_p_fmt[3, ],
  holm_fmt[3, ],
  bonf_fmt[3, ]
)

# Create final dataframe
final_df <- cbind(Row = rownames_combined, final_matrix)
rownames(final_df) <- NULL

I have done some calculations above (based on YT video data) and got this:

https://imgur.com/a/vgFbzvF
By the way: how to insert a picture so it is positioned in the post like beneath this text ?
Check this and correct please if I am wrong.
I have read this as well:
viewtopic.php?t=70,
and that:
viewtopic.php?f=12&t=822
and Maurizo wrote there:

I may ask you, but if you are interested in understanding the relationship between all three of your categorical variables without one necessarily being the "response", haven't you thought about trying a loglinear model?

So I would like to ask about that log-linear model as an alternative to chi square post hoc test, what are the advantages/benefits of using this ?

best regards and thank you,
Andrzej

Posted: **Thu Jul 03, 2025 8:58 am**

Andrzej_Andrew wrote: ↑Wed Jul 02, 2025 6:58 pm So I would like to ask about that log-linear model as an alternative to chi square post hoc test, what are the advantages/benefits of using this ?

Hey @Andrzej,
Looks like you jumped in the time machine and traveled back, so did you miss out on this detail in another comment of mine that might answer your request?
viewtopic.php?f=5&t=2018&hilit=Log+linear#p7369

So, to conclude my input here, I'll summarize...
Log-linear models are indeed a powerful alternative, or rather, an extension, to the standard chi-square post-hoc analysis, especially when dealing with more complex contingency tables involving three or more categorical variables.

The main advantages of using log-linear models are:
Analysis of Higher-Order Interactions: Unlike traditional two-way chi-square tests and their residuals, log-linear models allow you to explore and test not only pairwise associations but also more complex, higher-order interactions (e.g., A*B*C). This helps you understand if the relationship between two variables changes depending on the levels of a third variable.
Specific Hypothesis Testing: You can formulate and test precise hypotheses about the structure of relationships among your variables. By comparing different nested models, you can find the most parsimonious model that best explains your data.
Deeper Insight into Relationships: They offer a more profound understanding of the interdependencies among all categorical variables simultaneously, going beyond simply identifying significant cells.

While for a simple 2-way table, standardized and robust deviance residuals (like those provided in jamovi) are often sufficient to pinpoint deviating cells, log-linear models are the tool of choice when your research requires modeling and interpreting more nuanced, multivariate relationships.

Cheers,
Maurizio
https://www.jamovi.org/about.html

Posted: **Fri Jul 04, 2025 1:14 pm**

Much obliged, thank you @Maurizio,

best wishes,
Andrzej

jamovi

Chi square post hoc tests and correction for multiple testing

Chi square post hoc tests and correction for multiple testing

Re: Chi square post hoc tests and correction for multiple testing

Re: Chi square post hoc tests and correction for multiple testing

Re: Chi square post hoc tests and correction for multiple testing

Re: Chi square post hoc tests and correction for multiple testing

Re: Chi square post hoc tests and correction for multiple testing

Re: Chi square post hoc tests and correction for multiple testing

Re: Chi square post hoc tests and correction for multiple testing