jamovi

Hi !

I am running a multinomial logistic regression with 25 continuous predictors (reasons to prefer a treatment). Outcome variable has 6 levels (different preferred treatments).

In jamovi, I always turn on the Omnibus likelihood ratio tests, which (in my understanding) determines which of the predictors – if any - has a statistically significant effect on the choice of outcome variable across all its 6 levels, not just one pair of comparison (e.g., Treatment 1 preferred vs reference category).

However, changing the reference category in the exact same model not only changes the coefficients and p values of the omnibus tests but also the overall fit of the model itself (AIC value, chi-squre value, p value), which doesn't make any sense.

Could someone please explain why this is happening? Changing the reference category shouldn't change things, no? What am I missing?

Thank you!

Hey @mihaidricu,
why changing the reference category in a multinomial logistic regression can affect the overall model fit statistics (AIC, chi-square, p-value), even though it should not fundamentally alter the underlying relationships between predictors and outcomes?

A possible quick answer:
In multinomial logistic regression, you choose a category of the outcome variable as the reference category.
The coefficients for each predictor represent the log odds of choosing a specific outcome category relative to the reference category.
Changing the reference category shifts the perspective of comparison, leading to different coefficient values ​​and interpretations.

What impact can there be on the model fit statistics?
AIC (Akaike Information Criterion) is a measure of model fit that penalizes models with more parameters.
While the underlying relationships between predictors and outcomes remain the same, the specific parameter estimates (coefficients) change when the reference category changes.
This change in parameters can subtly affect the AIC value, even though the overall quality of the model may not differ substantially.
The chi-square statistic and its associated p-value typically assess the overall fit of the model.
These statistics are influenced by the model coefficients and how well the model predicts the observed outcomes.
Because changing the reference category alters the coefficients, it can affect the chi-square and p-value, even though the ability of the model to explain the data remains largely unchanged.

For a conceptual understanding:
Imagine comparing the heights of three groups: A, B, and C.
If you choose group A as the reference, you compare the heights of groups B and C to group A.
If you choose group B as the reference, you compare the heights of groups A and C to group B.
The comparisons change, but the underlying differences in heights between the groups remain the same.

In summary,
changing the reference category in a multinomial logistic regression primarily affects the interpretation of the coefficients and their associated p-values. While they should not dramatically alter the model's ability to fit the data, subtle changes in coefficients can affect fit statistics such as AIC, chi-square, and the overall p-value.

One suggestion is to focus on interpreting the coefficients by paying attention to how the coefficients change when you change the reference category and interpret them accordingly.

Consider comparing models.
If you are primarily interested in model fit, compare models using metrics such as AIC or BIC, but be aware of the potential for small changes due to changes in the reference category.
Check the model's predictions, evaluating how well the model predicts outcomes in different reference categories.
If the predictive accuracy remains consistent, it suggests that the quality of the underlying model is not substantially affected by the choice of reference category.

I hope this light explanation helps.

Cheers,
Maurizio
https://www.jamovi.org/about.html