ROC Curve P value

[Beatrice]

I was wondering if someone could help me with the following problem, I apologize in advance if my query is too silly, my statistical background is rather shallow.

I am doing a study on a lab test, and I want to check if it can discriminate well between 30 healthy and 30 diseased subjects.
I ran a ROC curve analysis using the PPDA module, and it returned an AUC 0.614, and a number of cutoffs.

Publishing journals require a p value for the AUC, and I can't figure out how to find that, I read everything on the subject online and I still don't know what to do.

I already ran the following tests
A univariate logistic regression analysis returned a p value 0.249
Independent sample T test Mann Whitney U =347.5, p value = 0.131
One sample T test: p<0.001
I read that the AUC p value is related to the Mann Whitney U statistic, and that pROC on R uses the wilcox.test function to calculate it. I honestly have zero knowledge of R, so I copied this code and using the R module, I ran the following:

wilcox.test(data$THRIL, y = NULL,
alternative = c("two.sided"),
mu = 0,
paired = FALSE,
exact = NULL,
correct = TRUE,
conf.int = TRUE,
conf.level = 0.95)


And received the following result:

Wilcoxon signed rank test with continuity correction

data: data$THRIL
V = 1830, p-value = 2e-11
alternative hypothesis: true location is not equal to 0
95 percent confidence interval:
6.269 9.616
sample estimates:
(pseudo)median
7.73

So...where can I get the required p value ?
If there is special R code please tell me exactly what to write, and where to get a required special module if any.

[MAgojam]

So...where can I get the required p value ?
If there is special R code please tell me exactly what to write, and where to get a required special module if any.

Hey Beatrice,
I see you're waiting.

It should come as no surprise that AUC can connect directly with the Mann-Whitney U statistic, which is commonly used in a robust, nonparametric alternative to the Student's t-test.
So, take your predictor variable (THRIL) and your dichotomous variable (outcome) and run a nonparametric test for independent data to get the Mann-Whitney U statistic.

The p-value is given by the test while your AUC will be:
AUC = 1 - U/(n1*n2)
Obviously your denominator will be 900, since you have 60 cases made up of 30 healthy subjects and 30 sick subjects.

I am attaching a jamovi file, with a simple example where you find, TestROC, binary logistic regression, Mann-Whitney U and a few lines of code in Rj and you will see that the resulting AUC does not change.

AUC_p-value.omv

(28.69 KiB) Downloaded 105 times

Cheers,
Maurizio Agosti
https://www.jamovi.org/about.html

[Beatrice]

Sir, I can't begin to thank you enough for your time responding to my query.

So this is what I understood after studying the file you attached :

1. The AUC doesn't change no matter the method you use to get it, TestROC, Regression or the Mann Whitney U dependent equation (AUC =1-U/n1*n2)
2. As for the p value, the one in the binomial logistic regression is the p value for the predictive ability of THRIL in the regression model itself and has nothing to do with the AUC.
3. If I want a p-value that the AUC is significantly different from 0.5 then i should use the p value for the mann whitney test.

Please tell me if I have misunderstood anything.

Which brings me to a theoretical question, if on the real data the logistic regression is non significant (p>0.05), and the Mann Whitney is significant (or the other way around), can that ever happen ?
In which case, if the AUC is, say, 0.61, do we say the lab test can or cannot discriminate between the healthy and the diseased ?

Many thanks again.
Best regards.

[MAgojam]

Which brings me to a theoretical question, if on the real data the logistic regression is non significant (p>0.05), and the Mann Whitney is significant (or the other way around), can that ever happen ?
In which case, if the AUC is, say, 0.61, do we say the lab test can or cannot discriminate between the healthy and the diseased ?

The AUC has an important statistical property: the AUC of a classifier is equivalent to the probability that the classifier will rank a randomly chosen positive instance higher than a randomly chosen negative instance.
This is equivalent to the "Mann-Whitney U statistic" or the "Wilcoxon rank sum statistic".

Why the AUC is showing small value and the p-value is high significant?

There is no direct relationship between AUC and p-value of Wilcoxon.
AUC measures, how well two distributions may be separated.
The p-value measures, to which extent it is clear that the distributions are different.
The test is significant, even if it is very clear that there is a tiny difference between the two distributions.

Thanks and cheers, Maurizio.

[Beatrice]

So can it be said that the AUC tells us about a difference between 2 distributions, and the p value tells us how certain we are that this difference is not due to chance (regardless of its size) ?

[MAgojam]

So can it be said that the AUC tells us about a difference between 2 distributions, and the p value tells us how certain we are that this difference is not due to chance (regardless of its size) ?

We could also say so, but for the AUC I would prefer to keep the term of separation rather than difference.
So, I'm not a p-value fanatic and I'd rather report AUC with its (much more informative) confidence intervals, obviously along with the other important indicators resulting from the ROC curve.

I'd close the post and some links to scientific publications, if you have the time and desire to take a look.
https://journals.plos.org/plosone/artic ... ne.0056669
https://journals.plos.org/plosone/artic ... ne.0165165
https://journals.plos.org/plosone/artic ... ne.0193235

Thanks and cheers, Maurizio.

[Beatrice]

I did look at the articles. The statistical methods sections were very informative. I have a much better grasp of the concepts now.
Thank you very much.

[istanbulemdoc]

Hi, those are great and informative. I have a question extending the ones before:

How can we find the 95% Confidence interval of the AUC?

Thanks

[seol]

Hi istanbulemdoc

You can use ROC analysis within snowCluster module to find 95% CI of the AUC.

Kind Regards
Seol