Yhyjyk

A colleague has recently made the claim that a large p-value was not more support for the null hypothesis than a low one. Of course, this is also what I learned (uniform distribution under the null hypothesis, we can only reject the null hypothesis...). But when I simulate two random normal distributions (100 samples in each group) in R, my p-value is correlated to the difference (averaged over 30 repetitions) between the two means (with for example a T test or a Mann & Whitney test).

Why is my p-value, above the threshold of 0.05, correlated to the difference between the means of my two groups?

With 1000 repetitions for each x (difference between means/2) value.

My R code in case this is just a silly mistake.

pvaluetot<-NULL
xtot<-NULL
seqx<-seq(0,5,0.01)
for (x in seqx)
 ptemp<-NULL
 pmean<-NULL
 a<-0

 repeat
 a<-a+1
 pop1<-rnorm(100,0+x,2)
 pop2<-rnorm(100,0-x,2)
 pvalue<-t.test(pop1,pop2)$p.value

 ptemp<-c(ptemp,pvalue)
 #print(ptemp)
 if (a==30)
 break
 

 pmean<-mean(ptemp)
 pvaluetot<-c(pvaluetot,pmean)
 xtot<-c(xtot,x)
 print(x)


pvaluetot
xtot
plot(pvaluetot,xtot)

Why would you expect anything else? You don't need a simulation to know this is going to happen. Look at the formula for the t-statistic:
$t = fracbarx_1 - barx_2 sqrt fracs^2_1n_1 + fracs^2_2n_2 $

Obviously if you increase the true difference of means you expect $barx_1 - barx_2$ will be larger. You are holding the variance and sample size constant, so the t-statistic must be larger and thus the p-value smaller.

I think you are confusing a philosophical rule about hypothesis testing with a mathematical fact. If the null hypothesis is true, you would expect a higher p-value. This has to be true in order for hypothesis testing to make any sense.

As you said, the p-value is uniformly distributed under the null hypothesis. That is, if the null hypothesis is really true, then upon repeated experiments we expect to find a fully random, flat distribution of p-values between [0, 1]. Consequently, a frequentist p-value says nothing about how likely the null hypothesis is to be true, since any p-value is equally probable under the null.

What you're looking at is the distribution of p-values under an alternative hypothesis. Depending on the formulation of this hypothesis, the resulting p-values can have any non-Uniform, positively skewed distribution between [0, 1]. But this doesn't tell you anything about the probability of the null. The reason is that the p-value expresses the probability of the evidence under the null hypothesis, i.e. $p(D|H_0)$, whereas you want to know $p(H_0|D)$. These two are related by Bayes' rule:
$$
p(H_0|D) = fracH_0)p(H_0)p(D
$$
This means that in order to calculate the probability you're interested in, you need to know and take into account the prior probability of the null being true ($p(H_0)$), the prior probability of the null being false ($p(neg H_0)$) and the probability of the data given that the null is false ($p(D|neg H_0)$). This is the purview of Bayesian, rather than frequentist statistics.

As for the correlation you observed: as I said above the p-values will be positively skewed under the alternative hypothesis. How skewed depends what that alternative hypothesis is. In the case of a two-sample t-test, the more you increase the difference between your population means, the more skewed the p-values will become. This reflects the fact that you're making your samples increasingly more different from what is plausible under the null, and so by definition the resulting p-values (reflecting the probability of the data under the null) must decrease.