P-rep or is a statistical alternative to the classic p-value. Whereas a p-value indicates the probability of obtaining a result by chance alone, p-rep estimates the probability of replicating an effect. The Association for Psychological Science now recommends that articles submitted to Psychological Science and their other journals report p-rep rather than the classic p-value.
The value of p-rep can be approximated based on the p-value using the following equation:
The mere fact that the p-rep has a one-to-one correspondence with the p-value makes it clear that this new measure doesn't bring any additional information on the significance of the result of a given experiment. However, according to Killeen who acknowledges this latter point, the main advantage of p-rep lies in the fact that it better captures the way experimenters naively think and conceptualize p-values and Null hypothesis statistical testing.
Among the criticisms of p-rep is the fact that it does not take prior probabilities into account (Macdonald, R. R. Psychological Science, 2005, 16, 1006-1008). For example, if an experiment on some unlikely paranormal phenomenon produced a p-rep of .75, most right-thinking people would not believe the probability of a replication is .75. Instead they would conclude that it is much closer to .50. Extraordinary claims require extraordinary evidence, and p-rep ignores this. This consideration undermines the argument that p-rep is easier to understand than a classical p value. The fact that p-rep requires assumptions about prior probabilities for it to be valid makes its interpretation complex. The classical p merely states the probability of an outcome (or more extreme outcome) given a null hypothesis and therefore is valid without regard to prior probabilities.
Critics have also underscored mathematical errors in the original paper by Killeen. For example, the formula relating the effect sizes from two replications of a given experiment erroneously use one of these random variables as a parameter of the probability distribution of the other while he previously hypothesized these two variables to be independent.
- Killeen PR (2005). "An alternative to null-hypothesis significance tests". Psychological science : a journal of the American Psychological Society / APS. 16 (5): 345–53. doi:10.1111/j.0956-7976.2005.01538.x. PMID 15869691.
- Excel spreadsheets for calculation of p-rep