Revisiting the replication crisis without false positives

Abstract

Efforts to replicate portions of the scientific literature have lead to widely varying and often low rates of replicability. This has raised concerns over a ``replication crisis’’ whereby many of the statistically significant claims in the published literature are thought to be false positives, due to some combination of publication bias and widespread use of questionable research practices. However, formal meta-scientific models invoking false positives lead to conclusions that often conflict with observational findings and require additional assumptions to reconcile varying rates of replicability across areas of research. Here, we present a minimal, alternative model of how replication failures can occur even in the absence of false positives. Using our model, we show that variation in estimates of replicability across social science is well explained as an artifact of replication sample size. We additionally demonstrate that key features of reformed science and multi-site replications can be explained without false positives. Our results are consistent with evidence suggesting that file-drawer sizes are likely much smaller, and Questionable Research Practices less abundant, than required by false positive models. We anticipate our findings will be a starting point for more formal and nuanced discussion of the health of the scientific literature and areas for improvement.