p-Hacking: Its Costs and When It Is Warranted

Abstract

p-Hacking, the use of analytic techniques that may lead to distorted research results, is widely condemned on epistemic and practical grounds. The prevalent position on this questionable research practice is that p-hacking should be avoided because it raises the probability of obtaining false-positive results, which can have harmful practical consequences. I have three aims in this paper. First, I offer a precise definition of p-hacking, something sorely needed in discussions of the practice. Second, I use philosophical tools from decision theory to articulate the prevalent position on p-hacking and demonstrate its flaws. While p-hacking can have epistemic and practical costs, a more nuanced approach to its consequences is necessary. Third, I argue that there are scenarios in which p-hacking can be warranted. The prevalent position neglects key factors in its overall evaluation of the consequences of p-hacking. Moreover, it disregards considerations that can lend support to decisions about analytic choices that amount to p-hacking. One important, but often neglected consideration is that, while p-hacking may lead to more false-positives, it also increases the chances of uncovering true-positives. I appeal to non-epistemic judgments to defend the view that there are situations in which p-hacking may be warranted.