Claims about scientific rigour require rigour

Abstract

Protzko et al.1 describe a project in which internal tests of pilot-tested hypotheses and independent replications embraced “rigour-enhancing practices” such as confirmatory tests, large sample sizes, preregistration and methodological transparency. The authors report a high estimate of replicability, which, in their appraisal, “justifies confidence in rigour-enhancing methods to increase the replicability of new discoveries” (see Abstract). However, replicability was not the original outcome of interest in the project, and analyses associated with replicability were not preregistered as claimed. Instead of replicability, the originally planned study set out to examine whether the mere act of scientifically investigating a phenomenon (data collection or analysis) could cause effect sizes to decline on subsequent investigation ( https://osf.io/ba8p7). This “decline effect” hypothesis, posited by one of the authors and not articulated in the published manuscript, invokes phenomena that, if found, could revise the “laws of reality”2. The project did not yield support for this preregistered hypothesis; the preregistered analyses on the decline effect and the resulting null findings were largely relegated to the supplement, and the published article instead focused on replicability, with a set of non-preregistered measures and analyses, despite claims to the contrary.