Definition: Analytic flexibility is a type of researcher degrees of freedom (Simmons, Nelson, & Simonsohn, 2011) that refers specifically to the large number of choices made during data preprocessing and statistical analysis. “[T]he range of analysis outcomes across different acceptable analysis methods” (Carp, 2012, p. 1). Analytic flexibility can be problematic, as this variability in analytic strategies can translate into variability in research outcomes, particularly when several strategies are applied, but not transparently reported (Masur, 2021).
Garden of forking paths,
Researcher degrees of freedom
References: Breznau et al. (2021), Carp (2012), Jones et al. (2020), Masur (2021), & Simmons et al. (2011)
Drafted and Reviewed by: Mariella Paul, Adrien Fillon, Bettina M. J . Kern, Adam Parker, Charlotte R. Pennington, Flávio Azevedo
Note that we are currently working on an automated mechanism to link references cited above with their full-length version that can be found at https://forrt.org/glossary/references with all references used so far.