Analytic Flexibility
Definition: Analytic flexibility is a type of researcher degrees of freedom (Simmons, Nelson, & Simonsohn, 2011) that refers specifically to the large number of choices made during data preprocessing and statistical analysis. “[T]he range of analysis outcomes across different acceptable analysis methods” (Carp, 2012, p. 1). Analytic flexibility can be problematic, as this variability in analytic strategies can translate into variability in research outcomes, particularly when several strategies are applied, but not transparently reported (Masur, 2021).
Related terms: Garden of forking paths, Multiverse analysis, Researcher degrees of freedom
References:
- Breznau, N. (2021). I saw you in the crowd: Credibility, reproducibility, and meta-utility. PS: Political Science & Politics, 54(2), 309–313. https://doi.org/10.1017/S1049096520000980
- Carp, J. (2012). On the plurality of (methodological) worlds: estimating the analytic flexibility of FMRI experiments. Frontiers in Neuroscience, 6, 149. https://doi.org/10.3389/fnins.2012.00149
- Jones, A., Dr, J., Duckworth, & Christiansen, P. (2020). May I have your attention, please? Methodological and Analytical Flexibility in the Addiction Stroop. https://doi.org/10.31234/osf.io/ws8xp
- Masur, P. K. (2020). Understanding the Effects of Analytical Choices on Finding the Privacy Paradox: A Specification Curve Analysis of Large-Scale Survey Data. Preprint. https://osf.io/m72gb/
- Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
Originally drafted by: Mariella Paul
Reviewed by: Adrien Fillon, Bettina M. J . Kern, Adam Parker, Charlotte R. Pennington, Flávio Azevedo