4 Pre-analysis Planning
5 sub-clusters · 42 referencesPre-analysis planning entails laying out a complete methodology and analysis before a study has been undertaken. This facilitates transparency and removes several potential QRPs. When teaching, students should attain knowledge regarding what a pre-registration entails, why it is important to remove potential QRPs and how to address deviations from preregistered plans. There are 5 sub-clusters which aim to further parse the learning and teaching process:
▸
Preregistration
14 / 14
When a researcher preregisters their work, they typically upload a detailed project plan before the start of data collection. This plan includes (but is not limited to) hypotheses, data collection procedures, measures and manipulated variables, and the analysis plan.
Merits and Limits of Preregistration for Visualization Research
This resource provides an overview of how preregistration can be integrated into visualization research, discussing both its potential to reduce questionable research practices and its practical challenges. It specifically addresses the nuances of applying open science standards to user studies and data visualization methodologies.
Evidence for prereg posters as a platform for preregistration
This study presents empirical evidence supporting the effectiveness of preregistration posters as a mechanism for improving research transparency and planning. It demonstrates how this specific format allows researchers to receive critical feedback on their designs before data collection, thereby strengthening the resulting study.
Centre for Open Science. (n.d.). Preregistration. Www.cos.io. https://www.cos.io/initiatives/prereg
DeHaven, A. (2017). 10 Tips for Making a Great Preregistration. www.cos.io. https://www.cos.io/blog/10-preregistration-tips
DeHaven, A. (2018). One Preregistration to Rule Them All? Www.cos.io. https://www.cos.io/blog/one-preregistration-rule-them-all
Preregistering qualitative research
This article explores how the concept of preregistration can be adapted for qualitative research by addressing the tension between data-driven interpretation and a priori planning. It provides a foundational framework for maintaining the inductive flexibility of qualitative work while leveraging preregistration to enhance research transparency and credibility.
Preregistering Qualitative Research: A Delphi Study
This study presents empirical findings from a Delphi consensus process aimed at determining which components of preregistration templates are most useful for qualitative researchers. It contributes evidence-based recommendations for designing flexible preregistration forms that accommodate the diverse methodological needs of the qualitative community.
Making the black box transparent: A template and tutorial for (pre-)registration of studies using Experience Sampling Methods (ESM)
This resource offers a specialized template and tutorial designed to guide researchers through the preregistration process for studies using Experience Sampling Methods (ESM) and Ecological Momentary Assessment (EMA). It aims to reduce the "black box" of researcher degrees of freedom by providing a structured framework for documenting complex longitudinal designs.
Campbell’s Law Explains the Replication Crisis: Pre-Registration Badges Are History Repeating
This article critiques the implementation of preregistration badges by framing them through the lens of Campbell’s Law, suggesting that high-stakes indicators can corrupt the very research processes they are meant to improve. It warns that mandating such practices may inadvertently prioritize the appearance of rigor over the actual quality of the science.
Preregistration of analyses of preexisting data
This resource addresses the unique challenges of applying preregistration to the analysis of preexisting data, a common practice in psychology that differs from original data collection. It makes the case for the utility of preregistering such studies to maintain transparency, even when researchers cannot control aspects like experimental manipulation or sample size.
The preregistration revolution
This paper argues for the fundamental importance of preregistration as a solution to the common human bias of mistaking postdiction for prediction. It advocates for the practice as a necessary step to protect the credibility of scientific findings by ensuring that research questions and analysis plans are defined before outcomes are observed.
Some Data Indicating that Editors and Reviewers Do Not Check Preregistrations during the Review Process
This empirical study investigates the behaviors of editors and reviewers by analyzing how often they engage with preregistration plans during the peer-review process. It contributes significant evidence of a 'compliance gap,' showing that even when studies are preregistered, the plans are rarely scrutinized during formal review.
Van den Akker, O. R., Weston, S., Campbell, L., Chopik, B., Damian, R., Davis-Kean, P., Hall, A., Kosie, J., Kruse, E., Olsen, J., Ritchie, S., Valentine, K., Van ’t Veer, A., & Bakker, M. (2021). Preregistration of secondary data analysis: A template and tutorial. Meta-Psychology, 5.
https://doi.org/10.15626/MP.2020.2625
Pre-registration in social psychology—A discussion and suggested template
This paper offers a specialized template and implementation guidelines for preregistration tailored to the methodological standards of social psychology. It discusses the relative advantages of reviewed and unreviewed preregistration models, providing researchers with actionable advice on how to structure their research plans for maximum transparency.
▸
Purpose of pre-analysis planning
10 / 10
Distinguishing exploratory and confirmatory analyses. One proposed benefit to this is transparency. FORRT does not propose one has more value over the other.
Making Prospective Registration of Observational Research a Reality
This article argues for the ethical and scientific necessity of prospectively registering observational studies in the health sciences, a domain where the practice is historically underutilized. It specifically identifies funders and ethics committees as the primary drivers for mandating the public disclosure of study protocols and analysis plans.
How can we make sound replication decisions?
This perspective piece introduces a conceptual framework to guide researchers and institutions in making strategic decisions about which findings should be prioritized for replication. It provides actionable criteria for weighing scientific values against practical constraints to ensure that limited research resources are allocated effectively.
Graf, C. (2017). 8 Answers About Registered Reports and Research Preregistration. Wiley Library. www.wiley.com. https://www.wiley.com/network/researchers/being-a-peer-reviewer/8-answers-about-registered-reports-research-preregistration-and-why-both-are-important
The Value of Preregistration for Psychological Science: A Conceptual Analysis
This article provides a philosophical defense of preregistration in psychological science by applying the principles of error statistics. It argues that preregistration is essential for maintaining the severity of a test, ensuring that the probability of finding support for a theory is not artificially inflated by post-hoc analytical choices.
Lin, W., & Green, D. P. (2016). Standard Operating Procedures: A Safety Net for Pre-Analysis Plans. Political Science and Politics, 49(3), 495–500.
https://doi.org/10.1017/S1049096516000810
Registered Reports
This resource introduces the Registered Reports publishing model, which shifts peer review to the stage before data collection to combat publication bias and the file-drawer problem. It presents a collection of social psychology replications as evidence of how this format successfully prioritizes methodological rigor and open science practices over the direction of the results.
The preregistration revolution
This paper argues for the fundamental importance of preregistration as a solution to the common human bias of mistaking postdiction for prediction. It advocates for the practice as a necessary step to protect the credibility of scientific findings by ensuring that research questions and analysis plans are defined before outcomes are observed.
How scientists fool themselves – and how they can stop
This resource explores the psychological factors that lead researchers to see patterns in noise and provides an overview of how cognitive biases affect data analysis. It introduces several preventative practices, including the use of blind analysis and the adoption of more formal pre-analysis planning to mitigate the risk of false discoveries.
Wagenmakers, E. J., & Dutilh, G. (2016). Seven selfish reasons for preregistration. APS Observer, 29(9). https://www.psychologicalscience.org/observer/seven-selfish-reasons-for-preregistration
Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking
This resource provides an extensive checklist of 34 specific researcher degrees of freedom that can lead to p-hacking across various stages of the research process. It serves as a practical tool for psychologists to preemptively identify and minimize opportunistic choices during study planning, data collection, analysis, and reporting.
▸
Registered reports
10 / 10
A publishing format consisting of a preregistration with a specific journal that undergoes peer review. Specifically, the journal reviews the introduction and methodology and upon in-principle acceptance (IPA, in stage 1), it agrees to publish the study assuming the preregistration is followed and deviations are reported (stage 2 acceptance).
Registered Replication Report
This resource provides empirical data from a large-scale multi-lab replication effort to estimate the true effect size of the verbal overshadowing phenomenon. It addresses discrepancies between original findings and subsequent research by using a pre-registered, standardized protocol across multiple sites to ensure a high-powered and unbiased assessment.
Anon. (n.d.). Wiley’s Reviewing Registered Reports. https://authorservices.wiley.com/Reviewers/journal-reviewers/how-to-perform-a-peer-review/reviewing-registered-reports.html
Registered Reports: A new publishing initiative at Cortex
This article introduces the "Registered Report" article format at the journal Cortex, explaining the policy of peer-reviewing research protocols before data collection. It details how this editorial shift aims to prioritize the quality of research questions and methods over the statistical significance of the results.
Registered Reports: Realigning incentives in scientific publishing
This foundational paper argues for Registered Reports as a mechanism to eliminate publication bias and p-hacking by decoupling the decision to publish from the study results. It makes the case for realigning scientific incentives so that professional success is determined by the quality of research questions and methods rather than the direction of findings.
D. Chambers, C., Feredoes, E., D. Muthukumaraswamy, S., & J. Etchells, P. (2014). Instead of “playing the game” it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond. AIMS Neuroscience, 1(1), 4–17.
https://doi.org/10.3934/Neuroscience.2014.1.4
Registered Reports: Realigning incentives in scientific publishing
This foundational paper argues for Registered Reports as a mechanism to eliminate publication bias and p-hacking by decoupling the decision to publish from the study results. It makes the case for realigning scientific incentives so that professional success is determined by the quality of research questions and methods rather than the direction of findings.
The past, present and future of Registered Reports
This article provides a comprehensive overview of the Registered Reports publication model, tracing its historical development and evaluating its effectiveness in mitigating publication bias. It serves as both a theoretical reflection on the format's impact and a practical guide for researchers, editors, and reviewers navigating the pre-acceptance process.
Registered Reports: A new chapter at <i>Ecology & Evolution</i>
This article provides a walkthrough of the Registered Report process within the field of ecology and evolution by documenting the experiences of an editor, author, and student. It serves as a practical guide for researchers in these disciplines to understand the logistical and pedagogical implications of adopting pre-accepted study protocols.
Registered reports for qualitative research
This resource advocates for the adoption of the Registered Reports publishing format within qualitative research to reduce publication bias and increase the visibility of non-quantitative studies. It explores how the model of pre-data collection peer review can be effectively tailored to support the rigor of qualitative research designs.
Zotero. (n.d.). OSF Collection of Registered Reports. Zotero. https://www.zotero.org/groups/479248/osf/collections/KEJP68G9/search/right/titleCreatorYear
▸
Study Design
1 / 1
Design elements should be before data collection so that confirmatory claims are credible and deviations are interpretable. This includes choosing appropriate sample sizes, conditions, and measures prior to registration, and planning how to handle deviations
When and How to Deviate from a Preregistration
This resource offers a principled approach to handling changes made after a study has been preregistered, focusing on how to report these deviations without undermining research integrity. It provides specific examples of how to document deviations and evaluate their impact on the severity of statistical tests and the validity of scientific inferences.
▸
Understanding the types of preregistration and writing one.
7 / 7
Practical materials and resources to move beyond the theory of preregistration, into conducting a preregistered study.
Call, M., DeHaven, A. C., Pfeiffer, N., Mellor, D. T., & Lowrey, O. (2023, October 19). Templates of OSF registration forms. OSF. https://osf.io/zab38/
Centre for Open Science. (n.d.). Registered Reports: Peer review before results are known to align scientific values and practices. Www.cos.io. https://www.cos.io/initiatives/registered-reports
When and How to Deviate from a Preregistration
This resource offers a principled approach to handling changes made after a study has been preregistered, focusing on how to report these deviations without undermining research integrity. It provides specific examples of how to document deviations and evaluate their impact on the severity of statistical tests and the validity of scientific inferences.
Preregistration of analyses of preexisting data
This resource addresses the unique challenges of applying preregistration to the analysis of preexisting data, a common practice in psychology that differs from original data collection. It makes the case for the utility of preregistering such studies to maintain transparency, even when researchers cannot control aspects like experimental manipulation or sample size.
The preregistration revolution
This paper argues for the fundamental importance of preregistration as a solution to the common human bias of mistaking postdiction for prediction. It advocates for the practice as a necessary step to protect the credibility of scientific findings by ensuring that research questions and analysis plans are defined before outcomes are observed.
Pre-registration and Registered Reports: a Primer from UKRN
This primer serves as an introductory guide to preregistration and Registered Reports, explaining how these practices help mitigate selective reporting and questionable research practices. It provides a foundational overview for researchers looking to incorporate these transparency-enhancing methodologies into their standard workflow.
Increasing the transparency of systematic reviews: presenting a generalized registration form
This resource provides a comprehensive and flexible registration form designed to enhance transparency in systematic reviews regardless of the discipline or review type. By prioritizing broad applicability over niche requirements, it allows researchers in diverse fields like economics, law, or physics to document their protocols effectively even when standard forms are not suitable for their specific data types.
