Framework for
Open and
Reproducible
Research
Training

Logo of FORRT is a fort.

Publications, Preprints, Policy Briefs & op-eds




Academic peer-reviewed publications



The Impact of Open Scholarship on Students’ Learning Outcomes (Scoping Review)


We have written a manuscript entitled Teaching open and reproducible scholarship: a critical review of the evidence base for current pedagogical methods and their outcomes. This manuscript describes the available (empirical) evidence of the impact (and importance) of integrating open scholarship into higher education, its benefits and challenges on three specific areas: students’ (a) scientific literacy; (b) engagement with science; and (c) attitudes towards science. This paper was borne out of colleagues reporting a need for something tangible to take to Heads of Departments to show pedagogical rationale for open scholarship (above and beyond the moral/‘good for science’ case that wasn’t always being heard on the ground). Cite as:


Pownall, M., Azevedo, F., König, L. M., Slack, H. R., Evans, T. R., Flack, Z., … & FORRT. (2023). Teaching open and reproducible scholarship: a critical review of the evidence base for current pedagogical methods and their outcomes. Royal Society Open Science, 10(5), Article 221255. https://doi.org/10.1098/rsos.221255


The preprint can be found here (osf.io/9e526).

________________________________





Improvements since the Replication Crisis: The Structural, Procedural, and Community Changes


We have written a manuscript entitled The replication crisis has led to positive structural, procedural, and community changes. This manuscript reviews how research structures, procedures and communities have changed in response to the replication crisis, showing that the replication “crisis” has been a positive credibility revolution. We also outline how improvements can continue to be made by accounting for all three levels. This manuscript was born out of the fact that much of the literature on the Replication Crisis has focused on understanding the potential underlying causes of low replicability, and framing solutions as a matter of individual researcher reform, and our community wanted to provide a wider perspective on what has been learnt since the crisis by highlighting the role of structural, procedural factors, and evidencing the rapid progress made by social scientists as a response. Cite as:


Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., Yu, S., Baker, B. J., O’Mahony, A., Olsnes, J. Ø.-S., Shaw, J. J., Gjoneska, B., Yamada, Y., Röer, J. P., Murphy, J., Alzahawi, S., Grinschgl, S., Oliveira, C. M., Wingen, T., Yeung, S. K., Liu, M., König, L. M., Albayrak-Aydemir, N., Lecuona, O., Micheli, L., & Evans, T. (2023). The replication crisis has led to positive structural, procedural, and community changes. Communications Psychology, 1, 3. https://doi.org/10.1038/s44271-023-00003-2.

The preprint can be found here (osf.io/r6cvx).

________________________________





Participatory Research Primer


We have written a manuscript entitled Opening up understanding of neurodiversity: A call for applying participatory and open scholarship practices. This manuscript gives a brief overview of what participatory research methods are and why they are important for promoting neurodiversity in academia. Cite as:


Gourdon-Kanhukamwe, A., Kalandadze, T., Yeung, S., Azevedo, F., Iley, B. J., Phan, J. M., … Elsherif, M. M. (2023). Opening up understanding of neurodiversity: A call for applying participatory and open scholarship practices. The Cognitive Psychology Bulletin, 8, 23-27. https://doi.org/10.53841/bpscog.2023.1.8.23


The preprint can be found here (osf.io/jq23s).

________________________________





FORRT’s Glossary


In response to the varied and plural new terminology introduced by the open scholarship movement, which has transformed academia’s lexicon, we have produced a community and consensus-based Glossary to facilitate education and effective communication between experts and newcomers. FORRT members have written a manuscript entitled A Community-Sourced Glossary of Open Scholarship Terms presenting the beta 0.1 version of our glossary of open scholarship terms. Cite as:


Parsons, S., Azevedo, F., Elsherif, M. M., Guay, S., Shahim, O. N., Govaart, G. H., Norris, E., O’Mahony, A., Parker, A. J., Todorovic, A., Pennington, C. R., Garcia-Pelegrin, E., Lazić, A., Robertson, O. M., Middleton, S. L., Valentini, B., McCuaig, J., Baker, B. J., Collins, E., … Aczel, B. (2022). A Community-Sourced Glossary of Open Scholarship Terms. Nature Human Behaviour, 6(3), 312-318. https://doi.org/10.1038/s41562-021-01269-4.


The preprint can be found here (osf.io/kdqcw) and here is the copy-edited version at the publisher’s website.

________________________________





The Role of Pedagogical Communities


We have written a manuscript entitled Towards a culture of open scholarship: The role of pedagogical communities describing (a) the the need to integrate open scholarship principles into research training within higher education; (b) the benefit of pedagogical communities and the role they play in fostering an inclusive culture of open scholarship; and (c) call for greater collaboration with pedagogical communities, paving the way for a much needed integration of top-down and grassroot open scholarship initiatives. Cite as:


Azevedo, F., Liu, M., Pennington, C. R., Pownall, M., Evans, T. R., Parsons, S., Elsherif, M. M., Micheli, L., Westwood, S., & FORRT. (2021). Towards a culture of open scholarship: The role of pedagogical communities. BMC Research Notes, 15(1), 1-5. https://doi.org/10.1186/s13104-022-05944-1


________________________________





FORRT’s Lesson Plans


Despite the growing awareness of the benefits of training reproducible methods within undergraduate and postgraduate curricula, there remains a lack of systematic incorporation of open scholarship practices in taught courses across Higher Education. While the potential reasons for this are diverse, one reason may be the dearth of available ready-to-use educational resources. To support instructors aiming to bridge that gap, FORRT’s community built open educational resources that can be integrated into taught courses ‘out of the box’. As creating or changing course content can be onerous and time-consuming, we aimed to make evidence-based, high-quality lesson plans and activities available to teaching faculty, thus reducing the labour required to develop and implement open scholarship content. We compiled lesson plans and activities, and categorized them based on their theme, learning outcome, and method of delivery, which are made publicly available here: FORRT’s Lesson Plans. Cite as:

Pownall, M., Azevedo, F., Aldoh, A., Elsherif, M. M., Vasilev, M. R., Pennington, C. R., … Parsons, S. (2021). Embedding open and reproducible science into teaching: A bank of lesson plans and resources. Scholarship of Teaching and Learning in Psychology. Advance online publication. https://doi.org/10.1037/stl0000307.

The preprint can be found here (osf.io/fgv79).





Preprints


The Replication Database: Documenting the Replicability of Psychological Science


We have written a manuscript about documenting and tracking replication efforts. In psychological science, replicability — repeating a study with a new sample achieving consistent results — is critical for affirming the validity of scientific findings. Despite its importance, replication efforts are few and far between in psychological science with many attempts failing to corroborate past findings. This scarcity, compounded by the difficulty in accessing replication data, jeopardizes the efficient allocation of research resources and impedes scientific advancement. Addressing this crucial gap, we present the Replication Database, a novel platform hosting >1,200 original findings paired with replication findings. The infrastructure of this database allows researchers to submit, access, and engage with replication findings. The database makes replications visible, easily findable via a graphical user interface, and tracks replication rates across various factors, such as publication year or journal. This will facilitate future efforts to evaluate the robustness of psychological research. More info about the project and other FORRT replication efforts can be found on the Replication Hub. This effort was also highlighted by the journal Science in their news at a glance!

Status: Our paper is preprinted in MetaArxiv. Cite as:

Röseler, L., Kaiser, L., Doetsch, C. A., Klett, N., Seida, C., Schütz, A., … Zhang, Y., Mr. (2024, April 11). The Replication Database: Documenting the Replicability of Psychological Science. https://doi.org/10.31222/osf.io/me2ub

________________________________

Neurodiversity & Open Scholarship in Academia


We have written a manuscript entitled Bridging Neurodiversity and Open Scholarship: How Shared Values Can Guide Best Practices for Research Integrity, Social Justice, and Principled Education explaining that Neurodiversity is fundamental to the understanding of human behaviour and cognition and that neurodivergent individuals are often stigmatised, devalued, and objectified. The authors describe systematic barriers, issue with disclosure, directions on prevalence and stigma, intersection of neurodiversity and open scholarship, and provide recommendations that can lead to personal and systematic changes to improve acceptance of neurodivergent individuals. We also present the perspectives of neurodivergent authors, the majority of whom have personal lived experiences of neurodivergence(s), and provide recommendations for how research and academia can and should be improved in terms of research integrity, inclusivity and diversity. We end with discussing systematic barriers, issue with disclosure, directions on prevalence and stigma, intersection of neurodiversity and open scholarship, and provide recommendations that can lead to personal and systematic changes to improve acceptance of neurodivergent individuals.

Status: Our paper is preprinted 🎉🥳 in MetaArxiv. Cite as:

Elsherif, M. M., Middleton, S. L., Phan, J. M., Azevedo, F., Iley, B. J., Grose-Hodge, M., … Dokovova, M. (2022, June 20). Bridging Neurodiversity and Open Scholarship: How Shared Values Can Guide Best Practices for Research Integrity, Social Justice, and Principled Education. https://doi.org/10.31222/osf.io/k7a9p.

________________________________





FORRT’s Manifesto


We have written a manuscript entitled Introducing a Framework for Open and Reproducible Research Training (FORRT) describing the importance of integrating open scholarship into higher education, its benefits and challenges, as well as about FORRT initiatives aiming to support educators in this endeavor.

The preprint can be found here (osf.io/bnh7p).

Status: Our manuscript is currently under review. Cite as:

Azevedo, F., Parsons, S., Micheli, L., Strand, J., Rinke, E., … & FORRT (2019, December 13). Introducing a Framework for Open and Reproducible Research Training (FORRT). https://doi.org/10.31219/osf.io/bnh7p





op-eds



📜 For The Psychologist, on open scholarship pedagogical communities.

Azevedo, F. (2023). Open scholarship pedagogical communities. The Psychologist, British Psychological Society.

📄 For the Observer, of the Association for Psychological Science, on Navigating Academia as Neurodivergent Researchers.

Azevedo, F., Middleton, S., Mai Phan, J., Kapp, S., Gourdon-Kanhukamwe, A., Iley, B., Elsherif, M., & Shaw, J. J. (2022). Navigating Academia as Neurodivergent Researchers: Promoting Neurodiversity Within Open Scholarship. APS Observer.

📃 For In-Mind Magazine, on What is Open Science and Why Does It Need a Glossary.

Azevedo, F., Baum, M., Hartmann, H., Micheli, L., Spitzer, L., & Wingen, T. (2022). What is “Open Science”? And why does it need a glossary? (Was ist „Offene Wissenschaft”? Und warum braucht sie ein Wörterbuch?). In-Mind Magazine.

📜 For the Center of Open Science Blog, on Integrating Open and Reproducible Science Principles into Higher Education.

Azevedo, F. (2022). Educating in the Open Scholarship Era: Integrating Open and Reproducible Science Principles into Higher Education. Center for Open Science (COS).



Policy Briefs


Policy | Open Scholarship in QAA Briefing


FORRT was asked by the UK’s Quality Assurance Agency for Higher Education (QAA) to produce a short briefing note in collaboration with the UK Reproducibility Network (UKRN) about the importance of embedding open scholarship into higher education. In the note, we argued that students need to understand and experience open and rigorous research processes to become effective and responsible consumers of evidence. We structured this around three themes (Thinking About Research, Doing Research, and Engaging With Research) and showed how this can be implemented across different fields.

The briefing note can be found here.

Status: Our briefing note is being used for QAA’s work around standards and quality in Higher Education. Cite as:

Azevedo, F., DeBruine, L., Evans, T. R. (2023). Open Scholarship in QAA Briefing. Framework for Open and Reproducible Research Training & UK Reproducibility Network. https://forrt.org/publications


________________________________

Policy | Building Community


FORRT responded to the the call from the UK House of Commons Science and Technology Committee on for evidence on reproducibility and research integrity, and the roles different institutions play in this arguing that one important component of research integrity that is often absent from discussion: the pedagogical consequences of how we teach, mentor, and supervise students through open scholarship. Our argument boils down to ‘If you want to go fast, go alone. If you want to go far, go together.’

The submission can be found here.

Status: Our submission is currently being considered by the UK’s Parliament Science and Technology Committee. Cite as:

Azevedo, F., Liu, M., Pennington, C. R., Pownall, M., Evans, T. R., Parsons, S., Elsherif, M. M., Micheli, L., Moreau, D., & FORRT. (2021). Towards a culture of open scholarship: The role of pedagogical communities. (Report No. RRE0080). Written evidence on reproducibility and research integrity. UK Parliament. [ UK Parliament, PDF, html]


________________________________

Policy | Briefing Note for Degree Accrediting Societies


We have written to UKRN (UK’s Reproducibility Network) and the British Psychological Society (BPS) a short note on our organization and goals, as well as providing 3 (three) recommendations:

  1. We ask that degree accrediting societies and higher education institutions endorse FORRT as a valuable tool for educators. Both as an educational self-assessment of current teaching content, and as a resource for successful implemented pedagogies.
  2. We ask that degree accrediting societies and higher education institutions support and promote the opening of teaching materials, to support educators to integrate principles of open and reproducible research in their own teaching and mentoring materials.
  3. We ask that representatives from degree accrediting societies and higher education institutions meet with FORRT to discuss partnerships and shared educational aspirations with FORRT.

The briefing note can be found here.

Status: Our briefing note was informative to UKRN & the British Psychological Society. Cite as:

Parsons, S., Azevedo, F., & FORRT (2020). Briefing note for degree accrediting societies. Framework for Open and Reproducible Research Training. https://forrt.org/publications


________________________________

Ongoing Projects


We are currently preparing additional manuscripts on a range of topics. To find out more about what we’re working on and how you can contribute, visit our Get Involved page.


________________________________

Stay updated

Subscribe to our newsletter from FORRT to get the latest news about our initiatives.