Educators' Corner | FORRT - Framework for Open and Reproducible Research Training

Meaningful results for meaningful hypotheses

Mon, 16 Feb 2026 00:00:00 -0300

Stop asking if an effect is “exactly zero” for experimental designs. It’s not a helpful question. Instead, concentrate on interpreting the actual meaning of the difference between two conditions using Bayesian inference.

How to decide if an effect is meaningful in Bayesian inference?

After years of null hypothesis significance testing (NHST), you might have finally made the switch to Bayesian inference. You set priors, run the models you want to run, and estimate posterior distributions. Great! But what now? When writing up their Bayesian results, people often wonder how to decide whether results are meaningful or not. We have been there ourselves.

Nowadays, more and more researchers move away from NHST toward Bayesian inference (e.g. van der Schoot et al. 2017) and are faced with the same conceptual hurdle because decision procedures are not an inherent part of the Bayesian workflow. There are several approaches to hypothesis testing within the Bayesian framework, of course. Unfortunately, many of them, such as various forms of calculating the Bayes Factor, are either conceptually challenging, computationally (too) costly, or both.

A computationally feasible and intuitive way

In our new tutorial, we walk you through a lesser known form of Bayes Factor calculation using the Savage-Dickey approximation ( Dickey & Lientz 1970). This form of inference is not only conceptually intuitive and computationally efficient, but it also allows us to tackle another conceptual issue with traditional hypothesis testing: Researchers coming from NHST tend to test whether differences between conditions are smaller or greater than exactly zero (i.e. testing point-0 hypotheses). But not every difference that is not zero is meaningful for either theoretical or practical purposes. In our tutorial, we use a data set on pitch perception by Korean speakers in formal and informal contexts. The researchers want to know if speakers use meaningfully higher or lower pitch in these two social contexts.

To test this hypothesis, we suggest the following workflow:

We define the smallest effect size of interest (SESOI, see Lakens 2017) and implement it as a region of practical equivalence (ROPE, see Kruschke 2018). Defining a SESOI is not easy and requires a good quantitative understanding of the phenomena. In our example, we define the ROPE in terms of the acoustic differences that lead to a certain accuracy performance in a forced-choice perception task (the so-called just-noticeable-difference). The simplified idea is that if you cannot reliably hear a certain acoustic difference, then the difference should not matter for communication.
For the difference between contexts, we quantify the proportion of posterior samples inside of the ROPE (i.e. the posteriors are practically equivalent to 0, indicating no meaningful results) relative to the proportion of posteriors outside of the ROPE (i.e. the posteriors represent meaningful differences). We do this for both the posterior distributions before observing the data (i.e., priors only) and after observing the data (i.e., priors combined with the likelihood).
By comparing these probabilities to each other, we can calculate how the evidence for a meaningful difference shifts when observing the data. This shift is the Bayes Factor, that lets us quantify evidence for and against a meaningful pitch difference between contexts.

The proposed workflow is not new ( Wagenmakers et al. 2010), but unfortunately not widely known across the behavioral sciences. This computationally lightweight decision procedure is conceptually intuitive and encourages us to critically reflect on what it means for an effect to be truly meaningful. Win-win.

Where to start if you are new to Bayesian inference?

Our tutorial is basically a long awaited 2nd sequel (!?) of Bodo Winter’s original introduction to linear mixed effects models from 2013. This tutorial was widely shared and cited because it was one of the first very accessible introductions to linear mixed effects models within NHST. Six years later, we wrote a sequel that used the original example problem and demonstrated how to implement it into a Bayesian workflow. Now, another 6 years later, we thought it was time to drop Part 3, sticking to the same phenomenon and data set as the original, finally bringing the trilogy to a “meaningful” conclusion.

Contact information

Timo B. Roettger
Email: timo.roettger@iln.uio.no
Bluesky: @timoroettger.bsky.social
LinkedIn: linkedin.com/in/timo-b-roettger/
Website: timo-b-roettger.github.io

Michael Franke
Email: mchfranke@gmail.com
Bluesky: @meanwhileina.bsky.social
Website: michael-franke.github.io/heimseite/

FORWARD: A UNESCO Decade of Science Programme

Thu, 11 Dec 2025 15:30:39 -0300

FORWARD (FORRT for World Development)

Sustainable Development Goals Addressed

In a landmark moment for global scientific equity, FORRT is one of the few initiatives worldwide to be formally endorsed as a Programme of the International Decade of Sciences for Sustainable Development (IDSSD, 2024–2033), led by UNESCO and the United Nations (UN). This endorsement reflects the highest level of UN policy recognition and positions FORRT’s work in Education & Pedagogy (Curriculum Hub), Metascience & Research (Replication Hub) and Social Justice & DEIA as an essential contribution to the UN’s strategy for achieving the Sustainable Development Goals (SDGs). Specifically, FORRT for World Development (FORWARD), is recognized as critical infrastructure for advancing Outcome 4 (widely and equitably practiced open science) and Outcome 1 (a globally science-literate society) of the IDSSD. Through this work, FORRT facilitates the international research community to meet its core commitments to rigor, transparency, and global equity while delivering a roadmap for democratizing science worldwide.

One of the few programmes of the IDSSD to focus on how research is done, FORWARD provides a concrete, multi-year, and community-driven framework for delivering on the mandate of the UNESCO Science Decade while feeding into the work of the other Programmes and beyond. As an endorsed Programme, we are committed to accountability and open knowledge sharing, ensuring that all outputs are openly accessible and utilized by the global community. The resources developed over the next 8 years will serve as policy tools for the Decade’s Secretariat and other high-level stakeholders to monitor and track progress on equity and quality in science. We are directly contributing to realizing the Decade’s vision of an inclusive and effective global science culture by 2033, with the ability to contribute policy recommendations to the World Science Forum, the UN Science Summit, the Global Research Council, the European Commission, and more.

The FORRT community is already hard at work producing tools, guidelines, recommendations and materials to democratize science. We invite aligned organizations and strategic national and international partners to join us and invest in FORRT’s mission, thereby directly supporting their national commitments to the UN Science Decade through our validated global roadmap.

Read the Official Announcement of FORWARD by the UNESCO Science Decade

Quantifying reproducibility: Lessons from an iRISE-FORRT collaborative project

Wed, 16 Jul 2025 15:30:39 -0300

Across research fields, initiatives and calls have emerged to improve or ensure the reproducibility of research studies and findings. Ideally, research should be designed and reported in ways that allow an independent team of researchers to replicate a study and arrive at consistent results.

But how exactly can we quantify reproducibility? And how can we assess whether the findings of a replication align with the original?

Understanding how reproducibility is measured matters not just for the individual researcher aiming to improve their own work, but also for meta-researchers who study research processes and results, and institutions, funders and science policymakers who want to assess research quality in meaningful ways. This was the motivation behind a recent collaboration between members of the iRISE project and the FORRT community. The results are now published in Royal Society Open Science [1].

The iRISE project

The iRISE (improving Reproducibility In SciencE) project, funded through the Horizon Europe WIDERA call “Increasing the Reproducibility of Scientific Results” (WIDERA-2022-ERA-01-41), explores the causes of poor reproducibility and evaluates – including primary research – the effectiveness of interventions to increase reproducibility.

Even before the start of the project, we noticed that terms like reproducibility, replicability and replication are often used inconsistently across research fields and groups. What one group calls for example reproducibility, another might label replicability. To support clear communication and a shared understanding within the consortium, we developed the iRISE glossary [2] with working definitions of the most relevant terms.

A natural next step was to investigate how different aspects of reproducibility can be measured and quantified.

A collaborative review of reproducibility metrics

To explore the question of how reproducibility is and should be measured, iRISE partners teamed up with members of the FORRT community to conduct a scoping review. Joris Frese, Helena Hartmann and Louise Townsin joined the team after a call for contributors was posted in the FORRT Slack channel. Our review’s goal was to identify and categorise metrics used to quantify reproducibility in large-scale replication projects and metrics that were suggested or discussed in the methodological literature.

Following a preregistered protocol [3], the team identified 49 large-scale replication projects and 97 methodological papers. A total of 50 reproducibility metrics were extracted from the included records. These ranged from statistical formulas and models (74%) to graphical representations (6%), algorithms (6%), frameworks (6%), and studies and questionnaires (8%). All metrics are summarised in Table 4 of our paper. An extended and interactive version of the table was also integrated in a small static website ( http://rachelhey.github.io/reproducibility_metrics/) which allows for easy searches and filtering of the table.

What we learned

We were surprised by the number and diversity of the identified reproducibility metrics, which reflects how complex it is to measure reproducibility. A few key takeaways stood out:

There is no single “best” reproducibility metric. Different metrics capture different aspects and serve different purposes.
Many large-scale replication projects did not clearly define what they mean by reproducibility, or how they planned to measure it.
The choice of a reproducibility metric(s) should depend on the study’s goals and research questions. It should further inform the design of the study and researchers should transparently describe their choices.

We hope that our reproducibility metrics table will be a practical tool for future replication teams. It offers guidance on which metric fits which research question, and what data inputs are required. As a next step, we plan to systematically evaluate the real-world applicability of the identified metrics, with particular attention to their underlying assumptions and limitations. To support this analysis, we plan to use real-world data examples from the FORRT Replication Database.

EDI considerations

We also explored how often dimensions related to equity, diversity and inclusion (EDI) were explicitly mentioned or addressed in the included literature. Unsurprisingly, only a few papers mentioned EDI dimensions, which highlights the need to make these considerations more explicit in future replication efforts. Within iRISE, EDI and research culture are not an afterthought; they are embedded by design into all our activities. If you are interested in other EDI efforts within iRISE, feel welcome to contact Stephanie Zellers at stephanie.zellers@helsinki.fi.

Dr. Flávio Azevedo, FORRT’s Director, sits on the iRISE Scientific and Ethical Advisory Board (SEAB) where he hopes to help iRISE’s SEAB to monitor ethical issues that might require attention over the course of the project, but also monitor the progress of our work, assess its scientific quality, and provide feedback to consortium members.

References

[1] Heyard R, Pawel S, Frese J, Voelkl B, Würbel H, McCann S, Held L, Wever KE, Hartmann H, Townsin L, Zellers S. (2025) A scoping review on metrics to quantify reproducibility: a multitude of questions leads to a multitude of metrics. R. Soc. Open Sci. 12: 242076 https://doi.org/10.1098/rsos.242076

[2] Voelkl B, Heyard R, Fanelli D, Wever K, Held L, Maniadis Z, McCann S, Zellers S, Würbel H. (2024) The iRISE reproducibility glossary. Open Science Framework. https://doi.org/10.17605/OSF.IO/BR9SP

[3] Heyard R, Pawel S, Wever K, Würbel H, Voelkl B, Held L. (2023) Reproducibility metrics - study protocol. Open Science Framework. https://doi.org/10.17605/OSF.IO/7VC4Z

iRISE receives funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No. 101094853. The project also receives funding from UK Research and Innovation (UKRI) and the Swiss State Secretariat for Education, Research and Innovation (SERI): Direct Funding for Collaborative Projects as part of the transitional measures.

LMU-FORRT Train-the-Trainer Program Press Release

Tue, 03 Jun 2025 15:30:39 -0300

The Open Science Center (OSC) of the Ludwig-Maximilians-Universität München (LMU) is excited to introduce the Train-the-Trainer program, funded by the VW Foundation. This initiative aims to promote open and reproducible research across academic disciplines. It equips researchers not only with hands-on Open Science skills for themselves, but also with the tools and knowledge to train other students and researchers in open research practices within their own fields.

The OSC staff on the Train-the-Trainer program are:

Sarah von Grebmer is the Train-the-Trainer Program Coordinator. She coordinates, develops and delivers the “Train-the-trainer” programme to future Open Science trainers, and facilitates interdisciplinary workshops, seminars, and networking events to engage and support participants in adopting open research practices.
Sara Lil Middleton is the Training Development Coordinator and Community Engagement Manager. She develops training content and manages and integrates contributions from the FORRT community. She will also co-develop auxiliary content to facilitate the longer-term engagement of Open Science practices beyond the completion of training courses.
Reema Gupta is the OSC’s FAIR data sharing expert. She specializes in data management, research data standards, and reproducible workflows. She creates training materials and provides consultations to researchers on these topics.

A key strength of the program lies in its close partnership with the Framework for Open and Reproducible Research Training (FORRT) community. This partnership reflects a shared commitment to systemic reform in research culture through education, grassroots collaboration, and equitable infrastructure. The FORRT project is an international and interdisciplinary community project that promotes and curates Open Science training resources and meta-research, and is committed to democratizing access to both learning and teaching materials in Open Science, ensuring equitable participation across career stages, disciplines, and geographies. For this partnership with the LMU OSC, FORRT contributes with expertise in Open Science pedagogy, inclusive education, participatory infrastructure-building, and metascience. Moreover, FORRT ensures that the Train-the-Trainer program also embeds diversity, equity, inclusion, and accessibility (DEIA) throughout. Its community developed resources, including training modules, lesson plans, and glossaries are integrated throughout to help ensure alignment with global Open Science values.

Over a three-year period, the program aims to train around 50 instructors, who will then go on to train students and researchers within their respective disciplines. The curriculum includes topics such as methodological foundations of Open Science, computational reproducibility, tools for collaborative working, data management and data sharing. In addition, the curriculum also includes crucial modules on pedagogical skills in teaching about Open Science, such as basic didactic skills, activation tools for student-trainer interactions, leadership skills as well as modules on broader issues connected to teaching Open Science such as diversity, accessibility and cultural awareness. It will be delivered through interactive flipped-classroom formats and as self-paced hands-on online materials. Participants will receive certification upon completion, ensuring academic recognition of their new skills. In this, the program aims to build sustainable capacity within departments and to create a multiplier effect, fostering systemic shifts toward transparency and openness.

“We want to create a freely accessible program covering not only the know-how of open research Practices, but also to provide program participants with the pedagogical tools to confidently pass on their knowledge about Open Science to different audiences.”

— Dr. Sarah von Grebmer zu Wolfsthurn, LMU OSC Train-the-Trainer Program Coordinator

“At FORRT, we believe that equipping educators with both the tools and the values of Open Science is key to changing how research is taught and practiced. This collaboration with LMU is about more than building skills— it’s about cultivating a culture of transparency, inclusion, and better research and teaching practices across disciplines and borders.”

— Dr. Flavio Azevedo, Director of FORRT

For any questions about the Train-the-Trainer program, please get in touch with the Train-the-Trainer program coordinator, Dr. Sarah von Grebmer zu Wolfsthurn ( s.grebmer@lmu.de).

Community-Augmented Meta-Analyses in Education (ELD CAMA)

Mon, 03 Mar 2025 15:30:39 -0300

Background of ELD CAMA

Systematic reviews are fundamental to empirical research and play a crucial role in effective policy-making. Indeed, their importance cannot be overstated. However, the methodological rigor and transparency do not seem to follow the weight of the evidence systematic reviews represent in academia and practice ( Nordström et al., 2023). Unfortunately, the extensiveness of the systematic review process and the speed of the publication proliferation make any systematic review essentially outdated by the time it gets to dissemination. To combat this, a concept of living reviews and meta-analyses (i.e., ongoing systematic reviews that are continuously updated with new studies) have been proposed and developed in various ways (e.g., Tsuji et al., 2014; Bosco et al., 2017).

Although living reviews have existed in various formats, they remain relatively unknown in educational sciences and are rarely used as practical tools. However, recent technological advancements now make it possible to implement such platforms at scale. The Evidence in Learning and Didactics Community-Augmented Meta-analysis (ELD CAMA) platform redefines how educational research is synthesized and applied. Unlike conventional CAMA platforms that rely on pre-curated meta-analyses, ELD CAMA empowers users to upload individual studies, breaking dependency on existing reviews and enabling up-to-date synthesis. This marks a major shift in evidence synthesis: ELD CAMA democratizes access to meta-analytic tools, transforming static reviews into a collaborative, iterative process. By bridging the gap between primary research and large-scale evidence integration, the platform supports faster, more transparent, and more inclusive advancements in educational science.

For researchers, practitioners, and policymakers alike, ELD CAMA turns the ideal of “living reviews” into reality — ensuring decisions are grounded in the most current, comprehensive, and community-validated data available. The database, shiny application and interface is built upon open source packages which enables researchers in other fields to set up a CAMA platform without spending countless hours in development. Simply copy the available repositories and adjust the database and applications to suit the needs in your field. We put effort in making the repository documented and reusable, but we encourage questions for anything unclear, as our goal is for researchers to reuse the available code!

Box 1.

Platform – and a template.

The community-augmented approach therefore extends beyond the content development. Like existing CAMAs that encourage forking and reusing established code, we aim to help meta-researchers with similar needs in developing their own database and platform with the template code we share. The beauty of it lies in the flexibility of the components. Users can take some components (e.g., only the SQL database) and build upon them to add new variables, improve the entire platform (e.g., add additional features or dynamic markdowns), or simply reuse it as is to build a platform in a different scientific field (e.g., medicine). This flexibility facilitates incremental knowledge and collaboration, which is what ELD CAMA and FORRT both strive for. Repository: https://github.com/metahag/ELD-CAMA_source.

How ELD CAMA and the FORRT reinforce each other

ELD CAMA fosters open science, research quality and integrity, and the main goal is to provide users with an open dataset of educational interventions in a FAIR (Findable, Accessible, Interoperable, and Reusable) way. FORRT is an excellent venue to reach like minded researchers who share the same interests and ideals. We hope to engage the educational science and meta-researchers to share the benefits and importance of updated, open, and high-quality systematic reviews. ELD CAMA and FORRT share the same goals but focus on different target groups — both aim to build an infrastructure that supports open scholarship and a credible, reproducible evidence base, developed in collaboration with educators. FORRT’s commitment to sharing educational resources and pedagogical tools that make learning accessible to all — through both physical access to infrastructure and cognitive access to content — makes it an ideal platform for ELD CAMA to connect with like-minded researchers who share similar values and interests. In contrast, ELD CAMA emphasizes high-quality evidence by implementing quality checks (e.g., risk of bias assessments) and promoting open collaboration (e.g., using the ORCID system for logging in), which align with FORRT’s core principles. Additionally, ELD CAMA supports FORRT’s primary goal by offering a platform that educators can use to inform their teaching practices or serve as a pedagogical tool.

For example, higher education teachers can use the platform for many various exercises – to teach meta-analysis, or explore the differences between frequentist and Bayesian statistical methods. Students can learn about the effects of heterogeneity in the population by comparing results from different subgroups. The benefit is not only in the platform’s content – with the open source code, educators can teach students how to code the database, or any of the shiny apps, which can make the platform valuable for data science, beyond its intended use.

ELD CAMA Mission

Evidence in Learning and Didactics (ELD) is a research methods lab focused on improving evidence synthesis methods and transparency. We conduct and plan various systematic reviews within education research, and have encountered all of the annoying parts of the process, but we especially focused on improving the final part of the process - data extraction and synthesis. Extracting data is tortuous and tedious, and there had to be some way to make this part more worthwhile. So we turned to the existing living review concepts and decided to create one for educational intervention research.

Within the lab, we developed a Community-Augmented Meta-Analysis (CAMA) platform that enables researchers to contribute to living reviews by simply uploading studies they read or have conducted. When we say simply, we mean - go to the platform, login and start contributing data through an upload form that guides you on how to upload data. Good, right? Well, we hope so, and if you think there is an even better way, we’d love to hear it!

Not only can you contribute to the database, you can benefit from the database by using the available data to conduct your own meta-analyses, without having to go through the process of finding the studies and extracting the data for each one. Our Shiny apps allow you to do frequentist or Bayesian meta-analysis on the data pulled directly from the database.

Quick start

Go to wanda.lnu.se and sign in with your ORCID.
Use the dashboard to explore educational intervention data and perform analyses.
Upload a study you’ve read or conducted — contribute to open educational science!

Call to Action

If you found the presented concept interesting, then we need your help! As the name implies, these platforms aim to be community augmented. Our platform is in early stages, and needs to be populated with data. If you’d like to test our upload form and have a perfect study in mind to add, you can do it here. Beyond that, we also look for feedback on the platform, any part - the shiny apps for the meta-analysis, database, user interface, code. Detailed information about the platform and links to repositories are available in the preprint (Box 2). If you decide to contribute, please email us, for contribution credit purposes.

Box 2.

Featured Publication

“… our commitment to open science practices means that this next generation of CAMA platforms can further develop in new areas in the spirit of open, transparent, and community-driven research.”

Kalmendal, A., Batinović, L., Claesson, F. & Carlsson, R. (2025, March 25). Living Review Platform for Educational Interventions: The Evidence in Learning Community Augmented Meta-analysis. https://doi.org/10.31234/osf.io/hjku5_v1

The development of ELD CAMA was made possible through the support of a grant from the Swedish Research Council. We thank all contributors throughout the development of the project, and are particularly grateful for Dr. Christina Bergmann’s help with the initiation of the ELD CAMA.

Contact information:

Lucija Batinović, lucija.batinovic@liu.se

Accessible Feedback in Research

Tue, 03 Sep 2024 15:30:59 -0400

Background

Did you know that only 27% of UK Professors are women, and less than 1% of UK Professors are black (HESA, 2020)? This significant underrepresentation highlights a problematic “leaky pipeline” in academia that affects various disciplines worldwide (Dubois-Shaik & Fesulier, 2015). Marginalised groups, including women and minorities, encounter systemic barriers that hinder their professional advancement, leading to fewer individuals in higher academic positions and reinforcing a cycle of inequality. Marginalised researchers consider work environment as the most common barrier to professional progression, yet perceive the support and guidance of others to be the most influential facilitator (Gregor et al., 2023).

Feedback has the potential to evaluate and promote research quality, helping academics to incrementally improve their research practices. However, structurally embedded inequalities in feedback systems are a key factor contributing to significant deficits in the recruitment and retention of diverse talent in higher education and research institutions (Kaplan et al., 2017). For example, the traditional feedback mechanism of peer review is the most prevalent type of feedback across disciplinary, geographical, and career boundaries (Chong & Masson, 2021). However, since peer review provides feedback to researchers after the work has been conducted, many researchers find their time and resources depleted before their efforts can be properly evaluated and improved. Furthermore, the peer review system is often led by individuals from privileged groups of senior academics and can be highly inconsistent (Hamilton et al., 2020). Additionally, biases in peer review have been shown to perpetuate the exclusion of minoritised groups (Smith et al., 2023), with similar trends across other feedback mechanisms like conference talks and questions, highlighting the urgent need for a more supportive research environment with accessible feedback processes for the researchers who could benefit most.

To drive this change, we are undertaking an international multi-disciplinary survey to comprehensively map out and evaluate diverse opportunities for researchers to gather feedback throughout the research cycle. Our survey aims to uncover cross-disciplinary feedback practices that can foster more timely and inclusive feedback. We hope that this will enable early-career and marginalised researchers to refine their work iteratively and efficiently, supporting their professional development.

Our mission and how it is working to contribute to the open scholarship community

Our mission is to foster a more inclusive and diverse research environment by addressing systemic barriers involved with feedback mechanisms and supporting the professional development of marginalised researchers. This initiative not only enhances individual professional growth but also contributes to the broader open scholarship community by promoting transparency, inclusivity, and collaboration in academic research.

Many advancements within the open scholarship movement are rooted in feedback-based practices, such as pre-printing, conducting premortems to identify methodological concerns, and performing code reviews to evaluate accessibility and processing quality (Evans, 2022). Although these practices are gradually reshaping norms around transparency and data sharing, the full scope of emerging feedback mechanisms within the open scholarship movement is not yet fully known or widely disseminated. For instance, many newer feedback initiatives, such as Registered Reports, which aim to improve the reproducibility and transparency of research by peer reviewing study proposals before data collection, remain underutilised and have only slowly expanded beyond the psychological and social sciences (Chambers & Tzavella, 2021). Our project seeks to uncover and promote these diverse feedback opportunities, ensuring they are more widely understood and adopted across various disciplines.

How our project and FORRT reinforce one another

Our project and FORRT reinforce each other by promoting a culture of transparency, inclusivity, and continuous improvement in academia. Both initiatives focus on breaking down systemic barriers and supporting the professional development of marginalised researchers, ultimately contributing to a more inclusive and diverse research environment.

By leveraging our strengths, we can accelerate the adoption of open scholarship principles and foster a research environment that values diversity, inclusivity, and high-quality research practices. This complementary approach ensures that researchers have the tools, knowledge, and support needed to navigate the challenges of research and thrive in an open and inclusive scholarly community.

Call to action

Now, we need your help.

Academic researchers from diverse disciplines and regions are invited to participate in this voluntary survey to share their experiences and strategies for seeking feedback, regardless of scale or origin. We’re interested in everything from internal discussions or external sources like advanced technologies such as AI for improving research readability. By mapping and evaluating these strategies, our project intends to compile accessible recommendations that empower researchers to effectively integrate feedback into their research processes.

As part of this initiative, we will be building a dedicated FORRT team to create a community-maintained e-book that will summarise the survey’s findings. This e-book will serve as a roadmap, highlighting and evaluating feedback opportunities at each stage of the research cycle. To make this resource as valuable and comprehensive as possible, it’s crucial that we gather insights from diverse disciplines and geographical regions. By doing so, we can facilitate the exchange of strategies across various fields and locations, enriching the recommendations and making them more applicable to a broader range of research contexts.

We invite you to join us on this journey to help drive positive societal change by enhancing the feedback culture in research and empowering researchers worldwide. Stay connected and look out for updates on the project through FORRT’s social media channels, newsletter, and other communications where we will be sharing opportunities to work with you.

Want to get started? The easiest first step is to take part in our survey: Online Survey Software | Qualtrics Survey Solutions

Contact information:

Project Lead: Dr Thomas Rhys Evans, thomas.evans@greenwich.ac.uk

Research Fellow: Roksana Sobolak, r.a.sobolak@greenwich.ac.uk

Research Fellow: Magda Skubera, m.skubera@greenwich.ac.uk

References

Chambers, C. D., & Tzavella, L. (2021). The past, present and future of Registered Reports. Nature Human Behaviour, 6, 29-42.
Chong, S. W., & Mason, S. (2021). Demystifying the process of scholarly peer-review: an autoethnographic investigation of feedback literacy of two award-winning peer reviewers. Humanities and Social Sciences Communications, 8(1), 1-11.
Dubois-Shaik & Fesulier (2015). Academic careers and gender inequality: Leaky pipeline and interrelated phenomena in seven European countries. Available at: https://eige.europa.eu/sites/default/files/garcia_working_paper_5_academic_careers_gender_inequality.pdf
Evans, T. R. (2022). Developments in Open Data Norms. Journal of Open Psychology Data, 10(1), 3.
Gregor, M., Dunn, M., Campbell-Halfaker, D., Martin-Fernandez, J., Ferrer, A., & Robinson, S. (2023). Plugging the leaky pipeline: A qualitative investigation of untenured female faculty in STEM. Journal of Career Development, 50(2), 425-444.
Hamilton, D. G., Fraser, H., Hoekstra, R., & Fidler, F. (2020). Meta-research: journal policies and editors’ opinions on peer review. Elife, 9, e62529.
HESA (2020) Higher Education Staff Statistics: UK, 2020/21. Available at: https://www.hesa.ac.uk/news/01-02-2022/sb261-higher-education-staff-statistics
Smith, O. M., Davis, K. L., Pizza, R. B., Waterman, R., Dobson, K. C., Foster, B., … & Davis, C. L. (2023). Peer review perpetuates barriers for historically excluded groups. Nature Ecology & Evolution, 7(4), 512-523.

Course on Open Science 101

Mon, 10 Jun 2024 00:50:51 -0400

Main Text

The replication crisis has raised many questions for researchers, but it also poses unique challenges for those teaching about science and the scientific process. We now know that many scientific studies cannot be accurately reproduced or their findings replicated, and this has raised concerns about the validity and reliability of scientific research, leading to calls for increased transparency and reproducibility in scientific inquiry.

For those involved in teaching science, it is therefore not possible to simply continue as we have been, for example, accepting and presenting published scientific works as the “truth” or final word on any given phenomenon. Rather, we need to take a step back and teach students about the entirety of the research process, both in its idealised form, and addressing the complex reality of a scientific process that is impacted by cognitive biases, human behaviour, and incentive structures.

Introducing Students to Open Science

These lectures and tutorials introduce students to the scientific process and open research practices, in order to ensure that psychological research is as rigorous, transparent and, ultimately, as reproducible as possible. We also consider problematic practices that have frequently occurred in psychological research, and discuss the impact they have had on psychological knowledge, as well as learning how to avoid such problematic practices in the future. The module content was designed to be delivered at a first-year undergraduate level, and can also be delivered in a pretty modular fashion, allowing the tutor to pick and choose which sessions best suit their programme or student needs.

Some might wonder if the first year of a degree programme is too early to be learning about open science. Shouldn’t we be teaching them the traditional approach first and then, once they have this grounding, get into open science practices later in their degree? I would strongly disagree with such an approach. There are several reasons why it’s important to teach open science to undergraduate students, and at the earliest possible opportunity. First, open science promotes transparency and reproducibility in research, which are essential principles of scientific inquiry. By teaching students about open science, we can help ensure that they are conducting research in a responsible and ethical manner. Why would we first teach them an approach to research that doesn’t integrate these ideals? It makes no sense to me.

In exposing students to open research and to questions of research integrity, they get to see not only the unvarnished nature of scientific inquiry, warts and all, but also to see the true potential of applying open research practices. They can see how open science can make research more collaborative and inclusive, for example by highlighting how to share data openly, so they can more easily share their findings with others, and also being able to build upon the work of their peers. Teaching open science can also help prepare students for the rapidly changing landscape of scientific research, where open and collaborative approaches are becoming increasingly important. Open research skills are frequently highlighted in academic job requirements, as well as cited as best practice by education institutions, charitable foundations, and research funders.

The Course Content

The content presented here was jointly developed with colleagues at Lancaster University (Dr. Marina Bazhydai, Dr. Sally Linkenauger, Dr. Neil McLatchie), as part of two undergraduate psychology modules, formally listed as Research Integrity and Open Science I (PSYC123) and Research Integrity and Open Science II (PSYC124).

Because the courses assume no knowledge of the research process, the course first focusses on the scientific method, thinking about experimental design, and questions of confounds, reliability, and validity. Once we have that under our belts we move on to contrasting “traditional” approaches to research vs an “open science” approach, and cover topics of study preregistration, replications, and sample size considerations.

We then turn to some of the thornier questions science and scientists have been faced with: How do we determine what is true in science? How do we know which theories are well supported by evidence and which ones are not? How can we tell if researchers are trying to pull the wool over our eyes? To start to think about these questions we look at learning how to spot and avoid questionable research practices, in favour of practices that are open, transparent and reproducible. In this way, we think about some of the problems faced by researchers, such as cognitive biases, researcher flexibility in reporting outcomes, and even fraud. We ask how we can assess research findings in the face of such problems?

PSYC123	PSYC124
Introduction and Scientific Method	Introduction to False positive psychology
Measurement (Distribution/Reliability/Validity)	Questionable Research Practices
Experimental Designs	Biases and incentive structures
Confounds and Extraneous	Introduction to meta-analysis
Traditional vs Open Science	Scientific Fraud
Study Preregistration	Errors and error detection
Replication	Principles of Open Data Sharing
Sample Size	Spotlight on Developmental Science

These modules will show students how scientific practices in the field have changed rapidly in recent years, with renewed interest in openness and transparency. While we focus on a range of problems that impact our interpretation of research findings, students will also develop an understanding of tools that can help overcome and prevent these issues from arising (e.g., study preregistration, sharing of data, meta-analyses etc.).

For tutors and instructors, lecture slides were designed to be covered in about 50 minutes, and the labs can also be completed in around 50 minutes, although some will take a little longer. It’s also worth noting that these modules don’t involve any statistical programming, focussing more on open research concepts, but can (and are) taught alongside modules more focussed on learning statistics through R. There is also some flexibility in the order that topics can be delivered. For example, Meta-Analysis is a standalone topic that could be moved later in the course, or handled separately as part of a more advanced research methods module.

The course content has an open licence (CC BY), so we encourage instructors to use and modify the content to meet their needs. For instructors, answer sheets are available for the majority of lab exercises, so please just email if you’d like a copy. If you do use any of the materials, we would love to hear about it! Indeed, we would be grateful for any feedback you have to offer.

Course content is available on the Open Science Framework, so to download everything go here for PSYC123, and here for PSYC124.

Contact information:

Dr. Dermot Lynott, Department of Psychology, Maynooth University (Email: dermot.lynott@mu.ie; Homepage.)

Being a Minoritized Early Career Researcher in Political Psychology

Mon, 10 Jun 2024 00:48:51 -0400

For the ISPP ECC virtual event titled “ Minoritized Early Career Researchers’ Perspectives and Experiences in Political Psychology,” one moderator and three panellists—all Early Career Researchers (ECRs) with intersectional minoritized identities—came together to discuss their experiences in academia, the barriers they face, and future projections to make political psychology a more inclusive field. This blog post summarises the discussion from the event held on May 8, 2024.

Navigating academia as a minoritized early career researcher (ECR) in political psychology reveals a deeply politicised existence. Our intersecting identities (e.g., being openly queer, people of colour (PoC), neurodivergent, etc.) significantly shape how we are treated and perceived. Despite holding some power as scholars, our status as ECRs emphasises our marginalisation, complicating our research and presence in an academic space marked by varying levels of privilege and power. Being visibly minoritized (e.g., PoC or disabled) further politicises our position. Additionally, being first-generation academics in Europe or America or being based in a Global South university adds another layer to our minoritized experience, making our very existence and expression in academia a political act, as we have to show our expertise and validate our existence in this power system daily. The current political climate, marked by xenophobia, white supremacy, and anti-immigration, exacerbates these challenges and makes our existence, scholarship, and actions politicised as our presence in academia and the world at large are continually questioned.

Growing up in economically and politically unstable environments or being marginalised from an early age instilled in us the belief that education could improve our lives and those of others. We saw academia as a saviour that could provide the material, intellectual, and political tools needed for desired social change towards more acceptance of people like us in privileged spaces. However, academia is far from the progressive haven it is often portrayed to be. The same reasons that made us see academia as a saviour also created constraints such as social and financial hardships, prolonging the start of our academic journey, and degree attainment. The intersection of these fundamental hardships with the precarious positions we occupy in academia reveals a significant gap between the theoretical commitments to diversity, equity, and inclusion and their practical implementation. These lived experiences emphasised the urgent need to address and dismantle these barriers and power structures.

Our journey into researching minoritized groups began with a belief in the importance of diversity. However, our perspective has evolved with our experiences as minoritized ECRs, leading us to understand that researching minoritized groups isn’t just about diversity; it’s about rethinking our approach to research itself. We must question our foundational assumptions and methods to ensure they don’t inadvertently perpetuate existing power structures. Intersectional approaches are crucial in this rethinking process. By examining the intersections of various identities, we can systematically uncover and challenge the assumptions embedded in our discipline.

As minoritized researchers, we are often expected to excel in established, less contentious areas to gain the credibility needed to pursue innovative, community-focused research. Diversifying the academic pipeline and conducting what we call “water is wet” research—validating what minoritized communities already know and experience—are key motivations for our work. Empirical evidence is necessary to substantiate lived experiences and influence policy and decision-making effectively. Political psychology still holds significant gaps, particularly in addressing intersectional differences and the “unknown of wet water.” Many foundational concepts in political psychology remain underexplored across different groups (i.e., the majority of the world). Addressing these gaps is essential for building robust, inclusive scientific knowledge production.

Living in a time where politics is forced upon you and your existence has also shaped our understanding of minoritized groups. Witnessing ongoing protests and social movements firsthand has highlighted the importance of recognizing the agency and knowledge within these communities. Rather than speaking for minoritized groups, we aim to amplify their voices and learn from their experiences. Researching minoritized groups requires constant ethical reflection. It’s crucial to ensure that our work benefits these communities and does not reinforce existing injustices and power structures. It is essential to acknowledge that most research on inequality and minoritization has been conducted by beneficiaries of the systems that produce inequality and minoritization. Therefore, the political biases of researchers while conducting these studies should not be ignored but systematically examined and recognized. Addressing these biases is essential for fostering a more equitable and inclusive academic environment by deconstructing what is seen as neutral, scientific, or a priori and showing that power structures are embedded in knowledge production. Conducting research on minoritized groups in political psychology demands a critical, intersectional approach that challenges existing power structures and amplifies marginalised voices. It’s about rethinking our methods, validating lived experiences, and ensuring our work contributes to social justice and equity.

The struggles faced by minoritized ECRs often originate from various mechanisms, including other researchers, institutions, and the dominant knowledge production systems in academia. These structures and incentives create significant challenges. For instance, we all, in varying ways and levels, have experienced a high degree of objectification, being frequently used for our expertise in “authentic” contexts and perceived as data collectors for privileged senior researchers who have no access to these contexts, making us feel like accessories rather than valued collaborators. Especially scholars from the Global South are often seen as a source of data and not valued for their substantive or methodological contributions. They are, often explicitly, considered as accessories to the research ideas of the Global North. This lack of recognition of their potential contributions and an enduring perception forces them to publish in lower-ranked, area-specific journals, limiting their recognition and career progression. We also encounter derogatory classifications in cross-cultural research, often marked with problematic and binary Western-oriented categorisation of the world, experiences, and emotions. When we try to highlight these issues, we are labelled as problem-makers: the hierarchical nature of academia further penalises us for disclosing our neurodiversity, PoC, or queer identities, making salient our marginalisation.

The handling of political and social crises by major academic organisations, such as genocides, wars, and conflicts, has further exposed moral shortcomings within the field of political psychology. The lack of decisive action and the tendency to adopt a so-called “neutral” approach have disillusioned many scholars, particularly minoritized ECRs. This issue highlights a broader moral, political, and generational gap within the field, with established academics with a privileged position and power of impact in academia often failing to adequately address or understand the perspectives and needs of younger, minoritized—and in turn politicised—researchers.

The legitimised narrative in political psychology restricts space for diverse research outputs. Convincing others that race, gender, class, and neurodiversity are universal issues, rather than niche areas, is particularly challenging. To dismantle power structures and hierarchies within academia, particularly for minoritized ECRs, we must recognize the systemic and everyday challenges we face and the actionable changes needed at both the systemic level and daily research practices. Academia often views minoritized ECRs as mere cogs in the machine, expecting them to fit predefined roles rather than valuing their unique expertise. This is particularly challenging for neurodiverse individuals who find the expectation to conform and the lack of genuine support alienating. The system’s tendency to replace those who don’t fit the mould exacerbates this issue, contributing to high attrition rates among marginalised scholars.

We’ve often faced financial and logistical barriers in attending international conferences. These challenges included visa delays, high costs for virtual presentations, and a lack of accommodation for low-income researchers from the Global South. Despite the academic community’s professed commitment to diversity, these scholars often encounter conditional support and red tape. The hierarchical and exclusionary nature of academic conferences and institutions marginalises scholars from the Global South and those with limited financial resources. We want to emphasise the need for more inclusive and democratised avenues for recognition and collaboration. Social media and horizontal networking among peers serve as effective alternatives to traditional, hierarchical networking methods like conferences. Nonetheless, the power of collective action and communal support is essential for overcoming these barriers and shifting away from reliance on traditional academic structures of conferences towards more inclusive practices. Community building emerges as a vital strategy for survival and success in academia for minoritized ECRs. Finding and nurturing a supportive network—whether through professional associations, annual conferences or communities centred around minoritized ECRs’ experiences and perspectives—provides essential emotional and professional support. These communities offer recognition and a sense of belonging that institutional structures often fail to provide.

In conclusion, while there are isolated examples of good practice, systemic change is required to dismantle the entrenched system of academic hierarchies and power structures. A path forward includes adopting more inclusive and context-sensitive review processes, providing genuine support for minoritized ECRs, fostering strong community networks, and addressing the moral failings within the discipline. Only through such comprehensive changes can we create a more equitable and supportive academic environment for all.

Playing this video loads it from YouTube (Google). See our privacy policy.

Uysal, M. S., Chan, J., Hudson, S., Mukherjee, A., & Azevedo, F. (2024). Being a minoritized early career researcher in political psychology. ISPP’s Political Psychology Blog. Link

Shifting the culture of peer review with Reviewer Zero

Fri, 10 May 2024 00:48:51 -0400

Background of Reviewer Zero

Any writer who puts their work into the world knows it can be a humbling and harsh experience. Negative experiences with peer review, as depicted in the picture below, are all too common. Comments like those illustrated – and others that are unduly harsh, belittling, racist, sexist, or otherwise inappropriate – have been received by scholars at all career stages during peer review. These experiences provide a starting point to question normative practices in scientific peer review, and identify the hidden and blatant consequences for early career scholars, particularly those from underserved groups. Most importantly, we can ask what each of us might do to shift this culture.

Reviewer Zero is a collective founded in 2020 to take concrete, data-driven, anti-racist action to improve the culture of scientific peer review. We were formed with guidance from SPARK Society — an organization that improves visibility and provides mentorship opportunities for cognitive scientists of color. Our broad aim is to change cultural norms in the peer review process, in order to improve the recruitment, retention, and advancement of minoritized scholars in psychology. Collectively, we are a group who has experienced hundreds of rejections and written hundreds of reviews. We’ve received decision letters accepting and rejecting our work, and we have written decision letters accepting or rejecting others’ work.

Like readers of this blog, Reviewer Zero members know that science can be better than it is, and that shifting the status quo is not easy. The culture of science matters for who engages with it, and normative experiences within a culture can produce different consequences if individuals see themselves as safe or threatened within that culture. The experience of peer review as hostile or negative sets the stage for disparate impacts for minoritized individuals. For example, a survey of peer review experience (Silbiger & Stubler, 2019) found that experiences of unprofessional peer review were quite common - reported by 50% of scientists in the survey. However, individuals from minoritized groups reported a stronger negative impact of these unprofessional reviews, compounding the impact of other biases against such individuals (see Aly et al., 2023, for further discussion preprint available).

To understand peer review from the perspective of early career researchers who are underserved by the status quo, Reviewer Zero conducted a survey in 2020 of early career researchers in psychology / neuroscience. Our results are not surprising: We found that cis-men of color reported less helpful feedback compared to White cis-men, and women/nonbinary respondents were more likely than cis-men to report that their peer review experience reduced their belonging in science. Qualitative results from a free-response question about the most memorable experiences with peer review were eye-opening, and heartbreaking. Respondents reported receiving harsh feedback, racist comments, and feeling so demotivated that they left those research topics behind. But there is also opportunity for change: This survey also showed that authors can feel motivated when the process is perceived as fair or when reviews note strengths of the work.

Harsh reviewer comments can have disparate impacts, demotivating early career researchers and driving them away from the field.

Photo by KOBU Agency on Unsplash

How Reviewer Zero’s Mission Intersects with the Open Science Community

As efforts to improve scientific practice and policy toward more openness and transparency take shape, it is essential to consider who is designing these systems and what perspectives they reflect. To build systems that are truly open and accessible for a global psychological science, we must take proactive steps to include different perspectives, identities, and voices in creating and changing systems. Here, we walk through examples of two kinds of openness that can have implications for minoritized or underserved groups. The essential questions are who is being asked to be “open,” and at what cost?

One innovation in peer review is open peer review, which includes disclosure of author and reviewer identities and reviews. Open peer review can include different levels of disclosure: For example, greater transparency without identification can occur if reviews are public but reviewers are not identified. Although identified peer review can have some advantages in that reviewers are clearly accountable to what they write in a review, this system may not be equitably open to all. Individuals who occupy less powerful positions, whether due to their career stage or underrepresented group identities, may experience more backlash or more negative consequences in offering critique – or their perspectives may simply be dismissed in favor of reviews from people who hold more power or come from majority groups. Transparent peer review, where reviews are public but reviewer identities are not, might offer greater security to reviewers. From an author perspective, though, individuals who are contending with identity threat may be unwilling (for good reason) to have critiques of their work openly aired (for further discussion, Aly et al., 2023). The very ability to engage in different aspects of open peer review can vary across individuals with different levels of power in a system.

Would greater openness in peer review make hostile and unprofessional reviews less likely, or would greater openness in peer review exacerbate negative consequences for members of historically excluded groups? We don’t know the answer to these questions, but we need to ask them.

Given that hostile or unprofessional reviews can be common, individuals from historically excluded groups might steer away from participating in open review processes. Navigating peer review is challenging enough without doing so in public. One example comes from Dr. Colleen Murphy, who tweeted in response to eLife’s move to drop accept/reject decisions:

The inequities present in peer review need to be addressed before there is any possibility of open review serving a wide range of scholars. Without a deliberate effort to change the existing culture in a way that advances diversity, equity, and inclusion, other efforts might perpetuate or exacerbate these inequities.

A different form of openness, transparency, and accountability in peer review comes from institutions or organizations. It is not yet routine for journals, societies, or editors to collect or share aggregate-level data about the demographic or identity characteristics of editors, editorial board members, reviewers, or authors. Yet knowing this information could provide valuable insights about who is served by the current practices and how that might change with reforms. What are the demographic or identity characteristics of the editorial board and reviewers? Who is submitting papers, who is being invited to resubmit papers, who appeals editorial decisions, and what are the outcomes? Data about these aspects of scientific publishing are rarely collected, rarely examined by journal staff, and are even more rarely shared with the community. Without transparency and accountability in terms of representation, editor and reviewer choices tend to rely on individuals’ own social networks and perpetuate gender and race homophily. This reliance on social networks may be exacerbated by the growing difficulties with finding scholars to agree to provide peer reviews (Petrescu & Krishen, 2022). Editors may rely on close colleagues who are willing to do them a favor by providing a review, which can in turn further narrow the pool of reviewers and the types of feedback authors get. To counteract this tendency, journals can publicize calls for individuals to register as new reviewers, so that editors have accessible and current listings of scholars with expertise and availability. Generally speaking, clearer and more transparent data about who engages with peer review and how they do so can inform innovations to ease burdens on editors and provide more diverse slates of reviewers.

How Reviewer Zero and FORRT work together

Like FORRT, Reviewer Zero seeks to illuminate the black box of scientific and academic culture. Peer review is central to scientific careers, but peer review is often left to the “hidden curriculum” of processes that people learn as they go without explicit discussion or training. To bring some light to these unknown processes, Reviewer Zero hosts workshops (both online and at in-person conferences) for both early-career scholars and for editors and reviewers. In the sessions aimed at trainees and early career scholars, we introduce the basics of peer review, to consider the emotional and motivational aspects of navigating peer review, and to provide concrete tools and strategies for responding to editorial decisions. In workshops with editorial teams, we lead discussions about the current culture of peer review and its implications for perpetuating disparities in who contributes to and advances in science, and we discuss concrete steps toward a more constructive and inclusive peer review culture.

Reviewer Zero is developing on-demand resources based on these workshops, and these can complement FORRT resources offered in the adopting principled education section of the website. We particularly advocate for collaboration and conversation moving forward, for identification and promotion of strategies that can advance diversity, equity, and inclusion both in peer review systems and the goals of open science. At times, as noted above, these goals might appear to be in tension with each other - but that should be the start rather than the end of the conversation.

Call to action

Each of us plays a part in maintaining or changing the culture of scientific peer review. We invite you to join us to stay updated on activities at www.reviewerzero.net and to explore our developing on-demand resources.

There are steps you can take today to move toward a more constructive and equitable peer review system. As with any culture change, shifting a culture requires multiple actors to begin to think and act differently, and cultural change is more possible when actors at different levels of the system take part (Hamedani & Markus, 2019). If you are someone with power in the system, with experience, and/or with a majority group identity, support minoritized and early career scholars to learn more about the unspoken aspects of peer review (Aly et al., 2024). If you have experience in scientific publication, speak about your rejections as well as your successes. When you deliver critique, remember there is a human on the other end of that communication, even if that person is not identified. If there is something promising or exciting about the work, note it explicitly. Recognize what power you have, and amplify the voices and perspectives of people whose voices are not yet central to the conversation.

If you are someone who has been underserved by the current system, know that there are communities and allies who deeply value your contributions to science. Find friends and mentors who can interpret and contextualize reviews with you. If you encounter hostile or unprofessional reviews, know that you have options (including alerting the editor of the journal), and that unfortunately you are not alone.

We at Reviewer Zero see equitable practices and reducing group disparities as essential to cultivating excellent science - science that is fair, meaningful, robust, and rigorous. Any move to improve scientific practice can ultimately succeed only if it centers building systems and spaces that represent a diverse range of identities, uphold equitable practices and outcomes, and prioritize inclusivity. Peer review is central to the practice and progress of science, and bettering it to serve a wider range of scientists will improve the rigor, quality, and contribution of science.

References

Aly, M., Colunga, E., Crockett, M. J., Goldrick, M., Gomez, P., Kung, F. Y. H., McKee, P. C., Pérez, M., Stilwell, S. M., & Diekman, A. B. (2023). Changing the culture of peer review for a more inclusive and equitable psychological science. Journal of Experimental Psychology: General, 152(12), 3546–3565. https://doi.org/10.1037/xge0001461

Hamedani, M. Y. G., & Markus, H. R. (2019). Understanding Culture Clashes and Catalyzing Change: A Culture Cycle Approach. Frontiers in Psychology, 10. https://www.frontiersin.org/article/10.3389/fpsyg.2019.00700

Petrescu, M., & Krishen, A. S. (2022). The evolving crisis of the peer-review process. Journal of Marketing Analytics, 10(3), 185-186.

Silbiger, N. J., & Stubler, A. D. (2019). Unprofessional peer reviews disproportionately harm underrepresented groups in STEM. PeerJ, 7, e8247. https://doi.org/10.7717/peerj.8247

Aly, M., Ansari, S., Colunga, E., Crockett, M. J., Diekman, A. B., Goldrick, M., Gomez, P., Kung, F. Y. H., McKee, P. C., Pérez, M., & Stilwell, S. M. (2024). Mentorship practices that improve the culture of peer review. Nature Reviews Psychology, 3(1), Article 1. https://doi.org/10.1038/s44159-023-00261-1

Author Biography

Reviewer Zero Organizing Committee. Contact information: admin@reviewerzero.net.

Funding

This material is based upon work supported by the National Science Foundation under Grant No. 2224777. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Academic Jobs in the US vs. the UK

Tue, 30 May 2023 16:00:00 +0100

Given the scarcity of full-time, permanent academic jobs, more and more PhDs are looking abroad for employment. When I was on the job market in 2020–21, I applied to positions in 7 countries. I landed at the University of Nottingham in the UK—which, despite being ostensibly close to my home country of the US in terms of culture and primary language, proved to be an entirely different world. This was apparent from the job application stage, yet at the time, there was no publicly available guidance on differences between the academic job markets in the US and UK.

Earlier this month, I set out to address this. I ran a webinar, “The Academic Job Market: US vs. UK,” aimed at demystifying both markets for applicants. Interest was so great that I added a second installment; in the end, over 200 people registered. You can watch a recording of the webinar below, which will remain freely accessible to all. The slides are also available to download from my website. I hope this can be a helpful resource to anyone looking to make sense of different market timelines, varying employment structures, and why you can’t address anyone below the rank of full professor in the UK as “Professor.” (This still trips me up constantly.)

Playing this video loads it from YouTube (Google). See our privacy policy.

Structural Factors Shape Employment

What I’d like to expand on here, however, is something only briefly touched on in the webinar: the structural factors shaping employment in higher education in the US and UK. Whereas much of the academic job market comparison in the two countries is apples and oranges, paying attention to the sociopolitical forces changing higher education reveals much common ground.

Let me start with a timely example. In mid-April, faculty (teaching staff in UK parlance), postdoctoral fellows, graduate students, and counselors at Rutgers University joined together in unprecedented collective action. In the US, different categories of university staff are represented by different unions, and these are at best loosely networked nationally—there is no equivalent of the University and College Union (UCU), which represents all research and teaching staff, professional services staff, and PhD students at every university in the UK. Yet despite organizational hurdles, some 9,000 Rutgers workers went on strike together for living wages and an end to race and gender pay gaps. I can recall no similar university-wide coordinated action in the US in my lifetime.

There are echoes in this struggle of the dispute UCU has waged since before the pandemic. Wages for UK university staff have fallen some 25% in real terms since 2009; the race and gender pay gaps remain unacceptable for universities who claim commitment to equality, diversity, and inclusion (EDI, the UK twist on the US’s DEI: diversity, equity and inclusion). Equally troubling are the echoes in both disputes’ resolutions. The deal at Rutgers, agreed to after a week of action, has been criticized for marginalizing graduate workers’ voices, just as the current proposal under consideration by UCU provides no concrete improvements for PhD students on casualized contracts. Solidarity at Rutgers, as in the UK, has been strongest among the most financially secure staff, with some of the least powerful university employees being left behind. Depending on the discipline, US PhDs will graduate into an academic job market where around 70 percent of jobs are temporary or contingent—a similar landscape to that facing UK PhDs.

Features, Not Bugs

Where do these analogous structural failings come from? The answer to this starts with deconstructing the idea that these are “failings,” rather than intentional components of a system that views degrees as commodities and people as cogs in machines. A system wherein the task of teaching is secondary to that of shoring up a university’s research reputation will not mind if teachers have to live in their cars or move from city to city every semester in search of work. Likewise, a system where glossy building façades matter more than what happens inside of those buildings will not mind using historically excluded groups, particularly women of color, as PR fodder while exploiting their labor until they burn out.

Others have written far more comprehensively about how these features of higher education (not bugs) have become ever more present in the past few decades. My point is simply that they are unavoidable, regardless of country of residence. Some might claim that coming to the UK will bring reduced research pressures compared to US research-intensive universities, which may be true in terms of the number of publications required but overlooks how the UK’s massive (and growing) grant-grubbing apparatus saps staff time and discourages counterhegemonic work. Others might note that US academic salaries are generally higher than in the UK, which may be true even at poorer institutions but masks the massive (and growing) disparities across and within US institutions. Regardless, the “it’s better over there” narrative sidesteps the reality of the academic job market for many: that, if they are able, they will go wherever hires them.

It’s easy to despair. What’s necessary—and what I hope sharing information across the pond helps encourage, however minimally—is turning collective despair into solidarity. Faculty in the US and UK face different day-to-day pressures, different career benchmarks, and different alphabet soups of acronyms. (Ask me sometime why Master’s students are PGTs but doctoral students are PGRs.) But underneath the surface, our struggles rhyme. It behooves those of us on both sides of the Atlantic to learn about how academia works in the US and UK, as well as in other countries. Our sector is transnational, and transparent, accessible information about that sector is an important foundation from which to exchange expertise, build campaigns, and push for change.

Author Biography

Anna A. Meier (she/her/hers) is Assistant Professor in the School of Politics and International Relations at the University of Nottingham. She is a committed advocate for graduate students, solidarity-building, and collective care in higher education. Email: anna.meier@nottingham.ac.uk; Twitter: @AnnaMeierPS; website: annameier.net.

A Student’s Guide to Open Science: Using the Replication Crisis to Reform Psychology

Thu, 09 Feb 2023 01:44:40 -0400

A child of the replication crisis

When Open University Press approached me to write a book on the “replication crisis in psychology”, I was filled with excitement as I knew that discussion of replication issues is scarce within teaching and psychology textbooks. If they don’t teach it, then I could! But I also knew it had to be more than the so-called ‘crisis’; a term associated with pessimism. It had to be about how far the discipline of psychology has come, and the opportunities that open research can bring to reform research training and the wider culture.

I am a ‘child of the replication crisis’. I completed my PhD in 2016 on ‘stereotype threat’; an influential phenomenon that suggests negative societal stereotypes deplete performance. But I got stuck time and time again trying to replicate these classic effects. I felt like a failure. I wish I had known what I know now: that the experiences I was facing were more common than what was gleaned from the published literature. Science is not perfect and nor should it be, but this was not what I was taught through my undergraduate education. So I wrote a book – A Student’s Guide to Open Science – for my younger self. In other words, for the talented and passionate students studying Psychology. By knowing our history – the good parts but also the messy parts – our students represent the grassroots that can make this discipline the best it can be!

What is this book about and why is it needed?

By now, many researchers have heard about the ‘replication crisis’ that has swept research disciplines, and the fast-paced improvements brought about by open research reform. However, there is still a gap in educational practices – in the teaching of the replication crisis – which FORRT and its members strive to overcome. Furthermore, when trying to learn about the replication crisis and open science, it can feel overwhelming and there are lots of different resources in many different places. With this in mind, I wrote a “A Student’s Guide to Open Science” as a way of pulling together all of these various pieces of information and useful resources that are currently out there.

The book walks students (and their educators) through a recent history of the replication crisis in psychology, the causes and drivers of crises (including academic incentive structures, questionable research practices, and cognitive biases), and open science reform, its benefits and associated challenges. Importantly, to ensure that students understand how to put these new practices into practice, it also includes a handy, digestible guide to implementing open science practices such as preprints, preregistration, Registered Reports, and open materials, data, and code. It is my hope that this book becomes an essential guide to navigating the replication crisis.

Pedagogic resources within the book

This book is primarily aimed at students studying psychology at both undergraduate and postgraduate level, but throughout I include useful pedagogic resources that educators can use to teach the replication crisis and open science reform. For example, there is an activity where students plan to replicate a study and can understand the various difficulties they may stumble upon in their quest. Here, they find their favourite article from the journal it was published and walk themselves through different questions to assess whether there is enough information to perform an exact replication – what is the aim of the study? Do the researchers specify their main hypotheses? What is the research design? Are the measures they used detailed enough to understand how they were scored? Do they report whether (and how) they screened or cleaned their data? (see Box 1).

Box 1. The seat of a replicator. An example pedagogic activity from “A Student’s Guide to Open Science”.

This same pedagogic activity can then be expanded to teach about open science reform. Specifically, using the notes that students have taken from their “seat of a replicator activity”, they can then have a go at preregistering their replication using a template from a verified repository (e.g., AsPredicted.org/Open Science Framework). These templates can be downloaded from each repository and students can either complete these as a formative (e.g., seminar activity) or summative assessment (e.g., written report; see Box 2).

Box 2. Pedagogic activity: preregistering a replication study.

In addition to these pedagogic resources, the book also contains top tips for implementing open science practices into student’s research training, including for preprints, preregistration, Registered Reports, and open materials, data, and code. Whilst the focus is mainly on quantitative research, it also provides links and resources to qualitative research. In addition, throughout it discusses the many different organisations and initiatives that have shaped my own learning of open science (see Box 3), thus providing a detailed overview of everything we have learnt so far from the advent of the replication crisis through to current open science reform.

Become a student of open science!

The replication crisis and open science reform is not currently embedded within the teaching of psychology. It is my hope that with all of these wonderful new and ever-evolving resources, today’s students become critical consumers of science and pave the way to creating a truly robust, transparent, replicable, reproducible and representative psychological science.

Author Bio & Contact:

Dr Charlotte Pennington is a Lecturer in Psychology at Aston University, Birmingham, a Fellow of the Higher Education Academy, and a member of FORRT. Her book, “A Student’s Guide to Open Science: Using the Replication Crisis to Reform Psychology” is available in all good bookstores and inspection copies are available from McGraw Hill for review. If you have trouble accessing it, please let Charlotte know via the contact details below and she will do her best to help.

Email: c.pennington@aston.ac.uk

Twitter: @drcpennington

Mastodon: @drcpennington@mastodon.social

Introducing Nowhere Lab

Thu, 01 Sep 2022 01:44:40 -0400

Nowhere Lab & FORRT. Nowhere Lab has partnered with FORRT as part of FORRT’s efforts to promote social justice and enable equality of access to open scholarship training! Nowhere Lab members benefit from weekly meetings where they can bring any questions they have about adopting any open scholarship practices (or anything else related to science and/or academia). Dr. Priya Silverstein (Nowhere Lab founder) is also available to answer any questions about engaging in other FORRT initiatives either during the weekly meetings, by reaching out via the Nowhere Lab or FORRT Slack workspaces, or by email. Nowhere Lab also operates informal mentorship – rather than matching mentors and mentees, people with mutual interests and skills are able to be connected and there are several channels in our Slack for seeking help or support for a variety of issues.

Background

Throughout my academic career, I loved being in a lab. I looked forward to biweekly lab meetings as a research assistant, hearing about what the Ph.D. students were up to and wondering about what my own Ph.D. would hold (if I ever got that far). When I finally got a Ph.D. position, I had not one but TWO weekly lab meetings with each of my two supervisors. In the first, we always held journal clubs – taking some of the newest and most exciting developmental science research and dissecting the theory and methods to bits. In the second, we had a mix of journal clubs, presentations, and catch-ups. A Ph.D. can be quite lonely – just you and your super-specific topic – so lab meetings were a welcome time to reflect, process, and think with labmates. During a pandemic-induced furlough from my postdoc, and when I took two alt-academic jobs after this, I really missed this chance to connect with labmates. So, I decided to create my own lab. That’s where Nowhere Lab began.

Who we are

Nowhere Lab ( http://nowherelab.com/) is an online community for people who would like the lab meeting experience but don’t currently have one. We have members from across all populated continents and career stages: undergraduate students, master’s students, Ph.D. students, postdocs, faculty members, and people working outside of academia. We hold weekly meetings and have an active Slack ( join here).

It has been a humbling and often emotional experience to hear the stories of those who have joined Nowhere Lab. Some members have joined when caught at a difficult and awkward time where they don’t feel embedded within their Ph.D. lab (e.g. they’re currently writing up) but also don’t have a new lab. Some members have joined when they get their first faculty job and don’t have any students for their lab yet. Some members have recently left academia and have mixed emotions about this transition. Some have joined because they’re considering postgraduate research but have never had any experience of research due to educational priorities in their country. One member even joined because health issues meant they weren’t able to take on new students and thus no longer had their own lab. We’re a home for the lost and are brought closer together by the stories of why we need Nowhere Lab.

What we do

What we do weekly is entirely dictated by members and has included job interview prep, advice on ongoing projects, discussions, statistical training, journal clubs, and much more. For example, we often have conversations where people who have done a Ph.D. share advice with members who are considering one. We decided to share this advice openly in a Ph.D. survival guide, in the hope that it will help people who are considering embarking on a Ph.D. or have just started. This guide covers things to know about doing a Ph.D., what’s great and not great about doing one, top tips, how to choose between two programmes, what to do when you think you’ve made the wrong choice, signs of a toxic environment, and how (not) to handle stress. As well as being useful, creating this guide was also a cathartic process for some members who had less than ideal Ph.D. experiences and wanted to share things they wished they’d known at the time.

Our goals

Nowhere Lab aims to open science up to anyone and everyone. For example, we have members who work in tech and nonprofits who would not normally consider themselves scientists or get to engage in a scientific community. In addition to advancing diversity as an essential part of open science, we often focus explicitly on methods-based open science skills and knowledge advancement too. For example, we often read papers on open science and metascience in our journal clubs.

Achievements

Although the Ph.D. survival guide is Nowhere Lab’s first “output”, we have many other success stories that have come from the lab. We have now had several Ph.D. vivas and graduations – we even got to virtually attend one member’s viva which was really interesting for members who’d never seen one before. Careers have been changed – one member got a job from a Nowhere Lab referral, and two members were offered jobs after doing interview prep with the lab. And friendships have been forged – one member even visited me and stayed at my home!

Join us

There are many reasons why someone might want to join Nowhere Lab:

New faculty who have no one in their lab yet
Ex-academics who now work in industry
Keen undergraduates and masters students
Freelance sci-commers/consultants
People who are in a toxic lab
People between jobs
…and many many more!

If you’re interested in joining Nowhere Lab, email me at priyasilverstein@gmail.com. We can’t wait to meet you!

Contact information:

Priya Silverstein [priyasilverstein@gmail.com]

Research and data lessons from a non-academic job:

Sat, 09 Jul 2022 01:44:40 -0400

Five lessons from a non-academic job

It’s been one year after leaving academia and starting to work for a charity organisation. For a bit of context, I did my PhD in Psychology, after which I spent four years across two postdocs (one in France, one in England) and I got to the point where I could no longer see my career progress in academia. There are many reasons why I came to this crossroad, but above all, I realised that my passion is with research and data, and less so with the other aspects of academic life. To my delight, my non-academic journey so far has been full of development and challenges and here are some of the lessons I shared, with some additional thoughts. Here are five things I learnt:

The three data lessons

I learnt a lot about data. Real life data is messy AF. I learnt how to join complicated datasets with missing data and how to deal with types of variables I rarely encountered like strings and dates. I cannot underestimate how neat data from academic research is compared to what real life data looks like.
I learnt how organisations store their data. Unlike any other datasets I worked with before, organisational data is dynamic. There isn’t just one database but rather multiple sets that are related to one another. You need tools like SQL to pull data from internal data systems. SQL queries are also used to power live dashboards which are used to display data in the simplest form. This is where most teams take their data from. Tools like PowerBi and Tableau are the major players here when it comes to data visualisation and internal data sharing. This was a completely new lesson to me because like most academics, I have been used to dealing with only static data, pulled from Qualtrics or arranged neatly within one Excel sheet. The good news about SQL is that I was able to pick it up relatively easily with my programming skills in R but what I still find tricky is finding the piece of data I need in columns and columns of data that aren’t always neatly labelled. I really miss having a neat codebook with all variable names listed!
I also had to revisit my thoughts on descriptive statistics. In academia, inferential stats are king because we infer something about a population from a sample. With organisational data, you have access to the whole population so descriptive stats become very powerful. I had to resist an urge to ask “yes, but is this increase statistically significant?” because if all 100 programme members answered the same question about the satisfaction in the following year, and we saw an increase of 5% in satisfaction, this in itself is informational and quantified in terms of the effect size - we don’t need inferential statistics to tell us if this difference was significant at the 0.01 level or at the 0.001 level and whether this would apply to the wider population - this isn’t the point.

The two (new) methodological approaches lessons.

I learnt about quasi-experimental approaches using matching. While in academia I often ran experiments, establishing causation to evidence the impact of established programmes and interventions is more tricky as you often can’t randomly assign participants to treatment and control. This led me to learning about quasi-experimental statistical approaches and methods like propensity score matching to estimate the likelihood of treatment and select the ideal matched pairs. Lack of random assignment is also the least of all issues. The environment in which programmes and interventions happen in the real world is far from the controlled experiments we run in labs/online. So how do you assess whether participating in a long-term teacher training programme has a real life impact for the pupils they teach? How do you establish causation and the process by which it happens? These are a lot more tricky questions than I ever thought I had to approach. It turns out there is a whole host of methodologies to address this issue - these are a lot more prevalent approaches in disciplines such as policy (think: policymakers often want to know what impact introducing policy X had on the local population but they aren’t always able to randomly assign two groups of population into an intervention with policy application versus not).
I learnt how to run a discrete choice experiment. This is a method that is often employed in marketing, I’ve rarely heard of it before but I found an excellent application of this method in the work we were doing. By designing a discrete choice experiment, we were able to assess what changes we could make to programmes we offer to increase their propensity. Would this programme be more popular if we introduced element X or increased Y? This technique looks at implicit preferences for various factors (not just two factors but many at the same time). It puts them directly in competition with one another to calculate propensity of each factor. It’s also often used in medical sciences and economics.I had a lot of fun designing this research and I am currently working with internal stakeholders to develop actions from this research - how can we turn this knowledge to attract more people to our programmes? It’s fascinating how quickly you can turn research into action.

I find the process of reflecting on these lessons somewhat therapeutic because this time last year, I was full of excitement but also filled with some worries as I was embarking on my non-academic path. These lessons tell me that I not only was able to build on my research and data skills that I developed during my academic career, but also this move has allowed me to expand my development in ways I didn’t anticipate before.

I originally went into academia because I loved this process of learning and developing, stumbling across new challenges and new approaches, but after 7 years I started getting worn out by discussing the same issues while using the same approaches and I craved new challenges. This first year of the non-academic pathway has certainly fulfilled that goal. I’m not saying that academia can’t be a fulfilling path with lots of challenges - I think it’s more about the fit. Digging deeper into what it is really about your job that makes you happy is going to lead you down the right pathways, academic or not.

Contact information: Karolina Urbanska ( k.urbanska@sheffield.ac.uk), Twitter @karo_urb, & website https://web.archive.org/web/20220607134246/https://www.karolinaurbanska.co.uk/, and link to the original thread: https://twitter.com/karo_urb/status/1529784303885885441.

Editor’s note: The present text is an adapted and expanded version of the widely shared Twitter thread by Karolina which resonated with so many of us. We invited the author if she would be willing to adapt & expand her thread to an Open Scholarship audience.

Addressing the grand challenges facing psychological science

Thu, 21 Apr 2022 01:44:40 -0400

What are the four grand challenges identified by APS members?

1. Globalization and diversity

Psychological science has been repeatedly described as WEIRD, meaning Western, Educated, Industrial, Rich, and Democratic. In particular, the persistent use of predominately White and North American college students to study human behaviour has raised concerns about the generalizability of our findings, as well as our legitimacy as a science (Arnett, 2008; Henrich, Heine, & Norenzayan, 2010; IJzerman et al., 2021; Thalmayer et al., 2021; Tindle, 2021). Others argue that we should get rid of the dichotomy altogether (Ghai, 2021).

Here’s what you can do to enhance globalization and diversity within psychological science:

Invite international speakers to participate in opportunities that historically privilege U.S. citizens (e.g., colloquium series, workshops, awards, fellowships)
Explicitly cite the achievements and ideas of Black, Indigenous, and People of Color within undergraduate and postgraduate courses, particularly Black women (e.g., see existing resources such as the BIPOC-authored Psychology Papers spreadsheet for a detailed list of available scholarship; reference Smith, et al., 2021 for a description of the Cite Black Women campaign)
Encourage people to engage with marginalized groups, including Disabled people; reference Educator’s Corner on Navigating Open scholarship for neurodivergent researchers
Acknowledge your positionality, meaning the social-historical-political aspects of a researcher that influence your orientations (see Bourke, 2014; Secules, et al., 2021) and include reflexive practice for all research pursuits, including quantitative research programmes (see Jamieson, Pownall, & Govaart, 2022)
Consider context and identity by reporting sample demographics (Sabik, et al., 2020), as well as incorporating informed frameworks (e.g., interpret results via critical race theory or open scholarship)

2. Research integrity and applicability

Many of us have been warned about the replication crisis in psychology (also commonly referred to as the replicability or reproducibility crisis, as well as the credibility revolution). Not unique to psychology, the crisis highlights that the results of many peer-reviewed scientific studies are either difficult or impossible to replicate (Open Science Collaboration, 2015). However, the legitimacy of the concern with regards to replication remains hotly contested (Feest, 2019; Maxwell, et al., 2015).

Here’s what you can do to promote research integrity and applicability within psychological science:

Challenge bias against open science as valuable or legitimate; incentivize early career and contingent scientists to participate by reshaping institutional norms or program milestones (e.g., encouraging students to pre-register dissertation studies; Bahlai, et al., 2019)
Shift mentality to encourage collaboration rather than competition (e.g., scientists working across departments or multiple working groups rather than in a singular lab at a particular campus)
Raise awareness about the unique experiences of intersectionally invisible participants (e.g., people who share two or more marginalized identities such as Black women; Coles & Pasek, 2020; Purdie-Vaughns & Eibach, 2008)
Use existing resources, such as the Open Science Framework or aspredicted.org to collaborate, document, archive, share and register research projects, materials, code, and data; join the Psychological Science Accelerator

3. Collaboration across fields and disciplines

As psychologists, we rarely cross borders, whether real or imagined (e.g., departments, universities, countries)! Collaboration across disciplines and fields, however, is extremely beneficial to psychological scientists seeking to identify connections between psychological processes, history, and broader context (2020). Take, for instance, work from Henderson and colleagues on Confederate monuments; they find that the number of lynching victims in a county is a positive and significant predictor of Confederate memorialization in that county (2021). An intersectional lens can be applied to their findings to illustrate how race and social forces such as the Civil War contributed to anti-Black racism in the U.S., particularly during Reconstruction in the South. Through this example, we can also see how the present work invites participation from other fields like American politics, history, and Africana studies.

Here’s what you can do to collaborate across fields and disciplines within psychological science:

Participate in scholarly events outside of psychology, such as the Minority Politics Online Seminar Series
Invite speakers from other disciplines to department- or University-led symposiums or conferences; provide remote options for attendance and accessibility grants to stimulate broader participation (e.g., the University of Virginia Diversifying Scholarship Conference)
Initiate cross-discipline collaborations (e.g., Gaither & Sims, 2022)
Leverage existing networks such as Academic Twitter to incorporate the views of people from relevant disciplines and achieve shared goals (e.g., a systematic co-creation of new knowledge on a specific topic)

4. Strengthening theory

In science, a theory is a well-supported explanation of natural phenomena, confirmed repeatedly through observation and experimentation. Theory changes, however, if and when evidence accumulates that the theory cannot explain. Adaptive research in psychology must therefore follow the assumption that our knowledge will expand along with our understanding of human diversity (and vice versa). Doing so will allow room for concepts such as intersectionality and open science to be integrated within psychological science. Take, for instance, Bronfenbrenner’s ecological systems theory. Frustrated with the centrality of white boys, Stern and colleagues developed a new version centered on Black girls (2021). You can see the model here:

Here’s what you can do to strengthen theory within psychological science:

Expand theory regarding the mind and behavior to represent human diversity; the development of reliable theories is essential to scientific progress (see Eronen & Bringmann, 2021 for insights into developing good psychological theories)
Compile evidence to advance new theories if and when they are no longer applicable or outdated due to history or context (e.g., COVID-19); create theories that center marginalized people and the real-world scenarios they experience in day-to-day life, controlling for relevant covariates

Join me!

It is possible that I am overoptimistic about the future of psychological science or naive to the systematic obstacles that must be overcome. Still, I would rather be naive than conform to a system that is stagnant to change. When we stand outside the Ivory Tower, aware that the system is corrupt but alone, helpless, and discouraged, we know that the master’s tools will never dismantle the master’s house (Lorde, 2018). We were meant to be kept out of the house. Ergo, the time has come to build a house of our own, a house founded on principles that reflect and affirm our identities as the marginalized, unconventional, and historically excluded. It is within our avant-garde that we will find our true power, and grapple with the many challenges requiring redress to create a stronger, more robust science.

Here’s what you can do:

Strengthen our existing community by joining our Slack group
Seek collaborations to pursue broader work exploring intersectionality, open science, and related topics and crowdsource manuscripts
Identify funding opportunities to advance our goals: research transparency, reproducibility, rigor, and ethics (e.g., Einstein Foundation Award for Promoting Quality in Research)
Spearhead initiatives (e.g., write a post for the Educators’ Corner)
Spread the word about FORRT (e.g., share this post with your colleagues, tag us on Twitter)

Contact information: Annalisa Myer ( amyer@gradcenter.cuny.edu), Twitter @MyerAnnalisa.

References

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716

Navigating Open scholarship for neurodivergent researchers

Sun, 23 Jan 2022 01:44:40 -0400

Navigating Open scholarship for neurodivergent researchers

Who we are and why is this important?

In academia, there has been much discussion about how open scholarship can benefit marginalised voices (e.g. Robertson, 2020; Pownall et al., 2020). However, neurodivergent individuals (e.g. dyslexic, autistic, ADHD) have received little attention. According to the Higher Education Statistics Agency, only 2.2% in 2003/04, 3.9% in 2012/13 to 5.5% in 2019/2020 of staff at universities in the UK disclosed having a physical or neurological disability. However, the true figures are likely to be higher. Despite increased coverage and interest regarding disability issues in academia, concerns regarding negative perceptions of disability disclosure remain. More open discussions are taking place about the lived experiences of disabilities and chronic illnesses, as people with disability are becoming less stigmatised, leading to more disclosures being discussed. However, there is still a question being asked: ‘where are the disabled academics?’ (We are here but you don’t see us!).

Playing this video loads it from YouTube (Google). See our privacy policy.

Disability? How do you have a disability? You do not have a deficit with your body, you can still function in society, earn a living and participate in community life. Disability is portrayed as a personal tragedy, with some heroic individuals “overcoming” their terrible handicap. So how are you disabled? The answer is that not all disabilities are visible. The relative proportion of disabled academics is widely underestimated (e.g. Farahar, 2021), likely due to the stigma attached to the concept of disability (Mellifont, 2021). In addition, there are too many assumptions about disability being a physical or intellectual impairment with no discussion on difficulties regarding cognition and mental health. This discussion also fails to encapsulate that the range of difficulties people face may be connected to an individual’s neurodevelopmental characteristics (Singer, 2017). This is in spite of the fact that there are conditions that are described by psychologists as disabling but people with these conditions are treated as if they are not disabled. This is the ‘neurodiverse’ group.

Neurodiversity is the non-pathological variation in the human brain regarding sociability, learning, attention, mood and other mental functions at a group level (Singer, 2017). An individual is neurodivergent if their neurology diverges from that of the neurological majority. Neurodiversity is critically relevant to the social sciences as it discusses the diverse cognitive behaviours forming the foundations of what it means to be unique and human. Importantly, the neurodiversity movement questions the assumption that all humans must conform to the same expectations in order to flourish.

So, you may still be wondering how neurodivergent individuals are disabled? The labelling of disability is a personal matter. It depends on what models we use to interpret it. Under the medical model, disability is portrayed as a flaw, weakness or a biological limitation of the individual. Put simply, it is a personal tragedy that was inflicted on you. Disability under the social theory of disability is defined not as a personal tragedy but as the result of barriers – environmental and social practices that disable, as opposed to enabling, the individual. In addition, the social model makes the distinction that an impairment is a property of an individual body, while disability is a social process. However, it is important to remember that both models are not mutually exclusive, but are required to be used together to open a constructive dialogue between able-bodied and disabled individuals, even among disabled individuals. The important message is that disabilities are dynamic and reasonable adjustments should be made for all groups.

In today’s academic environment, the economic and political system of universities, among many other industries, focus on productivity and profit. As a result, measures that focus on productivity and profit such as standards, norms, league tables, achievements and publications are becoming more and more important. Health and wellbeing and slow science are becoming less important, despite the fact that slow science and resting are important for creativity, critical thinking and understanding data, three crucial components to help accumulate knowledge. As a result of this development, burnout becomes more common and this is more prevalent among people with disabilities (Burns et al., 2021).

People with disabilities are more likely to be excluded from the academic workforce with its demands for speed, efficiency and productivity, leading to normalised, ingrained and internalised ableism to an extent that they desire to be able-bodied. As a result of this exclusion, able-bodied individuals may believe that disabled individuals do not contribute to the productivity of the community and argue that their unemployment is the ‘fate of the idle’ (Singer, 2017). This means that in academia and industry, people with disabilities become oppressed, mocked, ridiculed and perceived as a nuisance.

Many gatekeepers determine whether an individual is neurodivergent and these processes are driven by individuals who are neurotypical. As a result, referral time for these services vary widely, from 4 weeks to 201 weeks within the UK (Lloyd, 2019) and if a person does not fit the criteria, the individual can be ignored and may not receive the much-needed help that they require. This can lead to poor self-esteem, unemployment (e.g. around 22% in autistic people are in any type of employment; see Figure 2 in fact sheet). As a result, neurodivergent individuals may blame themselves for the difficulties they encounter, as opposed to the barriers that society has placed on them.

Despite this, people of different neurodivergent conditions or families of the people with the conditions have begun meeting and talking to each other about their experiences and one common shared experience is a history of misinterpretation and mistreatment by the dominant neurotypical cultures and its institutions such as academia. As a result of centuries of oppression of disabled people worldwide and a hyper-normalised environment, in addition to seeing the disproportionate impact of the coronavirus pandemic on disabled students (see this amazing paper by Dr Joanna Zawadka), many neurodivergent and disabled staff feel discouraged in an environment that should aim to support them. They do not feel like they belong, their differences are seen as an impairment and their voice does not seem to matter. They do not see themselves represented in psychological science, academia, business, teaching or elsewhere.

We are a group of early-career neurotypical and neurodivergent researchers that are a part of the Framework for Open and Reproducible Research Training (FORRT) community, aiming to make academia and the open scholarship community more open to neurodiversity. Everyone, no matter what they identify with, is welcome in this group. We aim to discuss how open scholarship can be intersected with the neurodiversity movement, and emphasise how differences should be highlighted and accepted, whilst supporting the idea of accessibility. Our neurodiversity team is a group that currently consists of individuals that have autism, dyspraxia/DCD, speech-language differences, ADHD, dyslexia, or are neurotypical allies. If you have these or other neurominorities and wish to be part of the team, you are more than welcome to join!

What do we want?

We want academia and open scholarship not to be covertly homophobic, racist, sexist and ableist. However, by attempting to adapt and work within the existing defined rules of academia and its power structures, what we do is reinforce that system, irrespective of intentions. Lorde (2018) illustrates this well in the metaphor: the master’s tools (i.e. the dynamics, language, and conceptual framework that create and maintain social inequities) never serve to dismantle the master’s house, but somehow end up building another extension of that mansion. Similarly, the fundamental assumption under the medical model of disability serves to both disempower the individual and strengthen societal perception that the neurodivergent person is the problem. Thus, we need lasting, sustainable, and widespread empowerment that can be obtained by making and propagating the shift from the medical to the social model, or use both in order to encourage a constructive dialogue. Put simply, in order to fulfil our potential, we as neurodivergent people cannot say that there is something wrong with us, and we must use the tools of the social model to temper or remove barriers provided by the medical model.

According to Feminist Standpoint theory (Harding, 1992; Pohlhaus, 2002), it is important that we empower under-represented and marginalised voices in knowledge production, and draw upon the lived experience when designing for under-represented voices. However, this is not possible, as we still follow psychology’s positivist (i.e., all authentic knowledge is scientific knowledge) tradition; this assumes that there is a fundamental truth of human diversity and that scientists should be objective. However, the research conducted on neurominorities is marred by the fact scientists are like everyone else. We are humans with different political views and lived experiences as well as biases (e.g. confirmation bias), which affect the questions that are asked and influence researchers’ degrees of freedom. Therefore, studies into neurodiversity are undertaken within structures that are characterised by power relationships, where colour, gender and neurological makeup are in no way neutral. This leads to neurodivergent individuals being treated as an _object _of the conversation, rather than the subject. This is analogous to the way the Eastern Societies and their members were treated by Western Researchers in the mid and late 20th Century (see Said’s Orientalism for further discussion; Said, 1976). “The Orient” was othered, misunderstood and seen as inferior, which led to a very skewed depiction which reflected Western biases. In post-colonial times, our awareness of power relations is, at least in some academic publications, greater than before. However, in the case of neurodiversity, there has been little change as most studies in the area are still conducted by those in possession of greater social power, with little input from neurodivergent researchers. This approach is not only patronising, but very dangerous as it will take centuries to counteract the discrimination it produces.

The power structure that dominates psychology and social sciences is from white, male and able-bodied people who treat neurodivergent people as an object of the conversation, not the subject. As argued by Jackson (1998, p.8), “Each person is at once a subject for himself or herself - a _who _- and an object for others - a what. And though individuals speak, act, and work toward belonging to a world of others, they simultaneously strive to experience themselves as world makers”. Once we consider that scientists are human, and neurodivergent people can be included in this research, then we can co-create projects that allow us to discover the truth about diversity from different perspectives. This is more commonly discussed in qualitative research but rarely even considered in quantitative research. In addition, with different perspectives, there is an emphasis on moving away from the typical White, Educated, Industrial, Rich, Democratic (WEIRD) samples that account for 80% of study samples but only 12% of the world population, despite the fact that this movement does not acknowledge the neurodiversity movement or neurominorities (e.g. autistic, ADHD, dyslexic). In addition, there is a lack of patient involvement in how to make the research more likely to improve the quality of life of neurodivergent people. The need to address this issue and to ensure disabled and marginalised individuals are directly included in research and policy-making decisions that affect them can be expressed by the commonly used slogan “Nothing about us without us”. Emancipatory and/or participatory approaches such as participatory action research (e.g. Bertilsdotter-Rosqvist et al., 2019; Fletcher-Watson et al., 2018; Grant & Kara, 2021; Leadbitter et al., 2021; Strang et al., 2019; Strang et al., 2021) have considerable potential for facilitating this type of collective knowledge creation and driving social change that benefits neurodivergent people in areas that may contrast with many mainstream research approaches.

A recent movement has become important in education: open scholarship. This reflects the idea that knowledge of all kinds should be openly shared, transparent, rigorous, reproducible, replicable, accumulative, and inclusive (allowing for all knowledge systems). Open scholarship includes all activities that are not solely limited to research such as teaching and pedagogy. One key foundation of open scholarship is accessibility, a key facet that also belongs to the neurodiverse movement (e.g. Brown & Leigh, 2018; Brown et al., 2018). Accessibility and inclusion is where your content, activities and all their components are accessible to all people with disabilities, learning differences, mental health conditions or other health conditions that may affect their learning or engagement with the materials and activities, research activities, clinical training, and teaching (Victor et al., 2021a). It highlights the importance of embracing diversity and making everyone feel welcome and valued (see information sheet). Discussions have been, however, scarce regarding not only how open scholarship can advance the neurodiverse movement, but also how it can benefit from it. It is thus a priority to build community to discuss how the neurodiversity movement can be included in open scholarship, as the lived experience of neurodivergent individuals (including encountered barriers) may help to enhance accessibility, allowing open scholarship to be truly open (Whitaker & Guest, 2020). This in turn may help to dismantle the harmful stereotypes about disabled individuals (Devendorf et al., 2021), providing more specific provisions for neurodivergent and/or disabled researchers (e.g. virtual conferences; see Levitis et al., 2021). Furthermore, including this population in academia will help promote work-life balance, by denormalising overwork and practices that lead to burnout.

There has been a recent shift towards the use of Universal Design for Learning (UDL) (i.e., an approach to teaching highlighting that academics/tutors should be proactively, not reactively, inclusive by making adjustments to their teaching without students having to disclose their disability to student disability services) in higher education (Burgstahler & Cory, 2010). UDL has several benefits: by offering a more flexible and inclusive practice, there is no need to disclose one’s disability, irrespective of student status (Clouder et al., 2020). In addition, making the assumption about the student’s intention based on your interpretation of their behaviour can be damaging for neurodivergent students’ self-esteem. University staff should recognise the different manners in which students may communicate and contribute, whilst being open to collaborating with students to find suitable approaches. Put simply, it can be described as neurodiversity involvement for pedagogy.

In addition, UDL allows students to engage in the material that best suits their learning. Traditionally, university students are assessed through essays/ dissertations, group/individual presentations or examinations with very little discussion but neurodivergent students may find these types of assignments challenging, even though they might otherwise succeed. UDL encourages educators to examine the students’ strengths, as opposed to weaknesses and allows students to have more choice. Learners could do a recorded presentation, as opposed to presenting in a group or present the skills they have acquired on the course in a different form that may suit them better. This would benefit the students in terms of better preparation for employment, by focusing on the student’s ability and professional values, as opposed to the challenges. This approach is also fully aligned with the UK Professional Standards Framework in Higher Education (UKPSF) as it facilitates educators’ continuous development (A5), focuses on respecting individual learners and diverse learning communities (V1), promotes participation in higher education and equality of opportunity for learners (V2) and acknowledges the wider context in which higher education operates recognising the implications for professional practice (V4). Another example is that attendance should not be used as a marker of class engagement, as there are several variables that predict class attendance (e.g. keeping pace with other students, attentional demands, and attendance being socially impossible). At the start of class, educators should signpost the expected outline and inform students before an activity is finished or changed. This lack of physical attendance and/or inability to follow attentional cues is not evidence of a lack of engagement but that students may engage in ways that are not expected and we should meet them where they are now, as opposed to where we expect them to be.

An important core property of UDL is to provide choice to allow students to develop agency in their own learning. Lecturers may feel that academic standards will not be maintained or students will not learn the learning outcomes but this choice aims to remove structural barriers that are faced when perhaps making a specific activity unnecessarily difficult, as opposed to reducing the academic level. For instance, lecture capture allows the student to learn in an environment that suits them and to learn at their own pace. Over decades, lecturers have questioned the effectiveness of lecture capture (Nordmann et al., 2021), but students such as dyslexic students, who otherwise struggled, may engage with learning and develop at their own speed (Nightingale et al., 2019). Rather than develop concerns on whether students will continue to engage, UDL offers students an opportunity to develop agency in their learning, a goal that lecturers should encourage. And applied more broadly to the academic employees’ relationships, UDL can also improve the academic culture, providing academic workers with opportunities to engage in their trade in a way that fits their neurocognitive style. Last but not least, UDL promotes and facilitates social justice and equality.

Finally, we want academia to approach neurodiversity in the same way that true cosmopolitans approach cultural diversity. We want academics to reject the idea that the lived experiences of neurominorities such as dyslexia, autism, ADHD, which differs from the neuromajority, should be pathologised. Rather, these experiences should be accepted as fundamental to the human experience, to allow us to have different perspectives to understand what it means to be human. As a result, by considering this perspective, ‘‘our strengths and deficits will shape, not deny, our humanity’ (Grinker, 2010, p.173).

Put simply, our team wants people in academia and the open scholarship community to understand that: “Being disabled does not make a person any less of a scientist. Actively listen to your students and/or peers if they disclose their disabilities, as they have entrusted you with sharing this important part about themselves. **If you are in a position of power and a disabled person asks for accommodations, give it to them. **Learn to recognise that disabilities are unique and dynamic. Ultimately, we should strive for universal design, of both our workspaces and pedagogy, as this gives everybody the best chance to fully be themselves and blossom in science” ( Middleton, 2021).

Our plans

Our overall aims are to: reduce the stigma neurodivergent individuals face by raising awareness of the contributions neurodivergent researchers have made, encourage neurodivergent researchers that there is no need to mask or hide our differences, and show that neurodiversity is an asset to open scholarship and academia. Open scholarship, like psychological science, benefits from neurodiversity and disabled perspectives, which have been growing in the past decade (e.g. Chown et al., 2017; Gillespie-Lynch et al., 2017; Grant & Kara, 2021; Kapp, 2020).

Currently, we are in the initial stages of creating a database of papers written by neurodivergent researchers. This will be a comprehensive crowd-sourced database of research written by neurodivergent researchers with the aim to allow educators to diversify their syllabi to include neurodivergent researchers to help counteract the bias towards able-bodied researchers. This database will help students feel that they can belong in academia, and that neurodivergent or disabled people can have some important strengths in research contexts (Grant & Kara, 2021). This project aims to be a “living“ resource, which will be regularly updated to include new studies authored by neurodivergent researchers.

In addition, we will engender a survey which will be similar to Victor et al. (2021b) and Devendorf et al. (2021). Their surveys investigated prevalence of lived experience, discussed the lived experiences and stigma (e.g. attitudes towards self-relevant research, help-seeking, disclosure) of mental health challenges among applied psychology researchers. The neurodiversity survey will detail the proportion of neurodivergent students and researchers (e.g. undergraduate students, graduate students, post-doctoral researchers) from different countries and different disciplines. It will investigate experiences in academia, possible concerns and barriers (e.g. external stigma, self stigma and difficulties in help-seeking) of neurodivergent students and researchers. In addition, we will discuss attitudes towards the neurodiversity movement, open scholarship, and the intersectionality between neurodiversity and open scholarship. We believe our work will improve neurodiverse representation and awareness. More importantly, we hope our work will also promote the inclusion of neurodiversity within the scientific community and the next generation of neurodivergent scientists. By including a neurodiverse population, “we have a huge opportunity to not only advance our science but also to equitably serve all of humanity” (Ghai, 2021, p2). Thus, we can encourage ourselves to broaden our minds and to truly be “open” in open scholarship.

Finally, we are also currently planning a manuscript that will take the form of a position statement, detailing how neurodiversity and open scholarship can benefit from each other. With these additional resources to combat the biases, prejudices and misconceptions of disability, you may be surprised what we can bring to the table and we hope these resources can enable us to do just that.

Join us!

We need to discuss the challenges faced by neurodivergent people, so they can feel seen. It is a wonderful experience to be seen and heard. We feel sad that not all neurodivergent people will encounter the same privileges that we have encountered. They feel these things so hard and this impacts their mental health, and in turn, leads to very upsetting ends. Neurodivergent people have been marginalised in so many ways that we have missed so many opportunities to make their lives that much better. We cannot change what has happened in the past; however, we hope that the resources we create will help improve representation early in students’ careers, so then they can hold their head high and smile, knowing they do not need to mask anymore. Also, they can feel proud that they can be themselves, achieve things with the correct reasonable adjustments and be treated as an equal, not an inferior. Put simply, we would like ally neurotypical academics of the neurodiverse movement and disabled academics to join us at FORRT, so we can provide “more effective support…[including] specific provisions for disabled researchers, online conferences and the ability to work from home” (Niedernhuber et al., 2021 p.34). These provisions need to continue post-COVID-19 pandemic because we cannot return back to normalcy. Not because it is not possible, but because the conversation about disability, race and gender has truly started and we are not letting this conversation end until everyone feels included in their environment. We want everyone not to just survive, but to thrive and flourish in their respective environment.

History of Team-Neurodiversity

This project began on the 24th June 2021 meeting of the Society for the Improvement of Psychological Science in a roundtable discussion, “How can open scholarship support evidence-based learning in people with neurodiverse conditions?” led by Dr. Tamara Kalandadze and Dr. Mahmoud Elsherif. During this roundtable discussion, Dr. Amélie Gourdon-Kanhukamwe attended this discussion. As a result of a fruitful discussion, an initial framework was developed that led to a discussion about how open scholarship and neurodiversity intersect, and on the 28th of June 2021, Team Neurodiversity was created. However, Team Neurodiversity did not have a home to discuss open scholarship and neurodiversity topics. The Framework for Open and Reproducible Research Training (FORRT) kindly offered us a base so we could establish ourselves and become a tour de force in this area. Although it started with five members, it has grown to a smashing total of 56 members based in more than 8 countries and is still increasing! The team has finished working on a database on neurodivergent scholars; a blog about open scholarship and its intersection with neurodiversity, which has been expanded to a position statement manuscript; a manuscript accepted in the British Psychological Society Cognitive Psychology Bulletin and a manuscript accepted in the Association for Psychological Science. Currently, we are developing a survey to assess masking and neurodivergence in academia and the convergence between open scholarship values and neurodivergent values. We have a rotating leadership team which changes every six months. To promote diverse and inclusive leadership, anyone can put themselves forward for this role, regardless of their experience. The current team leaders are Magdalena Grose-Hodge and Bethan Iley. Thank you to our previous team leaders: Amélie Gourdon-Kanhukamwe, Flávio Azevedo, and Mahmoud Elsherif.

References

Brown, N., Thompson, P., & Leigh, J. S. (2018). Making academia more accessible. Journal of Perspectives in Applied Academic Practice, 6(2), 82–90. https://doi.org/10.14297/jpaap.v6i2.348
Pownall, M., Talbot, C. V., Henschel, A., et al. (2020). Navigating Open Science as Early Career Feminist Researchers. https://doi.org/10.31234/osf.io/f9m47
Whitaker, K., & Guest, O. (2020). #bropenscience is broken science. The Psychologist, 33, 34–37.

Other references

Bertilsdotter Rosqvist, H., Kourti, M., Jackson-Perry, D., Brownlow, C., Fletcher, K., Bendelman, D., & O’Dell, L. (2019). Doing it differently: emancipatory autism studies within a neurodiverse academic space. Disability & Society, 34(7-8), 1082-1101. https://doi.org/10.1080/09687599.2019.1603102

Botha M. (2021). Academic, Activist, or Advocate? Angry, Entangled, and Emerging: A Critical Reflection on Autism Knowledge Production. Frontiers in psychology, 12, 727542. https://doi.org/10.3389/fpsyg.2021.727542

Brown, N., & Leigh, J. (2018). Ableism in academia: where are the disabled and ill academics?. Disability & Society, 33(6), 985-989. https:/doi.org/10.1080/09687599.2018.1455627

Burns, K. E., Pattani, R., Lorens, E., Straus, S. E., & Hawker, G. A. (2021). The impact of organizational culture on professional fulfillment and burnout in an academic department of medicine. PloS one, 16(6), e0252778. https://doi.org/10.1371/journal.pone.0252778

Burgstahler, S. E., & Cory, R. C. (2010). Universal design in higher education: From principles to practice. Harvard Education Press.

Chown, N., Robinson, J., Beardon, L., Downing, J., Hughes, L., Leatherland, J., … & MacGregor, D. (2017). Improving research about us, with us: a draft framework for inclusive autism research. Disability & society, 32(5), 720-734.https://doi.org/10.1080/09687599.2017.1320273

Clouder, L., Karakus, M., Cinotti, A., Ferreyra, M. V., Fierros, G. A., & Rojo, P. (2020). Neurodiversity in higher education: A narrative synthesis. Higher Education, 80(4), 757-778. https://doi.org/10.1007/s10734-020-00513-6

Devendorf, A., Victor, S. E., Rottenberg, J., Miller, R., Lewis, S., Muehlenkamp, J. J., & Stage, D. L. (2021, November 7). Stigmatizing our own: Self-relevant research is common but frowned upon in clinical, counseling, and school psychology. https://doi.org/10.31234/osf.io/szg5d

Farahar, C. (2021, May 13) How can we enable neurodivergent academics to thrive? LSE Higher Education Blog.

Fletcher-Watson, S., Adams, J., Brook, K., Charman, T., Crane, L., Cusack, J., … & Pellicano, E. (2019). Making the future together: Shaping autism research through meaningful participation. Autism, 23(4), 943-953. https://doi.org/10.1177/1362361318786721

Gillespie-Lynch, K., Kapp, S. K., Brooks, P. J., Pickens, J., & Schwartzman, B. (2017). Whose expertise is it? Evidence for autistic adults as critical autism experts. Frontiers in psychology, 8, 438. https://doi.org/10.3389/fpsyg.2017.00438

Grant, A., & Kara, H. (2021). Considering the Autistic advantage in qualitative research: the strengths of Autistic researchers. Contemporary Social Science, 16(5), 589-603. https://doi.org/10.1080/21582041.2021.1998589

Grinker R R. (2010). Commentary: On being autistic and social. Ethos, 38 (1), 172-8.

Harding, S. (1992). Rethinking standpoint epistemology: What is" strong objectivity?". The Centennial Review, 36(3), 437-470. https://doi.org/10.1371/journal.pone.0175967

Jackson, M. (1998) Minima Ethnographica: Intersubjectivity and the Anthropological Project. Chicago: University of Chicago Press.

Kapp, S. K. (2020). Autistic community and the neurodiversity movement: Stories from the frontline. Springer Nature.

Leadbitter, K., Buckle, K. L., Ellis, C., & Dekker, M. (2021). Autistic self-advocacy and the neurodiversity movement: Implications for autism early intervention research and practice. Frontiers in Psychology, 782.https://doi.org/10.3389/fpsyg.2021.635690

Levitis, E., Van Praag, C. D. G., Gau, R., Heunis, S., DuPre, E., Kiar, G., … & Maumet, C. (2021). Centering inclusivity in the design of online conferences—An OHBM–Open Science perspective. GigaScience, 10(8), giab051.https://doi.org/10.1093/gigascience/giab051

Lloyd, T. (2019). An audit of ADHD service provision for adults in England. https://www.adhdfoundation.org.uk/wp-content/uploads/2019/07/Takeda_Will-the-doctor-see-me-now_ADHD-Report.pdf

Lorde, A. (2018). The master’s tools will never dismantle the master’s house. Penguin UK.

Mellifont, D. (2021). Ableist ivory towers: a narrative review informing about the lived experiences of neurodivergent staff in contemporary higher education. Disability & Society, 1-22. https://doi.org/10.1080/09687599.2021.1965547

Niedernhuber, M., Haroon, H., & Brown, N. (2021). Disabled scientists’ networks call for more support. Nature, 591(7848), 34-34. https://doi.org/10.1038/d41586-021-00544-8

Nightingale, K. P., Anderson, V., Onens, S., Fazil, Q., & Davies, H. (2019). Developing the inclusive curriculum: Is supplementary lecture recording an effective approach in supporting students with Specific Learning Difficulties (SpLDs)?. Computers & Education, 130, 13-25. https://doi.org/10.1016/j.compedu.2018.11.006.

Nordmann, E., Clark, A., Spaeth, E., & MacKay, J. R. (2021). Lights, camera, active! appreciation of active learning predicts positive attitudes towards lecture capture. Higher Education, 1-22. https://doi.org/10.1007/s10734-020-00674-4

Pohlhaus, G. (2002). Knowing communities: An investigation of Harding’s standpoint epistemology. Social epistemology, 16(3), 283-293. https://doi.org/10.1080/0269172022000025633

Said, E.W. (1978). Orientalism. New York: Pantheon

Singer, J. (2017). NeuroDiversity: The Birth of an Idea. Amazon Kindle eBook, self-published.

Strang, J. F., Klomp, S. E., Caplan, R., Griffin, A. D., Anthony, L. G., Harris, M. C., … & van der Miesen, A. I. (2019). Community-based participatory design for research that impacts the lives of transgender and/or gender-diverse autistic and/or neurodiverse people. Clinical practice in pediatric psychology, 7(4), 396 .https://doi.org/10.1037/cpp0000310

Strang, J. F., Knauss, M., van der Miesen, A., McGuire, J. K., Kenworthy, L., Caplan, R., … & Anthony, L. G. (2021). A clinical program for transgender and gender-diverse neurodiverse/autistic adolescents developed through community-based participatory design. Journal of Clinical Child & Adolescent Psychology, 50(6), 730-745. https://doi.org/10.1080/15374416.2020.1731817

Victor, S. E., Schleider, J. L., Ammerman, B. A., Bradford, D. E., Devendorf, A., Gruber, J., … Stage, D. L. (2021a, July 12). Leveraging the Strengths of Psychologists with Lived Experience of Psychopathology. https://doi.org/10.31234/osf.io/ksnfd

Victor, S. E., Devendorf, A., Lewis, S., ROTTENBERG, J., Muehlenkamp, J. J., Stage, D. L., & Miller, R. (2021b, July 12). Only human: Mental health difficulties among clinical, counseling, and school psychology faculty and trainees. https://doi.org/10.31234/osf.io/xbfr6

The PaPOR TRaIL Course

Mon, 10 Jan 2022 01:44:40 -0400

Open research is an umbrella term incorporating a range of principles and practices that make research more transparent, reproducible and accessible to everyone in the society. Open and transparent research practices, including research conduct and dissemination of findings, are essential to both those who produce and those who engage with research can access knowledge. Such principles and practices are relevant at all stages of the research cycle and are applicable to all disciplines, though it has been predominantly highlighted in the sciences and medicine. Further, though the principles of open research are consistent across disciplines, open research practices can be implemented differently across disciplines, thereby enhancing the transparency and robustness of research within and across disciplines

There are benefits of open research for individual researchers, including improved research quality and discoverability, shorter publication embargo periods, and reduced engagement in questionable research practices (QRPs). Open research practices also support the democratisation of scientific knowledge, and enables a greater number of people to engage with research. Training and education in open research principles and practices at early stages of individuals research journey is essential to maximise use of open research practices contributing the aforementioned benefits. Undergraduate and postgraduate students in particular can benefit significantly from training and education in open research at these formative stages. To date, open research educational resources have typically been aimed at PhD students, postdoctoral and more senior researchers, rather than undergraduate and master’s-level students, despite this learning period serving as the foundation to everything that follows.

The Principles and Practices in Open Research: Teaching, Research, Impact and Learning (PaPOR TRaIL) course was developed to provide a foundation in best scientific practice that benefits students, universities, and research as a whole. Our focus was to develop an open educational resource that would provide a comprehensive introduction to open research, and to help students incorporate open research in their research projects.

PaPOR TRaIL is a freely available online course that was developed with students and research supervisors and funded by a National Forum for the Enhancement of Teaching and Learning in Higher Education award. Though primarily designed for undergraduate and Masters level students, the course is also applicable for PhD students and researchers looking for an introduction to open research. The course includes text-based materials, visuals, videos, templates, real-world examples, and activities for students to complete.

The course is comprised of an introductory module that introduces students to what open research is, why it is important, and how to do it. This introductory module can be completed by students as a standalone self-directed module that they complete in their own time. The introductory module is also designed so that it can be used as a Lecture that can be integrated into existing research courses by educators. The introductory module sections are shown below

Once students complete the introductory module they can complete any or all of the six practice-based modules that provide a more hands-on approach to _doing _open research Students do not need to complete all six modules, some modules may be more or less relevant to the stage of their research project and they can always come back to do other modules at a later date. When students complete the introductory and all practice modules, they receive a certificate of completion. The six practice-based modules are shown below:

The course is openly available to anyone who is interested, and is open for free enrolment here: PaPOR TRaIL

The course also has an open licence (CC BY), which means the course content can be re-used and remixed to suit different educator and institutional/organisation needs, provided attribution is given to the PaPOR TRaIL team.

To download course content, please enrol at the PaPOR TRaIL site.

All course materials will be made available on our OSF page in 2022, following any refinements required in the course based on initial student feedback.

Details of the course development can be found here: https://hrbopenresearch.org/articles/3-84

Further information, including our PaPOR TRaIL video, can be found here: https://osf.io/863ks/

Contact information

The PaPOR TRaIL Course

Dr Karen Matvienko-Sikar, , School of Public Health, University College Cork, on behalf of the PaPOR TRaIL team.

The Open Research Toolkit

Sat, 13 Nov 2021 10:44:40 -0400

Open research helps increase the visibility of important discoveries and extend them to new places. It also helps to increase participation in research by underrepresented and underfunded groups. The open research ecosystem extends beyond just open access to data and publications. The principles of openness in research should extend to the entire research lifecycle, including education. I have chosen the term ecosystem because open research is an interconnected system. This system consists of philosophies, concepts, principles, practices, tools, and resources which when taken together are intended to make the research process open, transparent, accessible, and reusable by anyone.

I have been the Data Curation Librarian at University of Tennessee, Knoxville, Libraries since 2013. During this time, I have helped hundreds of researchers meet the requirements of research funders and publishers to make their research data openly accessible. I have taught countless webinars and workshops on data management best practices, FAIR data principles, and open data. While open data is a major pillar of open research, it is only one component of the broad landscape of open research. I felt it would be a disservice to the research community I served if I did not situate my work within the larger context of open research and embrace the principles and practices in my own work. This began my quest to learn more about the open research landscape. To have the focused time to learn and to develop a resource for others, I applied and was approved for faculty development leave to devote one full semester to developing a series of training modules on various topics within open research. They were partly for my own benefit to learn the landscape and partly for the benefit of others who I was sure would be interested in learning more about the topic. The resulting product is the Open Research Toolkit (ORT).

Feedback on the ORT has been positive. My primary goal in creating this was to help librarians skill up on open research principles and practices. The feedback from this audience has been especially positive and grateful. Colleagues have indicated that the ORT’s publication is timely as they are teaching a course on open research or are implementing internal training in their library next semester. I have also heard from domain specialist scientists who have noted it would be helpful as a first introduction to open research concepts for non-academic audiences or early career researchers. My hope for the ORT is that it could be used as a continuing education resource or supplemental materials in courses across a wide range of sciences and social sciences disciplines. To encourage adoption and reuse, I decided to assign it the least restrictive Creative Commons License (CC-By) to all original materials.

The process of creating the ORT spanned about a year. I initially began reading and collecting resources on myriad open research topics around January of 2021. All resources were added to an EndNote library organized by topic. I then methodically went through these resources and created a list of main topics I wanted to cover. Once my faculty development leave officially began, I actively created the modules on the identified topics. The first step was creating an outline for each module which pulled in content from all the resources gathered. Next, I created PowerPoint slides for each module. Then I created a script for the modules for someone to follow if they wanted to deliver the module. Then I created a narrated YouTube video of the content of each module to which I added subtitles for accessibility. Finally, I created Google Slides of each module and narration files in PDF and Word formats. All these materials are available on the Open Science Framework.

Each module contains a slide deck in both PowerPoint and Google Slides, presentation notes in docx and pdf, a narrated video of the slides, and a bibliography of resources related to the topic.

All videos are publicly accessible on YouTube, and the video files are available for download from the Open Science Framework. Videos on YouTube contain both English and Spanish subtitles. ORT has been assigned a Creative Commons 4.0 Attribution ( CC-By) license to all original work in the ORT, so anyone can use and adapt however they like with attribution to the author.

The modules in the Open Research Toolkit currently cover the following topics. Links to each module’s materials are provided.

Module 1: The Open Research Ecosystem [ Video, PowerPoint, gSlides, Notes]
Module 2: Principles & Practices of Open Research [ Video, PowerPoint, gSlides, Notes]
Module 3: The Philosophical Underpinning of Open Research [ Video, PowerPoint, gSlides, Notes]
Module 4: Open Access to Published Research [ Video, PowerPoint, gSlides, Notes]
Module 5: Open Access to Research Data [ Video, PowerPoint, gSlides, Notes]
Module 6: FAIR Data Principles [ Video, PowerPoint, gSlides, Notes]
Module 7: Reproducibility of Research [ Video, PowerPoint, gSlides, Notes]
Module 8: The Five Rs of Open Research [ Video, PowerPoint, gSlides, Notes]
Module 9: Open Peer Review [ Video, PowerPoint, gSlides, Notes]
Module 10: Open Licensing of Data & Software [ Video, PowerPoint, gSlides, Notes]
Module 11: Open Advocacy [ Video, PowerPoint, gSlides, Notes]
Module 12: Citizen Science [ Video, PowerPoint, gSlides, Notes]
Module 13: Open Policies [ Video, PowerPoint, gSlides, Notes]
Module 14: Open Education Resources [ Video, PowerPoint, gSlides, Notes]

More resources and additional modules may be added over time. Future resources and modules may include topics such as open research ethics, open source software and collaborative platforms, and creating more detailed curricula for those interested in using the ORT for a course. Please let me know how you use the material so I can track its adoption and use.

In the spirit of openness, if you have ideas for additions or would like to collaborate on new modules, please contact me (info below).

All resources in the ORT can be found at the following DOI: https://doi.org/10.17605/OSF.IO/A4FTW.

All videos are on the ORT channel on YouTube at the following DOI: http://doi.org/10.7290/ORT_Videos.

Contact information:

Open Research Toolkit

Christopher Eaker

Data Curation Librarian, University of Tennessee Libraries

Innovative Mentoring

Mon, 13 Sep 2021 10:44:40 -0400

In academia there are some common ‘good practice’ mentoring things. Many of us do the ‘typical’ activities that good mentors do: weekly group meetings, individual meetings, open door policy, practice presentations, intellectual engagement, iterative writing feedback on theses, grants, and papers, collaborative interactions, and promoting work-life balance. But we can do more –much more– than the basics, without it taking a ton of time. Here are some innovative ways to up your mentoring game.

Get more out of group meetings. Like many labs, we meet each week. But we rotate what we do, each in approximately equal measure: standard journal article discussion, discussion of a research article on Equity/Diversity/Inclusion (EDI), “Slide Improv”, and Fact or Fiction.

EDI articles are chosen on a rotating basis by students. They are usually primary research articles. We conscientiously choose papers from diverse areas of EDI. These discussions have been fantastic, and have led to changes in how we do and discuss things. An added benefit is that this is a visible way to show the group that EDI is important to me as their supervisor.

Slide improv gives experience thinking on the fly. One of the greatest fears students have in presenting is “what if I screw up” or “what if I forget what to say,” and slide improv gives them actual practice in how to respond to, and be more comfortable with, those moments. To do slide improv, a student makes a five-slide presentation (no animations). Another student gives it. The topic is supposed to be on something that isn’t our main research area, but does need to be something in science. They typically are a presentation of the background and results of a single publication. The student who is presenting doesn’t have to be accurate, but what they say has to be believable. Students are nervous doing this the first few times, but over time I’ve seen a marked improvement in confidence and presenting skills.

Fact or fiction gives experience in critical thinking. A student gives (brief) details and results for three papers. But one of them is completely fabricated. As a group, we try to figure out which is the fake one. This is fun, and seeing the gears turn as people sort things out is great.

To get a handle on how my students are doing, and if there are issues in the lab that need addressing, I do an anonymous poll once per year. The poll covers a range of topics on the quality of life, what is working, and what isn’t. This has been VERY HELPFUL in identifying things that need fixing (and some surprises of things that I thought were issues but are fine). I use a variation of this:

This form comes from Google Forms. Loading it sends your IP address to them and they may set cookies. See our privacy policy.

Open on the provider's site

If you would like the direct link to the Google Forms, please follow this link.

Over time, I forget which new students I’ve given ‘key advice’ and expectations. So I made a document that I give to all new students. There is a different document for undergraduate Honours Thesis, MSc, and Ph.D. students, since my advice and expectations for those students differ. The document has basic things such as how many weeks ahead to give me drafts and that all students are expected to attend group meetings, and life things like you will work long hours sometimes, but this should be the exception, not the rule.

Reading papers is important but not urgent, so gets delayed. Every 6 months I require a list of papers the students have read, annotating why they might cite it in their thesis. The expected number of papers is scaled over time. The annotations are really helpful to keep on top of reading, as a searchable document when writing, and for me to see what they are reading. Plus, with a large lab studying diverse topics, this really helps me see some papers I otherwise may miss, too! Every single student I have had make these annotations has said this was incredibly useful when writing their thesis and that they were glad they had done it.

Lastly, don’t be afraid to try out new ways of mentoring, and toss things that don’t work. I’ve tried a lot of things that just didn’t work for my group, or didn’t work for this group. I touch base frequently to see what works and doesn’t, and student’s like that I ask and it’s adaptable.

Editor’s note: The present text is an adapted version of a widely shared Twitter thread that resonated with so many of us. We thought it was of general interest and deserved to be memorialized, and hence we approached Amanda to adapt the thread to post it here.

Qualitative Open Science Practices

Mon, 05 Jul 2021 10:44:40 -0400

Starting with Qualitative Open Science Practices

As a person who was trained primarily in research using quantitative methods but needed to do qualitative research to answer the next most pressing question, I was increasingly learning about qualitative research and how to conduct it properly. However, I had not seen much information about how to conduct qualitative research using open science practices. During my PhD, I had been introduced to open science practices as a way to improve educational research, but almost everything I saw did not seem to align with qualitative research. I asked colleagues what existed with regard to open qualitative research resources. Answer: no clue.

Call to action

We thus gathered people interested in qualitative open science research in education at the Virtual Unconference on Open Scholarship Practices in Education Research sponsored by the Center for Open Science. At the conference, we brought other educational researchers and open science fans together to debate what open science qualitative research might look like, put together a list of publicly available publications and tools, and figure out what to do with this information. By the end of a few of these hackathon sessions, we had a list of videos, websites, and publications to help the community understand how to do this work. The full list is located on the Open Educational Resources website but will be expanded upon here in this blog. This blog is intended to be a soft entry into this space of qualitative open science research, not a comprehensive journey; take the thoughts below as suggestions if they align with your philosophy, project, and Institutional Review Board (IRB).

Output

We started the resource list with articles that focus on the basics of open science qualitative research. These articles (and Twitter threads) dive into some of the reasons why open science work is important, such as opening the ‘Black box’ of research or helping replication in later studies. And others also touch upon some of the murky issues of doing this work, like the fact that quant and qual researchers often have different ways of viewing the world and what truth is or can be. This is a good starting point so that you know that there will likely not be consensus on how to do this work well. However, if you read those and still want to try to incorporate open science into your qualitative projects, you can try this in any or all of the following areas: transparency/rigor, open materials, open access, and ethics.
Transparency & Rigor. For transparency and rigor in qual research, this revolves around having enough information in your research so that other researchers A) know about the researcher(s) and their positionality and B) could do a similar study in the future, also known as replicating the study. For example, in the COREQ guidelines, there is an entire domain dedicated to transparency regarding the researchers and their backgrounds (see picture below). Researchers must also describe the context, questions, use of theory, sampling method, interview techniques, and analysis in a way that someone could imagine themselves performing those tasks. One open science practice that qualitative researchers could use to some degree before even conducting the study is preregistration. Essentially it’s like the dissertation proposal where you lay out your plan in advance. Preregistration (and registered reports, the preregistration done with a specific journal with a publication agreement) helps to ensure that a researcher does not change their stated methods after the study is completed. Explaining every detail of the study might be difficult if you are under space constraints (as with publication length requirements), which brings us to #3: open materials.

Open Materials. Thanks to the internet, researchers have websites and repositories where they can upload the tools for others to access. In qualitative research, this might mean interview protocols, memos, coding notebooks, tools (such as Nvivo or R packages), or even the data itself. This provides a sort of audit trail so others can verify the results of the research. There is no all or nothing here; open materials, much like the rest of these open science practices, exist along a spectrum. Not only what researchers share is on a spectrum; researchers can also dictate who may access the open materials. Perhaps it’s the entire public, but it could just be people who want to verify findings (i.e., dissertation committees, participants, reviewers). Below you can see how Bowman and Keene (2018) described open science practices as a layered onion with the innermost layer being the most transparent. However, no matter what or to whom materials are shared, researchers must include their plan within their consent procedures and IRB protocols to not violate any ethical boundaries.

Ethics. Ethics must be considered with any research, but given the often close relationships with participants, potentially sensitive data revealed, and the fact that large amounts of data could reveal participants’ identity, ethics are huge when it comes to qualitative research. Resources both for abstract and practical purposes can be found in each section of the resources document (i.e., ethics specific to data management) and general ethical considerations are only within this section.
Open Access. Finally, researchers can consider how to share their research study. Publishing within traditional academic journals can limit who can read their work, so journals have begun providing open access tiers where researchers can pay the publisher so that the article is available to the public. Alternatively, researchers can “publish” their work online in a preprint server before it is published in a traditional publication outlet. Pre-prints can help researchers get their work out and get community feedback, although certain publications will not accept articles that have been put in a preprint server or require authors to change the pre-print or the settings on it. Open access as an open science practice appears to be generally the same for both quantitative and qualitative work, so fewer resources are here as they can be found on most open science pages.

Contact us

Our team who created this (Rachel Renbarger, Sondra Stegenga, Thomas Lösch, Sebastian Karcher, & Crystal Steltenpohl) hopes that you’ll find this helpful for doing this work! We are continuing to explore this area further, so you can reach out to me at rachelrenbarger (at) gmail.com if you are interested in contributing to the conversation.

Making lemonade out of remote teaching

Mon, 24 May 2021 10:44:40 -0400

Making lemonade out of remote teaching

2020 was a year of big changes for the whole world. For the academic community, it was not different. The need to rapidly switch from in-person to online teaching represented a big challenge for most educators. However, with new challenges come also new and big opportunities. Assistant Professor of Economics at Stockholm University, Johannes Haushofer, saw in remote teaching the opportunity to open classes to students from low and middle-income countries.

Remote Student Exchange was then created as a volunteer and non-profit initiative to match professors willing to offer spots in their classes with interested students from low and middle-income countries. To facilitate the matching process, a website was launched. It has been a great success so far. Two weeks after its launch, there were 1701 students and 88 professors registered and more than 30 courses offered in varied disciplines such as Economics, Business, Psychology, Political Science, Neuroscience, and many others. You can browse the available courses here.

At FORRT, we believe that one of the most overlooked benefits of integrating open and reproducible scholarship into higher education is that of social justice. Academia is still a place of privileges that many cannot afford. We highly commend this initiative and in the hopes of helping it grow even further, we have invited Johannes Haushoffer to explain a bit more how this idea came to be and to share his initial experiences with the community.

The Remote Student Exchange in a nutshell

If you are teaching a remote course in the next few months, we highly encourage you to consider taking part in this initiative. Students can sign up free of charge, but they do not receive official grades or credits. This means that this initiative is informal and in most cases, it should not require permission from universities or department chairs.

To participate, visit the website https://web.archive.org/web/20260115081240/http://remotestudentexchange.org/. In the home page (see figure below) you can sign up as a professor.

After confirming your email account, you will be able to set up the details of the course you want to offer. As shown in the figure below, you will be asked to indicate the subject area, the level of the course (e.g., bachelor, master, Ph.D. level), and any prerequisites for attending the course (e.g., specific courses and areas of knowledge). You should also provide the link to where the classes take place (e.g., Zoom).

Importantly, you are flexible to decide how many students you can host and how they can participate (see the figure below showing the settings for course availability). You can accept students to actively participate with questions, discussions, and assignments, and/or you can accept students to audit silently only. You can also decide to review applications to your course and choose which students to accept or to admit enrolled students on a first-come-first-served basis.

Interested students also have to sign up on the website. They will be able to browse the courses being currently offered and apply to take part in a class, provided it still has free spots and that they fulfill the prerequisites (if any).

If you have more questions, make sure to access the FAQ and watch the video below where Johannes explains a bit more about this initiative. We hope that many more scholars consider joining it. Share this idea widely with other educators and with students that may be interested. You can find a link to the countries contemplated by this initiative here.

Playing this video loads it from YouTube (Google). See our privacy policy.

Video Interview Direct Link to YouTube

Teaching the why and how of replication studies

Tue, 04 May 2021 01:44:40 -0400

Psychological science is in the midst of a “credibility crisis” in which its practitioners re-examine their practices and re-define what constitutes study rigor. Replication studies have formed a critical role in motivating this sense of crisis – a sense of crisis that has led directly to the current movement to improve psychological science through a “credibility revolution”.

Despite this important role, when I was invited to hold a virtual two-day, 10-hour workshop on replication studies for the students and faculty of the Department of Social and Organizational Psychology at ISCTE in Lisbon, I realized that I did not have any ready-made teaching materials on this important topic. This blog shares the guiding principles of the workshop and my finished materials so that you, the reader, can learn from my experiences.

My workshop materials, including a syllabus, suggested readings, and exercises, are freely available at https://osf.io/m9bzh/

Workshop overview

My workshop describes the why and how of replication studies: why a researcher might want to conduct a replication study and how a researcher should go about conducting a replication study. It also emphasizes some issues that are often neglected in discussions of replication studies, at least in my experience, including the importance of choosing good replication targets, the importance of resource constraints in doing good sample size planning, and the use of simulation studies in sample size planning.

The workshop proceeded in four modules, as shown in this workshop slide:

Each module is structured around one or two learning goals, or big-picture takeaway points that I wanted the students to understand at the end of the module. Each module also breaks up lecture sections with independent or guided exercises. The exercises are intended to both reinforce the content of the lecture and give the students hands-on experience with a broad swathe of the skills that go into replication research.

The remaining sections of the blog will describe the content of each module, how this content reinforces the module’s learning goals, and the exercises I used for each module.

Module 1: The credibility crisis

Learning goals: The credibility crisis revealed weaknesses in research; these weaknesses are caused by hidden processes that don’t show up in published reports

This module is framed around a schematic version of how empirical psychological science works (adapted from Munafo et al., 2017).

In the schematic, you generate a hypothesis from theory, design a study to test the hypothesis, collect data based on the study design, analyze the data to test the hypothesis, interpret the results, publish the data, and begin the cycle anew. By using this process to compare discrepancies between your hypothesis and the data, you can identify flaws in your theoretical assumptions, which allows you to revise the theories and improve them.

However, a variety of events made psychologists aware that something about this cycle wasn’t working. First was the observation that only about 8% of the results published in psychology journals are negative ( Fanelli, 2010) – yet these negative results are necessary to identify flaws in theoretical assumptions. Second was the publication of a paper by the well-respected social psychologist Daryl Bem, who, in 2011, published a somewhat unusual paper in the world’s top journal for social psychology, the Journal of Personality and Social Psychology ( Bem, 2011). This paper used methods that met or exceeded the standards of rigor that were typical for the time, but advanced a claim that was patently absurd: that college students (and, by extension, everyday people) could be influenced by future events.

This paper suggested either that everything we knew about physics was wrong or (perhaps more likely) that the research methods in social psychology that we used to think were rigorous were somehow flawed.

I then describe how these observations spurred a credibility crisis (not a “replication crisis”, as the crisis is broader than just a lack of replicability) in which researchers investigated whether and how research methods in social psychology are flawed. I use two exercises to illustrate some of the problems.

In the first, the students use a free shiny app to simulate how the combination of the selective publication of positive results and low statistical precision create huge distortions in our understanding of the evidence supporting a particular psychological theory. This exercise demonstrates the concepts of statistical power, precision, and publication bias and demonstrates the general method of using simulation studies to understand what happens when some quantity (in the context of a simulation, a parameter) varies.

In the second, the students use FiveThirtyEight’s p-hacking app to illustrate what makes p-hacking possible. I randomly assigned students to either the Republican party or the Democratic party and to either make their assigned party look good or bad. This exercise illustrates how p-hacking can emerge from the combination of flexible definitions / measurement and a desire to obtain a certain result.

I close the module by describing how published research reports only constitute a subset of what goes into creating a certain research funding and how processes like p-hacking and publication bias, while hidden, can undermine a finding’s credibility. Thus, one way we can uncover the credibility of a finding is to make visible more aspects of the research process. The next module describes one way to achieve this greater level of visibility.

Module 2: Replications as a way of highlighting what is known

Learning goals: Close replications test the credibility of a finding. They work best with good documentation & structured ways of choosing replication targets

This module starts with the question of how to determine whether a particular finding is “credible”. One way is to fix in place the hypothesis and try to do the study again. When you do the next study, you can either vary certain aspects of the method or duplicate the past method as carefully as possible. Although I don’t yet introduce this terminology, these two sets of scenarios illustrate the ideas behind a close replication and a distant replication.

I then ask the students to complete an exercise to think about the conclusions that are reasonable in close and distant replications. The goal of this exercise is to illustrate how close replications (where you keep the method similar) are particularly informative when they give you results that differ from a set of past results because they call into question the credibility of the original results. However, they are less informative when you get results that are similar to the original.

In contrast, distant replications (where you vary some aspect of the method) are particularly informative when you get results similar to the original results because they allow you to generalize across the methodological feature that you varied. However, they are less informative when you get results that differ from the original because, in addition to all the explanations that apply when you do a close replication, the features that you intentionally varied could also have produced the different results.

I also use this exercise to highlight the importance of minimizing sampling error and fully documenting procedures; both sampling error and undocumented differences in procedure (“hidden moderators”) provide possible explanations for why replication results differ from original results. This highlights a somewhat hidden side benefit of replications: they force you to very carefully document a particular procedure.

To illustrate how to document the procedure behind a replication study, I introduce the students to the “replication recipe” (Brandt et al., 2014). The replication recipe provides a structured set of questions to guide the process of creating a method section for a replication study. As an exercise, I ask students to fill out the first section of the replication recipe with an article that I assign (I pre-selected two articles that are short and have a relatively simple research design). After the exercise, we discuss the process of using the replication recipe and identify issues that came up – including the poor reporting standards of most (but not all) psychology articles.

In the last part of this module, we discuss a way to choose replication targets. I teach a somewhat informal version of a framework developed by Isager and colleagues (2020). As an exercise, the students use the framework to rate the value, uncertainty, and cost of doing the replication study that I assigned them.

Module 3: Resource planning and piloting

Learning goals: Resources are critical for making your replication precise; you should think through the sample your resources allow & use those to calculate your power

This module subsumes most of the content that would normally be taught as “power analysis”. The reason I frame power analysis as “resource planning” is to emphasize the critical, and often unrecognized, role that resources (time, money, skills) play in the number of observations a replication study achieves. I teach two workflows for planning resources: a resource-first workflow based on identifying the practical constraints to one’s resources and determining the power those constraints allow to detect different effect sizes, and smallest-effect-size-of-interest workflow based on identifying a smallest effect size of interest and the number of observations (i.e., resources) required to achieve different levels of power to detect that effect.

The module relies heavily on faux, an R package for simulating fake data that fits a specific set of constraints. I link the idea of simulation studies back to the first module and tell the students that I am giving them a powerful set of tools to conduct their own simulation studies. I do not assume that students know how to use R, but rather wrote two different scripts that make simulating two-group and four-group designs easy, even for a complete R novice. My goal is to give the students some tools to do basic tasks in R, as well as resources to learn more about simulations in R if they have the time and interest.

A hidden goal of this module is to demonstrate how under-resourced most past research really is. For example, when I illustrate the resource-first workflow, I assume that we have enough resources to achieve 80 observations per cell in a two-group design – a number of resources that meets or exceeds the sample sizes used during the 2000s in social psychology ( Fraley & Vazire, 2014). This design yields abysmal power to detect most reasonably-sized effects.

As another example, when I illustrate the smallest-effect-size-of-interest workflow, I assume that the target effect is part of an interaction. Interactions require about 16 times the sample size to detect than main effects, a fact that is illustrated vividly in the simulation-based power curve from this part of the module.

One last issue that comes out of the simulations is the number of assumptions that one must make in the process of doing a simulation study. This includes both statistical assumptions, such as the size of the standard deviation of the outcome measure, and non-statistical assumptions, such as the length of time it takes for a typical participate in the study (a fact that is necessary to accurately estimate the number of participants who can participate in a lab-based study, for example). I argue that pilot studies are useful for developing good values for these assumptions. Pilot studies are not useful for directly estimating the value of the target effect size itself ( Albers & Lakens, 2018); in any case it is better to power to a smallest effect size of interest than the expected effect size.

Module 4: Preregistration

Learning goals: To preregister something, create an OSF project & put the replication recipe in the registry. There’s evidence this helps make research more credible

The bulk of this module is focused around completing a pre-registration for the article assigned to the students in the previous modules. Because the workshop participants have already completed the first part of the replication recipe (Brandt et al., 2014) for this article, they are already familiar with the article’s purpose and materials. For the first part of this module, the students complete the remainder of the replication recipe (or at least, as much as they can) as part of the last exercise of the workshop.

The replication recipe in hand, the students can complete a replication recipe-based preregistration on the Open Science Framework ( OSF). I walk the students through this process and introduce them to the basics of uploading materials on an OSF page. But the students already completed hard parts of preregistration as part of the previous exercises.

I spend the remainder of this module reviewing evidence of what a well-planned and well-executed preregistration can do for the credibility of research.

I also briefly cover Registered Reports, which circumvent publication bias through a staged review process. At stage 1, a proposal is reviewed prior to any data collection. At stage 2, the proposal is reviewed again, and as long as the researcher executes their accepted study protocol, published regardless of results. I give the students a list of journals that accepts registered reports if they’re interested in conducting a study with this publication format.

Conclusion: Use my materials!

Replications have played a critical, though I think sometimes misunderstood, role in spurring the credibility revolution: they are one way of investigating the credibility of a particular research finding. The primary way they do this, I think, is by serving as a tool to highlight previously unknown pieces of information, either during the process of documenting the procedure (so it can be successfully executed) or via the results themselves. This role deserves to be underscored in teaching materials. I also think replications can be a useful venue to highlight other parts of the research process, such as decisions about what to research and resource planning. The structure of my finished workshop reflects this general outlook.

If you find my materials useful, please use them! The materials are freely available in this repository. Let’s use our teaching to pass the lessons of the credibility revolution on to the next generation of behavioral scientists.

References

Brandt, M. J., IJzerman, H., Dijksterhuis, A., et al. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224. https://doi.org/10.1016/j.jesp.2013.10.005

Developing a comprehensive directory of tools and technologies for social science research methods

Mon, 04 Jan 2021 10:44:40 -0400

Often the search and exploration of tools and technologies in social science research is not part of the class curriculum in the same way as the systematic review of literature is. This, sadly, leaves the becoming researcher in a place of disadvantage, in my opinion. In their early research career, students will mostly rely on their supervisor or peers to advise on the tools they use, which is still a very limited sample. However, with strides in technological development, researchers could choose from a growing number of multivariate tools for social science methods rising from within the discipline itself, as well as borrowed from other disciplines or coming from the commercial sector.

Starting from this premise, we decided to build a tools directory for social scientists, a simple solution for a place where any researcher or student can come and find the right tool for what they need. In this piece, I explain how the tools directory was developed and how it can be used by educators, researchers and students.

Developing the tools directory

The initial list was based on software tools and tech platforms that we knew were popular among social science researchers because we’ve commissioned books about them, or they have been prominent within the community. We continued to ask academics, look through papers and other lists like the DiRT Directory from the Digital Humanities, the Digital Methods Initiative and SourceForge. Soon enough, the directory was growing out of control. What we thought would be a simple scroll down page, organised in a few basic categories, was not serving its purpose any longer.

With around three hundred different software packages and tools that we knew were used by some or many social science researchers in their work, a new challenge was becoming apparent. It was a paradox-of-choice situation. On one hand, it was increasingly clear why academics often rely solely on recommendations from their peers when choosing a tool. And on the other hand, we knew we needed to explore how one would choose the right tool from a list, and ultimately how to teach others to find the tool that fits their own purposes rather than simply recommending a tool they’ve used.

As the list grew, we enlisted the help of a few master students, and started collecting more data: who built these tools, were they free or paid, what cluster of similar tools would they belong to, when were they built, based on the information available could we tell whether they were up to date, scaling, or failed, could we find papers that cited these tools, were the creators recommending a citation etc.

When we hit 400 software packages/tools, we knew we had to promote this list and share it in a way that researchers would actually stumble upon it and have the opportunity to reference it in a lecture or paper. So we wrote a whitepaper summarizing the big trends on the development of tools and tech for social science research. We learned that both commercial and non-commercial tools are popular within the social sciences, but the ones that last longer and are more successful focus beyond the discipline and almost always have a person or teams of people dedicated to raising funds or expanding the community of users and contributors.

At 400 software packages/tools, we were still not sure the list was big enough. We then focused on specific methods and researched all the tools available to carry out that method or task within the research process. We looked at the evolution of technologies for that method in particular, as well as how it fits within the development of the method itself. We call these ‘deep dives’. We’ve done deep dives on tools for annotation, or tools for transcription, surveying tools, tools for studying social media data, and we kept finding more software applications within each of these areas. We concluded these deep dives to be quite useful, as they enabled sharing slightly more comprehensive sub-lists of tools that could be used in different modules. We have now 543 tools on the list, and the number keeps growing.

How to use the tools directory

The full directory is currently available on our GitHub repository as a csv file. We decided to host it on GitHub, in order to be able to update the directory when we come across new tools or after deep dives; ensure it’s always available for others to reuse in its most up-to-date form, and enable instructors, students and researchers to add tools that might be missing.

Educators teaching research methods or preparatory courses for students’ theses could present the full tools directory to students, so they are more flexible in finding the right tools for their needs and future projects.Students can browse through the list and filter for tools to find a tool that is most appropriate for a research project they are initiating. For example, a student transcribing interviews might look at the transcription tools to find alternatives. Similarly, educators that are teaching a more specialized course, such as introduction to text mining, data visualization, or social data mining, or running online experiments could filter out a sub-list of tools focusing on the explicit method. They could then share this sub-list as part of the course reference materials or assignments.

Fig. 1. The spread of 543 tools and technologies across methods and techniques.

Fig. 2. Filtering to find transcription tools. A student or instructor could filter by column F (the Competitive cluster which contains the method/technique/task/area that we used to categorize the tool) to get a sub-list of tools that could be broadly used for a particular process. If the cluster is too broad, the student can look through the technique (column E), that breaks it down further. For example for social media tools, the technique would include analysis, collection, visualisation etc. If looking for more recent tools, one can filter by the year the tool was launched (column M); or if the student is interested in something that is free, they can check the charges (column N).

While the csv file that contains the tools directory might be easy to update and share, we acknowledge that it might not be that easy to use within a classroom. We are experimenting with a variety of ways that would enable a better display and navigation of the directory, without losing from the ease of updating it.

In 2019 we did our first deep dive into the tools for social data science to support our SAGE Campus course on collecting social media data. We created a sublist to share for this course to help learners find the software that might be most appropriate for their own project, especially given the variety of social media platforms available. To render the sub-list in a more friendly way, we used the free version of airtable, which is a no-code app for relational databases with a colorful and modern interface. Students would navigate to this page (Fig 3) to see the sub-list on a single table. They can then find the right tool for their social media project by selecting the platform they want to collect their data from (twitter, instagram, facebook etc), whether they are happy to pay or looking for something that’s free, and the type of task they want to perform: whether they need the tool for collecting the data, analysis, or visualization. Once they have a filtered list, they can also look through the academic papers we’ve linked where each tool has been used, to explore further the potential of the tools.

Fig. 3. Screenshot of the sub-list containing social media tools via the free version of airtable. Similar to working with a csv file (as in Fig. 2), this interface lets the student filter the list down to narrow the choices for a tool they could use to either collect or analyse their data. This interface is web-based, and has a more inviting user experience than working with a csv file. A student can easily see the categories of tools, filter by multiple terms or concepts linked within each of the columns.

We envision this sub-list of social media tools to be a starting point, as it helps the learner filter down based on a limited number of criteria, such as: the task that can be achieved (collection, analysis), the social media platform that’s integrated, and the fees.

We’ve reused the same sub-list of social media tools with a different interface (pory.io, currently in beta) to render this list of tools more akin to a catalogue of records, that the student can search and filter. This rendering was used in a bootcamp on starting off with social media research. Similar to the airtable rendering, a student could filter based on the task they want to achieve and then click into the tool to get more information and explore which one would work better.

Fig. 4: Screenshot of the sub-list of social media tools rendered into a catalogue via the pory.io app. The user experience on this interface is friendlier than working with a table as in Fig. 2 & 3. A student can filter the list by the type of tool, which is immediately visible; for example they might be looking for tools to support their data collection. They can then use the search box to enter key terms and narrow down the list further, a process that is more familiar. The student can also browse the list of tools by opening the individual cards to find more information (see next figure).

Fig. 5: Once the student filters a list of tools, they can click one each card to get further information about each tool. Currently this includes a brief description, the platform supported, whether it’s free or not, and several academic papers that have used this tool.

Airtable and pory.io have different affordances for rendering the sub-lists of tools, and our experience so far is that both have been useful. We are hoping to learn more from these experiments, to understand the student’s journey as well as the data that would inform their exploration process.

The social media tools sub-list was part of a deep dive that we carried out in 2019. Since then, we dived into surveying tools and text mining. We have not created separate sub-lists for these, and encourage instructors to try other ways of representing these tools within their courses. If you are teaching text mining in the social sciences, for example, you can point your students to this overview of the text mining tools available (Fig. 6 & Fig. 7) and share a sub-list of the tools directory filtered for text mining with your students.

Fig. 6: Screenshot of the Text Mining section, an overview of tools available.

Fig. 7: Text mining tools and technologies based on the process they support.

Going forward

Going forward, we are quite interested in finding out what are the criteria people often use to filter down to their top tools, so we can build this list forward and continuously add the data that helps academics and students find the tools that fit their project best.

We understand that lists follow some form of a hype cycle, where there is a lot of work done at the start and some engagement from the community, and then the whole project slowly dies and it is forgotten. It becomes pretty unusable, because with the pace of research and technology, a lot of the tools are out of date and many new ones have popped up. A person must be dedicated to updating the list and for now we have that covered. Since the publication of the whitepaper in November 2019, we’ve added at least 100 more tools, mostly focusing on text and data mining. While it’s relatively easy to come across new tools, the hardest bit is updating the ones that are already on the list, and that’s where we are open for suggestions from the community. The list with updates to the whitepaper are available in this GitHub repository.

Finally, the locus of software tools and technologies within the research ecosystem remains a big challenge. Software tools are yet to gain the credit of research output. And that is why, among other reasons, software tools are rarely cited or referenced in papers. This is not only bad for reproducibility of research, but it also makes it difficult to help other researchers weigh in and compare different tools used for similar studies. We aim to promote and include the suggested citation of the tools in our list, and strongly encourage anyone to use https://citeas.org when unsure how to give credit to these.

We remain active and are continuously thinking of better ways to present and re-architecture the information about software tools and technologies we’ve gathered, to make it easier to navigate and explore. We hope these materials will help you and your students become more aware of the diversity of tools and technologies and will open new and potentially easier avenues to decide on the best software tool to use for your research.

Using the 'Transparency Checklist Guidelines' as an Educational tool

Sun, 04 Oct 2020 10:44:40 -0400

Transparency Checklist Guideline for Education

What is it for?

As an educational tool, the Checklist can be used to teach and improve the standards of transparency and credibility in research reports made by students. The aim is that students are embedded in transparent and open practices from the beginning of their training.

See here for an adapted version of the original checklist for educational purposes.

How to use it?

Supervisors and instructors can require students to create a Transparency Report along with any empirical research assignment.

Steps to follow

Inform the students about the Transparency Checklist.
Add the completion of the checklist to the requirements.
Make the checklist app available to students
Be ready to answer questions if the students are uncertain about a checklist item.
Read the Transparency Report along with the manuscript.

Optional steps

Explain each item why it is important for transparency.
In the classroom, ask the students to evaluate some (weakly and strongly transparent) published research papers. This can help them get familiar with the checklist and realize the importance of transparent reporting.

How to evaluate the Transparency Report

In the requirements, indicate which items should have a “yes” response. Should you want to make it easier for the students, you can require only the Short (12-item) Checklist: http://www.shinyapps.org/apps/ShortTransparencyChecklist/
When possible, check whether the manuscript is in line with the checklist answers.

Reference

The following paper introduces the Transparency Checklist and describes how an initial set of items was iteratively evaluated by 45 journal editors, as well as 18 open-science advocates until a consensus was reached about the content and form of the checklist.

Aczel, B., Szaszi, B., Sarafoglou, A. et al. A consensus-based transparency checklist. Nat Hum Behav 4, 4–6 (2020). https://www.nature.com/articles/s41562-019-0772-6

Addressing the issue of representation in an undergraduate psych class

Sat, 03 Oct 2020 10:44:40 -0400

I want to tell y’all about one of my most meaningful teaching experiences. Pull up a chair.

I’ve been teaching experimental psychology since 2006. It’s the flagship research methods course for undergraduate psychology majors at my institution. We now enroll 120 students each semester and, with the amazing help of 5 TAs, we offer lectures twice per week and a 2.5-hour lab once per week.

As many of you know, the shit hit the fan when it comes to psychology research during my time teaching this class. I felt pretty dizzy for a while during and after 2011, the year that marks scales falling from my eyes.

I didn’t really make many substantial changes to my course right away. I needed to feel like I was on firmer ground first. (I expressed that sentiment in this tweet thread below.)

But then, I came to realize that if I didn’t make changes to my teaching in addition to changes to my research, I was complicit. So, I got to work. Standing on the shoulders of the smart people who brought issues to light and modeled pathways to improvement, I shifted my content.

Building on traditional topics like internal, external, & construct validity, reliability, measurement, hypothesis testing, etc., I started teaching about transparency, openness, questionable research practices, and effect sizes.

We started to do IRB-approved, preregistered replication research some of which has seen or will see the light of day in published form ( like the example in this tweet).

We also started discussing efforts like the ManyLabs studies, the Psychological Science Accelerator ( @PsySciAcc), & the Collaborative Replications and Education Project ( @CREP_psych).

This has all reinvigorated my love of teaching in general and this class specifically. I honestly feel like I’m fulfilling a real need. We need people in our society to have the tools to consume research responsibly, whether they go on to produce research themselves or not.

But then 2020 happened. After centuries of oppression of Black, Indigenous, People of Color (BIPOC) in the United States, and seeing the disproportionate impact that the coronavirus is having on BIPOC folks, it finally clicked.

My students need more from me than validity and reliability and transparency and openness. They need to feel they belong. All of them. Every one of them needs to see themselves in psychological science if that’s how they want to spend their time.

So, this semester I introduced a single class session focused on the importance of representation in science. I borrowed ideas from Jessica Remedios ( see her terrific Twitter thread, which ends with her slides).

I had them read this work about the diversity-innovation paradox by Hofstra and colleagues.

I also invited them to watch the Picture a Scientist film and/or listen to the Everything Hertz podcast ( @hertzpodcast) about Diversity in science with Jess Wade.

I talked about psychology’s positivist tradition of believing there’s some sort of objective reality that we can discover if we cleave to principles like empiricism, transparency, & falsifiability and how that position is marred by biases (e.g., confirmation bias, availability).

It’s also marred by the fact that scientists are humans with standpoints that affect the questions they ask. And that power structures dominated by men and white people guide what we think is “normal” science. Such ideas aren’t commonly discussed in quantitative research circles.

I showed them clips from Picture a Scientist so they could appreciate what it’s like to be a woman in science, and the progress that’s been made to remediate gender bias (e.g., the MIT report from 1999).

We also talked about the idea that some seem to have been left out, women of color, in particular, but also disabled, LGBTQ, first generation, indigenous people, and all their intersections.

We talked about the loss to science, especially when innovative ideas put forward by women, people of color, and women of color are devalued (Hofstra et al., 2020).

I showed them powerful clips of one woman of color’s experience as a scientist, professor of chemistry, Raychelle Burks ( @DrRubidium). Dr. Burks captured so perfectly the importance of representation.

I talked about how these problems are not limited to male-dominated sciences but that these issues pervade psychological science too.

I showed them data from APA demonstrating the move from roughly 78% undergrad psychology majors being female to less than 50% of full professors being female. Lots of factors at play, including gender bias.

I shared with them an email that a female colleague of color received recently that illustrated the gendered racism she faces in our field. It referenced the hardships faced by white men. (Sorry, what now?)

I talked about my own standpoint as an educated white woman, a full professor with lots of mentorship from family members and powerful colleagues through the years. I had a lot of help getting where I am in part because of intersecting identities tied to systems of power.

I expressed my hope that they’ll think about their own intersecting identities and if science is something they love and psychology specifically they should go for it and make room for everybody’s voices.

I’d never talked about these ideas in my class before this week. I have also never received so much positive feedback from students. They felt seen. I don’t think I’d ever made them feel seen before, not quite like that. I feel good about that.

I also feel sad. Students feel these things so hard, especially those we’ve marginalized in so many ways. I’ve been in undergrad classrooms for 15 years. I’ve worked with literally hundreds of students. So many missed opportunities.

I can’t change that. But I can keep it up in future semesters. Urry is ON.

Meanwhile, here are some of my slides minus film clips and the email. Maybe they’ll help you or someone you know address representation as a foundational idea of science at one of the earliest career stages.

Editor’s note: The present text is an adapted version of widely shared Twitter thread which resonated with so many of us. We thought it is of general interest and deserved to be immortalized, and hence we approached Heather to adapt the thread to post it here. Importantly, in addition to writing extremely current and relevant threads, Heather has also inspired the creation of FORRT. Indeed, FORRT was initiated at the 2018 meeting of the Society for the Improvement of Psychological Science (SIPS) in Heather’s “Teaching replicable and reproducible science” hackathon with Kristin Lane.

Educators' Corner | FORRT - Framework for Open and Reproducible Research Training

Meaningful results for meaningful hypotheses

How to decide if an effect is meaningful in Bayesian inference?

A computationally feasible and intuitive way

Where to start if you are new to Bayesian inference?

Contact information

FORWARD: A UNESCO Decade of Science Programme

FORWARD (FORRT for World Development)

Quantifying reproducibility: Lessons from an iRISE-FORRT collaborative project

The iRISE project

A collaborative review of reproducibility metrics

What we learned

EDI considerations

References

LMU-FORRT Train-the-Trainer Program Press Release

The OSC staff on the Train-the-Trainer program are:

Community-Augmented Meta-Analyses in Education (ELD CAMA)

Background of ELD CAMA

How ELD CAMA and the FORRT reinforce each other

ELD CAMA Mission

Quick start

Call to Action

Contact information:

Accessible Feedback in Research

Background

Our mission and how it is working to contribute to the open scholarship community

How our project and FORRT reinforce one another

Call to action

Contact information:

References

Course on Open Science 101

Main Text

Introducing Students to Open Science

The Course Content

Contact information:

Being a Minoritized Early Career Researcher in Political Psychology

Shifting the culture of peer review with Reviewer Zero

Background of Reviewer Zero

How Reviewer Zero’s Mission Intersects with the Open Science Community

How Reviewer Zero and FORRT work together

Call to action

References

Author Biography

Funding

Academic Jobs in the US vs. the UK

Structural Factors Shape Employment

Features, Not Bugs

Trans-National Sharing

Author Biography

A Student’s Guide to Open Science: Using the Replication Crisis to Reform Psychology

A child of the replication crisis

What is this book about and why is it needed?

Pedagogic resources within the book

Become a student of open science!

Author Bio & Contact:

Introducing Nowhere Lab

Research and data lessons from a non-academic job:

Five lessons from a non-academic job

Addressing the grand challenges facing psychological science

What are the four grand challenges identified by APS members?

1. Globalization and diversity

2. Research integrity and applicability

3. Collaboration across fields and disciplines

4. Strengthening theory

References

Navigating Open scholarship for neurodivergent researchers

Navigating Open scholarship for neurodivergent researchers

Who we are and why is this important?

What do we want?

Our plans

Join us!

History of Team-Neurodiversity

References

Other references

The PaPOR TRaIL Course

The Open Research Toolkit

Innovative Mentoring

Qualitative Open Science Practices

Starting with Qualitative Open Science Practices

Call to action