Pedagogies | FORRT - Framework for Open and Reproducible Research Training

From mental-health breakfast interventions to evidence-based practice

Tue, 03 Sep 2024 15:30:59 -0400

We are thrilled to present our new Pedagogies, which features Dr. Thomas Rhys Evans. Thomas is Associate Professor of Occupational Psychology (from September 2024 he’ll be Professor in Organizational Psychology and Open Scholarship) at the University of Greenwich and a very active member of FORRT! His research focuses, among other things, on using meta-psychology and Open Science practices to understand the quality of evidence in Psychology. In this FORRT’s Pedagogies, Thomas shares insights on the 12-week module on evidence-based practice he developed and taught for the first time in 2023. We discussed the philosophy behind the course, how it relates to Open Science, his students’ reactions, and so much more. You can watch the interview in the video or read a summary of their main points below. Make sure to also check the 12-week module, which is fully available (including teaching materials, students’ reports etc) on the Open Science Framework (link below).

How did you come up with the idea and identified the need for this course?

Thomas: We all are involved in research and teaching because we want to do good and have a positive influence, but it can be really hard to know how you can help or what you can do. Especially when knowledge is so compartmentalized into different fields. The core thing for me was to try and break down what is shared across the ways in which different fields do research and the ways in which we teach people to think more about the broader skills that underlie, for example, open scholarship. It’s this need for more critical thinking, to look at the evidence and say, “is that right? Are there other alternative interpretations?” Critical thinking skills underpin so much of what we do, even in our general day-to-day life, not just our research life. For example, I’ve become increasingly vocal about things like climate change and the need to start creating better bodies of evidence. And to do that, we need to teach that better. We need to teach people, support people, and give them the confidence to be able to question things and not just accept a paper written by somebody from a prestigious university. And so this course was based upon the idea that we can do better in providing students and the general public with those critical skills that can be used in day to day life as well as in research.

Playing this video loads it from YouTube (Google). See our privacy policy.

Can you tell us a bit more about the course set up and organization?

Thomas: I set up a new module for our occupational psychology MSc at the University of Greenwich, which lasts twelve weeks. It consists of a one-hour pre-recorded lecture and a two-hour face-to-face practical seminar each week. The module broadly fits four themes from start to finish, which are to identify, evaluate, synthesize and apply evidence. In terms of my teaching philosophy, I think it is really important that students can label the skills that they develop. Basically, the modules cover these four dimensions from start to finish: How do we identify and generate evidence? How do we push it through the process to make sure that people, policymakers, and practitioners use that evidence? There was an interesting paper demonstrating that it can takes about ten years for research to actually evidence influence. That’s a long time. So it’s really important that we get the evidence right in the first case, developing high quality, robust, and rigorous evidence. And so that’s where the kind of Open Science content tends to trickle in.

Playing this video loads it from YouTube (Google). See our privacy policy.

How would you describe the teaching philosophy behind the course?

Thomas: I try to always think about inclusivity in the broadest sense and I also think about it in terms of evidence. When people think about evidence-based practice in action, they often think about academic sources of evidence. But practitioners have loads of experience too. It is often not written down, but that does not make it any less legitimate. And so we are changing the way of thinking, becoming more inclusive in how we integrate all these different forms of evidence. When we think about inclusion, we also do what is known as the class charter, which is a co-agreed statement between everybody involved in the course. I get a lot of students from a lot of different backgrounds and with a lot of different experiences. The idea is not to get everybody into the same conforming mindset. In fact, it is the opposite: to get people to be more open about different ideas and different notions, and to be inquisitive. Sometimes, students are encouraged to develop a mindset where you listen to the teacher and are asked to memorize what they say. That’s definitely not the goal here. I think it is very important that we build up confidence for people to be independent critical thinkers. Another core component of my philosophy is practical skills, because it is really important that we know what to do with the information we have, to adapt to circumstances, and let that inform judgments and decision-making. So I see practical skills as the most relevant components of my teaching, as it is a way of bringing everything together.

Playing this video loads it from YouTube (Google). See our privacy policy.

Research-wise, you are also quite involved in Open Science. Do you incorporate insights from Open Science research into your teaching? What Open Science principles did you integrate in this course?

Thomas: When we look at evidence, you can’t separate it from the way in which it was created. So, what I do within this course is trying to weave in Open Science dimensions throughout. We look at everything, starting with the standard things that you might expect ‒ like pre-registration, replications, open data sharing, and so on. But we also look at structural factors. That includes, for example, how research and researchers are evaluated. As soon as you look at the incentive structure of research, it becomes clear that it is not always going to lead to the best quality evidence. We also think about the values and the way of thinking associated with open scholarship. Often, it’s about prioritizing transparency. It’s about changing the focus from novelty to encouraging openness and reflection. For example, being open about both the limitations and the strengths of evidence. It’s not necessarily teaching Open Science, but it’s about getting that mindset: what are the factors I should be thinking about and what’s the style of thinking that I could adopt to get the answers I need?

Playing this video loads it from YouTube (Google). See our privacy policy.

Which were the most important skills that you wanted your students to develop in this course? Could you give us an example of how you taught students these skills?

Thomas: Let me give you one of my favorite examples of this. During the first face to face session, I gave my students evidence on how feedback (star ratings) on scented candles on Amazon changed over time, to encourage the application of critical thinking. For me, that was a really fun way of getting people to think about different things. For example, why would the reviews of scented candles suddenly plummet? And then, obviously, you build into the facts ‒ okay, here are the years that it happened in, here are the time frames, and here is a little bit of the data. This didn’t happen with unscented candles but it did happen with other fragrances. So what might that tell us? It turns out we were looking at the evidence behind COVID: reviews for scented candles were worse when people had COVID because they couldn’t smell the candles! I encouraged students to notice little details when evaluating evidence. In this course I wanted my students to develop core academic skills, such as critical reading and critical writing. These are fundamental skills to any research degree, but courses teaching these skills are usually so dry that students can’t see how they relate to practice. I always try to achieve this in a practice-oriented way.

Playing this video loads it from YouTube (Google). See our privacy policy.

In your course, you teach students how to gather and evaluate evidence-based knowledge. Do you integrate different research epistemologies? If so, how?

Thomas: Within this course, I do encourage people to think in different ways. I don’t ask my students to look at things from one perspective, but rather encourage them to use multiple points of view. A good example of what I’ve done is a book about the evidence behind specific HR practices (Evans, 2022). And the whole purpose of this book was to become the core reading of the module, because it encourages us to think in different ways and to adopt different epistemological viewpoints. This book deconstructs the evidence behind the so-called “best training.” I’ll give you an example. One of the students on the first version of this course was a trained massage therapist. They did their assignment looking at the evidence for the role of in-seat massages for workplace stress. The role of their professional experience in integrating academic evidence became a huge part of what they spoke about in their assignments. How does their experience match with what certain types of academic sources say? And again, there were lots of interesting differences between what quantitative research and qualitative research were saying. And then they had this interesting third point, which was their professional experience of this. This course is designed to facilitate those sorts of links in critical thinking.

Playing this video loads it from YouTube (Google). See our privacy policy.

For their assessment, students had to produce an evidence-based synthesis, evaluation and practical dissemination. Can you tell us a bit more about it?

Thomas: I wanted students to apply what they learned by looking into a workplace practice they were interested in and evaluate the evidence behind it. The goal of the assessment is for students to find sources of knowledge, such as academic sources and experts, identify the most important evidence we have about their chosen topic, and to bring all of that information together to form a message. The assessment has three main components, starting with a

curated literature review. The purpose is to demonstrate that you can prioritize the most important sources of knowledge, bringing the evidence together to say something. I encourage students to consider not just academic sources, but also practice, expert opinions, and other evidence-based resources.
Second, students evaluate the quality of evidence they have, using one of the quality evaluation or risk of bias tools they were introduced to during the course. For this task, students evaluate a piece of evidence that they find to be particularly important and then create a narrative to communicate their evaluation. For example, in the literature review, they might have highlighted a body of evidence or source of evidence that was extremely popular or well cited. But then in the quality evaluation they find that actually there’s a quite substantial risk of bias in a certain direction, and we reflect on how these concerns may impact our conclusions.
The third component is practical dissemination: a way for students to share with the public what they’ve found in the first two assignment sections. I encourage students to get creative with how they share this information, whether it be a poster, a blog, or a series of videos. Several students have allowed me to share their assignments as examples, which can be found here.

Playing this video loads it from YouTube (Google). See our privacy policy.

How did students respond to the course? Did you notice any changes in their attitudes towards science? Could you share your thoughts on how well the students grappled with the material and assignment in this course?

Thomas: I feel like I threw a range of topics at students, and they were always ready to engage with the material and participate in activities. As the course went on, the biggest change I noticed was the questions students asked. They became more willing to question different practices, and their questions were often in line with Open Science practices. For instance, they began to question why someone did not share their data or why a study wasn’t pre-registered. One take away from this course is that the quality of a justification is important for evaluating a choice, particularly when there is not a single right answer. For their assignment, there was no one right way to accomplish the goal because there was a lot of flexibility. This encouraged students to use critical thinking skills, which ultimately made for a good project. I like to share students’ assignments as examples because you can really see the range of ways students accomplished the task. In terms of criticisms, students felt that there was a lot of reading, which is true. However, I try to highlight the skills students get out of situations like these, such as prioritizing tasks, and to ask them to do something with the readings beyond summarizing them. Overall, I think emphasizing the skills students will gain, and making sure the activities focus on skills that will support them in their future employment, can help students get more out of the class.

Playing this video loads it from YouTube (Google). See our privacy policy.

Sometimes we think that incorporating Open Science into teaching means teaching a course on Open Science, but it’s not always possible to do that. What is your advice for teachers who have pre-defined courses/content but want to introduce more Open Science principles into their teaching?

Thomas: My advice is to not necessarily tackle everything at once, but to start small, and try embedding that critical way of thinking, encouraging students to start asking the right questions. For instance, sometimes this might mean breaking down the source of evidence to ask: “what can we take from this evidence if we acknowledge the role of that specific practice/bias/decision?”, “why did they use this sample or sampling strategy?”, “what would be the potential implications if they did this differently?” You are not teaching them what pre-registration is or why it’s useful to share data, but helping them gain that same way of thinking, that importance of critically reflecting on evidence. Open Science can also be incorporated in the way we grade students, again prioritizing the skills they developed in synthesizing evidence and thinking critically about it. In my course, the literature review has a criterion for criticality: have they demonstrated critical thoughts about the evidence gathered? Have they picked a diverse range of sources of evidence? If you’re starting a course from scratch, or even picking up a pre-existing module, it’s easy to embed little ideas and notions that get students thinking in that critical way. And this can be a really powerful mechanism to teach Open Science principles, especially because some people are openly averse to Open Science ways of thinking and specific practices. So, this just gives you that little first step of a positive kind of development in that direction.

Playing this video loads it from YouTube (Google). See our privacy policy.

Do you have a favorite part of the course? Do you think your students have a favorite too?

Thomas: I definitely have a favorite piece. It’s week one, lecture one, and my first encouragement to students to have a little bit of that flavor of critical thinking. I start off with a broad question: “Wouldn’t it be good if we could improve our mental health based upon what we ate for breakfast?” And I talk about a news article that said having Marmite on toast improves your mental health. It’s such a beautifully simplistic thing that could have such a positive impact. So everybody wants it to be right. My fun with this first session was to clinically take the fun out of it and deconstruct the evidence. We start by pondering what sort of evidence we would need to trust that claim: epidemiological, experimental, observational, etc… Turns out the study was an experimental trial with different types of multivitamin tablets, and it had nothing to do with Marmite whatsoever, but Marmite has a high amount of the certain type of vitamin they investigated. Then I really stretched this as far as I could by working out with them how much we would have to eat of a certain product to get the vitamin amount that they evaluated. Turns out that if you actually ate that much Marmite, you would more quickly die of sodium poisoning than you would of anything else because of the salt. So really, it wasn’t a very clever claim to make. Then we explore further and find out they also tested different vitamins, the study was underpowered, it didn’t declare the vitamin tablets given to the researchers for free as a conflict of interest, and that the distortion from tablets to marmite came somewhere between the university press release and the journalists. For me, this was a very fun exercise, because it is something we all hope would be real, but when we look at the evidence, it actually tells us a very different picture.

Playing this video loads it from YouTube (Google). See our privacy policy.

Resources

Check Thomas’ 12-week module class: https://osf.io/typ4z/
Evans, T. R. (2022). The evidence behind HR: An open evaluation of contemporary HR practices. Routledge.
Data enhancement and analysis of the REF 2021 Impact Case Studies (jisc.ac.uk)
YouTube video about evidence-based practice: Evidence Based Practice: A Toasty Introduction - YouTube
OSF link to “The ongoing and upcoming challenges for the role of news in science communication”: PsyArXiv Preprints | The Ongoing and Upcoming Challenges for the Role of News in Science Communication (osf.io)

Suggested citation:

Evans, T.R., Wolska, J., Andreolli, G., Woodward, A.M., Merighi, A., & Micheli, L. (2024). FORRT Pedagogies. From mental-health breakfast interventions to evidence-based practice: Teaching students how to critically engage with science. ( https://doi.org/10.17605/OSF.IO/SB5VE)

Team Pedagogies Contributors:

Julia Wolska, Manchester Metropolitan University, UK
Giorgia Andreolli, Verona University, Italy
Leticia Micheli, Leiden University, the Netherlands
Amanda Mae Woodward, University of Minnesota, United States
Alessio Merighi, Video Editor, alessiomerighi.com

Open Science education through student participation

Fri, 20 Oct 2023 10:10:49 +0000

We are back with a very special Pedagogies, this time including also the perspectives of students on learning about Open Science (OS)! We have three amazing guests: Prof. Dr. Jordan Wagge and her two outstanding undergraduate students, Jasmine Beltran and Amy Hernandez. Over the past years, Jordan has led the way in developing CREP so that it can i) accomplish much needed replications, and ii) provide tools/structure for instructors who desire to train their students in OS. In this FORRT’s Pedagogies, Jordan and her students share their experiences with CREP and their thoughts about the trials and tribulations of involving students in the process of OS. We hope that this can inspire and help many scholars wishing to incorporate more OS into their teaching, mentoring and research.

Jordan Wagge is a Professor of Psychology and Cognitive Science at the University of Avila (UA) and the Executive Director of CREP. Her research focuses on three areas including replication work, pedagogy, and critical work studies. She has a strong record supporting the OS movement at all levels - from her teaching philosophy, to the development of open teaching materials and resources, and the involvement of students in OS training and practice. But be sure to learn more about Jordan from her website (all links posted below). We are also joined by Jasmine Beltran and Amy Hernandez, Jordan’s students, who have experienced the implementation of CREP in their classrooms first hand. They discuss their perspective as a student, their challenges, and the benefits they gained through this process.

FORRT’s Team Pedagogies took the opportunity to ask Jordan, Jasmine, and Amy some questions on their journey with CREP (shoutout to the FORRT and academic Twitter community for sending so many great questions). You can watch or listen to the interview in the video or read a summary of their main points below!

Check out the full interview here

Playing this video loads it from YouTube (Google). See our privacy policy.

Could you tell us a little bit about yourself and how you got involved with Open Science and, more specifically, with CREP?

Jordan: I got involved with OS pretty naturally. In graduate school, I taught Statistics and Research Methods. When I taught, I would highlight best practices and how science works. There were all these topics that I felt hypocritical teaching because I would also say “Psychologists don’t do this in practice”. It led me to ask, why don’t we do these things in practice? I naturally fit into the OS and transparency movement because it made sense. I got involved with CREP after it was founded. At a conference, I was seeking co-presenters on how to teach undergraduate students research methods most effectively. I wanted to know how others maximized the limited amount of time given all the learning outcomes associated with a methods course. John Grahe responded. The angle he took is to make the course time to capitalize on a meaningful contribution to the field especially if students are already investing so much energy. At the same time, Mark Brandt and Hans Ijzerman from Tilburg University were interested in replication efforts and began CREP. I reached out and joined from there.

Jasmine: We’ve had Dr. Wagge as an instructor for a year now. Our Statistics course is a 2-semester course. During the first semester, we focused on doing a literature review and collecting material. This semester we are currently collecting data. We were required to engage with CREP but it’s been a fun, educational experience.

Amy: I am in the class as well as in Dr. Wagge’s lab, so I’ve double dipped with the CREP experience.

Could you give a brief description of what CREP does and an overview of the process from selecting studies to publication?

Jasmine: I can talk about the CREP steps from a student perspective. It’s a Psychology project designed to tackle the replication crisis as well as provide a way for undergraduate students to gain that quality experience in the research process which I think is super important. I am completing a senior research thesis [final year undergraduate dissertation project for undergraduates who want to complete a PhD]. It’s important to gain this experience from CREP before just diving into research without knowing really anything. I would have had no idea how the research process looks like. With CREP, the main steps we took were the following:

Studies are selected by the CREP Team. Across 9 disciplines, they select the most highly cited papers published in the previous 3 years. The studies undergo scrutiny by executive reviewers and are judged on how likely a student would have the ability to complete the replication in just 1 semester. The reviewers then do a final screening and reach out to the original authors of the selected studies to ensure the highest level of consistency with the original study.
Now, students enter the process. Students are first introduced to CREP by their instructor. They then create an Open Science Framework (OSF) page and assemble components.
Once the OSF page is drafted, it is submitted to a CREP executive reviewer to assess quality and provide feedback.
Students pursue Institutional Review Board (IRB) approval, revise and resubmit the OSF page if needed, pre-register the study and post that on OSF as well.
Data is collected, and once complete, students write up the results, discussion, submit materials, and sign a completion pledge.
Students are e-mailed completion certificates. Once enough data is collected by several labs, students are invited by the CREP Team to contribute to a meta-analysis manuscript, ideally in a 1st author position, which, as we know, is a pretty big deal for an undergraduate student.

Jordan: There’s a publication ( Wagge et al., 2019) which has a flow chart of how the process works from both the instructor’s end as well as the students (see image below). We also have a step by step manual (linked below) that walks students through their exact responsibilities. Of the 19 studies originally selected, we collected data on approximately 15 of them. Three of those studies led to a publication and one of them has a student first author. I am also working on writing another paper right now and it has 30 authors that started in 2013. There are several factors that impact the publication timeline. Some studies have few people choosing to collect data and other studies use software students may not have a background in, like Matlab. These lead to areas CREP is improving on. CREP publications need content experts to ensure high quality work. We started CREP with a sample size rule of thumb (2.5x the original); however, we want to move to conducting power analyses prior to data collection. We warn instructors that studies will eventually end after 1-2 years of being posted. That means that the publication timeline for a study will be at least 1.5 years until we receive a manuscript. Once a content expert has been obtained, they help mentor students writing it up. Independent sites who participate in CREP are able to publish their own collected data. We recommend they publish any extension hypotheses data; however, if they want to publish the data regarding the original hypothesis, they can. We simply note in the CREP manuscript that some of the data has been published already. We are constantly thinking of ways to make this process as efficient as possible.

Flowchart of the CREP Process (Wagge et al., 2019)

The focus on students is what sets CREP apart from other large-scale replication projects. Are students also involved in the publication process? How is authorship discussed and decided?

Jordan: We currently give independent site publications autonomy on student authorship. There have been no sites that have published their data, but we have thought of providing guidance through the OSF page’s bibliography. Currently OSF lists the graduate students first alphabetically, and I am last. Students can raise their position by contributing to more CREdiT categories (e.g., collecting data, contributing to the manuscript). For example, if several students write up the results section, the best writer’s draft is used and they are considered to give additional contributions meriting higher authorship. Ideally, we contact every person who has contributed to data. One challenge we face is if instructors still work at their original institution and if students’ emails still work if they graduated. When students are involved, they are also hesitant about their ability to provide meaningful feedback. In a recent project, we developed a survey and we sent the manuscript in that survey, along with guiding questions for students to provide feedback. We had several students contribute that way and that earns them authorship. We may not use all the feedback students provide, but it is considered and makes our manuscript stronger. We will see how this process goes and get students’ feedback on whether they felt they were contributing to authorship in that way.

There are many aspects of Open Science that can be taught to students. Could you tell us why CREP chose to focus on replications and what are the benefits of this approach?

Jordan: The reason we focus on replications with students is because - without wanting to sound too negative - student projects done in research methods courses are often not heavily based on theories or quality methodology. There is definite value in having this traditional method. Students learn how to test questions and think about how to break down a question through an empirical lens. I joke with my students that topics often chosen are usually pre-workout, sleep, stress, attachment styles, and the Big 5. These topics are interesting, but the questions are often not grounded in theory nor are they the cutting edge questions of what the field is doing. That’s to be expected from someone’s first research project. My own 1st project I conducted was extremely sloppy. I knew that I didn’t have any idea what I was doing. In other scientific fields, it is common for students to conduct replications of classic findings many, many times. Imagine an introductory chemistry student is asked to create a compound in the lab: they are given a manual, told the exact steps, and get the desired outcome. Chemistry instructors never ask their students to form their own hypothesis from the get-go, never ask to imagine what would happen if you mix Chemical X with Chemical Z. Instead, students are first trained by doing. In psychology, a student who wants to study depression is now exposed to several depression inventories and asked to uncover the exact methodology of a study. What CREP offers are curated studies with materials and the opportunity of providing a meaningful contribution to our field by replicating highly cited recent work. Instructors and students don’t have to worry about a lot of the decision making regarding the study. More time can be used in-class to discuss why we pursue the IRB, why this actual research study chose this method, and facing the challenges of being a replicator (e.g., ambiguous language for certain sections). CREP comes from an apprenticeship model and provides scaffolding.

Jasmine: When our class went over 5 or 6 studies we could do, it gave us freedom even if we didn’t get to do our own idea. We still had the choice of what to replicate and those days were my favorite. I appreciate the experience rather than being thrown in the deep-end and told ‘good luck!’.

What would you say are the major learning goals for students in these replication projects? For the students, what did you personally take away from this experience?

Amy: I was definitely scared of APA [American Psychological Association] style the first time, but I have improved my writing in APA since then. I am also less intimidated talking with faculty members and generally communicating with them.

Jasmine: I agree with the point about APA style as well as improving my ability to read scientific papers and their different sections. All students are contributing to every section of the research process, including the literature review. I believe the constant exposure to papers makes me feel more confident in understanding scientific papers, knowing I can write similarly… I think students need to build that confidence because nobody wants to sound stupid. By practicing in class, reading gets easier. Nobody is perfect at it, but practice helps get us there.

Jordan: I originally chose one study to replicate, but now I choose a pool of studies I am willing to lead and have students vote on which one they would like to do. All students in my class write their own papers and get exposure to all parts of the process. I see CREP as a model to implement your own learning outcomes, those of your department, and APA’s listed 2.0 outcomes. We are currently working on a resource putting together writing assignments that would align with different learning outcomes. Instructors can then pick and choose assignments that suit the learning outcomes of their courses or departments.

What are students’ initial reactions when they start with the replication projects? What are the major challenges they may face? For students, what was your experience when you started with CREP?

Jasmine: When we first started, I was intimidated and thought my semester would be difficult. As we continued to move into the course, we spent a good amount of time on each step of the CREP process. This helped us maximize our learning. Currently, I am about to start my senior thesis and I am not nervous about that now. The writing ability is definitely the biggest thing I took away from this process.

Amy: I was intimidated, too. And overwhelmed, thinking about all the things I would not know how to do. The challenge others and myself faced is that we were scared to make mistakes. Many of us wanted to be perfect at it the first time so that fear of ‘screwing up’ held us back from going on.

Imagine I would now start a replication project with a group of students at my university. Could you tell us what you think would be the advantage of doing these student-replications together with CREP? What are the benefits for instructors and for students?

Jordan: Having the structure, assignments, and pre-selected studies are important strengths of CREP. We do have lots of materials, for example tutorials on creating an OSF page. CREP also does have external reviewers who aim to get reviews back within 1-2 weeks. It is nice to have feedback from someone else on the project. We provide a huge advantage for replications in general. CREP can be thought of as a professional network. You meet collaborators and build relationships, and I’ve written letters of recommendations for Tenure and Promotion committees for those involved with CREP. And finally, you do have something that you can add for authorship. This is especially important for educators from small teaching schools, who might have difficulties maintaining their own lines of independent research besides the teaching duties. With that being said, if you are interested in doing replications with students, do it your way. Look at CREP materials, download the step by step guide and adapt it to your needs.

What are the different ways in which instructors can incorporate the work with CREP into their teaching/mentoring (e.g., lab course, seminar, other)? Which types of incorporation work best in your opinion? Which challenges do instructors usually experience?
8a. For you students, how was the collaboration with CREP set up? What was your role in the replication and what did you like the most? Do all students have the exact same roles, or do some students have unique roles?

Jordan: An instructor could implement replications with CREP in an Introductory to Psychology course up to a graduate level course. If the class is for earlier undergraduates, I recommend the instructor handling some of the ‘behind-the-scenes’ paperwork to make the process easier instead of having students be in charge of everything. Instructors could allow students to do their Master thesis on CREP replications; however, I personally ask my students to have extension hypotheses. I agree with work done by Daniel Quintana that student theses should be replications ( Quintana, 2021). Instructors can do replications with CREP on a variety of topics and disciplines, although some fields like biopsychology, developmental, or certain populations (e.g., children) might be more challenging. Instructors often face timeline issues from their institution such as slow IRBs. I would recommend getting IRB approval before the semester starts. It is also important to get familiar with the projects before proposing them to students. For example, knowing which type of resources and materials the replications require and whether you, as the instructor, would have access to those in your institution. I also think it can be easier for first time instructors to implement CREP either in smaller classes or for 1-1 mentoring.

Jasmine: I mentioned that our Statistics and Methods class was shown several pre-selected studies to choose from. My favorite part was to go through each study in detail and discuss how many participants we would need to collect, what potential problems could arise, and so on. We were given ~100 voting points and each of the 20 students could allocate their points to the different studies. I also did an additional contribution to the project which involved recording myself doing the study procedures for the OSF. Amy also contributed with coding for an ongoing study.

You recently got the National Science Foundation (NSF) grant to fund, evaluate, and enhance the activities of CREP. Congratulations! Can you share a little with us how you envision the future of CREP?

Jordan: It’s really exciting that we got NSF funding to do research on CREP and learning outcomes. Right now, we have a couple of classes at different universities that are doing pre- and post-testing, and we’re gonna keep doing that for the next couple of rounds. We are also implementing a ‘peer mentor’ system that pays students who have undergone a project through CREP a 1-year stipend. These students would then have office hours and mentor other students around the world currently undergoing a replication with CREP. This is because students can feel more comfortable asking other students when learning about the research process. Chosen peer mentors will also be taken to a conference. For any students who have participated in CREP prior, we are also doing a Summer CREP conference to give an opportunity for students who participated in CREP research to present their results. Some of the money is going to building infrastructure for a submission system. We also got a second grant that will be used to create instructor workshops on how to implement CREP in their curriculum. This will be really helpful in sort of getting people to incorporate CREP, but also build out materials that other people can use. One of the sessions will be dedicated to mentoring students in replications with extension hypotheses (specifically ones grounded in Diversity, Equity and Inclusion hypotheses). Lastly, we are investing in more teaching resources and material. We hope to make CREP more accessible so that students get authentic, genuine research experience. We want students to have the opportunity to be real scholars. So we are thinking of more ways to engage students with professional development.

What is the best way to keep updated with the results of your conducted replications? Is there for example a sort of metadata containing all replications and the results?

Jordan: That’s definitely on my wishlist items for the future. I’ve currently been posting results on Twitter ( @CREP_psych) to communicate these manuscripts. We are hoping to also improve the website in the future. This is also a good call for volunteers. If anybody wants to help set any of this stuff up, there are lots of places where people can help. Besides help with the website and communication, we also always need reviewers, who are paid 10$ for the review.

Since CREP started, what have you learned so far about teaching Open Science that you could share with us? Any specific tips for instructors/educators who would like to incorporate OS into their teaching and mentoring?

Jordan: The most surprising thing regarding replications is just how invested students were. I was really worried it would feel forced. I don’t have data to support this, but in my experience students seem as interested in these topics as in topics they choose on their own.

Regarding tips, I don’t believe I have to be the ‘perfect’ teacher. I would say, don’t worry about doing everything. Do what you can and don’t be afraid to reach out and ask people what/how they are doing their class. We have a wonderful community. Being involved in things like SIPS [Society for the Improvement of Psychological Science], FORRT and Twitter really helps to see what other people are saying and doing. My biggest tip is to reach out whenever you have a question. If you are facing an issue about how to run your classroom, there is someone out there who has faced something similar and can help.

CREP’s resources

CREP’s OSF page: The landing page for instructors who would like to start their journey with CREP. It includes CREP’s step-by-step directions and example syllabi that incorporate CREP
CREP’s Twitter page
Jordan Wagge’s Website: more information about Jordan and her Replication & Open Science Lab (ROSE)
Read more about Jordan’s thoughts on CREP from her interview given to Avila University: Helping students become better scientists through authentic research
See Dr. Fieke Wagemans from the University of Duisburg Essen lead a conference session introducing CREP: recorded on September 2018 at the University of Bordeaux
Youtube Playlist of CREP tutorials: information on how to sign up for CREP, how to create a class OSF page, how to prepare and submit to CREP, and how to complete a CREP project
Google Slides presentation on available studies for replication at CREP

Suggested citation

Wagge, J., Beltran, J., & Hernandez, A. (2023). Open Science education through student participation: Integrating replication training into the classroom. FORRT Pedagogies. https://doi.org/10.17605/OSF.IO/4JRFA

Team Pedagogies Contributors:

Jacob Miranda, California State University - East Bay, U.S.

Julia Wolska, Manchester Metropolitan University, UK.

Giorgia Andreolli, Verona University, Italy.

Leticia Micheli, Leiden University, the Netherlands.

Alessio Merighi, Video Editor, https://vimeo.com/user62908209

Developing principled pedagogies - An Open Science journey

Mon, 28 Nov 2022 10:44:40 -0400

FORRT’s Pedagogies are back with no less than Gilad Feldman. Over the past years, Gilad has completely remodelled his research and teaching to incorporate several aspects of Open Science in a way that is certainly exemplary to many of us. In this FORRT’s Pedagogies, Gilad shares his materials and his insights about his own journey in Open Science. We hope that this can inspire and help many scholars wishing to incorporate more Open Science into their teaching and research.

Gilad Feldman is Assistant Professor at the Department of Psychology at the University of Hong Kong (HKU). His research focuses on judgement and decision making and he strongly supports the Open Science movement at all levels - from his teaching philosophy, to the development of open teaching materials and resources, and the involvement of students in Open Science training and practice. One of Gilad’s most known projects is the student-led pre-registered replications and extensions, which you can find here. But be sure to check all his other amazing meta science resources such as his course syllabi and the “Check me, Replicate me” initiative (all links can be found below).

FORRT’s Team Pedagogies took the opportunity to ask Gilad some questions on his journey through and with Open Science (shoutout to the FORRT and academic Twitter community for sending so many great questions). You can watch or listen to the interview in the video or read a summary of his main points below! We hope you enjoy and learn as much as we did!

Check out the full interview here

Playing this video loads it from YouTube (Google). See our privacy policy.

You have an incredible amount of Open Science resources ranging from courses and replications to incentives to other researchers to replicate or find mistakes in your own work. How did you embark on this Open Science journey?

Gilad: In 2015, the big Science paper came out, showing replication rates that in my opinion were alarming. I didn’t understand how bad things really were until this paper came out. I remember being a postdoc and thinking that something big was happening and that I couldn’t in good faith continue with the research I was doing without figuring this out. I then started to think about my principles and decided on my main goals: my research has to be trustworthy, reproducible, replicable. Over time, I also started thinking about equity and inclusion issues. I set my own principles, such as to share materials, data and code. Back in 2016, this was scary stuff! I started to supervise students according to Open Science principles, and the CORE team (Collaborative Open-science and meta REsearch) developed from this.

You have managed to put Open Science at the core of your teaching and research. Can you briefly mention the materials and activities you have developed in your own work to accelerate Open Science?

Gilad: First of all, I have not developed this on my own. I’ve decided that every time I’m going to make a guide or a template, I’m going to make it collaborative. Everyone who contributes to the document puts their name down and if we submit to a journal, they’re co-authors - from students to professors! We have created a bunch of resources, such as research assessments, community resources (e.g., guides on how to do replications and extensions, how to calculate effect sizes, etc.) and collaborative open resources for statistical analysis, such as the Jamovi guide. Every time I see a need, I open a collaborative guide and share it with others (teaching assistants, students, external reviewers, the CORE team) and advertise it on Twitter. Guides have grown exponentially thanks to the community and now anybody can implement them.

Did you face any challenges or critical incidents when incorporating Open Science into your teaching or research? What can scholars expect when embarking on this Open Science journey?

Gilad: Being completely open can be intimidating, and it takes some time to convince students that this model works. At the beginning the major issue was the lack of ready-made resources, but I feel that many of the initial challenges have been overcome.

Psychologists study people, but we rarely point the research that we do to ourselves. I study judgment and decision-making and cognitive biases, and it is amazing how many of these biases we experience ourselves in the academic community and throughout the scientific process. There are lots of biases on how Open Science folks versus more status-quo researchers look at each other. I try to shield my team and students as much as possible from negative comments, but also want them to get a realistic view of the academic world, where there are lots of problems with ego, interests, cognitive biases and heuristics. I try to tell my students that in the end we all want the same thing: the science world is all of us working together. There are lots of challenges, we are humans, but we are learning as a community how to discuss these issues in the open. Humility is key. Taking a step back and listening can help overcome these biases and help us learn from each other.

Fostering collaboration: from the teaching portfolio to course (re)design

You have an amazing teaching portfolio where you detail all your teaching activities and present your teaching philosophy and goals. Could you tell us what you think are the key elements of your teaching philosophy?

Gilad: We really need to reorient what we do in education. I got a lot of inspiration from various projects, such as CREP and the Psychology Science Accelerator. I believe that students and early-career researchers are key actors to address the replication crisis in science. We need to move away from the model of the professor standing in front of students with their powerpoint slides. I believe in alternative approaches such as flipping the classroom. I’m also a fan of problem-based learning: you give problems to students and they work together to figure it out. This works brilliantly in the undergraduate courses that I’m teaching in Hong-Kong. We need to teach students how to learn. We need to understand that science is messy. These are the principles I set (see image below):

I decided to share everything I create: slides, lectures, videos… complete transparency! Everything students do is shared on OSF, which may be scary initially, but increases the sense of accountability. And students understand the importance of doing this: even if there’s the possibility of error, you want to be transparent about the process, so other people can help you do better or find your mistake. In fact, there are lots of reasons why I started with replications and extensions: they are more practical, measurable, systematic, valuable for students in their learning journeys and overall more instructive. I believe that this helps students develop their scientific thinking, whatever career path they decide to take.

Collaboration is one of the pillars of your pedagogy, and in your syllabi you mention a “team contract” that is created together with the students. Could you explain what this team contract is? Why and how investing in/learning team communication can be connected with Open Science practices?

Gilad: Team contract is something relatively new that really boosted collaborative work in my courses, mostly because it helps to align expectations. Students sometimes do not like to work together, and they’re used to multiple choice exams and a more structured model. But there is something very important about teamwork, also later when starting to work in a paid job. The most common complaint in my teaching evaluations at the beginning was: “we do not like to work in teams, we prefer to work individually, whenever we work in teams there are a lot of free-riders”. The team contract helps the team to solve issues students might face and it can be adjusted over the semester: basically, it’s about getting the team together to get to know each other and discuss their project following a flexible template. For example, as part of the team contract, students do a “pre-mortem”: they are asked to imagine an issue has happened, what they would do and how they could have prevented it from happening. At the end of the semester, we ask students to reflect on their learning journey (what they did compared to what they planned). Then, students are required to grade themselves. Everyone was very humble and understood that it is not about me telling them “you have to learn this way”. It is about them setting their own goals, holding themselves accountable, overcoming challenges and reflecting on their journey. In fact, they all got precisely the grade they assigned themselves.

Over time, how did you (re)design your courses to include Open Science principles?

Gilad: At the University of Hong Kong, I am very privileged. I have a lot of freedom and flexibility. It is not a big deal to include Open Science principles into my own courses. I just try things out. Whenever I see a good idea, I try to embed it in my teaching. My students are very open-minded and let me do this and evolve together with me.

Mass Replications & Extensions: workflow, students’ feedback and co-authorship

You have conducted several replications within your courses. Could you walk us through how it works from beginning to end? For example could you share the administrative/organisational side of it? How many students are involved in each course/replication and how do you motivate students to participate in replications?

Gilad: This is complex - I usually give 3 hour workshops on replications! In terms of numbers, some of the courses I teach are large, with 70 students per class. Recently, in the more advanced courses I got about 20 students. We do two types of projects. At the beginning, we completed pre-registered replications and extensions. Within one semester, students went from basic statistics skills to a completed manuscript that was ready to be submitted - with everything written up, from preregistration to data analysis and so on. In 2020, we moved to a model of Registered Reports Stage 1. We got a lot of rejection letters from journals telling us about all the issues in our replications, based on outcome bias and hindsight bias. This is something that Registered Reports tackle very well: I got students to write a Stage 1 Report with a simulated dataset (Qualtrics helps with this) and detailed methodology and analysis sections. The workflow is shown in the image below:

For pre-registered replications, the schedule is very tight, and everything is done within a month and a half. Then, I usually don’t sleep for two or three weeks and give two or three teams the same dataset, so that they can peer review one another. Finally, they produce a final report and an early-career researcher takes the lead as first author. Everybody takes credit and everything is written down. We still have about 25 unfinished projects - so if anybody in the FORRT community is interested, they can come and work with us!

A very similar model is applied to Stage 1 Registered Reports, but without the data collection, while the model for thesis students is a bit different. They have one year, and they do a registered report. We then submit everything to Peer Community in Registered Reports. In terms of impact, students not only gain a publication but they have their chances of being hired (both in academia and industry) increased. This can make a difference for them, so they are very enthusiastic and motivated. They are serious about the work and they know the course itself is different from other courses - hands-on, learning by doing… Some students appreciate this, but not everybody. Students with other preferences can drop out and choose other courses offered at HKU.

As one of your teaching goals, you have mentioned that “students experience the research process from beginning to end”. One of the most amazing things of the Mass Replications and Extensions is that students do make actual contributions to science. In your view, what is the role of students’ contributions to academic literature? Could you share with us the feedback that you got from students involved in replications?

Gilad: Students can feel a lot of pressure, a lot of tension, during the process. Especially at the beginning. It’s so different from everything they have done before, so this can lead to some frustrations… I get some emails from students who have graduated and transitioned to industry, and reflecting back they are able to see these courses as a meaningful part of their journey (for good or bad). Some of them understand that science is messy, and others had never imagined they could have a contribution as undergraduates. It’s about empowering students to take part in the process. One thing I try to show them is how many mistakes professors make and that everybody is capable of contributing, helping us do better. Overall, many students find it meaningful, if not memorable - and they’ve learned important skills in the end. This is my aim and what I was hoping for.

You published several replications that started out as replications in classrooms. Were students interested in contributing to the research papers as co-authors after the end of the course? Can you explain how authorship is attributed and agreed upon?

Gilad: This is an aspect that needs to be sorted out very early on, and we follow a very strict model. In my syllabus there’s a passage about “what it means to take part in this course”. Everybody is a co-author by default, but it’s possible to opt out if one makes it clear at the beginning (and reiterates it at each submission). Until now, and it’s been four years, nobody has opted out. In addition, we don’t want people to be forced by omission, and must ensure that the implications of this model are clear to them. The co-authorship model is explained to them early on, like a co-authorship contract: this is definitely very important to align expectations. I use a mandatory quiz on the course to collect their agreement and keep a written record (we also ask permission to submit articles in their name even after graduation). So, generally, every student is a co-author, with clear documentation of their contribution. We also have a credit contributorship where we keep track of what students, teaching assistants and early-career researchers bring in. Sometimes reviewers push back, so this document is useful to support and justify authorship attribution. Students did everything in their submissions. I guided them here and there but it’s really them who deserve the credit for the initial draft. It is open, transparent and collaborative work, with no room for misunderstandings.

In my PhD, I just remember a lot of interactions between students and professors where professors put themselves as first authors and some students were not even mentioned. I don’t understand how this is a sustainable system. It’s not fair, it’s not equitable, it’s not ethical. Undergraduate students, sometimes even high school students, can do science. There’s nothing about me having a professor title that makes me more eligible for co-authorship than a student. I believe in as much inclusion as possible, and it doesn’t matter where you come from, what rank you are, how old you are. Everybody should be acknowledged and included.

You also encourage other researchers to check and replicate your own work (Check me, Replicate me). Do you also incorporate this in your courses, letting students check your own work? Can you explain how that works?

Gilad: I started in 2019 to do assessments with students, because I understood that before we do replications students need to see how others do replications. I just wanted them to go and have a look at the top-notch replication work in our field. I realized that I really worry that in ten, twenty years somebody is going to find an error that I have made… and funding, millions of dollars, hundreds of hours are going to be wasted! This is partially why I open up everything: I want others to be able to check, for example, my code - which is something that, when you go through peer review, can be overlooked. So I turned this into a learning opportunity. In 2020, we checked our replication reports from 2019. In 2021, we checked our replication reports from 2020, and so on. I was hoping (praying!) that, on the one hand, students wouldn’t find mistakes. On the other hand, though, if there were mistakes I really wanted students to find them. Last year I created a section of my website where I wrote “I will pay you 5 USD for a minor error, and 50 USD for a major error”. To my great surprise, students found mistakes and I paid them quite a bit of money (which they decided to donate to charity). Win-win. I’m promoting this “Check me, Replicate me” as a pledge that other people can also incorporate and I hope that we’ll be able to do more of that in the future.

Do you have any general advice for new(ish) researchers wanting to be more open and transparent about their research and to educators wanting to implement Open Science in their teaching and mentoring?

Gilad: The best way is to just get started. I was able to put my research aside for a year. If you can’t, try to incorporate some Open Science principles in your work. If you can’t free yourself, start doing this with a masters student, as part of a collaboration. Join a team that does this kind of stuff. I suggest starting very, very simple, and building up. We started with a very simple replication, a very simple main effect. It doesn’t have to be complicated, just be as transparent as possible. If you’re too scared, invite me, and I’ll show you how to do it. But in general, start small, or join a big team and see what you can learn.

All links to Gilad Feldman’s teaching resources

Teaching Portfolio: a document detailing achievements, teaching philosophy, goals and practices at HKU (2021)
Course Materials: these include course summaries, syllabi, lecture slides, tutorials, teaching evaluations, and videos
Open Science Talks and Workshops (since 2020): recordings are available on OSF | Open Science Talks and Workshops and Gilad’s Youtube channel
A list of collaborative and open resources: guides and tutorials, examples of students’ work, templates, open projects for early-career researchers
Resources about Open Science and the Science Reform
Collaborative Open-science and meta REsearch: information on the project, mass replications and extensions, the team and how to get involved
The .pdf version of this page, with Gilad’s answers to the community’s question

Suggested Citation

Feldman, G. (2022). Developing principled pedagogies - An Open Science journey: Insights from incorporating open science into teaching and mentoring. FORRT Pedagogies. https://doi.org/10.17605/OSF.IO/QNCBF

Team-Pedagogy Contributors:

Giorgia Andreolli, Verona University, Italy

Julia Wolska, Manchester Metropolitan University, UK

Flavio Azevedo, Cambridge University, UK.

Leticia Micheli, Leiden University, the Netherlands.

Open and Reproducible Science walks into a classroom

Sat, 03 Oct 2020 10:44:40 -0400

Welcome to FORRT’s first Pedagogy. And what better way to kickstart this initiative than hosting a researcher and educator who has been widely recognized by her peers for her outstanding course and materials on theory of Open and Reproducible Science called Psychology’s Credibility Revolution?

FORRT is delighted to partner up with Julia Strand for its first Pedagogy!

Check out her course announcement (and teaser!)

Playing this video loads it from YouTube (Google). See our privacy policy.

Julia Strand is Associate Professor of Psychology at Carleton College and she teaches courses such as Introduction to Psychology, The Psychology of Spoken Words and Sensation and Perception. Julia has also recently prepared and taught a course on Psychology’s Credibility Revolution ( link) which excels in all aspects (e.g., content, creativity, visuals, & functionality).

FORRT has reached out to the academic community on social media to ask which questions other scholars would like Julia to answer about her teaching materials on the Psychology’s Credibility Revolution course (on both technical and non-technical spheres). Below you can find Julia’s answers to these questions. And you can also find in this OSF repository Julia’s complete teaching materials, from which everyone can learn, adapt, or repurpose.

Kudos to Julia for preparing such an amazing course and many thanks to her for being so open to partner up with FORRT to share her pedagogies with the wider community. We hope this can serve as an inspiration and help scholars, instructors and educational institutions interested in integrating open and reproducible scholarship tenets in their teaching and mentoring.

What is your teaching philosophy behind the course “Psychology’s Credibility Revolution”?

Julia: I’m trying to do two things with this class. The first is to familiarize students with concepts related to the replication crisis and subsequent credibility revolution. The second is to prepare my students to write their senior honors theses. At Carleton, all students complete a capstone project for their major (“comps”). In the Psychology department, the foundation of that comps is the term paper written in a seminar students take in the fall of their senior year. My Credibility Revolution course is one of those seminars. So in this course, I’m balancing teaching content with helping provide students with the tools they need to write their comps. The comps papers from my section are on a research topic of the student’s choosing, but written through the lens of something we’ve discussed. I’ve had students conduct systematic reviews, trace how findings have replicated (or not) on a particular topic, and evaluate the construct validity of measurement scales, among many others.

I want students to come away from this course with an appreciation for how research is actually conducted, how the incentive structure of academia can be at odds with scientific rigor, and how we can change our field to encourage science that is rigorous, transparent, and reproducible. I’ve found that teaching open science is a great way to help students be more critical consumers of the literature. I know that many of my students won’t go on to do research in Psychology themselves, but I want them all to have an appreciation for how to evaluate research.

Could you share your thoughts on how well the students grappled with the material and assignment in this course? Do you have an impression of whether this course influenced your students’ approaches to their own research projects?

Julia: Students can really get into this content! I’ve found that talking about open science has a relatively low barrier to entry, even for undergrads. Although it may take time to build up enough subject area expertise to evaluate psychological theories or models, students can readily understand why misplaced incentives will affect behavior, why questionable research practices lead to findings that can’t replicate, and so on. For example, I’ll describe two studies that are intended to test the same question (i.e., studies on age-related priming conducted by John Bargh vs Stéphane Doyen) but differ in methodology. Students then brainstorm all the reasons why the two studies may have reached different conclusions. They are incredibly thoughtful and thorough in identifying all the ways the choices that researchers make can influence the outcome of the study.

I ask my students for their impressions of how the course has changed how they engage with the literature, and they almost uniformly say they read more critically. They also regularly say (to my great delight!) that they pay much more attention to the methods sections.

Did you face any barriers to teaching about the credibility revolution? And if so, how did you overcome these?

Julia: I haven’t, really! My department is very supportive of letting faculty choose their own topics in these seminars and has been quite receptive to my incorporating open science principles in my teaching and research.

How many times have you taught this course? Could you give an indication of how much the course has developed over time?

Julia: I’ve taught it twice and will teach it again next year. The course has changed somewhat as I’ve looked for the right balance between teaching content and preparing my students for working on their senior theses. It also changes a bit each time because open science moves fast, so I need to update the content! But the basic bones of the course have been fairly consistent.

Is there anything that you didn’t have time to cover in this course but wish you did?

Julia: So many things. Given that Carleton is on trimesters and our terms are a short 9.5 weeks, and we also spend a substantial portion of that doing things related to comps, there are many topics I’d love to be able to include: the alpha wars, philosophy of science, computational and statistical reproducibility, meta-analyses, Bayesian vs frequentist approaches, and more. I ended up cutting content this year to take off pressure around the U.S election, as well.

What aspects of the course material do you think are the most important? For example if you only had 6 weeks to deliver this course.

Julia: It can be very easy for courses of this nature to be very pessimistic. The last thing I want is for students to come away with the sense “science is broken and everything is doomed.” So I spend a lot of time and energy helping them to understand all the ways our discipline, and science generally, has figured out to improve the practice of how we do science! I named this course Psychology’s Credibility Revolution rather than Psychology’s Replication Crisis because I really wanted to emphasize that the reforms of the last decade are changing how we do science, for the better.

Do you have a favorite part of the course? Do you think your students have a favorite too?

Julia: I love thinking about and teaching measurement and construct validation. I also love Halloween, and it works very nicely that my lessons on those topics tend to be around Halloween time. I created a fake scale that is intended to measure a construct called “Halloweenophilia” (an overwhelming love of all things halloween related) and ask students how they’d assess the construct validation of the instrument. It’s both spooky and fun.

Did you develop your materials from other teaching materials you found? And, what are your thoughts on adapting other’s teaching materials and making these available?

Julia: One reason I was keen to make my course publicly available is that others who teach similar courses have been so generous in sharing their materials! When I started designing this course, I combed through all the syllabi I could find online and made lots of notes about strategies I wanted to emulate and readings to adopt. So I’m happy to facilitate others doing the same.

A thing about teaching I wish I’d known earlier is that teaching doesn’t have to include a high proportion of invention. I adopt ideas from other teachers all the time. In fact, even if we don’t intend to, all of our teaching philosophies and approaches must be shaped by the courses we’ve taken and strategies we’ve seen others use.

Instructors need to curate the available resources/readings/assignments/approaches to achieve their course goals, fit their student population, etc, but they don’t need to design every single thing from scratch. In fact, adopting some materials from others frees you up to put more thought and energy into the aspects of your class that you create from scratch.

Do you have any general advice for a new(ish) lecturer wanting to organise a course similar to this?

Julia: Don’t be afraid to take inspiration from and borrow from others (see answer to question 8 above)!

In addition to the materials themselves (which are awesome), what should other teachers know or bear in mind when teaching this/similar material?

Julia: One challenge I faced is that much of this content is very interconnected—it’s hard to talk about low rates of replication without discussing publication bias, which is hard to talk about without mentioning null hypothesis significance testing, which then gets you into talking about alternatives to p-values….Which is all great, but it can be hard to wait weeks to talk about some topic that seems integral to understanding earlier content. So I think teaching this content requires coming to terms with the fact that you can’t do everything, and can’t do everything at once.

I’d also recommend using concrete examples from content areas you know well. My research is on spoken word recognition and listening effort (the cognitive resources necessary to understand speech), so when I teach content on measurement for example, I talk a lot about different ways of measuring listening effort. Working through examples that you are very familiar with can help students get an appreciation of the complexities of decisions about experiment design.

A few final, quick-fire technical questions: how did you record the lectures? How did you edit the lectures (e.g. to have your face and slides on screen)? How do you make the cartoon character of yourself?

See Julia’s response to all these technical questions in this great video she prepared!

Playing this video loads it from YouTube (Google). See our privacy policy.

All links

Suggested Citation

Strand, J. (2020) Open and Reproducible Science walks into a classroom: Insights from teaching a course on Psychology’s Credibility Revolution. FORRT Pedagogies. https://doi.org/10.17605/OSF.IO/ZPB8A

Team-Pedagogy Contributors:

Leticia Micheli, Leiden University, the Netherlands.

Flavio Azevedo, Cambridge University, UK.

Pedagogies | FORRT - Framework for Open and Reproducible Research Training

From mental-health breakfast interventions to evidence-based practice

How did you come up with the idea and identified the need for this course?

Can you tell us a bit more about the course set up and organization?

How would you describe the teaching philosophy behind the course?

Research-wise, you are also quite involved in Open Science. Do you incorporate insights from Open Science research into your teaching? What Open Science principles did you integrate in this course?

Which were the most important skills that you wanted your students to develop in this course? Could you give us an example of how you taught students these skills?

In your course, you teach students how to gather and evaluate evidence-based knowledge. Do you integrate different research epistemologies? If so, how?

For their assessment, students had to produce an evidence-based synthesis, evaluation and practical dissemination. Can you tell us a bit more about it?

How did students respond to the course? Did you notice any changes in their attitudes towards science? Could you share your thoughts on how well the students grappled with the material and assignment in this course?

Sometimes we think that incorporating Open Science into teaching means teaching a course on Open Science, but it’s not always possible to do that. What is your advice for teachers who have pre-defined courses/content but want to introduce more Open Science principles into their teaching?

Do you have a favorite part of the course? Do you think your students have a favorite too?

Resources

Suggested citation:

Team Pedagogies Contributors:

Open Science education through student participation

CREP’s resources

Suggested citation

Developing principled pedagogies - An Open Science journey

Fostering collaboration: from the teaching portfolio to course (re)design

Mass Replications & Extensions: workflow, students’ feedback and co-authorship

Open and Reproducible Science walks into a classroom