4 Documenting Decisions Transparently
This week is about ways to make your research more transparent. As you discovered earlier, transparency means being clear about exactly what you did at every stage of your research. However, over the course of an entire research project (typically months to years), it’s highly likely that you will forget certain aspects of how the study was conducted, and when and why decisions were made.
One of the most important ways you can make sure that others will be able to replicate your research is by keeping detailed records of all aspects of the project, and updating them as you go. It’s a bit like keeping a record of the twists and turns you made while going through a maze. Without detailed notes, remembering all the decisions you made at every stage of your research project can be difficult. You will build up a lot of information, which can be hard to compile once you get to the writing stage.
4.1 Preregistration: publishing your plans for a study
The most important distinction is whether decisions were made before or after data collection begins. This is because when researchers are able to look at the data they can be swayed by what they see. Preregistration (Registration in some fields) is the practice of publishing the plan for a study, including research questions, hypotheses, research design, and data analysis plans before the data has been collected or examined.
A preregistration document is time-stamped and typically registered with an independent party (e.g., an open access repository) so that it can be publicly shared with others. Preregistration provides a transparent documentation of what was planned at a certain time and allows third parties to assess what changes may have occurred afterwards. Importantly, it’s fine for changes to occur, it’s just important to know when and what these were, and why these changes were made.
Having a more detailed preregistration leaves fewer research degrees of freedom. In other words, the more detailed a preregistration is, the better third parties can assess any possible changes and how they may affect confidence in the results.
One platform for preregistering research is the Open Science Framework (OSF). On the OSF there are support videos and documentation to guide you through the process. You can use a variety of templates depending on your discipline and methodology to preregister your study in varying levels of detail. These templates include:
- Standard OSF template (good for most science disciplines)
- Social Psychology
- Qualitative
- Secondary Data
If none of the available templates suit you, you can write your own document and preregister this in an open-ended preregistration!
4.1.1 Confirmatory vs explanatory analysis
Confirmatory analyses refer to analyses that were set before data collection or examination, and that test whether a hypothesis is supported by the data. Exploratory analyses are carried out when some data have already been collected. They are useful for discovering patterns in that data or extending to new topics or subjects. They foster hypothesis development and refinement.
Preregistration often aims to clearly distinguish confirmatory from exploratory analyses. This is helpful because you won’t be able to convince yourself (or others) that you had hypotheses before you saw your data, when actually you added these ‘post-hoc’, after seeing the results.
If you are thinking of preregistering either type of research, here are some things to consider:
- Both quantitative and qualitative research can be confirmatory, and so preregistration for confirmatory research can be used for both.
- For exploratory research, preregistration can be a great way to document initial study plans, even if those later change through an iterative process.
- Preregistering exploratory studies can be useful for both quantitative and qualitative research, for all disciplines.
- If your discipline has another good way of keeping track of how study plans change over the course of the research, then this could also work well instead of preregistration.
It is important to distinguish between confirmatory and exploratory analysis so that results can be interpreted accordingly.
4.1.2 How to approach preregistration
Whether your work is confirmatory or exploratory, preregistering keeps a permanent record of your ideas at the design stage, before you start the analysis.
The process of preregistering involves answering a series of questions about your research. There are many templates of preregistration forms available. In the next activity, we will walk you through some typical questions.
When answering the questions, you should aim to be as precise and detailed as possible. By doing this, you are being transparent about your research plans from the outset. The benefits include establishing a clear and detailed plan for your research, that you can revisit and update as you make your way through your research project.
A detailed and comprehensive preregistration demonstrates that you haven’t engaged in the questionable research practices that you learned about in the last week, and can be useful for reviewers and readers when they assess the integrity of your research.
Activity 1: Preregistration
Allow about 30 minutes.
Time to get your notebook ready.
In the activity below, you will gain experience of the type of information you will need to provide when preregistering a research project. Please answer the questions based on one of your existing research projects, or a project you would like to do in the future.
1.1 What is your research project title?
Show / Hide Discussion
The title of your research project should, in ten to fifteen words, provide an informative description of the research being reported. When coming up with the title you might want to think about the variables, the design of the study, and the key findings of your research project.
1.2 Who is contributing to this research?
Show / Hide Discussion
If you are collaborating with people on this research project, you can list their names and affiliations here. Declaring these provides contextual information about your research, and the academic perspectives the people in your team are likely to bring to it.
1.3 Have any data been collected for this study already?
This question concerns whether you have already collected data. There are three possible options: select the one that best describes the stage you are at with your research, write your own notes on further details, then click ‘Show / Hide Discussion’ to see our comments.
Show / Hide Discussion
You may need to explain how much exposure to the data you've had: the general rule of thumb is the less involvement you’ve had with the data, the better. However this does depend on your project, and preregistration can still protect you from questionable research practices, even if some data has already started to come in.
Show / Hide Discussion
Great! When you are preregistering, the general rule of thumb is the less involvement you’ve had with data, the better. That way, you won’t be tempted to add post hoc justifications after you’ve seen the results.
Show / Hide Discussion
If you select this third option, you’ll need to explicitly state how, and to what capacity, you’ve been exposed to the data previously. When you are hoping to preregister, the general rule of thumb is the less involvement you’ve had with the data, the better, but a project that involves secondary data is a good example of how preregistration can still be valuable, even when a lot of data already exists.
Now let’s continue our walk through preregistration. The next question gets to the heart of the matter: what your research is about. You need to be clear and concise in your responses, so that when you return to your preregistration document, it will clearly encapsulate what your plans were at this point in time.
1.4 What is the main question being asked, or hypothesis being tested in the study?
Here are some tips As you write your notes on your research question, here are some tips to help with your responses:
Your research questions should be specific. If you have more than one hypothesis, you need to write multiple statements (one per hypothesis). It is helpful to write hypotheses in bulleted or numbered format: this forces you to be concise. If you are doing quantitative research, you should also state whether your hypothesis predicts a certain direction and if so what that direction is.
Show / Hide Discussion
Your response will vary according to your discipline and style of research. Try to ensure the research question or hypothesis is as focused as possible. Use simple language and avoid ambiguity. Here is an example:
Research question: Can we replicate the findings of Yoon, Johnson and Csibra [PNAS, 105, 36 (2008)] that nine-month-old infants retain qualitatively different information about novel objects in communicative and non-communicative contexts?
Hypothesis 1: In a communicative context (‘ostensive pointing’), infants will mentally process the identity of novel objects at the expense of mentally processing their location. We would expect longer looking times for changed objects than changed location.
Hypothesis 2: In a non-communicative context (‘non-ostensive reaching’), infants will mentally process the location of novel objects at the expense of encoding their identity. We would expect longer looking times to changed location than changed identity.
Next, you’ll be asked questions about the design of your study. The first of these questions relates to quantitative research. If your research is not quantitative, you may wish to go to question 6 directly.
These are some questions you might want to ask yourself when answering: - What are your independent variables and your dependent variables? How do these variables relate to each other? - How will they be measured (a self-scale report, a behavioural task)? - What is your sample size and criteria? - How did you determine your sample size? - Do you have a Between, within or mixed study design? - Are you using Counterbalancing?
1.5. Describe the design, key variables, and sample, specifying how they will be measured and collected.
In your notes, reflect on why you think this information is important for preregistration.
Show / Hide Discussion
Your response will vary according to your discipline and style of research. Here’s one example:
Independent variable: Our study investigates two conditions: communicative (ostensive pointing) and non-communicative (ostensive reaching).
Dependent variable: Duration of first looks and total looking time, measured using a Tobii eye-tracker. We may also hand-code looking offline (blind) to increase confidence in our results.
Design: Previous research has only found a significant effect for duration of first look, so we only predict differences in this. However, we are still including total looking time: although previous research data were not significant for this variable, they appear to be in the predicted direction.
Measurement: Duration from first video frame when object is revealed to when infant first looks off-screen.
Sampling: We will run the study until twenty four infants that meet the criteria for the experiment have been tested. This will exclude excessive fussiness preventing completion of study or resulting in uncodable eye movement, experimenter and equipment error, caretaker interference, or infants looking off-screen.
If your research is qualitative, you will also be asked to specify exactly how you plan to conduct your research, although your answers are likely to be a little different.
For instance, you might be interested in the experience of parenting a child prodigy. How will you approach the task? With a questionnaire? An interview? A focus group? Open or closed questions? Or supposing you are interested in changes in depictions of families through the twentieth century. What evidence will you use? Newspapers and magazines? Archive photographs? How will you analyse them? Discourse analysis? Visual analysis?
1.6. Describe the study design and how data will be sampled and collected.
As you write your own notes documenting your design and data collection plan, here are some tips to help with your responses:
- What methodologies are you using, e.g.: case study, ethnography?
- What is your sampling, recruiting or case selection strategy?
- What type of data are you interested in, and what is your method for collecting or generating the data?
- You may also want to describe tools, instruments, plans or schedules (e.g.: interview schedule or archival search plan)?
- What criteria need to be reached in order to stop data collection or generation?
Show / Hide Discussion
Your response will vary according to your discipline and style of research. But asking yourself questions like this helps to focus on what you want to know, and how you will be exploring it. Here is one example of a preregistered qualitative study.
We hope this activity has helped you feel confident about your next preregistration. These questions have concentrated on your methodology plan, but you will also need to provide information about your analysis plans. We will explore the analysis stage in more detail in the next week.
4.2 Reporting guidelines
We’ve talked about preregistration as a way to be transparent before you collect your data. How about once you’ve collected your data and are writing up your research? You should be honest about how you conducted your study, and anything that has changed since you planned (and perhaps preregistered) it. One way of being transparent when writing up your research is to use reporting guidelines.
Reporting guidelines are sets of rules or standards that help researchers present their findings clearly and transparently. They’re like a checklist that ensures all-important information about a study is included in a research paper. These guidelines vary depending on the type of study or field of research, but they generally help researchers communicate their methods, results and conclusions effectively, making it easier for others to understand and evaluate their work.
Here are some reporting guidelines for different fields:
STROBE (STrengthening the Reporting of OBservational studies in Epidemiology)
STROBE provides a checklist to enhance the reporting of observational studies in epidemiology, encompassing key aspects such as study design, participant selection, data collection methods, and statistical analysis.
COREQ (COnsolidated criteria for REporting Qualitative research)
COREQ provides a checklist of items that researchers should address when reporting qualitative research, covering aspects such as study design, data collection, analysis, and interpretation.
EQUATOR (Enhancing the QUAlity and Transparency Of health Research)
EQUATOR provides a variety of reporting guideline templates for various branches of health research, including reporting guidelines for randomised trials, observational studies, systematic reviews, qualitative research, animal studies, economic evaluations, and more.
There are many benefits to using reporting guidelines. Most obviously, they help researchers to clearly and comprehensively communicate all the important information about their study. This is helpful for the researcher themselves, and for anyone else who wants to read, understand, and potentially build upon their work. However, if you’re unable to find reporting guidelines for your particular field, being as transparent as possible and including as much detail as possible is your best bet!
Activity 2
Allow about 30 minutes.
In the activity below, you get the chance to practise writing your own simple set of guidelines.
Imagine that your friend has some very important news to tell you. Create a set of reporting guidelines for them, so that they can make sure to include all relevant information about what happened and the people involved when telling you the news. Fill the reporting guidelines out for the piece of news to make sure the guidelines include everything you would need.
Show / Hide Discussion
Here’s an example of what this might look like:
1. Briefly describe the news: Sanjay is moving to Argentina!
2. Outline short descriptions of all people involved in this piece of news:
Sanjay is a 30-year-old sociology researcher who currently lives in the UK.
3. Outline important dates relevant to the news: Sanjay will be moving in September 2026.
4. Provide any background reasoning for the news: Sanjay has been offered a research job in Argentina.
4.3 Applying open research in your own work: The open research decision tree
So far, you have learned a lot about the principles of open research and how they are applied. You may be planning to begin incorporating open research practices into your own research immediately, or perhaps you will want to do so in the future. To help you navigate more quickly to the information you need, the course team have developed an open research decision tree. This has been developed as a companion to the course helps you remind yourself of the principles of open research, and how to take open research actions.
Write down your thoughts:
In what ways are these principles Transparency, Integrity, Accessibility connected to your work, or your field?
What actions can you plan for your research to ensure replicability?
4.4 Quiz 4
- What is preregistration in the context of research? (Select one)
Feedback: Preregistration documents the plan for a study. This is usually before data have been collected, but it can be done after data collection, or when you are using data that already exist (secondary data).
- Documenting the plan for a study Correct
- Publishing the final report of a study Incorrect
- Submitting a research proposal for funding Incorrect
- Registering for a research conference Incorrect
- Which are the most important benefits of preregistration? (Select one or more)
Feedback: Providing a transparent documentation of planned research means that others can know when certain decisions about the research were made, and can assess any changes from your original plan.
- It allows for more flexible research designs Incorrect
- It guarantees funding for the study Incorrect
- It ensures data privacy Incorrect
- It can facilitate collaboration Correct
- It provides transparent documentation of planned research Correct
- Why is it important to distinguish between confirmatory and exploratory analyses? (Select one)
Feedback: Differentiating between which analyses were pre-planned and which were data-driven means that you won’t be able to convince yourself (or others) that you had hypotheses before you saw your data, when actually you thought of these explanations after seeing the results. This helps you avoid questionable research practices.
- To determine the funding sources for each analysis Incorrect
- To limit the scope of the study Incorrect
- To clarify which analyses were pre-planned and which were data-driven Correct
- To enhance the complexity of the research Incorrect
- What should researchers do when writing up their research after data collection? (Select one)
Feedback: Researchers should always be honest about all aspects of a research project. It’s fine to deviate from your original plans, as long as you explain any changes clearly. You are absolutely free to report unexpected findings and exploratory analyses beyond what you specified in your preregistration, these should just be discussed separately from the preregistered analyses.
- Ensure all results align with the initial hypotheses Incorrect
- Be honest about how the study was conducted and any changes since preregistration Correct
- Minimise the reporting of unexpected findings Incorrect
- Alter their preregistered plans to match the results Incorrect
- What are reporting guidelines? (Select one)
Feedback: Reporting guidelines outline how you should write up the results of your studies, not how you should conduct your studies. They are another way (like preregistration) or helping researchers to write clearly and transparently about their research.
- “Sets of rules for how to conduct studies” Incorrect
- “Standards to help researchers present their findings clearly and transparently” Correct
- “Regulations for obtaining research grants” Incorrect
- “Instructions for writing literature reviews” Incorrect
4.5 Summary
In this week, you have dug deeper into transparency in research – documenting how your study was conducted and when and why decisions were made. You learned about preregistration and reporting guidelines: how these can increase transparency, and potentially help you avoid questionable practices.
You were also introduced to the open research interactive decision tree, which you can continue to use throughout the course. You can return to the decision tree after you have finished the course as needed.
In Week 5, you’ll move on to consider the trustworthiness or believability of research findings.
