Glossary

    Community-written definitions of open scholarship terms, described in Nature Human Behaviour (Parsons et al., 2022). How to cite the glossary.

    About the glossary project, and how to cite it
    English Deutsch (German) عربي (Arabic) Türkçe (Turkish) 中文 (Chinese)

    Also consider the Re-SearchTerms app to explore alternative definitions.

    Introduction

    In the last decade, the Open Science movement has introduced and modified many research practices. The breadth of these initiatives can be overwhelming, and digestible introductions to these topics are valuable (e.g. Crüwell et al. 2019; Kathawalla, Silverstein, & Syed, 2020). Creating a shared understanding of the purposes of these initiatives facilitates discussions of the strengths and weaknesses of each practice, ultimately helping us work towards a research utopia (Nosek & Bar-Anan, 2012).

    Accompanying this cultural shift towards increased transparency and rigour has been a wealth of terminology within the zeitgeist of research practice and culture. For those unfamiliar, the new nomenclature can be a barrier to follow and join the discussions; for those familiar, potentially vague or competing definitions can cause confusion and misunderstandings. For example, even the “classic” 2015 paper “Estimating the reproducibility of psychological science” (Open Science Collaboration, 2015) can be argued to assess the replicability of research findings.

    In order to reduce barriers to entry and understanding, we present a Glossary of terms relating to open scholarship. We aim that the glossary will help clarify terminologies, including where terms are used differently/interchangeably or where terms are less known in some fields or among students. We also hope that this glossary will be a welcome resource for those new to these concepts, and that it helps grow their confidence in navigating discussions of open scholarship. We also hope that this glossary aids in mentoring and teaching, and allows newcomers and experts to communicate efficiently.

    If you use this glossary in any form, please cite the associated paper:

    Parsons, S., Azevedo, F., Elsherif, M. M., et al. (2022). A Community-Sourced Glossary of Open Scholarship Terms. Nature Human Behaviour, 6(3), 312–318. https://doi.org/10.1038/s41562-021-01269-4

    👀 [The Open Access link to the copy-edited version at the publisher’s website, and here’s the postprint which is the same as the published version.] ⬅️

    The list of terms we have drafted and reviewed can be found on the left if you are viewing this page on a computer screen or bigger, otherwise they can be found at the bottom of the page. If you hover a word, you will be able to read the full description of the term. To know more about a term, including references, simply click on it and it will bring you to the term page.

    Project Status


    We successfully arrived at the end of Phase 1 🎉. This means we managed to go from an ambitious idea to a full blown crowd-sourced project in which more than 110 collaborators defined via consensus after much discussion and reviewed upwards of 250 Open Scholarship terms. We published our manuscript at Nature Human Behaviour, where all contributors are co-authors. 🥳

    Phase 2


    🚨 Phase 2 of the FORRT glossary project is now open ! 🚨


    Following the success of Phase 1, we invite you to help us continue to improve this resource. We are interested in a wide range of contributions to improve existing definitions, extend the scope of the terms, as well as translating terms to improve accessibility. We have opened four live working documents (see the landing page for instructions and links to working documents). Please read the instructions for contributors. We have prepared these to help guide constructive feedback and facilitate a smooth editorial process.

    We aim to regularly implement suggested changes and improvements. If you believe an existing definition is incorrect please contact the project leads, we aim to correct any mistakes as quickly as possible. We see the glossary as a potential starting point for other projects and resources the community feels may be needed. Please contact us if you have suggestions for publications or have ideas for related projects that could use or adapt the glossary.

    To receive updates please join FORRT’s Slack channel. You can also contact FORRT, and project leads Sam Parsons and Flávio Azevedo. For information on Phase 1 of FORRT’s Glossary Project, see below.

    Glossary Versioning

    We plan to continuously improve, extend, and update our items with community involvement. Versioning will allow the study of the evolution of the terminology. The version-controlled source code of the official releases of the complete Glossary is archived on FORRT’s GitHub, OSF, and Zenodo. The present glossary is the Beta 0.1 version and can be accessed from within FORRT website (as will its updates) below:

    1. Current Version: FORRT’s Glossary | Phase 1 | Beta version 0.1

    Phase 1


    Expand to learn more about details of the Phase 1

    Phase 1 - from an ambitious idea to a crowd-sourced project

    Phase 1 had three parts, A, B, and C. Below you find the explanations of each of them and the instructions given to the contributors.

    Part A

    Project methods and guidelines

    1. Concept

    At the start of Phase 1, the lead writing team developed the overall project concept, including the first version of the Glossary skeleton outlining how we would like to proceed with facilitating and recognizing contributions from the community.

    Through this process, the community-driven glossary development procedure deliberately centred the Open Scholarship ethos of accessibility, diversity, equity, and inclusion. And hence, we aimed to capture the wide scope of Open Scholarship, including terms related to education, diversity, equity, and inclusivity.

    The sentence below, by one of our members, captures the ethos of this project.

    Hey there world, we are doing this glossary thing hoping it is useful. We hope we got most things right, but please let us know when we didn’t and how to improve it (we expect there’s lots to improve, hence a Phase 2). And please be mindful that our goal isn’t to provide definitive definitions but rather create an educational resource aiming at decreasing the burden of educators trying to integrate open and reproducible principles into their teaching as well as increasing accessibility to niche knowledge about Open Scholarship.

    1. The Definitions

    Each entry (or term) should follow a standard format (provided below). The definitions should be concise, ideally no more than three or four sentences, using non-technical language (as much as possible). They must also contain enough information to be useful. Please include supporting information (e.g., citations) for an appropriate reference that gives more detail or an example of the term in practice. If possible, please add the APA formatted reference to the references section –or provide enough information for one of the lead writing team to find it (e.g., the page number being quoted from).

    Where there are several, potentially competing definitions for a term (e.g. some fields use reproducibility and replicability in opposing ways), please enter this as an alternative definition. Alternative definitions should be distinct in some way, and not rephrasing of other definitions. Where there are alternative definitions, it would be maximally beneficial to include a reference for all possible definitions: remember that the goal is to educate on existing terms rather than asserting authority about what is the correct definition.

    1. Community contributions

    In this phase we aim to populate the glossary section. We will share an open invite for contributions via the FORRT community and social media. We invite all interested to: write definitions, comment on existing definitions, add alternative definitions where applicable, and suggest relevant references. If you feel that key terms are missing, please add it - you can let us know, or ask contact us with suggestions in the FORRT slack or email sam.parsons@psy.ox.ac.uk and flavio.azevedo@uni-jena.de. Once all terms have been added, the lead writing team (Parsons, Azevedo, & Elsherif) will develop an abridged version to submit as a manuscript. We outline the kinds of contributions and their correspondence to authorship in more detail in the next section. Don’t forget to add your name and details to the contributions spreadsheet.

    1. Manuscript development and submission

    There are two outputs for this project. First, the entire glossary will appear on the FORRT website. Second, an abridged version will be submitted for publication. The lead writing team will handle the overall manuscript development, project administration, formatting, etc. For the manuscript submission, the lead writing team will be considered joint first authors. A final version will be shared so that all contributors have the chance to check that they are happy with the final version of the manuscript.

    1. Contributions and Authorship

    In this project we will use the CREDIT taxonomy ( https://credit.niso.org/) in this prepared contributors spreadsheet. Please add your details (including ORCID) and contributions as you make them. This will facilitate the development of this project, allow us to easily communicate with all contributors, and ensure that all contributions are recognized.

    Every few days, one of the team will review this document to finalize definitions that have had sufficient input.

    We invite several specific contributions: original draft preparation, and review & editing. To help decide what contributions to select, please refer to these outlines. Please add your details to the contributor spreadsheet as you make any contributions. This will also allow us to contact you as we enter later stages of the manuscript development. It is important to note that it is not our aim to distinguish these contributions in terms of prestige. If you are uncertain, please contact one of the lead writing team members.

    • Writing | Original Draft Preparation: We consider this contribution as, for example, writing at least one full glossary entry. If you wrote the original draft for an entry, please add your name to the “Drafted by” field and be sure to tick the “Original Draft Preparation” checkbox in the contributors spreadsheet.

    • Writing | Review & Editing: We consider this contribution as, for example, providing constructive comments, feedback, and approval, on more than 5 glossary entries (we acknowledge that towards the end of the project the main contribution will be checking definitions for agreement and so it may be difficult for some people to make large writing contributions). Please remember to add your name to the “Reviewed by” field and be sure to tick the “Review & Editing” checkbox in the contributors spreadsheet.

    1. Template & Example

    The first block below is the blank format for an entry (XXX marks each field to fill in); the CRediT entry after it is a completed example.

    Term: XXX

    Definition: XXX

    Related terms: XXX

    Alternative definition: (if applicable)

    Related terms to alternative definition: (if applicable)

    Reference(s): XXX

    Drafted by: XXX

    Reviewed (or Edited) by: XXX; XXX; XXX

    Term: CRediT

    Definition: The Contributor Roles Taxonomy (CRediT; https://credit.niso.org/) is a high-level taxonomy, including 14 roles, that can be used to indicate the roles typically adopted by contributors to scientific scholarly output. The roles describe each contributor’s specific contribution to the scholarly output. They can be assigned multiple times to different authors and one author can also be assigned multiple roles. CRediT includes the following roles: Conceptualization, Data curation, Formal Analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing. A description of the different roles can be found in the work of Brand et al. (2015).
    Related terms: Authorship
    Alternative definition: (if applicable)
    Related terms to alternative definition: (if applicable)
    Reference(s): Brand et al. (2015); Holcombe (2019); https://credit.niso.org/
    Drafted by: Sam Parsons
    Reviewed (or Edited) by: Myriam A. Baum; Matt Jaquiery; Connor Keating; Yuki Yamada


    Part B

    We completely filled the original G-doc with comments and so have moved the project into two fresh documents (retaining your open comments, but not the resolved ones). Please see the links below to keep discussing and working on the terms. Both documents contain all instructions for contributors/authors. If you have any trouble, please contact sam.parsons@psy.ox.ac.uk or flavio.azevedo@uni-jena.de or check on the FORRT Slack channel.


    Part C

    We then worked on our manuscript, published in Nature Human Behaviour in 2022, as well as its implementation in FORRT’s website.

    Editorial advice was given to us and it suggested us to choose 50 items to go into a ‘box’ (a sort of a table that doesn’t have word limits). However, it is of fundamental importance to note that these 50 terms are not the community’s conception —or leading authors’— of ‘main’ terms, or ‘core’ terms, or ‘most important terms’. We tried as much as possible —and in line with FORRT’s mission, FORRT’s Code of Conduct, and FORRT’s Manuscript— to choose items that give representation to a variety of past, present and future issue of Open Scholarship. The chosen 50 terms reflect the diversity and plurality of terms for the broader OS, not only for this or that discipline, or this or that view of what Open Scholarship is. Now, that’s not to say these 50 comprise a perfect list. It is not, and we are bound to disagree on which terms should have made the list and which shouldn’t have. And that’s both normal and OK 😊

    After the manuscript’s submission and the display of defined terms in FORRT’s Glossary webpage, we proceeded to Phase 2, which aims to improve upon existing definitions, extend the scope of terms defined, and translate it to other languages to increase access.

    Feedback

    Would you like to give feedback, help us review terms, or add terms? You can do so by watching this space, joining FORRT’s Slack channel, contacting FORRT, or contacting project leads Sam Parsons and Flávio Azevedo.

    The complete reference list for the glossary can be viewed here.

    A

    • Abstract Bias

      The tendency to report only significant results in the abstract, while reporting non-significant results within the main body of the manuscript (not …

    • Academic Impact

      The contribution that a research output (e.g., published manuscript) makes in shifting understanding and advancing scientific theory, method, and …

    • Accessibility

      Accessibility refers to the ease of access and re-use of materials (e.g., data, code, outputs, publications) for academic purposes, particularly the …

    • Ad hominem bias

      From Latin meaning “to the person”; Judgment of an argument or piece of work influenced by the characteristics of the person who forwarded it, not the …

    • Adversarial (collaborative) commentary

      A commentary in which the original authors of a work and critics of said work collaborate to draft a consensus statement. The aim is to draft a …

    • Adversarial collaboration

      A collaboration where two or more researchers with opposing or contradictory theoretical views —and likely diverging predictions about study results— …

    • Affiliation bias

      This bias occurs when one’s opinions or judgements about the quality of research are influenced by the affiliation of the author(s). When publishing …

    • Aleatoric uncertainty

      Variability in outcomes due to unknowable or inherently random factors. The stochastic component of outcome uncertainty that cannot be reduced through …

    • Altmetrics

      Departing from traditional citation measures, altmetrics (short for “alternative metrics”) provide an assessment of the attention and broader impact …

    • AMNESIA

      AMNESIA is a free anonymization tool to remove identifying information from data. After uploading a dataset that contains personal data, the original …

    • Analytic Flexibility

      Analytic flexibility is a type of researcher degrees of freedom (Simmons, Nelson, & Simonsohn, 2011) that refers specifically to the large number of …

    • Anonymity

      Anonymising data refers to removing, generalising, aggregating or distorting any information which may potentially identify participants, …

    • ARRIVE Guidelines

      The ARRIVE guidelines (Animal Research: Reporting of In Vivo Experiments) are a checklist-based set of reporting guidelines developed to improve …

    • Article Processing Charge (APC)

      An article (sometimes author) processing charge (APC) is a fee charged to authors by a publisher in exchange for publishing and hosting an open access …

    • Authorship

      Authorship assigns credit for research outputs (e.g. manuscripts, data, and software) and accountability for content (McNutt et al. 2018; Patience et …

    • Auxiliary Hypothesis

      All theories contain assumptions about the nature of constructs and how they can be measured. However, not all predictions are derived from theories …

    B

    • Badges (Open Science)

      Badges are symbols that editorial teams add to published manuscripts to acknowledge open science practices and act as incentives for researchers to …

    • Bayes Factor

      A continuous statistical measure for model selection used in Bayesian inference, describing the relative evidence for one model over another, …

    • Bayesian Inference

      A method of statistical inference based upon Bayes’ theorem, which makes use of epistemological (un)certainty using the mathematical language of …

    • Bayesian Parameter Estimation

      A Bayesian approach to estimating parameter values by updating a prior belief about model parameters (i.e., prior distribution) with new evidence …

    • BIDS data structure

      The Brain Imaging Data Structure (BIDS) describes a simple and easy-to-adopt way of organizing neuroimaging, electrophysiological, and behavioral data …

    • BIZARRE

      This acronym refers to Barren, Institutional, Zoo, and other Rare Rearing Environments (BIZARRE). Most research for chimpanzees is conducted on this …

    • Bottom-up approach (to Open Scholarship)

      Within academic culture, an approach focusing on the intrinsic interest of academics to improve the quality of research and research culture, for …

    • Bracketing Interviews

      Bracketing interviews are commonly used within qualitative approaches. During these interviews researchers explore their personal subjectivities and …

    • Bropenscience

      A tongue-in-cheek expression intended to raise awareness of the lack of diverse voices in open science (Bahlai, Bartlett, Burgio et al. 2019; Onie, …

    C

    • CARKing

      Critiquing After the Results are Known (CARKing) refers to presenting a criticism of a design as one that you would have made in advance of the …

    • Center for Open Science (COS)

      A non-profit technology organization based in Charlottesville, Virginia with the mission “to increase openness, integrity, and reproducibility of …

    • Citation bias

      A biased selection of papers or authors cited and included in the references section. When citation bias is present, it is often in a way which would …

    • Citation Diversity Statement

      A current effort trying to increase awareness and mitigate the citation bias in relation to gender and race is the Citation Diversity Statement, a …

    • Citizen Science

      Citizen science refers to projects that actively involve the general public in the scientific endeavour, with the goal of democratizing science. …

    • CKAN

      The Comprehensive Knowledge Archive Network (CKAN) is an open-source data platform and free software that aims to provide tools to streamline …

    • Co-production

      An approach to research where stakeholders who are not traditionally involved in the research process are empowered to collaborate, either at the …

    • COAR Community Framework for Good Practices in Repositories

      A framework which identifies best practices for scientific repositories and evaluation criteria for these practices. Its flexible and multidimensional …

    • Code review

      The process of checking another researcher’s programming (specifically, computer source code) including but not limited to statistical code and data …

    • Codebook

      A codebook is a high-level summary that describes the contents, structure, nature and layout of a data set. A well-documented codebook contains …

    • Collaborative Replication and Education Project (CREP)

      The Collaborative Replication and Education Project (CREP) is an initiative designed to organize and structure replication efforts of highly-cited …

    • Committee on Best Practices in Data Analysis and Sharing (COBIDAS)

      The Organization for Human Brain Mapping (OHBM) neuroimaging community has developed a guideline for best practices in neuroimaging data acquisition, …

    • Communality

      The common ownership of scientific results and methods and the consequent imperative to share both freely. Communality is based on the fact that every …

    • Community Projects

      Collaborative projects that involve researchers from different career levels, disciplines, institutions or countries. Projects may have different …

    • Compendium

      A collection of files prepared by a researcher to support a report or publication that include the data, metadata, programming code, software …

    • Computational reproducibility

      Ability to recreate the same results as the original study (including tables, figures, and quantitative findings), using the same input data, …

    • Conceptual replication

      A replication attempt whereby the primary effect of interest is the same but tested in a different sample and captured in a different way to that …

    • Confirmation bias

      The tendency to seek out, interpret, favor and recall information in a way that supports one’s prior values, beliefs, expectations, or hypothesis.

    • Confirmatory analyses

      Part of the confirmatory-exploratory distinction (Wagenmakers et al., 2012), where confirmatory analyses refer to analyses that were set a priori and …

    • Conflict of interest

      A conflict of interest (COI, also ‘competing interest’) is a financial or non-financial relationship, activity or other interest that might compromise …

    • Consortium authorship

      Only the name of the consortium or organization appears in the author column, and the individuals’ names do not appear in the literature: For example, …

    • Constraints on Generality (COG)

      A statement that explicitly identifies and justifies the target population, and conditions, for the reported findings. Researchers should be explicit …

    • Construct validity

      When used in the context of measurement and testing, construct validity refers to the degree to which a test measures what it claims to be measuring. …

    • Content validity

      The degree to which a measurement includes all aspects of the concept that the researcher claims to measure; “A qualitative type of validity where the …

    • Corrigendum

      A corrigendum (pl. corrigenda, Latin: ’to correct’) documents one or multiple errors within a published work that do not alter the central claim or …

    • Creative Commons (CC) license

      A set of free and easy-to-use copyright licences that define the rights of the authors and users of open data and materials in a standardized way. CC …

    • Creative destruction approach to replication

      Replication efforts should seek not just to support or question the original findings, but also to replace them with revised, stronger theories with …

    • Credibility revolution

      The problems and the solutions resulting from a growing distrust in scientific findings, following concerns about the credibility of scientific claims …

    • CRediT

      The Contributor Roles Taxonomy (CRediT) is a high-level taxonomy used to indicate the roles typically adopted by contributors to scientific scholarly …

    • Criterion validity

      The degree to which a measure corresponds to other valid measures of the same concept. Criterion validity is usually established by calculating …

    • Crowdsourced Research

      Crowdsourced research is a model of the social organisation of research as a large-scale collaboration in which one or more research projects are …

    • Cultural taxation

      The additional labor expected or demanded of members of underrepresented or marginalized minority groups, particularly scholars of color. This labor …

    • Cumulative science

      Goal of any empirical science, it is the pursuit of “the construction of a cumulative base of knowledge upon which the future of the science may be …

    D

    • Data Access and Research Transparency (DA-RT)

      Data Access and Research Transparency ( DA-RT) is an initiative promoting data access and research transparency across different methodological and …

    • Data management plan (DMP)

      A structured document that describes the process of data acquisition, analysis, management and storage during a research project. It also describes …

    • Data sharing

      collection of practices, technologies, cultural elements and legal frameworks that are relevant to the practice of making data used for scholarly …

    • Data visualisation

      Graphical representation of data or information. Data visualisation takes advantage of humans’ well-developed visual processing capacity to convey …

    • Decolonisation

      Coloniality can be described as the naturalisation of concepts such as imperialism, capitalism, and nationalism. Together these concepts can be …

    • Demarcation criterion

      A criterion for distinguishing science from non-science which aims to indicate an optimal way for knowledge of the world to grow. In a Popperian …

    • Direct replication

      As ‘direct replication’ does not have a widely-agreed technical meaning nor there is no clear cut distinction between a direct and conceptual …

    • Diversity

      Diversity refers to between-person (i.e., interindividual) variation in humans, e.g. ability, age, beliefs, cognition, country, disability, ethnicity, …

    • DOI (digital object identifier)

      Digital Object Identifiers (DOI) are alpha-numeric strings that can be assigned to any entity, including: publications (including preprints), …

    • DORA

      The San Francisco Declaration on Research Assessment (DORA) is a global initiative aiming to reduce dependence on journal-based metrics (e.g. journal …

    • Double consciousness

      An identity confusion, as the individual feels like they have two distinct identities. One is to assimilate to the dominant culture at university when …

    • Double-blind peer review

      Evaluation of research products by qualified experts where both the author(s) and reviewer(s) are anonymous to each other. “This approach conceals the …

    E

    • Early career researchers (ECRs)

      A label given to researchers who “range from senior doctoral students to postdoctoral workers who may have up to 10 years postdoctoral education; the …

    • Economic and societal impact

      The contribution a research item makes to the broader economy and society. It also captures the benefits of research to individuals, organisations, …

    • Embargo Period

      Applied to Open Scholarship, in academic publishing, the period of time after an article has been published and before it can be made available as …

    • Epistemic uncertainty

      Systematic uncertainty due to limited data, measurement precision, model or process specification, or lack of knowledge. That is, uncertainty due to …

    • Epistemology

      Alongside ethics, logic, and metaphysics, epistemology is one of the four main branches of philosophy. Epistemology is largely concerned with nature, …

    • Equity

      Different individuals have different starting positions (cf. “opportunity gaps”) and needs. Whereas equal treatment focuses on treating all …

    • Equivalence Testing

      Equivalence tests statistically assess the null hypothesis that a given effect exceeds a minimum criterion to be considered meaningful. Thus, …

    • Error detection

      Broadly refers to examining research data and manuscripts for mistakes or inconsistencies in reporting. Commonly discussed approaches include: …

    • Evidence Synthesis

      This is a type of research method which aims to draw general conclusions to address a research question on a certain topic, phenomenon or effect by …

    • Exploratory data analysis

      Exploratory Data Analysis (EDA) is a well-established statistical tradition that provides conceptual and computational tools for discovering patterns …

    • External Validity

      Whether the findings of a scientific study can be generalized to other contexts outside the study context (different measures, settings, people, …

    F

    • Face validity

      A subjective judgement of how suitable a measure appears to be on the surface, that is, how well a measure is operationalized. For example, judging …

    • FAIR principles

      Describes making scholarly materials Findable, Accessible, Interoperable and Reusable (FAIR). ‘Findable’ and ‘Accessible’ are concerned with where …

    • Feminist psychology

      With a particular focus on gender and sexuality, feminist psychology is inherently concerned with representation, diversity, inclusion, accessibility, …

    • First-last-author-emphasis norm (FLAE)

      An authorship system that assigns the order of authorship depending on the contributions of a given author while simultaneously valuing the first and …

    • FORRT

      Framework of Open Reproducible Research and Teaching. It aims to provide a pedagogical infrastructure designed to recognize and support the teaching …

    • Free Our Knowledge Platform

      A collective action platform aiming to support the open science movement by obtaining pledges from researchers that they will implement certain …

    G

    • Gaming (the system)

      Adopting questionable research practices (QRPs, e.g., salami slicing of an academic paper) that would align with academic incentive structures that …

    • Garden of forking paths

      The typically-invisible decision tree traversed during operationalization and statistical analysis given that ‘there is a one-to-many mapping from …

    • General Data Protection Regulation (GDPR)

      A legal framework of seven principles implemented across the European Union (EU) that aims to safeguard individuals’ information. The framework seeks …

    • Generalizability

      Generalizability refers to how applicable a study’s results are to broader groups of people, settings, or situations they study and how the findings …

    • Gift (or Guest) Authorship

      The inclusion in an article’s author list of individuals who do not meet the criteria for authorship. As authorship is associated with benefits …

    • Git

      A software package for tracking changes in a local set of files (local version control), initially developed by Linus Torvalds. In general, it is used …

    • Goodhart’s Law

      A term coined by economist Charles Goodhart to refer to the observation that measuring something inherently changes user behaviour. In relation to …

    • GPower

      Free to use statistical software for performing power analyses. The user specifies the desired statistical test (e.g. t-test, regression, ANOVA), and …

    H

    • H-index

      Hirsch’s index, abbreviated as H-index, intends to measure both productivity and research impact by combining the number of publications and the …

    • Hackathon

      An organized event where experts, designers, or researchers collaborate for a relatively short amount of time to work intensively on a project or …

    • HARKing

      A questionable research practice termed ‘Hypothesizing After the Results are Known’ (HARKing). “HARKing is defined as presenting a post hoc hypothesis …

    • Hidden Moderators

      Contextual conditions that can, unbeknownst to researchers, make the results of a replication attempt deviate from those of the original study. Hidden …

    • Hypothesis

      A hypothesis is an unproven statement relating the connection between variables (Glass & Hall, 2008) and can be based on prior experiences, scientific …

    I

    • i10-index

      A research metric created by Google Scholar that represents the number of publications a researcher has with at least 10 citations.

    • Ideological bias

      The idea that pre-existing opinions about the quality of research can depend on the ideological views of the author(s). One of the many biases in the …

    • Incentive structure

      The set of evaluation and reward mechanisms (explicit and implicit) for scientists and their work. Incentivised areas within the broader structure …

    • Inclusion

      Inclusion, or inclusivity, refers to a sense of welcome and respect within a given collaborative project or environment (such as academia) where …

    • Induction

      “Reasoning by drawing a conclusion not guaranteed by the premises; for example, by inferring a general rule from a limited number of observations. …

    • Interaction Fallacy

      A statistical error in which a formal test is not conducted to assess the difference between a significant and non-significant correlation (or other …

    • Interlocking

      An analysis at the core of intersectionality to analyse power, inequality and exclusion, as efforts to reform academic culture cannot be completed by …

    • Internal Validity

      An indicator of the extent to which a study’s findings are representative of the true effect in the population of interest and not due to research …

    • Intersectionality

      A term which derives from Black feminist thought and broadly describes how social identities exist within ‘interlocking systems of oppression’ and …

    J

    • JabRef

      An open-sourced, cross-platform citation and reference management tool that is available free of charge. It allows editing BibTeX files, importing …

    • Jamovi

      Free and open source software for data analysis based on the R language. The software has a graphical user interface and provides the R code to the …

    • JASP

      Named after Sir Harold Jeffreys, JASP stands for Jeffrey’s Amazing Statistics Program. It is a free and open source software for data analysis. JASP …

    • Journal Impact Factor™

      The mean number of citations to research articles in that journal over the preceding two years. It is a proprietary and opaque calculation marketed by …

    • JSON file

      JavaScript Object Notation (JSON) is a data format for structured data that can be used to represent attribute-value pairs. Values thereby can contain …

    K

    • Knowledge acquisition

      The process by which the mind decodes or extracts, stores, and relates new information to existing information in long term memory. Given the complex …

    L

    • Likelihood function

      A statistical model of the data used in frequentist and Bayesian analyses, defined up to a constant of proportionality. A likelihood function …

    • Likelihood Principle

      The notion that all information relevant to inference contained in data is provided by the likelihood. The principle suggests that the likelihood …

    • Literature Review

      Researchers often review research records on a given topic to better understand effects and phenomena of interest before embarking on a new research …

    M

    • Manel

      Portmanteau for ‘male panel’, usually to refer to speaker panels at conferences entirely composed of (usually caucasian) males. Typically discussed in …

    • Many authors

      Large-scale collaborative projects involving tens or hundreds of authors from different institutions. This kind of approach has become increasingly …

    • Many Labs

      A series of crowdsourced replication projects in which independent research teams use shared study protocols to examine the replicability of published …

    • Massive Open Online Courses (MOOCs)

      Exclusively online courses which are accessible to any learner at any time, are typically free to access (while not necessarily openly licensed), and …

    • Massively Open Online Papers (MOOPs)

      Unlike the traditional collaborative article, a MOOP follows an open participatory and dynamic model that is not restricted by a predetermined list of …

    • Matthew effect (in science)

      Named for the ‘rich get richer; poor get poorer’ paraphrase of the Gospel of Matthew. Eminent scientists and early-career researchers with a …

    • Meta-analysis

      A meta-analysis is a statistical synthesis of results from a series of studies examining the same phenomenon. A variety of meta-analytic approaches …

    • Meta-science or Meta-research

      The scientific study of science itself with the aim to describe, explain, evaluate and/or improve scientific practices. Meta-science typically …

    • Metadata

      Structured data that describes and synthesises other data. Metadata can help find, organize, and understand data. Examples of metadata include …

    • Model (computational)

      Computational models aim to mathematically translate the phenomena under study to better understand, communicate and predict complex behaviours.

    • Model (philosophy)

      The process by which a verbal description is formalised to remove ambiguity, while also constraining the dimensions a theory can span. The model is …

    • Model (statistical)

      A mathematical representation of observed data that aims to reflect the population under study, allowing for the better understanding of the …

    • Multi-Analyst Studies

      In typical empirical studies, a single researcher or research team conducts the analysis, which creates uncertainty about the extent to which the …

    • Multiplicity

      Potential inflation of Type I error rates (incorrectly rejecting the null hypothesis) because of multiple statistical testing, for example, multiple …

    • Multiverse analysis

      Multiverse analyses are based on all potentially equally justifiable data processing and statistical analysis pipelines that can be employed to test a …

    N

    O

    • Objectivity

      The idea that scientific claims, methods, results and scientists themselves should remain value-free and unbiased, and thus not be affected by …

    • Ontology (Artificial Intelligence)

      A set of axioms in a subject area that help classify and explain the nature of the entities under study and the relationships between them.

    • Open access

      “Free availability of scholarship on the public internet, permitting any users to read, download, copy, distribute, print, search, or link to the full …

    • Open Code

      Making computer code (e.g., programming, analysis code, stimuli generation) freely and publicly available in order to make research methodology and …

    • Open Data

      Open data refers to data that is freely available and readily accessible for use by others without restriction, “Open data and content can be freely …

    • Open Educational Resources (OER) Commons

      OER Commons (with OER standing for open educational resources) is a freely accessible online library allowing teachers to create, share and remix …

    • Open Educational Resources (OERs)

      Learning materials that can be modified and enhanced because their creators have given others permission to do so. The individuals or organizations …

    • Open Licenses

      Open licenses are provided with open data and open software (e.g., analysis code) to define how others can (re)use the licensed material. In setting …

    • Open Material

      Author’s public sharing of materials that were used in a study, “such as survey items, stimulus materials, and experiment programs” (Kidwell et al., …

    • Open Peer Review

      A scholarly review mechanism providing disclosure of any combination of author and referee identities, as well as peer-review reports and editorial …

    • Open Scholarship

      ‘Open scholarship’ is often used synonymously with ‘open science’, but extends to all disciplines, drawing in those which might not traditionally …

    • Open Scholarship Knowledge Base

      The Open Scholarship Knowledge Base (OSKB) is a collaborative initiative to share knowledge on the what, why and how of open scholarship to make this …

    • Open Science

      An umbrella term reflecting the idea that scientific knowledge of all kinds, where appropriate, should be openly accessible, transparent, rigorous, …

    • Open Science Framework

      A free and open source platform for researchers to organize and share their research project and to encourage collaboration. Often used as an open …

    • Open Source software

      A type of computer software in which source code is released under a license that permits others to use, change, and distribute the software to anyone …

    • Open washing

      Open washing, termed after “greenwashing”, refers to the act of claiming openness to secure perceptions of rigor or prestige associated with open …

    • OpenNeuro

      A free platform where researchers can freely and openly share, browse, download and re-use brain imaging data (e.g., MRI, MEG, EEG, iEEG, ECoG, ASL, …

    • Optional Stopping

      The practice of (repeatedly) analyzing data during the data collection process and deciding to stop data collection if a statistical criterion (e.g. …

    • ORCID (Open Researcher and Contributor ID)

      A organisation that provides a registry of persistent unique identifiers (ORCID iDs) for researchers and scholars, allowing these users to link their …

    • Overlay Journal

      Open access electronic journals that collect and curate articles available from other sources (typically preprint servers, such as arXiv). Article …

    P

    • P-curve

      P-curve is a tool for identifying potential publication bias and makes use of the distribution of significant p-values in a series of independent …

    • p-hacking

      Exploiting techniques that may artificially increase the likelihood of obtaining a statistically significant result by meeting the standard …

    • p-value

      A statistic used to evaluate the outcome of a hypothesis test in Null Hypothesis Significance Testing (NHST). It refers to the probability of …

    • Papermill

      An organization that is engaged in scientific misconduct wherein multiple papers are produced by falsifying or fabricating data, e.g. by editing …

    • Paradata

      Data that are captured about the characteristics and context of primary data collected from an individual - distinct from metadata. Paradata can be …

    • PARKing

      PARKing (preregistering after results are known) is defined as the practice where researchers complete an experiment (possibly with infinite …

    • Participatory Research

      Participatory research refers to incorporating the views of people from relevant communities in the entire research process to achieve shared goals …

    • Patient and Public Involvement (PPI)

      Active research collaboration with the population of interest, as opposed to conducting research “about” them. Researchers can incorporate the lived …

    • Paywall

      A technological barrier that permits access to information only to individuals who have paid - either personally, or via an organisation - a …

    • PCI (Peer Community In)

      Peer Community In (PCI) is a non-profit organisation that coordinates free, transparent peer review of preprints through communities of researchers in …

    • PCI Registered Reports

      An initiative launched in 2021 dedicated to receiving, reviewing, and recommending Registered Reports (RRs) across the full spectrum of Science, …

    • Plan S

      Plan S is an initiative, launched in September 2018 by cOAlition S, a consortium of research funding organisations, which aims to accelerate the …

    • Positionality

      The contextualization of both the research environment and the researcher, to define the boundaries within the research was produced (Jafar, 2018). …

    • Positionality Map

      A reflexive tool for practicing explicit positionality in critical qualitative research. The map is to be used “as a flexible starting point to guide …

    • Post Hoc

      Post hoc is borrowed from Latin, meaning “after this”. In statistics, post hoc (or post hoc analysis) refers to the testing of hypotheses not …

    • Post Publication Peer Review

      Peer review that takes place after research has been published. It is typically posted on a dedicated platform (e.g., PubPeer). It is distinct from …

    • Posterior distribution

      A way to summarize one’s updated knowledge in Bayesian inference, balancing prior knowledge with observed data. In statistical terms, posterior …

    • Predatory Publishing

      Predatory (sometimes “vanity”) publishing describes a range of business practices in which publishers seek to profit, primarily by collecting article …

    • PREPARE Guidelines

      The PREPARE guidelines and checklist (Planning Research and Experimental Procedures on Animals: Recommendations for Excellence) aim to help the …

    • Preprint

      A publicly available version of any type of scientific manuscript/research output preceding formal publication, considered a form of Green Open …

    • Preregistration

      The practice of publishing the plan for a study, including research questions/hypotheses, research design, data analysis before the data has been …

    • Preregistration Pledge

      In a “collective action in support of open and reproducible research practices’’, the preregistration pledge is a campaign from the Project Free Our …

    • Prior distribution

      Beliefs held by researchers about the parameters in a statistical model before further evidence is taken into account. A ‘prior’ is expressed as a …

    • PRO (peer review openness) initiative

      The agreement made by several academics that they will not provide a peer review of a manuscript unless certain conditions are met. Specifically, the …

    • Pseudonymisation

      Pseudonymisation refers to a technique that involves replacing or removing any information that could lead to identification of research subjects’ …

    • Pseudoreplication

      When there is a lack of statistical independence presented in the data and thus artificially inflating the number of samples (i.e. replicates). For …

    • Psychometric meta-analysis

      Psychometric meta-analyses aim to correct for attenuation of the effect sizes of interest due to measurement error and other artifacts by using …

    • Public Trust in Science

      Trust in the knowledge, guidelines and recommendations that has been produced or provided by scientists to the benefit of civil society (Hendriks et …

    • Publication bias (File Drawer Problem)

      The failure to publish results based on the “direction or strength of the study findings” (Dickersin & Min, 1993, p. 135). The bias arises when the …

    • Publish or Perish

      An aphorism describing the pressure researchers feel to publish academic manuscripts, often in high prestige academic journals, in order to have a …

    • PubPeer

      A website that allows users to post anonymous peer reviews of research that has been published (i.e. post-publication peer review).

    • Python

      An interpreted general-purpose programming language, intended to be user-friendly and easily readable, originally created by Guido van Rossum in 1991. …

    Q

    R

    • R

      R is a free, open-source programming language and software environment that can be used to conduct statistical analyses and plot data. R was created …

    • Red Teams

      An approach that integrates external criticism by colleagues and peers into the research process. Red teams are based on the idea that research that …

    • Reflexivity

      The process of reflexivity refers to critically considering the knowledge that we produce through research, how it is produced, and our own role as …

    • Registered Report

      A scientific publishing format that includes an initial round of peer review of the background and methods (study design, measurement, and analysis …

    • Registry of Research Data Repositories

      A global registry of research data repositories from different academic disciplines. It includes repositories that enable permanent storage of, …

    • Reliability

      The extent to which repeated measurements lead to the same results. In psychometrics, reliability refers to the extent to which respondents have …

    • Repeatability

      Synonymous with test-retest reliability. It refers to the agreement between the results of successive measurements of the same measure. Repeatability …

    • Replicability

      An umbrella term, used differently across fields, covering concepts of: direct and conceptual replication, computational …

    • Replication Markets

      A replication market is an environment where users bet on the replicability of certain effects. Forecasters are incentivized to make accurate …

    • RepliCATs project

      Collaborative Assessment for Trustworthy Science. The repliCATS project’s aim is to crowdsource predictions about the reliability and replicability of …

    • Reporting Guideline

      A reporting guideline is a “checklist, flow diagram, or structured text to guide authors in reporting a specific type of research, developed using …

    • Repository

      An online archive for the storage of digital objects including research outputs, manuscripts, analysis code and/or data. Examples include preprint …

    • ReproducibiliTea

      A grassroots initiative that helps researchers create local journal clubs at their universities to discuss a range of topics relating to open research …

    • Reproducibility

      A minimum standard on a spectrum of activities (“reproducibility spectrum”) for assessing the value or accuracy of scientific claims based on the …

    • Reproducibility crisis (aka Replicability or replication crisis)

      The finding, and related shift in academic culture and thinking, that a large proportion of scientific studies published across disciplines do not …

    • Reproducibility Network

      A reproducibility network is a consortium of open research working groups, often peer-led. The groups operate on a wheel-and-spoke model across a …

    • Research Contribution Metric (p)

      Type of semantometric measure assessing similarity of publications connected in a citation network. This method uses a simple formula to assess …

    • Research Cycle

      Describes the circular process of conducting scientific research, with “researchers working at various stages of inquiry, from more tentative and …

    • Research Data Management

      Research Data Management (RDM) is a broad concept that includes processes undertaken to create organized, documented, accessible, and reusable quality …

    • Research integrity

      Research integrity is defined by a set of good research practices based on fundamental principles: honesty, reliability, respect and accountability …

    • Research Protocol

      A detailed document prepared before conducting a study, often written as part of ethics and funding applications. The protocol should include …

    • Research workflow

      The process of conducting research from conceptualisation to dissemination. A typical workflow may look like the following: Starting with …

    • Researcher degrees of freedom

      refers to the flexibility often inherent in the scientific process, from hypothesis generation, designing and conducting a research study to …

    • Responsible Research and Innovation

      An approach that considers societal implications and expectations, relating to research and innovation, with the aim to foster inclusivity and …

    • Reverse p-hacking

      Exploiting researcher degrees of freedom during statistical analysis in order to increase the likelihood of accepting the null hypothesis (for …

    • RIOT Science Club

      The RIOT Science Club is a multi-site seminar series that raises awareness and provides training in Reproducible, Interpretable, Open & Transparent …

    • Robustness (analyses)

      The persistence of support for a hypothesis under perturbations of the methodological/analytical pipeline In other words, applying different …

    S

    • Salami slicing

      A questionable research/reporting practice strategy, often done post hoc, to increase the number of publishable manuscripts by ‘slicing’ up the data …

    • Scooping

      The act of reporting or publishing a novel finding prior to another researcher/team. Survey-based research indicates that fear of being scooped is an …

    • Semantometrics

      A class of metrics for evaluating research using full publication text to measure semantic similarity of publications and highlighting an article’s …

    • Sensitive research

      Research that poses a threat to those who are or have been involved in it, including the researchers, the participants, and the wider society. This …

    • Sequence-determines-credit approach (SDC)

      An authorship system that assigns authorship order based on the contribution of each author. The names of the authors are listed according to their …

    • Sherpa Romeo

      An online resource that collects and presents open access policies from publishers, from across the world, providing summaries of individual journal’s …

    • Single-blind peer review

      Evaluation of research products by qualified experts where the reviewer(s) knows the identity of the author(s), but the reviewer(s) remains anonymous …

    • Slow science

      Adopting Open Scholarship practices leads to a longer research process overall, with more focus on transparency, reproducibility, replicability and …

    • Social class

      Social class is usually measured using both objective and subjective measurements, as recommended by the American Psychological Association (American …

    • Social integration

      Social integration is a multi-dimensional construct. In an academic context, social integration is related to the quantity and quality of the social …

    • Society for Open, Reliable, and Transparent Ecology and Evolutionary biology (SORTEE)

      SORTEE ( https://www.sortee.org/) is an international society with the aim of improving the transparency and reliability of research results in the …

    • Society for the Improvement of Psychological Science (SIPS)

      A membership society founded to further promote improved methods and practices in the psychological research field. The society aims to complete its …

    • Specification Curve Analysis

      An analytic approach that consists of identifying, calculating, visualising and interpreting results (through inferential statistics) for all …

    • Statistical Assumptions

      Analytical approaches and models assume certain characteristics of one’s data (e.g., statistical independence, random samples, normality, equal …

    • Statistical power

      Statistical power is the long-run probability that a statistical test correctly rejects the null hypothesis if the alternative hypothesis is true. It …

    • Statistical significance

      A property of a result using Null Hypothesis Significance Testing (NHST) that, given a significance level, is deemed unlikely to have occurred given …

    • Statistical validity

      The extent to which conclusions from a statistical test are accurate and reflective of the true effect found in nature. In other words, whether or not …

    • STRANGE

      The STRANGE “framework” is a proposal and series of questions to help animal behaviour researchers consider sampling biases when planning, performing …

    • StudySwap

      A free online platform through which researchers post brief descriptions of research projects or resources that are available for use (“haves”) or …

    • Systematic Review

      A form of literature review and evidence synthesis. A systematic review will usually include a thorough, repeatable (reproducible) search strategy …

    T

    • Tenzing

      tenzing is an online webapp and R package that helps researchers to track and report the contributions of each team member using the CRediT taxonomy …

    • The Troubling Trio

      Described as a combination of low statistical power, a surprising result, and a p-value only slightly lower than .05.

    • Theory

      A theory is a unifying explanation or description of a process or phenomenon, which is amenable to repeated testing and verifiable through scientific …

    • Theory building

      The process of creating and developing a statement of concepts and their interrelationships to show how and/or why a phenomenon occurs. Theory …

    • Transparency

      Having one’s actions open and accessible for external evaluation. Transparency pertains to researchers being honest about theoretical, methodological, …

    • Transparency Checklist

      The transparency checklist is a consensus-based, comprehensive checklist that contains 36 items that cover the prepregistration, methods, results and …

    • Trim-and-fill method

      A method used in meta-analysis to assess how publication bias might affect the pooled effect estimate. It assumes that missing studies on one side of …

    • Triple-blind peer review

      Evaluation of research products by qualified experts where the author(s) are anonymous to both the reviewer(s) and editor(s). “Blinding of the authors …

    • TRUST Principles

      A set of guiding principles that consider Transparency, Responsibility, User focus, Sustainability, and Technology (TRUST) as the essential components …

    • Type I error

      “Incorrect rejection of a null hypothesis” (Simmons et al., 2011, p. 1359), i.e. finding evidence to reject the null hypothesis that there is no …

    • Type II error

      A false negative result occurs when the alternative hypothesis is true in the population but the null hypothesis is accepted as part of the analysis …

    • Type M error

      A Type M error occurs when a researcher concludes that an effect was observed with magnitude lower or higher than the real one. For example, a type M …

    • Type S error

      A Type S error occurs when a researcher concludes that an effect was observed with an opposite sign than real one. For example, a type S error occurs …

    U

    • Under-representation

      Not all voices, perspectives, and members of the community are adequately represented. Under-representation typically occurs when the voices or …

    • Universal design for learning (UDL)

      A framework for improving learning and optimising teaching based upon scientific insights of how humans learn. It aims to make learning inclusive and …

    V

    • Validity

      Validity refers to the application of statistical principles to arrive at well-founded —i.e., likely corresponding accurately to the real world— …

    • Version control

      The practice of managing and recording changes to digital resources (e.g. files, websites, programmes, etc.) over time so that you can recall specific …

    W

    • Webometrics

      Webometrics involves the study of online content. Webometrics focuses on the numbers and types of hyperlinks between different online sites. Such …

    Z

    • Z-Curve

      Computing a Z-score is a statistical approach mainly used to obtain the ‘Estimated Replication Rate’ (ERR) and ‘Expected Discovery Rate’ (EDR) for a …

    • Zenodo

      An open science repository where researchers can deposit research papers, reports, data sets, research software, and any other research-related …

    Last updated on Nov 9, 2026