Robustness (analyses)

Definition: The persistence of support for a hypothesis under perturbations of the methodological/analytical pipeline In other words, applying different methods/analysis pipelines to examine if the same conclusion is supported under analytical different conditions.

Alternative definition: “Robustness refers to the stability of experimental conclusions to variations in either baseline assumptions or experimental procedures. It is somewhat related to the concept of generalizability (also known as transportability), which refers to the persistence of an effect in settings different from and outside of an experimental framework […] Whether a study design is similar enough to the original to be considered a replication, a “robustness test,” or some of many variations of pure replication that have been identified, particularly in the social sciences (for example, conceptual replication, pseudoreplication), is an unsettled question” (Goodman et al., 2016).

Related terms: Many Labs, Multiverse analysis, Sensitivity analyses, Specification Curve Analysis

References: Goodman et al. (2016) (alternative), & Nosek and Errington (2020)

Drafted and Reviewed by: Tina Lonsdorf, Flávio Azevedo, Gilad Feldman, Adrien Fillon, Helena Hartmann, Timo Roettger

Note that we are currently working on an automated mechanism to link references cited above with their full-length version that can be found at with all references used so far.