Definition: A property of a result using Null Hypothesis Significance Testing (NHST) that, given a significance level, is deemed unlikely to have occurred given the null hypothesis. Tenny and Abdelgawad (2017) defined it as “a measure of the probability of obtaining your data or more extreme data assuming the null hypothesis is true, compared to a pre-selected acceptable level of uncertainty regarding the true answer” (p. 1). Conventions for determining the threshold vary between applications and disciplines but ultimately depend on the considerations of the researcher about an appropriate error margin. The American Statistical Association’s statement (Wasserstein & Lazar, 2016) notes that “Researchers often wish to turn a p-value into a statement about the truth of a null hypothesis, or about the probability that random chance produced the observed data. The p-value is neither. It is a statement about data in relation to a specified hypothetical explanation, and is not a statement about the explanation itself” (p. 131).
Null Hypothesis Significance Testing (NHST),
Type I error
References: Cassidy et al. (2019), Tenny and Abdelgawad (2021), & Wasserstein and Lazar (2016)
Drafted and Reviewed by: Alaa AlDoh, Flávio Azevedo, James E. Bartlett, Alexander Hart, Annalise A. LaPlume, Charlotte R. Pennington, Graham Reid, Timo Roettger, Suzanne L. K. Stewart
Note that we are currently working on an automated mechanism to link references cited above with their full-length version that can be found at https://forrt.org/glossary/references with all references used so far.