Framework for
Open and
Reproducible
Research
Training
Replications of previous scientific work are at the core of the Open Scholarship movement. However, as replication efforts become more widespread, it can be challenging to scholars and educators to keep themselves up to date with which effects in their field replicate and which do not. FORRT’s replications and reversals aims to collate replications and specifically so-called reversal effects in social science. Reversals are—in the context of a replication—effects that have their original direction flipped. The extent of such reversals and non-replicated effects is already apparent in the social science literature, with even replicated effects being only half of the originally reported effect (Ioannidis, 2005; Open Science Collaboration, 2015). Although such failures to replicate are far less costly to society than for example medical ones (Prasad & Cifu, 2011), they broadly hinder science’s goal of accumulating knowledge and contribute to waste of scarce resources. This resource aims to be a “living”, freely available, crowd-sourced, and community-driven collection of effects that have either not been replicated or even reversed through empirical research across social sciences. Scholars from varied backgrounds and areas of social science are invited to contribute with prevalent effects in their respective fields.
The purpose of collating these reversal effects in social science is to encourage educators to incorporate replications of these effects into their students' project (e.g., third-year, thesis, course work) to provide them the opportunity to experience the research process directly, assess their ability to perform and report scientific research, and to help evaluate the robustness of the original study, thereby also helping them become good consumers of research. The below crowdsourced and community-curated resource aims to satisfy three of FORRT’s Goals:
and four of FORRT’s Mission:
This is a dynamic project that is organized in four stages for its 1st Phase. Currently, we are in stage 2 of phase 1:
Proof of Concept Phase (adaptation of original project into FORRT, inclusion of effects from social and cognitive psychology, using Gavin Leech’s collection as a basis) → ~150 entries finished in 2021.
Team Science Expansion Phase Across Disciplines (crowd-sourcing entries and refine existing entries), started at the end of 2021 and planned until the end of June 2022. Draft first ‘output’ piece.
Review Phase (open review to identify inconsistencies, missing data, and errors), planned for the end of 2022. Finish first ‘output’ piece. End of Phase 1.
Regular Update Phases (dynamically adding new effects), planned for 2023 and beyond.
Anyone can add reversals or replications by joining our initiative on Slack and then following the instructions in our reversals g-doc.
To search whether an effect already exists in our collection, use Ctrl-F and a keyword in relation to the effect (e.g. “Macbeth” or “Priming”). Please note that not all effects contain all available information, as this is a work in progress.
No good evidence for many forms of priming, automatic behavior change from ‘related’ (often only metaphorically related) stimuli. Semantic priming is still solid, but the effect lasts only seconds.
Elderly priming, that hearing about old age makes people walk slower.
Hostility priming (unscrambled sentences). Exposing participants to more hostility-related stimuli caused them subsequently to interpret ambiguous behaviours as more hostile.
Intelligence priming (contemplation), alt term = professor priming. Participants primed with a category associated with intelligence (e.g. “professor”) performed 13% better on a trivia test than participants primed with a category associated with a lack of intelligence (“soccer hooligans”).
Moral priming (contemplation). Participants exposed to a moral-reminder prime would demonstrate reduced cheating.
Moral priming (cleanliness). Participants exposed to physical cleanliness were shown to reduce the severity of their moral judgments. Direct, well-powered replications did not find evidence for the phenomenon.
Distance priming. Participants primed with distance compared to closeness produced greater enjoyment of media depicting embarrassment (Study 1), less emotional distress from violent media (Study 2), lower estimates of the number of calories in unhealthy food (Study 3), and weaker reports of emotional attachments to family members and hometowns (Study 4).
Flag priming. Participants primed by a flag are more likely to be more in conservative positions than those in the control condition.
Fluency priming. Objects that are fluent (e.g., conceptually fluent, visually fluent) are perceived more concretely than objects that are disfluent (disfluent objects are perceived more abstractly).
Money priming. “Images or phrases related to money cause increased faith in capitalism, and the belief that victims deserve their fate”.
Commitment priming (recall). Participants exposed to a high-commitment prime would exhibit greater forgiveness.
Death priming, alt term = Mortality Salience/Terror Management Theory. Participants not exposed to mortality primes would show higher fear of death.
Spatial priming for emotional closeness. Spatial distances cues were used as a prime for participants’ feelings regarding their emotional closeness to their families (Williams & Bargh, 2008). Participants were asked to plot points on a grid on a paper, either closer or further apart. Then they were asked to rate how emotionally close they feel towards their family members.
Verbal framing (temporal tense). Participants who read what a person was doing (relative to those who read what person did) showed enhanced accessibility of intention-related concepts and attributed more intentionality to the person.
Prosocial spending. Spending money on other people leads to greater happiness than spending money on oneself.
Gustatory disgust on moral judgement. Gustatory disgust triggers a heightened sense of moral wrongness.
Macbeth effect. Moral aspersions induce literal physical hygiene.
Signing at the beginning rather than end makes ethics salient. Signing a statement of honest intent before providing information rather than after can reduce dishonesty.
Stanford Prison Experiment employed a simulation of a prison environment to examine the psychological effects of coercive situations. Utilizing role-playing, labeling and social expectations it showed that one third of participants in the role of prison guards displayed aggressive and dehumanizing behaviour.
Milgram experiment was a study examining the influence of authority on the immoral behaviour. Participants were assigned the role of ‘teachers’ and they were instructed by the experimentator to administer electric shocks of 15-450 V voltage, whenever the ‘learner’ made a mistake. There were various variants of the study. In the most basic one, 100% of participants agree to administer a 300 V shock and 65% agreed to apply to maximum shock od 450 V.
Robbers Cave Study. Utilized arbitrary groupings to demonstrate that tribalism between groups arises spontaneously, and depending on the context, it can result in group competition (e.g., in case of scarce resources) or group cooperation (e.g., in case of superordinate goals and common obstacles)**. **
Digital technology use and adolescent wellbeing. Adolescents who spent more time on new media (including social media and electronic devices such as smartphones) are more likely to report mental health issues.
Anthropomorphism. Individuals who are lonely are more likely than people who are not lonely to attribute humanlike traits (e.g., free will) to nonhuman agents (e.g., an alarm clock),to fulfill unmet needs for belongingness.
Female-named hurricanes are more deadly than male-named ones. Original effect size was a 176% increase in deaths, driven entirely by four outliers; reanalysis using a greatly expanded historical dataset found a nonsignificant decrease in deaths from female named storms.
Implicit bias testing for racism. Implicit bias scores poorly predict actual bias, r = 0.15. The operationalisations used to measure that predictive power are often unrelated to actual discrimination (e.g. ambiguous brain activations). Test-retest reliability of 0.44 for race, which is usually classed as “unacceptable”. This isn’t news; the original study also found very low test-criterion correlations.
The Pygmalion effect, the effect of a teacher’s expectations on a student’s performance, is at most small, temporary, and inconsistent, r<0.1 with a reset after weeks. Rosenthal’s original claims about massive IQ gains, persisting for years, are straightforwardly false (“The largest gain… 24.8 IQ points in excess of the gain shown by the controls.”), and used an invalid test battery. Jussim: “90%–95% of the time, students are unaffected by teacher expectations”.
Stereotype threat on Asian women’s mathematical performance, i.e. the interaction between race, gender and stereotyping. This study found that Asian-American women performed better on a math test when their ethnic identity was activated, but worse when their gender identity was activated, compared with a control group who had neither identity activated.
Stereotype threat on gender differences in political knowledge, the idea that making gender stereotypes about political knowledge salient decrease womens’ performance on political knowledge tests. The replication effort showed no significant effect of gender stereotype activation on womens’ performance on a political knowledge test.
Increase in narcissism (leadership, vanity, entitlement) in young people over the last thirty years. It’s an ancient hypothesis. The basic counterargument is that they’re misidentifying an age effect as a cohort effect (The narcissism construct apparently decreases by about a standard deviation between adolescence and retirement.) “every generation is Generation Me”
Minimal group effect (MGE), alt-term = Minimal group paradigm. Ingroup bias (i.e., the tendency to prefer ingroup members) that appears when participants are assigned to previously unfamiliar, experimentally created and largely meaningless social identities.
Solomon Asch’s conformity study. The study investigated the degree to which a person’s own opinions are influenced by those of a group. The original study is regarded as credible and the main effect has been confirmed multiple times in many cultural contexts (see, Bond and Smith, 1996). Nevertheless, the main effect of an original study had been widely misinterpreted and incorrectly referred to in both academic and popular literature.
Dynamic norms. Information about increasing minority norms increases interest/engagement in minority behaviour.
Social comparison. No robust evidence for an interaction effect between body dissatisfaction and social comparison on fat talk.
Bystander effect. Claims that the feeling of responsibility diffuses with an increasing number of other observers. Research about the bystander effect was sparked by the 1964 murder of Catherine “Kitty” Genovese. See this New York Times article for details. Here’s a more detailed
resource.
Color red on attractiveness. Viewing the color red enhances men’s attraction to women. In a lingua franca this effect may reflect the amorous meaning in the human mating game.
Big brother effect. An original study reported that being watched makes someone more likely to cooperate. People who viewed by a pair of eyes (even when a picture of eyes and not a real person) were three times more likely to contribute to an honesty box used to collect money for drinks (compared to participants who instead saw a picture of flowers), but later meta-analyses did not find this result using very large sample sizes.
Decoy effect
Imagined Contact: the claim that imagining contact (instead of having actual contact) with someone from an outgroup (based on e.g., ethnicity, sexuality, religion, age) can increase contact intention.
Complementarity (opposites attract)
Stereotype susceptibility effects, Awareness of stereotypes about a person’s in-group can affect a person’s behavior and performance when they complete a stereotype-relevant task.
Power pose. Taking on a power pose lowers cortisol and risk tolerance, while it raises testosterone and feelings of power.
Facial Feedback. Smiling causes a good mood, while pouting produces a bad mood.
| Study | Publication status | N | d |
| Andréasson & Dimberg (2008) | published | 112 | -0.22 |
| Andréasson (2010) Study 3 | unpublished | 48 | -0.05 |
| Andréasson (2010) Study 3 | unpublished | 48 | -0.35 |
| Andréasson (2010) Study 4 | unpublished | 44 | 0.49 |
| Andréasson (2010) Study 4 | unpublished | 44 | 0.31 |
| Baumeister et al. (2016) | published | 10 | 1.26 |
| Baumeister et al. (2016) | published | 10 | 0.63 |
| Bodenhausen et al. (1994) | published | 51 | 0.55 |
| Bush et al. (1989) | published | 69 | 0.16 |
| Butler et al. (2003) Study 1 | published | 24 | -0.1 |
| Butler et al. (2003) Study 2 | published | 42 | -0.83 |
| Butler et al. (2006) | published | 69 | -0.03 |
| Cai et al. (2016) | published | 68 | -0.08 |
| Ceschi & Scherer (2003) | published | 64 | 0.74 |
| Clapp (2012) | unpublished | 99 | 0.69 |
| Clapp (2012) | unpublished | 93 | 0.08 |
| Clapp (2012) | unpublished | 93 | 0.17 |
| Clapp (2012) | unpublished | 99 | 0.27 |
| Laird & Crosby (1974) Study 1 | published | 26 | -0.13 |
| Laird & Crosby (1974) Study 2 | published | 26 | 0.35 |
| Davey et al. (2013) Study 1 | published | 28 | 0.41 |
| Davey et al. (2013) Study 1 | published | 14 | 0.62 |
| Davey et al. (2013) Study 1 | published | 28 | 0.52 |
| Davey et al. (2013) Study 1 | published | 14 | 0.13 |
| Davey et al. (2013) Study 1 | published | 28 | 0.69 |
| Davey et al. (2013) Study 1 | published | 14 | 0.42 |
| Davey et al. (2013) Study 1 | published | 28 | 0.35 |
| Davey et al. (2013) Study 1 | published | 14 | 0.14 |
| Davey et al. (2013) Study 2 | published | 29 | 0.73 |
| Davey et al. (2013) Study 2 | published | 15 | 0.63 |
| Davey et al. (2013) Study 2 | published | 29 | 0.4 |
| Davey et al. (2013) Study 2 | published | 15 | 0 |
| Davey et al. (2013) Study 2 | published | 29 | 0.08 |
| Davey et al. (2013) Study 2 | published | 15 | -0.25 |
| Davey et al. (2013) Study 2 | published | 29 | 0.03 |
| Davey et al. (2013) Study 2 | published | 15 | -0.06 |
| Davis (2008) Study 1 | unpublished | 28 | 0.99 |
| Davis (2008) Study 1 | unpublished | 28 | 0.87 |
| Davis (2008) Study 2 | unpublished | 31 | 0.26 |
| Davis (2008) Study 2 | unpublished | 30 | -0.19 |
| Davis et al. (2009) | published | 69 | 0.07 |
| Davis et al. (2009) | published | 69 | 0.51 |
| Davis et al. (2010) | published | 68 | 0.1 |
| Davis et al. (2010) | published | 68 | 0.05 |
| Davis et al. (2010) | published | 68 | -0.15 |
| Davis et al. (2015) | published | 18 | -0.16 |
| Demaree et al. (2004) | published | 53 | 0.62 |
| Demaree et al. (2004) | published | 50 | 0.16 |
| Demaree et al. (2006) | published | 32 | -0.64 |
| Demaree et al. (2006) | published | 35 | 0.06 |
| Demaree et al. (2006) | published | 37 | -0.38 |
| Dillon et al. (2007) | published | 36 | 0.11 |
| Dimberg & Söderkvist (2011) Study 1 | published | 48 | 0.51 |
| Dimberg & Söderkvist (2011) Study 2 | published | 96 | 0.1 |
| Dimberg & Söderkvist (2011) Study 2 | published | 96 | 0.32 |
| Dimberg & Söderkvist (2011) Study 3 | published | 61 | 0.06 |
| Dimberg & Söderkvist (2011) Study 3 | published | 61 | 0.31 |
| Dimberg & Söderkvist (2011) Study 3 | published | 61 | 0.34 |
| Duncan & Laird (1977) | published | 31 | 0.44 |
| Duncan & Laird (1977) | published | 31 | 0.38 |
| Duncan & Laird (1977) | published | 31 | 0.51 |
| Duncan & Laird (1980) | published | 60 | 0.59 |
| Duncan & Laird (1980) | published | 60 | 0.44 |
| Dzokoto et al. (2014) | published | 70 | 1.02 |
| Dzokoto et al. (2014) | published | 59 | 0.07 |
| Dzokoto et al. (2014) | published | 35 | 1.07 |
| Dzokoto et al. (2014) | published | 51 | 0.2 |
| Flack, Laird & Cavallaro (1999b) Study 1 | published | 60 | 1.2 |
| Flack, Laird & Cavallaro (1999b) Study 1 | published | 60 | 0.7 |
| Flack, Laird & Cavallaro (1999b) Study 1 | published | 60 | 0.31 |
| Flack, Laird & Cavallaro (1999b) Study 1 | published | 60 | 0.86 |
| Flack, Laird & Cavallaro (1999b) Study 1 | published | 60 | 1.31 |
| Flack, Laird & Cavallaro (1999b) Study 2 | published | 29 | 0.39 |
| Flack, Laird & Cavallaro (1999b) Study 2 | published | 29 | 0.23 |
| Flack, Laird & Cavallaro (1999b) Study 2 | published | 29 | -0.16 |
| Flack, Laird & Cavallaro (1999b) Study 2 | published | 29 | -0.49 |
| Flack, Laird & Cavallaro (1999b) Study 2 | published | 29 | 0.25 |
| Flack, Laird & Cavallaro (1999a) | published | 54 | 1.41 |
| Flack, Laird & Cavallaro (1999a) | published | 54 | 0.29 |
| Flack, Laird & Cavallaro (1999a) | published | 54 | 1.18 |
| Flack, Laird & Cavallaro (1999a) | published | 54 | 1.21 |
| Flack (2006) | published | 51 | 0.72 |
| Flack (2006) | published | 51 | 0.35 |
| Flack (2006) | published | 51 | 0.59 |
| Flack (2006) | published | 51 | 0.68 |
| Gan et al. (2015) | published | 34 | -0.11 |
| Goldin et al. (2008) | published | 17 | 0.8 |
| Gross & Levenson (1993) | published | 85 | 0.04 |
| Gross & Levenson (1997) | published | 180 | 0.37 |
| Gross & Levenson (1997) | published | 180 | 0.16 |
| Gross (1993) | unpublished | 180 | 0.37 |
| Gross (1993) | unpublished | 180 | 0.09 |
| Gross (1993) | unpublished | 180 | 0.2 |
| Gross (1993) | unpublished | 180 | 0.16 |
| Gross (1993) | unpublished | 180 | -0.23 |
| Gross (1998) | published | 80 | 0.18 |
| Harris (2001) | published | 36 | 0.07 |
| Hawk et al. (2012) | published | 41 | 0.85 |
| Helt & Fein (2016) | published | 43 | 0.42 |
| Hendricks & Buchanan (2016) | published | 79 | -0.08 |
| Hendricks (2013) | unpublished | 79 | 0.02 |
| Henry et al. (2007) | published | 30 | -0.49 |
| Henry et al. (2007) | published | 30 | 0.25 |
| Henry et al. (2009)a | published | 26 | -0.05 |
| Henry et al. (2009)a | published | 26 | 0.53 |
| Henry et al. (2009)b | published | 20 | -0.05 |
| Henry et al. (2009)b | published | 20 | 0.48 |
| Hess et al. (1992) | published | 28 | -0.28 |
| Hess et al. (1992) | published | 28 | 0.14 |
| Hess et al. (1992) | published | 28 | -0.26 |
| Hess et al. (1992) | published | 28 | -0.16 |
| Hofmann et al. (2009) | published | 134 | -0.03 |
| Ito et al. (2006) | published | 40 | -0.39 |
| Ito et al. (2006) | published | 33 | -0.25 |
| Kalokerinos et al. (2015) Study 1 | published | 133.67b | -0.06 |
| Kalokerinos et al. (2015) Study 1 | published | 133.67b | -0.02 |
| Kalokerinos et al. (2015) Study 2 | published | 295 | 1.32 |
| Kalokerinos et al. (2015) Study 2 | published | 295 | 0.2 |
| Kao et al. (2017) | published | 41 | 0.09 |
| Kao et al. (2017) | published | 41 | -0.39 |
| Kao et al. (2017) | published | 41 | 0.8 |
| Kao et al. (2017) | published | 41 | -0.34 |
| Kao et al. (2017) | published | 41 | 0.98 |
| Kao et al. (2017) | published | 41 | -0.67 |
| Kircher et al. (2012) | published | 27 | 1.89 |
| Kircher et al. (2012) | published | 27 | 1.14 |
| Korb et al. (2012) | published | 22 | 0.21 |
| Labott & Teleha (1996) | published | 19 | 0.04 |
| Labott & Teleha (1996) | published | 16 | 0.91 |
| Laird (1974) Study 1 | published | 38 | 0.46 |
| Laird (1974) Study 1 | published | 38 | 0.44 |
| Laird (1974) Study 1 | published | 38 | 0.39 |
| Laird (1974) Study 2 | published | 26 | 0.55 |
| Laird (1974) Study 2 | published | 26 | 0.13 |
| Lalot et al. (2014) | published | 45 | -0.17 |
| Larsen et al. (1992) | published | 27 | 0.43 |
| Lee (2011) | unpublished | 52 | 0.48 |
| Lee (2011) | unpublished | 44 | 0.17 |
| Lee (2011) | unpublished | 52 | -0.27 |
| Lee (2011) | unpublished | 44 | -0.26 |
| Lewis & Bowler (2009) | published | 25 | 1.35 |
| Lewis (2012) | published | 24 | 0.71 |
| Lewis (2012) | published | 24 | 0.56 |
| Ma (2011) | unpublished | 42.67b | -0.21 |
| Ma (2011) | unpublished | 42.67b | -0.21 |
| Ma (2011) | unpublished | 42.67b | -0.21 |
| Ma (2011) | unpublished | 42.67b | -0.21 |
| Maldonado et al. (2015) | unpublished | 157.33b | 0.12 |
| Marmolejo-Ramos & Dunn (2013) Study 1 | published | 100 | -0.07 |
| Marmolejo-Ramos & Dunn (2013) Study 2 | published | 106 | -0.07 |
| Marmolejo-Ramos & Dunn (2013) Study 3 | published | 104 | -0.07 |
| Marmolejo-Ramos & Dunn (2013) Study 4 | published | 100 | -0.07 |
| Marmolejo-Ramos & Dunn (2013) Study 5 | published | 66 | 0.27 |
| Marmolejo-Ramos & Dunn (2013) Study 6 | published | 67 | 0.38 |
| Martijn et al. (2002) | published | 33 | -0.24 |
| McCanne & Anderson (1987) | published | 30 | -2.16 |
| McCanne & Anderson (1987) | published | 30 | -2.07 |
| McCanne & Anderson (1987) | published | 30 | 4.73 |
| McCanne & Anderson (1987) | published | 30 | 1.67 |
| McCanne & Anderson (1987) | published | 30 | 2.48 |
| McCanne & Anderson (1987) | published | 30 | -0.25 |
| McCaul et al. (1982) | published | 27 | 0.25 |
| McIntosh et al. (1997) | published | 26 | 0.54 |
| Meeten et al. (2015) | published | 71 | 0.49 |
| Miyamoto (2006) Study 1 | unpublished | 40 | 0.17 |
| Miyamoto (2006) Study 1 | unpublished | 40 | 0.53 |
| Miyamoto (2006) Study 2 | unpublished | 77 | 0.49 |
| Moore & Zoellner(2012) | published | 23.33b | -0.87 |
| Kappas (1989) | unpublished | 32 | 0.08 |
| Kappas (1989) | unpublished | 32 | 0.26 |
| Kappas (1989) | unpublished | 32 | 0.27 |
| Kappas (1989) | unpublished | 32 | 0.1 |
| Kappas (1989) | unpublished | 32 | 0.17 |
| Kappas (1989) | unpublished | 32 | 0.52 |
| Kappas (1989) | unpublished | 32 | 0.62 |
| Kappas (1989) | unpublished | 32 | 0.74 |
| Kappas (1989) | unpublished | 32 | 0.18 |
| Kappas (1989) | unpublished | 32 | 0.42 |
| Ohira & Kurono (1993) Study 1 | published | 20 | 1.23 |
| Ohira & Kurono (1993) Study 1 | published | 20 | 0.31 |
| Ohira & Kurono (1993) Study 2 | published | 20 | 1.61 |
| Ohira & Kurono (1993) Study 2 | published | 20 | -1.38 |
| Paredes et al. (2013) | published | 31 | 0.85 |
| Paul et al. (2013) | published | 20 | 0.91 |
| Pedder et al. (2016) | published | 68 | 0.7 |
| Pedder et al. (2016) | published | 68 | 0.22 |
| Phillips et al. (2008) | published | 32 | 0.18 |
| Phillips et al. (2008) | published | 32 | 0.08 |
| Reisenzein & Studtmann (2007) Study 1 | published | 53 | 0.18 |
| Reisenzein & Studtmann (2007) Study 1 | published | 55 | 0.34 |
| Reisenzein & Studtmann (2007) Study 1 | published | 55 | -0.08 |
| Reisenzein & Studtmann (2007) Study 1 | published | 55 | 0.3 |
| Reisenzein & Studtmann (2007) Study 1 | published | 53 | -0.12 |
| Reisenzein & Studtmann (2007) Study 1 | published | 53 | 0.22 |
| Reisenzein & Studtmann (2007) Study 1 | published | 52 | -0.04 |
| Reisenzein & Studtmann (2007) Study 1 | published | 52 | -0.09 |
| Reisenzein & Studtmann (2007) Study 3 | published | 40 | -0.74 |
| Richards, Butler & Gross (2003) | published | 59 | 0.19 |
| Richards, Butler & Gross (2003) | published | 59 | -0.12 |
| Richards & Gross (1999) Study 1 | published | 58 | -0.1 |
| Richards & Gross (1999) Study 1 | published | 58 | 0.25 |
| Richards & Gross (1999) Study 1 | published | 58 | 0.36 |
| Richards & Gross (1999) Study 2 | published | 85 | 0.13 |
| Richards & Gross (1999) Study 2 | published | 85 | 0.24 |
| Richards & Gross (1999) Study 2 | published | 85 | 0.06 |
| Richards & Gross (2000) Study 1 | published | 53 | -0.12 |
| Richards & Gross (2000) Study 2 | published | 61 | 0.39 |
| Richards & Gross (2006) | published | 131 | 0.34 |
| Roberts et al. (2008) | published | 160 | 0.07 |
| Robinson & Demaree (2009) | published | 102 | -0.04 |
| Robinson & Demaree (2009) | published | 102 | 0.03 |
| Robinson & Demaree (2009) | published | 102 | 0 |
| Robinson & Demaree (2009) | published | 102 | 0 |
| Roemer (2014) | unpublished | 44 | 0.58 |
| Roemer (2014) | unpublished | 44 | 0.29 |
| Rohrmann et al. (2009) | published | 36 | 0.16 |
| Rohrmann et al. (2009) | published | 36 | 0.13 |
| Rummer et al. (2014) | published | 74 | 0.57 |
| Rummer et al. (2014) | published | 74 | 0.46 |
| Schmeichel , Vohs, & Baumeister (2003) | published | 37 | -0.23 |
| Schmeichel et al. (2008) | published | 50 | 0.1 |
| Söderkvist & Dimberg (unpublished) | unpublished | 32 | 0.36 |
| Söderkvist et al. (2018) Study 1a | unpublished | 32 | 0.34 |
| Söderkvist et al. (2018) Study 2a | unpublished | 64 | 0.17 |
| Soussignan (2002) | published | 33 | -0.17 |
| Soussignan (2002) | published | 33 | 0.48 |
| Soussignan (2002) | published | 33 | 0.47 |
| Soussignan (2002) | published | 33 | 0.44 |
| Soussignan (2002) | published | 32 | 0.53 |
| Soussignan (2002) | published | 32 | 1.1 |
| Soussignan (2002) | published | 32 | 1.11 |
| Soussignan (2002) | published | 32 | 0.94 |
| Stel et al. (2008) Study 2 | published | 18.67b | 1.11 |
| Stel et al. (2008) Study 3 | published | 24 | 1 |
| Strack et al. (1988) Study 1 | published | 76.67b | 0.43 |
| Strack et al. (1988) Study 2 | published | 83 | -0.15 |
| Strack et al. (1988) Study 2 | published | 41.5 | 0.55 |
| Strack et al. (1988) Study 2 | published | 41.5 | -0.51 |
| Tamir et al. (2004) | published | 72 | -0.16 |
| Tourangeau & Ellsworth (1979) | published | 20.5b | 0.3 |
| Tourangeau & Ellsworth (1979) | published | 20.5b | 0.3 |
| Tourangeau & Ellsworth (1979) | published | 20.5b | 0.3 |
| Tourangeau & Ellsworth (1979) | published | 20.5b | 0.3 |
| Trent (2010) | unpublished | 107.33b | -0.22 |
| Trent (2010) | unpublished | 107.33b | -0.22 |
| Trent (2010) | unpublished | 107.33b | -0.06 |
| Trent (2010) | unpublished | 107.33b | -0.06 |
| Vieillard et al. (2015) | published | 31 | 0.25 |
| Vieillard et al. (2015) | published | 31 | 0.66 |
| Vieillard et al. (2015) | published | 30 | 0.21 |
| Vieillard et al. (2015) | published | 30 | 0.14 |
| Vieillard et al. (2015) | published | 31 | -0.05 |
| Vieillard et al. (2015) | published | 31 | -0.5 |
| Vieillard et al. (2015) | published | 30 | 0.07 |
| Vieillard et al. (2015) | published | 30 | -0.12 |
| Wagenmakers et al. (2016) Albohn site | published | 139 | 0.09 |
| Wagenmakers et al. (2016) Allard site | published | 125 | 0.09 |
| Wagenmakers et al. (2016) Benning site | published | 115 | -0.01 |
| Wagenmakers et al. (2016) Bulnes site | published | 101 | 0.09 |
| Wagenmakers et al. (2016) Capaldi site | published | 117 | -0.07 |
| Wagenmakers et al. (2016) Chasten site | published | 94 | -0.04 |
| Wagenmakers et al. (2016) Holmes site | published | 99 | 0.15 |
| Wagenmakers et al. (2016) Koch site | published | 100 | -0.14 |
| Wagenmakers et al. (2016) Korb site | published | 101 | 0.01 |
| Wagenmakers et al. (2016) Lynott site | published | 126 | 0.23 |
| Wagenmakers et al. (2016) Oosterwijk site | published | 110 | -0.17 |
| Wagenmakers et al. (2016) Ozdogru site | published | 87 | -0.3 |
| Wagenmakers et al. (2016) Pacheco-Unguetti site | published | 120 | -0.08 |
| Wagenmakers et al. (2016) Talarico site | published | 112 | 0.02 |
| Wagenmakers et al. (2016) Wagenmakers site | published | 130 | 0.13 |
| Wagenmakers et al. (2016) Wayand site | published | 110 | -0.14 |
| Wagenmakers et al. (2016) Zeelenberg site | published | 108 | 0.25 |
| Wittmer (1985) | unpublished | 30 | -0.36 |
| Wittmer (1985) | unpublished | 30 | -0.21 |
| Yartz (2004) | unpublished | 28 | -0.05 |
| Yartz (2004) | unpublished | 30 | -0.18 |
| Yartz (2004) | unpublished | 28 | -0.08 |
| Yartz (2004) | unpublished | 30 | -0.09 |
| Yartz (2004) | unpublished | 28 | 0.04 |
| Yartz (2004) | unpublished | 30 | 0.5 |
| Zajonc et al. (1989) Study 3 | published | 37 | 1.27 |
| Zajonc et al. (1989) Study 4 | published | 26 | 0.47 |
| Zajonc et al. (1989) Study 4 | published | 26 | 0.31 |
| Zariffa et al. (2014) | published | 24 | -0.57 |
| Zariffa et al. (2014) | published | 24 | -0.14 |
| Zhu et al. (2015) | published | 55 | 1.74 |
No good evidence for Blue Monday, that the third week in January is the peak of depression or low affect ‘as measured by a simple mathematical formula developed on behalf of Sky Travel’. You’d need a huge sample size, in the thousands, to detect the effect reliably and this has never been done.
Reason to be cautious about mindfulness for mental health. Most studies are low quality and use inconsistent designs, there’s higher heterogeneity than other mental health treatments, and there’s strong reason to suspect reporting bias (see also this relevant review; Van Dam, 2017). None of the 36 meta-analyses before 2016 mentioned publication bias.
Mindfulness for wellbeing - effectiveness of MBIs on wellbeing
Mindfulness and mental health - Correlates of mindfulness and mental health
Mindfulness and wellbeing - Correlates of mindfulness and well-being
Ego depletion, that willpower is limited in a muscle-like fashion.
Dunning-Kruger effect. No evidence for the “Mount Stupid” misinterpretation.
Depressive realism effect, of increased predictive accuracy or decreased cognitive bias among the clinically depressed.
Hungry judge effect, of massively reduced acquittals (d=2) just before lunch. Case order isn’t independent of acquittal probability (“unrepresented prisoners usually go last and are less likely to be granted parole”); favourable cases may take predictably longer and so are pushed until after recess; effect size is implausible on priors; explanation involved ego depletion.
No good evidence for multiple intelligences (in the sense of statistically independent components of cognition). Gardner, the inventor: “Nor, indeed, have I carried out experiments designed to test the theory… I readily admit that the theory is no longer current. Several fields of knowledge have advanced significantly since the early 1980s.
At most weak evidence for brain training (that is, “far transfer” from daily training games to fluid intelligence) in general, in particular from the Dual n-Back game.
Generalized cognitive improvements following brief interventions. Cognitive improvements elicited by many interventions are not reliable, and their ecological validity remains limited ( Moreau 2021). In general, be highly suspicious of anything that claims a positive permanent effect on adult or children IQ. Even in children the absolute maximum is 4- 15 points for a powerful single intervention (iodine supplementation during pregnancy in deficient populations).
At most weak evidence for brain training, as in “far transfer” from daily computer training games to fluid intelligence in general, in particular from the Dual n-Back game.
Music lessons improve intelligence. An original experimental study found an increase in IQ for children who received a year of music lessons, compared to children who were randomly assigned to drama lessons or no lessons. Later studies did not replicate this when comparing music lessons to other forms of training such as dance lessons. However, observational studies do suggest that adult musicians have enhanced cognitive abilities compared to non-musicians or even bilinguals ( D’Souza et al., 2018), but this work is correlational, and one possible reason of such positive effects is that the personality traits of people who self-select into long-term music training (e.g., conscientiousness) also drives them to success in other areas. This finding is a good case study of the problem of miscommunication of scientific research in the press ( Mehr, 2015), because the original study has not been replicated in over five attempts, yet it continues to be cited and used by the media and educators.
Brain training, that video gaming enhances cognitive ability.
Bilingual advantages in executive control. The popular hypothesis was that speaking two languages also improves general cognitive control processes (executive control). However, this was challenged by a growing body of systematic studies, which showed no bilingual advantage across different executive control tasks, or even a small bilingual disadvantage. The lack of an effect was even found in exact replications of the original tasks, especially as sample size increases ( Paap et al., 2015), and after accounting for the main moderators proposed by the bilingualism literature (Gunnerud et al., 2020).+
Mozart effect. Listening to Mozart’s sonata for two pianos in D major (KV 448) enhances performance on spatial tasks in standardized tests.
Enhancing effects on IQ
Automatic imitation
Congruence Sequence effect
Action-sentence Compatibility Effect (ACE), participants’ movements are faster when the sentence meaning is consistent with their moving direction.
The attentional spatial-numerical association of response codes (Att-SNARC) effect is the finding that participants had quicker detects to left-side targets preceded by small numbers and to the right-side targets preceded by large numbers. This finding triggered many assumptions about the number representations grounded in body experience.
Scarcity effect. Having too little resources leads individuals to misallocate attention, leading to consequences such as overborrowing. Study 1 examined whether scarcity causes greater cognitive fatigue, measured by poorer performance on a cognitive ability task.
Scarcity effect - Meaning in life. Threats to people’s sense that they can afford things that they need in the present and foreseeable future, undermines perceptions of meaning in life.
Scarcity effect - Discounting. A negative income shock was associated with increased discounting rates for gains and loses.
Scarcity effect - Physical pain. The higher the economic insecurity is associated with the higher the physical pain.
Scarcity effect - **_Self expansion. _**Lower self-concept clarity (conceptualized as a finite resource) is associated with lower self-expansion.
Scarcity effect **-_ Wellbeing. _**Imagining having less time available in one’s current city is positively associated with well-being.
Scarcity effect - _Decision making. _ Lacking time or money can lead to making worse decisions.
Scarcity effect - Opportunity costs. Poor people are more likely to consider opportunity costs spontaneously.
**Scarcity effect -_ Conscious thoughts._** Thoughts triggered by financial concerns intrude more often into consciousness of poorer individuals than for wealthier individuals.
**Scarcity effect -_ Absoluteness of losses_.** Poorer individuals view losses in more absolute, rather than relative, terms than do wealthier individuals.
Bottomless soup bowl. Visual cues related to portion size increase intake volume of soup.
Simon effect. Faster responses are observed when the stimulus and response are on the same side than when the stimulus and response are on opposite sides.
ERP and lie detectors
Evaluative conditioning
Bilingual deficit in lexical retrieval. Compared to monolinguals, bilinguals have often been found to be slower or less accurate in accessing the meaning of a certain word or the word for a certain representation under certain conditions.
Visiting a place will cue memories
Spacing effect
False memories
Motor priming
Associative priming
Repetition priming
Shape simulation
Flanker task
Mere Exposure Effect, the mere exposure effect refers to the finding that participants who are repeatedly exposed to the same stimuli rate them more positively than stimuli that have not been presented before.
Cocktail Party Effect, the cocktail party effect refers to the finding that approximately one third of participants hear their own name being presented in the irrelevant message during a dichotic listening task. Sometimes the impression is given that all participants demonstrate the effect. This is mentioned for example by Conway, Cowan & Bunting (2001): “Contrary to popular belief, not all subjects demonstrate this cocktail party effect.”. However, both in the original study and in the replications, less than half of the participants reported hearing their own name (29-43 percent).
Some evidence for a tiny effect of growth mindset (thinking that skill is improvable) on attainment. Really we should distinguish the correlation of the mindset with attainment vs. the effect of a 1-hour class about the importance of growth-mindset on attainment. I cover the latter but check out Sisk for evidence against both.
“Expertise attained after 10,000 hours practice” (Gladwell). Disowned by the supposed proponents.
Tailoring teaching to students’ preferred learning styles has any effect on objective measures of attainment.
Neonate imitation. Babies are born with the ability to imitate.
Watching violence on aggression
2D:4D ratio of the fingers and its correlation with psychological traits. This ratio was used as a predictor for different interindividual (e.g., intelligence) and especially gender differences.
Personality traits and consequential life outcomes
Reading prevents cancer
Graphology
In general, the replication success rate of Judgment and Decision Making (JDM) is decent, 68% according to Collaborative Open-science Research (2022) led by Dr. Gilad Feldman, a mass replication project. That said, similar to other subfields in social sciences, many effects are overestimated and there are publication biases.
The effect of “nudges” (choice architecture that promotes beneficial decisions) may be exaggerated in general. One comprehensive review by DellaVigna and Linos (2020) found average effects by Nudge Units were six times smaller than studies published in academic journals. (Not saying there are no big effects.), see another systematic review by Szaszi et al. (2018) as well as Hummel and Maedche (2019).
Identifiable victim effect is much weaker and less robust than previously thought, with lots of mixed findings, failed replications, null findings and numerous boundary conditions. See Lee and Feeley (2016) meta-analysis, with r = .05.
Psychophysical numbing. People prefer to save lives if they are a higher proportion of the total (e.g. people prefer to save 4,500 lives out of 11,000 or 4,500 lives out of 250,000?).
Here are a few cautionary pieces on whether, aside from the pure question of reproducibility, behavioural science is ready to steer policy.
Moving the signature box to the top of forms does not decrease dishonest reporting in the rest of the form.
Loss aversion. The subjective value of losses exceeds the subjective values of gains.
Unconscious Thought Advantage, or “deliberation-without-attention”, the idea that for complex choices (with more features to take into account), not deliberating leads to better decisions (as defined by the research team, i.e., normatively).
Self-interest is Overestimated: how much do personal benefits affect policy preferences and behaviors?
Marshmallow experiment
Differential reinforcement
Extinction bursts
Functional communication training
Derived relational responding
Schedules of R+
Above- and Below Average Effect. Above-and-below-average effects arise when comparing oneself to others, whereby people rate themselves as above average for easy abilities and below average for difficult abilities. All standardized beta-values from multiple regression predicting judgmental weight of own and others’ abilities from mean comparative ability estimates in different ability domains were consistent with the original. Additionally, different statistical tests resulted in slightly smaller effects in the same direction of the original.
Accuracy of information (truth discernment), asking people to think about the accuracy of a single headline improves “truth discernment” of intentions to share news headlines about COVID-19.
OpenMKTG.org records all retractions and replications in marketing.
Brian Wansink accidentally admitted gross malpractice; fatal errors were found in 50 of his lab’s papers. These include flashy results about increased portion size massively reducing satiety.
Choice overload, the idea that giving people too many choices can lead to certain undesirable consequences such as reduced purchasing intentions, is in doubt, but most people don’t consider it to be discredited.
Mate guarding, the idea that women use conspicuous luxurious goods to deter female rivals by signaling to other women they have a devoted partner.
Super size me. Larger food options are associated with higher status.
Scarcity effect - Overborrowing. Perceived financial scarcity causes consumers to overborrow.
Scarcity effect - Resource allocation. Poor economic conditions favour resource allocations to daughters over sons.
Scarcity effect - Planning. Consumers who feel resource constrained shift to engage in relatively more priority planning, rather than efficiency planning.
Scarcity effect - Competition/threat. Exposure to limited-quantity promotion advertising prompts consumers to perceive other shoppers as competitive threats.
Scarcity effect - Brand attitudes. Observing luxury brand consumers whose consumption arises from unearned financial resources reduces observers’ brand attitudes when observers place a high value on fairness.
Scarcity effect - Product use creativity. Scarcity salience is associated with greater creativity.
Scarcity effect - Wage rates. The difference in implied wage rates based on a time elicitation versus a money elicitation procedure is reduced as the time horizon increases.
Scarcity effect - Selfishness. Reminders of scarcity causes selfish behaviour to a greater extent in people with low social value orientation.
Scarcity effect - Preference for material goods. Scarcity leads to a preference for material goods over experiential goods.
Scarcity effect - Preference polarization. Perceived scarcity leads to greater preference polarization and stronger preference for a preferred option over a less preferred option.
One mind per hemisphere. The corpus callosotomy studies which purported to show “two consciousnesses” inhabiting the same brain were badly overinterpreted.
Existence of high-functioning (IQ ~ 100) hydrocephalic people. The hypothesis begins from extreme prior improbability; the effect of massive volume loss is claimed to be on average positive for cognition; the case studies are often questionable and involve little detailed study of the brains (e.g. 1970 scanners were not capable of the precision claimed).
Readiness potentials seem to be actually causal, not diagnostic. So Libet’s studies also do not show what they purport to. We still don’t have free will (since random circuit noise can tip us when the evidence is weak), but in a different way.
No good evidence for left/right hemisphere dominance correlating with personality differences. No clear hemisphere dominance at all in this study.
Oxytocin on trust. Intranasal administration of oxytocin increases trust in strangers in a laboratory setting.
Structural brain-behaviour correlations - the association between behavioural activation and white matter integrity. Individual differences in the sensitivity to signals of reward as indexed by BAS-Total and in the tendency to seek out potentially rewarding experiences as measured by BAS-Fun are positively correlated with diffusion measures of several white matter pathways.
Structural brain behaviour correlations - The association between social network size and grey matter volume. Individual differences in the number of Facebook friends (FBN) are positively correlated with grey matter volume in several brain areas: left middle temporal gyrus (MTG), right superior temporal sulcus (STS), rich entorhinal cortex (EC), left and right amygdala.
Structural brain-behaviour correlations - the association between distractibility and grey matter volume. Variability in self-reported distractibility is positively correlated with grey matter volume in the left superior parietal lobule (SPL) and negatively correlated with grey matter volume in medial pre-frontal cortex (mPFC).
Structural brain-behaviour correlations - the association between attention and cortical thickness. Individual differences in executive control are negatively correlated with cortical thickness in left anterior cingulate cortex (ACC), left superior temporal gyrus (STG), and right middle temporal gyrus (MTG), whereas variation in alerting scores is negatively correlated with cortical thickness in the left superior parietal lobule (SPL).
Structural brain-behaviour correlations - the association between control over speed/accuracy of perceptual decisions and white matter tracts strength. Individual differences in control over speed and accuracy of perceptual decisions are positively correlated with the strength of white matter tracts between the right presupplementary motor area (pre-SMA) and the right striatum.
At most extremely weak evidence that psychiatric hospitals (of the 1970s) could not detect sane patients in the absence of deception.
Low self-esteem on poor mental health/psychological outcomes. Small amount of slightly mixed evidence for some outcomes but not supported for most outcomes e.g. alcohol/smoking/drug use etc.
Rorschach Test, as a diagnostic tool for psychiatric conditions.
Lunar effect, alt term = Transylvania effect, this suggests there is a correlation between the full moon and strange occurances, particularly human behaviour. This is thought to have existed as a folk belief for centuries, and is widely believed today (e.g. According to Owen & McGowan, 81% of mental health professionals believe in this effect, with 69% of mental health nurses believing that full moons are associated with an increase in patient admissions - Francescani & Bacon, 2008).
Lack of a Theory of Mind is universal in autism. All autistic people fail to understand that other people have a mind or that they themselves have a mind.
Romantic priming, that looking at attractive women increases men’s conspicuous consumption, time discount, risk-taking. Weak, despite there being 43 independent confirmatory studies!: one of the strongest publication biases / p-hacking ever found.
Implicit religious priming. Implicitly priming god concepts by unscrambling sentences with words relating to religion increases prosocial behaviour in an anonymous economic game.
Implicit analytic priming, that implicitly priming analytic thinking by seeing a photo of Auguste Rodin’s The Thinker decreases belief in God.
Menstrual cycle version of the dual-mating-strategy hypothesis (that “heterosexual women show stronger preferences for uncommitted sexual relationships [with more masculine men] during the high-fertility ovulatory phase of the menstrual cycle, while preferring long-term relationships at other points”). Studies are usually tiny (median n=34, mostly over one cycle). Funnel plot looks ok though.
Menstrual cycle and lunar influence
Large parents have more sons (Kanazawa); original analysis makes several errors and reanalysis shows near-zero effect. (Original effect size: 8% more likely.)
Men’s strength in particular predicts opposition to egalitarianism.
No good evidence that 5-HTTLPR is strongly linked to depression, insomnia, PTSD, anxiety, and more. See also COMT and APOE for intelligence, BDNF for schizophrenia, 5-HT2a for everything.
Be very suspicious of any such “candidate gene” finding (post-hoc data mining showing large >1% contributions from a single allele). 0/18 replications in candidate genes for depression. 73% of candidates failed to replicate in psychiatry in general. One big journal won’t publish them anymore without several accompanying replications. A huge GWAS, n=1 million: “We find no evidence of enrichment for genes previously hypothesized to relate to risk tolerance.”
Critical period hypothesis. How grammar-learning ability changes with age, finding that it is intact to the crux of adulthood (17.4 years) and then declines steadily.
Motivational role of L2 vision. Mental imagery of oneself as a successful language user in the future can enhance one’s motivation and performance.
Flipped learning, students learn better if they do homework about a lesson before coming to class to study that lesson.
Mindsets, people’s beliefs about whether their talents and abilities are subject to growth and improvement. According to the meta-analysis by Sisk and colleagues (
2018), the relationship between mindsets and academic achievement is weak: Of the 129 studies that they analyzed, only 37% found a positive relationship between mindset and academic outcomes. Furthermore, 58% of the studies found no relationship and 6% found a negative relationship between mindset and academic outcomes. Evidence on the efficacy of mindset interventions is not promising: of the 29 studies reviewed, only 12% had a positive effect, 86% of the studies found no effect of the intervention and 2% found a negative effect of the intervention. It should be noted that interventions seemed to work for low SES populations.
First instinct fallacy
Strephosymbolia/ mirror reading and writing in dyslexia
Pen mightier than the screen
Sleep assisted learning
Stereotype threat on gender differences in political knowledge. Making gender stereotypes about political knowledge salient decreases womens’ performance on political knowledge tests.
Avoidance of dissonance-arousing situations
Depressed-entitlement effect among women
Moral foundations across political spectrum
Bilingualism and stuttering, Bilingual children had an increased risk of stuttering and a lower chance of recovery from stuttering than language exclusive and monolingual speakers.
Self esteem and stuttering. Children who stutter have higher self-esteem than children who do not stutter. However, the self-esteem of children who stutter declines once they reach adolescence.
Multi-Analyst Projects collated by Balazs Aczel
A review of 2500 social science papers, showing the lack of correlation between citations and replicability, between journal status and replicability, and the apparent lack of improvement since 2009.
Diederick Stapel: 58 retractions here.
Discussion on Everything Hertz, Hacker News, Andrew Gelman, some star data thugs comment.
See also the popular literature with uncritical treatments of the original studies: