Replications of previous scientific work are at the core of the Open Scholarship movement. However, as replication efforts become more widespread, it can be challenging to scholars and educators to keep themselves up to date with which effects in their field replicate and which do not. FORRT’s replications and reversals aims to collate replications and specifically so-called reversal effects in social science. Reversals are—in the context of a replication—effects that have their original direction flipped. The extent of such reversals and non-replicated effects is already apparent in the social science literature, with even replicated effects being only half of the originally reported effect (Ioannidis, 2005; Open Science Collaboration, 2015). Although such failures to replicate are far less costly to society than for example medical ones (Prasad & Cifu, 2011), they broadly hinder science’s goal of accumulating knowledge and contribute to waste of scarce resources. This resource aims to be a “living”, freely available, crowd-sourced, and community-driven collection of effects that have either not been replicated or even reversed through empirical research across social sciences. Scholars from varied backgrounds and areas of social science are invited to contribute with prevalent effects in their respective fields.
The purpose of collating these reversal effects in social science is to encourage educators to incorporate replications of these effects into their students' project (e.g., third-year, thesis, course work) to provide them the opportunity to experience the research process directly, assess their ability to perform and report scientific research, and to help evaluate the robustness of the original study, thereby also helping them become good consumers of research. The below crowdsourced and community-curated resource aims to satisfy three of FORRT’s Goals:
and four of FORRT’s Mission:
This is a dynamic project that is organized in four stages. Currently, we are in stage 2:
Proof of Concept Phase (adaptation of original project into FORRT, inclusion of effects from social and cognitive psychology, using Gavin Leech’s collection as a basis) → ~150 entries finished in 2021.
Team Science Expansion Phase Across Disciplines (crowd-sourcing entries and refine existing entries), started at the end of 2021 and planned until mid of 2023. Draft of first ‘output’ piece. Currently a total of 320+ effects documented, 220+ of those finished.
Review Phase (open review to identify inconsistencies, missing data, and errors), planned for the end of 2023. Finish first ‘output’ piece.
Regular Update Phases (dynamically adding new effects), planned for 2024 and beyond.
For the latest updates, sent in October 2022 to every contributor, please check this summarizing email
Anyone can add new effects or edit existing effects by joining our initiative on Slack and then following the instructions in our reversals g-doc. Currently we are focusing on finishing already existing effects.
You can find a list of all effects we are working on here. To search whether an effect already exists in our collection, use Ctrl-F and a keyword in relation to the effect (e.g. “Macbeth” or “Priming”). Please note that not all effects are listed here or contain all available information, as this is a work in progress (last site update: 28th of November 2022)
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
Study | Publication status | N | d |
Andréasson & Dimberg (2008) | published | 112 | -0.22 |
Andréasson (2010) Study 3 | unpublished | 48 | -0.05 |
Andréasson (2010) Study 3 | unpublished | 48 | -0.35 |
Andréasson (2010) Study 4 | unpublished | 44 | 0.49 |
Andréasson (2010) Study 4 | unpublished | 44 | 0.31 |
Baumeister et al. (2016) | published | 10 | 1.26 |
Baumeister et al. (2016) | published | 10 | 0.63 |
Bodenhausen et al. (1994) | published | 51 | 0.55 |
Bush et al. (1989) | published | 69 | 0.16 |
Butler et al. (2003) Study 1 | published | 24 | -0.1 |
Butler et al. (2003) Study 2 | published | 42 | -0.83 |
Butler et al. (2006) | published | 69 | -0.03 |
Cai et al. (2016) | published | 68 | -0.08 |
Ceschi & Scherer (2003) | published | 64 | 0.74 |
Clapp (2012) | unpublished | 99 | 0.69 |
Clapp (2012) | unpublished | 93 | 0.08 |
Clapp (2012) | unpublished | 93 | 0.17 |
Clapp (2012) | unpublished | 99 | 0.27 |
Laird & Crosby (1974) Study 1 | published | 26 | -0.13 |
Laird & Crosby (1974) Study 2 | published | 26 | 0.35 |
Davey et al. (2013) Study 1 | published | 28 | 0.41 |
Davey et al. (2013) Study 1 | published | 14 | 0.62 |
Davey et al. (2013) Study 1 | published | 28 | 0.52 |
Davey et al. (2013) Study 1 | published | 14 | 0.13 |
Davey et al. (2013) Study 1 | published | 28 | 0.69 |
Davey et al. (2013) Study 1 | published | 14 | 0.42 |
Davey et al. (2013) Study 1 | published | 28 | 0.35 |
Davey et al. (2013) Study 1 | published | 14 | 0.14 |
Davey et al. (2013) Study 2 | published | 29 | 0.73 |
Davey et al. (2013) Study 2 | published | 15 | 0.63 |
Davey et al. (2013) Study 2 | published | 29 | 0.4 |
Davey et al. (2013) Study 2 | published | 15 | 0 |
Davey et al. (2013) Study 2 | published | 29 | 0.08 |
Davey et al. (2013) Study 2 | published | 15 | -0.25 |
Davey et al. (2013) Study 2 | published | 29 | 0.03 |
Davey et al. (2013) Study 2 | published | 15 | -0.06 |
Davis (2008) Study 1 | unpublished | 28 | 0.99 |
Davis (2008) Study 1 | unpublished | 28 | 0.87 |
Davis (2008) Study 2 | unpublished | 31 | 0.26 |
Davis (2008) Study 2 | unpublished | 30 | -0.19 |
Davis et al. (2009) | published | 69 | 0.07 |
Davis et al. (2009) | published | 69 | 0.51 |
Davis et al. (2010) | published | 68 | 0.1 |
Davis et al. (2010) | published | 68 | 0.05 |
Davis et al. (2010) | published | 68 | -0.15 |
Davis et al. (2015) | published | 18 | -0.16 |
Demaree et al. (2004) | published | 53 | 0.62 |
Demaree et al. (2004) | published | 50 | 0.16 |
Demaree et al. (2006) | published | 32 | -0.64 |
Demaree et al. (2006) | published | 35 | 0.06 |
Demaree et al. (2006) | published | 37 | -0.38 |
Dillon et al. (2007) | published | 36 | 0.11 |
Dimberg & Söderkvist (2011) Study 1 | published | 48 | 0.51 |
Dimberg & Söderkvist (2011) Study 2 | published | 96 | 0.1 |
Dimberg & Söderkvist (2011) Study 2 | published | 96 | 0.32 |
Dimberg & Söderkvist (2011) Study 3 | published | 61 | 0.06 |
Dimberg & Söderkvist (2011) Study 3 | published | 61 | 0.31 |
Dimberg & Söderkvist (2011) Study 3 | published | 61 | 0.34 |
Duncan & Laird (1977) | published | 31 | 0.44 |
Duncan & Laird (1977) | published | 31 | 0.38 |
Duncan & Laird (1977) | published | 31 | 0.51 |
Duncan & Laird (1980) | published | 60 | 0.59 |
Duncan & Laird (1980) | published | 60 | 0.44 |
Dzokoto et al. (2014) | published | 70 | 1.02 |
Dzokoto et al. (2014) | published | 59 | 0.07 |
Dzokoto et al. (2014) | published | 35 | 1.07 |
Dzokoto et al. (2014) | published | 51 | 0.2 |
Flack, Laird & Cavallaro (1999b) Study 1 | published | 60 | 1.2 |
Flack, Laird & Cavallaro (1999b) Study 1 | published | 60 | 0.7 |
Flack, Laird & Cavallaro (1999b) Study 1 | published | 60 | 0.31 |
Flack, Laird & Cavallaro (1999b) Study 1 | published | 60 | 0.86 |
Flack, Laird & Cavallaro (1999b) Study 1 | published | 60 | 1.31 |
Flack, Laird & Cavallaro (1999b) Study 2 | published | 29 | 0.39 |
Flack, Laird & Cavallaro (1999b) Study 2 | published | 29 | 0.23 |
Flack, Laird & Cavallaro (1999b) Study 2 | published | 29 | -0.16 |
Flack, Laird & Cavallaro (1999b) Study 2 | published | 29 | -0.49 |
Flack, Laird & Cavallaro (1999b) Study 2 | published | 29 | 0.25 |
Flack, Laird & Cavallaro (1999a) | published | 54 | 1.41 |
Flack, Laird & Cavallaro (1999a) | published | 54 | 0.29 |
Flack, Laird & Cavallaro (1999a) | published | 54 | 1.18 |
Flack, Laird & Cavallaro (1999a) | published | 54 | 1.21 |
Flack (2006) | published | 51 | 0.72 |
Flack (2006) | published | 51 | 0.35 |
Flack (2006) | published | 51 | 0.59 |
Flack (2006) | published | 51 | 0.68 |
Gan et al. (2015) | published | 34 | -0.11 |
Goldin et al. (2008) | published | 17 | 0.8 |
Gross & Levenson (1993) | published | 85 | 0.04 |
Gross & Levenson (1997) | published | 180 | 0.37 |
Gross & Levenson (1997) | published | 180 | 0.16 |
Gross (1993) | unpublished | 180 | 0.37 |
Gross (1993) | unpublished | 180 | 0.09 |
Gross (1993) | unpublished | 180 | 0.2 |
Gross (1993) | unpublished | 180 | 0.16 |
Gross (1993) | unpublished | 180 | -0.23 |
Gross (1998) | published | 80 | 0.18 |
Harris (2001) | published | 36 | 0.07 |
Hawk et al. (2012) | published | 41 | 0.85 |
Helt & Fein (2016) | published | 43 | 0.42 |
Hendricks & Buchanan (2016) | published | 79 | -0.08 |
Hendricks (2013) | unpublished | 79 | 0.02 |
Henry et al. (2007) | published | 30 | -0.49 |
Henry et al. (2007) | published | 30 | 0.25 |
Henry et al. (2009)a | published | 26 | -0.05 |
Henry et al. (2009)a | published | 26 | 0.53 |
Henry et al. (2009)b | published | 20 | -0.05 |
Henry et al. (2009)b | published | 20 | 0.48 |
Hess et al. (1992) | published | 28 | -0.28 |
Hess et al. (1992) | published | 28 | 0.14 |
Hess et al. (1992) | published | 28 | -0.26 |
Hess et al. (1992) | published | 28 | -0.16 |
Hofmann et al. (2009) | published | 134 | -0.03 |
Ito et al. (2006) | published | 40 | -0.39 |
Ito et al. (2006) | published | 33 | -0.25 |
Kalokerinos et al. (2015) Study 1 | published | 133.67b | -0.06 |
Kalokerinos et al. (2015) Study 1 | published | 133.67b | -0.02 |
Kalokerinos et al. (2015) Study 2 | published | 295 | 1.32 |
Kalokerinos et al. (2015) Study 2 | published | 295 | 0.2 |
Kao et al. (2017) | published | 41 | 0.09 |
Kao et al. (2017) | published | 41 | -0.39 |
Kao et al. (2017) | published | 41 | 0.8 |
Kao et al. (2017) | published | 41 | -0.34 |
Kao et al. (2017) | published | 41 | 0.98 |
Kao et al. (2017) | published | 41 | -0.67 |
Kircher et al. (2012) | published | 27 | 1.89 |
Kircher et al. (2012) | published | 27 | 1.14 |
Korb et al. (2012) | published | 22 | 0.21 |
Labott & Teleha (1996) | published | 19 | 0.04 |
Labott & Teleha (1996) | published | 16 | 0.91 |
Laird (1974) Study 1 | published | 38 | 0.46 |
Laird (1974) Study 1 | published | 38 | 0.44 |
Laird (1974) Study 1 | published | 38 | 0.39 |
Laird (1974) Study 2 | published | 26 | 0.55 |
Laird (1974) Study 2 | published | 26 | 0.13 |
Lalot et al. (2014) | published | 45 | -0.17 |
Larsen et al. (1992) | published | 27 | 0.43 |
Lee (2011) | unpublished | 52 | 0.48 |
Lee (2011) | unpublished | 44 | 0.17 |
Lee (2011) | unpublished | 52 | -0.27 |
Lee (2011) | unpublished | 44 | -0.26 |
Lewis & Bowler (2009) | published | 25 | 1.35 |
Lewis (2012) | published | 24 | 0.71 |
Lewis (2012) | published | 24 | 0.56 |
Ma (2011) | unpublished | 42.67b | -0.21 |
Ma (2011) | unpublished | 42.67b | -0.21 |
Ma (2011) | unpublished | 42.67b | -0.21 |
Ma (2011) | unpublished | 42.67b | -0.21 |
Maldonado et al. (2015) | unpublished | 157.33b | 0.12 |
Marmolejo-Ramos & Dunn (2013) Study 1 | published | 100 | -0.07 |
Marmolejo-Ramos & Dunn (2013) Study 2 | published | 106 | -0.07 |
Marmolejo-Ramos & Dunn (2013) Study 3 | published | 104 | -0.07 |
Marmolejo-Ramos & Dunn (2013) Study 4 | published | 100 | -0.07 |
Marmolejo-Ramos & Dunn (2013) Study 5 | published | 66 | 0.27 |
Marmolejo-Ramos & Dunn (2013) Study 6 | published | 67 | 0.38 |
Martijn et al. (2002) | published | 33 | -0.24 |
McCanne & Anderson (1987) | published | 30 | -2.16 |
McCanne & Anderson (1987) | published | 30 | -2.07 |
McCanne & Anderson (1987) | published | 30 | 4.73 |
McCanne & Anderson (1987) | published | 30 | 1.67 |
McCanne & Anderson (1987) | published | 30 | 2.48 |
McCanne & Anderson (1987) | published | 30 | -0.25 |
McCaul et al. (1982) | published | 27 | 0.25 |
McIntosh et al. (1997) | published | 26 | 0.54 |
Meeten et al. (2015) | published | 71 | 0.49 |
Miyamoto (2006) Study 1 | unpublished | 40 | 0.17 |
Miyamoto (2006) Study 1 | unpublished | 40 | 0.53 |
Miyamoto (2006) Study 2 | unpublished | 77 | 0.49 |
Moore & Zoellner(2012) | published | 23.33b | -0.87 |
Kappas (1989) | unpublished | 32 | 0.08 |
Kappas (1989) | unpublished | 32 | 0.26 |
Kappas (1989) | unpublished | 32 | 0.27 |
Kappas (1989) | unpublished | 32 | 0.1 |
Kappas (1989) | unpublished | 32 | 0.17 |
Kappas (1989) | unpublished | 32 | 0.52 |
Kappas (1989) | unpublished | 32 | 0.62 |
Kappas (1989) | unpublished | 32 | 0.74 |
Kappas (1989) | unpublished | 32 | 0.18 |
Kappas (1989) | unpublished | 32 | 0.42 |
Ohira & Kurono (1993) Study 1 | published | 20 | 1.23 |
Ohira & Kurono (1993) Study 1 | published | 20 | 0.31 |
Ohira & Kurono (1993) Study 2 | published | 20 | 1.61 |
Ohira & Kurono (1993) Study 2 | published | 20 | -1.38 |
Paredes et al. (2013) | published | 31 | 0.85 |
Paul et al. (2013) | published | 20 | 0.91 |
Pedder et al. (2016) | published | 68 | 0.7 |
Pedder et al. (2016) | published | 68 | 0.22 |
Phillips et al. (2008) | published | 32 | 0.18 |
Phillips et al. (2008) | published | 32 | 0.08 |
Reisenzein & Studtmann (2007) Study 1 | published | 53 | 0.18 |
Reisenzein & Studtmann (2007) Study 1 | published | 55 | 0.34 |
Reisenzein & Studtmann (2007) Study 1 | published | 55 | -0.08 |
Reisenzein & Studtmann (2007) Study 1 | published | 55 | 0.3 |
Reisenzein & Studtmann (2007) Study 1 | published | 53 | -0.12 |
Reisenzein & Studtmann (2007) Study 1 | published | 53 | 0.22 |
Reisenzein & Studtmann (2007) Study 1 | published | 52 | -0.04 |
Reisenzein & Studtmann (2007) Study 1 | published | 52 | -0.09 |
Reisenzein & Studtmann (2007) Study 3 | published | 40 | -0.74 |
Richards, Butler & Gross (2003) | published | 59 | 0.19 |
Richards, Butler & Gross (2003) | published | 59 | -0.12 |
Richards & Gross (1999) Study 1 | published | 58 | -0.1 |
Richards & Gross (1999) Study 1 | published | 58 | 0.25 |
Richards & Gross (1999) Study 1 | published | 58 | 0.36 |
Richards & Gross (1999) Study 2 | published | 85 | 0.13 |
Richards & Gross (1999) Study 2 | published | 85 | 0.24 |
Richards & Gross (1999) Study 2 | published | 85 | 0.06 |
Richards & Gross (2000) Study 1 | published | 53 | -0.12 |
Richards & Gross (2000) Study 2 | published | 61 | 0.39 |
Richards & Gross (2006) | published | 131 | 0.34 |
Roberts et al. (2008) | published | 160 | 0.07 |
Robinson & Demaree (2009) | published | 102 | -0.04 |
Robinson & Demaree (2009) | published | 102 | 0.03 |
Robinson & Demaree (2009) | published | 102 | 0 |
Robinson & Demaree (2009) | published | 102 | 0 |
Roemer (2014) | unpublished | 44 | 0.58 |
Roemer (2014) | unpublished | 44 | 0.29 |
Rohrmann et al. (2009) | published | 36 | 0.16 |
Rohrmann et al. (2009) | published | 36 | 0.13 |
Rummer et al. (2014) | published | 74 | 0.57 |
Rummer et al. (2014) | published | 74 | 0.46 |
Schmeichel , Vohs, & Baumeister (2003) | published | 37 | -0.23 |
Schmeichel et al. (2008) | published | 50 | 0.1 |
Söderkvist & Dimberg (unpublished) | unpublished | 32 | 0.36 |
Söderkvist et al. (2018) Study 1a | unpublished | 32 | 0.34 |
Söderkvist et al. (2018) Study 2a | unpublished | 64 | 0.17 |
Soussignan (2002) | published | 33 | -0.17 |
Soussignan (2002) | published | 33 | 0.48 |
Soussignan (2002) | published | 33 | 0.47 |
Soussignan (2002) | published | 33 | 0.44 |
Soussignan (2002) | published | 32 | 0.53 |
Soussignan (2002) | published | 32 | 1.1 |
Soussignan (2002) | published | 32 | 1.11 |
Soussignan (2002) | published | 32 | 0.94 |
Stel et al. (2008) Study 2 | published | 18.67b | 1.11 |
Stel et al. (2008) Study 3 | published | 24 | 1 |
Strack et al. (1988) Study 1 | published | 76.67b | 0.43 |
Strack et al. (1988) Study 2 | published | 83 | -0.15 |
Strack et al. (1988) Study 2 | published | 41.5 | 0.55 |
Strack et al. (1988) Study 2 | published | 41.5 | -0.51 |
Tamir et al. (2004) | published | 72 | -0.16 |
Tourangeau & Ellsworth (1979) | published | 20.5b | 0.3 |
Tourangeau & Ellsworth (1979) | published | 20.5b | 0.3 |
Tourangeau & Ellsworth (1979) | published | 20.5b | 0.3 |
Tourangeau & Ellsworth (1979) | published | 20.5b | 0.3 |
Trent (2010) | unpublished | 107.33b | -0.22 |
Trent (2010) | unpublished | 107.33b | -0.22 |
Trent (2010) | unpublished | 107.33b | -0.06 |
Trent (2010) | unpublished | 107.33b | -0.06 |
Vieillard et al. (2015) | published | 31 | 0.25 |
Vieillard et al. (2015) | published | 31 | 0.66 |
Vieillard et al. (2015) | published | 30 | 0.21 |
Vieillard et al. (2015) | published | 30 | 0.14 |
Vieillard et al. (2015) | published | 31 | -0.05 |
Vieillard et al. (2015) | published | 31 | -0.5 |
Vieillard et al. (2015) | published | 30 | 0.07 |
Vieillard et al. (2015) | published | 30 | -0.12 |
Wagenmakers et al. (2016) Albohn site | published | 139 | 0.09 |
Wagenmakers et al. (2016) Allard site | published | 125 | 0.09 |
Wagenmakers et al. (2016) Benning site | published | 115 | -0.01 |
Wagenmakers et al. (2016) Bulnes site | published | 101 | 0.09 |
Wagenmakers et al. (2016) Capaldi site | published | 117 | -0.07 |
Wagenmakers et al. (2016) Chasten site | published | 94 | -0.04 |
Wagenmakers et al. (2016) Holmes site | published | 99 | 0.15 |
Wagenmakers et al. (2016) Koch site | published | 100 | -0.14 |
Wagenmakers et al. (2016) Korb site | published | 101 | 0.01 |
Wagenmakers et al. (2016) Lynott site | published | 126 | 0.23 |
Wagenmakers et al. (2016) Oosterwijk site | published | 110 | -0.17 |
Wagenmakers et al. (2016) Ozdogru site | published | 87 | -0.3 |
Wagenmakers et al. (2016) Pacheco-Unguetti site | published | 120 | -0.08 |
Wagenmakers et al. (2016) Talarico site | published | 112 | 0.02 |
Wagenmakers et al. (2016) Wagenmakers site | published | 130 | 0.13 |
Wagenmakers et al. (2016) Wayand site | published | 110 | -0.14 |
Wagenmakers et al. (2016) Zeelenberg site | published | 108 | 0.25 |
Wittmer (1985) | unpublished | 30 | -0.36 |
Wittmer (1985) | unpublished | 30 | -0.21 |
Yartz (2004) | unpublished | 28 | -0.05 |
Yartz (2004) | unpublished | 30 | -0.18 |
Yartz (2004) | unpublished | 28 | -0.08 |
Yartz (2004) | unpublished | 30 | -0.09 |
Yartz (2004) | unpublished | 28 | 0.04 |
Yartz (2004) | unpublished | 30 | 0.5 |
Zajonc et al. (1989) Study 3 | published | 37 | 1.27 |
Zajonc et al. (1989) Study 4 | published | 26 | 0.47 |
Zajonc et al. (1989) Study 4 | published | 26 | 0.31 |
Zariffa et al. (2014) | published | 24 | -0.57 |
Zariffa et al. (2014) | published | 24 | -0.14 |
Zhu et al. (2015) | published | 55 | 1.74 |
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
Multi-Analyst Projects collated by Balazs Aczel
A review of 2500 social science papers, showing the lack of correlation between citations and replicability, between journal status and replicability, and the apparent lack of improvement since 2009.
Diederick Stapel: 58 retractions here.
Discussion on Everything Hertz, Hacker News, Andrew Gelman, some star data thugs comment.
See also the popular literature with uncritical treatments of the original studies: