Image: Vmenkov, CC BY-SA 3.0, via Wikimedia CommonsWhat happened when psychologists repeated 100 experiments
Between 2011 and 2015, a group that grew to 270 researchers picked 100 studies published in 2008 in three leading journals, Psychological Science, the Journal of Personality and Social Psychology, and the Journal of Experimental Psychology: Learning, Memory, and Cognition, and ran each of them again. Each replication followed the original method, used the original materials where the authors would share them, and was reviewed by the original authors before it was run. The samples were on average larger than the originals. The results appeared in Science in August 2015 under the name of the Open Science Collaboration, and the paper is at the journal, with every original, every replication, and every dataset on the Open Science Framework.
The figures
97 of the 100 original studies had reported a statistically significant result. Of the 97 replications, 35 did, which is 36%. The average effect size of the replications was about half that of the originals: a mean correlation of 0.20 against 0.40. In 47% of the studies, the original effect size fell inside the 95% confidence interval of the replication. When the replicating teams were asked for a judgement, they said that 39% of the studies had replicated.
The rate differed by field. Studies in cognitive psychology replicated about half the time; studies in social psychology about a quarter of the time. Studies with a stronger original result, a smaller original p-value, and a larger original effect replicated more often, which is what you would expect if the problem lay in weak evidence rather than in anything about the replicators. The expertise of the replicating team and the judged difficulty of the study made little difference.
Those are the figures that Unit 1 and Unit 8 of the masterclass use, and they are easy to misread in both directions. The project did not show that 64% of published findings are false. A replication can fail because the original was a false positive, because the effect is real but smaller than reported, because something in the method or the population differed, or by chance. It also did not show that the three journals had published unusually bad work; they were chosen because they were good. The finding was that a single significant result in a good journal is weaker evidence than most readers, including most psychologists, had assumed.
Where the false positives came from
The causes had been described four years earlier. In 2011, Joseph Simmons, Leif Nelson, and Uri Simonsohn published a paper called "False-Positive Psychology", free at the journal, in which they ran a real experiment and found, with p below 0.05, that listening to the Beatles song "When I'm Sixty-Four" made undergraduates younger. They had not falsified anything. They had used the ordinary freedoms of analysis: measuring several outcomes and reporting the one that worked, adding participants until the result crossed the line, trying the analysis with and without a covariate. With four such choices, the chance of finding a significant result where none exists rises from 5% to about 60%. The practices are now called questionable research practices, or p-hacking, and surveys found that most researchers admitted to at least one.
Two other things made the problem worse. Journals preferred positive results, so studies that found nothing went in the file drawer and the published record was a biased sample. And samples were small. A study with 20 participants per group has, for an effect of medium size, less than a 50% chance of detecting it even when it is real, and the significant results it does produce overstate the effect. The same year as the false-positive paper, a leading journal published a study claiming evidence for precognition, and a Dutch social psychologist was found to have invented data in dozens of papers. The three events together, and the failure of several famous priming studies to replicate, are what people mean by the replication crisis.
What changed
The reforms are still going on, and a student reading a paper published after about 2018 is reading a different kind of paper from one published in 2008.
Preregistration. A researcher writes down the hypotheses, the sample size, the measures, and the analysis before collecting the data, and deposits the plan with a time stamp on a public registry. The analysis can still be changed afterwards, but the change has to be declared. A registered report goes further: the journal reviews and accepts the plan before the data exist, so the paper is published whatever the result. By 2024 more than 300 journals offered the format, and an analysis of the first 71 registered reports in psychology found that about 44% of their hypotheses were confirmed, against more than 90% in the standard literature.
Open data and materials. Many journals now require the data and the analysis code to be deposited, so that a reader can rerun the analysis. The badges that Psychological Science introduced for open data in 2014 raised the share of papers with available data from about 3% to about 40% within two years.
Larger samples and many laboratories. The Many Labs projects ran the same study in dozens of laboratories around the world and found that most classic effects replicated everywhere with much the same size, while several newer ones replicated nowhere. The Psychological Science Accelerator, a network of hundreds of laboratories, now runs studies with samples in the tens of thousands.
A harder look at significance. In 2016 the American Statistical Association issued a statement on what a p-value does and does not mean, and in 2018 a group of 72 researchers proposed moving the conventional threshold from 0.05 to 0.005 for claims of new discoveries. Neither changed the rules of any journal, but both changed how reviewers read a result that scrapes under 0.05 with a small sample.
Brian Nosek, who led the 2015 project, reviewed the decade with colleagues in 2022 in the Annual Review of Psychology; the preprint is free. Their reading is that the replication rate of new work has probably risen, that the practices which produced the 2015 figures are no longer standard, and that the field is in a better state than most of the sciences that have not yet tried to measure their own replicability. Medicine, economics, and cancer biology ran similar projects and found similar or worse figures.
How to read a study now
For a student, the practical lessons are short. Look for the sample size before the p-value. Look for the effect size and its confidence interval, and ask whether the lower end would still matter. Check whether the study was preregistered and whether the data are available. Prefer a meta-analysis or a multi-site replication to any single study, and treat a surprising finding with a small sample in a famous journal as a hypothesis until someone has repeated it. A finding that has been replicated by independent groups with large samples, such as the testing effect or the Stroop effect, deserves more confidence than it got before 2011, because it has now been tested in a way that the field did not use to apply.
The masterclass states the replication status of every contested finding it teaches, and the first unit sets out the tools, from the standard deviation to the confidence interval, that make the 2015 figures readable. The key findings sheet collects them.


