We have big changes to announce! Read to learn about… simpler definition of replication more markets paying out simpler surveys Simpler Definition of Replication SCORE has adopted a simpler, unified definition of replication: Replication is testing the same claims using data that was not used in the original study. That required some changes from us.…
The social and behavioral sciences, some say, have a “replication crisis.” When researchers attempt to reproduce results of published papers, sometimes the original conclusions do not hold up. Replication studies are conducted using the same methods and standards laid out in the published paper. They use new participants and often a larger sample size. Researchers…
It’s really easy not to find any mind-changing evidence… just don’t go looking for it. Cassie Kozyrkov When testing hypotheses, there are two types of errors one can make: finding a relationship where one does not exist (false positive), and finding no relationship where one does exist (false negative). Statistical power measures how hard you…
In a number of past research projects, we have used prediction markets to elicit information on the replicability of research studies. In these markets, participants traded contracts with payoffs tied to the outcome of replications in large-scale replication projects, and thereby created forecasts – similar to bettors in sports betting markets who create forecasts by…
“Standing on the shoulders of giants” – this phrase, attributed to Bernard of Chartres and prominently used by Isaac Newton, emphasises the role of past discoveries in ongoing scientific research. For modern research, where communication is predominately done through scientific publications, a high reliability of published findings is instrumental. Yet surprisingly little is known about…
Anna Dreber Almenberg, part of Replication Markets’ experiment design team, is the Johan Björkman Professor of Economics at the Stockholm School of Economics. She took time recently to answer a few questions about her interest in the field of research replication. What got you thinking about replications and markets? For me it all started when…
If you’ve just read What is a high-quality replication, you know that our replications will be high-power — usually much higher than the original study — but not quite the “100 replications” ideal set out by DARPA: Assume 100 replications of this study were performed, and a weighted average of their results (i.e., a meta-analysis) was…
What is high quality replication?
In brief: A good-faith, high-power attempt to reproduce a previously-observed finding.
[Updated Jan. 2020 to cover unified definition, and clarify] Concept Replication is when you repeat a previous study to see if you get the same results. Ideally this happens a lot in science. In practice, not as much. But what counts? According to Brian Nosek (“What is replication?”, 2019), people commonly say replication is repeating…
Replication Markets is a game that is part crowd-sourcing, part playing a market, and part legal gambling. The bets are placed on research results: can a study be replicated?
