Dear Forecasters: Wow! What a year it’s been! Did you know you completed 54,000 surveys and 42,000 trades on 3,000 studies? Plus the current COVID-19 Round 11? That’s stunning. Replication Markets Recap: The View From 3,000 Claims The forecasts are in. The results are not. But there’s quite a bit we can say – if you don’t mind preliminary results.…
The use of p-values and hypothesis testing is ubiquitous across science. Everyone remembers the rules they learnt in their first statistics courses; 0.05 is the magic cut-off for statistical significance. Statistical evidence with p-values below 0.05 are accepted – models with p-values higher than this threshold are discarded. However, this statistical practice has come…
In a number of past research projects, we have used prediction markets to elicit information on the replicability of research studies. In these markets, participants traded contracts with payoffs tied to the outcome of replications in large-scale replication projects, and thereby created forecasts – similar to bettors in sports betting markets who create forecasts by…
Anna Dreber Almenberg, part of Replication Markets’ experiment design team, is the Johan Björkman Professor of Economics at the Stockholm School of Economics. She took time recently to answer a few questions about her interest in the field of research replication. What got you thinking about replications and markets? For me it all started when…
If you’ve just read What is a high-quality replication, you know that our replications will be high-power — usually much higher than the original study — but not quite the “100 replications” ideal set out by DARPA: Assume 100 replications of this study were performed, and a weighted average of their results (i.e., a meta-analysis) was…
What is high quality replication?
In brief: A good-faith, high-power attempt to reproduce a previously-observed finding.
[Updated Jan. 2020 to cover unified definition, and clarify] Concept Replication is when you repeat a previous study to see if you get the same results. Ideally this happens a lot in science. In practice, not as much. But what counts? According to Brian Nosek (“What is replication?”, 2019), people commonly say replication is repeating…
Replication Markets is a government funded research project to forecast the outcomes of thousands of potential social science replications to be done from mid-2019 to mid-2020.
Replication Markets is a game that is part crowd-sourcing, part playing a market, and part legal gambling. The bets are placed on research results: can a study be replicated?
