Replication Markets recently estimated replication probabilities for 100 COVID-19 studies from the social and behavioral sciences. We don’t have all the results yet, but let’s see what our forecasters thought. (Note: We just launched another COVID-19 project, we hope you’ll sign-up to participate or learn more).
~75% of studies (everything to the right of the blue line) are estimated to have a better than 50% chance of replicating successfully.
In prior research only ~50% of studies replicate successfully, so it’s interesting to note that our forecasters are more optimistic about the studies in this sample.
Comparing Surveys and Markets
Participants in Replication Markets express their beliefs about the probability of successful replication two ways: with prediction surveys and markets. We’ll compare the forecasts we get from each of these methods
The market estimates are slightly lower (by 1.9%)
The market estimates are more varied (ratio of variances = 1.5)
Fig 1: Note that each point corresponds to the estimated replication probability for a different study.
We evaluate our replication forecasts with replication results. As of today we have only 8 replication results. Of the 100 we forecast, we expect 20-50 to eventually receive a replication efforts (as part of our research program, DARPA SCORE).
All 8 studies replicated successfully, a positive surprise!
We assigned > 0.6 probability of replication success to each successful replication.
Despite being 8 for 8, we are certain our forecasts are imperfect. The 8 studies replicated look “more obvious” than most. We will learn more about where we were right and wrong over time as more replication results arrive.