Will COVID-19 Studies Replicate?

Replication Markets recently estimated replication probabilities for 100 COVID-19 studies from the social and behavioral sciences. We don’t have all the results yet, but let’s see what our forecasters thought. (Note: We just launched another COVID-19 project, we hope you’ll sign-up to participate or learn more).

Reviewing our Previous COVID-19 Forecasts

Big picture

• ~75% of studies (everything to the right of the blue line) are estimated to have a better than 50% chance of replicating successfully.
• In prior research only ~50% of studies replicate successfully, so it’s interesting to note that our forecasters are more optimistic about the studies in this sample.

Comparing Surveys and Markets

Participants in Replication Markets express their beliefs about the probability of successful replication two ways: with prediction surveys and markets. We’ll compare the forecasts we get from each of these methods

• The market estimates are slightly lower (by 1.9%)
• The market estimates are more varied (ratio of variances = 1.5)

Fig 1: Note that each point corresponds to the estimated replication probability for a different study.

Do surveys and markets agree about individual studies?

• The plot below indicates a fair amount of agreement on replication probabilities between the surveys and the market.
• The points above the line are studies for which the survey was more optimistic than the market.
• We hope combining surveys and markets can improve the accuracy of our forecasts

Are these forecasts accurate?

We evaluate our replication forecasts with replication results. As of today we have only 8 replication results. Of the 100 we forecast, we expect 20-50 to eventually receive a replication efforts (as part of our research program, DARPA SCORE).

• All 8 studies replicated successfully, a positive surprise!
• We assigned > 0.6 probability of replication success to each successful replication.
• Despite being 8 for 8, we are certain our forecasts are imperfect. The 8 studies replicated look “more obvious” than most. We will learn more about where we were right and wrong over time as more replication results arrive.

If you find this interesting, please sign-up to participate or learn more.