Using Markets to Forecast Replications

In a number of past research projects, we have used prediction markets to elicit information on the replicability of research studies. In these markets, participants traded contracts with payoffs tied to the outcome of replications in large-scale replication projects, and thereby created forecasts – similar to bettors in sports betting markets who create forecasts by placing wagers on the outcome of sports events.

Our results show that information on the replicability of individual research claims is present in the research community and can be “crowd-sourced” through prediction markets.

But what is the point of forecasting replication outcomes, if outcomes will be observed anyway?

Besides providing a proof-of-principle, the advantage becomes clear with the SCORE project, where only 3% of the studies will be validated through direct replications. Since it is unknown which studies will be validated through replication, we can still set up markets to provide forecasts for all studies.

Investments in studies which are not selected for replication will not have pay-outs. However, in compensation, payoffs for studies which are selected will be increased. 

Our past prediction markets showed how reliable scientific communities can forecast replication success. We are looking forward to see if we can achieve similar results in a much larger setting.

Contribute to the discussion...

This site uses Akismet to reduce spam. Learn how your comment data is processed.

May we send you invitations to future research projects? 

Add your email address to our low-volume distribution list.

Note: We cannot re-use participant lists for new recruiting. Please opt-in (above) for occasional announcements about related future studies.

We’re sorry to see you go! Please visit our social media sites.

This site uses cookies to provide you with a better browsing experience.

Visit our Privacy Policy for more information.