In a number of past research projects, we have used prediction markets to elicit information on the replicability of research studies. In these markets, participants traded contracts with payoffs tied to the outcome of replications in large-scale replication projects, and thereby created forecasts – similar to bettors in sports betting markets who create forecasts by placing wagers on the outcome of sports events.
Our results show that information on the replicability of individual research claims is present in the research community and can be “crowd-sourced” through prediction markets.
But what is the point of forecasting replication outcomes, if outcomes will be observed anyway?
Besides providing a proof-of-principle, the advantage becomes clear with the SCORE project, where only 3% of the studies will be validated through direct replications. Since it is unknown which studies will be validated through replication, we can still set up markets to provide forecasts for all studies.
Investments in studies which are not selected for replication will not have pay-outs. However, in compensation, payoffs for studies which are selected will be increased.
Our past prediction markets showed how reliable scientific communities can forecast replication success. We are looking forward to see if we can achieve similar results in a much larger setting.
