Replication Markets' New Experiment: Evaluating COVID-19 Preprints

Call to Action Evaluation!

Replication Markets (RM) invites citizen scientists to make forecasts about COVID-19 preprints that have been discussed on social media. We need a variety of perspectives and voices like yours. As a participant, you will predict which preprints are:

  • Most likely to be published
  • Most likely to be cited
  • Most likely to be replicated
  • Most likely to have a public health impact

We need your viewpoint, as well as insights from people with different backgrounds. Nobody has expertise in every relevant field, and experts are sometimes outperformed by generalists.


Thousands of scientists have responded to the pandemic by studying the virus, the disease, and the impact on society, writing preprints far faster than they can be reviewed. Today there are nearly 10,000 COVID-19 preprints registered on two of the largest preprint servers, and We are experimenting with additional mechanisms to discover and evaluate scientific work.

Our findings will contribute to the wider body of scientific knowledge about forecasting, and possibly to the more immediate cause of rating and reviewing preprints. RM will make all its data available for analysis.

Track Record

Our research team, and others, have already demonstrated the scientific potential of crowdsourcing forecasts through surveys and prediction markets. You can read a bit about our recently completed forecasting for 3,000 research claims, funded by DARPA SCORE, or review prior published Replication Forecasting research:

How it Works

You will be asked to use both surveys and a play-money market, with real payouts totaling over $14,000. Payouts are distributed among the most accurate of its anticipated 100+ forecasters. There is no cost to play the Replication Markets. Average winnings will depend on the number of participants, and their accuracy.

In the Surveys, you will be asked to answer a handful of questions about 10 papers. You may go back for as many extra batches of 10 papers as you like.

In the Markets, you will receive points to bet as you wish across the 400 papers. Successful forecasters generally adjust their assessments in response to further news, discussions, and other bets.

Our project needs forecasters like you with knowledge and judgment. We hope you will enjoy participating.

Mike Bishop


There is NO COST to participate in Replication Markets.

This research is supported (in part) by the Fetzer Franklin Fund of the John E. Fetzer Memorial Trust. It uses a platform developed for DARPA SCORE, and some staff are supported by SCORE while working on this. We are grateful for their support.

We’re sorry to see you go! Please visit our social media sites.

This site uses cookies to provide you with a better browsing experience.

Visit our Privacy Policy for more information.