Common Questions

Our forecasters bet on the chance that a research claim will replicate. They have the opportunity to read the original paper and discuss their analyses, and can adjust their predictions until the market closes. The most accurate forecasters will earn money. 

Prior to the markets, there are also private surveys on the same claims, with a separate prize pool.

Prediction markets are an alternative to surveys for predicting outcomes of events in politics, sports, research replication studies, and other domains

In a prediction market, participants invest points to say how likely they think different outcomes are. Forecasters who more accurately predict the true outcome receive a gain and those who are less accurate incur a loss. So, participants have an incentive to speak up when they know, and stay quiet when they don’t. 

Unlike surveys, there is no extra averaging step, just the current estimate (market value). Markets tend to outperform simple averaging; they roughly match sophisticated averaging.

To learn more about prediction markets for science, see our publications listed here.

For more details, read the Explanation of Payouts.

In this project:

A claim is an assertion in social science, such as “Imagining eating M&Ms 30 times makes people eat fewer M&Ms.” Typically, a claim is disputed or in doubt, and we want to know if it will replicate when tested. Participants forecast replications using both surveys and markets.

Each claim has a market where forecasters invest points to move the market estimate (usually a probability) up or down. That is equivalent to buying and selling shares that “pay 1 point if the replication succeeds” or “pay 1 point if it fails”. The price of the shares traded in a market is usually supposed to reflect the crowd’s belief about the probability that a claim will be replicated. We show it as a probability.

We also use “the market” (or “the survey”), to mean the collection of all market (or survey) claims, for example “this claim is now on the market”. We will also have markets (and surveys) on other kinds of claims like effect sizes, or overall replication rates.

Prediction markets have historically performed better than surveys for crowdsourced information. 

At RM, we take a survey of forecasts for each claim, and use that as a ground truth to assess the accuracy of the corresponding market.

In Round 0, you will start with 100 points to play. In all subsequent rounds, you will start the same number of points as there are claims.

To learn how to earn more points to play with, see Points Distribution.

SUMMARY FOR HUMANS:

• No purchase necessary to win.
• You must be 18yo and not affiliated with the project.
• You may participate even if declining or ineligible for prizes.
• One user, one account, except by written agreement.
• Follow the rules. Play nice. Don’t cheat. Don’t try to lose.
• Contest begins 12-AUG-2019 and runs approximately through 28-SEP-2020.
• Contest will forecast 3,000 claims from social & behavioral science.
• Forecasts will be in Rounds of about 300 claims, roughly every 4 weeks. 
• Each Round has:
  • about a week for private surveys in Batches of 10 claims
  • about two weeks for public markets on all 300 together
• Only about 5% of claims will resolve. Out of 3000 total claims:
  • We expect about 50 claims to resolve from Rounds 1-5 (only direct replications count).
  • We expect about 125 claims to resolve from Rounds 6-10 (any replication counts).
• Neither RM nor forecasters will know which claims resolve.
• A total of about US $195,000 in prizes will be awarded. Prizes are divided among the ~10 Rounds as follows.
  • Surveys: $160 per survey Batch, paid per Round to the Top 4. Each batch awards $80, $40, $20, and $20, to the first, second, third, and fourth place for that batch. Rounds will generally have 30 batches. Total ~US$48,000.
  • Markets in Rounds 1-5 keep their original plan:
    • Markets pay about $9,000 per Round, paid on resolution, proportional to winning shares/round. On average, that is $900 per resolving market, paid on resolution, Total ~US$45,000.
  • Markets in Rounds 6+ pay more per round and less per claim:
    • About $18,750 per Round, assuming 25 resolving claims.
    • Actual Round prize pool is $750 x # resolving claims.
    • (We used project reserves to boost per-claim $ following cut required by definition change.)
  • Amounts per round assume ~300 claims per round and will be adjusted proportionally if this number changes.
• RM will use PayPal to distribute funds to all winners.
• Recipients may be required to verify identity.
• Taxes and fees are paid by recipients.
• Usual liability limitations, and final authority with Sponsor.

From Rules v.1.7

The full text of the consent form is available here. (It was updated in R11 to allow for the possibility of longer Rounds having slightly different structure.) A complete version history will be available from OSF.io after project completion.