
Menu
© 2020 All rights Reserved
Replication Markets is one part of the larger DARPA SCORE program to assign confidence scores to social & behavioral science claims. (See also this Wired Article.)
SCORE sponsors the claim selection, the replications, two crowd forecasting teams, and (TBD) machine learning.
The Center for Open Science selected studies from 62 journals. For a complete list, click here.
Science is about testing one’s ideas. Therefore, a valid scientific finding must be reproducible: if you do the same thing, you are supposed to get the same result, at least statistically, and ideally (but rarely) always. As we use the terms:
A reproduction is an attempt to do get the same result with the original data — this can fail due to errors, lack of data, or missing steps.
As of Round 6, replication is testing the claim using data that was not used in the original study. The data could be pre-existing, or collected for the replication study. (See “What is a High Quality Replication” below for details.)
Finally, a successful replication, for us and all of SCORE, is “simply” getting a statistically significant result in the same direction as the original claim. (See “What is a High Quality Replication” below.)
Take a look at our Recommended Reading to learn more about replication.
Short answer: A good-faith, high-power attempt to reproduce a previously-observed finding.
Slightly longer answer: A good-faith attempt to reproduce a previously-observed finding with a sample large enough to find the effect if there.
For history of our evolving definition, and further details, read our long post about High Quality Replication!
The advent of the internet has made it possible for large numbers of people to contribute to a collective project in a timely manner. Wikipedia is one of the most famous examples. Website product reviews, community news sites, and prediction markets are other types of crowdsourcing. The power of crowdsourcing builds on the wisdom of crowds, which has been shown to be more effective in producing the true answer in comparison to an elite group. Particularly, when knowledge is widely distributed and hard to locate, crowdsourcing has been shown to be surprisingly effective.
Yes! Our project is being conducted in the spirit of open science and transparency, and therefore de-identified data will be published when the project concludes. We are also pre-registering our work with the Center for Open Science.
As for what we do with personally identifying data, we do cover this information in the informed consent details available on the registration page: https://predict.replicationmarkets.com/main/#!/users/register
Our forecasters bet on the chance that a research claim will replicate. They have the opportunity to read the original paper and discuss their analyses, and can adjust their predictions until the market closes. The most accurate forecasters will earn money.
Prior to the markets, there are also private surveys on the same claims, with a separate prize pool.
Prediction markets are an alternative to surveys for predicting outcomes of events in politics, sports, research replication studies, and other domains
In a prediction market, participants invest points to say how likely they think different outcomes are. Forecasters who more accurately predict the true outcome receive a gain and those who are less accurate incur a loss. So, participants have an incentive to speak up when they know, and stay quiet when they don’t.
Unlike surveys, there is no extra averaging step, just the current estimate (market value). Markets tend to outperform simple averaging; they roughly match sophisticated averaging.
To learn more about prediction markets for science, see our publications listed here.
Rounds 1-5
We expect about 50 claims to resolve from Rounds 1-5 (only direct replications count).
Rounds 6-10
We expect about 125 claims to resolve from Rounds 6-10 (any replication counts). These Markets pay more per round and less per claim:
For more details, read the Explanation of Payouts.
In this project:
A claim is an assertion in social science, such as “Imagining eating M&Ms 30 times makes people eat fewer M&Ms.” Typically, a claim is disputed or in doubt, and we want to know if it will replicate when tested. Participants forecast replications using both surveys and markets.
Each claim has a market where forecasters invest points to move the market estimate (usually a probability) up or down. That is equivalent to buying and selling shares that “pay 1 point if the replication succeeds” or “pay 1 point if it fails”. The price of the shares traded in a market is usually supposed to reflect the crowd’s belief about the probability that a claim will be replicated. We show it as a probability.
We also use “the market” (or “the survey”), to mean the collection of all market (or survey) claims, for example “this claim is now on the market”. We will also have markets (and surveys) on other kinds of claims like effect sizes, or overall replication rates.
Prediction markets have historically performed better than surveys for crowdsourced information.
At RM, we take a survey of forecasts for each claim, and use that as a ground truth to assess the accuracy of the corresponding market.
In Round 0, you will start with 100 points to play. In all subsequent rounds, you will start the same number of points as there are claims.
To learn how to earn more points to play with, see Points Distribution.
SUMMARY FOR HUMANS:
The full text of the consent form is available here. (It was updated in R11 to allow for the possibility of longer Rounds having slightly different structure.) A complete version history will be available from OSF.io after project completion.
Note: We cannot re-use participant lists for new recruiting. Please opt-in (above) for occasional announcements about related future studies.
© 2020 All rights Reserved
This site uses cookies to provide you with a better browsing experience.
Visit our Privacy Policy for more information.