Update on Resolutions

The Replication Markets team expects to begin prize announcements by 8-NOV. 

We are double-checking points distribution and prize values, and will do a last check that there are no new replication results we could use. So far we have 126 forecasters who will receive prizes for Rounds 1-10.

Winners will receive emails, the winners list will be posted on this blog, and you’ll see information in our Twitter feed and Reddit channel.

Until then, enjoy this general preview! (Final numbers subject to change.)

Reminder about the Replication Markets SCORE structure:

Round 0: Meta-questions – still working on those resolutions, as they depend on others

Rounds 1-5: About 1500 claims, only direct replications count towards market resolutions.

Rounds 6-10: About 1500 claims, both direct replications and data analytic replications count.

Round 11: 100 COVID-related preprints, direct and data analytic replications count

We ignore computational reproductions. Rounds 1-10 have had 87 replications so far:

  • 33 data analytic replication results (15 successful, or 45%)
  • 54 direct replication results (23 successful, or 43%)

We can use 66 of those for prizes (dropping data analytic replications in Rounds 1-5): 












# Replications 












  • 29 of these 66 claims successfully replicated 
  • 37 of these 66 claims did NOT replicate 

 (There was one more eligible replication, but RM forecasters never saw the claim because it was one of 19 judged inappropriate for betting on.) 

Winning positions: 

These are the numbers most subject to change. At the moment, it seems 126 forecasters have net winning positions in R1-10. 

  • 108 users have 1, 2, or 3 winning positions 
  • Maximum number of winning positions is 15 

Prizes are computed based on the proportion of points in a single Round.  

  • Maximum proportion of winning points is 100% ! 
  • Median proportion of winning points is 13% 
  • Mean proportion of winning points is 23% 

6 thoughts on “Update on Resolutions”

  • Awesome to finally get some news.

    Could you clarify how the payouts will be calculated? The wording here (“net winning positions”) seems to imply that losing predictions are netted out from winning ones. The rules OTOH seem to imply that the gross winning shares will be used (“allocated proportionally to the winning shares in the round […] Winning shares are ‘Yes’ shares of a successful direct replication, and ‘No’ shares of an unsuccessful direct replication.”)

  • Awesome to finally have some news!

    Could you elaborate on how the prizes will be calculated? This post seems to imply losing shares will be netted out from winning ones. On the other hand, the rules seem to imply gross winning shares will be used (“prize pool will be allocated proportionally to the winning shares in the round […] Winning shares are ‘Yes’ shares of a successful direct replication, and ‘No’ shares of an unsuccessful direct replication.”)

  • Charles Twardy says:

    You have only one position in a claim, regardless how many trades you made to get there. If that position favored the correct outcome, those are winning shares and they go into the prize pool, regardless how badly you did on other claims.

    We first group by user-claim to get your net position in each claim. Then we drop rows with ≤0 gain, so claims you lost don’t count against you. Next we group by Round and sum by user, yielding winning shares per user for the Round. Total cash prize for that Round is divided proportional to their shares.

    But of course we don’t ignore losing shares _per trade_, or everyone could get arbitrarily large winning shares at no risk by just raising 50% ➛ 60% and then immediately reversing. (RM doesn’t let you buy “Yes” separate from selling “No”.)

    That help?

    • Thanks, that clears it up. So does “Maximum proportion of winning points is 100%” mean that only a single user had any net winning shares, I’m guessing on the one question that resolved from round 8?

      • Charles Twardy says:

        Yes. Hoping a last tree-shake will turn up another replication and spread things out there.

        • Charles Twardy says:

          Happily the last tree-shake turned up enough replications that the minimum replication count is now Round 9 with 2 replications: replication_counts = [18, 14, 7, 7, 5, 9, 9, 4, 2, 3].

          Plus the 30 covid replications in SCORE Round 11. (Not the the Fetzer-funded covid preprint market, which was mostly about publication & citation.)

Contribute to the discussion...

This site uses Akismet to reduce spam. Learn how your comment data is processed.

May we send you invitations to future research projects? 

Add your email address to our low-volume distribution list.

Note: We cannot re-use participant lists for new recruiting. Please opt-in (above) for occasional announcements about related future studies.

We’re sorry to see you go! Please visit our social media sites.

This site uses cookies to provide you with a better browsing experience.

Visit our Privacy Policy for more information.