The Replication Markets team expects to begin prize announcements by 8-NOV.
We are double-checking points distribution and prize values, and will do a last check that there are no new replication results we could use. So far we have 126 forecasters who will receive prizes for Rounds 1-10.
Winners will receive emails, the winners list will be posted on this blog, and you’ll see information in our Twitter feed and Reddit channel.
Until then, enjoy this general preview! (Final numbers subject to change.)
Reminder about the Replication Markets SCORE structure:
Round 0: Meta-questions – still working on those resolutions, as they depend on others
Rounds 1-5: About 1500 claims, only direct replications count towards market resolutions.
Rounds 6-10: About 1500 claims, both direct replications and data analytic replications count.
Round 11: 100 COVID-related preprints, direct and data analytic replications count
We ignore computational reproductions. Rounds 1-10 have had 87 replications so far:
- 33 data analytic replication results (15 successful, or 45%)
- 54 direct replication results (23 successful, or 43%)
We can use 66 of those for prizes (dropping data analytic replications in Rounds 1-5):
Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
# Replications | 14 | 11 | 7 | 7 | 4 | 8 | 9 | 1 | 2 | 3 |
Summarizing:
- 29 of these 66 claims successfully replicated
- 37 of these 66 claims did NOT replicate
(There was one more eligible replication, but RM forecasters never saw the claim because it was one of 19 judged inappropriate for betting on.)
Winning positions:
These are the numbers most subject to change. At the moment, it seems 126 forecasters have net winning positions in R1-10.
- 108 users have 1, 2, or 3 winning positions
- Maximum number of winning positions is 15
Prizes are computed based on the proportion of points in a single Round.
- Maximum proportion of winning points is 100% !
- Median proportion of winning points is 13%
- Mean proportion of winning points is 23%
Awesome to finally get some news.
Could you clarify how the payouts will be calculated? The wording here (“net winning positions”) seems to imply that losing predictions are netted out from winning ones. The rules OTOH seem to imply that the gross winning shares will be used (“allocated proportionally to the winning shares in the round […] Winning shares are ‘Yes’ shares of a successful direct replication, and ‘No’ shares of an unsuccessful direct replication.”)
Awesome to finally have some news!
Could you elaborate on how the prizes will be calculated? This post seems to imply losing shares will be netted out from winning ones. On the other hand, the rules seem to imply gross winning shares will be used (“prize pool will be allocated proportionally to the winning shares in the round […] Winning shares are ‘Yes’ shares of a successful direct replication, and ‘No’ shares of an unsuccessful direct replication.”)
You have only one position in a claim, regardless how many trades you made to get there. If that position favored the correct outcome, those are winning shares and they go into the prize pool, regardless how badly you did on other claims.
We first group by user-claim to get your net position in each claim. Then we drop rows with ≤0 gain, so claims you lost don’t count against you. Next we group by Round and sum by user, yielding winning shares per user for the Round. Total cash prize for that Round is divided proportional to their shares.
But of course we don’t ignore losing shares _per trade_, or everyone could get arbitrarily large winning shares at no risk by just raising 50% ➛ 60% and then immediately reversing. (RM doesn’t let you buy “Yes” separate from selling “No”.)
That help?
Thanks, that clears it up. So does “Maximum proportion of winning points is 100%” mean that only a single user had any net winning shares, I’m guessing on the one question that resolved from round 8?
Yes. Hoping a last tree-shake will turn up another replication and spread things out there.
Happily the last tree-shake turned up enough replications that the minimum replication count is now Round 9 with 2 replications: replication_counts = [18, 14, 7, 7, 5, 9, 9, 4, 2, 3].
Plus the 30 covid replications in SCORE Round 11. (Not the the Fetzer-funded covid preprint market, which was mostly about publication & citation.)