When Replication Markets launched in 2019, we could not have expected the global interruptions that affected our project timeline. Finally, we are ready to resolve and pay out our markets! One silver lining: in waiting to announce results, we have more resolutions to pay out.
More resolutions means a larger sample size to answer the meta-questions of Round 0.
Our patience has produced more resolutions from claims in Rounds 1-10. RM’s users traded on 77 of the 79 claims that qualified for payout.
We made the best we could of the pandemic by adding Round 11 for preprints of COVID-related papers. This has paid off — both for our research and for RM’s traders: 30 of the 100 claims from Round 11 have resolved.
While RM knows the results of the replications, we are not allowed to share the individual findings. We can share the information in aggregate, which we present in the tables below. Click on a link to jump to the section.
Resolutions per Round
Note that Claims is the number of eligible resolved claims that users traded on.
| Round | Claims |
| 1 | 18 |
| 2 | 14 |
| 3 | 7 |
| 4 | 7 |
| 5 | 5 |
| 6 | 9 |
| 7 | 9 |
| 8 | 4 |
| 9 | 2 |
| 10 | 3 |
| 11 | 30 |
Leaderboard for users and their prizes
- Winning users should look for an email from domain @replicationmarkets.com by Wednesday, November 17.
- Make sure to add our domain to your allow-list so the email lands in your in-box!
| username | prize total |
| BradleyJBaker | $5,721 |
| meaning.mosaic.curtain | $4,229 |
| unipedal | $4,020 |
| mVranka | $3,903 |
| physwiz | $3,615 |
| mbulutay | $3,172 |
| sattuma | $2,827 |
| ejorgenson | $2,822 |
| CPM | $2,692 |
| Nokta | $2,610 |
| namk | $2,572 |
| jc20 | $2,557 |
| JonS | $2,303 |
| Scalene1234 | $2,290 |
| RobC | $2,226 |
| stanlaurel | $2,119 |
| galbuquerque | $2,055 |
| DavidRhysBernard | $1,974 |
| levent | $1,841 |
| Taku | $1,789 |
| Vitorper | $1,533 |
| Luthor113 | $1,483 |
| Matho | $1,425 |
| martin_kosiik | $1,414 |
| AchinToBe | $1,413 |
| gordon gekko | $1,364 |
| andi | $1,353 |
| GeritPfuhl | $1,333 |
| guilfoos | $1,188 |
| ted | $1,175 |
| CesarMant | $1,172 |
| curious | $1,149 |
| yetAnotherTrader | $1,148 |
| Mptg2 | $1,143 |
| acct | $1,137 |
| DMS | $1,105 |
| ncerutti | $1,090 |
| tanzor | $1,063 |
| JozefBavolar | $1,060 |
| maalsc | $1,045 |
| wetpaint | $1,027 |
| Sceptic | $1,017 |
| Sjaakharrie | $983 |
| denns | $956 |
| kushkan | $944 |
| sflicht | $934 |
| Poison | $925 |
| Franzi | $919 |
| pbog | $903 |
| antipodeanlurker | $894 |
| shkaff | $889 |
| felixbunny | $878 |
| Miss_prediction | $875 |
| friendlysnail | $875 |
| zultan | $839 |
| NunoNunez | $809 |
| adambear91 | $781 |
| relihan | $772 |
| Bitschnau | $750 |
| Pavol | $742 |
| Hygglot | $738 |
| BBalmford | $735 |
| cgravert | $731 |
| benthamite | $714 |
| rm_user | $713 |
| cmeinel | $704 |
| bjaeger | $693 |
| Frantisek | $664 |
| smingers | $653 |
| NLA01 | $653 |
| Cory | $638 |
| educatedguess | $630 |
| xoph09 | $626 |
| jonathanstray | $608 |
| Econ101 | $599 |
| EirikStroemland | $561 |
| Miranda | $560 |
| Troof_ | $559 |
| ekarpinski | $548 |
| jacobjacob | $540 |
| dschwarz | $533 |
| Ilei | $533 |
| Pepe | $532 |
| smeinz | $504 |
| e4e6 | $497 |
| Hmassz | $489 |
| buddhabrot | $481 |
| bmandl | $474 |
| lincoln | $468 |
| drl1764 | $467 |
| chiefi_lux | $455 |
| thekalby | $454 |
| catrachohansen | $447 |
| prorgan | $443 |
| smetis | $443 |
| Kirstin | $439 |
| Marol198 | $425 |
| Merkus90 | $421 |
| gingervitiz | $406 |
| dhitch | $399 |
| Sclamons | $398 |
| ClaudiusMaier | $391 |
| riarm | $370 |
| braoi | $363 |
| kongk | $363 |
| Illmatic | $352 |
| lukasneville | $352 |
| Reviewer3 | $345 |
| cwilliams | $343 |
| renebekkers | $343 |
| noob_trader | $336 |
| mspape | $325 |
| vaquinha | $315 |
| ulrich | $312 |
| Lenkafiala | $307 |
| Botilicious | $299 |
| radiator | $292 |
| wangtong1235813 | $289 |
| bgfarrar | $268 |
| jpheffne | $264 |
| paul_henne | $258 |
| unpredictified | $256 |
| WhatCouldHappen | $250 |
| MariaC | $244 |
| mg | $240 |
| Humboldt | $238 |
| pwaffle | $238 |
| cavanbonner | $237 |
| yeli23 | $236 |
| abaillon | $229 |
| wellfleetOyster | $220 |
| GKarreskog | $216 |
| sutych | $213 |
| bjoerksb | $212 |
| felize2000 | $212 |
| nurit.nobel | $210 |
| andrewsmith | $193 |
| philip_p | $192 |
| KABrekke | $191 |
| adamlgreen | $188 |
| nicolenightingale | $186 |
| NPXY | $185 |
| Sampotarmo | $182 |
| vtevelde | $179 |
| PTan | $177 |
| rboehm | $174 |
| Dima | $168 |
| jaynbaxter | $167 |
| benkenobi | $167 |
| robert.thibault | $166 |
| scchen | $158 |
| Poleres | $149 |
| John Prowse | $148 |
| DCH | $147 |
| jornstein | $145 |
| SBNDad | $144 |
| mds | $143 |
| hugopedder | $142 |
| FloraCozzi | $141 |
| nesealkan | $141 |
| schniter | $138 |
| cbrandolino | $137 |
| Brett | $135 |
| mf483 | $134 |
| SMGifford | $130 |
| alexrjl | $127 |
| Matti Heino | $127 |
| mpaaso | $127 |
| insatiableurge | $126 |
| SeaSantiago | $124 |
| nathan98000 | $119 |
| dankee_yoodle | $119 |
| mvernholmen | $115 |
| Pheck100 | $113 |
| 8435207df859e854 | $112 |
| Garonmae | $112 |
| mattathias | $104 |
| ocean01234 | $102 |
| jjxw | $101 |
| SarahSchoch | $100 |
| caroroux | $98 |
| shig_ono | $96 |
| pboenisch | $96 |
| Davidoj | $95 |
| Machery | $95 |
| Nimchuk | $94 |
| simoncolumbus | $94 |
| spahen | $89 |
| An1lam | $89 |
| Bvschuer | $89 |
| HWillem | $88 |
| warsquid | $86 |
| Laura | $86 |
| gracecsmith | $83 |
| Jo.Weiner | $83 |
| slantedlight80 | $80 |
| JCmo | $75 |
| HABK | $75 |
| ytan830 | $74 |
| andrasm | $73 |
| enr1co | $73 |
| wscampbell | $70 |
| rpeeters | $68 |
| Tamay | $67 |
| bravodelta | $67 |
| CHartling | $63 |
| sharedvariance | $60 |
| bphillab | $59 |
| dstauffer11 | $58 |
| 2e10e122 | $56 |
| Aniaw81 | $54 |
| jbhender | $53 |
| Spy | $52 |
| opowell | $47 |
| sarah.jacobson | $46 |
| Azaza | $46 |
| ecotico | $46 |
| c_cap | $46 |
| chewy | $45 |
| grpennycook | $42 |
| lohsej | $42 |
| boris | $41 |
| cmadan | $40 |
| pthiemann | $38 |
| matthias | $38 |
| mvs | $37 |
| Danva85 | $37 |
| Danielf089 | $35 |
| gstupp | $34 |
| adamwhlr | $33 |
| funashi | $32 |
| j19 | $32 |
| joshfoster | $32 |
| nwblack | $31 |
| m.stroom | $31 |
| ajaxecles | $31 |
| ekimbrough | $26 |
| brungl | $26 |
| TomL | $25 |
| oneAtATime | $25 |
| TDZ | $24 |
| drichter | $23 |
| tankred | $23 |
| malsh | $23 |
| JohnH | $23 |
| Zubrin | $23 |
| AMW | $23 |
| alf847 | $22 |
| OliverGenschow | $21 |
| baxpr | $20 |
| bshor | $20 |
| paulalab | $19 |
| jwulff | $19 |
| cflexman | $19 |
| Wolfschi | $19 |
| Vlad | $17 |
| ytmytm | $17 |
| benthinkin | $16 |
Round 0 Meta-questions
- ‘What will be the average replication rate in score?’
- ‘What will be the average replication rate in economics?’
- ‘What will be the average replication rate in political sciences?’
- ‘What will be the average replication rate in psychology?’
- ‘What will be the average replication rate in education research?’
- ‘What will be the average replication rate in sociology and criminology?’
- ‘What will be the average replication rate in marketing, management and related areas?’
- ‘What will be the average replication rate for papers published in 2009/10?’
- ‘What will be the average replication rate for papers published in 2011/12?’
- ‘What will be the average replication rate for papers published in 2013/14?’
- ‘What will be the average replication rate for papers published in 2015/16?’
- ‘What will be the average replication rate for papers published in 2017/18?’
Awesome. Would be interesting to see (a) the aggregate replication rates (with error bars) answering the round 0 questions as well as (b) the trading volumes of each of the winners or similar accuracy metrics. Curious to gauge how much of the difference in payout is due to more trades made vs better judgment.
Definitely interesting – just trying to figure out how much we can release without violating SCORE embargo.
We are revealing the overall replication rate (so far) was 54% — using all 100 of the non-covid resolutions we have received. (Uses both direct and data analytic replications for a better estimate.) That’s unofficial but the the sample is large enough it should be pretty stable and non-identifying.
The per-field rates varied quite a bit, but some samples are small (<10), so we're holding those until we can confer more with COS.