Review In late 2020 we asked forecasters to predict the fate of 400 popular COVID-19 preprints posted to bioRxiv and medRxiv between January and September of that year. For each we asked: Would it be published within a year of posting, and if so, would it be in a high-impact (JIF≥10) journal? [No / High / Low]…
We expected to announce prizes this week, but co-pilot efforts just ran slowly and after verifying Rounds 1-11, today we found a small discrepancy in our Round 0 (meta-claims) calculations that wasn’t sorted by end of day. This will certainly be embarrassingly obvious with a fresh look on Monday. In the meantime, we can say…
SCORE Markets (Rounds 1-10, plus 0 & 11): I waited as long as possible to maximize the number of completed replications, but we now have about two months to resolve claims, pay prizes, and wrap up. There are now max 88 qualifying claims: better than in February, but less than we would like. It is…
Hello forecasters! A quick status update Survey prizes paid out as planned, but we’re still awaiting final December resolutions to let us pay out the main SCORE markets. COS says we should have it shortly. There will be far fewer resolutions than expected, but we cannot wait a year for pandemic-halted experiments. Updated what-to-expect:• We…
Dear Forecasters: Wow! What a year it’s been! Did you know you completed 54,000 surveys and 42,000 trades on 3,000 studies? Plus the current COVID-19 Round 11? That’s stunning. Replication Markets Recap: The View From 3,000 Claims The forecasts are in. The results are not. But there’s quite a bit we can say – if you don’t mind preliminary results.…
Happily, we have seen little evidence of fraud during RM. However, due diligence requires we check at least monthly, and in Round 8 we disqualified 11 accounts for improper collaboration. We cannot offer details, but we can say several would have won survey prizes, and that warning flags included clear ties to a single established…
The new trade widget has the same functionality as the one you are familiar with from Rounds 1-5, with an added dimension. You will have a better sense of the point cost/gain of a trade as you adjust the slider. The balloon value shows the adjusted probability of replication; the red curves show how many…
Cross-posted from RM’s PI’s blog. Nature Communications has an inspiring piece about a DARPA project baking replication into the original study. The DARPA Biological Control program applies independent validation & verification (IV&V) to synthetic biology. The article is a Lessons Learned piece. Although DARPA oversight presumably mitigated the element of competition, the IV&V teams had at least…
SCORE’s change in definition of “replication” (see our blog post on this topic) more than doubled the number of claims that will resolve in Round 6 and later. This has affected our project’s budget. We have adjusted market payouts as described below. However, a change to incentives has to be approved by our Institutional Review…
We have big changes to announce! Read to learn about… simpler definition of replication more markets paying out simpler surveys Simpler Definition of Replication SCORE has adopted a simpler, unified definition of replication: Replication is testing the same claims using data that was not used in the original study. That required some changes from us.…
