REPLICATION MARKETS PRIZES: OFFICIAL CONTEST RULES v.1.7 (ChangeLog)
The Replication Markets Prize Contest (the “Contest”) is sponsored by the Replication Markets Project(the “Sponsor”), having an address of:
Jacobs
KeyW Corporation, a wholly owned subsidiary of Jacobs
2551 Dulles View Drive,
Suite 700
Herndon, Virginia 20171.
SUMMARY FOR HUMANS:
ELIGIBILITY:
NO PURCHASE OR PAYMENT NECESSARY TO ENTER OR WIN. CONTEST OPEN ONLY TO THOSE WHO ARE 18 YEARS OF AGE OR OLDER AT TIME OF ENTRY. THE CONTEST IS VOID WHERE PROHIBITED OR RESTRICTED BY LAW. The following individuals are also ineligible to participate in this contest: employees of Replication Markets (RM) and any officers, directors, investors, agents, and representatives of RM and their parent organizations, subsidiaries, or affiliates, including any sponsors and affiliates and advertising and promotion agencies of RM, and other performers on the DARPA SCORE program (“Ineligible Affiliates”). This restriction also applies to any immediate family members (defined to include parents, spouse, or children) of those Ineligible Affiliates, and to any person residing in the same household as any of those Ineligible Affiliates. The contest is also not available to any Users previously removed from the Service by RM. RM reserves the right at its sole and absolute discretion to limit the number of contests you may participate in at any given time, should we offer other contests.
CONTEST PERIODS:
The Contest will occur from August 12 at 12:00 pm UTC, 2019 to approximately September 28, 06:00 UTC July 1, 2020 at 12:00 AM UTC, as announced. Contest occurs in Rounds of 2-4 weeks. Each round has crisp start and stop times, announced beforehand.
HOW TO ENTER:
Prior to or during the Contest Period, visit https://predict.replicationmarkets.com and either register or log into your account. Completely and accurately submit the required online member registration form, including a reliable email address. Take the required intake survey and optionally the other intake surveys. Then make forecasts in order to receive virtual points as explained below and in our pre-registration.
You may only use a single account at RM during the duration of the Contest Period. If you already have an Account, you do not need to create a new Account to participate in the Contest. If you have more than one account, notify RM to have the other(s) deactivated or monitored.
Please note that by registering an Account, you will be subject to all notices posted on the Site, including but not limited to the Site’s Informed Consent and Privacy Policy.
By entering the Contest, you agree that your Portfolio and any other information provided by you or collected by Sponsor in connection with the Contest may be used by Sponsor in accordance with Sponsor’s Privacy Policy.
By registering for the Contest, all entrants agree to abide by these Official Contest Rules.
By registering for the Contest, you agree not to allow any other person to use or access your account for any purpose. You are responsible for maintaining the security of your Account information, including your password.
HOW NOT TO PLAY:
Sponsor reserves the right to penalize, suspend, disable, and/or destroy the account(s) of any entrant that it believes in good faith has engaged in any negative or cheating behavior, attempted to interfere with or undermine the Contest or the accuracy of the forecasts, or damage the website, even if those attempts are unsuccessful.
These attempts may be a violation of criminal or civil laws. Should such an attempt be made, RM reserves the right to seek damages from any such individual to the fullest extent permitted by applicable law and/or to take other actions deemed necessary by Sponsor in its sole discretion to protect its rights
Examples of violations include:
– Registering more than once, creating and/or managing more than one account without written authorization from Sponsor for legitimate purposes.
– Coordinating one’s survey or trading activity with any other entrant(s), other than to maximize accuracy-based gains.
– Providing incomplete, non-current, false, or inaccurate data in the registration form.
– Engaging, or planning or threatening to engage in any activity which may result in restricting or inhibiting any other entrant from using and enjoying the Contest, including, but not restricted to boycotts, law suits, computer attacks, and harassment.
– Using third party software robots, scripts, or any other automated system to participate in the Contest outside the approved scripting API & terms, or in a manner that interferes with other forecasters or the accessibility of the system.
– Engaging in any kind of transaction meant primarily to transfer virtual funds (points) from one account to another.
– Engaging in any kind of purposefully losing behavior with respect to ground truth. This includes most forms of collusion and manipulation in surveys and markets.
– Using the discussion forums or comment areas to spread unlawful, threatening, abusive, libelous, defamatory, obscene, vulgar, pornographic, profane or indecent information of any kind.
– Using the discussion forums or comment areas for off-topic criticism of the Sponsor, its partners, staff, or other affiliated parties. (As opposed for example to on-topic objections to a judgment call, feature or rule.)
– Using the discussion forums to transmit any kind of advertising.
The nature and timing of penalties and all other determinations will be entirely at Sponsor’s sole discretion and may be applied without prior warning.
PRIZES:
A minimum of US$150,000 in prizes will be awarded ($195,000 now expected), and announced as they are determined: surveys by the end of the next Round, and markets when the ground truth is revealed by DARPA. Winners will receive money via Google Pay or suitable replacement chosen by the Sponsor (PayPal). Requirements for each prize and amount of prize are described on the Contest recruiting announcement on the RM home page https://replicationmarkets.com/ with pertinent revisions and clarifications posted to the RM blog linked from that page (and currently https://replicationmarkets.com/index.php/rm-blog-home/).
The total prize pool will be split into one part dedicated to the prediction markets (~two thirds), and one part dedicated to surrogate scoring of survey questions (~one third) (now about 3/4 Market, 1/4 Survey). Prizes for surrogate scoring will be paid out after each round. The prize pool for the prediction markets will be paid out at the end of the project, as the results of direct replication become known.
Decisions of the Sponsor are final and binding with respect to all matters related to the Contest. In no event shall the Sponsor be obligated to award any prizes other than the Prizes specified in these Official Contest Rules. The Prizes are non-transferable and no substitutions are permitted. Winners shall be responsible and liable for all federal, state and local taxes on the value of the Prizes. Transfer fees (if any) will be deducted from the prize amount. Sponsor may consider requests to defer small payments.
Winners lists will be posted to the blog, by username. To receive a complete list of winners or a copy of the Official Contest Rules, within six (6) months after the end of a Contest Period, send a self-addressed, stamped envelope to the RM address listed above.
Specify winner’s list or rules on your request.
PRIZE SELECTION:
Surrogate scoring pays prizes on all 3,000 claims (in batches of 10). Market scoring pays only on the (a) 100 50 or so claims that are tested with direct replication in Rounds 1-5, and (b) the 125 or so tested claims in Rounds 6 and beyond. RM will not know which claims those are until resolution.
For more details about claims and resolutions, see the pre-registration and the website.
Survey Prizes (Surrogate Scoring)
The payouts for surrogate scoring will be made after the markets on a round close to ensure that information on performance in surrogate scoring does not affect investments in the market. The prize pool for surrogate scoring will be allocated into prize pools of US$160 for the individual batches: The top four participants for any given batch will receive fixed prices of US$80, US$40, US$20, and US$20. The prize pool for the meta-surveys (Round 0) is US$960, allocated in prizes of US$80 for the top 6 participants, US$40 for participants ranked 7-12, and US$20 for participants ranked 13-24. Top ranks are determined by a method specified in a sealed portion of the pre-registration, revealed at the end of the contest. With high probability, top ranks will go to the most accurate forecasters.
Market Prizes
Round 0: The prize pool for the meta-market is USD$4,320 (USD$360 for each of the 12 meta-claims) and is allocated proportionally to the points earnedwinning shares in these 12 markets.
Rounds 1-5: The prize pool for each Round is about US$9,000 (US$900 per resolving claim, with 10 resolutions expected per Round). This prize pool will be allocated proportionally to the points that participants earn from their investments winning shares in the round: if Round 3 has 9 resolving claims, and you have 5% of the winning shares, you receive 5% of US$8,100, or US$405.
Rounds 6+: The prize pool for each Round is about US$18,750 (US$750 x # of resolving claims, with 25 resolutions expected per Round). Also allocated proportional to total winning shares in the Round (not per-claim).
Participants in the prediction markets do not receive any payouts for points that have not been invested into forecasts.
Participants earn 1 point for each Winning shares are ‘Yes’ shares of a successful direct replication, and 1 point for each ‘No’ shares of an unsuccessful direct replication. (Further details in the pre-registration.) Meta-claims resolve as the proportion of successful direct replications, so a meta-claim resolving as 60% would pay count as 0.6 points per ‘Yes’ shares and 0.4 points per ‘No’ shares.
Winners
Winners will be selected after all eligible activity, all ranks or points are calculated, all corrections (if any) are made, and all site maintenance is performed. RM will confirm of eligibility and compliance with these Official Contest Rules. Winners must be able to receive payments, and comply with any required tax or legal notifications.
The Contest will be conducted under the supervision of the Sponsor. The decisions of the Sponsor are final and binding in all matters relating to this Contest. Winners will be notified by email; however, the Sponsor reserves the right to determine an alternate method of notification. Winners must claim their Prize within 30 days after the date of notification of such Prize. A Winner’s failure to respond to the Prize notification within the specified 30 days will be considered such Winner’s forfeiture of the Prize and an alternate Winner may be selected from the pool of eligible entries. If an entrant is found to be ineligible, an alternate Winner may also be selected from the pool of eligible entries.
In the event of a dispute regarding the identity of the potential Winner, the potential Winner will be deemed to be the person in whose name the applicable Account is registered, and proof of being the authorized account holder may be required. Decision of the Sponsor as to the identity of the entrant shall be final.
Survey winners are selected at the end of each round, 4 per batch of surveys (24 in Round 0), as described above. They resolve ahead of claim resolution, but with high probability, the top ranks will also be the most accurate.
CLAIM RESOLUTION:
Only resolved claims are eligible for Market prizes. Claim resolutions are provided by the Center for Open Science. Other questions such as meta-claims state their resolution conditions and are decided by the Sponsor as stated: meta-claims forecast proportion of regular claim resolutions.
For regular resolutions, shares in the correct answer earn 1 point. For mixture resolutions, the final probabilities will represent the proportion of weight given to each outcome. Scaled questions resolve to the final value and pay as rescaled mixture resolutions.
In the case of any disagreement, ambiguity or differing interpretations among such sources, or of disputed outcomes or any other valuation-related issues, Sponsor reserves the right to make final and binding decisions on all matters, and such decisions will not be subject to appeal. RM provides a 48-hour discussion period after nominating a potential resolution. Forecasters are encouraged to register any objections and supporting arguments during that period.
Sponsor may extend the advertised expiration of a forecast question if the outcome is still unknown / unknowable as of the expiration date. In RM it is not anticipated that claims could resolve before their expiration date.
RM expects 100 175 claims to resolve by June 31, 2020, but this date is beyond our control. If DARPA extends the SCORE PHASE 1 timeline, RM will adapt. If the number of resolved claims changes from anticipated 100, RM will attempt to keep the total at least $900 per claim. RM has some prize money in reserve and may make claims worth more than $900, or may introduce other prizes after January 2020. (The definition change in early 2020 more than doubled the number of resolving claims from Round 6 on. By moving other funds to prizes, we were able to double the per-Round prize pool, though the per-claim average drops to $750. Special thanks here to the generous folks at Stockholm School of Economics.)
DETAILS:
Market prizes are paid proportional to winning shares in a Round’s resolving claims. Shares are earned by trading in the market. The market uses a logarithmic market scoring rule (LMSR) engine with base 2 and liquidity parameter of b=100. For a trade on the correct outcome:
gain = 100 x log2 ( p’ / p )
Where p is the old probability and p’ is the new probability, for that outcome. If p’ > p, gain is positive. If p’ < p, gain is negative. The market is always coherent, so raising (lowering) the probability of one outcome must lower (raise) that of others: all bets are hedged. By default the other probabilities move proportionally. (On a binary claim, moving Yes to p’ necessarily moves No to 1-p’.
Initial prices will be informed by p-values of the original study, as described in the pre-registration.
The default leaderboard calculates the expected value of your portfolio, with the expectation taken at current market prices. However, it may be of limited use in RM, as few claims will resolve, and only at the end. At best it will be a noisy estimate.
Unlike Available Assets, your portfolio value (or expected value or score) fluctuates with the market on forecasts for which the outcome is not yet known.
Notes:
In later rounds, we will enable combinatorial features. Then, the possible outcomes are defined on the full joint space that has been allowed by linking related questions. If you did not use assumptions when forecasting on a question, then your expected value is based only on the market probability for that question (which itself may change based on linked forecasts). If you made an assumption, and that assumption has not resolved, then your expected gain (or loss) is tempered by the market probability of the assumption failing to hold.
The constant ‘100’ above is the market liquidity parameter. RM has held that fixed, but it might change that for some claims or Rounds. Such changes would be noted in the questions’ Background, or the Round notes.
GENERAL CONDITIONS AND RELEASES
Limitations on Liability
By entering (including making a forecast or completing any tasks that might make you eligible to win a prize), you agree to indemnify and hold harmless Sponsor from any claim or cause of action arising out of your participation in the contest. Any entry obtained outside of authorized, legitimate channels, not in compliance with these Official Rules will void all such entries and disqualify any entrant using such methods. If, for any reason, the contest is not capable of running as planned by reason of, but not limited to, tampering, unauthorized intervention, fraud, technical failures, or any other causes which Sponsor deems could corrupt or affect the administration, security, fairness or proper conduct of this contest, Sponsor reserves the right at its sole discretion to cancel, modify, suspend, or terminate the contest and select the winners from among all eligible entries received prior to the action taken, or as otherwise may be deemed fair and equitable by Sponsor, in its sole discretion. Sponsor shall post notice of such action at the blog (currently hosted on https://replicationmarkets.com) or by email to the addresses provided by registered participants. Each entrant agrees to indemnify, hold harmless, and defend Sponsor against any and all third party claims or demands and all related liabilities, damages, settlements, costs, and expenses (including but not limited to, reasonably attorney fees) arising out of any act, or omission of entrant or violation of these Official Rules.
Sponsor assumes no responsibility for lost, late, mutilated, incomplete, stolen, misdirected, illegible or otherwise unacceptable entries, or for any computer, telephone, cable, network, satellite, electronic or Internet hardware or software malfunctions, failures, delays, interruptions or garbled or jumbled transmissions, or service provider/Internet/web site accessibility or availability, traffic congestion, or unauthorized human or nonhuman intervention that might interfere with your entry to or participation in the contest. Sponsor website availability during the Contest Period will be at the sole and absolute discretion of SciCast Replication Markets.
IN NO EVENT WILL SPONSOR, ITS LICENSEES AND OR THEIR PARENTS, AFFILIATES, SUBSIDIARIES AND RELATED COMPANIES, THEIR ADVERTISING OR PROMOTION AGENCIES OR THEIR RESPECTIVE OFFICERS, DIRECTORS, EMPLOYEES, REPRESENTATIVES AND AGENTS, BE RESPONSIBLE OR LIABLE FOR ANY DAMAGES OR LOSSES OF ANY KIND, INCLUDING DIRECT, INDIRECT, INCIDENTAL, CONSEQUENTIAL OR PUNITIVE DAMAGES ARISING OUT OF YOUR ACCESS TO AND USE OF THE RM INTERNET SITE (replicationmarkets.com) OR THE DOWNLOADING FROM AND/OR PRINTING MATERIAL DOWNLOADED FROM SAID SITE. WITHOUT LIMITING THE FOREGOING, EVERYTHING ON THIS SITE IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT. SOME JURISDICTIONS MAY NOT ALLOW THE LIMITATIONS OR EXCLUSION OF LIABILITY FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OR EXCLUSION OF IMPLIED WARRANTIES SO SOME OF THE ABOVE LIMITATIONS OR EXCLUSIONS MAY NOT APPLY TO YOU. CHECK YOUR LOCAL LAWS FOR ANY RESTRICTIONS OR LIMITATIONS REGARDING THESE LIMITATIONS OR EXCLUSIONS.
