Key Information
The following is a short summary of this study to help you decide whether or not to be a part of this study. More detailed information is listed later on in this form.

Why am I being invited to take part in a research study?

We invite researchers in social sciences and statistics, or people who are generally knowledgeable about social sciences to take part in this research study.

You must be 18 or older.

What should I know about a research study?
• Someone will explain this research study to you.
• Whether or not you participate is up to you.
• Your participation is completely voluntary.
• You can choose not to take part.
• You can agree to take part and later change your mind.
• Your decision will not be held against you.
• Your refusal to participate will not result in any consequences or any loss of benefits that you are otherwise entitled to receive.
• You can ask all the questions you want before you decide.
Why is this research being done?

We do this research to understand how accurately groups can predict the outcomes for independent tests of published research claims. We will ask for your judgment about this research using surveys and markets (online platforms that allow participants to buy and sell “financial” contracts).

How long will the research last and what will I need to do?

This is an extended study with regular forecasting. You may leave at any time without penalty. How much time you spend is up to you: after the initial intake survey, you can spend anywhere from a few minutes to several hours per week, depending on your preferences and competitiveness. The basic structure is:
1. One intake survey (about 30 minutes).
2. Roughly eleven forecasting rounds. The first ten rounds are anticipated to be 4 weeks each. Round eleven could last as long as 3-4 months.
3. A 20-minutes exit survey when you leave, or at the end.

Is there any way being in this study could be bad for me?

We don’t believe there are any risks from participating in this research.
Will being in this study help me in any way?

We cannot promise any benefits to you or others from your taking part in this research. However, you may learn about and evaluate diverse research claims.

Detailed Information
The following is more detailed information about this study in addition to the information listed above.

What is the purpose of this research?
We hope to improve the ability to forecast replications and identify effective tools to obtain confidence on social science claims. The term “replication” refers to whether repetition of a research study by some independent research group will lead to the same result as the original study. In addition, we hope to learn which signals are important, and provide labeled training data for automated confidence scores. Ultimately, we hope to help improve the quality of social science research.

Who is eligible? (Detailed Version)
We invite researchers in social sciences and statistics, or people who are generally knowledgeable about social sciences to take part in this research study.
You must be 18 or older.
Caveats:
1. Members of SCORE TA1, TA2, and TA3 teams are not allowed.
2. Anyone involved in SCORE replication efforts may participate, but not forecast on claims they are replicating, nor may anyone who obtains early results from such people.
3. You may not participate simultaneously in any other SCORE forecasting project, nor may you forecast on the same claim in two projects. You may switch projects so long as you avoid forecasting the same claims in both.
4. In the unlikely event we are oversubscribed, we reserve the right to limit participation in any round, and to prioritize based on education, field, intake scores, past performance, and other measures associated with accurate crowd forecasts.

How long will I take part in this research?
This is our Phase I study, and continues through the end of 2020. We hope you will remain enrolled until the end of Phase 1. You may leave at any time, or pause your participation and resume later.

How much time will it take?
1. That’s up to you. There is a single 30-minute intake survey, and then about 11 monthly forecasting rounds. In each round, you may spend anything from about an hour to as long as you like. For the first 10 rounds, the first week of each round is surveys about scientific claims; the second two weeks are markets about those claims; week 4 is normally downtime, but may have a small bonus round. Round 11, which will be on pandemic-related topics, could last as long as 3-4 months. Some forecasters will be invited to complete a short, optional exit survey.

What can I expect if I take part in this research?
This is an online study. At the beginning of the study, you will be asked to first fill out an online intake survey. That survey will ask you some demographic questions and some quiz questions that help us describe our population, and measure factors that might be associated with good performance. (See “Privacy” for more information.) We will provide tutorials, training material, and an email helpdesk.
Then, you will participate in about 11 monthly forecasting rounds, online. The actual tempo and number may vary a bit, but the structure of the first 10 rounds will be:
• About a week to answer surveys about the outcome of social science experiments (“claims”). Surveys are in batches of about 10 claims. You may do as many or as few batches as you like. (More batches means more chances at survey prizes.) Surveys are blind – you will not see others’ forecasts.
• About two weeks to participate in a play-money prediction market containing several hundred “claims”, including the ones you saw in the survey. The markets are open – everyone can see each other’s’ forecasts. They are pseudonymous – you can choose your username and how much you reveal to other forecasters. You may return to the market as often as you like during the open period.
• A week of down time, possibly with optional side forecasts to improve calibration.
In the surveys and markets, you will be asked to give a probabilistic assessment on, for example, how likely a replication study, if carried out, will reproduce the results in the original study. You will have a chance to learn how the market works. Round 11, which will be on pandemic-related topics, could last as long as 3-4 months. Round 11 is expected to have 100 claims at a time, every couple of weeks, for an expected total of 400-600 claims.
At the end of our study or when you decide to leave our platform, you may be invited to complete an optional exit survey to provide feedback for our study and our platform.

What happens if I say yes, but I change my mind later?
You can leave the research at any time it will not be held against you. You will still be eligible for any prizes. Unless you opt out, you may be contacted about opportunities to participate in future research. You may opt out by emailing support@replicationmarkets.com. You may unsubscribe from mailings the same way, or by clicking the “unsubscribe” link in our emails.

Is there any way being in this study could be bad for me? (Detailed Risks)
We don’t believe there are any risks from participating in this research.

Privacy: If I take part in this research, how will my privacy be protected? What happens to the information you collect?
Your participation in this research study is completely voluntary. Your identifying data (PII) will only be reported in the aggregate and will not be linked to any identifiable personal information in any way.

However, de-identified data from your participation will become “open data”, stored in a publicly-available online database (likely Open Science Framework, osf.io). This will allow anyone interested in the research to conduct their own analysis of the de-identified data. By default your comments and pseudonym will be available in that data! Although other information is not identifying per-se, there is a risk that your answers, especially combined with other online information, could identify you

You have the right to withdraw from the study at any time without penalty and to have all identifying data related to your participation erased. To withdraw and have your data erased, contact support@replicationmarkets.com using the contact information below. (Your market forecasts may remain, but completely de-identified; your comments will be sanitized or replaced with “Comment Deleted”.)

If you opt into the prizes, any additional payment information we may need is stored completely separately, and is available only to those involved in prize distribution and fraud detection.

Efforts will be made to limit the use and disclosure of your Personal Information, including research study records, to people who have a need to review this information. We cannot promise complete secrecy. Organizations that may inspect and copy your information include the IRB and other representatives of this organization. Authorized representatives from the United States Department of the Navy Human Research Protection Program may review and/or obtain identifiable information related to your participation in this research study as part of their responsibility to protect human research subject volunteers.

Surveys, including the intake survey, may be conducted via secure connection to a major commercial survey provider like Qualtrics with appropriate safeguards.

Additional electronic information such as IP address and browser information may be collected and stored short-term for fraud detection and system operation, including system monitoring and optimization.

Compensation –
There is a prize pool of at least $12,500 per round (assuming 12 round), with most paid out according to actual outcomes in the market, when they become known (expected late 2020). The remainder are paid as per-round incentives for correlates of good forecasting. Details are provided as official contest Rules prominently available on the website. The reward plan changed in February 2020. Please see the “Rules” on the website for complete details.

Payment will be managed by Jacobs (formerly KeyW). Where possible, we will use online payments such as electronic gift cards, Paypal, or similar. Taxes, if any, are the participant’s responsibility.

What else do I need to know?
This research is funded by Defense Advanced Research Projects Agency (DARPA).
Who can I talk to?
If you have questions, concerns, or complaints, or think the research has hurt you, talk to the research team at support@replicationmarkets.com. Failing that, you may reach the Co-Principal Investigator (Charles Twardy) at ctwardy@replicationmarkets.com, or (703) 817-4773. This research has been reviewed and approved by the Harvard University Area Institutional Review Board (“IRB”). You may talk to them at (617) 496-2847 or cuhs@harvard.edu if:
• Your questions, concerns, or complaints are not being answered by the research team.
• You cannot reach the research team.
• You want to talk to someone besides the research team.
• You have questions about your rights as a research subject.
• You want to get information or provide input about this research.

[radio button] I am eighteen (18) years of age or older and have read, understand and agree to the informed consent statement and the informed consent details.

[NEXT]

GDPR ADDENDUM

The researchers will collect information about you. This form calls such information your “Personal Information” and it will include your name, demographic information, your responses to any tests, surveys or procedures described in this informed consent form.

If you withdraw your permission, you will no longer be able to participate in the study. No new information will be collected about you or from you by the study team. Your withdrawal has no effect on the lawfulness of the data processing that occurred prior to your withdrawal.

Your Personal Information that has already been collected to the time of your withdrawal will be kept and used to guarantee the integrity of the study and/or for any other purposes permitted under applicable data protection and privacy laws.

Your Personal Information will not be used for further research. However, if permitted by applicable law, your Personal Information may be anonymized so that the information does not identify you personally, and such anonymized information may be used for further research.

Your Personal Information will be treated in compliance with applicable data protection laws. Harvard is the controller of your Personal Information collected for this study.

Harvard and some of the other people using your Personal Information may be based in countries other than your country, including the United States. The European Commission has determined that the data protection laws of the United States do not protect personal information to the same extent as those of the European Economic Area. By signing this consent form, you consent to the transfer of your information to the U.S. Harvard and those working with Harvard will take steps to maintain the confidentiality of your Personal Information.

Harvard, the IRB(s) and IEC(s), collaborating institutions of this project, including Jacobs (formerly KeyW), Gold Brand Software, Massey University of New Zealand, Stockholm School of Economics, and University of California at Santa Cruz, will obtain and use your Personal Information to conduct and manage this study, and to comply with legal or regulatory requirements, including to:

• provide you with payment or prize, if allowed by the study, for your time, effort and performance related to your participation;
• verify that the study is conducted correctly and that study data are accurate;
• answer questions from IRB(s), IEC(s), or government or regulatory agencies;
• contact you during and after the study (if necessary); and
• answer your data protection requests (if any).

Your Personal Information may also be used by the individuals and groups listed above to:
• Publish summaries of the study results in academic journals, on the internet or at educational meetings of other researchers. You will not be directly identified in any publication or report of the study. But, some journal representatives may need access to your Personal Information to verify the study results and ensure the research meets the journal’s quality standards. Also, journals may require that certain data from the study that does not directly identify you (i.e., de-identified survey responses) be made available to other researchers for further research projects.
• Improve the quality, design and safety of this study and other research studies.
• Conduct additional studies with the data collected in this study to advance scientific research and public health. At this time, we do not know the specific details of these future research projects. If your Personal Information is used for additional studies, specific safeguards will be used to protect the data, which may include:

o Using only information from which your direct identifiers have been removed instead of information that readily identifies you.
o Limiting access to specific individuals who are obligated to keep the information confidential.
o Using security measures to avoid data loss and unauthorized access.
o Anonymizing the data by destroying the link between the information and your personal identifiers.
o When required by applicable law, ensuring that the scientific research has the approval of IECs, IRBs, or other similar review groups.

Harvard will retain your Personal Information (including your Coded Information) for the period necessary to fulfill the purposes outlined in this informed consent form, unless a different retention period is required or permitted by law.

Your rights related to your Personal Information collected under the study are described below. If you wish to exercise any of these rights, you must contact EEAdatasubjectrequest@harvard.edu

• You have the right to see the information being collected about you in the study.
• You have the right to correct or update your Personal Information if it is inaccurate.
• You have the right to limit the collection and use of your Personal Information under certain circumstances (for example, if you think that the information is inaccurate).
• You have the right to receive your Personal Information in a structured, common computer format (for example, in a readable text electronic file or chart) for your own purposes or for giving it to others, as required by applicable data protection laws. You may not have the right to receive your Personal Information that has been used for public interest purposes or in the exercise of official authority vested in Harvard.
• You have the right to request the deletion of your Personal Information if you are no longer participating in the study. However, there are limits on your ability to request deletion of your Personal Information. Harvard may keep and use some or all of your Personal Information if deletion would seriously impair the study (for example, if deletion would affect the consistency of study results) or if your Personal Information is needed to comply with legal requirements.
• You have the right to make a complaint to a data protection authority within the EU (http://ec.europa.eu/justice/data-protection/article-29/structure/data-protection-authorities/index_en.htm).

☐ Your checking this box documents that you have freely given your consent to the use of Personal Information as described in this GDPR Addendum.