Program Sponsors and Applications Technical Background Participation in CREATE

10) Who can participate in CREATE?
11) What’s the time commitment for participating? (How much time every day, how often visit the site)
12) What will I be asked to do?
13) Will problems have right and wrong answers, or varying degrees of correctness? How will I know how I’m doing on them?
14) What kind of information will I be able to use to analyze problems? Can I search the internet?
15) How long will I have to respond to problems?
16) What size group will I be working with?
17) Do I get to choose what team I’m on? Can I change teams?
18) What personal information will I be asked to provide, and who will have access to it?
19) What will I gain by participating in CREATE? Are there any costs?
20) Can I participate using my tablet or mobile phone?
21) Do I need to download anything or install anything on my computer?
22) How long is the CREATE program? What happens after this study is over?
23) Can I still participate in CREATE
Program Sponsors and Applications
1) What is IARPA? What is their interest in this research?
The Intelligence Advanced Research Projects Activity (IARPA) invests in high-risk, high-payoff research programs to tackle some of the most difficult challenges facing the US Intelligence Community. CREATE seeks to develop and test tools for improving analytic reasoning . IARPA is part of the U.S. Office of the Director of National Intelligence (ODNI).
The Intelligence Community (IC), like many organizations, typically conducts analysis in traditional ways: individual analysts review information sources, think through issues, confer with colleagues, conduct their analysis, and embody their results in written products. This approach is time-tested, intuitive, and requires no special training in methods. But it has drawbacks. The Iraq WMD Commission noted, “Perhaps most troubling, we found an Intelligence Community in which analysts have a difficult time stating their assumptions up front, explicitly explaining their logic, and, in the end, identifying unambiguously for policymakers what they do not know.”
CREATE tools are designed to help analysts work together to improve all these areas—and come to more accurate conclusions and avoid costly errors.
2) Who else is involved in CREATE?
IARPA programs fund two types of organizations: Performers Teams, and Test and Evaluation (T&E) Teams. Performer Teams develop innovative tools and techniques that are intended to achieve program goals. Independent Test and Evaluation (T&E) teams then test those tools against metrics provided by IARPA.
The CREATE program is funding four interdisciplinary, multi-institution Performer Teams for Phase 1, led by primary investigators at:
  • George Mason University
  • Monash University
  • Syracuse University
  • University of Melbourne
  • T&E is led by two organizations:
  • John Hopkins University Applied Physics Lab (JHUAPL), a University Affiliated Research Center with extensive experience in independent testing
  • Good Judgment Incorporated (GJI), a small company founded by a successful team from IARPA’s ACE program, specializing in innovative crowdsourcing and analysis.
  • 3) How will the US Intelligence community use this research?
    CREATE is a little over one year into a 4.5 year program, so it’s not yet clear what range of capabilities the final set of tools will have . However, IC members have expressed interest in providing CREATE tools to analysts, adding them to the portfolio of methods used to improve analysis and provide better answers to America’s most pressing intelligence questions.
    CREATE tools are designed to address two basic needs for improving analysis. First: analysts are always looking for more accurate answers to intelligence questions, and better ways to avoid error and bias. However, the importance of good analysis goes beyond accuracy. Decision makers need to understand the reasoning behind good conclusions, including what alternative explanations were considered, what assumptions were made, how evidence was evaluated, and how confident analysts are in their findings. Clear and thorough reasoning allows the readers of analytic products to make better decisions about how to act on them—and to understand what went wrong if errors are made.
    4) Who will have access to the software developed through CREATE? Will it be open source?
    All software developed for IARPA includes Government Purpose Rights: all U.S. Government agencies and organizations can use it without further cost beyond IARPA’s original funding. However, IARPA doesn’t place any other restrictions on how CREATE software can be used, and many teams from past programs have gone on to further develop their software for public use.
    IARPA encourages, but does not require, the software it funds to be open source, and several of the tools developed for CREATE are meeting this goal.
    5) What else could the CREATE research be used for?
    Almost every profession carries out some form of analysis. CREATE tools, while developed for the IC, should be useful for medical diagnosis, legal analysis, education, and any other field where people need to gather and interpret evidence, compare possible explanations, and come to conclusions based on that evidence.
    Because the applications are so broad, and because analytic practice is similar across disciplines, some of the problems used in CREATE testing may not look much like intelligence questions. By separating the underlying thought processes from specific topics, T&E test better reasoning in ways that will apply across fields.
    Technical Background
    6) What past research suggests that this can work?
    Structured guidance can improve individual reasoning efforts—if it’s used well. But this can be prohibitively difficult or time-consuming when answers are needed quickly. Meanwhile, research shows that the “wisdom of crowds” can improve and speed up answers to many types of questions—but poorly organized collaboration can also lead to less creative answers, or to missing input from quieter team members.
    By combining state-of-the-art structured analytic techniques and collaboration methods, CREATE seeks to leverage the strengths of both, while compensating for their potential weaknesses. Recent research demonstrates that increased structure can indeed improve the results of online collaboration , and that social interaction improves argumentation and reasoning .
    7) Does this relate to the Delphi Method or other collaboration techniques?
    Some CREATE tools may include variations on existing techniques such as Delphi, collaborative document development, etc. However, all Performer teams have built on earlier work to develop new tools and methods, and no previous experience with any collaboration method is necessary to participate in the study.
    8) Are you using AI to supplement human input in this study?
    While CREATE focuses on improving the results of human reasoning, several systems incorporate AI-based techniques to help analysts mitigate bias, check for error, calculate probability and model causality.
    9) What disciplines do the researchers come from?
    CREATE performer teams draw on many disciplines, and include experimental psychologists, political scientists, software developers, statisticians, philosophers, and experts in informal reasoning and critical thinking—among many others.
    Participation in CREATE
    10) Who can participate in CREATE?
    CREATE testing is open to anyone who:
  • Is 18 or older
  • Is able to read and write English fluently
  • Have at least some college education
  • Has regular internet access
  • Is willing to accept browser cookies
  • There are no restrictions on national origin and you can participate from anywhere in the world. If you like solving problems, and coming up with better arguments to convince people of your point of view, we’d love to have you!
    11) What’s the time commitment for participating? (How much time every day, how often visit the site)
    We expect problem-solving to take about 2-3 hours/person each week. Some of the CREATE tools work best if you log in briefly each day, while others work well with a couple of longer session each week.
    12) What will I be asked to do?
    Participants will work on their own or in teams to analyze problems that require different types of reasoning. Some of these will look a lot like intelligence problems—you’ll learn about a political situation and answer questions about motivations or likely outcomes. Others may focus on different topics or more specific questions. Some problems may ask you to make simple calculations or estimate probabilities (but none of them will just be math problems). For this study, none of the problems involve real-world situations and all the information you need to analyze them will be provided.
    Depending on your assignment, you may work on your own or with a team. You may be working with one of the new tools developed for CREATE, or you may be trying to beat their performance with standard tools like Google Docs.
    For each problem, you or your team will be asked to create a written analysis (usually about 1-5 pages) that explains your conclusions and the reasoning behind them. These analyses can include graphics or just written explanations.
    13) Will problems have right and wrong answers, or varying degrees of correctness? How will I know how I’m doing on them?
    Some problems will have definite right answers. Others may be less certain—for example, if you don’t have enough evidence to give a definite answer, a good analysis might offer two answers that explain the evidence reasonably well. Sometimes good answers may include probability or confidence information (e.g., “The answer is 80% likely to be A” or “The answer is almost certain to be A”). The best analysts never try to sound more sure than they really are!
    Because we’re collecting such a large number of analyses, we won’t be able to provide personal feedback right away. After you submit your analysis, we’ll send you an example of what a good answer might look like. As our expert raters complete their evaluation, we’ll provide more personalized feedback, and we’ll keep you updated on what we’re learning from the study .
    14) What kind of information will I be able to use to analyze problems? Can I search the internet?
    For this study, problems will be fictional and all the information needed to analyze them will be provided. We’ll ask you not to search the internet for help with your analysis, and details gathered online will invalidate submissions.
    15) How long will I have to respond to problems?
    For most conditions, one problem is released every week, and you’ll be given two weeks to analyze each problem. Some conditions may offer more problems at any given time, but each problem will still be open for two weeks.
    16) What size group will I be working with?
    Depending on your assignment, you may be working on your own or with a team. Teams may include anywhere from 3-50 people .
    17) Do I get to choose what team I’m on? Can I change teams?
    So that we can tell how useful the tools are when everything else is held constant, all teams will be randomly assigned and you will not be able to change teams. However, all participants will be required to follow a code of conduct, and T&E moderators will remove team members who are disruptive or treat their colleagues badly.
    18) What personal information will I be asked to provide, and who will have access to it?
    You’ll need to provide Good Judgment Incorporated with your name and e-mail address. These will not be shared with IARPA or anyone else outside the two T&E organizations. You will receive a CREATE e-mail account with a study-provided username for study-related communications. If you are assigned to a team-based condition, you’ll be given the opportunity to pick a screen name, and encouraged to use that for your interactions with teammates.
    19) What will I gain by participating in CREATE? Are there any costs?
    CREATE is an all-volunteer study. You’ll be among the first to try out cutting-edge analytic tools, and you’ll learn more about how real analysis works. You may also learn better strategies for reasoning, debate, and collaboration.
    There is no cost to participate, aside from the cost of internet access and data connections, and the value of your volunteer time.
    20) Can I participate using my tablet or mobile phone?
    Most of the CREATE tools are optimized for a desktop computer. You may be able to access them through your mobile device’s internet browser, but they haven’t been developed specifically for this format. Some Performer Teams are developing mobile apps, so depending on what tool you’re assigned to it’s possible that you’ll have this option available.
    21) Do I need to download anything or install anything on my computer?
    No, you don’t need to download or install any software to participate in CREATE testing.
    22) How long is the CREATE program? What happens after this study is over?
    CREATE started in January 2017 and is planned to run for a total of 4.5 years and 3 phases. Performer Teams spent the first year developing their systems, and we’re now ready to test them—which is where you come in!
    We’ll use data from this study to decide which tools should continue development in Phase 2, and find out how they can most usefully be improved. We expect to run the next round of testing, with more advanced systems and more complex analytic questions, in 2019. We’ll then move on to Phase 3, when we’ll develop and test the final systems, capable of handling real-world levels of analytic complexity.
    23) Can I still participate in CREATE?
    Recruiting for the CREATE Phase 1 study is complete and the study is now underway. In Phase 1 participants use cutting-edge analytic tools to solve a series of problems. We are beginning to plan for Phase 2 and anticipate that the next round of testing, with improved tools and even more challenging problems, will begin in Summer 2019. If you would like to receive notification when we begin recruiting for Phase 2 testing, please e-mail info@createbetterreasoning.com.
    Barlow, J. B. (June 2017). Collective intelligence and its relationship to individual intelligence. Presented at Collective Intelligence, New York City, NY.
    Mercier, H., & Sperber, D. (2017). The Enigma of Reason. Harvard University Press: Cambridge MA.
    To participate in CREATE
    Please click on the link below
    Participate in CREATE