The Central Intelligence Agency’s “Good Judgement Project” hires thousands of average citizen non-experts to predict future disaster — and it’s more accurate than the highly-classified intelligence data accessible to agency analysts.
Good Judgement employs about 3,000 ordinary people to weigh in on questions ranging from North Korea’s missile project to Russia’s potential invasion of Ukraine, and its participants’ guesses are surprisingly accurate all the way down to in-depth issues like Venezuelan gas subsidies and internal North Korean politics.
“I’m just a pharmacist,” project participant Elaine Rich of Maryland told NPR. “Nobody cares about me, nobody knows my name, I don’t have a professional reputation at stake. And it’s this anonymity which actually gives me freedom to make true forecasts.”
Rich is in the top one percent of forecasters, all of whom log onto a website with very specific questions about potential global events deemed important to intelligence officials, and enter numeric probabilities that amount to little more than guesses. The results are often more accurate than agencies’ traditional data-gathering techniques and analyses.
“The first two years I did this, all you do is choose numbers,” Rich said. “You don’t have to say anything about what you’re thinking, you don’t have to justify your numbers. You just choose numbers and then see how your numbers work out.”
Rich has since been put on a team of 30 “superforecasters,” whose predictions are up to 30 percent more accurate than intelligence analysts. The pharmacist in her 60s uses a simple Google search to find the information she uses to make her predictions.
The project seems to indicate that experts analyzing classified intelligence may have far less of an edge than one would otherwise expect over a large group of average citizens using publicly accessible data from popular online search engines.
Good Judgment was developed by psychologists and officials across multiple federal intelligence agencies, and has been ongoing for three years. According to one of the founders, pooling a group of people, tracking and providing feedback on their accuracy is far more effective than drawing analyses from one smart expert.
By averaging large numbers of predictions together, developers found outlaying errors often cancel each out, leaving a central core of predictions that are very accurate.
“They’ve shown that you can significantly improve the accuracy of geopolitical forecasts, compared to methods that had been the state of the art before this project started,” intelligence community member Jason Matheny said.
Good Judgement plans to take on even more recruits in 2014, and though intelligence officials don’t believe it will replace traditional intelligence gathering and analysis methods, it will likely become a complimentary method.