Do you know what the price of gas will be in six months? How about the extent of the expected U.S. troop drawdown in Afghanistan by year’s end? As an individual, predicting such things with any degree of confidence may seem the stuff of science fiction, but UC Irvine cognitive scientists say it’s possible.
“As a group, people are collectively more intelligent than you might think,” says professor Mark Steyvers. “We all possess knowledge about different things, some more than others. If we pool these pieces of knowledge together, we can get a pretty accurate look at the big picture.”
It’s called the “wisdom of crowds” effect, and researchers are hoping to get your help putting it to the test.
Steyvers and fellow UCI cognitive sciences professors Michael Lee and William Batchelder are part of a national team given a grant by the Intelligence Advanced Research Projects Activity to develop statistical models based on the crowds concept that can, with increasing precision, forecast the future.
Working with the UCI trio are scientists from the University of Maryland, the University of Michigan, The Ohio State University, Fordham University, Wake Forest University, Wichita State University and private company Applied Research Associates.
They constitute one of five national teams to be funded by IARPA and must meet yearly goals to continue receiving support from the government agency.
In July, the UCI/multi-university team launched its models via a software program called Forecasting Ace. Through volunteer participation, the program will collect individual opinions on the likelihood of certain events within a specified time frame.
Before providing input, contributors must complete a short questionnaire in which they rate their subject-matter expertise in areas including science and technology, business and the economy, politics and policy, military and security, and sports and health.
After the self-assessment, participants can choose which questions about the future they’d like to answer. The software then aggregates and analyzes the collected data using models of human decision making created by the UCI scientists.
“Finding wisdom in crowds is fundamentally a cognitive science problem, because it’s about how people acquire knowledge and make judgments,” says Lee. The concept, while relatively recent, is similar to cultural consensus theory, which Batchelder has been developing for 30 years.
The newer computer and cognitive aspects draw on the expertise of Lee and Steyvers, who have been perfecting these modeling methods at UCI over the past four years through numerous lab experiments and analyses of real-world behavioral data. The professors, along with two student researchers, won a best-paper prize for this work at the July conference of the Cognitive Science Society.
The qualitative and quantitative results from these various studies have helped them construct novel ways of identifying experts by combining self-reported expertise levels and behavioral responses in their models.
As the time frame for a forecasting challenge – such as the likelihood of federal legislators agreeing to raise the U.S. debt ceiling by Aug. 2 – draws near, the UCI team runs the collected data through the models. During this process, “ace analysts” – true experts in their field – are identified, and their opinions are appropriately weighted, helping researchers derive a relatively sound prediction of the upcoming event’s outcome.
They currently don’t know a prediction’s degree of accuracy until the event does or does not occur, but continued data aggregation lets them hone the models to account for more complex and precise forecasting.
The resulting applied project has implications for government agencies interested in improving the accuracy of intelligence analysis. It may also have applications in the fields of business, medicine and policy.
Funding for the endeavor began in May and runs through April 2012, at which time it may be extended. During that period, the research program will strive to achieve a 20 percent increase in forecast accuracy compared with state-of-the-art alternatives. Subsequent years will aim for growth of 35, 50 and more percentage points.