UCI researchers develop hybrid human-machine framework for building smarter AI
Model uses human and algorithmic predictions and confidence scores to boost accuracy
Irvine, Calif., March 7, 2022 – From chatbots that answer tax questions to algorithms that drive autonomous vehicles and dish out medical diagnoses, artificial intelligence undergirds many aspects of daily life. Creating smarter, more accurate systems requires a hybrid human-machine approach, according to researchers at the University of California, Irvine. In a study published this month in Proceedings of the National Academy of Sciences, they present a new mathematical model that can improve performance by combining human and algorithmic predictions and confidence scores.
“Humans and machine algorithms have complementary strengths and weaknesses. Each uses different sources of information and strategies to make predictions and decisions,” said co-author Mark Steyvers, UCI professor of cognitive sciences. “We show through empirical demonstrations as well as theoretical analyses that humans can improve the predictions of AI even when human accuracy is somewhat below [that of] the AI – and vice versa. And this accuracy is higher than combining predictions from two individuals or two AI algorithms.”
To test the framework, researchers conducted an image classification experiment in which human participants and computer algorithms worked separately to correctly identify distorted pictures of animals and everyday items – chairs, bottles, bicycles, trucks. The human participants ranked their confidence in the accuracy of each image identification as low, medium or high, while the machine classifier generated a continuous score. The results showed large differences in confidence between humans and AI algorithms across images.
“In some cases, human participants were quite confident that a particular picture contained a chair, for example, while the AI algorithm was confused about the image,” said co-author Padhraic Smyth, UCI Chancellor’s Professor of computer science. “Similarly, for other images, the AI algorithm was able to confidently provide a label for the object shown, while human participants were unsure if the distorted picture contained any recognizable object.”
When predictions and confidence scores from both were combined using the researchers’ new Bayesian framework, the hybrid model led to better performance than either human or machine predictions achieved alone.
“While past research has demonstrated the benefits of combining machine predictions or combining human predictions – the so-called ‘wisdom of the crowds’ – this work forges a new direction in demonstrating the potential of combining human and machine predictions, pointing to new and improved approaches to human-AI collaboration,” Smyth said.
This interdisciplinary project was facilitated by the Irvine Initiative in AI, Law, and Society. The convergence of cognitive sciences – which are focused on understanding how humans think and behave – with computer science – in which technologies are produced – will provide further insight into how humans and machines can collaborate to build more accurate artificially intelligent systems, the researchers said.
Additional co-authors include Heliodoro Tejeda, a UCI graduate student in cognitive sciences, and Gavin Kerrigan, a UCI Ph.D. student in computer science.
Funding for this study was provided by the National Science Foundation under award numbers 1927245 and 1900644 and the HPI Research Center in Machine Learning and Data Science at UCI.
About UCI’s Brilliant Future campaign: Publicly launched on Oct. 4, 2019, the Brilliant Future campaign aims to raise awareness and support for UCI. By engaging 75,000 alumni and garnering $2 billion in philanthropic investment, UCI seeks to reach new heights of excellence in student success, health and wellness, research and more. The School of Social Sciences and the Donald Bren School of Information & Computer Sciences play vital roles in the success of the campaign. Learn more at https://brilliantfuture.uci.edu/uci-school-of-social-sciences and https://brilliantfuture.uci.edu/donald-bren-school-of-ics.
About the University of California, Irvine: Founded in 1965, UCI is the youngest member of the prestigious Association of American Universities and is ranked among the nation’s top 10 public universities by U.S. News & World Report. The campus has produced five Nobel laureates and is known for its academic achievement, premier research, innovation and anteater mascot. Led by Chancellor Howard Gillman, UCI has more than 36,000 students and offers 224 degree programs. It’s located in one of the world’s safest and most economically vibrant communities and is Orange County’s largest employer, contributing $7 billion annually to the local economy and $8 billion statewide. For more on UCI, visit www.uci.edu.
Media access: Radio programs/stations may, for a fee, use an on-campus ISDN line to interview UCI faculty and experts, subject to availability and university approval. For more UCI news, visit news.uci.edu. Additional resources for journalists may be found at communications.uci.edu/for-journalists.