{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,3]],"date-time":"2026-08-03T09:35:32Z","timestamp":1785749732525,"version":"3.56.0"},"reference-count":61,"publisher":"SAGE Publications","issue":"7","license":[{"start":{"date-parts":[[2023,8,8]],"date-time":"2023-08-08T00:00:00Z","timestamp":1691452800000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"The Swedish Foundation for the Humanities and Social Sciences"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Hum Factors"],"published-print":{"date-parts":[[2024,7]]},"abstract":"<jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>We manipulate the presence, skill, and display of artificial intelligence (AI) recommendations in a strategy game to measure their effect on users\u2019 performance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Many applications of AI require humans and AI agents to make decisions collaboratively. Success depends on how appropriately humans rely on the AI agent. We demonstrate an evaluation method for a platform that uses neural network agents of varying skill levels for the simple strategic game of Connect Four.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We report results from a 2 \u00d7 3 between-subjects factorial experiment that varies the format of AI recommendations (categorical or probabilistic) and the AI agent\u2019s amount of training (low, medium, or high). On each round of 10 games, participants proposed a move, saw the AI agent\u2019s recommendations, and then moved.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Participants\u2019 performance improved with a highly skilled agent, but quickly plateaued, as they relied uncritically on the agent. Participants relied too little on lower skilled agents. The display format had no effect on users\u2019 skill or choices.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>The value of these AI agents depended on their skill level and users\u2019 ability to extract lessons from their advice.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Application<\/jats:title>\n                    <jats:p>Organizations employing AI decision support systems must consider behavioral aspects of the human-agent team. We demonstrate an approach to evaluating competing designs and assessing their performance.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/00187208231190459","type":"journal-article","created":{"date-parts":[[2023,8,8]],"date-time":"2023-08-08T19:40:40Z","timestamp":1691523640000},"page":"1914-1927","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":6,"title":["When Do Humans Heed AI Agents\u2019 Advice? When Should They?"],"prefix":"10.1177","volume":"66","author":[{"given":"Richard E.","family":"Dunning","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Baruch","family":"Fischhoff","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alex L.","family":"Davis","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2023,8,8]]},"reference":[{"key":"bibr1-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1016\/j.acra.2004.05.012"},{"key":"bibr2-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1108\/SHR-04-2019-0024"},{"key":"bibr3-00187208231190459","volume-title":"The Complete Book of Connect 4: History, Strategy, Puzzles","author":"Allen J. D.","year":"2010"},{"key":"bibr4-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1016\/j.giq.2020.101490"},{"key":"bibr72-00187208231190459","author":"Bansal G.","year":"2019","journal-title":"The Seventh AAAI Conference on Human Computation and Crowdsourcing (HCOMP-19)"},{"key":"bibr5-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1177\/0018720817700258"},{"key":"bibr6-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1109\/CDC.2011.6160994"},{"key":"bibr7-00187208231190459","doi-asserted-by":"publisher","DOI":"10.32614\/RJ-2017-066"},{"key":"bibr8-00187208231190459","doi-asserted-by":"crossref","unstructured":"Brown N., Sandholm T. (2017). Libratus: The superhuman AI for no-limit poker. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17), pp. 5226\u20135228.","DOI":"10.24963\/ijcai.2017\/772"},{"key":"bibr9-00187208231190459","unstructured":"Buolamwini J., Gebru T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency, pp. 77\u201391.http:\/\/proceedings.mlr.press\/v81\/buolamwini18a\/buolamwini18a.pdf"},{"key":"bibr10-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1177\/00187208211009995"},{"key":"bibr11-00187208231190459","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v034.i02"},{"key":"bibr12-00187208231190459","first-page":"41","volume-title":"Autonomous Weapon Systems: An Exploration of Issues and Recommendations","author":"Defense Science Board","year":"2016"},{"key":"bibr13-00187208231190459","volume-title":"Proceedings of the 2000 Command and Control Research and Technology Symposium","author":"Dzindolet M. T.","year":"2000"},{"key":"bibr14-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1518\/0018720024494856"},{"key":"bibr15-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1177\/0018720816681350"},{"key":"bibr16-00187208231190459","doi-asserted-by":"publisher","DOI":"10.21236\/ADA562448"},{"key":"bibr17-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1016\/0749-5978(90)90002-Q"},{"key":"bibr18-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1080\/0266476042000214501"},{"key":"bibr19-00187208231190459","unstructured":"Future of Life Institute (2023). Pause open AI experiments: An open letter. https:\/\/futureoflife.org\/open-letter\/pause-giant-ai-experiments\/"},{"key":"bibr20-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1007\/s10796-015-9611-0"},{"issue":"31","key":"bibr73-00187208231190459","volume":"4","author":"Gaube S.","year":"2021","journal-title":"npj Digital Medicine"},{"key":"bibr21-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1038\/nature.2016.20784"},{"issue":"1","key":"bibr22-00187208231190459","first-page":"32","volume":"38","author":"Gifford R.","year":"2018","journal-title":"The Proctor"},{"key":"bibr23-00187208231190459","unstructured":"Gilmer J., Schoenholz S. S., Riley P. F., Vinyals O., Dahl G. E. (2017). Neural message passing for quantum chemistry. In 34th International Conference on Machine Learning, ICML 2017, pp. 2053\u20132070."},{"key":"bibr24-00187208231190459","unstructured":"Graybill F. A. (1961). An Introduction to Linear Statistical Models (Vol 1). McGraw-Hill."},{"key":"bibr25-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-022-31236-0"},{"key":"bibr26-00187208231190459","author":"Green B.","year":"2021","journal-title":"Social Science Research Network"},{"key":"bibr27-00187208231190459","volume-title":"Signal detection theory and psychophysics","author":"Green D. M.","year":"1966"},{"key":"bibr28-00187208231190459","unstructured":"Hasbro (2009). Connect four instructions. https:\/\/www.hasbro.com\/common\/documents\/dad2614d1c4311ddbd0b0800200c9a66\/1EF6874419B9F36910222EB9858E8CB8.pdf"},{"key":"bibr29-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1177\/0018720811417254"},{"key":"bibr30-00187208231190459","volume-title":"Fundamentals of Artificial Neural Networks","author":"Hassoun M.","year":"2003"},{"issue":"0","key":"bibr31-00187208231190459","first-page":"1","volume":"00","author":"Kaplan A. D.","year":"2021","journal-title":"Human Factors"},{"key":"bibr32-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1191\/1471082X03st053oa"},{"key":"bibr34-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1518\/hfes.46.1.50_30392"},{"key":"bibr35-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1177\/0272989X07307271"},{"key":"bibr36-00187208231190459","doi-asserted-by":"publisher","DOI":"10.21273\/HORTSCI.38.1.117"},{"key":"bibr76-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-3242-6"},{"key":"bibr37-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1177\/0018720817743223"},{"key":"bibr38-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1518\/hfes.46.2.196.37335"},{"key":"bibr39-00187208231190459","doi-asserted-by":"crossref","unstructured":"Meyer J., Lee D. (2013). Trust, Reliance, and Compliance, The Oxford Handbook of Cognitive Engineering (pp. 109\u2013124). Oxford Library of Psychology, Oxford University Press. Chapter 6.","DOI":"10.1093\/oxfordhb\/9780199757183.013.0007"},{"key":"bibr40-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287596"},{"key":"bibr41-00187208231190459","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1261"},{"key":"bibr42-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1080\/001401300409125"},{"key":"bibr43-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1518\/001872097778543886"},{"key":"bibr44-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-9280.2006.01720.x"},{"key":"bibr45-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMra1814259"},{"key":"bibr75-00187208231190459","author":"Schrittwieser J.","year":"2020","journal-title":"arXiv preprint arXiv:1911.08265v2"},{"key":"bibr47-00187208231190459","volume-title":"The linear model and hypotheses","author":"Seber G. A. F.","year":"1966"},{"key":"bibr48-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1126\/science.aar6404"},{"key":"bibr50-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1037\/1082-989X.11.1.54"},{"key":"bibr51-00187208231190459","volume-title":"Biometry: The principles and practices of statistics in biological research","author":"Sokal R. R.","year":"1969","edition":"1"},{"key":"bibr53-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1724-z"},{"key":"bibr54-00187208231190459","doi-asserted-by":"publisher","DOI":"10.3102\/1076998610396895"},{"key":"bibr55-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1518\/001872008X288394"},{"key":"bibr74-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1080\/14639220500370105"},{"key":"bibr56-00187208231190459","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3389\/fpsyg.2019.00519","volume":"10","author":"Wiczorek R.","year":"2019","journal-title":"Frontiers in Psychology"},{"key":"bibr57-00187208231190459","unstructured":"Xiong W., Droppo J., Huang X., Seide F., Seltzer M., Stolcke A., Zweig G. (2017). Achieving human parity in conversational speech recognition. Microsoft. Report: MSR-TR-2016-71.https:\/\/doi.org\/10.48550\/arXiv.1610.05256"},{"key":"bibr58-00187208231190459","volume-title":"AZFour: Connect four powered by the alphazero algorithm","author":"Young A.","year":"2018"},{"key":"bibr59-00187208231190459","doi-asserted-by":"publisher","DOI":"10.3758\/BF03331120"},{"key":"bibr60-00187208231190459","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372852"}],"container-title":["Human Factors: The Journal of the Human Factors and Ergonomics Society"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/00187208231190459","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/00187208231190459","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/00187208231190459","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T07:53:56Z","timestamp":1777449236000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/00187208231190459"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,8]]},"references-count":61,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2024,7]]}},"alternative-id":["10.1177\/00187208231190459"],"URL":"https:\/\/doi.org\/10.1177\/00187208231190459","relation":{},"ISSN":["0018-7208","1547-8181"],"issn-type":[{"value":"0018-7208","type":"print"},{"value":"1547-8181","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,8]]}}}