{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T16:21:22Z","timestamp":1783614082516,"version":"3.55.0"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2023,9,23]],"date-time":"2023-09-23T00:00:00Z","timestamp":1695427200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput.-Hum. Interact."],"published-print":{"date-parts":[[2023,10,31]]},"abstract":"<jats:p>Human-AI collaboration for decision-making strives to achieve team performance that exceeds the performance of humans or AI alone. However, many factors can impact success of Human-AI teams, including a user\u2019s domain expertise, mental models of an AI system, trust in recommendations, and more. This article reports on a study that examines users\u2019 interactions with three simulated algorithmic models, all with equivalent accuracy rates but each tuned differently in terms of true positive and true negative rates. Our study examined user performance in a non-trivial blood vessel labeling task where participants indicated whether a given blood vessel was flowing or stalled. Users completed 140 trials across multiple stages, first without an AI and then with recommendations from an AI-Assistant. Although all users had prior experience with the task, their levels of proficiency varied widely.<\/jats:p><jats:p>Our results demonstrated that while recommendations from an AI-Assistant can aid in users\u2019 decision making, several underlying factors, including user base expertise and complementary human-AI tuning, significantly impact the overall team performance. First, users\u2019 base performance matters, particularly in comparison to the performance level of the AI. Novice users improved, but not to the accuracy level of the AI. Highly proficient users were generally able to discern when they should follow the AI recommendation and typically maintained or improved their performance. Mid-performers, who had a similar level of accuracy to the AI, were most variable in terms of whether the AI recommendations helped or hurt their performance. Second, tuning an AI algorithm to complement users\u2019 strengths and weaknesses also significantly impacted users\u2019 performance. For example, users in our study were better at detecting flowing blood vessels, so when the AI was tuned to reduce false negatives (at the expense of increasing false positives), users were able to reject those recommendations more easily and improve in accuracy. Finally, users\u2019 perception of the AI\u2019s performance relative to their own performance had an impact on whether users\u2019 accuracy improved when given recommendations from the AI. Overall, this work reveals important insights on the complex interplay of factors influencing Human-AI collaboration and provides recommendations on how to design and tune AI algorithms to complement users in decision-making tasks.<\/jats:p>","DOI":"10.1145\/3534561","type":"journal-article","created":{"date-parts":[[2023,3,10]],"date-time":"2023-03-10T11:45:23Z","timestamp":1678448723000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":97,"title":["Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Making"],"prefix":"10.1145","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5661-4659","authenticated-orcid":false,"given":"Kori","family":"Inkpen","sequence":"first","affiliation":[{"name":"Microsoft Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7042-5766","authenticated-orcid":false,"given":"Shreya","family":"Chappidi","sequence":"additional","affiliation":[{"name":"University of Virginia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9249-0139","authenticated-orcid":false,"given":"Keri","family":"Mallari","sequence":"additional","affiliation":[{"name":"University of Washington"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7554-8586","authenticated-orcid":false,"given":"Besmira","family":"Nushi","sequence":"additional","affiliation":[{"name":"Microsoft Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2281-8172","authenticated-orcid":false,"given":"Divya","family":"Ramesh","sequence":"additional","affiliation":[{"name":"University of Michigan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9819-7221","authenticated-orcid":false,"given":"Pietro","family":"Michelucci","sequence":"additional","affiliation":[{"name":"Human Computation Institute"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3592-9453","authenticated-orcid":false,"given":"Vani","family":"Mandava","sequence":"additional","affiliation":[{"name":"Microsoft Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0672-6633","authenticated-orcid":false,"given":"Libu\u0161e Hannah","family":"Vep\u0159ek","sequence":"additional","affiliation":[{"name":"LMU Munich"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-7656-7601","authenticated-orcid":false,"given":"Gabrielle","family":"Quinn","sequence":"additional","affiliation":[{"name":"Western Washington University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,9,23]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Stall Catchers. 2019. Join a global game that\u2019s trying to cure Alzheimer\u2019s. Retrieved from https:\/\/stallcatchers.com\/. Accessed 3\/26\/2023."},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2018.07.026"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300233"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v7i1.5285"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33012429"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445717"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-020-65908-y"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377325.3377498"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3449287"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICHI.2015.26"},{"key":"e_1_3_2_12_2","doi-asserted-by":"crossref","unstructured":"Chun-Wei Chiang and Ming Yin. 2021. You\u2019d better stop! Understanding human reliance on machine learning models under covariate shift. In Proceedings of the 13th ACM Web Science Conference .","DOI":"10.1145\/3447535.3462487"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.34.7.571"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376638"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1037\/xge0000033"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.2016.2643"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1080\/014492999118832"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1080\/014492998119526"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1518\/0018720024494856"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1207\/s15327752jpa5003_8"},{"key":"e_1_3_2_21_2","first-page":"973","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence","volume":"17","author":"Elkan Charles","year":"2001","unstructured":"Charles Elkan. 2001. The foundations of cost-sensitive learning. In Proceedings of the International Joint Conference on Artificial Intelligence, Vol. 17. Lawrence Erlbaum Associates Ltd, 973\u2013978."},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0235691"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302265"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533193"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-021-00385-9"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/2899475.2899478"},{"key":"e_1_3_2_27_2","article-title":"Human evaluation of spoken vs. visual explanations for open-domain QA","author":"Gonzalez Ana Valeria","year":"2021","unstructured":"Ana Valeria Gonzalez, Gagan Bansal, Angela Fan, Robin Jia, Yashar Mehdad, and Srinivasan Iyer. 2021. Human evaluation of spoken vs. visual explanations for open-domain QA. In Proceedings of the ACL.","journal-title":"Proceedings of the ACL"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1007\/11538059_91"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1754-9434.2008.00058.x"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1177\/0018720814547570"},{"key":"e_1_3_2_31_2","first-page":"4070","volume-title":"Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI\u201916)","author":"Kamar Ece","year":"2016","unstructured":"Ece Kamar. 2016. Directions in hybrid intelligence: Complementing AI systems with human intelligence. In Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI\u201916). AAAI Press, 4070\u20134073."},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.5555\/2343576.2343643"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.94.2.211"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300641"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287590"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1518\/hfes.46.1.50.30392"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2015.5231"},{"key":"e_1_3_2_38_2","unstructured":"Greg Lipstein. 2020. Meet the winners of the Clog Loss Challenge for Alzheimer\u2019s Research. (Sept.2020). Retrieved from https:\/\/www.drivendata.co\/blog\/clog-loss-alzheimers-winners. Section: Blog. Accessed 3\/26\/2023."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.obhdp.2018.12.005"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445562"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41551-018-0304-0"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1080\/14639220500337708"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1353\/nib.2019.0013"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1080\/14792779943000044"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.2505\/4\/ss18_041_07_12"},{"issue":"4","key":"e_1_3_2_46_2","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1080\/00368555.2021.12293593","article-title":"CITIZEN SCIENCE: Accelerating Alzheimer\u2019s research with stall catchers","volume":"88","author":"Nugent J.","year":"2021","unstructured":"J. Nugent. 2021. CITIZEN SCIENCE: Accelerating Alzheimer\u2019s research with stall catchers. The Science Teacher 88, 4 (2021), 16\u201319. https:\/\/www.jstor.org\/stable\/27135481.","journal-title":"The Science Teacher"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445315"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1002\/for.2464"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3512930"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2017.2648849"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302308"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/2567948.2579215"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1097\/PAS.0000000000001151"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.biocon.2013.11.003"},{"key":"e_1_3_2_55_2","unstructured":"Sarah Tan Julius Adebayo Kori Inkpen and Ece Kamar. 2018. Investigating human+ machine complementarity for recidivism predictions. arXiv:1808.09123. Retrieved from https:\/\/arxiv.org\/abs\/1808.09123."},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.46.2.186.11926"},{"key":"e_1_3_2_57_2","unstructured":"Dayong Wang Aditya Khosla Rishab Gargeya Humayun Irshad and Andrew H. Beck. 2016. Deep learning for identifying metastatic breast cancer. arXiv:1606.05718. Retrieved from https:\/\/arxiv.org\/abs\/1606.05718."},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41551-021-00704-1"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397481.3450650"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2020\/212"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1002\/bdm.2118"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300509"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372852"}],"container-title":["ACM Transactions on Computer-Human Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3534561","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3534561","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:00:10Z","timestamp":1750186810000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3534561"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,23]]},"references-count":62,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,10,31]]}},"alternative-id":["10.1145\/3534561"],"URL":"https:\/\/doi.org\/10.1145\/3534561","relation":{},"ISSN":["1073-0516","1557-7325"],"issn-type":[{"value":"1073-0516","type":"print"},{"value":"1557-7325","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,23]]},"assertion":[{"value":"2021-07-02","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-04-28","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-09-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}