{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T06:29:15Z","timestamp":1782282555933,"version":"3.54.5"},"reference-count":53,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2023,4,14]],"date-time":"2023-04-14T00:00:00Z","timestamp":1681430400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"UL Research Institutes through the Center for Advancing Safety of Machine Intelligence"},{"name":"CMU Block Center","award":["3680.1.5007718"],"award-info":[{"award-number":["3680.1.5007718"]}]},{"name":"UT Austin Good Systems"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2023,4,14]]},"abstract":"<jats:p>In many real world contexts, successful human-AI collaboration requires humans to productively integrate complementary sources of information into AI-informed decisions. However, in practice human decision-makers often lack understanding of what information an AI model has access to, in relation to themselves. There are few available guidelines regarding how to effectively communicate aboutunobservables: features that may influence the outcome, but which are unavailable to the model. In this work, we conducted an online experiment to understand whether and how explicitly communicating potentially relevant unobservables influences how people integrate model outputs and unobservables when making predictions. Our findings indicate that presenting prompts about unobservables can change how humans integrate model outputs and unobservables, but do not necessarily lead to improved performance. Furthermore, the impacts of these prompts can vary depending on decision-makers' prior domain expertise. We conclude by discussing implications for future research and design of AI-based decision support tools.<\/jats:p>","DOI":"10.1145\/3579628","type":"journal-article","created":{"date-parts":[[2023,4,16]],"date-time":"2023-04-16T17:23:07Z","timestamp":1681665787000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":31,"title":["Toward Supporting Perceptual Complementarity in Human-AI Collaboration via Reflection on Unobservables"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6730-922X","authenticated-orcid":false,"given":"Kenneth","family":"Holstein","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2297-3308","authenticated-orcid":false,"given":"Maria","family":"De-Arteaga","sequence":"additional","affiliation":[{"name":"The University of Texas at Austin, Austin, TX, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8205-7825","authenticated-orcid":false,"given":"Lakshmi","family":"Tumati","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0321-2853","authenticated-orcid":false,"given":"Yanghuidi","family":"Cheng","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,4,16]]},"reference":[{"key":"e_1_2_1_1_1","first-page":"16","article-title":"If you give a judge a risk score: Evidence from Kentucky bail decisions","volume":"85","author":"Albright Alex","year":"2019","unstructured":"Alex Albright. 2019. If you give a judge a risk score: Evidence from Kentucky bail decisions. Harvard John M. Olin Fellow's Discussion Paper, Vol. 85 (2019), 16.","journal-title":"Harvard John M. Olin Fellow's Discussion Paper"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376277"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445717"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"5","author":"Zana Bucc","year":"2021","unstructured":"Zana Bucc inca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, Vol. 5, CSCW1 (2021), 1--21."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3501831"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447535.3462487"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-11644-5_30"},{"key":"e_1_2_1_8_1","volume-title":"Conference on Fairness, Accountability and Transparency. PMLR, 134--148","author":"Chouldechova Alexandra","year":"2018","unstructured":"Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Conference on Fairness, Accountability and Transparency. PMLR, 134--148."},{"key":"e_1_2_1_9_1","volume-title":"Fairer and more accurate, but for whom? arXiv preprint arXiv:1707.00046","author":"Chouldechova Alexandra","year":"2017","unstructured":"Alexandra Chouldechova and Max G'Sell. 2017. Fairer and more accurate, but for whom? arXiv preprint arXiv:1707.00046 (2017)."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0210966"},{"key":"e_1_2_1_11_1","volume-title":"Leveraging expert consistency to improve algorithmic decision support. arXiv preprint arXiv:2101.09648","author":"De-Arteaga Maria","year":"2021","unstructured":"Maria De-Arteaga, Artur Dubrawski, and Alexandra Chouldechova. 2021. Leveraging expert consistency to improve algorithmic decision support. arXiv preprint arXiv:2101.09648 (2021)."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376638"},{"key":"e_1_2_1_13_1","article-title":"Ames, Iowa: Alternative to the Boston housing data as an end of semester regression project","volume":"19","author":"Cock Dean De","year":"2011","unstructured":"Dean De Cock. 2011. Ames, Iowa: Alternative to the Boston housing data as an end of semester regression project. Journal of Statistics Education, Vol. 19, 3 (2011).","journal-title":"Journal of Statistics Education"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.18608\/jla.2021.7153"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1037\/xge0000033"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3195106.3195133"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479572"},{"key":"e_1_2_1_18_1","volume-title":"A case for humans-in-the-loop: Decisions in the presence of misestimated algorithmic scores. Available at SSRN 4050125","author":"Fogliato Riccardo","year":"2022","unstructured":"Riccardo Fogliato, Maria De-Arteaga, and Alexandra Chouldechova. 2022. A case for humans-in-the-loop: Decisions in the presence of misestimated algorithmic scores. Available at SSRN 4050125 (2022)."},{"key":"e_1_2_1_19_1","volume-title":"Min Kyung Lee, and Matthew Lease","author":"Gao Ruijiang","year":"2021","unstructured":"Ruijiang Gao, Maytal Saar-Tsechansky, Maria De-Arteaga, Ligong Han, Min Kyung Lee, and Matthew Lease. 2021. Human-AI Collaboration with Bandit Feedback. arXiv preprint arXiv:2105.10614 (2021)."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287563"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359152"},{"key":"e_1_2_1_22_1","volume-title":"On the Effect of Information Asymmetry in Human-AI Teams. arXiv preprint arXiv:2205.01467","author":"Hemmer Patrick","year":"2022","unstructured":"Patrick Hemmer, Max Schemmer, Niklas K\u00fchl, Michael V\u00f6ssing, and Gerhard Satzger. 2022. On the Effect of Information Asymmetry in Human-AI Teams. arXiv preprint arXiv:2205.01467 (2022)."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1002\/aaai.12058"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-52237-7_20"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.18608\/jla.2019.62.3"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3419249.3420149"},{"key":"e_1_2_1_27_1","volume-title":"Automating the B2B salesperson pricing decisions: Can machines replace humans and when. Available at SSRN","author":"Karlinsky-Shichor Yael","year":"2019","unstructured":"Yael Karlinsky-Shichor and Oded Netzer. 2019. Automating the B2B salesperson pricing decisions: Can machines replace humans and when. Available at SSRN, Vol. 3368402 (2019)."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517439"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3532106.3533556"},{"key":"e_1_2_1_30_1","volume-title":"Human decisions and machine predictions. The quarterly journal of economics","author":"Kleinberg Jon","year":"2018","unstructured":"Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2018. Human decisions and machine predictions. The quarterly journal of economics, Vol. 133, 1 (2018), 237--293."},{"key":"e_1_2_1_31_1","volume-title":"The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive science","author":"Koedinger Kenneth R","year":"2012","unstructured":"Kenneth R Koedinger, Albert T Corbett, and Charles Perfetti. 2012. The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive science, Vol. 36, 5 (2012), 757--798."},{"key":"e_1_2_1_32_1","volume-title":"Building machines that learn and think like people. Behavioral and brain sciences","author":"Lake Brenden M","year":"2017","unstructured":"Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017. Building machines that learn and think like people. Behavioral and brain sciences, Vol. 40 (2017)."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098066"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1518\/hfes.46.1.50.30392"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445472"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1037\/bul0000324"},{"key":"e_1_2_1_37_1","volume-title":"Algorithms and decision-making in the public sector. arXiv preprint arXiv:2106.03673","author":"Levy Karen","year":"2021","unstructured":"Karen Levy, Kyla Chasalow, and Sarah Riley. 2021. Algorithms and decision-making in the public sector. arXiv preprint arXiv:2106.03673 (2021)."},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the Workshop on Fair and Responsible AI at CHI.","author":"Lurie Emma","year":"2020","unstructured":"Emma Lurie and Deirdre K Mulligan. 2020. Crowdworkers are not judges: Rethinking crowdsourced vignette studies as a risk assessment evaluation technique. In Proceedings of the Workshop on Fair and Responsible AI at CHI."},{"key":"e_1_2_1_39_1","first-page":"6147","article-title":"Predict responsibly: improving fairness and accuracy by learning to defer","volume":"31","author":"Madras David","year":"2018","unstructured":"David Madras, Toni Pitassi, and Richard Zemel. 2018. Predict responsibly: improving fairness and accuracy by learning to defer. NeurIPS, Vol. 31 (2018), 6147--6157.","journal-title":"NeurIPS"},{"key":"e_1_2_1_40_1","volume-title":"International Conference on Machine Learning. PMLR, 6765--6774","author":"Marx Charles","year":"2020","unstructured":"Charles Marx, Flavio Calmon, and Berk Ustun. 2020. Predictive multiplicity in classification. In International Conference on Machine Learning. PMLR, 6765--6774."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03193227"},{"key":"e_1_2_1_42_1","doi-asserted-by":"crossref","unstructured":"Bhavik N Patel Louis Rosenberg Gregg Willcox David Baltaxe Mimi Lyons Jeremy Irvin Pranav Rajpurkar Timothy Amrhein Rajan Gupta Safwan Halabi et al. 2019. Human--machine partnership with artificial intelligence for chest radiograph diagnosis. NPJ digital medicine Vol. 2 1 (2019) 1--10.","DOI":"10.1038\/s41746-019-0189-7"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445315"},{"key":"e_1_2_1_44_1","volume-title":"A Unifying Framework for Combining Complementary Strengths of Humans and ML toward Better Predictive Decision-Making. arXiv preprint arXiv:2204.10806","author":"Rastogi Charvi","year":"2022","unstructured":"Charvi Rastogi, Liu Leqi, Kenneth Holstein, and Hoda Heidari. 2022. A Unifying Framework for Combining Complementary Strengths of Humans and ML toward Better Predictive Decision-Making. arXiv preprint arXiv:2204.10806 (2022)."},{"key":"e_1_2_1_45_1","unstructured":"Steven Ritter Michael Yudelson Stephen Fancsali and Susan R Berman. 2016. Towards integrating human and automated tutoring systems.. In EDM. Citeseer 626--627."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376229"},{"key":"e_1_2_1_47_1","volume-title":"Algorithmic risk assessment in the hands of humans. Available at SSRN 3489440","author":"Stevenson Megan T","year":"2021","unstructured":"Megan T Stevenson and Jennifer L Doleac. 2021. Algorithmic risk assessment in the hands of humans. Available at SSRN 3489440 (2021)."},{"key":"e_1_2_1_48_1","volume-title":"Investigating human machine complementarity for recidivism predictions. arXiv preprint arXiv:1808.09123","author":"Tan Sarah","year":"2018","unstructured":"Sarah Tan, Julius Adebayo, Kori Inkpen, and Ece Kamar. 2018. Investigating human machine complementarity for recidivism predictions. arXiv preprint arXiv:1808.09123 (2018)."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445432"},{"key":"e_1_2_1_50_1","volume-title":"Measurement error and latent variables. A companion to theoretical econometrics","author":"Wansbeek Tom","year":"2001","unstructured":"Tom Wansbeek and Erik Meijer. 2001. Measurement error and latent variables. A companion to theoretical econometrics (2001), 162--179."},{"key":"e_1_2_1_51_1","volume-title":"Learning to Complement Humans. arXiv","author":"Wilder Bryan","year":"2020","unstructured":"Bryan Wilder, Eric Horvitz, and Ece Kamar. 2020. Learning to Complement Humans. arXiv (2020)."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300468"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858373"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3579628","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3579628","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:48:43Z","timestamp":1750182523000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3579628"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,14]]},"references-count":53,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2023,4,14]]}},"alternative-id":["10.1145\/3579628"],"URL":"https:\/\/doi.org\/10.1145\/3579628","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,14]]},"assertion":[{"value":"2023-04-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}