{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,6]],"date-time":"2026-04-06T15:20:28Z","timestamp":1775488828922,"version":"3.50.1"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"7","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2025,10,18]]},"abstract":"<jats:p>Understanding the dynamics of human-AI interaction in question answering is crucial for enhancing collaborative efficiency. Extending from our initial formative study, which revealed challenges in human utilization of conversational AI support, we designed two configurations for prompt guidance: a Nudging approach, where the AI suggests potential responses for human agents, and a Highlight strategy, emphasizing crucial parts of reference documents to aid human responses. Through two controlled experiments, the first involving 31 participants and the second involving 106 participants, we compared these configurations against traditional human-only approaches, both with and without AI assistance. Our findings suggest that effective human-AI collaboration can enhance response quality, though merely combining human and AI efforts does not ensure improved outcomes. In particular, the Nudging configuration was shown to help improve the quality of the output when compared to AI alone. This paper delves into the development of these prompt guidance paradigms, offering insights for refining human-AI collaborations in conversational question-answering contexts and contributing to a broader understanding of human perceptions and expectations in AI partnerships.<\/jats:p>","DOI":"10.1145\/3757486","type":"journal-article","created":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T17:32:00Z","timestamp":1760635920000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Interaction Configurations and Prompt Guidance in Conversational AI for Question Answering in Human-AI Teams"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-6428-020X","authenticated-orcid":false,"given":"Jaeyoon","family":"Song","sequence":"first","affiliation":[{"name":"Massachusetts Institute of Technology, Cambridge, MA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0686-7911","authenticated-orcid":false,"given":"Zahra","family":"Ashktorab","sequence":"additional","affiliation":[{"name":"IBM Research, Yorktown Heights, NY, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0437-1736","authenticated-orcid":false,"given":"Qian","family":"Pan","sequence":"additional","affiliation":[{"name":"IBM Research, Cambridge, MA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1508-2091","authenticated-orcid":false,"given":"Casey","family":"Dugan","sequence":"additional","affiliation":[{"name":"IBM Research, Cambridge, MA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4699-5026","authenticated-orcid":false,"given":"Werner","family":"Geyer","sequence":"additional","affiliation":[{"name":"IBM Research, Cambridge, MA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7005-1482","authenticated-orcid":false,"given":"Thomas W.","family":"Malone","sequence":"additional","affiliation":[{"name":"Massachusetts Institute of Technology, Cambridge, MA, USA"}]}],"member":"320","published-online":{"date-parts":[[2025,10,16]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12525-020-00414-7"},{"key":"e_1_2_1_2_1","article-title":"The question answering systems: A survey","volume":"2","author":"Nabil Allam Ali Mohamed","year":"2012","unstructured":"Ali Mohamed Nabil Allam and Mohamed Hassan Haggag. 2012. The question answering systems: A survey. International Journal of Research and Reviews in Information Sciences (IJRRIS), Vol. 2, 3 (2012).","journal-title":"International Journal of Research and Reviews in Information Sciences (IJRRIS)"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300233"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300484"},{"key":"e_1_2_1_5_1","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"4","author":"Ashktorab Zahra","year":"2020","unstructured":"Zahra Ashktorab, Q Vera Liao, Casey Dugan, James Johnson, Qian Pan, Wei Zhang, Sadhana Kumaravel, and Murray Campbell. 2020. Human-ai collaboration in a cooperative game setting: Measuring social perception and outcomes. Proceedings of the ACM on Human-Computer Interaction, Vol. 4, CSCW2 (2020), 1-20."},{"key":"e_1_2_1_6_1","volume-title":"International Conference on Artificial Intelligence in Medicine","author":"Baniecki Hubert","unstructured":"Hubert Baniecki, Bartlomiej Sobieski, Przemys\u0142aw Bombi'nski, Patryk Szatkowski, and Przemys\u0142aw Biecek. 2023. Hospital Length of Stay Prediction Based on Multi-modal Data Towards Trustworthy Human-AI Collaboration in Radiomics. In International Conference on Artificial Intelligence in Medicine. Springer, 65-74."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1111\/puar.13293"},{"key":"e_1_2_1_8_1","volume-title":"Improving human-AI collaboration with descriptions of AI behavior. arXiv preprint arXiv:2301.06937","author":"Cabrera \u00c1ngel Alexander","year":"2023","unstructured":"\u00c1ngel Alexander Cabrera, Adam Perer, and Jason I Hong. 2023. Improving human-AI collaboration with descriptions of AI behavior. arXiv preprint arXiv:2301.06937 (2023)."},{"key":"e_1_2_1_9_1","volume-title":"A test for evaluating performance in human-computer systems. arXiv preprint arXiv:2206.12390","author":"Campero Andres","year":"2022","unstructured":"Andres Campero, Michelle Vaccaro, Jaeyoon Song, Haoran Wen, Abdullah Almaatouq, and Thomas W Malone. 2022. A test for evaluating performance in human-computer systems. arXiv preprint arXiv:2206.12390 (2022)."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3580959"},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the 28th International Conference on Computational Linguistics. 4167-4178","author":"Chatterjee Ajay","year":"2020","unstructured":"Ajay Chatterjee and Shubhashis Sengupta. 2020. Intent Mining from past conversations for Conversational Agent. In Proceedings of the 28th International Conference on Computational Linguistics. 4167-4178."},{"key":"e_1_2_1_12_1","volume-title":"QuAC: Question answering in context. arXiv preprint arXiv:1808.07036","author":"Choi Eunsol","year":"2018","unstructured":"Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. arXiv preprint arXiv:1808.07036 (2018)."},{"key":"e_1_2_1_13_1","doi-asserted-by":"crossref","first-page":"720","DOI":"10.1177\/0165551515590096","article-title":"Answers or no answers: Studying question answerability in stack overflow","volume":"41","author":"Chua Alton YK","year":"2015","unstructured":"Alton YK Chua and Snehasish Banerjee. 2015. Answers or no answers: Studying question answerability in stack overflow. Journal of Information Science, Vol. 41, 5 (2015), 720-731.","journal-title":"Journal of Information Science"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ssaho.2022.100342"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3545945.3569823"},{"key":"e_1_2_1_16_1","unstructured":"Anne R Diekema Ozgur Yilmazel and Elizabeth D Liddy. 2004. Evaluation of restricted domain question-answering systems. (2004)."},{"key":"e_1_2_1_17_1","first-page":"242","article-title":"The proposed uscf rating system, its development, theory, and applications","volume":"22","author":"Elo Arpad E","year":"1967","unstructured":"Arpad E Elo. 1967. The proposed uscf rating system, its development, theory, and applications. Chess Life, Vol. 22, 8 (1967), 242-247.","journal-title":"Chess Life"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 2008 ACM conference on Computer supported cooperative work. 609-618","author":"Erickson Thomas","year":"2008","unstructured":"Thomas Erickson, Catalina M Danis, Wendy A Kellogg, and Mary E Helander. 2008. Assistance: the work practices of human administrative assistants and their implications for it and organizations. In Proceedings of the 2008 ACM conference on Computer supported cooperative work. 609-618."},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","first-page":"308","DOI":"10.1111\/j.1365-2656.2009.01634.x","article-title":"Analysis of variance with unbalanced data: an update for ecology & evolution","volume":"79","author":"Hector Andy","year":"2010","unstructured":"Andy Hector, Stefanie Von Felten, and Bernhard Schmid. 2010. Analysis of variance with unbalanced data: an update for ecology & evolution. Journal of animal ecology, Vol. 79, 2 (2010), 308-316.","journal-title":"Journal of animal ecology"},{"key":"e_1_2_1_20_1","first-page":"36","article-title":"The use of artificial intelligence in gauging the risk of recidivism","volume":"58","author":"Hillman Noel L","year":"2019","unstructured":"Noel L Hillman. 2019. The use of artificial intelligence in gauging the risk of recidivism. Judges J., Vol. 58 (2019), 36.","journal-title":"Judges J."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3579628"},{"key":"e_1_2_1_22_1","first-page":"7","article-title":"Intention to Use Intelligent Conversational Agents in e-Commerce among Malaysian SMEs: An Integrated Conceptual Framework Based on Tri-theories including Unified Theory of Acceptance, Use of Technology (UTAUT), and T-O-E","volume":"9","author":"Ikumoro Abayomi Oluwaseyi","year":"2019","unstructured":"Abayomi Oluwaseyi Ikumoro and M. Jawad. 2019. Intention to Use Intelligent Conversational Agents in e-Commerce among Malaysian SMEs: An Integrated Conceptual Framework Based on Tri-theories including Unified Theory of Acceptance, Use of Technology (UTAUT), and T-O-E. Social Sciences, Vol. 9, 1 (2019), 7.","journal-title":"Social Sciences"},{"key":"e_1_2_1_23_1","volume-title":"Libu\u0161e Hannah Vep\u0159ek, and Gabrielle Quinn","author":"Inkpen Kori","year":"2022","unstructured":"Kori Inkpen, Shreya Chappidi, Keri Mallari, Besmira Nushi, Divya Ramesh, Pietro Michelucci, Vani Mandava, Libu\u0161e Hannah Vep\u0159ek, and Gabrielle Quinn. 2022. Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Making. ACM Transactions on Computer-Human Interaction (2022)."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3334480.3383104"},{"key":"e_1_2_1_25_1","volume-title":"Personalization in goal-oriented dialog. arXiv preprint arXiv:1706.07503","author":"Joshi Chaitanya K","year":"2017","unstructured":"Chaitanya K Joshi, Fei Mi, and Boi Faltings. 2017. Personalization in goal-oriented dialog. arXiv preprint arXiv:1706.07503 (2017)."},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-17","author":"Kabir Samia","year":"2024","unstructured":"Samia Kabir, David N Udo-Imeh, Bonan Kou, and Tianyi Zhang. 2024. Is stack overflow obsolete? an empirical study of the characteristics of chatgpt answers to stack overflow questions. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-17."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v41i3.5257"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572006"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3640543.3645200"},{"key":"e_1_2_1_30_1","volume-title":"Burcu Karagol Ayan, and Deepak Ramachandran","author":"Kim Najoung","year":"2021","unstructured":"Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran. 2021. Which linguist invented the lightbulb? presupposition verification for question-answering. arXiv preprint arXiv:2101.00391 (2021)."},{"key":"e_1_2_1_31_1","unstructured":"Sotiris Kotsiantis Dimitris Kanellopoulos Panayiotis Pintelas et al. 2006. Handling imbalanced datasets: A review. GESTS international transactions on computer science and engineering Vol. 30 1 (2006) 25-36."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00276"},{"key":"e_1_2_1_33_1","doi-asserted-by":"crossref","unstructured":"Yi Lai Atreyi Kankanhalli and Desmond Ong. 2021. Human-AI collaboration in healthcare: A review and research agenda. (2021).","DOI":"10.24251\/HICSS.2021.046"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11747-022-00892-5"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300268"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581225"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1162\/coli.2007.33.1.41"},{"key":"e_1_2_1_38_1","volume-title":"Collaborative Human-AI Decision-Making Systems with Numerical Channels","author":"Mulesa O.","year":"2022","unstructured":"O. Mulesa, Mykola Kotsipak, Sergey Dolgikh, Y. Bilak, T. Radivilova, and Oleksii Baranovskyi. 2022. Collaborative Human-AI Decision-Making Systems with Numerical Channels. Blekinge Institute of Technology (2022). http:\/\/bth.diva-portal.org\/smash\/get\/diva2:1711729\/FULLTEXT01"},{"key":"e_1_2_1_39_1","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"7","author":"Munyaka Imani","year":"2023","unstructured":"Imani Munyaka, Zahra Ashktorab, Casey Dugan, J Johnson, and Qian Pan. 2023. Decision Making Strategies and Team Efficacy in Human-AI Teams. Proceedings of the ACM on Human-Computer Interaction, Vol. 7, CSCW1 (2023), 1-24."},{"key":"e_1_2_1_40_1","first-page":"2045","article-title":"Human-Computer Interaction in Customer Service","volume":"12","author":"Nicolescu L.","year":"2022","unstructured":"L. Nicolescu and Monica Teodora Tudorache. 2022. Human-Computer Interaction in Customer Service: The Experience with AI Chatbots-A Systematic Literature Review. Symmetry, Vol. 12, 12 (2022), 2045.","journal-title":"The Experience with AI Chatbots-A Systematic Literature Review. Symmetry"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbef.2017.12.004"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2017.2724035"},{"key":"e_1_2_1_43_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3343172","article-title":"PlayeRank: data-driven performance evaluation and player ranking in soccer via a machine learning approach","volume":"10","author":"Pappalardo Luca","year":"2019","unstructured":"Luca Pappalardo, Paolo Cintia, Paolo Ferragina, Emanuele Massucco, Dino Pedreschi, and Fosca Giannotti. 2019. PlayeRank: data-driven performance evaluation and player ranking in soccer via a machine learning approach. ACM Transactions on Intelligent Systems and Technology (TIST), Vol. 10, 5 (2019), 1-27.","journal-title":"ACM Transactions on Intelligent Systems and Technology (TIST)"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3311957.3359433"},{"key":"e_1_2_1_45_1","volume-title":"Know what you don't know: Unanswerable questions for SQuAD. arXiv preprint arXiv:1806.03822","author":"Rajpurkar Pranav","year":"2018","unstructured":"Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. arXiv preprint arXiv:1806.03822 (2018)."},{"key":"e_1_2_1_46_1","volume-title":"100,000 questions for machine comprehension of text. arXiv preprint arXiv:1606.05250","author":"Rajpurkar Pranav","year":"2016","unstructured":"Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000 questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 (2016)."},{"key":"e_1_2_1_47_1","first-page":"7421","article-title":"Literature review process: Measuring the effective usage of knowledge management systems in customer support organizations","volume":"2582","author":"Sadashiva Reddy Hima Bindu","year":"2022","unstructured":"Hima Bindu Sadashiva Reddy, Roopesh Reddy Sadashiva Reddy, and Ratnaditya Jonnalagadda. 2022. Literature review process: Measuring the effective usage of knowledge management systems in customer support organizations. Journal homepage: www. ijrpr. com ISSN, Vol. 2582 (2022), 7421.","journal-title":"Journal homepage: www. ijrpr. com ISSN"},{"key":"e_1_2_1_48_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-022-18751-2","article-title":"Experimental evidence of effective human-AI collaboration in medical decision-making","volume":"12","author":"Reverberi C.","year":"2022","unstructured":"C. Reverberi, T. Rigon, A. Solari, C. Hassan, P. Cherubini, et al., 2022. Experimental evidence of effective human-AI collaboration in medical decision-making. Scientific Reports, Vol. 12, 1 (2022), 1-14. https:\/\/www.nature.com\/articles\/s41598-022-18751-2.pdf","journal-title":"Scientific Reports"},{"key":"e_1_2_1_49_1","first-page":"4","article-title":"Customer-support service from a relationship perspective: Best practice for Telecom","volume":"5","author":"Roos Inger","year":"2013","unstructured":"Inger Roos, Martin L\u00f6fgren, and Bo Edvardsson. 2013. Customer-support service from a relationship perspective: Best practice for Telecom. Management Research and Practice, Vol. 5, 2 (2013), 4.","journal-title":"Management Research and Practice"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-023-01445-8"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376229"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3462204.3481771"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3564752"},{"key":"e_1_2_1_54_1","first-page":"725","volume-title":"Proceedings of the International AAAI Conference on Web and Social Media","volume":"7","author":"Tian Qiongjie","year":"2013","unstructured":"Qiongjie Tian, Peng Zhang, and Baoxin Li. 2013. Towards predicting the best answers in community-based question-answering services. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 7. 725-728."},{"key":"e_1_2_1_55_1","doi-asserted-by":"crossref","first-page":"256","DOI":"10.3390\/su12010256","article-title":"The Effect of Social Presence and Chatbot Errors on Trust","volume":"11","author":"Toader D.","year":"2019","unstructured":"D. Toader, G. Boca, Rita Toader, Mara Macelaru, Cezar Toader, D. Ighian, and A. Radulescu. 2019. The Effect of Social Presence and Chatbot Errors on Trust. Sustainability, Vol. 11, 1 (2019), 256.","journal-title":"Sustainability"},{"key":"e_1_2_1_56_1","doi-asserted-by":"crossref","unstructured":"Dakuo Wang Elizabeth Churchill Pattie Maes Xiangmin Fan Ben Shneiderman Yuanchun Shi and Qianying Wang. 2020. From human-human collaboration to Human-AI collaboration: Designing AI systems that can work together with people. In Extended abstracts of the 2020 CHI conference on human factors in computing systems. 1-6.","DOI":"10.1145\/3334480.3381069"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359313"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581282"},{"key":"e_1_2_1_59_1","volume-title":"From human-computer interaction to human-AI Interaction: new challenges and opportunities for enabling human-centered AI. arXiv preprint arXiv:2105.05424","author":"Xu Wei","year":"2021","unstructured":"Wei Xu, Marvin J Dainoff, Liezhong Ge, and Zaifeng Gao. 2021. From human-computer interaction to human-AI Interaction: new challenges and opportunities for enabling human-centered AI. arXiv preprint arXiv:2105.05424, Vol. 5 (2021)."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2022.2041900"},{"key":"e_1_2_1_61_1","volume-title":"AI customer service: Task complexity, problem-solving ability, and usage intention. Australasian marketing journal","author":"Xu Yingzi","year":"2020","unstructured":"Yingzi Xu, Chih-Hui Shieh, Patrick van Esch, and I-Ling Ling. 2020. AI customer service: Task complexity, problem-solving ability, and usage intention. Australasian marketing journal, Vol. 28, 4 (2020), 189-199."},{"key":"e_1_2_1_62_1","first-page":"4525","article-title":"Smarter Response with Proactive Suggestion","author":"Yan Rui","year":"2018","unstructured":"Rui Yan and Dongyan Zhao. 2018. Smarter Response with Proactive Suggestion: A New Generative Neural Conversation Paradigm. In IJCAI. 4525-4531.","journal-title":"A New Generative Neural Conversation Paradigm. In IJCAI."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581388"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3757486","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T17:46:14Z","timestamp":1760636774000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3757486"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,16]]},"references-count":63,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2025,10,18]]}},"alternative-id":["10.1145\/3757486"],"URL":"https:\/\/doi.org\/10.1145\/3757486","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,16]]},"assertion":[{"value":"2025-10-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}