{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T22:17:54Z","timestamp":1776118674884,"version":"3.50.1"},"reference-count":81,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2024,4,17]],"date-time":"2024-04-17T00:00:00Z","timestamp":1713312000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100006374","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62172252"],"award-info":[{"award-number":["62172252"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2024,4,17]]},"abstract":"<jats:p>Conversational Agents (CAs) can facilitate information elicitation in various scenarios, such as semi-structured interviews. Current CAs can ask predetermined questions but lack skills for asking follow-up questions. Thus, we designed three approaches for CAs to automatically ask follow-up questions, i.e., follow-ups on concepts, follow-ups on related concepts, and general follow-ups. To investigate their effects, we conducted a user study (N=26) in which a CA interviewer asked follow-up questions generated by algorithms and crafted by human wizards. Our results showed that the CA's follow-up questions were readable and effective in information elicitation. The follow-ups on concepts and related concepts achieved a lower drop rate and better relevance, while the general follow-ups elicited more informative responses. Further qualitative analysis of the human-CA interview data revealed algorithm drawbacks and identified follow-up question techniques used by the human wizards. We provided design implications for improving information elicitation of future CAs based on the results.<\/jats:p>","DOI":"10.1145\/3637320","type":"journal-article","created":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T10:05:31Z","timestamp":1714385131000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Designing the Conversational Agent: Asking Follow-up Questions for Information Elicitation"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9988-6101","authenticated-orcid":false,"given":"Jiaxiong","family":"Hu","sequence":"first","affiliation":[{"name":"Tsinghua University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4466-7435","authenticated-orcid":false,"given":"Jingya","family":"Guo","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-2605-9000","authenticated-orcid":false,"given":"Ningjing","family":"Tang","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9847-7784","authenticated-orcid":false,"given":"Xiaojuan","family":"Ma","sequence":"additional","affiliation":[{"name":"The Hong Kong University of Science and Technology, Hong Kong SAR, Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6282-342X","authenticated-orcid":false,"given":"Yuan","family":"Yao","sequence":"additional","affiliation":[{"name":"Beijing Jiaotong University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0065-6272","authenticated-orcid":false,"given":"Changyuan","family":"Yang","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-6304-4262","authenticated-orcid":false,"given":"Yingqing","family":"Xu","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,4,26]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Research Methods for Human-Computer Interaction","author":"Adams Anne","year":"1911","unstructured":"Anne Adams, Peter Lunt, and Paul Cairns. 2008. A qualitative approach to HCI research. In Research Methods for Human-Computer Interaction, Paul Cairns and Anna Cox (Eds.). Cambridge University Press, Cambridge, UK, 138--157. http:\/\/oro.open.ac.uk\/11911\/"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1002\/9781119171386.ch19"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300484"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2648.1994.tb01088.x"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSLP.1996.607176"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/K18-1022"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1017\/S1049096502001166"},{"key":"e_1_2_2_8_1","volume-title":"Proceedings of the SIGDIAL 2011 Conference","author":"Bohus Dan","year":"2011","unstructured":"Dan Bohus and Eric Horvitz. 2011. Multiparty Turn Taking in Situated Dialog: Study, Lessons, and Directions. In Proceedings of the SIGDIAL 2011 Conference (Portland, Oregon) (SIGDIAL '11). Association for Computational Linguistics, USA, 98--109. https:\/\/dl.acm.org\/doi\/abs\/10.5555\/2132890.2132903"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASRU.2005.1566504"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/978--1--4020--6821--8_6"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.csl.2008.10.001"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449281"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3392838"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.58"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19--1423"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3555112"},{"key":"e_1_2_2_17_1","volume-title":"Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305","author":"Dodge Jesse","year":"2020","unstructured":"Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305 (2020)."},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3472749.3474784"},{"key":"e_1_2_2_19_1","volume-title":"Basic interviewing skills","author":"Gorden Raymond L","unstructured":"Raymond L Gorden. 1998. Basic interviewing skills. Waveland Press."},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3412330"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445129"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445569"},{"key":"e_1_2_2_23_1","volume-title":"Question Design in Conversation. The handbook of conversation analysis","author":"Hayano Kaoru","year":"2013","unstructured":"Kaoru Hayano. 2013. Question Design in Conversation. The handbook of conversation analysis (2013), 395--408."},{"key":"e_1_2_2_24_1","doi-asserted-by":"crossref","unstructured":"Michael Heilman and Noah A Smith. 2009. Question generation via overgenerating transformations and ranking. Technical Report. Carnegie-Mellon Univ Pittsburgh pa language technologies insT.","DOI":"10.21236\/ADA531042"},{"key":"e_1_2_2_25_1","volume-title":"Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics","author":"Heilman Michael","unstructured":"Michael Heilman and Noah A. Smith. 2010. Good Question! Statistical Ranking for Question Generation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Los Angeles, California) (HLT '10). Association for Computational Linguistics, USA, 609--617."},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2493432.2493502"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00153988"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2022.3205919"},{"key":"e_1_2_2_29_1","volume-title":"Dialogflow, Microsoft Bot Framework, Twilio, and Alexa Skills","author":"Janarthanam Srini","unstructured":"Srini Janarthanam. 2017. Hands-on chatbots and conversational UI development: build chatbots and voice user interfaces with Chatfuel, Dialogflow, Microsoft Bot Framework, Twilio, and Alexa Skills. Packt Publishing Ltd."},{"key":"e_1_2_2_30_1","volume-title":"TARDIS-A simulation platform with an affective virtual recruiter for job interviews. IDGEI (Intelligent Digital Games for Empowerment and Inclusion)","author":"Jones Haza\u00ebl","year":"2013","unstructured":"Haza\u00ebl Jones and Nicolas Sabouret. 2013. TARDIS-A simulation platform with an affective virtual recruiter for job interviews. IDGEI (Intelligent Digital Games for Empowerment and Inclusion) (2013)."},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449161"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300316"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2007.sigdial-1.8"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.3115\/1620932.1620935"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.5555\/1944506.1944564"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274371"},{"key":"e_1_2_2_37_1","volume-title":"Mixout: Effective regularization to finetune large-scale pretrained language models. arXiv preprint arXiv:1909.11299","author":"Lee Cheolhyoung","year":"2019","unstructured":"Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. 2019. Mixout: Effective regularization to finetune large-scale pretrained language models. arXiv preprint arXiv:1909.11299 (2019)."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3502087"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3392836"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376175"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1017\/S1049096502001129"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11859-007-0038-4"},{"key":"e_1_2_2_43_1","volume-title":"Confiding in and Listening to Virtual Agents: The Effect of Personality. arxiv","author":"Li Jingyi","year":"1811","unstructured":"Jingyi Li, Michelle X. Zhou, Huahai Yang, and Gloria Mark. 2018. Confiding in and Listening to Virtual Agents: The Effect of Personality. arxiv: 1811.00746 [cs.AI]"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3395260.3395275"},{"key":"e_1_2_2_45_1","volume-title":"Proceedings of the International Conference: Research in Applied Linguistics.","author":"Louw Stephen","year":"2011","unstructured":"Stephen Louw, Richard Watson Todd, and Pattamawan Jimakorn. 2011. Active listening in qualitative research interviews. In Proceedings of the International Conference: Research in Applied Linguistics."},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449091"},{"key":"e_1_2_2_47_1","unstructured":"G.A. Moerman. 2010. Probing behaviour in open interviews: A field experiment on the effects of Probing Tactics on Quality and Content of the Received Information. (2010)."},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397481.3450689"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3311957.3359502"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.12928\/telkomnika.v13i4.2389"},{"key":"e_1_2_2_51_1","volume-title":"MS MARCO: A human generated machine reading comprehension dataset. In CoCo@ NIPs.","author":"Nguyen Tri","year":"2016","unstructured":"Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In CoCo@ NIPs."},{"key":"e_1_2_2_52_1","volume-title":"Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088","author":"Phang Jason","year":"2018","unstructured":"Jason Phang, Thibault F\u00e9vry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088 (2018)."},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-2124"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1264"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00266"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3161187"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","unstructured":"Herbert J Rubin and Irene S Rubin. 2005. Preparing follow-up questions. 173--200 pages. https:\/\/doi.org\/10.4135\/9781452226651","DOI":"10.4135\/9781452226651"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3555587"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1631\/FITEE.1700826"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1108\/JOSM-11--2019-0341"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v31i1.11164"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2019-1300"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2018.8461494"},{"key":"e_1_2_2_64_1","doi-asserted-by":"crossref","unstructured":"Sandeep Subramanian Tong Wang Xingdi Yuan Saizheng Zhang Yoshua Bengio and Adam Trischler. 2018. Neural Models for Key Phrase Detection and Question Generation. arxiv: 1706.04560 [cs.CL]","DOI":"10.18653\/v1\/W18-2609"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1427"},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3231644.3231654"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3555606"},{"key":"e_1_2_2_68_1","volume-title":"Cognitive interviewing and questionnaire design: A training manual","author":"Willis Gordon Bruce","unstructured":"Gordon Bruce Willis. 1994. Cognitive interviewing and questionnaire design: A training manual. US Department of Health and Human Services, Centers for Disease Control and ?."},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3462204.3481756"},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376131"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302264"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3381804"},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1237"},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.coling-main.563"},{"key":"e_1_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3449975"},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415220"},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3468889"},{"key":"e_1_2_2_78_1","volume-title":"Kdconv: A chinese multi-domain dialogue dataset towards multi-turn knowledge-driven conversation. arXiv preprint arXiv:2004.04100","author":"Zhou Hao","year":"2020","unstructured":"Hao Zhou, Chujie Zheng, Kaili Huang, Minlie Huang, and Xiaoyan Zhu. 2020. Kdconv: A chinese multi-domain dialogue dataset towards multi-turn knowledge-driven conversation. arXiv preprint arXiv:2004.04100 (2020)."},{"key":"e_1_2_2_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3232077"},{"key":"e_1_2_2_80_1","volume-title":"Natural Language Processing and Chinese Computing","author":"Zhou Qingyu","unstructured":"Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. 2018. Neural Question Generation from Text: A Preliminary Study. In Natural Language Processing and Chinese Computing, Xuanjing Huang, Jing Jiang, Dongyan Zhao, Yansong Feng, and Yu Hong (Eds.). Springer International Publishing, Cham, 662--671."},{"key":"e_1_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1622"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637320","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3637320","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T17:13:53Z","timestamp":1755882833000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637320"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,17]]},"references-count":81,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2024,4,17]]}},"alternative-id":["10.1145\/3637320"],"URL":"https:\/\/doi.org\/10.1145\/3637320","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,17]]},"assertion":[{"value":"2024-04-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}