{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T12:25:28Z","timestamp":1783427128446,"version":"3.54.6"},"publisher-location":"New York, NY, USA","reference-count":71,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,5,6]],"date-time":"2021-05-06T00:00:00Z","timestamp":1620259200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,5,6]]},"DOI":"10.1145\/3411764.3445569","type":"proceedings-article","created":{"date-parts":[[2021,5,8]],"date-time":"2021-05-08T05:03:13Z","timestamp":1620450193000},"page":"1-15","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":53,"title":["Designing Effective Interview Chatbots: Automatic Chatbot Profiling and Design Suggestion Generation for Chatbot Debugging"],"prefix":"10.1145","author":[{"given":"Xu","family":"Han","sequence":"first","affiliation":[{"name":"University of Colorado Boulder, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michelle","family":"Zhou","sequence":"additional","affiliation":[{"name":"Juji, Inc., United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew J.","family":"Turner","sequence":"additional","affiliation":[{"name":"University of Colorado at Boulder, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tom","family":"Yeh","sequence":"additional","affiliation":[{"name":"University of Colorado Boulder, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,5,7]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"Wikipedia 2020. The Wikipedia Corpus. Wikipedia. https:\/\/www.english-corpora.org\/wiki\/  Wikipedia 2020. The Wikipedia Corpus. Wikipedia. https:\/\/www.english-corpora.org\/wiki\/"},{"key":"e_1_3_2_2_2_1","unstructured":"ManyChat 2020. ManyChat. ManyChat. https:\/\/manychat.com  ManyChat 2020. ManyChat. ManyChat. https:\/\/manychat.com"},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1177\/174701610900500203"},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.5555\/1518321.1518367"},{"key":"e_1_3_2_2_5_1","unstructured":"Susan\u00a0E Brennan. 1990. Conversation as direct manipulation: An iconoclastic view. The art of human-computer interface design n.a. (1990) 393\u2013404.  Susan\u00a0E Brennan. 1990. Conversation as direct manipulation: An iconoclastic view. The art of human-computer interface design n.a. (1990) 393\u2013404."},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-008-9057-x"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11162-011-9217-4"},{"key":"e_1_3_2_2_8_1","first-page":"3835","article-title":"Interactive feedback can improve the quality of responses in web surveys. In Proc","author":"Conrad G","year":"2005","unstructured":"Frederick\u00a0 G Conrad , Mick\u00a0 P Couper , Roger Tourangeau , Mirta Galesic , and T Yan . 2005 . Interactive feedback can improve the quality of responses in web surveys. In Proc . Surv. Res. Meth. Sect. Am. Statist. Ass. Reseachgate, n.a. , 3835 \u2013 3840 . Frederick\u00a0G Conrad, Mick\u00a0P Couper, Roger Tourangeau, Mirta Galesic, and T Yan. 2005. Interactive feedback can improve the quality of responses in web surveys. In Proc. Surv. Res. Meth. Sect. Am. Statist. Ass. Reseachgate, n.a., 3835\u20133840.","journal-title":"Surv. Res. Meth. Sect. Am. Statist. Ass. Reseachgate, n.a."},{"key":"e_1_3_2_2_9_1","unstructured":"Google Cloud 2020. Data Loss Prevention. Google Cloud. https:\/\/cloud.google.com\/dlp  Google Cloud 2020. Data Loss Prevention. Google Cloud. https:\/\/cloud.google.com\/dlp"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v11i1.14955"},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1177\/0894439315573744"},{"key":"e_1_3_2_2_12_1","unstructured":"Jan Deriu Alvaro Rodrigo Arantxa Otegi Guillermo Echegoyen Sophie Rosset Eneko Agirre and Mark Cieliebak. 2019. Survey on evaluation methods for dialogue systems. n.a. (2019) n.a.arXiv:1905.04071  Jan Deriu Alvaro Rodrigo Arantxa Otegi Guillermo Echegoyen Sophie Rosset Eneko Agirre and Mark Cieliebak. 2019. Survey on evaluation methods for dialogue systems. n.a. (2019) n.a.arXiv:1905.04071"},{"key":"e_1_3_2_2_13_1","unstructured":"Google Cloud 2020. Dialogflow. Google Cloud. https:\/\/cloud.google.com\/dialogflow  Google Cloud 2020. Dialogflow. Google Cloud. https:\/\/cloud.google.com\/dialogflow"},{"key":"e_1_3_2_2_14_1","unstructured":"Amazon Alexa 2020. Establish and Maintain Trust. Amazon Alexa. https:\/\/developer.amazon.com\/en-US\/docs\/alexa\/alexa-design\/trustbusters.html  Amazon Alexa 2020. Establish and Maintain Trust. Amazon Alexa. https:\/\/developer.amazon.com\/en-US\/docs\/alexa\/alexa-design\/trustbusters.html"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1177\/0081175016637890"},{"key":"e_1_3_2_2_16_1","volume-title":"Eighth International Conference on Weblogs and Social Media (ICWSM-14)","author":"Gilbert CHE","year":"2014","unstructured":"CHE Gilbert and Erric Hutto . 2014 . Vader: A parsimonious rule-based model for sentiment analysis of social media text . In Eighth International Conference on Weblogs and Social Media (ICWSM-14) . Available at (20\/04\/16) http:\/\/comp.social.gatech.edu\/papers\/icwsm14.vader.hutto.pdf, Vol.\u00a081. AAAI, Palo Alto, California, USA, 82. CHE Gilbert and Erric Hutto. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Eighth International Conference on Weblogs and Social Media (ICWSM-14). Available at (20\/04\/16) http:\/\/comp.social.gatech.edu\/papers\/icwsm14.vader.hutto.pdf, Vol.\u00a081. AAAI, Palo Alto, California, USA, 82."},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1163\/9789004368811_003"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300439"},{"key":"e_1_3_2_2_19_1","unstructured":"Fenfei Guo Angeliki Metallinou Chandra Khatri Anirudh Raju Anu Venkatesh and Ashwin Ram. 2018. Topic-based evaluation for conversational bots. n.a. (2018) n.a.arXiv:1801.03622  Fenfei Guo Angeliki Metallinou Chandra Khatri Anirudh Raju Anu Venkatesh and Ashwin Ram. 2018. Topic-based evaluation for conversational bots. n.a. (2018) n.a.arXiv:1801.03622"},{"key":"e_1_3_2_2_20_1","volume-title":"Evaluating survey quality in health services research: a decision framework for assessing nonresponse bias. Health services research 48, 3","author":"Halbesleben RB","year":"2013","unstructured":"Jonathon\u00a0 RB Halbesleben and Marilyn\u00a0 V Whitman . 2013. Evaluating survey quality in health services research: a decision framework for assessing nonresponse bias. Health services research 48, 3 ( 2013 ), 913\u2013930. Jonathon\u00a0RB Halbesleben and Marilyn\u00a0V Whitman. 2013. Evaluating survey quality in health services research: a decision framework for assessing nonresponse bias. Health services research 48, 3 (2013), 913\u2013930."},{"key":"e_1_3_2_2_21_1","unstructured":"Xu Han and Tom Yeh. 2019. Evaluating Voice Skills by Design Guidelines Using an Automatic Voice Crawler. n.a. (2019) n.a.arXiv:1906.01122  Xu Han and Tom Yeh. 2019. Evaluating Voice Skills by Design Guidelines Using an Automatic Voice Crawler. n.a. (2019) n.a.arXiv:1906.01122"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1093\/ijpor\/edl028"},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3278721.3278777"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jocm.2013.03.005"},{"key":"e_1_3_2_2_25_1","unstructured":"IBM 2020. IBMwatson. IBM. https:\/\/www.ibm.com\/watson  IBM 2020. IBMwatson. IBM. https:\/\/www.ibm.com\/watson"},{"key":"e_1_3_2_2_26_1","unstructured":"simplenlg 2020. simplenlg. simplenlg. https:\/\/github.com\/simplenlg\/simplenlg  simplenlg 2020. simplenlg. simplenlg. https:\/\/github.com\/simplenlg\/simplenlg"},{"key":"e_1_3_2_2_27_1","unstructured":"Juji 2020. Juji-api. Juji. https:\/\/juji.io\/ai-chatbot-api\/  Juji 2020. Juji-api. Juji. https:\/\/juji.io\/ai-chatbot-api\/"},{"key":"e_1_3_2_2_28_1","unstructured":"Juji 2020. Juji-chatbot-design-tips. Juji. https:\/\/juji.io\/docs\/chatbot-design-tips\/#create-natural-and-engaging-conversations  Juji 2020. Juji-chatbot-design-tips. Juji. https:\/\/juji.io\/docs\/chatbot-design-tips\/#create-natural-and-engaging-conversations"},{"key":"e_1_3_2_2_29_1","volume-title":"Dialog systems and chatbots. Speech and language processing 3","author":"Jurafsky Daniel","year":"2017","unstructured":"Daniel Jurafsky and James\u00a0 H Martin . 2017. Dialog systems and chatbots. Speech and language processing 3 ( 2017 ), Chapter 24. Daniel Jurafsky and James\u00a0H Martin. 2017. Dialog systems and chatbots. Speech and language processing 3 (2017), Chapter 24."},{"key":"e_1_3_2_2_30_1","volume-title":"Design and analysis: A researcher\u2019s handbook","author":"Keppel Geoffrey","unstructured":"Geoffrey Keppel . 1991. Design and analysis: A researcher\u2019s handbook . Prentice-Hall, Inc , n.a. Geoffrey Keppel. 1991. Design and analysis: A researcher\u2019s handbook. Prentice-Hall, Inc, n.a."},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300316"},{"key":"e_1_3_2_2_32_1","first-page":"90","article-title":"Review on determining number of Cluster in K-Means Clustering","volume":"1","author":"Kodinariya M","year":"2013","unstructured":"Trupti\u00a0 M Kodinariya and Prashant\u00a0 R Makwana . 2013 . Review on determining number of Cluster in K-Means Clustering . International Journal 1 , 6 (2013), 90 \u2013 95 . Trupti\u00a0M Kodinariya and Prashant\u00a0R Makwana. 2013. Review on determining number of Cluster in K-Means Clustering. International Journal 1, 6 (2013), 90\u201395.","journal-title":"International Journal"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/3477.764879"},{"key":"e_1_3_2_2_34_1","volume-title":"Design thinking research","author":"Kulkarni Chinmay","unstructured":"Chinmay Kulkarni , Steven\u00a0 P Dow , and Scott\u00a0 R Klemmer . 2014. Early and repeated exposure to examples improves creative work . In Design thinking research . Springer , n.a., 49\u201362. Chinmay Kulkarni, Steven\u00a0P Dow, and Scott\u00a0R Klemmer. 2014. Early and repeated exposure to examples improves creative work. In Design thinking research. Springer, n.a., 49\u201362."},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2724660.2724670"},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376327"},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025171.3025206"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173577"},{"key":"e_1_3_2_2_39_1","volume-title":"Bigham Carnegie Mellon University","author":"Mitchell Gordon University of Rochester, Rochester, NY, USA, Jeffrey P.","year":"2020","unstructured":"Mitchell Gordon University of Rochester, Rochester, NY, USA, Jeffrey P. Bigham Carnegie Mellon University , Pittsburgh, PA, USA , and Walter S. Lasecki University of Michigan , Ann Arbor, MI, USA. 2020 . LegionTools | Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. https:\/\/dl.acm.org\/doi\/abs\/10.1145\/2815585.2815729. Accessed: 2020-2-21. Mitchell Gordon University of Rochester, Rochester, NY, USA, Jeffrey P. Bigham Carnegie Mellon University, Pittsburgh, PA, USA, and Walter S. Lasecki University of Michigan, Ann Arbor, MI, USA. 2020. LegionTools | Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. https:\/\/dl.acm.org\/doi\/abs\/10.1145\/2815585.2815729. Accessed: 2020-2-21."},{"key":"e_1_3_2_2_40_1","volume-title":"Ways of Knowing in HCI","author":"Muller Michael","unstructured":"Michael Muller . 2014. Curiosity , creativity, and surprise as analytic tools: Grounded theory method . In Ways of Knowing in HCI . Springer , n.d., 25\u201348. Michael Muller. 2014. Curiosity, creativity, and surprise as analytic tools: Grounded theory method. In Ways of Knowing in HCI. Springer, n.d., 25\u201348."},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379336.3381489"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/2.241424"},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.20801"},{"key":"e_1_3_2_2_44_1","unstructured":"Actions on Google. 2019. Conversation Design. Google. https:\/\/designguidelines.withgoogle.com\/conversation\/  Actions on Google. 2019. Conversation Design. Google. https:\/\/designguidelines.withgoogle.com\/conversation\/"},{"key":"e_1_3_2_2_45_1","unstructured":"Marije Oudejans and Leah\u00a0Melani Christian. 2010. Using interactive features to motivate and probe responses to open-ended questions. Social and behavioral research and the internet: Advances in applied methods and research strategies n.a. n.a. (2010) 304\u2013332.  Marije Oudejans and Leah\u00a0Melani Christian. 2010. Using interactive features to motivate and probe responses to open-ended questions. Social and behavioral research and the internet: Advances in applied methods and research strategies n.a. n.a. (2010) 304\u2013332."},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"crossref","unstructured":"Marije Oudejans and Leah\u00a0Melani Christian. 2011. Using interactive features to motivate and probe responses to open-ended questions. Social and behavioral research and the internet: Advances in applied methods and research strategies n.d. (2011) 215\u201344.  Marije Oudejans and Leah\u00a0Melani Christian. 2011. Using interactive features to motivate and probe responses to open-ended questions. Social and behavioral research and the internet: Advances in applied methods and research strategies n.d. (2011) 215\u201344.","DOI":"10.4324\/9780203844922-9"},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"crossref","unstructured":"Grice\u00a0H Paul 1975. Logic and conversation. Syntax and semantics 3(1975) 41\u201358.  Grice\u00a0H Paul 1975. Logic and conversation. Syntax and semantics 3(1975) 41\u201358.","DOI":"10.1163\/9789004368811_003"},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"e_1_3_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1002\/0471654728"},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1093\/poq\/nfh008"},{"key":"e_1_3_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/1111449.1111475"},{"key":"e_1_3_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043962"},{"key":"e_1_3_2_2_53_1","doi-asserted-by":"crossref","unstructured":"Hannah Rashkin Eric\u00a0Michael Smith Margaret Li and Y-Lan Boureau. 2018. Towards empathetic open-domain conversation models: A new benchmark and dataset. n.a. (2018) n.a.arXiv:1811.00207  Hannah Rashkin Eric\u00a0Michael Smith Margaret Li and Y-Lan Boureau. 2018. Towards empathetic open-domain conversation models: A new benchmark and dataset. n.a. (2018) n.a.arXiv:1811.00207","DOI":"10.18653\/v1\/P19-1534"},{"key":"e_1_3_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753687"},{"key":"e_1_3_2_2_55_1","unstructured":"Abigail See Stephen Roller Douwe Kiela and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. n.a. (2019) n.a.arXiv:1902.08654  Abigail See Stephen Roller Douwe Kiela and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. n.a. (2019) n.a.arXiv:1902.08654"},{"key":"e_1_3_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1631\/FITEE.1700826"},{"key":"e_1_3_2_2_57_1","unstructured":"Tableau 2020. Tableau. Tableau. https:\/\/www.tableau.com  Tableau 2020. Tableau. Tableau. https:\/\/www.tableau.com"},{"key":"e_1_3_2_2_58_1","unstructured":"Wikipedia 2020. Tay(bot). Wikipedia. https:\/\/en.wikipedia.org\/wiki\/Tay_(bot)  Wikipedia 2020. Tay(bot). Wikipedia. https:\/\/en.wikipedia.org\/wiki\/Tay_(bot)"},{"key":"e_1_3_2_2_59_1","unstructured":"Juji 2020. Juji. Juji. https:\/\/juji.io\/no-coding-ai-chatbot-builder  Juji 2020. Juji. Juji. https:\/\/juji.io\/no-coding-ai-chatbot-builder"},{"key":"e_1_3_2_2_60_1","unstructured":"Chatfuel 2020. Chatfuel. Chatfuel. https:\/\/chatfuel.com  Chatfuel 2020. Chatfuel. Chatfuel. https:\/\/chatfuel.com"},{"key":"e_1_3_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376877"},{"key":"e_1_3_2_2_62_1","volume-title":"PARADISE: A framework for evaluating spoken dialogue agents. n.a.","author":"Walker A","year":"1997","unstructured":"Marilyn\u00a0 A Walker , Diane\u00a0 J Litman , Candace\u00a0 A Kamm , and Alicia Abella . 1997 . PARADISE: A framework for evaluating spoken dialogue agents. n.a. (1997), n.a.arXiv:cmp-lg\/9704004 Marilyn\u00a0A Walker, Diane\u00a0J Litman, Candace\u00a0A Kamm, and Alicia Abella. 1997. PARADISE: A framework for evaluating spoken dialogue agents. n.a. (1997), n.a.arXiv:cmp-lg\/9704004"},{"key":"e_1_3_2_2_63_1","unstructured":"Wikipedia contributors. 2020. Information content. https:\/\/en.wikipedia.org\/w\/index.php?title=Information_content&oldid=960984818. Accessed: 2020-8-24.  Wikipedia contributors. 2020. Information content. https:\/\/en.wikipedia.org\/w\/index.php?title=Information_content&oldid=960984818. Accessed: 2020-8-24."},{"key":"e_1_3_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173662"},{"key":"e_1_3_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376131"},{"key":"e_1_3_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302264"},{"key":"e_1_3_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3381804"},{"key":"e_1_3_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531604"},{"key":"e_1_3_2_2_69_1","doi-asserted-by":"crossref","unstructured":"Saizheng Zhang Emily Dinan Jack Urbanek Arthur Szlam Douwe Kiela and Jason Weston. 2018. Personalizing dialogue agents: I have a dog do you have pets too?n.a. (2018) n.a.arXiv:1801.07243  Saizheng Zhang Emily Dinan Jack Urbanek Arthur Szlam Douwe Kiela and Jason Weston. 2018. Personalizing dialogue agents: I have a dog do you have pets too?n.a. (2018) n.a.arXiv:1801.07243","DOI":"10.18653\/v1\/P18-1205"},{"key":"e_1_3_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1162\/coli_a_00368"},{"key":"e_1_3_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3232077"}],"event":{"name":"CHI '21: CHI Conference on Human Factors in Computing Systems","location":"Yokohama Japan","acronym":"CHI '21","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411764.3445569","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3411764.3445569","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:31:25Z","timestamp":1750195885000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411764.3445569"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,6]]},"references-count":71,"alternative-id":["10.1145\/3411764.3445569","10.1145\/3411764"],"URL":"https:\/\/doi.org\/10.1145\/3411764.3445569","relation":{},"subject":[],"published":{"date-parts":[[2021,5,6]]},"assertion":[{"value":"2021-05-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}