{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T23:24:11Z","timestamp":1780529051021,"version":"3.54.1"},"publisher-location":"New York, NY, USA","reference-count":53,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,6,28]],"date-time":"2021-06-28T00:00:00Z","timestamp":1624838400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,6,28]]},"DOI":"10.1145\/3461778.3462147","type":"proceedings-article","created":{"date-parts":[[2021,6,28]],"date-time":"2021-06-28T20:26:48Z","timestamp":1624912008000},"page":"1458-1469","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Chatbot or Chat-Blocker: Predicting Chatbot Popularity before Deployment"],"prefix":"10.1145","author":[{"given":"Mingkun","family":"Gao","sequence":"first","affiliation":[{"name":"Department of Computer Science University of Illinois at Urbana-Champaign, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaotong","family":"Liu","sequence":"additional","affiliation":[{"name":"IBM Research - Almaden, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anbang","family":"Xu","sequence":"additional","affiliation":[{"name":"IBM Research - Almaden, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rama","family":"Akkiraju","sequence":"additional","affiliation":[{"name":"IBM Research - Almaden, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,6,28]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"2016. New DigitasLBi Research Shows More than 1 in 3 Americans are willing to make Purchases via Chatbots. digitas.com.  2016. New DigitasLBi Research Shows More than 1 in 3 Americans are willing to make Purchases via Chatbots. digitas.com."},{"key":"e_1_3_2_1_2_1","unstructured":"2017. Enterprises by business size. data.oecd.org.  2017. Enterprises by business size. data.oecd.org."},{"key":"e_1_3_2_1_3_1","unstructured":"2019. The Ultimate Guide for chatbot conversation flow. rakebots.com.  2019. The Ultimate Guide for chatbot conversation flow. rakebots.com."},{"key":"e_1_3_2_1_4_1","unstructured":"2020. 20+ Metrics for Chatbot Analytics in 2020: The Ultimate Guide. research.aimultiple.com.  2020. 20+ Metrics for Chatbot Analytics in 2020: The Ultimate Guide. research.aimultiple.com."},{"key":"e_1_3_2_1_5_1","unstructured":"2021. 8 Epic Chatbot\/Conversational Bot Failures. research.aimultiple.com.  2021. 8 Epic Chatbot\/Conversational Bot Failures. research.aimultiple.com."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISIICT.2011.6149594"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300484"},{"key":"e_1_3_2_1_8_1","unstructured":"Paul Boutin. 2017. Why Most Chatbots Fail. chatbotsmagazine.com.  Paul Boutin. 2017. Why Most Chatbots Fail. chatbotsmagazine.com."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR.2010.764"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025919"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300320"},{"key":"e_1_3_2_1_12_1","unstructured":"Mark Cirillo. 2019. 6 Examples of SMART Chatbot to Improve Your Customer Services. leadsbridge.com.  Mark Cirillo. 2019. 6 Examples of SMART Chatbot to Improve Your Customer Services. leadsbridge.com."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1515\/text-2014-0018"},{"key":"e_1_3_2_1_14_1","first-page":"158","article-title":"A tool of conversation","volume":"5","author":"Dahiya Menal","year":"2017","unstructured":"Menal Dahiya . 2017 . A tool of conversation : Chatbot. International Journal of Computer Sciences and Engineering 5 , 5(2017), 158 \u2013 161 . Menal Dahiya. 2017. A tool of conversation: Chatbot. International Journal of Computer Sciences and Engineering 5, 5(2017), 158\u2013161.","journal-title":"Chatbot. International Journal of Computer Sciences and Engineering"},{"key":"e_1_3_2_1_15_1","unstructured":"Edgar Dale and Jeanne\u00a0S Chall. 1948. A formula for predicting readability: Instructions. Educational research bulletin(1948) 37\u201354.  Edgar Dale and Jeanne\u00a0S Chall. 1948. A formula for predicting readability: Instructions. Educational research bulletin(1948) 37\u201354."},{"key":"e_1_3_2_1_16_1","unstructured":"Christopher Dossman. 2019. AI Scholar: Chatbots that improve after deployment. towardsdatascience.com.  Christopher Dossman. 2019. AI Scholar: Chatbots that improve after deployment. towardsdatascience.com."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8640.1992.tb00371.x"},{"key":"e_1_3_2_1_18_1","unstructured":"Akash Dubey. 2018. Feature Selection Using Random forest. towardsdatascience.com.  Akash Dubey. 2018. Feature Selection Using Random forest. towardsdatascience.com."},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3240925.3240965"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858535"},{"key":"e_1_3_2_1_21_1","first-page":"195","article-title":"A black-box approach for response quality evaluation of conversational agent systems","volume":"3","author":"Goh Ong\u00a0Sing","year":"2007","unstructured":"Ong\u00a0Sing Goh , Cemal Ardil , Wilson Wong , and Chun\u00a0Che Fung . 2007 . A black-box approach for response quality evaluation of conversational agent systems . International Journal of Computational Intelligence 3 , 3(2007), 195 \u2013 203 . https:\/\/doi.org\/10.5281\/zenodo.1076854 10.5281\/zenodo.1076854 Ong\u00a0Sing Goh, Cemal Ardil, Wilson Wong, and Chun\u00a0Che Fung. 2007. A black-box approach for response quality evaluation of conversational agent systems. International Journal of Computational Intelligence 3, 3(2007), 195\u2013203. https:\/\/doi.org\/10.5281\/zenodo.1076854","journal-title":"International Journal of Computational Intelligence"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300439"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1177\/002194366900600202"},{"key":"e_1_3_2_1_24_1","unstructured":"Lakisha Hall. 2018. 6 steps to successful conversational design. ibm.com.  Lakisha Hall. 2018. 6 steps to successful conversational design. ibm.com."},{"key":"e_1_3_2_1_25_1","volume-title":"2018 International Symposium on Educational Technology (ISET). IEEE, 52\u201356","author":"Ho Chan\u00a0Chun","year":"2018","unstructured":"Chan\u00a0Chun Ho , Ho\u00a0Lam Lee , Wing\u00a0Kwan Lo , and Kwok Fai\u00a0Andrew Lui . 2018 . Developing a chatbot for college student programme advisement . In 2018 International Symposium on Educational Technology (ISET). IEEE, 52\u201356 . https:\/\/doi.org\/10.1109\/ISET.2018.00021 10.1109\/ISET.2018.00021 Chan\u00a0Chun Ho, Ho\u00a0Lam Lee, Wing\u00a0Kwan Lo, and Kwok Fai\u00a0Andrew Lui. 2018. Developing a chatbot for college student programme advisement. In 2018 International Symposium on Educational Technology (ISET). IEEE, 52\u201356. https:\/\/doi.org\/10.1109\/ISET.2018.00021"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/IEEM.2017.8289883"},{"key":"e_1_3_2_1_27_1","volume-title":"RE-WOCHAT: Workshop on Collecting and Generating Resources for Chatbots and Conversational Agents-Development and Evaluation Workshop Programme (May 28 th","author":"Kaleem Mohammed","year":"2016","unstructured":"Mohammed Kaleem , Omar Alobadi , James O\u2019Shea , and Keeley Crockett . 2016 . Framework for the formulation of metrics for conversational agent evaluation . In RE-WOCHAT: Workshop on Collecting and Generating Resources for Chatbots and Conversational Agents-Development and Evaluation Workshop Programme (May 28 th , 2016). 20. Mohammed Kaleem, Omar Alobadi, James O\u2019Shea, and Keeley Crockett. 2016. Framework for the formulation of metrics for conversational agent evaluation. In RE-WOCHAT: Workshop on Collecting and Generating Resources for Chatbots and Conversational Agents-Development and Evaluation Workshop Programme (May 28 th, 2016). 20."},{"key":"e_1_3_2_1_28_1","volume-title":"Decisions with multiple objectives: preferences and value trade-offs","author":"Keeney L","unstructured":"Ralph\u00a0 L Keeney and Howard Raiffa . 1993. Decisions with multiple objectives: preferences and value trade-offs . Cambridge university press . https:\/\/doi.org\/10.1017\/CBO9781139174084 10.1017\/CBO9781139174084 Ralph\u00a0L Keeney and Howard Raiffa. 1993. Decisions with multiple objectives: preferences and value trade-offs. Cambridge university press. https:\/\/doi.org\/10.1017\/CBO9781139174084"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300316"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"crossref","unstructured":"J\u00a0Peter Kincaid Robert\u00a0P Fishburne\u00a0Jr Richard\u00a0L Rogers and Brad\u00a0S Chissom. 1975. Derivation of new readability formulas (automated readability index fog count and flesch reading ease formula) for navy enlisted personnel. (1975).  J\u00a0Peter Kincaid Robert\u00a0P Fishburne\u00a0Jr Richard\u00a0L Rogers and Brad\u00a0S Chissom. 1975. Derivation of new readability formulas (automated readability index fog count and flesch reading ease formula) for navy enlisted personnel. (1975).","DOI":"10.21236\/ADA006655"},{"key":"e_1_3_2_1_31_1","volume-title":"Commercial chatbot: Performance evaluation, usability metrics and quality standards of embodied conversational agents","author":"Kuligowska Karolina","year":"2015","unstructured":"Karolina Kuligowska . 2015. Commercial chatbot: Performance evaluation, usability metrics and quality standards of embodied conversational agents . Professionals Center for Business Research 2 ( 2015 ). Karolina Kuligowska. 2015. Commercial chatbot: Performance evaluation, usability metrics and quality standards of embodied conversational agents. Professionals Center for Business Research 2 (2015)."},{"key":"e_1_3_2_1_32_1","unstructured":"Justin Lee. 2018. The practical guide to chatbot metrics and analytics. blog.growthbot.org.  Justin Lee. 2018. The practical guide to chatbot metrics and analytics. blog.growthbot.org."},{"key":"e_1_3_2_1_33_1","first-page":"175","article-title":"Conversation modelling for chatbots: current approaches and future directions","volume":"2018","author":"McTear Michael","year":"2018","unstructured":"Michael McTear . 2018 . Conversation modelling for chatbots: current approaches and future directions . Studientexte zur Sprachkommunikation: Elektronische Sprachsignalverarbeitung 2018 (2018), 175 \u2013 185 . Michael McTear. 2018. Conversation modelling for chatbots: current approaches and future directions. Studientexte zur Sprachkommunikation: Elektronische Sprachsignalverarbeitung 2018 (2018), 175\u2013185.","journal-title":"Studientexte zur Sprachkommunikation: Elektronische Sprachsignalverarbeitung"},{"key":"e_1_3_2_1_34_1","volume-title":"Proceedings of the world congress on engineering, Vol.\u00a01. 1\u20138.","author":"Meira M","year":"2015","unstructured":"M Meira and AM\u00a0de\u00a0 P Canuto . 2015 . Evaluation of emotional agents\u2019 architectures: an approach based on quality metrics and the influence of emotions on users . In Proceedings of the world congress on engineering, Vol.\u00a01. 1\u20138. M Meira and AM\u00a0de\u00a0P Canuto. 2015. Evaluation of emotional agents\u2019 architectures: an approach based on quality metrics and the influence of emotions on users. In Proceedings of the world congress on engineering, Vol.\u00a01. 1\u20138."},{"key":"e_1_3_2_1_35_1","unstructured":"Isaac Oswalt. 2017. What Will a Chatbot Cost Me - And Is It Worth It?21handshake.com.  Isaac Oswalt. 2017. What Will a Chatbot Cost Me - And Is It Worth It?21handshake.com."},{"key":"e_1_3_2_1_36_1","unstructured":"David\u00a0J Pasta. 2009. Learning when to be discrete: continuous vs. categorical predictors. In SAS Global Forum Vol.\u00a0248.  David\u00a0J Pasta. 2009. Learning when to be discrete: continuous vs. categorical predictors. In SAS Global Forum Vol.\u00a0248."},{"key":"e_1_3_2_1_37_1","first-page":"2001","article-title":"Linguistic inquiry and word count: LIWC 2001. Mahway","volume":"71","author":"Pennebaker W","year":"2001","unstructured":"James\u00a0 W Pennebaker , Martha\u00a0 E Francis , and Roger\u00a0 J Booth . 2001 . Linguistic inquiry and word count: LIWC 2001. Mahway : Lawrence Erlbaum Associates 71 , 2001 (2001), 2001 . James\u00a0W Pennebaker, Martha\u00a0E Francis, and Roger\u00a0J Booth. 2001. Linguistic inquiry and word count: LIWC 2001. Mahway: Lawrence Erlbaum Associates 71, 2001 (2001), 2001.","journal-title":"Lawrence Erlbaum Associates"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3167132.3167362"},{"key":"e_1_3_2_1_39_1","unstructured":"Nicole\u00a0M Radziwill and Morgan\u00a0C Benton. 2017. Evaluating quality of chatbots and intelligent conversational agents. arXiv preprint arXiv:1704.04579(2017).  Nicole\u00a0M Radziwill and Morgan\u00a0C Benton. 2017. Evaluating quality of chatbots and intelligent conversational agents. arXiv preprint arXiv:1704.04579(2017)."},{"key":"e_1_3_2_1_40_1","unstructured":"Prateek Saxena. 2018. How Much Does it Cost to Develop A Chatbot. appinventiv.com.  Prateek Saxena. 2018. How Much Does it Cost to Develop A Chatbot. appinventiv.com."},{"key":"e_1_3_2_1_41_1","first-page":"19","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations). 60\u201365","author":"Sedoc Joao","year":"2019","unstructured":"Joao Sedoc , Daphne Ippolito , Arun Kirubarajan , Jai Thirani , Lyle Ungar , and Chris Callison-Burch . 2019 . Chateval: A tool for chatbot evaluation . In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations). 60\u201365 . https:\/\/doi.org\/10.18653\/v1\/N 19 - 4011 10.18653\/v1 Joao Sedoc, Daphne Ippolito, Arun Kirubarajan, Jai Thirani, Lyle Ungar, and Chris Callison-Burch. 2019. Chateval: A tool for chatbot evaluation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations). 60\u201365. https:\/\/doi.org\/10.18653\/v1\/N19-4011"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"crossref","unstructured":"Bayan\u00a0Abu Shawar and Eric Atwell. 2007. Chatbots: are they really useful?. In Ldv forum Vol.\u00a022. 29\u201349.  Bayan\u00a0Abu Shawar and Eric Atwell. 2007. Chatbots: are they really useful?. In Ldv forum Vol.\u00a022. 29\u201349.","DOI":"10.21248\/jlcl.22.2007.88"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.5555\/1556328.1556341"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1631\/FITEE.1700826"},{"key":"e_1_3_2_1_46_1","volume-title":"Applications of random forest feature selection for fine-scale genetic population assignment. Evolutionary applications 11, 2","author":"Sylvester VA","year":"2018","unstructured":"Emma\u00a0 VA Sylvester , Paul Bentzen , Ian\u00a0 R Bradbury , Marie Cl\u00e9ment , Jon Pearce , John Horne , and Robert\u00a0 G Beiko . 2018. Applications of random forest feature selection for fine-scale genetic population assignment. Evolutionary applications 11, 2 ( 2018 ), 153\u2013165. https:\/\/doi.org\/10.1111\/eva.12524 10.1111\/eva.12524 Emma\u00a0VA Sylvester, Paul Bentzen, Ian\u00a0R Bradbury, Marie Cl\u00e9ment, Jon Pearce, John Horne, and Robert\u00a0G Beiko. 2018. Applications of random forest feature selection for fine-scale genetic population assignment. Evolutionary applications 11, 2 (2018), 153\u2013165. https:\/\/doi.org\/10.1111\/eva.12524"},{"key":"e_1_3_2_1_47_1","volume-title":"On evaluating and comparing conversational agents. arXiv preprint arXiv:1801.03625 4","author":"Venkatesh Anu","year":"2018","unstructured":"Anu Venkatesh , Chandra Khatri , Ashwin Ram , Fenfei Guo , Raefer Gabriel , Ashish Nagar , Rohit Prasad , Ming Cheng , Behnam Hedayatnia , Angeliki Metallinou , 2018. On evaluating and comparing conversational agents. arXiv preprint arXiv:1801.03625 4 ( 2018 ), 60\u201368. Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Behnam Hedayatnia, Angeliki Metallinou, 2018. On evaluating and comparing conversational agents. arXiv preprint arXiv:1801.03625 4 (2018), 60\u201368."},{"key":"e_1_3_2_1_48_1","volume-title":"Software quality and software testing in internet times","author":"Vetter Michael","unstructured":"Michael Vetter . 2002. Quality aspects of bots . In Software quality and software testing in internet times . Springer , 165\u2013184. https:\/\/doi.org\/10.1007\/978-3-642-56333-1_11 10.1007\/978-3-642-56333-1_11 Michael Vetter. 2002. Quality aspects of bots. In Software quality and software testing in internet times. Springer, 165\u2013184. https:\/\/doi.org\/10.1007\/978-3-642-56333-1_11"},{"key":"e_1_3_2_1_49_1","volume-title":"PARADISE: A Framework for Evaluating Spoken Dialogue Agents. In 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European","author":"Walker A","year":"1997","unstructured":"Marilyn\u00a0 A Walker , Diane\u00a0 J Litman , Candace\u00a0 A Kamm , and Alicia Abella . 1997 . PARADISE: A Framework for Evaluating Spoken Dialogue Agents. In 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics. 271\u2013280. https:\/\/doi.org\/10.3115\/976909.979652 10.3115\/976909.979652 Marilyn\u00a0A Walker, Diane\u00a0J Litman, Candace\u00a0A Kamm, and Alicia Abella. 1997. PARADISE: A Framework for Evaluating Spoken Dialogue Agents. In 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics. 271\u2013280. https:\/\/doi.org\/10.3115\/976909.979652"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/365153.365168"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025496"},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1162\/coli_a_00368"},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3232077"},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025800"}],"event":{"name":"DIS '21: Designing Interactive Systems Conference 2021","location":"Virtual Event USA","acronym":"DIS '21","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Designing Interactive Systems Conference 2021"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3461778.3462147","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3461778.3462147","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:17:34Z","timestamp":1750191454000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3461778.3462147"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,28]]},"references-count":53,"alternative-id":["10.1145\/3461778.3462147","10.1145\/3461778"],"URL":"https:\/\/doi.org\/10.1145\/3461778.3462147","relation":{},"subject":[],"published":{"date-parts":[[2021,6,28]]},"assertion":[{"value":"2021-06-28","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}