{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T13:26:59Z","timestamp":1790947619370,"version":"4.1.0"},"publisher-location":"New York, NY, USA","reference-count":47,"publisher":"ACM","license":[{"start":{"date-parts":[[2018,6,8]],"date-time":"2018-06-08T00:00:00Z","timestamp":1528416000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2018,6,8]]},"DOI":"10.1145\/3196709.3196735","type":"proceedings-article","created":{"date-parts":[[2018,6,11]],"date-time":"2018-06-11T12:36:20Z","timestamp":1528720580000},"page":"895-906","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":288,"title":["Evaluating and Informing the Design of Chatbots"],"prefix":"10.1145","author":[{"given":"Mohit","family":"Jain","sequence":"first","affiliation":[{"name":"IBM Research & University of Washington, Seattle, WA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pratyush","family":"Kumar","sequence":"additional","affiliation":[{"name":"IBM Research, Bangalore, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ramachandra","family":"Kota","sequence":"additional","affiliation":[{"name":"Realtor.com & IBM Research, Vancouver, BC, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shwetak N.","family":"Patel","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,6,8]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Foundation website. (2002). Retrieved","author":"E.","year":"2017","unstructured":"2002. A.L.I.C. E. Foundation website. (2002). Retrieved January 4, 2017 from http:\/\/alicebot.org 2002. A.L.I.C.E. Foundation website. (2002). Retrieved January 4, 2017 from http:\/\/alicebot.org"},{"key":"e_1_3_2_1_2_1","volume-title":"Retrieved","year":"2013","unstructured":"2013. Mitsuku. ( 2013 ). Retrieved January 4, 2017 from http:\/\/www.mitsuku.com 2013. Mitsuku. (2013). Retrieved January 4, 2017 from http:\/\/www.mitsuku.com"},{"key":"e_1_3_2_1_3_1","volume-title":"Retrieved","year":"2013","unstructured":"2013. Rose. ( 2013 ). Retrieved January 4, 2017 from http:\/\/brilligunderstanding.com\/rosedemo.html 2013. Rose. (2013). Retrieved January 4, 2017 from http:\/\/brilligunderstanding.com\/rosedemo.html"},{"key":"e_1_3_2_1_4_1","volume-title":"Facebook Messenger bots. (2016). Retrieved","year":"2016","unstructured":"2016. Facebook Messenger bots. (2016). Retrieved Dec 1, 2016 from https:\/\/chatbottle.co\/bots\/messenger 2016. Facebook Messenger bots. (2016). Retrieved Dec 1, 2016 from https:\/\/chatbottle.co\/bots\/messenger"},{"key":"e_1_3_2_1_5_1","volume-title":"Retrieved","year":"2017","unstructured":"2017. Facebook Messenger Alterra. (2017) . Retrieved January 30, 2017 from https:\/\/www.messenger.com\/t\/alterra.cc 2017. Facebook Messenger Alterra. (2017). Retrieved January 30, 2017 from https:\/\/www.messenger.com\/t\/alterra.cc"},{"key":"e_1_3_2_1_6_1","volume-title":"Retrieved","year":"2017","unstructured":"2017. Facebook Messenger Call of Duty. (2017) . Retrieved January 30, 2017 from https:\/\/www.messenger.com\/t\/CallofDuty 2017. Facebook Messenger Call of Duty. (2017). Retrieved January 30, 2017 from https:\/\/www.messenger.com\/t\/CallofDuty"},{"key":"e_1_3_2_1_7_1","volume-title":"Retrieved","year":"2017","unstructured":"2017. Facebook Messenger chatShopper. (2017) . Retrieved January 30, 2017 from https:\/\/www.messenger.com\/t\/chatShopper 2017. Facebook Messenger chatShopper. (2017). Retrieved January 30, 2017 from https:\/\/www.messenger.com\/t\/chatShopper"},{"key":"e_1_3_2_1_8_1","volume-title":"Retrieved","year":"2017","unstructured":"2017. Facebook Messenger CNN. (2017) . Retrieved January 30, 2017 from https:\/\/www.messenger.com\/t\/cnn 2017. Facebook Messenger CNN. (2017). Retrieved January 30, 2017 from https:\/\/www.messenger.com\/t\/cnn"},{"key":"e_1_3_2_1_9_1","volume-title":"Retrieved","year":"2017","unstructured":"2017. Facebook Messenger Hi Poncho. (2017) . Retrieved January 30, 2017 from https:\/\/www.messenger.com\/t\/hiponcho 2017. Facebook Messenger Hi Poncho. (2017). Retrieved January 30, 2017 from https:\/\/www.messenger.com\/t\/hiponcho"},{"key":"e_1_3_2_1_10_1","volume-title":"Retrieved","year":"2017","unstructured":"2017. Facebook Messenger Pandorabots. (2017) . Retrieved January 30, 2017 from https:\/\/www.messenger.com\/t\/chatbots.io 2017. Facebook Messenger Pandorabots. (2017). Retrieved January 30, 2017 from https:\/\/www.messenger.com\/t\/chatbots.io"},{"key":"e_1_3_2_1_11_1","volume-title":"Retrieved","year":"2017","unstructured":"2017. Facebook Messenger Swelly. (2017) . Retrieved January 30, 2017 from https:\/\/www.messenger.com\/t\/swell.bot 2017. Facebook Messenger Swelly. (2017). Retrieved January 30, 2017 from https:\/\/www.messenger.com\/t\/swell.bot"},{"key":"e_1_3_2_1_12_1","volume-title":"Retrieved","year":"2017","unstructured":"2017. Facebook Messenger Trivia Blast. (2017) . Retrieved January 30, 2017 from https:\/\/www.messenger.com\/t\/triviablast1 2017. Facebook Messenger Trivia Blast. (2017). Retrieved January 30, 2017 from https:\/\/www.messenger.com\/t\/triviablast1"},{"key":"e_1_3_2_1_13_1","volume-title":"Bickmore and Justine Cassell","author":"Timothy","year":"2005","unstructured":"Timothy W. Bickmore and Justine Cassell . 2005 . Social dialongue with embodied conversational agents. In Advances in natural multimodal dialogue systems. Springer , 23--54. Timothy W. Bickmore and Justine Cassell. 2005. Social dialongue with embodied conversational agents. In Advances in natural multimodal dialogue systems. Springer, 23--54."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1067860.1067867"},{"key":"e_1_3_2_1_15_1","volume-title":"Rudnicky","author":"Bohus Dan","year":"2003","unstructured":"Dan Bohus and Alexander I . Rudnicky . 2003 . Ravenclaw: dialog management using hierarchical task decomposition and an expectation agenda.. In INTERSPEECH. ISCA. Dan Bohus and Alexander I. Rudnicky. 2003. Ravenclaw: dialog management using hierarchical task decomposition and an expectation agenda.. In INTERSPEECH. ISCA."},{"key":"e_1_3_2_1_16_1","volume-title":"Conversation as Direct Manipulation: An Iconoclastic View. The Art of Human-Computer Interface Design","author":"Brennan Susan","year":"1990","unstructured":"Susan Brennan . 1990. Conversation as Direct Manipulation: An Iconoclastic View. The Art of Human-Computer Interface Design ( 1990 ). Susan Brennan. 1990. Conversation as Direct Manipulation: An Iconoclastic View. The Art of Human-Computer Interface Design (1990)."},{"key":"e_1_3_2_1_17_1","volume-title":"Embodied conversational agents","author":"Cassell Justine","unstructured":"Justine Cassell . 2000. Embodied conversational agents . MIT press . Justine Cassell. 2000. Embodied conversational agents. MIT press."},{"key":"e_1_3_2_1_18_1","volume-title":"Retrieved","author":"Chaykowski Kathleen","year":"2016","unstructured":"Kathleen Chaykowski . 2016 . More Than 11,000 Bots Are Now On Facebook Messenger. (2016) . Retrieved Dec 28, 2016 from http: \/\/www.forbes.com\/sites\/kathleenchaykowski\/2016\/07\/01\/ more-than-11000-bots-are-now-on-facebook-messenger\/ Kathleen Chaykowski. 2016. More Than 11,000 Bots Are Now On Facebook Messenger. (2016). Retrieved Dec 28, 2016 from http: \/\/www.forbes.com\/sites\/kathleenchaykowski\/2016\/07\/01\/ more-than-11000-bots-are-now-on-facebook-messenger\/"},{"key":"e_1_3_2_1_19_1","volume-title":"Facebook Messenger chief says platform's 34,000 chatbots are finally improving user experience. (2016). Retrieved","author":"Chris O' Brien","year":"2017","unstructured":"O' Brien Chris . 2016. Facebook Messenger chief says platform's 34,000 chatbots are finally improving user experience. (2016). Retrieved February 7, 2017 from http:\/\/venturebeat.com\/2016\/11\/11\/ facebook-messenger-chief-says-platforms-34000\/ -chatbots-are-finally-improving-user-experience\/ O' Brien Chris. 2016. Facebook Messenger chief says platform's 34,000 chatbots are finally improving user experience. (2016). Retrieved February 7, 2017 from http:\/\/venturebeat.com\/2016\/11\/11\/ facebook-messenger-chief-says-platforms-34000\/ -chatbots-are-finally-improving-user-experience\/"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985715"},{"key":"e_1_3_2_1_21_1","volume-title":"Retrieved","author":"Elimeliah Craig","year":"2016","unstructured":"Craig Elimeliah . 2016 . Why chatbots are replacing apps. (2016) . Retrieved January 20, 2017 from http:\/\/venturebeat.com\/2016\/08\/02\/ why-chatbots-are-replacing-apps\/ Craig Elimeliah. 2016. Why chatbots are replacing apps. (2016). Retrieved January 20, 2017 from http:\/\/venturebeat.com\/2016\/08\/02\/ why-chatbots-are-replacing-apps\/"},{"key":"e_1_3_2_1_22_1","volume-title":"Retrieved","year":"2017","unstructured":"Facebook. 2017 . Discover. (2017) . Retrieved May 31, 2017 from https:\/\/developers.facebook.com\/docs\/ messenger-platform\/discover Facebook. 2017. Discover. (2017). Retrieved May 31, 2017 from https:\/\/developers.facebook.com\/docs\/ messenger-platform\/discover"},{"key":"e_1_3_2_1_23_1","volume-title":"The Current State of Chatbots","author":"Grech Matt","year":"2017","unstructured":"Matt Grech . 2017. The Current State of Chatbots in 2017 . (2017). Retrieved Jan 5, 2018 from https:\/\/getvoip.com\/blog\/2017\/04\/21\/ the-current-state-of-chatbots-in-2017\/ Matt Grech. 2017. The Current State of Chatbots in 2017. (2017). Retrieved Jan 5, 2018 from https:\/\/getvoip.com\/blog\/2017\/04\/21\/ the-current-state-of-chatbots-in-2017\/"},{"key":"e_1_3_2_1_24_1","volume-title":"Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in psychology 52","author":"Hart Sandra G","year":"1988","unstructured":"Sandra G Hart and Lowell E Staveland . 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in psychology 52 ( 1988 ), 139--183. Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in psychology 52 (1988), 139--183."},{"key":"e_1_3_2_1_25_1","volume-title":"First time bot users deserve good bots. (2017). Retrieved","author":"Hive Orange","year":"2018","unstructured":"Orange Hive . 2017. First time bot users deserve good bots. (2017). Retrieved Jan 5, 2018 from https:\/\/unfiltered.orangehive.de\/ first-time-bot-users-deserve-good-bots\/ Orange Hive. 2017. First time bot users deserve good bots. (2017). Retrieved Jan 5, 2018 from https:\/\/unfiltered.orangehive.de\/ first-time-bot-users-deserve-good-bots\/"},{"key":"e_1_3_2_1_26_1","volume-title":"How to pass the Turing test by cheating. School of Electrical, Electronic and Computer Engineering research report TR97-05","author":"Hutchens Jason L","unstructured":"Jason L Hutchens . 1996. How to pass the Turing test by cheating. School of Electrical, Electronic and Computer Engineering research report TR97-05 . Perth : University of Western Australia (1996) . Jason L Hutchens. 1996. How to pass the Turing test by cheating. School of Electrical, Electronic and Computer Engineering research report TR97-05. Perth: University of Western Australia (1996)."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174042"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741669"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/11550617_28"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2901790.2901842"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173577"},{"key":"e_1_3_2_1_32_1","volume-title":"IRE Transactions on Human Factors in Electronics HFE-1 (March","author":"Licklider J. C. R.","year":"1960","unstructured":"J. C. R. Licklider . 1960. IRE Transactions on Human Factors in Electronics HFE-1 (March 1960 ), 4--11. J. C. R. Licklider. 1960. IRE Transactions on Human Factors in Electronics HFE-1 (March 1960), 4--11."},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858288"},{"key":"e_1_3_2_1_34_1","volume-title":"The Design of Everyday Things","author":"Norman Donald A.","unstructured":"Donald A. Norman . 2002. The Design of Everyday Things . Basic Books, Inc. , New York, NY, USA . Donald A. Norman. 2002. The Design of Everyday Things. Basic Books, Inc., New York, NY, USA."},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207684"},{"key":"e_1_3_2_1_36_1","volume-title":"Traum","author":"Robinson Susan","year":"2010","unstructured":"Susan Robinson , Antonio Roque , and David R . Traum . 2010 . Dialogues in Context : An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue. In 7th International Conference on Language Resources and Evaluation (LREC). Valletta, Malta . http:\/\/people.ict. usc.edu\/~traum\/Papers\/Robinson-LREC2010.pdf Susan Robinson, Antonio Roque, and David R. Traum. 2010. Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue. In 7th International Conference on Language Resources and Evaluation (LREC). Valletta, Malta. http:\/\/people.ict. usc.edu\/~traum\/Papers\/Robinson-LREC2010.pdf"},{"key":"e_1_3_2_1_37_1","volume-title":"Language Resources and Evaluation Conference (LREC). Marrakech (Morocco). http:\/\/people.ict.usc.edu\/~traum\/Papers\/ Blackwell-LREC08","author":"Robinson Susan","year":"2008","unstructured":"Susan Robinson , David R. Traum , Midhun Ittycheriah , and Joe Henderer . 2008 . What would you ask a conversational agent? Observations of Human-Agent dialogues in a museum setting . In Language Resources and Evaluation Conference (LREC). Marrakech (Morocco). http:\/\/people.ict.usc.edu\/~traum\/Papers\/ Blackwell-LREC08 .pdf Susan Robinson, David R. Traum, Midhun Ittycheriah, and Joe Henderer. 2008. What would you ask a conversational agent? Observations of Human-Agent dialogues in a museum setting. In Language Resources and Evaluation Conference (LREC). Marrakech (Morocco). http:\/\/people.ict.usc.edu\/~traum\/Papers\/ Blackwell-LREC08.pdf"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/384076.384085"},{"key":"e_1_3_2_1_39_1","unstructured":"Bayan Abu Shawar and Eric Atwell. 2002. A comparison between ALICE and Elizabeth chatbot systems. (2002).  Bayan Abu Shawar and Eric Atwell. 2002. A comparison between ALICE and Elizabeth chatbot systems. (2002)."},{"key":"e_1_3_2_1_40_1","volume-title":"Most popular global mobile messenger apps as of","year":"2017","unstructured":"Statista. 2017. Most popular global mobile messenger apps as of January 2017 . (2017). Retrieved February 7, 2017 from https:\/\/www.statista.com\/statistics\/258749\/ most-popular-global-mobile-messenger-apps\/ Statista. 2017. Most popular global mobile messenger apps as of January 2017. (2017). Retrieved February 7, 2017 from https:\/\/www.statista.com\/statistics\/258749\/ most-popular-global-mobile-messenger-apps\/"},{"key":"e_1_3_2_1_41_1","volume-title":"Proceedings. 3rd Asia Pacific Computer Human Interaction (Cat. No.98EX110)","author":"Suzuki N.","unstructured":"N. Suzuki , K. Ishii , and M. Okada . 1998. Talking Eye: autonomous creature as accomplice for human . In Proceedings. 3rd Asia Pacific Computer Human Interaction (Cat. No.98EX110) . 409--414. N. Suzuki, K. Ishii, and M. Okada. 1998. Talking Eye: autonomous creature as accomplice for human. In Proceedings. 3rd Asia Pacific Computer Human Interaction (Cat. No.98EX110). 409--414."},{"key":"e_1_3_2_1_42_1","volume-title":"Proceedings of the International Conference on Human-Computer Interaction (HCI) (INTERACT '17)","author":"Thies Indrani M","year":"2017","unstructured":"Indrani M Thies , Nandita Menon , Sneha Magapu , Manisha Subramony , and Jacki O'Neill . 2017 . How do you want your chatbot? An exploratory Wizard-of-Oz study with young, urban Indians . In Proceedings of the International Conference on Human-Computer Interaction (HCI) (INTERACT '17) . IFIP, 20. Indrani M Thies, Nandita Menon, Sneha Magapu, Manisha Subramony, and Jacki O'Neill. 2017. How do you want your chatbot? An exploratory Wizard-of-Oz study with young, urban Indians. In Proceedings of the International Conference on Human-Computer Interaction (HCI) (INTERACT '17). IFIP, 20."},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"crossref","unstructured":"Marilyn A. Walker John S. Aberdeen Julie E. Boland Elizabeth Owen Bratt John S. Garofolo Lynette Hirschman Audrey N. Le Sungbok Lee Shrikanth S. Narayanan Kishore Papineni Bryan L. Pellom Joseph Polifroni Alexandros Potamianos P. Prabhu Alexander I. Rudnicky Gregory A. Sanders Stephanie Seneff David Stallard and Steve Whittaker. 2001. DARPA communicator dialog travel planning systems: the june 2000 data collection. In INTERSPEECH.  Marilyn A. Walker John S. Aberdeen Julie E. Boland Elizabeth Owen Bratt John S. Garofolo Lynette Hirschman Audrey N. Le Sungbok Lee Shrikanth S. Narayanan Kishore Papineni Bryan L. Pellom Joseph Polifroni Alexandros Potamianos P. Prabhu Alexander I. Rudnicky Gregory A. Sanders Stephanie Seneff David Stallard and Steve Whittaker. 2001. DARPA communicator dialog travel planning systems: the june 2000 data collection. In INTERSPEECH.","DOI":"10.21437\/Eurospeech.2001-355"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/365153.365168"},{"key":"e_1_3_2_1_45_1","volume-title":"Young","author":"Wen Tsung-Hsien","year":"2015","unstructured":"Tsung-Hsien Wen , Milica Gasic , Nikola Mrksic , Pei-hao Su, David Vandyke , and Steve J . Young . 2015 . Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems. CoRR abs\/1508.01745 (2015). http:\/\/arxiv.org\/abs\/1508.01745 Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-hao Su, David Vandyke, and Steve J. Young. 2015. Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems. CoRR abs\/1508.01745 (2015). http:\/\/arxiv.org\/abs\/1508.01745"},{"key":"e_1_3_2_1_46_1","volume-title":"Psychological, Ethical, and Design Issues","author":"Wilks Yorick","unstructured":"Yorick Wilks . 2010. Close Engagements with Artificial Companions: Key Social , Psychological, Ethical, and Design Issues . John Benjamins Publishing Company , Amsterdam . Yorick Wilks. 2010. Close Engagements with Artificial Companions: Key Social, Psychological, Ethical, and Design Issues. John Benjamins Publishing Company, Amsterdam."},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1109\/79.536824","article-title":"A review of large-vocabulary continuous-speech","volume":"13","author":"Young Steve","year":"1996","unstructured":"Steve Young . 1996 . A review of large-vocabulary continuous-speech . IEEE Signal Processing Magazine 13 , 5 (Sept 1996), 45--. Steve Young. 1996. A review of large-vocabulary continuous-speech. IEEE Signal Processing Magazine 13, 5 (Sept 1996), 45--.","journal-title":"IEEE Signal Processing Magazine"}],"event":{"name":"DIS '18: Designing Interactive Systems Conference 2018","location":"Hong Kong China","acronym":"DIS '18","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the 2018 Designing Interactive Systems Conference"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3196709.3196735","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3196709.3196735","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T01:08:40Z","timestamp":1750208920000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3196709.3196735"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,6,8]]},"references-count":47,"alternative-id":["10.1145\/3196709.3196735","10.1145\/3196709"],"URL":"https:\/\/doi.org\/10.1145\/3196709.3196735","relation":{},"subject":[],"published":{"date-parts":[[2018,6,8]]},"assertion":[{"value":"2018-06-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}