{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T01:43:13Z","timestamp":1787017393537,"version":"3.56.0"},"publisher-location":"New York, NY, USA","reference-count":55,"publisher":"ACM","license":[{"start":{"date-parts":[[2024,5,20]],"date-time":"2024-05-20T00:00:00Z","timestamp":1716163200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100006374","name":"Coordena\u00e7\u00e3o de Aperfei\u00e7oamento de Pessoal de N\u00edvel Superior","doi-asserted-by":"publisher","award":["C\u00f3digo Financeiro 001"],"award-info":[{"award-number":["C\u00f3digo Financeiro 001"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100006374","name":"Funda\u00e7\u00e3o de Amparo \u00e0 Pesquisa do Estado do Rio Grande do Sul","doi-asserted-by":"publisher","award":["processo 22\/2551-0000606-0"],"award-info":[{"award-number":["processo 22\/2551-0000606-0"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,5,20]]},"DOI":"10.1145\/3658271.3658324","type":"proceedings-article","created":{"date-parts":[[2024,5,23]],"date-time":"2024-05-23T10:26:23Z","timestamp":1716459983000},"page":"1-11","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["What Do We Know About Usability Evaluation for Chatbots?: A Systematic Mapping Study"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-6489-6340","authenticated-orcid":false,"given":"Lucas Vieira de","family":"Carvalho","sequence":"first","affiliation":[{"name":"Federal University of Pampa (UNIPAMPA), Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6929-7557","authenticated-orcid":false,"given":"Pedro Henrique Dias","family":"Valle","sequence":"additional","affiliation":[{"name":"Universidade Federal de Juiz de Fora, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-9173-5764","authenticated-orcid":false,"given":"Bhruno Roan","family":"Leifheit","sequence":"additional","affiliation":[{"name":"Federal University of Pampa (UNIPAMPA), Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6008-6537","authenticated-orcid":false,"given":"Lu\u00eds Enrique Rivero","family":"Cabrejos","sequence":"additional","affiliation":[{"name":"Universidade Federal do Maranh\u00e3o, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5451-3109","authenticated-orcid":false,"given":"Walter","family":"Nakamura","sequence":"additional","affiliation":[{"name":"Universidade Tecnol\u00f3gica Federal do Paran\u00e1, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4979-5831","authenticated-orcid":false,"given":"Guilherme Corredato","family":"Guerino","sequence":"additional","affiliation":[{"name":"Universidade Estadual do Paran\u00e1, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-9811-6058","authenticated-orcid":false,"given":"Renato De Souza","family":"Garcia","sequence":"additional","affiliation":[{"name":"Federal University of Pampa (UNIPAMPA), Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1849-2675","authenticated-orcid":false,"given":"Williamson","family":"Silva","sequence":"additional","affiliation":[{"name":"Federal University of Pampa (UNIPAMPA), Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,5,23]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"2018. Ergonomics of human-system interaction \u2014 Part 11: Usability: Definitions and concepts."},{"key":"e_1_3_2_1_2_1","unstructured":"Sherif Abdelhamid and Andrew Katz. 2020. Using chatbots as smart teaching assistants for first-year engineering students. In 2020 First-Year Engineering Experience."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2018.10.006"},{"key":"e_1_3_2_1_4_1","volume-title":"Grand research challenges in information systems in brazil 2016\u20132026","author":"Ara\u00fajo Renata","year":"2017","unstructured":"Renata Ara\u00fajo and R Suzana. 2017. Grand research challenges in information systems in brazil 2016\u20132026. Brazilian Computer Society. Clodis Boscarioli Renata Araujo and Rita Suzana (2017), 2016\u20132026."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"crossref","unstructured":"Marcus Barbosa Walter\u00a0Takashi Nakamura Pedro Valle Guilherme\u00a0Corredato Guerino Alice\u00a0Fonseca Finger Gabriel\u00a0Machado Lunardi and Williamson Silva. 2022. UX of Chatbots: An Exploratory Study on Acceptance of User Experience Evaluation Methods.. In ICEIS (2). 355\u2013363.","DOI":"10.5220\/0011090100003179"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.2196\/18607"},{"key":"e_1_3_2_1_7_1","volume-title":"Use of chatbots for customer service in MSMEs. Applied Computing and Informatics","author":"Cordero Jorge","year":"2022","unstructured":"Jorge Cordero, Luis Barba-Guaman, and Franco Guam\u00e1n. 2022. Use of chatbots for customer service in MSMEs. Applied Computing and Informatics (2022)."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.csi.2018.12.003"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.3390\/info10120366"},{"key":"e_1_3_2_1_10_1","volume-title":"Usability assessment of conversational agents in healthcare: a literature review. Challenges of Trustable AI and Added-Value on Health","author":"Denecke Kerstin","year":"2022","unstructured":"Kerstin Denecke and Richard May. 2022. Usability assessment of conversational agents in healthcare: a literature review. Challenges of Trustable AI and Added-Value on Health (2022), 169\u2013173."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2976796.2976850"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2007.59"},{"key":"e_1_3_2_1_13_1","volume-title":"HCI 2021, Held as Part of the 23rd HCI International Conference, HCII 2021, Virtual Event, July 24\u201329, 2021, Proceedings, Part I 23","author":"Federici Stefano","year":"2021","unstructured":"Stefano Federici, Maria\u00a0Laura Mele, Marco Bracalenti, Maria\u00a0Laura De\u00a0Filippis, Rosa Lanzilotti, Giuseppe Desolda, Simone Borsci, Giancarlo Gaudino, Antonello Cocco, Massimo Amendola, 2021. A chatbot solution for eGLU-Box Pro: the usability evaluation platform for Italian public administrations. In Human-Computer Interaction. Theory, Methods and Tools: Thematic Area, HCI 2021, Held as Part of the 23rd HCI International Conference, HCII 2021, Virtual Event, July 24\u201329, 2021, Proceedings, Part I 23. Springer, 268\u2013279."},{"key":"e_1_3_2_1_14_1","volume-title":"Usability evaluation methods for the web: A systematic mapping study. Information and software Technology 53, 8","author":"Fernandez Adrian","year":"2011","unstructured":"Adrian Fernandez, Emilio Insfran, and Silvia Abrah\u00e3o. 2011. Usability evaluation methods for the web: A systematic mapping study. Information and software Technology 53, 8 (2011), 789\u2013817."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3342775.3342784"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICETA.2018.8572054"},{"key":"e_1_3_2_1_17_1","volume-title":"Calla Beauty Assistant: Beauty Advisory Chatbot. In 2021 8th International Conference on Advanced Informatics: Concepts, Theory and Applications (ICAICTA). IEEE, 1\u20136.","author":"Gambetta Zalikha\u00a0Adiera","year":"2021","unstructured":"Zalikha\u00a0Adiera Gambetta, Lestari\u00a0Dessi Puji, and Niwanputri\u00a0Ginar Santika. 2021. Calla Beauty Assistant: Beauty Advisory Chatbot. In 2021 8th International Conference on Advanced Informatics: Concepts, Theory and Applications (ICAICTA). IEEE, 1\u20136."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357074"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.5220\/0010450204630473"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3422392.3422421"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3335082.3335094"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-74796-3_58"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.2196\/25787"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09872-1"},{"key":"e_1_3_2_1_25_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction, Vol.\u00a02 (CSCW). 1\u201320","author":"Johnson B.","unstructured":"B. Johnson and et al.2018. Exploring Usability of Chatbots in Mobile Contexts: A Literature Review. In Proceedings of the ACM on Human-Computer Interaction, Vol.\u00a02 (CSCW). 1\u201320."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.3390\/a13090217"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2022.2074645"},{"key":"e_1_3_2_1_28_1","volume-title":"On the pragmatic design of literature studies in software engineering: an experience-based guideline. Empirical software engineering 22, 6","author":"Kuhrmann Marco","year":"2017","unstructured":"Marco Kuhrmann, Daniel\u00a0M\u00e9ndez Fern\u00e1ndez, and Maya Daneva. 2017. On the pragmatic design of literature studies in software engineering: an experience-based guideline. Empirical software engineering 22, 6 (2017), 2852\u20132891."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.3390\/jpm12050828"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10209-019-00666-x"},{"key":"e_1_3_2_1_31_1","volume-title":"Chatbot Market Size, Share and Global Market Forecast to","author":"Markets Marketsand","year":"2026","unstructured":"Marketsand Markets. 2020. Chatbot Market Size, Share and Global Market Forecast to 2026. COVID-19 Impact Analysis | MarketsandMarkets. Dispon\u00edvel em:https:\/\/www.marketsandmarkets.com\/Market-Reports\/smart-advisor-market-72302363.html, Consultado em 16 de outubro de 2023 (2020)."},{"key":"e_1_3_2_1_32_1","volume-title":"Consultado em 16 de outubro de 2023","author":"Markets Marketsand","year":"2021","unstructured":"Marketsand Markets. 2021. Grand View Research. Chatbot Market Growth & Trends Disponivel em:https:\/\/www.grandviewresearch.com\/industry-analysis\/chatbot-market, Consultado em 16 de outubro de 2023 (2021)."},{"key":"e_1_3_2_1_33_1","volume-title":"Web usability: Principles and evaluation methods. Web engineering","author":"Matera Maristella","year":"2006","unstructured":"Maristella Matera, Francesca Rizzo, and Giovanni\u00a0Toffetti Carughi. 2006. Web usability: Principles and evaluation methods. Web engineering (2006), 143\u2013180."},{"key":"e_1_3_2_1_34_1","unstructured":"Ligia Maza-Jimenez and Pablo Torres-Carri\u00f3n. 2020. Usability evaluation of a Conversational Artificial Intelligence Interface."},{"key":"e_1_3_2_1_35_1","volume-title":"Chatbot Research and Design: Third International Workshop, CONVERSATIONS","author":"McAllister Patrick","year":"2019","unstructured":"Patrick McAllister, James Kerr, Michael McTear, Maurice Mulvenna, Raymond Bond, Karen Kirby, Joseph Morning, and Danni Glover. 2020. Towards chatbots to support bibliotherapy preparation and delivery. In Chatbot Research and Design: Third International Workshop, CONVERSATIONS 2019, Amsterdam, The Netherlands, November 19\u201320, 2019, Revised Selected Papers 3. Springer, 127\u2013142."},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110607"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICOIACT53268.2021.9563926"},{"key":"e_1_3_2_1_38_1","volume-title":"Evaluating the Usability of Conversational Agents: A Systematic Literature Review. In 2021 IEEE 15th International Conference on Research Challenges in Information Science (RCIS). IEEE, 1\u201312","author":"Isabella","unstructured":"Isabella C.\u00a0R. Pereira and Raquel\u00a0O. Prates. 2021. Evaluating the Usability of Conversational Agents: A Systematic Literature Review. In 2021 IEEE 15th International Conference on Research Challenges in Information Science (RCIS). IEEE, 1\u201312."},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2015.03.007"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2015.03.007"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.4258\/hir.2020.26.4.255"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-77943-6_14"},{"key":"e_1_3_2_1_43_1","first-page":"1","article-title":"A import\u00e2ncia da qualidade dos chatbots para o sucesso dos neg\u00f3cios","volume":"1","year":"2023","unstructured":"Qualitor. 2023. A import\u00e2ncia da qualidade dos chatbots para o sucesso dos neg\u00f3cios. Qualitor Blog 1, 1 (2023), 1\u20135.","journal-title":"Qualitor Blog"},{"key":"e_1_3_2_1_44_1","volume-title":"Evaluating quality of chatbots and intelligent conversational agents. arXiv preprint arXiv:1704.04579","author":"Radziwill M","year":"2017","unstructured":"Nicole\u00a0M Radziwill and Morgan\u00a0C Benton. 2017. Evaluating quality of chatbots and intelligent conversational agents. arXiv preprint arXiv:1704.04579 (2017)."},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2021.102630"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"crossref","unstructured":"Ranci Ren John\u00a0W Castro and Silvia\u00a0Teresita Acu\u00f1a. 2021. Evaluation of Chatbots Usability Experimentation (S).. In SEKE. 281\u2013284.","DOI":"10.18293\/SEKE2021-053"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2022.3145323"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2021.104473"},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/SBES.2012.31"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1002\/eat.23798"},{"key":"e_1_3_2_1_51_1","volume-title":"Thomas\u00a0Akselberg Hatlebrekke, Nina Str\u00f8msvik, Anniken Hamang, Sigrid\u00a0Tronsli Hovland, Jill\u00a0Walker Rettberg, Vidar\u00a0M Steen, and Cathrine Bjorvatn.","author":"Siglen Elen","year":"2022","unstructured":"Elen Siglen, Hildegunn\u00a0H\u00f8berg Vetti, Aslaug Beathe\u00a0Forberg Lunde, Thomas\u00a0Akselberg Hatlebrekke, Nina Str\u00f8msvik, Anniken Hamang, Sigrid\u00a0Tronsli Hovland, Jill\u00a0Walker Rettberg, Vidar\u00a0M Steen, and Cathrine Bjorvatn. 2022. Ask Rosa\u2013The making of a digital genetic conversation tool, a chatbot, about hereditary breast and ovarian cancer. Patient education and counseling 105, 6 (2022), 1488\u20131494."},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.5220\/0005377701050113"},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSSA48701.2019.8985467"},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12205-020-2044-4"},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3501855"}],"event":{"name":"SBSI '24: XX Brazilian Symposium on Information Systems","location":"Juiz de Fora Brazil","acronym":"SBSI '24"},"container-title":["Proceedings of the 20th Brazilian Symposium on Information Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3658271.3658324","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3658271.3658324","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T13:07:19Z","timestamp":1755868039000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3658271.3658324"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,20]]},"references-count":55,"alternative-id":["10.1145\/3658271.3658324","10.1145\/3658271"],"URL":"https:\/\/doi.org\/10.1145\/3658271.3658324","relation":{},"subject":[],"published":{"date-parts":[[2024,5,20]]},"assertion":[{"value":"2024-05-23","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}