{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T05:43:37Z","timestamp":1775886217490,"version":"3.50.1"},"reference-count":87,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,7,21]],"date-time":"2021-07-21T00:00:00Z","timestamp":1626825600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,7,21]],"date-time":"2021-07-21T00:00:00Z","timestamp":1626825600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/T51729X\/1"],"award-info":[{"award-number":["EP\/T51729X\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Pers Ubiquit Comput"],"published-print":{"date-parts":[[2022,2]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Standardised tools to assess a user\u2019s satisfaction with the experience of using chatbots and conversational agents are currently unavailable. This work describes four studies, including a systematic literature review, with an overall sample of 141 participants in the survey (experts and novices), focus group sessions and testing of chatbots to (i) define attributes to assess the quality of interaction with chatbots and (ii) the designing and piloting a new scale to measure satisfaction after the experience with chatbots. Two instruments were developed: (i) A diagnostic tool in the form of a checklist (BOT-Check). This tool is a development of previous works which can be used reliably to check the quality of a chatbots experience in line with commonplace principles. (ii) A 15-item questionnaire (BOT Usability Scale, BUS-15) with estimated reliability between .76 and .87 distributed in five factors. BUS-15 strongly correlates with UMUX-LITE by enabling designers to consider a broader range of aspects usually not considered in satisfaction tools for non-conversational agents, e.g. conversational efficiency and accessibility, quality of the chatbot\u2019s functionality and so on. Despite the convincing psychometric properties, BUS-15 requires further testing and validation. Designers can use it as a tool to assess products, thus building independent databases for future evaluation of its reliability, validity and sensitivity.<\/jats:p>","DOI":"10.1007\/s00779-021-01582-9","type":"journal-article","created":{"date-parts":[[2021,7,21]],"date-time":"2021-07-21T11:14:27Z","timestamp":1626866067000},"page":"95-119","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":188,"title":["The Chatbot Usability Scale: the Design and Pilot of a Usability Scale for Interaction with AI-Based Conversational Agents"],"prefix":"10.1007","volume":"26","author":[{"given":"Simone","family":"Borsci","sequence":"first","affiliation":[]},{"given":"Alessio","family":"Malizia","sequence":"additional","affiliation":[]},{"given":"Martin","family":"Schmettow","sequence":"additional","affiliation":[]},{"given":"Frank","family":"van der Velde","sequence":"additional","affiliation":[]},{"given":"Gunay","family":"Tariverdiyeva","sequence":"additional","affiliation":[]},{"given":"Divyaa","family":"Balaji","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2122-8077","authenticated-orcid":false,"given":"Alan","family":"Chamberlain","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,7,21]]},"reference":[{"key":"1582_CR1","unstructured":"Amazon. (2018). Voice Design Checklist. Amazon. Retrieved 26th April from https:\/\/developer.amazon.com\/docs\/alexa-design\/checklists.html#voice-design-checklist"},{"key":"1582_CR2","volume-title":"New technologies and mixed-use convergence: How humans and algorithms are adapting to each other","author":"SA Applin","year":"2015","unstructured":"Applin SA, Fischer MD (2015) New technologies and mixed-use convergence: How humans and algorithms are adapting to each other. IEEE International Symposium on Technology and Society (ISTAS), Dublin"},{"issue":"8","key":"1582_CR3","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1080\/10447318.2015.1064648","volume":"31","author":"S Borsci","year":"2015","unstructured":"Borsci S, Federici S, Bacci S, Gnaldi M, Bartolucci F (2015) Assessing User Satisfaction in the Era of User Experience: Comparison of the SUS, UMUX, and UMUX-LITE as a Function of Product Experience. International Journal of Human\u2013Computer Interaction 31(8):484\u2013495. https:\/\/doi.org\/10.1080\/10447318.2015.1064648","journal-title":"International Journal of Human\u2013Computer Interaction"},{"key":"1582_CR4","doi-asserted-by":"crossref","unstructured":"Brandtzaeg, P. B., & F\u00f8lstad, A. (2017). Why people use chatbots International Conference on Internet Science","DOI":"10.1007\/978-3-319-70284-1_30"},{"issue":"5","key":"1582_CR5","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1145\/3236669","volume":"25","author":"PB Brandtzaeg","year":"2018","unstructured":"Brandtzaeg PB, F\u00f8lstad A (2018) Chatbots: changing user needs and motivations. Interactions 25(5):38\u201343","journal-title":"Interactions"},{"issue":"194","key":"1582_CR6","first-page":"4","volume":"189","author":"J Brooke","year":"1996","unstructured":"Brooke J (1996) SUS-A quick and dirty usability scale. Usability evaluation in industry 189(194):4\u20137","journal-title":"Usability evaluation in industry"},{"key":"1582_CR7","unstructured":"Businesswire. (2019). The Global Chatbot Market is Forecast to Reach $5.63 Billion by 2023 -- Asia Pacific to Witness the Highest Growth - ResearchAndMarkets.com. Businesswire. Retrieved 20th of August from https:\/\/www.businesswire.com\/news\/home\/20190314005364\/en\/Global-Chatbot-Market-Forecast-Reach-5.63-Billion"},{"issue":"7\/8","key":"1582_CR8","doi-asserted-by":"publisher","first-page":"811","DOI":"10.1108\/03090560210430818","volume":"36","author":"A Caruana","year":"2002","unstructured":"Caruana A (2002) Service loyalty. European Journal of Marketing 36(7\/8):811\u2013828. https:\/\/doi.org\/10.1108\/03090560210430818","journal-title":"European Journal of Marketing"},{"key":"1582_CR9","unstructured":"Charlton JI (2000) Nothing about us without us: Disability oppression and empowerment. Univ of California Press"},{"key":"1582_CR10","doi-asserted-by":"crossref","unstructured":"Chaves, A. P., & Gerosa, M. A. (2019). How should my chatbot interact? A survey on human-chatbot interaction design. arXiv preprint arXiv:1904.02743.","DOI":"10.1080\/10447318.2020.1841438"},{"key":"1582_CR11","doi-asserted-by":"crossref","unstructured":"Chopra, S., & Chivukula, S. (2017). My phone assistant should know I am an Indian: influencing factors for adoption of assistive agents Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, Vienna, Austria.","DOI":"10.1145\/3098279.3122137"},{"issue":"9","key":"1582_CR12","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1109\/MC.2017.3571053","volume":"50","author":"H Chung","year":"2017","unstructured":"Chung H, Iorga M, Voas J, Lee S (2017) Alexa, can I trust you? Computer 50(9):100\u2013104","journal-title":"Computer"},{"key":"1582_CR13","doi-asserted-by":"publisher","first-page":"587","DOI":"10.1016\/j.jbusres.2018.10.004","volume":"117","author":"M Chung","year":"2020","unstructured":"Chung M, Ko E, Joung H, Kim SJ (2020) Chatbot e-service and customer satisfaction regarding luxury brands. Journal of Business Research 117:587\u2013595. https:\/\/doi.org\/10.1016\/j.jbusres.2018.10.004","journal-title":"Journal of Business Research"},{"key":"1582_CR14","doi-asserted-by":"crossref","unstructured":"Cohen, D., & Lane, I. (2016). An oral exam for measuring a dialog system\u2019s capabilities 13th AAAI Conference on Artificial Intelligence, Phoenix, US.","DOI":"10.1609\/aaai.v30i1.10060"},{"issue":"5","key":"1582_CR15","doi-asserted-by":"publisher","first-page":"545","DOI":"10.1515\/text-2014-0018","volume":"34","author":"D Coniam","year":"2014","unstructured":"Coniam D (2014) The linguistic accuracy of chatbots: usability from an ESL perspective. Text & Talk 34(5):545\u2013567","journal-title":"Text & Talk"},{"issue":"1","key":"1582_CR16","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1016\/j.jeconom.2014.06.008","volume":"183","author":"G Conti","year":"2014","unstructured":"Conti G, Fr\u00fchwirth-Schnatter S, Heckman JJ, Piatek R (2014) Bayesian exploratory factor analysis. Journal of econometrics 183(1):31\u201357","journal-title":"Journal of econometrics"},{"issue":"5","key":"1582_CR17","doi-asserted-by":"publisher","first-page":"811","DOI":"10.1017\/S1351324916000243","volume":"22","author":"R Dale","year":"2016","unstructured":"Dale R (2016) The return of the chatbots. Natural Language Engineering 22(5):811\u2013817","journal-title":"Natural Language Engineering"},{"key":"1582_CR18","doi-asserted-by":"crossref","unstructured":"De Souza CS (2005) The semiotic engineering of human-computer interaction. MIT press","DOI":"10.7551\/mitpress\/6175.001.0001"},{"issue":"1","key":"1582_CR19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.2200\/S00173ED1V01Y200901HCI002","volume":"2","author":"CS De Souza","year":"2009","unstructured":"De Souza CS, Leit\u00e3o CF (2009) Semiotic engineering methods for scientific research in HCI. Synthesis Lectures on Human-Centered Informatics 2(1):1\u2013122","journal-title":"Synthesis Lectures on Human-Centered Informatics"},{"issue":"2","key":"1582_CR20","first-page":"9","volume":"4","author":"DC Derrick","year":"2013","unstructured":"Derrick DC, Meservy TO, Jenkins JL, Burgoon JK, Nunamaker JF Jr (2013) Detecting deceptive chat-based communication using typing behavior and message cues. ACM Transactions on Management Information Systems (TMIS) 4(2):9","journal-title":"ACM Transactions on Management Information Systems (TMIS)"},{"key":"1582_CR21","unstructured":"DeVellis, R. F. (2016). Scale development: Theory and applications (Vol. 26). Sage publications."},{"key":"1582_CR22","unstructured":"Dillon A (2001) Beyond Usability: Process, Outcome and Affect in human computer interactions. Canadian Journal of Information and Library Science 26(4)"},{"key":"1582_CR23","doi-asserted-by":"publisher","unstructured":"Federici, S., de Filippis, M. L., Mele, M. L., Borsci, S., Bracalenti, M., Gaudino, G., Cocco, A., Amendola, M., & Simonetti, E. (2020). Inside pandora\u2019s box: a systematic review of the assessment of the perceived quality of chatbots for people with disabilities or special needs. Disability and Rehabilitation: Assistive Technology, 1-6. https:\/\/doi.org\/10.1080\/17483107.2020.1775313,","DOI":"10.1080\/17483107.2020.1775313"},{"key":"1582_CR24","unstructured":"Feine, J., Morana, S., & Gnewuch, U. (2019). Measuring service encounter satisfaction with customer service chatbots using sentiment analysis 14th International Conference on Wirtschaftsinformatik (WI2019), Siegen, Germany. https:\/\/aisel.aisnet.org\/wi2019\/"},{"issue":"5","key":"1582_CR25","doi-asserted-by":"publisher","first-page":"323","DOI":"10.1016\/j.intcom.2010.04.004","volume":"22","author":"K Finstad","year":"2010","unstructured":"Finstad K (2010) The usability metric for user experience. Interacting with Computers 22(5):323\u2013327","journal-title":"Interacting with Computers"},{"issue":"2","key":"1582_CR26","doi-asserted-by":"publisher","DOI":"10.2196\/mental.7785","volume":"4","author":"KK Fitzpatrick","year":"2017","unstructured":"Fitzpatrick KK, Darcy A, Vierhile M (2017) Delivering Cognitive Behavior Therapy to Young Adults With Symptoms of Depression and Anxiety Using a Fully Automated Conversational Agent (Woebot): A Randomized Controlled Trial. JMIR Ment Health 4(2):e19. https:\/\/doi.org\/10.2196\/mental.7785","journal-title":"JMIR Ment Health"},{"issue":"4","key":"1582_CR27","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1145\/3085558","volume":"24","author":"A F\u00f8lstad","year":"2017","unstructured":"F\u00f8lstad A, Brandtz\u00e6g PB (2017) Chatbots and the new world of HCI. Interactions 24(4):38\u201342","journal-title":"Interactions"},{"key":"1582_CR28","doi-asserted-by":"crossref","unstructured":"F\u00f8lstad, A., Skjuve, M., & Brandtzaeg, P. B. (2019). Different Chatbots for Different Purposes: Towards a Typology of Chatbots to Understand Interaction Design International Conference on Internet Science, Cham.","DOI":"10.1007\/978-3-030-17705-8_13"},{"key":"1582_CR29","doi-asserted-by":"crossref","unstructured":"Fr\u00f8kj\u00e6r, E., Hertzum, M., & Hornb\u00e6k, K. (2000). Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? SIGCHI conference on Human Factors in Computing Systems, The Hague, The Netherlands.","DOI":"10.1145\/332040.332455"},{"key":"1582_CR30","doi-asserted-by":"crossref","unstructured":"Gaudiano, P., & Kater, K. (2000). ALife-WebGuide: an intelligent user interface for Web site navigation Proceedings of the 5th international conference on Intelligent user interfaces, New Orleans, Louisiana, USA.","DOI":"10.1145\/325737.325800"},{"key":"1582_CR31","unstructured":"Gnewuch, U., Morana, S., & Maedche, A. (2017). Towards designing cooperative and social conversational agents for customer service."},{"key":"1582_CR32","doi-asserted-by":"publisher","first-page":"304","DOI":"10.1016\/j.chb.2019.01.020","volume":"97","author":"E Go","year":"2019","unstructured":"Go E, Sundar SS (2019) Humanizing chatbots: The effects of visual, identity and conversational cues on humanness perceptions. Computers in Human Behavior 97:304\u2013316. https:\/\/doi.org\/10.1016\/j.chb.2019.01.020","journal-title":"Computers in Human Behavior"},{"key":"1582_CR33","unstructured":"Google. (2017). Learn about conversation. Google. Retrieved 13th of March from https:\/\/designguidelines.withgoogle.com\/conversation\/conversation-design\/learn-about-conversation.html#learn-about-conversation-the-cooperative-principle"},{"key":"1582_CR34","unstructured":"Harkous, H., Fawaz, K., Shin, K. G., & Aberer, K. (2016). Pribots: Conversational privacy with chatbots 12th Symposium on Usable Privacy and Security, Denver, Colorado, US."},{"issue":"1","key":"1582_CR35","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1080\/19312450709336664","volume":"1","author":"AF Hayes","year":"2007","unstructured":"Hayes AF, Krippendorff K (2007) Answering the call for a standard reliability measure for coding data. Communication methods and measures 1(1):77\u201389","journal-title":"Communication methods and measures"},{"issue":"5","key":"1582_CR36","doi-asserted-by":"publisher","first-page":"575","DOI":"10.1016\/S0953-5438(02)00023-1","volume":"14","author":"M Hertzum","year":"2002","unstructured":"Hertzum M, Andersen HH, Andersen V, Hansen CB (2002) Trust in information sources: seeking information from people, documents, and virtual agents. Interacting with Computers 14(5):575\u2013599","journal-title":"Interacting with Computers"},{"key":"1582_CR37","doi-asserted-by":"crossref","unstructured":"Hinrichs, H., & Le, N.-T. (2018). Which text-mining technique would detect most accurate user frustration in chats with conversational agents? 32nd International BCS Human Computer Interaction Conference, Belfast, United Kingdom.","DOI":"10.14236\/ewic\/HCI2018.211"},{"key":"1582_CR38","doi-asserted-by":"crossref","unstructured":"Hone, K. S., & Graham, R. (2000). Towards a tool for the subjective assessment of speech system interfaces (SASSI).","DOI":"10.1017\/S1351324900002497"},{"issue":"4","key":"1582_CR39","doi-asserted-by":"publisher","first-page":"537","DOI":"10.1177\/0013164417709314","volume":"78","author":"H Hoofs","year":"2018","unstructured":"Hoofs H, van de Schoot R, Jansen NWH, Kant I (2018) Evaluating Model Fit in Bayesian Confirmatory Factor Analysis With Large Samples: Simulation Study Introducing the BRMSEA. Educational and psychological measurement 78(4):537\u2013568. https:\/\/doi.org\/10.1177\/0013164417709314","journal-title":"Educational and psychological measurement"},{"key":"1582_CR40","doi-asserted-by":"crossref","unstructured":"Huang, C.-Y., & Ku, L.-W. (2018). EmotionPush: Emotion and Response Time Prediction Towards Human-Like Chatbots IEEE Global Communications Conference (GLOBECOM), Abu Dhabi, United Arab Emirates.","DOI":"10.1109\/GLOCOM.2018.8647331"},{"key":"1582_CR41","unstructured":"IBM Conversational UX (2018). Talk meets technology - Conversation design guidelines. IBM. Retrieved 2nd May from http:\/\/conversational-ux.mybluemix.net\/design\/conversational-ux\/practices\/"},{"key":"1582_CR42","doi-asserted-by":"crossref","unstructured":"Io, H. N., & Lee, C. B. (2017, 10-13 Dec. 2017). Chatbots and conversational agents: A bibliometric analysis. 2017 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM),","DOI":"10.1109\/IEEM.2017.8289883"},{"key":"1582_CR43","unstructured":"ISO. (2018). ISO 9241-11:2018 Ergonomic requirements for office work with visual display terminals \u2013 Part 11: Guidance on usability. In. Brussels, BE: CEN."},{"issue":"10","key":"1582_CR44","doi-asserted-by":"publisher","first-page":"785","DOI":"10.1145\/358413.358430","volume":"26","author":"B Ives","year":"1983","unstructured":"Ives B, Olson MH, Baroudi JJ (1983) The measurement of user information satisfaction. Communications of the ACM 26(10):785\u2013793","journal-title":"Communications of the ACM"},{"key":"1582_CR45","doi-asserted-by":"crossref","unstructured":"Jain, M., Kumar, P., Kota, R., & Patel, S. N. (2018). Evaluating and informing the design of chatbots Designing Interactive Systems Conference, Hong Kong.","DOI":"10.1145\/3196709.3196735"},{"key":"1582_CR46","doi-asserted-by":"publisher","unstructured":"Kirakowski J, O\u2019Donnell P, Yiu A (2009) Establishing the hallmarks of a convincing chatbot-human dialogue. In: Maurtua I (ed) Human-Computer Interaction. IntechOpen, pp 49\u201356. https:\/\/doi.org\/10.5772\/7741","DOI":"10.5772\/7741"},{"issue":"2","key":"1582_CR47","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18483\/PCBR.22","volume":"2","author":"K Kuligowska","year":"2015","unstructured":"Kuligowska K (2015) Commercial chatbot: Performance evaluation, usability metrics and quality standards of embodied conversational agents. Professionals Center for Business Research 2(2):1\u201316. https:\/\/doi.org\/10.18483\/PCBR.22","journal-title":"Professionals Center for Business Research"},{"key":"1582_CR48","doi-asserted-by":"publisher","first-page":"95","DOI":"10.1016\/j.ijhcs.2017.02.005","volume":"103","author":"S Lee","year":"2017","unstructured":"Lee S, Choi J (2017) Enhancing user experience with conversational agent for movie recommendation: Effects of self-disclosure and reciprocity. International Journal of Human-Computer Studies 103:95\u2013105. https:\/\/doi.org\/10.1016\/j.ijhcs.2017.02.005","journal-title":"International Journal of Human-Computer Studies"},{"key":"1582_CR49","doi-asserted-by":"crossref","unstructured":"Lemon, O., Gruenstein, A., Battle, A., & Peters, S. (2002). Multi-tasking and collaborative activities in dialogue systems 3rd SIGdial Workshop on Discourse and Dialogue, Philadelpia, US.","DOI":"10.3115\/1118121.1118137"},{"key":"1582_CR50","unstructured":"Lewis, J. R. (2001). Psychometric properties of the mean opinion scale. Proceedings of HCI International 2001: Usability Evaluation and Interface Design, 149-153."},{"key":"1582_CR51","doi-asserted-by":"publisher","first-page":"001872081988131","DOI":"10.1177\/0018720819881312","volume":"001872081988131","author":"JR Lewis","year":"2019","unstructured":"Lewis JR (2019) Measuring User Experience With 3, 5, 7, or 11 Points: Does It Matter? Human Factors, Online first 0018720819881312:001872081988131. https:\/\/doi.org\/10.1177\/0018720819881312","journal-title":"Human Factors, Online first"},{"issue":"3","key":"1582_CR52","doi-asserted-by":"publisher","first-page":"479","DOI":"10.1007\/s10772-015-9289-1","volume":"18","author":"JR Lewis","year":"2015","unstructured":"Lewis JR, Hardzinski ML (2015) Investigating the psychometric properties of the Speech User Interface Service Quality questionnaire. International Journal of Speech Technology 18(3):479\u2013487. https:\/\/doi.org\/10.1007\/s10772-015-9289-1","journal-title":"International Journal of Speech Technology"},{"key":"1582_CR53","doi-asserted-by":"crossref","unstructured":"Lewis, J. R., Utesch, B. S., & Maher, D. E. (2013). UMUX-LITE: when there's no time for the SUS SIGCHI Conference on Human Factors in Computing Systems, Paris, France.","DOI":"10.1145\/2470654.2481287"},{"key":"1582_CR54","doi-asserted-by":"crossref","unstructured":"Lindgaard, G., & Dudek, C. (2002). User Satisfaction, Aesthetics and Usability: Beyond Reductionism IFIP 17th World Computer Congress - TC13 Stream on Usability: Gaining a Competitive Edge, Deventer, The Netherlands.","DOI":"10.1007\/978-0-387-35610-5_16"},{"issue":"2","key":"1582_CR55","doi-asserted-by":"publisher","first-page":"12:12","DOI":"10.1145\/1985347.1985353","volume":"2","author":"DH Mcknight","year":"2011","unstructured":"Mcknight DH, Carter M, Thatcher JB, Clay PF (2011) Trust in a specific technology: An investigation of its components and measures. ACM Transactions on Management Information Systems (TMIS) 2(2):12:12\u201312:25. https:\/\/doi.org\/10.1145\/1985347.1985353","journal-title":"ACM Transactions on Management Information Systems (TMIS)"},{"key":"1582_CR56","doi-asserted-by":"publisher","unstructured":"McTear MF, Callejas Z, Griol D (2016) Speech Input and Output. In: McTear MF, Callejas Z, Griol D (eds) The Conversational Interface Talking to Smart Devices. Springer, pp 75\u201392. https:\/\/doi.org\/10.1007\/978-3-319-32967-3_5","DOI":"10.1007\/978-3-319-32967-3_5"},{"key":"1582_CR57","unstructured":"Meira, M., & Canuto, A. D. P. (2015). Evaluation of emotional agents' architectures: an approach based on quality metrics and the influence of emotions on users World congress on engineering (WCE 2015), London, UK."},{"issue":"5","key":"1582_CR58","doi-asserted-by":"publisher","first-page":"619","DOI":"10.1001\/jamainternmed.2016.0400","volume":"176","author":"AS Miner","year":"2016","unstructured":"Miner AS, Milstein A, Schueller S, Hegde R, Mangurian C, Linos E (2016) Smartphone-based conversational agents and responses to questions about mental health, interpersonal violence, and physical health. JAMA Internal Medicine 176(5):619\u2013625","journal-title":"JAMA Internal Medicine"},{"issue":"4","key":"1582_CR59","doi-asserted-by":"publisher","first-page":"264","DOI":"10.7326\/0003-4819-151-4-200908180-00135","volume":"151","author":"D Moher","year":"2009","unstructured":"Moher D, Liberati A, Tetzlaff J, Altman DG (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Annals of Internal Medicine 151(4):264\u2013269","journal-title":"Annals of Internal Medicine"},{"key":"1582_CR60","doi-asserted-by":"crossref","unstructured":"Morrissey, K., & Kirakowski, J. (2013). \u2018Realness\u2019 in Chatbots: Establishing Quantifiable Criteria International Conference on Human-Computer Interaction,","DOI":"10.1007\/978-3-642-39330-3_10"},{"key":"1582_CR61","unstructured":"Munteanu, C., & Boldea, M. (2000). MDWOZ: A Wizard of Oz Environment for Dialog Systems Development International Conference on Language Resources and Evaluation (LREC\u201900), Athens, Greece."},{"key":"1582_CR62","doi-asserted-by":"crossref","unstructured":"Paikari, E., & van der Hoek, A. (2018). A framework for understanding chatbots and their future The 11th International Workshop on Cooperative and Human Aspects of Software Engineering, Gothenburg, Sweden.","DOI":"10.1145\/3195836.3195859"},{"issue":"3","key":"1582_CR63","doi-asserted-by":"publisher","first-page":"115","DOI":"10.3109\/14015439.2013.810303","volume":"38","author":"S Pauletto","year":"2013","unstructured":"Pauletto S, Balentine B, Pidcock C, Jones K, Bottaci L, Aretoulaki M, Wells J, Mundy DP, Balentine J (2013) Exploring expressivity and emotion with artificial voice and speech technologies. Logopedics Phoniatrics Vocology 38(3):115\u2013125","journal-title":"Logopedics Phoniatrics Vocology"},{"key":"1582_CR64","unstructured":"Peras, D. (2018). Chatbot evaluation metrics. Economic and Social Development: Book of Proceedings, 89-97"},{"key":"1582_CR65","doi-asserted-by":"crossref","unstructured":"Pereira J, D\u00edaz \u00d3 (2019) What Matters for Chatbots? Analyzing Quality Measures for Facebook Messenger\u2019s 100 Most Popular Chatbots. In: Majchrzak TA, Mateos C, Poggi F, Gr\u00f8nli T-M (eds) Towards Integrated Web, Mobile, and IoT Technology. Springer International Publishing, pp 67\u201382","DOI":"10.1007\/978-3-030-28430-5_4"},{"key":"1582_CR66","unstructured":"Piatek, R. (2017). BayesFM: Bayesian Inference for Factor Modeling. R package version 0.1, 2."},{"key":"1582_CR67","doi-asserted-by":"crossref","unstructured":"Polkosky, M. D. (2008). Machines as mediators: The challenge of technology for interpersonal communication theory and research. In Mediated interpersonal communication (pp. 48-71). Routledge.","DOI":"10.4324\/9780203926864-11"},{"issue":"1","key":"1582_CR68","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1017\/S1351324908004919","volume":"15","author":"S Quarteroni","year":"2009","unstructured":"Quarteroni S, Manandhar S (2009) Designing an interactive open-domain question answering system. Natural Language Engineering 15(1):73\u201395","journal-title":"Natural Language Engineering"},{"key":"1582_CR69","unstructured":"Radziwill, N. M., & Benton, M. C. (2017). Evaluating quality of chatbots and intelligent conversational agents. arXiv preprint arXiv:1704.04579."},{"issue":"2012","key":"1582_CR70","first-page":"1","volume":"3","author":"W Revelle","year":"2011","unstructured":"Revelle W (2011) An overview of the psych package. Department of Psychology Northwestern University. Accessed on March 3(2012):1\u201325","journal-title":"Accessed on March"},{"key":"1582_CR71","unstructured":"Saenz, J., Burgess, W., Gustitis, E., Mena, A., & Sasangohar, F. (2017). The usability analysis of chatbot technologies for internal personnel communications Industrial and Systems Engineering Conference Pittsburgh, Pennsylvania, US."},{"key":"1582_CR72","doi-asserted-by":"crossref","unstructured":"Sankar, G. R., Greyling, J., Vogts, D., & Plessis, M. C. D. (2008). Models towards a hybrid conversational agent for contact centres Annual research conference of the South African Institute of Computer Scientists and Information Technologists on IT research in developing countries: riding the wave of technology, Wilderness, South Africa.","DOI":"10.1145\/1456659.1456683"},{"key":"1582_CR73","unstructured":"Sauro, J. (2017). Measuring usability: From the SUS to the UMUX-LITE. measuringU. Retrieved 3rd February from https:\/\/measuringu.com\/umux-lite\/"},{"key":"1582_CR74","unstructured":"Staven, T. (2017). What Makes a Good Bot or Not? unit4. Retrieved 15th of April from https:\/\/www.unit4.com\/blog\/2017\/03\/what-makes-a-good-bot-or-not"},{"key":"1582_CR75","unstructured":"Tay L, Jebb A (2017) Scale Creation. In: Rogelberg SG (ed) The SAGE encyclopedia of industrial and organizational psychology, 2nd edn. SAGE Publications, Inc."},{"issue":"10","key":"1582_CR76","doi-asserted-by":"publisher","DOI":"10.1111\/lnc3.12253","volume":"11","author":"C Thorne","year":"2017","unstructured":"Thorne C (2017) Chatbots for troubleshooting: A survey. Language and Linguistics Compass 11(10):e12253. https:\/\/doi.org\/10.1111\/lnc3.12253","journal-title":"Language and Linguistics Compass"},{"issue":"7","key":"1582_CR77","doi-asserted-by":"publisher","first-page":"456","DOI":"10.1177\/0706743719828977","volume":"64","author":"AN Vaidyam","year":"2019","unstructured":"Vaidyam AN, Wisniewski H, Halamka JD, Kashavan MS, Torous JB (2019) Chatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape. The Canadian Journal of Psychiatry 64(7):456\u2013464. https:\/\/doi.org\/10.1177\/0706743719828977","journal-title":"The Canadian Journal of Psychiatry"},{"key":"1582_CR78","doi-asserted-by":"crossref","unstructured":"Val\u00e9rio, F. A. M., Guimar\u00e3es, T. G., Prates, R. O., & Candello, H. (2017). Here's What I Can Do: Chatbots' Strategies to Convey Their Features to Users The XVI Brazilian Symposium on Human Factors in Computing Systems, Joinville, Brazil.","DOI":"10.1145\/3160504.3160544"},{"key":"1582_CR79","unstructured":"Van Eeuwen, M. (2017). Mobile conversational commerce: messenger chatbots as the next interface between businesses and consumers University of Twente]."},{"issue":"3","key":"1582_CR80","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1111\/jcc4.12066","volume":"19","author":"T Verhagen","year":"2014","unstructured":"Verhagen T, van Nes J, Feldberg F, van Dolen W (2014) Virtual Customer Service Agents: Using Social Presence and Personalization to Shape Online Service Encounters. Journal of Computer-Mediated Communication 19(3):529\u2013545. https:\/\/doi.org\/10.1111\/jcc4.12066","journal-title":"Journal of Computer-Mediated Communication"},{"key":"1582_CR81","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1007\/978-3-642-56333-1_11","volume-title":"Software Quality and Software Testing in Internet Times","author":"M Vetter","year":"2002","unstructured":"Vetter M (2002) Quality Aspects of Bots. In: Meyerhoff D, Laibarra B, van der Pouw Kraan R, Wallet A (eds) Software Quality and Software Testing in Internet Times. Springer, Berlin, pp 165\u2013184. https:\/\/doi.org\/10.1007\/978-3-642-56333-1_11"},{"key":"1582_CR82","doi-asserted-by":"crossref","unstructured":"Walker, M. A., Litman, D. J., Kamm, C. A., & Abella, A. (1997). PARADISE: A framework for evaluating spoken dialogue agents. arXiv preprint cmp-lg\/9704004.","DOI":"10.3115\/979617.979652"},{"key":"1582_CR83","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/978-3-642-32323-2_1","volume-title":"Believable Bots: Can Computers Play Like People?","author":"A Weiss","year":"2012","unstructured":"Weiss A, Tscheligi M (2012) Rethinking the Human\u2013Agent Relationship: Which Social Cues Do Interactive Agents Really Need to Have? In: Hingston P (ed) Believable Bots: Can Computers Play Like People? Springer, Berlin, pp 1\u201328. https:\/\/doi.org\/10.1007\/978-3-642-32323-2_1"},{"issue":"4","key":"1582_CR84","first-page":"14","volume":"58","author":"HJ Wilson","year":"2017","unstructured":"Wilson HJ, Daugherty P, Bianzino N (2017) The jobs that artificial intelligence will create. MIT Sloan Management Review 58(4):14","journal-title":"MIT Sloan Management Review"},{"key":"1582_CR85","doi-asserted-by":"crossref","unstructured":"Yang, F., Heeman, P. A., & Kun, A. (2008). Switching to real-time tasks in multi-tasking dialogue 22nd International Conference on Computational Linguistics, Manchester, UK.","DOI":"10.3115\/1599081.1599210"},{"key":"1582_CR86","unstructured":"Yang, X., Oikarinen, E.-L., & Saraniemi, S. (2020). Understanding Chatbot Service Encounters: Consumers\u2019 satisfactory and dissatisfactory experiences University of Oulu]. Oulu Business School."},{"key":"1582_CR87","doi-asserted-by":"crossref","unstructured":"Zamora, J. (2017). I\u2019m Sorry, Dave, I\u2019m Afraid I Can\u2019t Do That: Chatbot Perception and Expectations 5th International Conference on Human Agent Interaction, Bielefeld, Germany.","DOI":"10.1145\/3125739.3125766"}],"container-title":["Personal and Ubiquitous Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00779-021-01582-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00779-021-01582-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00779-021-01582-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,6]],"date-time":"2023-11-06T09:04:19Z","timestamp":1699261459000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00779-021-01582-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,21]]},"references-count":87,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,2]]}},"alternative-id":["1582"],"URL":"https:\/\/doi.org\/10.1007\/s00779-021-01582-9","relation":{},"ISSN":["1617-4909","1617-4917"],"issn-type":[{"value":"1617-4909","type":"print"},{"value":"1617-4917","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,21]]},"assertion":[{"value":"22 November 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 May 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 July 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}