{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,9]],"date-time":"2026-04-09T12:12:43Z","timestamp":1775736763541,"version":"3.50.1"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2022,8,4]],"date-time":"2022-08-04T00:00:00Z","timestamp":1659571200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,8,4]],"date-time":"2022-08-04T00:00:00Z","timestamp":1659571200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/V00784X\/1"],"award-info":[{"award-number":["EP\/V00784X\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/S035362\/1"],"award-info":[{"award-number":["EP\/S035362\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Pers Ubiquit Comput"],"published-print":{"date-parts":[[2023,4]]},"abstract":"<jats:title>Abstract\n<\/jats:title><jats:p>The Bot Usability Scale (BUS) is a standardised tool to assess and compare the satisfaction of users after interacting with chatbots to support the development of usable conversational systems. The English version of the 15-item BUS scale (BUS-15) was the result of an exploratory factorial analysis; a confirmatory factorial analysis tests the replicability of the initial model and further explores the properties of the scale aiming to optimise this tool seeking for the stability of the original model, the potential reduction of items, and testing multiple language versions of the scale. BUS-15 and the usability metrics for user experience (UMUX-LITE), used here for convergent validity purposes, were translated from English to Spanish, German, and Dutch. A total of 1292 questionnaires were completed in multiple languages; these were collected from 209 participants interacting with an overall pool of 26 chatbots. BUS-15 was acceptably reliable; however, a shorter and more reliable solution with 11 items (BUS-11) emerged from the data. The satisfaction ratings obtained with the translated version of BUS-11 were not significantly different from the original version in English, suggesting that the BUS-11 could be used in multiple languages. The results also suggested that the age of participants seems to affect the evaluation when using the scale, with older participants significantly rating the chatbots as less satisfactory, when compared to younger participants. In line with the expectations, based on reliability, BUS-11 positively correlates with UMUX-LITE scale. The new version of the scale (BUS-11) aims to facilitate the evaluation with chatbots, and its diffusion could help practitioners to compare the performances and benchmark chatbots during the product assessment stage. This tool could be a way to harmonise and enable comparability in the field of human and conversational agent interaction.<\/jats:p>","DOI":"10.1007\/s00779-022-01690-0","type":"journal-article","created":{"date-parts":[[2022,8,4]],"date-time":"2022-08-04T09:03:32Z","timestamp":1659603812000},"page":"317-330","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":45,"title":["A confirmatory factorial analysis of the Chatbot Usability Scale: a multilanguage validation"],"prefix":"10.1007","volume":"27","author":[{"given":"Simone","family":"Borsci","sequence":"first","affiliation":[]},{"given":"Martin","family":"Schmettow","sequence":"additional","affiliation":[]},{"given":"Alessio","family":"Malizia","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2122-8077","authenticated-orcid":false,"given":"Alan","family":"Chamberlain","sequence":"additional","affiliation":[]},{"given":"Frank","family":"van der Velde","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,8,4]]},"reference":[{"key":"1690_CR1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2019.103978","volume":"132","author":"AA Abd-Alrazaq","year":"2019","unstructured":"Abd-Alrazaq AA, Alajlani M, Alalwan AA, Bewick BM, Gardner P, Househ M (2019) An overview of the features of chatbots in mental health: a scoping review. Int J Med Informatics 132:103978","journal-title":"Int J Med Informatics"},{"key":"1690_CR2","doi-asserted-by":"publisher","DOI":"10.1016\/j.apergo.2019.103007","volume":"84","author":"S Borsci","year":"2020","unstructured":"Borsci S, Buckle P, Walne S (2020) Is the LITE version of the usability metric for user experience (UMUX-LITE) a reliable tool to support rapid assessment of new healthcare technology? Appl Ergon 84:103007. https:\/\/doi.org\/10.1016\/j.apergo.2019.103007","journal-title":"Appl Ergon"},{"issue":"8","key":"1690_CR3","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1080\/10447318.2015.1064648","volume":"31","author":"S Borsci","year":"2015","unstructured":"Borsci S, Federici S, Bacci S, Gnaldi M, Bartolucci F (2015) Assessing user satisfaction in the era of user experience: comparison of the SUS, UMUX, and UMUX-LITE as a function of product experience. Int J Human-Comput Inter 31(8):484\u2013495. https:\/\/doi.org\/10.1080\/10447318.2015.1064648","journal-title":"International Journal of Human-Computer Interaction"},{"issue":"5","key":"1690_CR4","doi-asserted-by":"publisher","first-page":"519","DOI":"10.1080\/0144929X.2018.1541255","volume":"38","author":"S Borsci","year":"2019","unstructured":"Borsci S, Federici S, Malizia A, De Filippis ML (2019) Shaking the usability tree: why usability is not a dead end, and a constructive way forward. Behav Inform Technol 38(5):519\u2013532. https:\/\/doi.org\/10.1080\/0144929X.2018.1541255","journal-title":"Behaviour & Information Technology"},{"key":"1690_CR5","doi-asserted-by":"publisher","DOI":"10.1007\/s00779-021-01582-9","author":"S Borsci","year":"2021","unstructured":"Borsci S, Malizia A, Schmettow M, van der Velde F, Tariverdiyeva G, Balaji D, Chamberlain A (2021) The Chatbot Usability Scale: the design and pilot of a usability scale for interaction with AI-based conversational agents. Pers Ubiquit Comput. https:\/\/doi.org\/10.1007\/s00779-021-01582-9","journal-title":"Pers Ubiquit Comput"},{"key":"1690_CR6","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1007\/978-3-319-70284-1_30","volume-title":"International conference on internet science","author":"PB Brandtzaeg","year":"2017","unstructured":"Brandtzaeg PB, F\u00f8lstad A (2017) Why People Use Chatbots. In: Kompatsiaris I, Cave J, Satsiou A, Carle G, Passani A, Kontopoulos E, Diplaris S, McMillan D (eds) International conference on internet science. Springer International Publishing, pp 377\u2013392"},{"issue":"194","key":"1690_CR7","first-page":"4","volume":"189","author":"J Brooke","year":"1996","unstructured":"Brooke J (1996) SUS-A quick and dirty usability scale. Usabil Evaluat Indust 189(194):4\u20137","journal-title":"Usabil Evaluat Indust"},{"issue":"4","key":"1690_CR8","doi-asserted-by":"publisher","first-page":"584","DOI":"10.1037\/0022-006X.55.4.584","volume":"55","author":"DA Cole","year":"1987","unstructured":"Cole DA (1987) Utility of confirmatory factor analysis in test validation research. J Consult Clin Psychol 55(4):584","journal-title":"J Consult Clin Psychol"},{"issue":"1","key":"1690_CR9","first-page":"7","volume":"10","author":"AB Costello","year":"2005","unstructured":"Costello AB, Osborne J (2005) Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Pract Assess Res Eval 10(1):7","journal-title":"Pract Assess Res Eval"},{"issue":"5","key":"1690_CR10","doi-asserted-by":"publisher","first-page":"811","DOI":"10.1017\/S1351324916000243","volume":"22","author":"R Dale","year":"2016","unstructured":"Dale R (2016) The return of the chatbots. Nat Lang Eng 22(5):811\u2013817","journal-title":"Nat Lang Eng"},{"key":"1690_CR11","doi-asserted-by":"crossref","unstructured":"De Filippis ML, Federici S, Mele ML, Borsci S, Bracalenti M, Gaudino G, ..., Simonetti E (2020) Preliminary results of a systematic review: quality assessment of conversational agents (chatbots) for people with disabilities or special needs. Paper presented at the International Conference on Computers Helping People with Special Needs","DOI":"10.1007\/978-3-030-58796-3_30"},{"key":"1690_CR12","doi-asserted-by":"crossref","unstructured":"Dev J, Camp LJ (2020) User engagement with chatbots: a discursive psychology approach. Paper presented at the Proceedings of the 2nd Conference on Conversational User Interfaces","DOI":"10.1145\/3405755.3406165"},{"key":"1690_CR13","unstructured":"Dillon A (2001) Beyond usability: process, outcome and affect in human computer interactions. Can J Inform Library Sci 26(4)"},{"issue":"1","key":"1690_CR14","doi-asserted-by":"publisher","first-page":"126","DOI":"10.1037\/0033-2909.101.1.126","volume":"101","author":"WR Dillon","year":"1987","unstructured":"Dillon WR, Kumar A, Mulani N (1987) Offending estimates in covariance structure analysis: Comments on the causes of and solutions to Heywood cases. Psychol Bull 101(1):126","journal-title":"Psychol Bull"},{"issue":"3","key":"1690_CR15","doi-asserted-by":"publisher","first-page":"272","DOI":"10.1037\/1082-989X.4.3.272","volume":"4","author":"LR Fabrigar","year":"1999","unstructured":"Fabrigar LR, Wegener DT, MacCallum RC, Strahan EJ (1999) Evaluating the use of exploratory factor analysis in psychological research. Psychol Methods 4(3):272","journal-title":"Psychol Methods"},{"key":"1690_CR16","doi-asserted-by":"publisher","unstructured":"Federici S, de Filippis ML, Mele ML, Borsci S, Bracalenti M, Gaudino G, ..., Simonetti E (2020) Inside pandora\u2019s box: a systematic review of the assessment of the perceived quality of chatbots for people with disabilities or special needs. Disabil Rehabil Assist Technol 15(7):832-837. https:\/\/doi.org\/10.1080\/17483107.2020.1775313","DOI":"10.1080\/17483107.2020.1775313"},{"key":"1690_CR17","doi-asserted-by":"crossref","unstructured":"Federici S, Mele ML, Bracalenti M, De Filippis ML, Lanzilotti R, Desolda G, ... Simonetti E (2021) A Chatbot Solution for eGLU-Box Pro: The Usability Evaluation Platform for Italian Public Administrations. Paper presented at the Human-Computer Interaction. Theory, Methods and Tools, Cham","DOI":"10.1007\/978-3-030-78462-1_20"},{"issue":"5","key":"1690_CR18","doi-asserted-by":"publisher","first-page":"323","DOI":"10.1016\/j.intcom.2010.04.004","volume":"22","author":"K Finstad","year":"2010","unstructured":"Finstad K (2010) The usability metric for user experience. Interact Comput 22(5):323\u2013327","journal-title":"Interact Comput"},{"issue":"4","key":"1690_CR19","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1145\/3085558","volume":"24","author":"A F\u00f8lstad","year":"2017","unstructured":"F\u00f8lstad A, Brandtz\u00e6g PB (2017) Chatbots and the new world of HCI. Interactions 24(4):38\u201342","journal-title":"Interactions"},{"key":"1690_CR20","doi-asserted-by":"crossref","unstructured":"Fr\u00f8kj\u00e6r E, Hertzum M, Hornb\u00e6k K (2000) Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? Paper presented at the SIGCHI conference on Human Factors in Computing Systems, The Hague, The Netherlands","DOI":"10.1145\/332040.332455"},{"key":"1690_CR21","volume-title":"Multivariate data analysis","author":"JF Hair","year":"2010","unstructured":"Hair JF, Black WC, Babin BJ, Anderson RE, Tatham RL (2010) Multivariate data analysis, 7th edn. Prentice Hall, Upper Saddle River, New Jersey","edition":"7"},{"key":"1690_CR22","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2022.102788","volume":"161","author":"IKF Haugeland","year":"2022","unstructured":"Haugeland IKF, F\u00f8lstad A, Taylor C, Alexander C (2022) Understanding the user experience of customer service chatbots: an experimental study of chatbot interaction design. Int J Hum Comput Stud 161:102788. https:\/\/doi.org\/10.1016\/j.ijhcs.2022.102788","journal-title":"Int J Hum Comput Stud"},{"issue":"1","key":"1690_CR23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/10705519909540118","volume":"6","author":"LT Hu","year":"1999","unstructured":"Hu LT, Bentler PM (1999) Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives.\u00a0Struct Equ Modeling 6(1):1\u201355","journal-title":"Structural Equation Modeling: A Multidisciplinary Journal"},{"key":"1690_CR24","unstructured":"ISO (2019) ISO 9241\u2013210 Ergonomics of human-system interaction \u2014 Part 210: Human-centred design for interactive systems. In. Brussels, BE: CEN"},{"issue":"10","key":"1690_CR25","doi-asserted-by":"publisher","first-page":"785","DOI":"10.1145\/358413.358430","volume":"26","author":"B Ives","year":"1983","unstructured":"Ives B, Olson MH, Baroudi JJ (1983) The measurement of user information satisfaction. Commun ACM 26(10):785\u2013793","journal-title":"Commun ACM"},{"issue":"3","key":"1690_CR26","doi-asserted-by":"publisher","first-page":"486","DOI":"10.1177\/0049124114543236","volume":"44","author":"DA Kenny","year":"2015","unstructured":"Kenny DA, Kaniskan B, McCoach DB (2015) The performance of RMSEA in models with small degrees of freedom.\u00a0Sociol Methods Res\u00a044(3):486\u2013507","journal-title":"Sociological methods & research"},{"key":"1690_CR27","doi-asserted-by":"publisher","first-page":"001872081988131","DOI":"10.1177\/0018720819881312","volume":"63","author":"JR Lewis","year":"2019","unstructured":"Lewis JR (2019) Measuring user experience with 3, 5, 7, or 11 points: does it matter? Hum Factors 63:0018720819881312. https:\/\/doi.org\/10.1177\/0018720819881312","journal-title":"Hum Factors"},{"key":"1690_CR28","doi-asserted-by":"crossref","unstructured":"Lewis JR, Utesch BS, Maher DE (2013) UMUX-LITE: when there\u2019s no time for the SUS. Paper presented at the SIGCHI Conference on Human Factors in Computing Systems, Paris, France","DOI":"10.1145\/2470654.2481287"},{"key":"1690_CR29","doi-asserted-by":"crossref","unstructured":"Lindgaard G, Dudek C (2002) User satisfaction, aesthetics and usability: beyond reductionism. Paper presented at the IFIP 17th World Computer Congress - TC13 Stream on Usability: Gaining a Competitive Edge, Deventer, The Netherlands","DOI":"10.1007\/978-0-387-35610-5_16"},{"key":"1690_CR30","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1007\/978-3-319-32967-3_5","volume-title":"The conversational interface talking to smart devices","author":"MF McTear","year":"2016","unstructured":"McTear MF, Callejas Z, Griol D (2016) Speech input and output. In: McTear MF, Callejas Z, Griol D (eds) The conversational interface talking to smart devices. Springer, Switzerland, pp 75\u201392"},{"issue":"3","key":"1690_CR31","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1093\/iwc\/iwz022","volume":"31","author":"CB Nordheim","year":"2019","unstructured":"Nordheim CB, F\u00f8lstad A, Bj\u00f8rkli CA (2019) An initial model of trust in chatbots for customer service\u2014findings from a questionnaire study. Interact Comput 31(3):317\u2013335","journal-title":"Interact Comput"},{"issue":"1","key":"1690_CR32","first-page":"15","volume":"17","author":"JW Osborne","year":"2012","unstructured":"Osborne JW, Fitzpatrick DC (2012) Replication analysis in exploratory factor analysis: What it is and why it makes your analysis better. Pract Assess Res Eval 17(1):15","journal-title":"Pract Assess Res Eval"},{"key":"1690_CR33","doi-asserted-by":"crossref","unstructured":"Paikari E, van der Hoek A (2018) A framework for understanding chatbots and their future. Paper presented at the The 11th International Workshop on Cooperative and Human Aspects of Software Engineering, Gothenburg, Sweden","DOI":"10.1145\/3195836.3195859"},{"issue":"2","key":"1690_CR34","doi-asserted-by":"publisher","first-page":"33","DOI":"10.15346\/hc.v8i2.114","volume":"8","author":"M Portela","year":"2021","unstructured":"Portela M (2021) Interfacing participation in citizen science projects with conversational agents. Human Comput 8(2):33\u201353","journal-title":"Human Computation"},{"key":"1690_CR35","unstructured":"Sauro J (2017) Measuring Usability: From the SUS to the UMUX-Lite. measuringu.com. Retrieved from https:\/\/measuringu.com\/umux-lite\/"},{"key":"1690_CR36","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-46380-9","volume-title":"New statistics for design researchers","author":"M Schmettow","year":"2021","unstructured":"Schmettow M (2021) New statistics for design researchers. Springer International Publishing, Cham"},{"issue":"11","key":"1690_CR37","doi-asserted-by":"publisher","first-page":"1732","DOI":"10.1080\/10826084.2021.1949609","volume":"56","author":"Z Su","year":"2021","unstructured":"Su Z, Schneider JA, Young SD (2021) The role of conversational agents for substance use disorder in social distancing contexts. Subst Use Misuse 56(11):1732\u20131735. https:\/\/doi.org\/10.1080\/10826084.2021.1949609","journal-title":"Subst Use Misuse"},{"key":"1690_CR38","doi-asserted-by":"crossref","unstructured":"Val\u00e9rio FAM, Guimar\u00e3es TG, Prates RO, Candello H (2018) Chatbots explain themselves: designers\u2019 strategies for conveying chatbot features to users. J Interact Syst 9(3)","DOI":"10.5753\/jis.2018.710"},{"key":"1690_CR39","doi-asserted-by":"crossref","unstructured":"van der Goot MJ, Pilgrim T (2020) Exploring age differences in motivations for and acceptance of chatbot communication in a customer service context. Paper presented at the Chatbot Research and Design, Cham","DOI":"10.1007\/978-3-030-39540-7_12"},{"issue":"5","key":"1690_CR40","doi-asserted-by":"publisher","first-page":"777","DOI":"10.1177\/00131640121971518","volume":"61","author":"J-W Van Prooijen","year":"2001","unstructured":"Van Prooijen J-W, Van Der Kloot WA (2001) Confirmatory analysis of exploratively obtained factor structures. Educ Psychol Measur 61(5):777\u2013792","journal-title":"Educ Psychol Measur"},{"key":"1690_CR41","doi-asserted-by":"publisher","first-page":"84","DOI":"10.2307\/270754","volume":"8","author":"B Wheaton","year":"1977","unstructured":"Wheaton B, Muthen B, Alwin DF, Summers GF (1977) Assessing reliability and stability in panel models. Sociol Methodol 8:84\u2013136","journal-title":"Sociol Methodol"}],"container-title":["Personal and Ubiquitous Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00779-022-01690-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00779-022-01690-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00779-022-01690-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,4,19]],"date-time":"2023-04-19T10:10:57Z","timestamp":1681899057000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00779-022-01690-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,4]]},"references-count":41,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,4]]}},"alternative-id":["1690"],"URL":"https:\/\/doi.org\/10.1007\/s00779-022-01690-0","relation":{},"ISSN":["1617-4909","1617-4917"],"issn-type":[{"value":"1617-4909","type":"print"},{"value":"1617-4917","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,4]]},"assertion":[{"value":"4 October 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 July 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 August 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}