{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T00:23:46Z","timestamp":1774398226397,"version":"3.50.1"},"reference-count":70,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,3,4]],"date-time":"2024-03-04T00:00:00Z","timestamp":1709510400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Comput. Sci."],"abstract":"<jats:p>User experience (UX) research relies heavily on survey scales to measure users' subjective experiences with technology. However, repeatedly raised concerns regarding the improper use of survey scales in UX research and adjacent fields call for a systematic review of current measurement practices. Therefore, we conducted a systematic literature review, screening 153 papers from four years of the ACM Conference on Human Factors in Computing Systems proceedings (ACM CHI 2019 to 2022), of which 60 were eligible empirical studies using survey scales to study users' experiences. We identified 85 different scales and 172 distinct constructs measured. Most scales were used once (70.59%), and most constructs were measured only once (66.28%). The System Usability Scale was the most popular scale, followed by the User Experience Questionnaire, and the NASA Task Load Index. Regarding constructs, usability was the most frequently measured, followed by attractiveness, effort, and presence. Furthermore, results show that papers rarely contained complete rationales for scale selection (20.00%) and seldom provided all scale items used (30.00%). More than a third of all scales were adapted (34.19%), while only one-third of papers reported any scale quality investigation (36.67%). On the basis of our results, we highlight questionable measurement practices in UX research and suggest opportunities to improve scale use for UX-related constructs. Additionally, we provide six recommended steps to promote enhanced rigor in following best practices for scale-based UX research.<\/jats:p>","DOI":"10.3389\/fcomp.2024.1368860","type":"journal-article","created":{"date-parts":[[2024,3,3]],"date-time":"2024-03-03T23:53:22Z","timestamp":1709510002000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":17,"title":["Measurement practices in user experience (UX) research: a systematic quantitative literature review"],"prefix":"10.3389","volume":"6","author":[{"given":"Sebastian A. C.","family":"Perrig","sequence":"first","affiliation":[]},{"given":"Lena Fanya","family":"Aeschbach","sequence":"additional","affiliation":[]},{"given":"Nicolas","family":"Scharowski","sequence":"additional","affiliation":[]},{"given":"Nick","family":"von Felten","sequence":"additional","affiliation":[]},{"given":"Klaus","family":"Opwis","sequence":"additional","affiliation":[]},{"given":"Florian","family":"Br\u00fchlmann","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2024,3,4]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.1145\/3474660","article-title":"Transparency in measurement reporting: a systematic literature review of CHI PLAY","author":"Aeschbach","year":"2021","journal-title":"Proceedings ACM Human-Computer Interaction 5(CHI PLAY)"},{"key":"B2","doi-asserted-by":"publisher","first-page":"452","DOI":"10.1038\/533452a","article-title":"1,500 scientists lift the lid on reproducibility","volume":"533","author":"Baker","year":"2016","journal-title":"Nature"},{"key":"B3","first-page":"2689","article-title":"Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '11","author":"Bargas-Avila","year":"2011"},{"key":"B4","first-page":"1","article-title":"A chinese-language validation of the video game demand scale (VGDS-C): measuring the cognitive, emotional, physical, and social demands of video games","volume-title":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI '21","author":"Bowman","year":"2021"},{"key":"B5","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1016\/0005-7916(94)90063-9","article-title":"Measuring emotion: the self-assessment manikin and the semantic differential","volume":"25","author":"Bradley","year":"1994","journal-title":"J. Behav. Ther. Exper. Psychiat"},{"key":"B6","first-page":"189","article-title":"SUS: a \u201cquick and dirty\u201d usability scale","volume":"189","author":"Brooke","year":"1996","journal-title":"Usab. Eval. Ind"},{"key":"B7","volume-title":"Confirmatory Factor Analysis for Applied Research","author":"Brown","year":"2015"},{"key":"B8","first-page":"981","article-title":"Local standards for sample size at CHI","volume-title":"Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI '16","author":"Caine","year":"2016"},{"key":"B9","volume-title":"Measuring Experiences, chapter 5","author":"Cairns","year":"2018"},{"key":"B10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3173574.3173715","article-title":"HARK no more: on the preregistration of CHI experiments","volume-title":"Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems","author":"Cockburn","year":"2018"},{"key":"B11","doi-asserted-by":"publisher","first-page":"297","DOI":"10.1007\/BF02310555","article-title":"Coefficient alpha and the internal structure of tests","volume":"16","author":"Cronbach","year":"1951","journal-title":"Psychometrika"},{"key":"B12","doi-asserted-by":"publisher","first-page":"982","DOI":"10.1287\/mnsc.35.8.982","article-title":"User acceptance of computer technology: a comparison of two theoretical models","volume":"35","author":"Davis","year":"1989","journal-title":"Manag. Sci"},{"key":"B13","volume-title":"Scale Development: Theory and Applications","author":"DeVellis","year":"2017"},{"key":"B14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1155\/2021\/5518722","article-title":"UX evaluation with standardized questionnaires in ubiquitous computing and ambient intelligence: a systematic literature review","volume":"2021","author":"D\u00edaz-Oreiro","year":"2021","journal-title":"Adv. Hum. Comput. Inter"},{"key":"B15","doi-asserted-by":"publisher","first-page":"399","DOI":"10.1111\/bjop.12046","article-title":"From alpha to omega: a practical solution to the pervasive problem of internal consistency estimation","volume":"105","author":"Dunn","year":"2014","journal-title":"Br. J. Psychol"},{"key":"B16","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511820724","volume-title":"The Measurement of Affect, Mood, and Emotion: A Guide for Health-Behavioral Research","author":"Ekkekakis","year":"2013"},{"key":"B17","doi-asserted-by":"publisher","first-page":"456","DOI":"10.1177\/2515245920952393","article-title":"Measurement schmeasurement: questionable measurement practices and how to avoid them","volume":"3","author":"Flake","year":"2020","journal-title":"Adv. Methods Pract. Psychol. Sci"},{"key":"B18","doi-asserted-by":"publisher","first-page":"370","DOI":"10.1177\/1948550617693063","article-title":"Construct validation in social and personality research: Current practice and recommendations","volume":"8","author":"Flake","year":"2017","journal-title":"Soc. Psychol. Person. Sci"},{"key":"B19","doi-asserted-by":"crossref","DOI":"10.4135\/9781446287866","volume-title":"Scale Construction and Psychometrics for Social and Personality Psychology","author":"Furr","year":"2011"},{"key":"B20","doi-asserted-by":"publisher","first-page":"366","DOI":"10.1080\/08919402.1907.10532551","article-title":"A history of the questionnaire method of research in psychology","volume":"14","author":"Gault","year":"1907","journal-title":"Pedag. Semin"},{"key":"B21","first-page":"1","article-title":"Isness: using multi-person VR to design peak mystical type experiences comparable to psychedelics","volume-title":"Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20","author":"Glowacki","year":"2020"},{"key":"B22","volume-title":"Google Scholar: Top publications on Human Computer Interaction","year":"2022"},{"key":"B23","first-page":"111","article-title":"Usability evaluation considered harmful (some of the time)","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '08","author":"Greenberg","year":"2008"},{"key":"B24","doi-asserted-by":"publisher","first-page":"904","DOI":"10.1177\/154193120605000909","article-title":"Nasa-task load index (NASA-TLX); 20 years later","author":"Hart","year":"2006","journal-title":"Proceedings of the Human Factors and Ergonomics Society Annual Meeting"},{"key":"B25","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1016\/S0166-4115(08)62386-9","article-title":"Development of NASA-TLX (task load index): results of empirical and theoretical research","author":"Hart","year":"1988","journal-title":"Human Mental Workload"},{"key":"B26","doi-asserted-by":"publisher","first-page":"481","DOI":"10.1207\/S15327590IJHC1304_07","article-title":"The effect of perceived hedonic quality on product appealingness","volume":"13","author":"Hassenzahl","year":"2001","journal-title":"Int. J. Hum. Comput. Inter"},{"key":"B27","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1007\/978-3-322-80058-9_19","article-title":"AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualit\u00e4t [AttrakDiff: A questionnaire for measuring perceived hedonic and pragmatic quality]","volume-title":"Mensch &Computer 2003: Interaktion in Bewegung","author":"Hassenzahl","year":"2003"},{"key":"B28","volume-title":"Educational and Psychological Measurement and Evaluation","author":"Hopkins","year":"1998"},{"key":"B29","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1016\/j.ijhcs.2005.06.002","article-title":"Current practice in measuring usability: challenges to usability studies and research","volume":"64","author":"Hornb\u00e6k","year":"2006","journal-title":"Int. J. Hum. Comput. Stud"},{"key":"B30","doi-asserted-by":"crossref","first-page":"617","DOI":"10.1145\/1240624.1240722","article-title":"Meta-analysis of correlations among usability measures","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '07","author":"Hornb\u00e6k","year":"2007"},{"key":"B31","doi-asserted-by":"publisher","first-page":"166","DOI":"10.1177\/2515245919882903","article-title":"Hidden invalidity among 15 commonly used measures in social and personality psychology","volume":"3","author":"Hussey","year":"2020","journal-title":"Adv. Methods Pract. Psychol. Sci"},{"key":"B32","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1207\/S15327566IJCE0401_04","article-title":"Foundations for an empirically determined scale of trust in automated systems","volume":"4","author":"Jian","year":"2000","journal-title":"Int. J. Cogn. Ergon"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.1037\/t07550-000","article-title":"Big Five Inventory (BFI) [Database record]","author":"John","year":"1991","journal-title":"APA PsycTests"},{"key":"B34","doi-asserted-by":"publisher","first-page":"1015","DOI":"10.1183\/09031936.00110209","article-title":"Validated questionnaires should not be modified","volume":"34","author":"Juniper","year":"2009","journal-title":"Eur. Respir. J"},{"key":"B35","volume-title":"Interpretation of Educational Measurements","author":"Kelley","year":"1927"},{"key":"B36","doi-asserted-by":"publisher","first-page":"441","DOI":"10.1007\/s11257-011-9118-4","article-title":"Explaining the user experience of recommender systems","volume":"22","author":"Knijnenburg","year":"2012","journal-title":"User Model. User-Adapt. Inter"},{"key":"B37","first-page":"1","article-title":"Negative emotions, positive outcomes? Exploring the communication of negativity in serious data stories","volume-title":"Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI '22","author":"Lan","year":"2022"},{"key":"B38","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"B39","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1007\/978-3-540-89350-9_6","article-title":"Construction and evaluation of a user experience questionnaire","volume-title":"HCI and Usability for Education and Work","author":"Laugwitz","year":"2008"},{"key":"B40","doi-asserted-by":"publisher","first-page":"526","DOI":"10.1016\/j.ijhcs.2013.09.006","article-title":"Attitudes towards user experience (UX) measurement","volume":"72","author":"Law","year":"2014","journal-title":"Int. J. Hum. Comput. Stud"},{"key":"B41","first-page":"1","article-title":"Obfuscation remedies harms arising from content flagging of photos","volume-title":"Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI '22","author":"Li","year":"2022"},{"key":"B42","first-page":"3553","article-title":"Chi 1994\u20132013: Mapping two decades of intellectual progress through co-word analysis","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '14","author":"Liu","year":"2014"},{"key":"B43","first-page":"3","article-title":"Measuring usability with the use questionnaire","volume":"8","author":"Lund","year":"2001","journal-title":"Usabil. Interf"},{"key":"B44","doi-asserted-by":"publisher","first-page":"323","DOI":"10.1177\/0092070303031003011","article-title":"The dangers of poor construct conceptualization","volume":"31","author":"MacKenzie","year":"2003","journal-title":"J. Acad. Market. Sci"},{"key":"B45","doi-asserted-by":"publisher","first-page":"48","DOI":"10.1080\/02701367.1989.10607413","article-title":"Psychometric properties of the intrinsic motivation inventory in a competitive sport setting: a confirmatory factor analysis","volume":"60","author":"McAuley","year":"1989","journal-title":"Res. Quart. Exer. Sport"},{"key":"B46","volume-title":"Test Theory: A Unified Treatment","author":"McDonald","year":"1999"},{"key":"B47","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1186\/2046-4053-4-1","article-title":"Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement","volume":"4","author":"Moher","year":"2015","journal-title":"System. Rev"},{"key":"B48","volume-title":"Psychometric Theory","author":"Nunnally","year":"1978"},{"key":"B49","doi-asserted-by":"publisher","first-page":"n71","DOI":"10.1136\/bmj.n71","article-title":"The PRISMA 2020 statement: an updated guideline for reporting systematic reviews","volume":"372","author":"Page","year":"2021","journal-title":"BMJ"},{"key":"B50","doi-asserted-by":"publisher","DOI":"10.17605\/OSF.IO\/DGQ6B","article-title":"Code book (CHI 19 - 22 coded) - Data from \u201cMeasurement practices in user experience UX research: A systematic quantitative literature review\u201d","author":"Perrig","year":"2023"},{"key":"B51","doi-asserted-by":"publisher","DOI":"10.31234\/osf.io\/3jz67","article-title":"Measurement practices in UX research: a systematic quantitative literature review","author":"Perrig","year":"2022","journal-title":"PsyArXiv"},{"key":"B52","first-page":"1","article-title":"A bermuda triangle? A review of method application and triangulation in user experience evaluation","volume-title":"Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI '18","author":"Pettersson","year":"2018"},{"key":"B53","doi-asserted-by":"publisher","first-page":"534","DOI":"10.1080\/07294360.2013.841651","article-title":"The benefits of publishing systematic quantitative literature reviews for PhD candidates and other early-career researchers","volume":"33","author":"Pickering","year":"2014","journal-title":"Higher Educ. Res. Dev"},{"key":"B54","doi-asserted-by":"publisher","first-page":"1756","DOI":"10.1080\/03075079.2014.914907","article-title":"Publishing not perishing: how research students transition from novice to knowledgeable using systematic quantitative literature reviews","volume":"40","author":"Pickering","year":"2015","journal-title":"Stud. Higher Educ"},{"key":"B55","volume-title":"R: A Language and Environment for Statistical Computing","year":"2019"},{"key":"B56","first-page":"46","article-title":"The one number you need to grow","volume":"81","author":"Reichheld","year":"2003","journal-title":"Harvard Bus. Rev"},{"key":"B57","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3313831.3376668","article-title":"Better because it's new: the impact of perceived novelty on the added value of mid-air haptic feedback","volume-title":"Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems","author":"Rutten","year":"2020"},{"key":"B58","first-page":"1","article-title":"Changes in research ethics, openness, and transparency in empirical studies between CHI 2017 and CHI 2022","volume-title":"Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23","author":"Salehzadeh Niksirat","year":"2023"},{"key":"B59","first-page":"1609","article-title":"Correlations among prototypical usability metrics: Evidence for the construct of usability","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '09","author":"Sauro","year":"2009"},{"key":"B60","doi-asserted-by":"publisher","first-page":"266","DOI":"10.1162\/105474601300343603","article-title":"The experience of presence: factor analytic insights","volume":"10","author":"Schubert","year":"2001","journal-title":"Presence"},{"key":"B61","doi-asserted-by":"publisher","first-page":"298","DOI":"10.1111\/j.1468-2958.2010.01377.x","article-title":"Personalization versus customization: the importance of agency, privacy, and power usage","volume":"36","author":"Sundar","year":"2010","journal-title":"Hum. Commun. Res"},{"key":"B62","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1080\/07370024.2017.1298038","article-title":"The usability construct: a dead end?","volume":"33","author":"Tractinsky","year":"2018","journal-title":"Hum. Comput. Inter"},{"key":"B63","first-page":"1","article-title":"Self-determination theory in HCI games research: current uses and open questions","volume-title":"Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20","author":"Tyack","year":"2020"},{"key":"B64","doi-asserted-by":"publisher","first-page":"102370","DOI":"10.1016\/j.ijhcs.2019.102370","article-title":"Development and validation of the player experience inventory: a scale to measure player experiences at the level of functional and psychosocial consequences","volume":"135","author":"Vanden Abeele","year":"2020","journal-title":"Int. J. Hum. Comput. Stud"},{"key":"B65","first-page":"1","article-title":"Transparency of CHI research artifacts: Results of a self-reported survey","volume-title":"Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20","author":"Wacharamanotham","year":"2020"},{"key":"B66","doi-asserted-by":"publisher","first-page":"1063","DOI":"10.1037\/0022-3514.54.6.1063","article-title":"Development and validation of brief measures of positive and negative affect: the PANAS scales","volume":"54","author":"Watson","year":"1988","journal-title":"J. Person. Soc. Psychol"},{"key":"B67","doi-asserted-by":"publisher","first-page":"225","DOI":"10.1162\/105474698565686","article-title":"Measuring presence in virtual environments: a presence questionnaire","volume":"7","author":"Witmer","year":"1998","journal-title":"Presence"},{"key":"B68","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1145\/2907069","article-title":"Research contributions in human-computer interaction","volume":"23","author":"Wobbrock","year":"2016","journal-title":"Interactions"},{"key":"B69","first-page":"1","article-title":"The trusted listener: the influence of anthropomorphic eye design of social robots on user's perception of trustworthiness","volume-title":"Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI '22","author":"Zhu","year":"2022"},{"key":"B70","volume-title":"The construction of a scale to measure perceived effort","author":"Zijlstra","year":"1985"}],"container-title":["Frontiers in Computer Science"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fcomp.2024.1368860\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,3]],"date-time":"2024-03-03T23:53:40Z","timestamp":1709510020000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fcomp.2024.1368860\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,4]]},"references-count":70,"alternative-id":["10.3389\/fcomp.2024.1368860"],"URL":"https:\/\/doi.org\/10.3389\/fcomp.2024.1368860","relation":{"has-preprint":[{"id-type":"doi","id":"10.31234\/osf.io\/3jz67","asserted-by":"object"}]},"ISSN":["2624-9898"],"issn-type":[{"value":"2624-9898","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,4]]},"article-number":"1368860"}}