{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T23:35:55Z","timestamp":1784331355153,"version":"3.55.0"},"reference-count":51,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2021,5,17]],"date-time":"2021-05-17T00:00:00Z","timestamp":1621209600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Samsung Electronics of Amazonia Ltda","award":["Agreement n\u00ba 003\/2019"],"award-info":[{"award-number":["Agreement n\u00ba 003\/2019"]}]},{"name":"Coordination for the Improvement of Higher Education Personnel (CAPES)","award":["Financial code 001"],"award-info":[{"award-number":["Financial code 001"]}]},{"name":"Amazonas State Research Support Foundation (FAPEAM)","award":["Process 062.00150\/2020"],"award-info":[{"award-number":["Process 062.00150\/2020"]}]},{"name":"National Council for Scientific and Technological Development (CNPq)","award":["Processes 204081\/2018\u00ad1\/PDE and 314174\/2020-6"],"award-info":[{"award-number":["Processes 204081\/2018\u00ad1\/PDE and 314174\/2020-6"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The success of a software application is related to users\u2019 willingness to keep using it. In this sense, evaluating User eXperience (UX) became an important part of the software development process. Researchers have been carrying out studies by employing various methods to evaluate the UX of software products. Some studies reported varied and even contradictory results when applying different UX evaluation methods, making it difficult for practitioners to identify which results to rely upon. However, these works did not evaluate the developers\u2019 perspectives and their impacts on the decision process. Moreover, such studies focused on one-shot evaluations, which cannot assess whether the methods provide the same big picture of the experience (i.e., deteriorating, improving, or stable). This paper presents a longitudinal study in which 68 students evaluated the UX of an online judge system by employing AttrakDiff, UEQ, and Sentence Completion methods at three moments along a semester. This study reveals contrasting results between the methods, which affected developers\u2019 decisions and interpretations. With this work, we intend to draw the HCI community\u2019s attention to the contrast between different UX evaluation methods and the impact of their outcomes in the software development process.<\/jats:p>","DOI":"10.3390\/s21103480","type":"journal-article","created":{"date-parts":[[2021,5,17]],"date-time":"2021-05-17T05:45:13Z","timestamp":1621230313000},"page":"3480","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Are UX Evaluation Methods Providing the Same Big Picture?"],"prefix":"10.3390","volume":"21","author":[{"given":"Walter Takashi","family":"Nakamura","sequence":"first","affiliation":[{"name":"Institute of Computing (IComp), Federal University of Amazonas (UFAM), Avenida Rodrigo Ot\u00e1vio 6200, Manaus 69067-005, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Iftekhar","family":"Ahmed","sequence":"additional","affiliation":[{"name":"Department of Informatics, University of California Irvine (UCI), Irvine, CA 92697, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Redmiles","sequence":"additional","affiliation":[{"name":"Department of Informatics, University of California Irvine (UCI), Irvine, CA 92697, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Edson","family":"Oliveira","sequence":"additional","affiliation":[{"name":"Secretaria de Estado da Fazenda do Amazonas\u2013SEFAZ\/AM, Avenida Andre Araujo 150, Manaus 69060-000, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Fernandes","sequence":"additional","affiliation":[{"name":"Institute of Computing (IComp), Federal University of Amazonas (UFAM), Avenida Rodrigo Ot\u00e1vio 6200, Manaus 69067-005, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elaine H. T.","family":"de Oliveira","sequence":"additional","affiliation":[{"name":"Institute of Computing (IComp), Federal University of Amazonas (UFAM), Avenida Rodrigo Ot\u00e1vio 6200, Manaus 69067-005, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tayana","family":"Conte","sequence":"additional","affiliation":[{"name":"Institute of Computing (IComp), Federal University of Amazonas (UFAM), Avenida Rodrigo Ot\u00e1vio 6200, Manaus 69067-005, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,5,17]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Bargas-Avila, J.A., and Hornb\u00e6k, K. (2011, January 7\u201312). Old wine in new bottles or novel challenges: A critical analysis of empirical studies of user experience. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vancouver, BC, USA.","DOI":"10.1145\/1978942.1979336"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Law, E.L.C., Roto, V., Hassenzahl, M., Vermeeren, A.P.O.S., and Kort, J. (2009, January 4\u20139). Understanding, scoping and defining user experience: A survey approach. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Boston, MA, USA.","DOI":"10.1145\/1518701.1518813"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.ijhcs.2017.07.005","article-title":"The effects of interaction sequencing on user experience and preference","volume":"108","author":"Cockburn","year":"2017","journal-title":"Int. J. Human Comput. Stud."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Pettersson, I., Lachner, F., Frison, A.K., Riener, A., and Butz, A. (2018, January 21\u201326). A Bermuda triangle? A Review of method application and triangulation in user experience evaluation. Proceedings of the CHI Conference on Human Factors in Computing Systems, Montr\u00e9al, QC, Canada.","DOI":"10.1145\/3173574.3174035"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"542","DOI":"10.1016\/j.ijhcs.2013.10.004","article-title":"Investigating and promoting UX practice in industry: An experimental study","volume":"72","author":"Ardito","year":"2014","journal-title":"Int. J. Human Comput. Stud."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Rivero, L., and Conte, T. (2017, January 23\u201327). A systematic mapping study on research contributions on UX evaluation technologies. Proceedings of the XVI Brazilian Symposium on Human Factors in Computing Systems\u2014IHC 2017, Joinville, Brazil.","DOI":"10.1145\/3160504.3160512"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Marques, L.C., Nakamura, W.T., Valentim, N., Rivero, L., and Conte, T. (2018, January 1\u20133). Do scale type techniques identify problems that affect user experience? User experience evaluation of a mobile application (S). Proceedings of the the 30th International Conference on Software Engineering and Knowledge Engineering, Redwood, CA, USA.","DOI":"10.18293\/SEKE2018-161"},{"key":"ref_8","first-page":"104","article-title":"Are scale-based techniques enough for learners to convey their UX when using a learning management system?","volume":"27","author":"Nakamura","year":"2019","journal-title":"Rev. Bras. Inf. Educ."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"de Andrade Cardieri, G., and Zaina, L.M. (2018, January 22\u201326). Analyzing user experience in mobile web, native and progressive web applications: A User and HCI specialist perspectives. Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems-IHC 2018, Bel\u00e9m, Brazil.","DOI":"10.1145\/3274192.3274201"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1080\/10447318.2015.1064648","article-title":"Assessing user satisfaction in the era of user experience: Comparison of the SUS, UMUX, and UMUX-LITE as a function of product experience","volume":"31","author":"Borsci","year":"2015","journal-title":"Int. J. Human Comput. Interact."},{"key":"ref_11","unstructured":"Abascal, J., Barbosa, S., Fetter, M., Gross, T., Palanque, P., and Winckler, M. (2015). Mind the gap! Comparing Retrospective and concurrent ratings of emotion in user experience evaluation. Proceedings of the Human-Computer Interaction\u2014INTERACT 2015, Springer."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Soleimani, S., and Law, E.L.C. (2017, January 10\u201314). What can self-reports and acoustic data analyses on emotions tell us?. Proceedings of the 2017 Conference on Designing Interactive Systems, Edinburg, UK.","DOI":"10.1145\/3064663.3064770"},{"key":"ref_13","first-page":"22","article-title":"Lab testing beyond usability: Challenges and recommendations for assessing user experiences","volume":"12","author":"Lallemand","year":"2017","journal-title":"J. Usability Stud."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Szwillus, G., and Ziegler, J. (2003). AttrakDiff: Ein fragebogen zur messung wahrgenommener hedonischer und pragmatischer qualit\u00e4t. Mensch & Computer 2003, Springer.","DOI":"10.1007\/978-3-322-80058-9"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Holzinger, A. (2008). Construction and evaluation of a user experience questionnaire. HCI and Usability for Education and Work, Springer.","DOI":"10.1007\/978-3-540-89350-9"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1016\/j.intcom.2010.07.002","article-title":"An integrated model of interaction experience for information retrieval in a web-based encyclopaedia","volume":"23","author":"Ling","year":"2011","journal-title":"Interact. Comput."},{"key":"ref_17","unstructured":"Van der Heijden, H., and S\u00f8rensen, L.S. (2003, January 16\u201321). Measuring attitudes towards mobile information services: An empirical validation of the HED\/UT Scale. Proceedings of the ECIS, 2003, Naples, Italy."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Karapanos, E. (2013). Modeling Users\u2019 Experiences with Interactive Systems, Springer. [1st ed.].","DOI":"10.1007\/978-3-642-31000-3"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Kujala, S., Roto, V., V\u00e4\u00e4n\u00e4nen-Vainio-Mattila, K., and Sinnel\u00e4, A. (2011, January 22\u201325). Identifying hedonic factors in long-term user experience. Proceedings of the 2011 Conference on Designing Pleasurable Products and Interfaces\u2014DPPI 2011, Milano, Italy.","DOI":"10.1145\/2347504.2347523"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"659","DOI":"10.1002\/hfm.20583","article-title":"How user experience changes over time: A case study of social network services: How user experience changes over time","volume":"25","author":"Kim","year":"2015","journal-title":"Human Fact. Ergon. Manuf. Serv. Ind."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Harbich, S., and Hassenzahl, M. (2016). User experience in the work domain: A longitudinal field study. Interacti Comput.","DOI":"10.1093\/iwc\/iww022"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., and Winckler, M. (2011). User experience evaluation\u2014Which method to choose?. Human-Computer Interaction\u2014INTERACT 2011, Springer.","DOI":"10.1007\/978-3-642-23765-2"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Vermeeren, A.P.O.S., Law, E.L.C., Roto, V., Obrist, M., Hoonhout, J., and V\u00e4\u00e4n\u00e4nen-Vainio-Mattila, K. (2010, January 16\u201320). User experience evaluation methods: Current state and development needs. Proceedings of the 6th Nordic Conference on Human-Computer Interaction, Extending Boundaries, New York, NY, USA.","DOI":"10.1145\/1868914.1868973"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"D\u00edaz-Oreiro, I., L\u00f3pez, G., Quesada, L., and Guerrero, L.A. (2019). Standardized questionnaires for user experience evaluation: A Systematic literature review. Proceedings, 31.","DOI":"10.3390\/proceedings2019031014"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1080\/10447318.2015.1065696","article-title":"An exploration of the relation between expectations and user experience","volume":"31","author":"Michalco","year":"2015","journal-title":"Int. J. Human Comput. Interact."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Somrak, A., Poga\u010dnik, M., and Guna, J. (2021). Suitability and comparison of questionnaires assessing virtual reality-induced symptoms and effects and user experience in virtual environments. Sensors, 21.","DOI":"10.3390\/s21041185"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Burger, G., Guna, J., and Poga\u010dnik, M. (2018). Suitability of inexpensive eye-tracking device for user experience evaluations. Sensors, 18.","DOI":"10.3390\/s18061822"},{"key":"ref_28","unstructured":"Marcus, A. (2014). Changing paradigm\u2014Changing experience? Comparative usability evaluation of Windows 7 and Windows 8. Design, User Experience, and Usability. Theories, Methods, and Tools for Designing the User Experience, Springer."},{"key":"ref_29","first-page":"58","article-title":"Measuring user experience of the student-centered e-learning environment","volume":"13","author":"Santoso","year":"2016","journal-title":"J. Educ. Online"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Gaeta, E., Beltr\u00e1n-Jaunsaras, M.E., Cea, G., Spieler, B., Burton, A., Garc\u00eda-Betances, R.I., Cabrera-Umpi\u00e9rrez, M.F., Brown, D., Boulton, H., and Arredondo Waldmeyer, M.T. (2019). Evaluation of the create@school game-based learning\u2013teaching approach. Sensors, 19.","DOI":"10.3390\/s19153251"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Mast, M., Burmester, M., Graf, B., Weisshardt, F., Arbeiter, G., \u0160pan\u011bl, M., Materna, Z., Smr\u017e, P., and Kronreif, G. (2015). Design of the human-robot interaction for a semi-autonomous service robot to assist elderly people. Ambient Assisted Living, Springer.","DOI":"10.1007\/978-3-319-11866-6_2"},{"key":"ref_32","first-page":"10","article-title":"The hedonic\/pragmatic model of user experience","volume":"10","author":"Hassenzahl","year":"2007","journal-title":"UX Manif."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"238","DOI":"10.1093\/iwc\/iwt036","article-title":"Sentence completion for understanding users and evaluating user experience","volume":"26","author":"Kujala","year":"2013","journal-title":"Interact. Comput."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Blythe, M., and Monk, A. (2018). The thing and I (summer of \u201917 remix). Funology 2: From Usability to Enjoyment, Springer.","DOI":"10.1007\/978-3-319-68213-6"},{"key":"ref_35","first-page":"223","article-title":"Projective techniques for social science and business research","volume":"26","author":"Soley","year":"2008","journal-title":"J. Consum. Mark."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1016\/j.jrp.2013.09.008","article-title":"Caring about carelessness: Participant inattention and its effects on research","volume":"48","author":"Maniaci","year":"2014","journal-title":"J. Res. Pers."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1016\/j.jesp.2015.07.006","article-title":"Methods for the detection of carelessly invalid responses in survey data","volume":"66","author":"Curran","year":"2016","journal-title":"J. Exp. Soc. Psychol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1002\/job.1962","article-title":"Best practice recommendations for data screening: Data screening","volume":"36","author":"DeSimone","year":"2015","journal-title":"J. Organ. Behav."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1007\/s10869-011-9231-8","article-title":"Detecting and deterring insufficient effort responding to surveys","volume":"27","author":"Huang","year":"2012","journal-title":"J. Bus. Psychol."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1273","DOI":"10.1007\/s11165-016-9602-2","article-title":"The use of cronbach\u2019s \u03b1 when developing and reporting research instruments in science education","volume":"48","author":"Taber","year":"2018","journal-title":"Res. Sci. Educ."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"478","DOI":"10.1207\/S15327663JCP1304_14","article-title":"Advancing \u03b1: Measuring reliability with confidence","volume":"13","author":"Iacobucci","year":"2003","journal-title":"J. Consum. Psychol."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1002\/pri.66","article-title":"The use and interpretation of the friedman test in the analysis of ordinal-scale data in repeated measures designs","volume":"1","author":"Sheldon","year":"1996","journal-title":"Physiother. Res. Int."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1277","DOI":"10.1177\/1049732305276687","article-title":"Three approaches to qualitative content analysis","volume":"15","author":"Hsieh","year":"2005","journal-title":"Qual. Health Res."},{"key":"ref_44","first-page":"103","article-title":"Design and evaluation of a short version of the user experience questionnaire (UEQ-S)","volume":"4","author":"Schrepp","year":"2017","journal-title":"Int. J. Interact. Multimed. Artif. Intell."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1002\/bdm.516","article-title":"Hedonic versus informational evaluations: Task dependent preferences for sequences of outcomes","volume":"19","author":"Zauberman","year":"2006","journal-title":"J. Behav. Decis. Making"},{"key":"ref_46","unstructured":"Blythe, M., and Monk, A. (2018). The thing and I: Understanding the relationship between user and product. Funology 2, Springer."},{"key":"ref_47","unstructured":"Marcus, A., and Wang, W. (2019). Which Instrument should I use? Supporting decision-making about the evaluation of user experience. Design, User Experience, and Usability. Practice and Case Studies, Springer."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Nakamura, W.T., de Oliveira, E.H.T., and Conte, T. (2019). Negative emotions, positive experience: What are we doing wrong when evaluating the UX?. Proceedings of the Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems\u2014CHI EA \u201919, ACM Press.","DOI":"10.1145\/3290607.3313000"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"526","DOI":"10.1016\/j.ijhcs.2013.09.006","article-title":"Attitudes towards user experience (UX) measurement","volume":"72","author":"Law","year":"2014","journal-title":"Int. J. Human Comput. Stud."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Almeida, R.L., Andrade, R.M., Darin, T.G., and Paiva, J.O. (2020, January 6\u201311). CHASE: Checklist to assess user experience in IoT environments. Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results, Seoul, Korea.","DOI":"10.1145\/3377816.3381737"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1016\/j.chb.2014.10.048","article-title":"User experience: A concept without consensus? Exploring practitioners\u2019 Perspectives through an international survey","volume":"43","author":"Lallemand","year":"2015","journal-title":"Comput. Human Behav."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/10\/3480\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:02:32Z","timestamp":1760162552000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/10\/3480"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,17]]},"references-count":51,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2021,5]]}},"alternative-id":["s21103480"],"URL":"https:\/\/doi.org\/10.3390\/s21103480","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,17]]}}}