{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,8]],"date-time":"2025-12-08T22:33:20Z","timestamp":1765233200078,"version":"build-2065373602"},"reference-count":43,"publisher":"MDPI AG","issue":"21","license":[{"start":{"date-parts":[[2021,10,29]],"date-time":"2021-10-29T00:00:00Z","timestamp":1635465600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004916","name":"Funda\u00e7\u00e3o de Amparo \u00e0 Pesquisa do Estado do Amazonas","doi-asserted-by":"publisher","award":["062.00150\/2020","062.00638\/2019"],"award-info":[{"award-number":["062.00150\/2020","062.00638\/2019"]}],"id":[{"id":"10.13039\/501100004916","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002322","name":"Coordena\u00e7\u00e3o de Aperfeicoamento de Pessoal de N\u00edvel Superior","doi-asserted-by":"publisher","award":["001"],"award-info":[{"award-number":["001"]}],"id":[{"id":"10.13039\/501100002322","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003593","name":"National Council for Scientific and Technological Development","doi-asserted-by":"publisher","award":["204081\/2018-1\/PDE","314174\/2020-6"],"award-info":[{"award-number":["204081\/2018-1\/PDE","314174\/2020-6"]}],"id":[{"id":"10.13039\/501100003593","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Samsung (Brazil)","award":["003\/2019","001\/2020"],"award-info":[{"award-number":["003\/2019","001\/2020"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>User experience (UX) is a quality aspect that considers the emotions evoked by the system, extending the usability concept beyond effectiveness, efficiency, and satisfaction. Practitioners and researchers are aware of the importance of evaluating UX. Thus, UX evaluation is a growing field with diverse approaches. Despite various approaches, most of them produce a general indication of the experience as a result and do not seek to capture the problem that gave rise to the bad UX. This information makes it difficult to obtain relevant results to improve the application, making it challenging to identify what caused a negative user experience. To address this gap, we developed a UX evaluation technique called UX-Tips. This paper presents UX-Tips and reports two empirical studies performed in an academic and an industrial setting to evaluate it. Our results show that UX-Tips had good performance in terms of efficiency and effectiveness, making it possible to identify the causes that led to a negative user experience, and it was easy to use. In this sense, we present a new technique suitable for use in both academic and industrial settings, allowing UX evaluation and finding the problems that may lead to a negative experience.<\/jats:p>","DOI":"10.3390\/s21217183","type":"journal-article","created":{"date-parts":[[2021,11,1]],"date-time":"2021-11-01T22:24:22Z","timestamp":1635805462000},"page":"7183","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Understanding UX Better: A New Technique to Go beyond Emotion Assessment"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3645-7606","authenticated-orcid":false,"given":"Leonardo","family":"Marques","sequence":"first","affiliation":[{"name":"Institute of Computing (IComp), Federal University of Amazonas (UFAM), Avenida Rodrigo Ot\u00e1vio 6200, Manaus 69067-005, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9230-3620","authenticated-orcid":false,"given":"Patr\u00edcia Gomes","family":"Matsubara","sequence":"additional","affiliation":[{"name":"Institute of Computing (IComp), Federal University of Amazonas (UFAM), Avenida Rodrigo Ot\u00e1vio 6200, Manaus 69067-005, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5451-3109","authenticated-orcid":false,"given":"Walter Takashi","family":"Nakamura","sequence":"additional","affiliation":[{"name":"Institute of Computing (IComp), Federal University of Amazonas (UFAM), Avenida Rodrigo Ot\u00e1vio 6200, Manaus 69067-005, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2895-1054","authenticated-orcid":false,"given":"Bruna Moraes","family":"Ferreira","sequence":"additional","affiliation":[{"name":"Department of Informatics, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rua Marqu\u00eas de S\u00e3o Vicente 225, Rio de Janeiro 22451-900, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9943-5570","authenticated-orcid":false,"given":"Igor Scaliante","family":"Wiese","sequence":"additional","affiliation":[{"name":"Academic Department of Computing (DACOM), Federal University of Technology of Paran\u00e1 (UTFPR), Campo Mour\u00e3o 87301-899, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7007-5209","authenticated-orcid":false,"given":"Bruno Freitas","family":"Gadelha","sequence":"additional","affiliation":[{"name":"Institute of Computing (IComp), Federal University of Amazonas (UFAM), Avenida Rodrigo Ot\u00e1vio 6200, Manaus 69067-005, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1736-544X","authenticated-orcid":false,"given":"Luciana Martinez","family":"Zaina","sequence":"additional","affiliation":[{"name":"Departament of Computing, Federal University of S\u00e3o Carlos (UFSCar), Sorocaba 18052-780, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1370-7123","authenticated-orcid":false,"given":"David","family":"Redmiles","sequence":"additional","affiliation":[{"name":"Department of Informatics, University of California Irvine (UCI), Irvine, CA 92697, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6436-3773","authenticated-orcid":false,"given":"Tayana Uch\u00f4a","family":"Conte","sequence":"additional","affiliation":[{"name":"Institute of Computing (IComp), Federal University of Amazonas (UFAM), Avenida Rodrigo Ot\u00e1vio 6200, Manaus 69067-005, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,10,29]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Pettersson, I., Lachner, F., Frison, A.K., Riener, A., and Butz, A. (2018, January 21\u201326). A bermuda triangle?\u2014A review of method application and triangulation in user experience evaluation. Proceedings of the Conference on Human Factors in Computing Systems\u2014Proceedings, Association for Computing Machinery, Montreal, QC, Canada.","DOI":"10.1145\/3173574.3174035"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1080\/01449290500330331","article-title":"User experience\u2014A research agenda","volume":"25","author":"Hassenzahl","year":"2006","journal-title":"Behav. Inf. Technol."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Hassenzahl, M. (2008, January 2\u20135). User experience (UX): Towards an experiential perspective on product quality. Proceedings of the 20th Conference on l\u2019Interaction Homme-Machine, Metz, France.","DOI":"10.1145\/1512714.1512717"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Kou, Y., and Gray, C.M. (2019, January 4\u20139). A practice-led account of the conceptual evolution of UX knowledge. Proceedings of the Conference on Human Factors in Computing Systems\u2014Proceedings, Glasgow, UK.","DOI":"10.1145\/3290605.3300279"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1007\/978-3-540-73105-4_19","article-title":"Emotional experiences and quality perceptions of interactive products","volume":"Volume 4550 LNCS","author":"Mahlke","year":"2007","journal-title":"Lecture Notes in Computer Science"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Bargas-Avila, J.A., and Hornb\u00e6k, K. (2011, January 7\u201312). Old wine in new bottles or novel challenges? A critical analysis of empirical studies of User Experience. Proceedings of the Conference on Human Factors in Computing Systems\u2014Proceedings, Vancouver, BC, Canada.","DOI":"10.1145\/1978942.1979336"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Marques, L., Nakamura, W., Valentim, N., Rivero, L., and Conte, T. (2018, January 1\u20133). Do scale type techniques identify problems that affect user experience? User experience evaluation of a mobile application. Proceedings of the International Conference on Software Engineering and Knowledge Engineering, Redwood City, CA, USA.","DOI":"10.18293\/SEKE2018-161"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Vermeeren, A.P., Law, E.L.C., Roto, V., Obrist, M., Hoonhout, J., and V\u00e4\u00e4n\u00e4nen-Vainio-Mattila, K. (2010, January 16\u201320). User experience evaluation methods: Current state and development needs. Proceedings of the NordiCHI 2010: Extending Boundaries\u2014Proceedings of the 6th Nordic Conference on Human\u2013Computer Interaction, Reykjavik, Iceland.","DOI":"10.1145\/1868914.1868973"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Rivero, L., and Conte, T. (2017, January 23\u201327). A systematic mapping study on research contributions on UX evaluation technologies. Proceedings of the XVI Brazilian Symposium on Human Factors in Computing Systems, Joinville, Brazil.","DOI":"10.1145\/3160504.3160512"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1016\/j.chb.2014.10.048","article-title":"User experience: A concept without consensus? Exploring practitioners\u2019 perspectives through an international survey","volume":"43","author":"Lallemand","year":"2015","journal-title":"Comput. Hum. Behav."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"542","DOI":"10.1016\/j.ijhcs.2013.10.004","article-title":"Investigating and promoting UX practice in industry: An experimental study","volume":"72","author":"Ardito","year":"2014","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Marques, L., Matsubara, P., Nakamura, W., Wiese, I., Zaina, L., and Conte, T. (2019, January 23\u201327). UX-Tips: A UX evaluation technique to support the identification of software application problems. Proceedings of the XXXIII Brazilian Symposium on Software Engineering, Salvador, Brazil.","DOI":"10.1145\/3350768.3350783"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"460","DOI":"10.1007\/978-3-319-60492-3_44","article-title":"Beyond \u201cUsability and user experience\u201d, towards an integrative heuristic inspection: From accessibility to persuasiveness in the UX evaluation a case study on an insurance prospecting tablet application","volume":"Volume 607","author":"Urrutia","year":"2018","journal-title":"Advances in Intelligent Systems and Computing"},{"key":"ref_14","unstructured":"ISO (2021, October 20). 9241-11\u2014Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs)\u2014Guidance on Usability. Available online: https:\/\/www.iso.org\/obp\/ui\/#iso:std:iso:9241:-11:ed-1:v1:en."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"132","DOI":"10.1016\/j.csi.2019.03.003","article-title":"Developing usability heuristics with PROMETHEUS: A case study in virtual learning environments","volume":"65","author":"Figueroa","year":"2019","journal-title":"Comput. Stand. Interfaces"},{"key":"ref_16","first-page":"1","article-title":"To cite this article: Bernd Schmitt (1999) Experiential Marketing","volume":"15","author":"Schmitt","year":"2010","journal-title":"J. Mark. Manag."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Hassenzahl, M. (2018). The Thing and I (Summer of \u201917 Remix). Funology 2, Springer.","DOI":"10.1007\/978-3-319-68213-6_2"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Norman, D., Miller, J., and Henderson, A. (1995, January 7\u201311). What you see, some of what\u2019s in the future, and how we go about doing it: HI at apple computer. Proceedings of the Conference on Human Factors in Computing Systems\u2014Proceedings, Denver, CO, USA.","DOI":"10.1145\/223355.223477"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Hassenzahl, M. (2003). The Thing and I: Understanding the Relationship between User and Product, Springer.","DOI":"10.1007\/1-4020-2967-5_4"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Bach, P.M., Deline, R., and Carroll, J.M. (2009, January 4\u20139). Designers wanted: Participation and the user experience in open source software development. Proceedings of the Conference on Human Factors in Computing Systems\u2014Proceedings, Boston, MA, USA.","DOI":"10.1145\/1518701.1518852"},{"key":"ref_21","unstructured":"V\u00e4\u00e4n\u00e4nen-Vainio-Mattila, K., Roto, V., and Hassenzahl, M. (2008). Towards Practical User Experience Evaluation Methods. Meaningful Measures: Valid Useful User Experience Measurement, Institute of Research in Informatics of Toulouse (IRIT)."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Law, E.L., Roto, V., Hassenzahl, M., Vermeeren, A.P., and Kort, J. (2009, January 4\u20139). Understanding, scoping and defining user experience: A survey approach. Proceedings of the Conference on Human Factors in Computing Systems\u2014Proceedings, Boston, MA, USA.","DOI":"10.1145\/1518701.1518813"},{"key":"ref_23","unstructured":"Roto, V., Law, E., Vermeeren, A., and Hoonhout, J. (2021, October 20). User Experience White Paper: Bringing Clarity to the Concept of User Experience. Result from Dagstuhl Seminar on Demarcating User Experience. Available online: http:\/\/www.allaboutux.org\/files\/UX-WhitePaper.pdf."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Roto, V., Obrist, M., and V\u00e4\u00e4n\u00e4nen-Vainio-Mattila, K. (2009, January 4\u20139). User Experience Evaluation Methods in Academic and Industrial Contexts. Proceedings of the Workshop UXEM, Boston, MA, USA.","DOI":"10.1145\/1520340.1520401"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1002\/asi.21229","article-title":"The development and evaluation of a survey to measure user engagement","volume":"61","author":"Toms","year":"2010","journal-title":"J. Assoc. Inf. Sci. Technol."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Vatanasuk, N., Chomputawat, A., Chomputawat, S., and Chatwiriya, W. (2015, January 23\u201325). Mobile Crime Incident Reporting System using UX dimensions guideline. Proceedings of the ACDT 2015\u2014Proceedings: The 1st Asian Conference on Defence Technology, Hua Hin, Thailand.","DOI":"10.1109\/ACDT.2015.7111609"},{"key":"ref_27","unstructured":"Hao, Y., Chong, W., Man, K.L., Liu, O., and Shi, X. (2016, January 16\u201318). Key Factors Affecting User Experience of Mobile Crowdsourcing Applications. Proceedings of the International MultiConference of Engineers and Computer Scientists, Hong Kong, China."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Schrepp, M., Hinderks, A., and Thomaschewski, J. (2014). Applying the user experience questionnaire (UEQ) in different evaluation scenarios. Lecture Notes in Computer Science, Springer.","DOI":"10.1007\/978-3-319-07668-3_37"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"272","DOI":"10.1007\/978-3-319-07515-0_28","article-title":"Defining and structuring the dimensions of user experience with interactive products","volume":"Volume 8532 LNAI","author":"Robert","year":"2014","journal-title":"Lecture Notes in Computer Science"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"399","DOI":"10.1007\/978-3-642-39229-0_43","article-title":"The dimensions of positive and negative user experiences with interactive products","volume":"Volume 8012 LNCS","author":"Provost","year":"2013","journal-title":"Lecture Notes in Computer Science"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Cavalcante, E., Rivero, L., and Conte, T. (2015, January 6\u20138). MAX: A Method for Evaluating the Post-use User eXperience through Cards and a Board. Proceedings of the 27th International Conference on Software Engineering and Knowledge Engineering, Pittsburgh, PA, USA.","DOI":"10.18293\/SEKE2015-136"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Maya, J., and Ariza, N. (2014, January 19\u201322). Towards an empirical model of the UX: A factor analysis study. Proceedings of the Design and Emotion 2014, Dubrovnik, Croatia.","DOI":"10.11608\/sgnj.2014.19.006"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1007\/s10111-017-0406-6","article-title":"Sustainable Usage through Emotional Engagement: A User Experience Analysis of an Adaptive Driving School Application","volume":"19","author":"Dirin","year":"2017","journal-title":"Technol. Work. J."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1080\/0144929X.2012.751708","article-title":"Must evaluation methods be about usability? Devising and assessing the utility inspection method","volume":"33","author":"Johannessen","year":"2014","journal-title":"Behav. Inf. Technol."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1016\/j.tourman.2015.09.020","article-title":"In search of patterns among travellers\u2019 hotel ratings in TripAdvisor","volume":"53","author":"Banerjee","year":"2016","journal-title":"Tour. Manag."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Hvannberg, E.T., and Law, E.L.C. (2017, January 26\u201329). Novice evaluators\u2019 behavior when consolidating usability problems individually or collaboratively. Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems, EICS 2017, Lisbon, Portugal.","DOI":"10.1145\/3102113.3102129"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Fernandez, A., Abrah\u00e3o, S., Insfran, E., and Matera, M. (2012, January 19\u201320). Further analysis on the validation of a usability inspection method for model-driven web development. Proceedings of the International Symposium on Empirical Software Engineering and Measurement, Lund, Sweden.","DOI":"10.1145\/2372251.2372278"},{"key":"ref_38","unstructured":"Wohlin, C., Runeson, P., H\u00f6st, M., Ohlsson, M.C., Regnell, B., and Wessl\u00e9n, A. (2014). Experimentation in Software Engineering, Springer."},{"key":"ref_39","unstructured":"Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences, Lawrence Earlbaum Associates. [2nd ed.]."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1277","DOI":"10.1177\/1049732305276687","article-title":"Three approaches to qualitative content analysis","volume":"15","author":"Hsieh","year":"2005","journal-title":"Qual. Health Res."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Juristo, N., and Moreno, A.M. (2001). Basics of Software Engineering Experimentation, Springer.","DOI":"10.1007\/978-1-4757-3304-4"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1214\/aoms\/1177730491","article-title":"On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other","volume":"18","author":"Mann","year":"1947","journal-title":"Ann. Math. Stat."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1016\/j.chb.2018.04.019","article-title":"The state of immersive technology research: A literature analysis","volume":"86","author":"Suh","year":"2018","journal-title":"Comput. Hum. Behav."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/21\/7183\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:22:25Z","timestamp":1760167345000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/21\/7183"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,29]]},"references-count":43,"journal-issue":{"issue":"21","published-online":{"date-parts":[[2021,11]]}},"alternative-id":["s21217183"],"URL":"https:\/\/doi.org\/10.3390\/s21217183","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2021,10,29]]}}}