{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,5]],"date-time":"2026-05-05T21:34:01Z","timestamp":1778016841534,"version":"3.51.4"},"reference-count":58,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2022,10,20]],"date-time":"2022-10-20T00:00:00Z","timestamp":1666224000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Nowadays, web designers are forced to have an even deeper perception of how users approach their products in terms of user experience and usability. Remote Usability Testing (RUT) is the most appropriate tool to assess the usability of web platforms by measuring the level of user attention, satisfaction, and productivity. RUT does not require the physical presence of users and evaluators, but for this very reason makes data collection more difficult. To simplify data collection and analysis and help RUT moderators collect and analyze user\u2019s data in a non-intrusive manner, this research work proposes a low-cost comprehensive framework based on Deep Learning algorithms. The proposed framework, called Miora, employs facial expression recognition, gaze recognition, and analytics algorithms to capture data about other information of interest for in-depth usability analysis, such as interactions with the analyzed software. It uses a comprehensive evaluation methodology to elicit information about usability metrics and presents the results in a series of graphs and statistics so that the moderator can intuitively analyze the different trends related to the KPI used as usability indicators. To demonstrate how the proposed framework could facilitate the collection of large amounts of data and enable moderators to conduct both remote formative and summative tests in a more efficient way than traditional lab-based usability testing, two case studies have been presented: the analysis of an online shop and of a management platform. Obtained results suggest that this framework can be employed in remote usability testing to conduct both formative and summative tests.<\/jats:p>","DOI":"10.3390\/info13100505","type":"journal-article","created":{"date-parts":[[2022,10,20]],"date-time":"2022-10-20T22:49:46Z","timestamp":1666306186000},"page":"505","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["A Test Management System to Support Remote Usability Assessment of Web Applications"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6173-9616","authenticated-orcid":false,"given":"Andrea","family":"Generosi","sequence":"first","affiliation":[{"name":"Department of Industrial Engineering and Mathematical Sciences, Universit\u00e0 Politecnica delle Marche, Via Brecce Bianche 12, 60131 Ancona, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6475-2533","authenticated-orcid":false,"given":"Jos\u00e9 Yuri","family":"Villafan","sequence":"additional","affiliation":[{"name":"Department of Industrial Engineering and Mathematical Sciences, Universit\u00e0 Politecnica delle Marche, Via Brecce Bianche 12, 60131 Ancona, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Luca","family":"Giraldi","sequence":"additional","affiliation":[{"name":"Department of Economics, Universit\u00e0 degli Studi di Macerata, Piazza San Vincenzo Maria Strambi 1, 62100 Macerata, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8916-1262","authenticated-orcid":false,"given":"Silvia","family":"Ceccacci","sequence":"additional","affiliation":[{"name":"Department of Education, Cultural Heritage and Tourism, Universit\u00e0 degli Studi di Macerata, P.le Luigi Bertelli 1, 62100 Macerata, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Maura","family":"Mengoni","sequence":"additional","affiliation":[{"name":"Department of Industrial Engineering and Mathematical Sciences, Universit\u00e0 Politecnica delle Marche, Via Brecce Bianche 12, 60131 Ancona, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,10,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"100375","DOI":"10.1016\/j.cosrev.2021.100375","article-title":"Neurological and physiological measures to evaluate the usability and user-experience (UX) of information systems: A systematic literature review","volume":"40","author":"Zaki","year":"2021","journal-title":"Comput. Sci. Rev."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1016\/j.indmarman.2018.07.005","article-title":"The drivers of success in new-product development","volume":"76","author":"Cooper","year":"2018","journal-title":"Ind. Mark. Manag."},{"key":"ref_3","first-page":"65","article-title":"Reducing the risks of new product development","volume":"47","author":"Piller","year":"2006","journal-title":"MIT Sloan Manag. Rev."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Nielsen, J. (1994). Usability Engineering, Morgan Kaufman.","DOI":"10.1016\/B978-0-08-052029-2.50009-7"},{"key":"ref_5","unstructured":"International Organization for Standardization (2018). Ergonomics of Human-System Interaction\u2014Part 11: Usability: Definitions and Concepts (ISO 9241-11:2018) (Standard No. ISO 9241-11:2018). Available online: https:\/\/www.iso.org\/standard\/63500.html."},{"key":"ref_6","unstructured":"Tullis, T., and Albert, B. (2013). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics, Morgan Kaufmann. [2nd ed.]."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Bevan, N., Carter, J., Earthy, J., Geis, T., and Harker, S. (2016). New ISO Standards for Usability, Usability Reports and Usability Measures. Human-Computer Interaction. Theory, Design, Development and Practice, Proceedings of the 2016 International Conference on Human-Computer Interaction (HCI\u20192016), Toronto, ON, Canada, 17\u201322 July 2016, Springer.","DOI":"10.1007\/978-3-319-39510-4_25"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1111\/j.1467-8535.1993.tb00076.x","article-title":"SUMI: The Software Usability Measurement Inventory","volume":"24","author":"Kirakowski","year":"1993","journal-title":"Br. J. Educ. Technol."},{"key":"ref_9","first-page":"4","article-title":"SUS: A \u2019Quick and Dirty\u2019 Usability Scale","volume":"189","author":"Brooke","year":"1996","journal-title":"Usability Eval. Ind."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Altieri, A., Ceccacci, S., and Mengoni, M. (2019). Emotion-Aware Ambient Intelligence: Changing Smart Environment Interaction Paradigms Through Affective Computing. International Conference on Human-Computer Interaction, Springer.","DOI":"10.1007\/978-3-030-21935-2_20"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Munim, K.M., Islam, I., Khatun, M., Karim, M., and Islam, M.N. (2017, January 7\u20139). Towards developing a tool for UX evaluation using facial expression. Proceedings of the 2017 3rd International Conference on Electrical Information and Communication Technology (EICT), Khulna, Bangladesh.","DOI":"10.1109\/EICT.2017.8275227"},{"key":"ref_12","first-page":"113","article-title":"Development of Integrated Analysis System and Tool of Perception, Recognition, and Behavior for Web Usability Test: With Emphasis on Eye-Tracking, Mouse-Tracking, and Retrospective Think Aloud","volume":"Volume 4559","author":"Kim","year":"2007","journal-title":"International Conference on Usability and Internationalization"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Hussain, J., Khan, W.A., Hur, T., Bilal, H.S.M., Bang, J., Hassan, A.U., Afzal, M., and Lee, S. (2018). A Multimodal Deep Log-Based User Experience (UX) Platform for UX Evaluation. Sensors, 18.","DOI":"10.3390\/s18051622"},{"key":"ref_14","unstructured":"Franco, R.Y.D.S., De Freitas, A.A., Lima, R.S.D.A.D., Mota, M.P., dos Santos, C.G.R., and Meiguins, B.S. (2019, January 2\u20135). UXmood\u2014A Tool to Investigate the User Experience (UX) Based on Multimodal Sentiment Analysis and Information Visualization (InfoVis). Proceedings of the 2019 23rd International Conference Information Visualisation (IV), Paris, France."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Sivaji, A., and Ahmad, W.F.W. (2014). Benefits of Complementing Eye-Tracking Analysis with Think-Aloud Protocol in a Multilingual Country with High Power Distance. Current Trends in Eye Tracking Research, Springer.","DOI":"10.1007\/978-3-319-02868-2_21"},{"key":"ref_16","unstructured":"Dixit, S., and Padmadas, V. (2016, January 26\u201329). Automated Usability Evaluation of Web Applications. Proceedings of the International Congress on Information and Communication Technology, Singapore."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., and Torralba, A. (2016, January 27\u201330). Eye Tracking for Everyone. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.239"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1016\/j.chb.2017.02.035","article-title":"Diagnosing user perception and acceptance using eye tracking in web-based end-user development","volume":"72","author":"Tzafilkou","year":"2017","journal-title":"Comput. Hum. Behav."},{"key":"ref_19","unstructured":"Moro, R., Daraz, J., and Bielikova, M. (2014, January 1\u20134). Visualization of Gaze Tracking Data for UX Testing on the Web. Proceedings of the Late-breaking Results, Doctoral Consortium and Workshop Proceedings of the 25th ACM Hypertext and Social Media Conference, Santiago, Chile."},{"key":"ref_20","unstructured":"Barnum, C.M. (2020). Usability Testing Essentials: Ready, Set\u2026 Test!, Morgan Kaufman. [2nd ed.]."},{"key":"ref_21","first-page":"63","article-title":"Remote Web Site Usability Testing\u2014Benefits Over Traditional Methods","volume":"3","author":"Gardner","year":"2007","journal-title":"Int. J. Public Inf. Syst."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1177\/1541931213571043","article-title":"Validation of the System Usability Scale (SUS)","volume":"57","author":"Peres","year":"2013","journal-title":"Proc. Hum. Factors Ergon. Soc. Annu. Meet."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3299096","article-title":"How UX Practitioners Produce Findings in Usability Testing","volume":"26","author":"Reeves","year":"2019","journal-title":"ACM Trans. Comput. Interact."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Jabbar, J., Urooj, I., JunSheng, W., and Azeem, N. (2019, January 9\u201311). Real-time Sentiment Analysis On E-Commerce Application. Proceedings of the IEEE 16th International Conference on Networking, Sensing and Control (ICNSC), Banff, AB, Canada.","DOI":"10.1109\/ICNSC.2019.8743331"},{"key":"ref_25","unstructured":"Nielsen, J. (2022, October 18). Why You Only Need to Test with 5 Users. Available online: https:\/\/www.nngroup.com\/articles\/why-you-only-need-to-test-with-5-users\/."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Sivaji, A., and Tzuaan, S.S. (2012, January 9\u201312). Website user experience (UX) testing tool development using Open Source Software (OSS). Proceedings of the Southeast Asian Network of Ergonomics Societies Conference (SEANES), Langkawi, Malaysia.","DOI":"10.1109\/SEANES.2012.6299576"},{"key":"ref_27","unstructured":"Chin, J.P., Diehl, V.A., and Norman, K.L. (1998, January 15\u201319). Development of an instrument measuring user satisfaction of the human-computer interface. Proceedings of the SIGCHI \u201888, Washington, DC, USA."},{"key":"ref_28","unstructured":"Claridge, N., and Kirakowski, J. (2022, July 27). WAMMI: Website Analysis and Measurement Inventory Questionnaire. Available online: http:\/\/www.wammi.com\/index.html."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Ceccacci, S., Generosi, A., Giraldi, L., and Mengoni, M. (2018, January 26\u201329). An Emotion Recognition System for monitoring Shopping Experience. Proceedings of the 11th Pervasive Technologies Related to Assistive Environments Conference, Corfu, Greece.","DOI":"10.1145\/3197768.3201518"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1195","DOI":"10.1109\/TAFFC.2020.2981446","article-title":"Deep Facial Expression Recognition: A Survey","volume":"13","author":"Li","year":"2020","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Ekman, P. (1999). Basic Emotions. Handbook of Cognition and Emotions, John Wiley & Sons.","DOI":"10.1002\/0470013494.ch3"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1109\/TAFFC.2017.2713783","article-title":"Audio-Visual Emotion Recognition in Video Clips","volume":"10","author":"Noroozi","year":"2017","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1177\/1529100619832930","article-title":"Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements","volume":"20","author":"Barrett","year":"2019","journal-title":"Psychol. Sci. Public Interes."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Kaur, H., McDuff, D., Williams, A.C., Teevan, J., and Iqbal, S.T. (May, January 29). \u201cI Didn\u2019t Know I Looked Angry\u201d: Characterizing Observed Emotion and Reported Affect at Work. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, New York, NY, USA.","DOI":"10.1145\/3491102.3517453"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1326","DOI":"10.1080\/02699931.2013.777660","article-title":"On the valence of surprise","volume":"27","author":"Noordewier","year":"2013","journal-title":"Cogn. Emot."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Basu, S., Bag, A., Aftabuddin, M., Mahadevappa, M., Mukherjee, J., and Guha, R. (2016, January 16\u201318). Effects of emotion on physiological signals. Proceedings of the 2016 IEEE Annual India Conference (INDICON), Bangalore, India.","DOI":"10.1109\/INDICON.2016.7839091"},{"key":"ref_37","first-page":"081046","article-title":"Emotion Recognition based on EEG using LSTM Recurrent Neural Network","volume":"8","author":"Alhagry","year":"2017","journal-title":"Int. J. Adv. Comput. Sci. Appl."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Athavipach, C., Pan-Ngum, S., and Israsena, P. (2019). A Wearable In-Ear EEG Device for Emotion Monitoring. Sensors, 19.","DOI":"10.3390\/s19184014"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Zhang, X., Sugano, Y., Fritz, M., and Bulling, A. (2015, January 12). Appearance-Based Gaze Estimation in the Wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299081"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"23","DOI":"10.3233\/SW-140163","article-title":"Eye tracking the user experience\u2014An evaluation of ontology visualization techniques","volume":"8","author":"Fu","year":"2016","journal-title":"Semant. Web"},{"key":"ref_41","unstructured":"De Bleecker, I., and Okoroji, R. (2018). Remote Usability Testing: Actionable Insights in User Behavior across Geographies and Time Zones, Packt Publishing."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"663","DOI":"10.1080\/10447318.2014.930311","article-title":"Usability: Lessons Learned \u2026 and Yet to Be Learned","volume":"30","author":"Lewis","year":"2014","journal-title":"Int. J. Hum.-Comput. Interact."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1080\/10447318.2021.1938391","article-title":"A Comparison of Synchronous and Asynchronous Remote Usability Testing Methods","volume":"38","author":"Alhadreti","year":"2021","journal-title":"Int. J. Human-Computer Interact."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"477","DOI":"10.1016\/j.tourman.2010.03.015","article-title":"Google Analytics for measuring website performance","volume":"32","author":"Plaza","year":"2011","journal-title":"Tour. Manag."},{"key":"ref_45","unstructured":"(2022, February 24). User Research & UX Testing Solution. Available online: https:\/\/www.userzoom.com\/."},{"key":"ref_46","unstructured":"(2022, February 24). Validately. Available online: https:\/\/faq.validately.com\/hc\/en-us."},{"key":"ref_47","unstructured":"(2022, February 24). The Human Insight Platform. Available online: https:\/\/www.usertesting.com\/."},{"key":"ref_48","unstructured":"(2022, February 24). Remote User Testing Platform. Available online: https:\/\/www.userlytics.com."},{"key":"ref_49","unstructured":"(2022, February 24). Website Heatmaps & Behavior Analytics Tools. Available online: https:\/\/www.hotjar.com."},{"key":"ref_50","unstructured":"(2022, February 24). Celebrating 20 Years\u2014Global Leader in Eye Tracking. Available online: https:\/\/www.tobii.com\/."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"185","DOI":"10.25046\/aj050622","article-title":"A Toolkit for the Automatic Analysis of Human Behavior in HCI Applications in the Wild","volume":"5","author":"Generosi","year":"2020","journal-title":"Adv. Sci. Technol. Eng. Syst. J."},{"key":"ref_52","first-page":"1755","article-title":"Dlib-ml: A Machine Learning Toolkit","volume":"10","author":"King","year":"2009","journal-title":"J. Mach. Learn. Res."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Talipu, A., Generosi, A., Mengoni, M., and Giraldi, L. (2019, January 19\u201321). Evaluation of Deep Convolutional Neural Network architectures for Emotion Recognition in the Wild. Proceedings of the 2019 IEEE 23rd International Symposium on Consumer Technologies (ISCT), Ancona, Italy.","DOI":"10.1109\/ISCE.2019.8900994"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Ceccacci, S., Mengoni, M., Andrea, G., Giraldi, L., Carbonara, G., Castellano, A., and Montanari, R. (2020). A Preliminary Investigation Towards the Application of Facial Expression Analysis to Enable an Emotion-Aware Car Interface. International Conference on Human-Computer Interaction, Springer.","DOI":"10.1007\/978-3-030-49108-6_36"},{"key":"ref_55","unstructured":"Ceccacci, S., Generosi, A., Cimini, G., Faggiano, S., Giraldi, L., and Mengoni, M. (2021, January 21\u201322). Facial coding as a mean to enable continuous monitoring of student\u2019s behavior in e-Learning. Proceedings of the CEUR Workshop Proceedings, Foggia, Italy."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Koonsanit, K., and Nishiuchi, N. (2020, January 16\u201319). Classification of User Satisfaction Using Facial Expression Recognition and Machine Learning. Proceedings of the 2020 IEEE REGION 10 CONFERENCE (TENCON), Osaka, Japan.","DOI":"10.1109\/TENCON50793.2020.9293912"},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Landowska, A., and Miler, J. (2016, January 11\u201314). Limitations of emotion recognition in software user experience evaluation context. Proceedings of the 2016 Federated Conference on Computer Science and Information Systems (FedCSIS), Gdansk, Poland.","DOI":"10.15439\/2016F535"},{"key":"ref_58","unstructured":"(2022, April 22). Available online: https:\/\/justadudewhohacks.github.io\/face-api.js\/docs\/index.html."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/10\/505\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:57:45Z","timestamp":1760144265000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/10\/505"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,20]]},"references-count":58,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2022,10]]}},"alternative-id":["info13100505"],"URL":"https:\/\/doi.org\/10.3390\/info13100505","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,20]]}}}