{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T19:13:04Z","timestamp":1782846784178,"version":"3.54.5"},"reference-count":33,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T00:00:00Z","timestamp":1782777600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"name":"Austrian Research Promotion Agency","award":["915086"],"award-info":[{"award-number":["915086"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>\n                    <jats:italic toggle=\"yes\">Usability<\/jats:italic>\n                    describes quality attributes of application\n                    <jats:italic toggle=\"yes\">user interfaces<\/jats:italic>\n                    that determine how effectively users can interact with them. Traditional usability evaluation methods require considerable expertise and resources, which can be challenging, especially for small teams and organizations. Automating usability evaluation could make it more accessible and help to improve the user experience. The recent emergence of powerful\n                    <jats:italic toggle=\"yes\">multimodal large language models (MLLMs)<\/jats:italic>\n                    has opened new opportunities for automating usability evaluation and recommendation of improvements. These models can process visual inputs such as images and videos alongside textual context, which enables the identification of usability issues and the generation of actionable suggestions to resolve these issues.\n                  <\/jats:p>\n                  <jats:p>\n                    In this paper, we present a novel automated approach that uses limited application context and screen recordings of user interactions as input to an MLLM. The model automatically identifies and describes usability issues based on\n                    <jats:italic toggle=\"yes\">Nielsen\u2019s usability heuristics<\/jats:italic>\n                    , and provides corresponding explanations and improvement recommendations. To reduce the developer effort of manual prioritization, the recommendations are ranked by severity. The quality and practical usefulness of the generated recommendations were evaluated based on a user study that involved software engineers as participants. The evaluation focused on the highest-ranked suggestions provided by the model. The results demonstrate the potential of our approach to provide low-effort usability improvement recommendations. This makes it a promising complement to traditional evaluation methods, especially in settings with limited access to usability experts. In this sense, the approach serves as a basis for future integration into development tools to enable automated usability evaluation within software engineering workflows.\n                  <\/jats:p>","DOI":"10.1145\/3797121","type":"journal-article","created":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:06:14Z","timestamp":1782839174000},"page":"568-589","source":"Crossref","is-referenced-by-count":0,"title":["Recommending Usability Improvements with Multimodal Large Language Models"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5024-3786","authenticated-orcid":false,"given":"Sebastian","family":"Lubos","sequence":"first","affiliation":[{"name":"Graz University of Technology, Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0108-3146","authenticated-orcid":false,"given":"Alexander","family":"Felfernig","sequence":"additional","affiliation":[{"name":"Graz University of Technology, Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-0993-0911","authenticated-orcid":false,"given":"Damian","family":"Garber","sequence":"additional","affiliation":[{"name":"Graz University of Technology, Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5778-975X","authenticated-orcid":false,"given":"Viet-Man","family":"Le","sequence":"additional","affiliation":[{"name":"Graz University of Technology, Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-4644-0836","authenticated-orcid":false,"given":"Manuel","family":"Henrich","sequence":"additional","affiliation":[{"name":"UNiQUARE Software Development, Krumpendorf am W\u00f6rthersee, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,30]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549118"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.29"},{"key":"e_1_2_1_3_1","volume-title":"Rojas","author":"Castro John W.","year":"2022","unstructured":"John W. Castro, Ignacio Garnica, and Luis A. Rojas. 2022. Automated Tools for Usability Evaluation: A Systematic Mapping Study. In Social Computing and Social Media: Design, User Experience and Impact, Gabriele Meiselwitz (Ed.). Springer International Publishing, Cham, 28-46."},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 37th International Conference on Neural Information Processing Systems","author":"Deng Xiang","year":"2023","unstructured":"Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. MIND2WEB: towards a generalist agent for the web. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, Article 1220, 24 pages."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642782"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-44943-7_2"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-032-05005-2_20"},{"key":"e_1_2_1_8_1","volume-title":"Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust.","author":"Gur Izzeddin","year":"2024","unstructured":"Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2024. A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis. arXiv:2307.12856 [cs.LG] https:\/\/arxiv.org\/abs\/2307.12856"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-96906-0_6"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1297144.1297200"},{"key":"e_1_2_1_11_1","unstructured":"Eduard Kuric Peter Demcak Matus Krajcovic and Jan Lang. 2025. Systematic Literature Review of Automation and Artificial Intelligence in Usability Issue Detection. arXiv:2504.01415 [cs.HC] https:\/\/arxiv.org\/abs\/2504.01415"},{"key":"e_1_2_1_12_1","volume-title":"Intelligent Data Engineering and Automated Learning -IDEAL","author":"Li Baoli","year":"2013","unstructured":"Baoli Li and Liping Han. 2013. Distance Weighted Cosine Similarity Measure for Text Classification. In Intelligent Data Engineering and Automated Learning -IDEAL 2013, Hujun Yin, Ke Tang, Yang Gao, Frank Klawonn, Minho Lee, Thomas Weise, Bin Li, and Xin Yao (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 611-618."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3660803"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416547"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.19498008"},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the 12th Joint Workshop on Interfaces and Human Decision Making for Recommender Systems (IntRS 2025) (CEUR Workshop Proceedings","volume":"7","author":"Lubos Sebastian","year":"2025","unstructured":"Sebastian Lubos, Alexander Felfernig, Damian Garber, Viet Man Le, and Thi Ngoc Trang Tran. 2025. Towards LLM- Based Usability Analysis for Recommender User Interfaces. In Proceedings of the 12th Joint Workshop on Interfaces and Human Decision Making for Recommender Systems (IntRS 2025) (CEUR Workshop Proceedings, Vol. 4027). CEUR-WS, Aachen. https:\/\/ceur-ws.org\/Vol-4027\/paper7.pdf"},{"key":"e_1_2_1_17_1","unstructured":"Sebastian Lubos Alexander Felfernig Damian Garber Gerhard Leitner Julian Schwazer and Manuel Henrich. 2026. Investigating Multimodal Large Language Models to Support Usability Evaluation. arXiv:2508.16165 [cs.SE] https: \/\/arxiv.org\/abs\/2508.16165"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468604"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3278154"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1080\/0144929032000173951"},{"key":"e_1_2_1_21_1","volume-title":"Design, User Experience, and Usability: UX Research and Design, Marcelo M","author":"Namoun Abdallah","unstructured":"Abdallah Namoun, Ahmed Alrehaili, and Ali Tufail. 2021. A Review of Automated Website Usability Evaluation Tools: Research Issues and Challenges. In Design, User Experience, and Usability: UX Research and Design, Marcelo M. Soares, Elizabeth Rosenzweig, and Aaron Marcus (Eds.). Springer International Publishing, Cham, 292-311."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/191666.191729"},{"key":"e_1_2_1_23_1","first-page":"04","article-title":"Usability 101: Introduction to Usability. https:\/\/www.nngroup.com\/articles\/usability-101- introduction-to-usability\/","volume":"22","author":"Nielsen Jakob","year":"2012","unstructured":"Jakob Nielsen. 2012. Usability 101: Introduction to Usability. https:\/\/www.nngroup.com\/articles\/usability-101- introduction-to-usability\/. Accessed: 22.04.2025.","journal-title":"Accessed"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2023.3266377"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE55347.2025.00138"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1764810.1764814"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3473109"},{"key":"e_1_2_1_28_1","volume-title":"Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805 [cs.CL] https:\/\/arxiv.org\/abs\/2312.11805","author":"Team Gemini","year":"2024","unstructured":"Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2024. Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805 [cs.CL] https:\/\/arxiv.org\/abs\/2312.11805"},{"key":"e_1_2_1_29_1","volume-title":"A Comprehensive Model of Usability","author":"Winter Sebastian","unstructured":"Sebastian Winter, Stefan Wagner, and Florian Deissenboeck. 2008. A Comprehensive Model of Usability. In Engineering Interactive Systems, Jan Gulliksen, Morton Borup Harning, Philippe Palanque, Gerrit C. van der Veer, and Janet Wesson (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 106-122."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1093\/nsr\/nwae403"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST60714.2024.00020"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616329"},{"key":"e_1_2_1_33_1","unstructured":"Ruican Zhong David W. McDonald and Gary Hsieh. 2025. Synthetic Heuristic Evaluation: A Comparison between AI- and Human-Powered Usability Evaluation. arXiv:2507.02306 [cs.HC] https:\/\/arxiv.org\/abs\/2507.02306"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3797121","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:34:22Z","timestamp":1782844462000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3797121"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,30]]},"references-count":33,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3797121"],"URL":"https:\/\/doi.org\/10.1145\/3797121","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,30]]}}}