{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,16]],"date-time":"2026-05-16T15:09:56Z","timestamp":1778944196701,"version":"3.51.4"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"2","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>Inclusive usability testing, such as the GenderMag method, wants to identify gender-related usability problems in digital interfaces. Large Language Models (LLMs) have been used by usability engineers in usability evaluations but their contribution is still underexplored, especially regarding inclusive usability testing. Research has shown that GenderMag workshops can produce valuable insights but are resource intense and might show effects from the evaluators\u2019 ability to embody personas with a different cognitive style. Therefore, we need to assess if LLM-agent based testing can aid human-based evaluations. This study evaluates an LLM-agent system for GenderMag persona-based usability testing, and compares its performance to traditional human-led evaluations. The agent system integrates GenderMag persona facets into three LLM-agents, which analyze usability issues of four web interfaces, three generic and one intentionally flawed interface with gender-related usability issues. We quantitatively and qualitatively compare the types, severity, and relevance of usability problems LLM-agents identified to those produced in three GenderMag workshops involving nine participants. Findings show a broad overlap in detecting usability issues of humans and LLM-agents that are not gender specific via the generic interfaces. Agents thereby consistently assign significantly higher severity and relevance ratings. For the intentionally flawed interface, humans and LLM-agents assign similar ratings but the overlap between human and agent gender-related usability issues was low, with each missing issues the other caught. The agent system is an efficient tool for gender-related usability evaluation that humans may overlook, thereby expanding the coverage of evaluation.<\/jats:p>","DOI":"10.1145\/3801978","type":"journal-article","created":{"date-parts":[[2026,3,13]],"date-time":"2026-03-13T16:39:14Z","timestamp":1773419954000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["AI Can See What You Can\u2019t See: How LLM-Agents Complement Human-Based Gender-Inclusive Usability Testing"],"prefix":"10.1145","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-6723-8253","authenticated-orcid":false,"given":"Joy","family":"Geuenich","sequence":"first","affiliation":[{"name":"Chemnitz University of Technology, Chemnitz, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4421-1737","authenticated-orcid":false,"given":"Frank","family":"Joublin","sequence":"additional","affiliation":[{"name":"Honda Research Institute Europe GmbH, Offenbach, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1075-459X","authenticated-orcid":false,"given":"Antonello","family":"Ceravola","sequence":"additional","affiliation":[{"name":"Honda Research Institute Europe GmbH, Offenbach, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7628-5247","authenticated-orcid":false,"given":"Stefan","family":"Brandenburg","sequence":"additional","affiliation":[{"name":"Chemnitz University of Technology, Chemnitz, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-5175-5962","authenticated-orcid":false,"given":"Jonathan","family":"Dodge","sequence":"additional","affiliation":[{"name":"Penn State University, University Park, Pennsylvania, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,5,11]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3663740"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i16.29720"},{"key":"e_1_3_2_4_2","unstructured":"Inc. BigBlueButton. 2025. Retrieved August 22 2025 from https:\/\/bigbluebutton.org\/"},{"key":"e_1_3_2_5_2","unstructured":"Jonathan Bratt and Jon Harmon. 2019. RBERT: Implementation of BERT in R (Version 0.1.11). [R package]. Retrieved from https:\/\/github.com\/jonathanbratt\/RBERT"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3399715.3400871"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858274"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1093\/iwc\/iwv046"},{"key":"e_1_3_2_9_2","unstructured":"Antonello Ceravola Frank Joublin Ahmed R. Sadik Bram Bolder and Juha-Pekka Tolvanen. 2024. HyperGraphOS: A meta operating system for science and engineering. arXiv:2412.04923. Retrieved from https:\/\/arxiv.org\/abs\/2412.04923"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbvi.2021.e00276"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3632620.3671117"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00128"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.5555\/553473"},{"key":"e_1_3_2_14_2","unstructured":"Intel Corporation. 2025. Dynamic Analysis vs. Static Analysis. Retrieved December 19 2025 from https:\/\/www.intel.com\/content\/www\/us\/en\/docs\/inspector\/user-guide-windows\/2024-2\/dynamic-analysis-vs-static-analysis.html"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-49304-6_6"},{"key":"e_1_3_2_16_2","unstructured":"Digitalzentrum Fokus Mensch. 2025. Fairness und Diversit\u00e4t. Retrieved December 4 2024 from https:\/\/www. digitalzentrum-fokus-mensch.de\/themen\/fairness-und-diversitaet"},{"key":"e_1_3_2_17_2","unstructured":"DIN. 2019. DIN EN ISO 9241-210:2019 Ergonomie der Mensch-System-Interaktion\u2014Teil 210: Menschzentrierte Gestaltung interaktiver Systeme. ISO 9241-210:2019."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3691620.3695599"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3027063.3053354"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642782"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3660515.3661329"},{"key":"e_1_3_2_22_2","doi-asserted-by":"crossref","unstructured":"Tao Ge Xin Chan Xiaoyang Wang Dian Yu Haitao Mi and Dong Yu. 2024. Scaling synthetic data creation with 1 000 000 000 personas. arXiv:2406.20094. Retrieved from https:\/\/arxiv.org\/abs\/2406.20094","DOI":"10.14218\/GE.2024.00035"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1177\/1541931213601853"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2016.09.009"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4842-9651-6_27"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3772074"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-68213-6_19"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1080\/0144929X.2013.783114"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025609"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3373625.3416996"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.3390\/electronics13234633"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/503112.503114"},{"key":"e_1_3_2_33_2","first-page":"223","article-title":"Nouvelles recherches sur la distribution florale","volume":"44","author":"Jaccard Paul","year":"1908","unstructured":"Paul Jaccard. 1908. Nouvelles recherches sur la distribution florale. Bulletin de la Societe Vaudoise des Sciences Naturelles 44 (1908), 223\u2013270.","journal-title":"Bulletin de la Societe Vaudoise des Sciences Naturelles"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3708359.3712160"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510458.3513016"},{"key":"e_1_3_2_36_2","unstructured":"A. Baki Kocaballi. 2023. Conversational AI-powered design: ChatGPT as designer user and product. arXiv:2302.07406. Retrieved from https:\/\/arxiv.org\/abs\/2302.07406"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3695765"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.3233\/EFI-240014"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3501825"},{"key":"e_1_3_2_40_2","unstructured":"Yuxuan Lu Bingsheng Yao Hansu Gu Jing Huang Jessie Wang Laurence Li Jiri Gesi Qi He Toby Jia-Jun Li and Dakuo Wang. 2025. UXAgent: An LLM agent-based usability testing framework for web design. arXiv:2502.12561. Retrieved from https:\/\/arxiv.org\/abs\/2502.12561"},{"key":"e_1_3_2_41_2","first-page":"115","volume-title":"Proceedings of the AHFE 2016 International Conference on Ergonomics Modeling, Usability & Special Populations","author":"Minge Michael","year":"2016","unstructured":"Michael Minge, Manfred Th\u00fcring, Ingmar Wagner, and Carina V. Kuhr. 2016. The meCUE questionnaire: A modular tool for measuring user experience. In Proceedings of the AHFE 2016 International Conference on Ergonomics Modeling, Usability & Special Populations. Springer, 115\u2013128."},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639097"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/259963.260531"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/97243.97281"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.dss.2013.05.003"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3526113.3545616"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/1124772.1124942"},{"key":"e_1_3_2_48_2","volume-title":"The Persona Lifecycle: Keeping People in Mind Throughout Product Design","author":"Pruitt John","year":"2005","unstructured":"John Pruitt and Tamara Adlin. 2005. The Persona Lifecycle: Keeping People in Mind Throughout Product Design. Morgan Kaufmann Publishers Inc."},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.18420\/MUC2017-UP-0161"},{"key":"e_1_3_2_50_2","volume-title":"Artificial Intelligence: A Modern Approach","author":"Russell Stuart J.","year":"2010","unstructured":"Stuart J. Russell, Peter Norvig, and Ernest Davis. 2010. Artificial Intelligence: A Modern Approach (3rd ed.). Prentice Hall, Upper Saddle River.","edition":"3"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613905.3650860"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1098\/rsos.231053"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.3102\/10769986027001077"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1080\/00207590701396674"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445497"},{"key":"e_1_3_2_56_2","unstructured":"Dakuo Wang Ting-Yao Hsu Yuxuan Lu Hansu Gu Limeng Cui Yaochen Xie William Headean Bingsheng Yao Akash Veeragouni Jiapeng Liu et al. [n. d.]. AgentA\/B: Automated and scalable web A\/BTesting with interactive LLM agents. arXiv:2504.09723. Retrieved from https:\/\/arxiv.org\/abs\/2504.09723"},{"key":"e_1_3_2_57_2","doi-asserted-by":"crossref","unstructured":"Ronald L. Wasserstein and Nicole A. Lazar. 2016. The ASA statement on p-values: Context process and purpose. The American Statistician 70 (2016) 129\u2013133.","DOI":"10.1080\/00031305.2016.1154108"},{"key":"e_1_3_2_58_2","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, Vol. 35, 24824\u201324837.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/1952383.1952384"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642481"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3706598.3714327"},{"key":"e_1_3_2_62_2","first-page":"11809","volume-title":"Advances in Neural Information Processing Systems","author":"Yao Shunyu","year":"2023","unstructured":"Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, Vol. 36, 11809\u201311822. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2023\/file\/271db9922b8d1f4dd7aaef84ed5ac703-Paper-Conference.pdf"},{"key":"e_1_3_2_63_2","volume-title":"Case Study Research: Design and Methods (Applied Social Research Methods)","author":"Yin Robert K.","year":"2008","unstructured":"Robert K. Yin. 2008. Case Study Research: Design and Methods (Applied Social Research Methods) (4th ed.). Sage Publications. Retrieved from http:\/\/www.amazon.de\/Case-Study-Research-Methods-Applied\/dp\/1412960991%3FSubscriptionId%3D13CT5CVB80YFWJEPWS02%26tag%3Dws%26linkCode%3Dxm2%26camp%3D2025%26creative%3D165953%26creativeASIN%3D1412960991","edition":"4"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3664476.3670943"}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3801978","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,16]],"date-time":"2026-05-16T14:32:42Z","timestamp":1778941962000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3801978"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,11]]},"references-count":63,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3801978"],"URL":"https:\/\/doi.org\/10.1145\/3801978","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"value":"2160-6455","type":"print"},{"value":"2160-6463","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,11]]},"assertion":[{"value":"2025-08-25","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-03-02","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-05-11","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}