{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T16:08:16Z","timestamp":1789574896767,"version":"build-2803163510"},"reference-count":36,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T00:00:00Z","timestamp":1789516800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Psychol."],"abstract":"\n Introduction<\/jats:title>\n Cognitive models in interactive environments often rely on hard-coded symbolic task descriptions, pre-specified interface objects, and idealized action execution. This paper's primary contribution is EVisiTor, an AI-augmented extension of the VisiTor eyes-and-hands tool for ACT-R that automates two forms of work commonly performed by the modeler: perception of the live interface and acquisition of task knowledge.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Its visual-perception pathway converts interactive interface objects into ACT-R-compatible visicon features, while its knowledge-acquisition pathway converts human-readable instructions into executable declarative structures. A motor-execution pathway connects ACT-R actions to the operating system and supports probabilistic misexecution with perceptually grounded checking and correction. We use a previously studied Excel procedure as a behavioral case study.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Relative to an idealized perfect-user model, the EVisiTor-enabled model reduced task-level RMSE from 29.19 s to 17.37 s, human-SD-normalized RMSE from 1.24 to 0.66, and mean absolute per-subtask error from 23.1 s to 10.5 s, with smaller absolute error on 12 of 14 subtasks. A spreadsheet knowledge-acquisition analysis showed that prompting the LLMs with the instruction and UIA grounding can produce the intended action and live visual-object names for every scannable target; the file-name input was the sole exception because it has no stable UIA label. A six-task Windows analysis then tested the feasibility of EVisiTor beyond Excel. Broad instructions often resulted in alternative valid methods: exact action agreement ranged from 42.5%\u201363.6% without UIA and 18.8%\u201370.6% with UIA, although 41 of 42 generated parses in each condition were functionally sufficient. Detailed guides raised exact agreement to 70.0%\u201385.0% without UIA and 76.2%\u201390.0% with UIA, and validated chunks drove the task-independent model through all six tasks in live Windows applications.<\/jats:p>\n <\/jats:sec>\n \n Discussion<\/jats:title>\n Together, the results demonstrate EVisiTor's potential for automating interface perception and instruction-based knowledge acquisition for a symbolic cognitive model, while identifying instruction specificity and individual variability as targets for future work.<\/jats:p>\n <\/jats:sec>","DOI":"10.3389\/fpsyg.2026.1913953","type":"journal-article","created":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T15:36:25Z","timestamp":1789572985000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["AI-augmented ACT-R modeling in real interfaces: automating visual perception, knowledge acquisition, and motor misexecution"],"prefix":"10.3389","volume":"17","author":[{"given":"Amirreza","family":"Bagherzadeh","sequence":"first","affiliation":[{"name":"Department of Industrial and Manufacturing Engineering, The Pennsylvania State University, University Park","place":["PA, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Farnaz","family":"Tehranchi","sequence":"additional","affiliation":[{"name":"School of Engineering Design and Innovation, The Pennsylvania State University, University Park","place":["PA, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,9,16]]},"reference":[{"key":"B1","doi-asserted-by":"crossref","DOI":"10.1093\/acprof:oso\/9780195324259.001.0001","volume-title":"How Can the Human Mind Occur in the Physical Universe","author":"Anderson","year":"2007"},{"key":"B2","doi-asserted-by":"publisher","first-page":"1036","DOI":"10.1037\/0033-295X.111.4.1036","article-title":"An integrated theory of the mind","volume":"111","author":"Anderson","year":"2004","journal-title":"Psychol. Rev"},{"key":"B3","unstructured":"Windows agent arena: evaluating multi-modal OS agents at scale\n \n \n \n Bonatti\n R.\n \n \n Zhao\n D.\n \n \n Bonacci\n F.\n \n \n Dupont\n D.\n \n \n Abdali\n S.\n \n \n Li\n Y.\n \n \n arXiv\n \n 2024"},{"key":"B4","article-title":"\u201cCognitive architectures in HCI: present work and future directions,\u201d","volume-title":"Proceedings of the 11th international conference on human computer interaction","author":"Byrne","year":"2005"},{"key":"B5","doi-asserted-by":"publisher","first-page":"225","DOI":"10.1518\/155723409X448071","article-title":"A history and primer of human performance modeling","volume":"5","author":"Byrne","year":"2009","journal-title":"Rev. Hum. Factors Ergon"},{"key":"B6","doi-asserted-by":"publisher","first-page":"9313","DOI":"10.18653\/v1\/2024.acl-long.505","article-title":"\u201cSeeClick: harnessing GUI grounding for advanced visual GUI agents,\u201d","author":"Cheng","year":"2024","journal-title":"Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024)"},{"key":"B7","doi-asserted-by":"crossref","DOI":"10.52202\/075280-1220","article-title":"\u201cMind2Web: towards a generalist agent for the web,\u201d","volume-title":"Advances in Neural Information Processing Systems 36 (NeurIPS 2023)","author":"Deng","year":"2023"},{"key":"B8","doi-asserted-by":"publisher","first-page":"959","DOI":"10.1037\/0033-2909.130.6.959","article-title":"The role of representative design in an ecological approach to cognition","volume":"130","author":"Dhami","year":"2004","journal-title":"Psychol. Bull"},{"key":"B9","first-page":"205","article-title":"\u201cACT-CV: bridging the gap between cognitive models and the outer world,\u201d","volume-title":"Grundlagen und Anwendungen der Mensch-Technik-Interaktion: 10. Berliner Werkstatt Mensch-Maschine-Systeme","author":"Halbr\u00fcgge","year":"2013"},{"key":"B10","doi-asserted-by":"publisher","first-page":"721","DOI":"10.3389\/fpsyg.2020.00721","article-title":"The \u2018single real-world approach' and its problems: a critique of the term ecological validity","volume":"11","author":"Holleman","year":"2020","journal-title":"Front. Psychol"},{"key":"B11","first-page":"14281","article-title":"\u201cCogAgent: a visual language model for GUI agents,\u201d","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (CVPR 2024)","author":"Hong","year":"2024"},{"key":"B12","doi-asserted-by":"publisher","first-page":"1007","DOI":"10.3758\/s13428-013-0425-z","article-title":"Simplifying the interaction between cognitive models and task environments with the JSON network interface","volume":"46","author":"Hope","year":"2014","journal-title":"Behav. Res. Methods"},{"key":"B13","unstructured":"Computer use data - paradigm Shift AI [Dataset]\n \n \n \n Howland\n A.\n \n \n Thinnappan\n A.\n \n \n Mohammed\n J. S.\n \n \n Hugging Face\n \n 2025"},{"key":"B14","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1007\/s10462-018-9646-y","article-title":"40 years of cognitive architectures: core cognitive abilities and practical applications","volume":"53","author":"Kotseruba","year":"2020","journal-title":"Artif. Intell. Rev"},{"key":"B15","unstructured":"Laird\n J. E.\n \n \n The Soar Cognitive Architecture\n \n 2019"},{"key":"B16","unstructured":"Omniparser for pure vision based gui agent\n \n \n \n Lu\n Y.\n \n \n Yang\n J.\n \n \n Shen\n Y.\n \n \n Awadallah\n A.\n \n \n arXiv\n \n 2024"},{"key":"B17","volume-title":"Unified Theories of Cognition","author":"Newell","year":"1990"},{"key":"B18","unstructured":"gpt-oss-120b\n \n 2025"},{"key":"B19","unstructured":"Introducing GPT-5.5"},{"key":"B20","volume-title":"A novel training paradigm for knowledge and skills acquisition: hybrid schedules lead to better learning for some but not all tasks","author":"Paik","year":"2011"},{"key":"B21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2776891","article-title":"Predicting user performance and learning in human-computer interaction with the Herbal compiler","volume":"22","author":"Paik","year":"2015","journal-title":"ACM Trans. Comput. Hum. Interact"},{"key":"B22","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1007\/978-0-85729-883-6_5","article-title":"\u201cDetermining the number of simulation runs: treating simulations as theories by not sampling their behavior,\u201d","volume-title":"Human-in-the-Loop Simulations: Methods and Practice","author":"Ritter","year":"2011"},{"key":"B23","doi-asserted-by":"publisher","first-page":"e1488","DOI":"10.1002\/wcs.1488","article-title":"ACT-R: a cognitive architecture for modeling cognition","volume":"10","author":"Ritter","year":"2019","journal-title":"Wiley Interdiscipl. Rev. Cogn. Sci"},{"key":"B24","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1006\/ijhc.2001.0470","article-title":"A perception\/action substrate for cognitive modeling in HCI","volume":"55","author":"St. Amant","year":"2001","journal-title":"Int. J. Hum. Comput. Stud"},{"key":"B25","article-title":"\u201cImage processing in cognitive models with SegMan,\u201d","volume-title":"Proceedings of HCI international 2005: Vol. 4. Theories, models, and processes in HCI (Paper No. 1869)","author":"St. Amant","year":"2005"},{"key":"B26","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1007\/s41233-023-00059-2","article-title":"Towards enhancing ecological validity in user studies: a systematic review of guidelines and implications for QoE research","volume":"8","author":"Subramanian","year":"2023","journal-title":"Qual. User Exp"},{"key":"B27","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1093\/acprof:oso\/9780199794553.003.0001","article-title":"\u201cWhat is a cognitive architecture?,\u201d","author":"Sun","year":"2016","journal-title":"Anatomy of the Mind: Exploring Psychological Mechanisms and Processes With the Clarion Cognitive Architecture"},{"key":"B28","volume-title":"An eyes and hands model: extending visual and motor modules for cognitive architectures","author":"Tehranchi","year":"2020"},{"key":"B29","doi-asserted-by":"publisher","first-page":"e27","DOI":"10.1017\/S089006042300015X","article-title":"A user model to directly compare two unmodified interfaces: a study of including errors and error corrections in a cognitive user model","volume":"37","author":"Tehranchi","year":"2023","journal-title":"Artif. Intell. Eng. Des. Anal. Manuf"},{"key":"B30","first-page":"15","article-title":"\u201cAn eyes and hands model for cognitive architectures to interact with user interfaces,\u201d","volume-title":"MAICS 2017: the 28th modern artificial intelligence and cognitive science conference","author":"Tehranchi","year":"2017"},{"key":"B31","first-page":"162","article-title":"\u201cModeling visual search in interactive graphic interfaces: adding visual pattern matching algorithms to ACT-R,\u201d","volume-title":"Proceedings of the 16th International Conference on Cognitive Modeling (ICCM 2018)","author":"Tehranchi","year":"2018"},{"key":"B32","doi-asserted-by":"publisher","first-page":"30","DOI":"10.5898\/JHRI.2.1.Trafton","article-title":"ACT-R\/E: an embodied cognitive architecture for human-robot interaction","volume":"2","author":"Trafton","year":"2013","journal-title":"J. Hum. Robot Interact"},{"key":"B33","doi-asserted-by":"crossref","DOI":"10.52202\/079017-1650","article-title":"\u201cOSWorld: benchmarking multimodal agents for open-ended tasks in real computer environments,\u201d","volume-title":"Advances in neural information processing systems 37 (NeurIPS 2024)","author":"Xie","year":"2024"},{"key":"B34","doi-asserted-by":"publisher","first-page":"559","DOI":"10.1109\/TSE.2010.50","article-title":"GUI interaction testing: incorporating event context","volume":"37","author":"Yuan","year":"2011","journal-title":"IEEE Trans. Softw. Eng"},{"key":"B35","unstructured":"UFO: a UI-focused agent for Windows OS interaction\n \n \n \n Zhang\n C.\n \n \n Li\n L.\n \n \n He\n S.\n \n \n Zhang\n X.\n \n \n Qiao\n B.\n \n \n Qin\n S.\n \n \n arXiv\n \n 2024"},{"key":"B36","unstructured":"WebArena: a realistic web environment for building autonomous agents\n \n \n \n Zhou\n S.\n \n \n Xu\n F. F.\n \n \n Zhu\n H.\n \n \n Zhou\n X.\n \n \n Lo\n R.\n \n \n Sridhar\n A.\n \n \n arXiv\n \n 2023"}],"container-title":["Frontiers in Psychology"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpsyg.2026.1913953\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T15:36:34Z","timestamp":1789572994000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpsyg.2026.1913953\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,16]]},"references-count":36,"alternative-id":["10.3389\/fpsyg.2026.1913953"],"URL":"https:\/\/doi.org\/10.3389\/fpsyg.2026.1913953","relation":{},"ISSN":["1664-1078"],"issn-type":[{"value":"1664-1078","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,16]]},"article-number":"1913953"}}