{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T05:04:56Z","timestamp":1750309496975,"version":"3.41.0"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2025,3,17]],"date-time":"2025-03-17T00:00:00Z","timestamp":1742169600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100013209","name":"Hellenic Foundation for Research and Innovation","doi-asserted-by":"crossref","award":["15706 RV4THINGS"],"award-info":[{"award-number":["15706 RV4THINGS"]}],"id":[{"id":"10.13039\/501100013209","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Austrian Science Fund (FWF) for the project \u201cHigh-dimensional statistical learning: New methods to advance economic and sustainability policies\u201d","award":["MUR PRIN project 20228FT78M DREAM"],"award-info":[{"award-number":["MUR PRIN project 20228FT78M DREAM"]}]},{"name":"iNEST"},{"name":"European Union Next-GenerationEU","award":["D.D. 1058 23\/06\/2022, ECS_00000043"],"award-info":[{"award-number":["D.D. 1058 23\/06\/2022, ECS_00000043"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Web"],"published-print":{"date-parts":[[2025,5,31]]},"abstract":"<jats:p>\n            Application development for the modern web involves sophisticated engineering workflows\u2014including user interface (UI) aspects. Such user interfaces comprise web elements that are typically created with HTML\/CSS markup and JavaScript-like languages, yielding web documents. Their testing entails performing checks to examine visual and structural parts of the resulting UI software against requirements such as usability, accessibility, performance, or, increasingly, compliance with standards. However, current techniques are largely\n            <jats:italic>ad hoc<\/jats:italic>\n            and tailor-made to specific classes of requirements or web technologies and extensively require human-in-the-loop qualitative evaluations. Web UI evaluation so far has lacked formal foundations, which would provide assurances of compliance with requirements in an automatic manner. To this end, we devise a methodology and accompanying technical framework for web UIs. In our approach, requirements are formally specified in a spatio-temporal logic able to capture both the layout of visual components as well as how they change over time, as a user interacts with them. The technique we advocate is independent of the underlying technologies a web application may be developed with, as well as the browser and operating system used. To concretely support the specification and evaluation of UI requirements, our framework is grounded on open source tools for instrumenting, analyzing, and reporting spatio-temporal behaviors in webpages. We demonstrate our approach in practice over web accessibility standards posing challenges for automated verification.\n          <\/jats:p>","DOI":"10.1145\/3708512","type":"journal-article","created":{"date-parts":[[2025,1,28]],"date-time":"2025-01-28T11:01:09Z","timestamp":1738062069000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Automated Monitoring of Web User Interfaces"],"prefix":"10.1145","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1146-4850","authenticated-orcid":false,"given":"Ennio","family":"Visconti","sequence":"first","affiliation":[{"name":"TU Wien, Wien, Austria"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9493-3404","authenticated-orcid":false,"given":"Christos","family":"Tsigkanos","sequence":"additional","affiliation":[{"name":"Department of Aerospace Science and Technology, University of Athens, Athens, Greece"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2263-9342","authenticated-orcid":false,"given":"Laura","family":"Nenzi","sequence":"additional","affiliation":[{"name":"University of Trieste, Trieste, Italy"}]}],"member":"320","published-online":{"date-parts":[[2025,3,17]]},"reference":[{"key":"e_1_3_2_2_2","article-title":"Towards a mathematics formalisation assistant using large language models","volume":"2211","author":"Agrawal Ayush","year":"2022","unstructured":"Ayush Agrawal, Siddhartha Gadgil, Navin Goyal, Ashvni Narayanan, and Anand Tadipatri. 2022. Towards a mathematics formalisation assistant using large language models. arXiv: 2211.07524. Retrieved from https:\/\/arxiv.org\/abs\/2211.07524","journal-title":"arXiv"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/2381716.2381782"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2018.00026"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3464974.3468448"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCNS.2016.2609138"},{"key":"e_1_3_2_7_2","first-page":"417","volume-title":"Runtime Verification","author":"Bartocci Ezio","year":"2020","unstructured":"Ezio Bartocci, Luca Bortolussi, Michele Loreti, Laura Nenzi, and Simone Silvetti. 2020. MoonLight: A lightweight tool for monitoring spatio-temporal properties. In Runtime Verification, Jyotirmoy Deshmukh and Dejan Ni\u010dkovi\u0107 (Eds.). Springer International, Cham, 417\u2013428."},{"key":"e_1_3_2_8_2","first-page":"60","volume-title":"Post-proceedings of the 7th Seminar on Advanced Techniques and Tools for Software Evolution (SATToSE\u201914) (CEUR Workshop Proceedings)","volume":"1354","author":"Bauersfeld Sebastian","year":"2014","unstructured":"Sebastian Bauersfeld and Tanja E. J. Vos. 2014. User interface level testing with TESTAR; What about more sophisticated action specification and selection? In Post-proceedings of the 7th Seminar on Advanced Techniques and Tools for Software Evolution (SATToSE\u201914) (CEUR Workshop Proceedings), Davide Di Ruscio and Vadim Zaytsev (Eds.), Vol. 1354. CEUR-WS.org, 60\u201378."},{"key":"e_1_3_2_9_2","unstructured":"Donald Beaver. 2020. Applied awareness: Test-driven GUI development using computer vision and cryptography. arXiv:2006.03725. Retrieved from https:\/\/arxiv.org\/abs\/2006.03725"},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1007\/3-540-49059-0_14","volume-title":"Tools and Algorithms for the Construction and Analysis of Systems","author":"Biere Armin","year":"1999","unstructured":"Armin Biere, Alessandro Cimatti, Edmund Clarke, and Yunshan Zhu. 1999. Symbolic model checking without BDDs. In Tools and Algorithms for the Construction and Analysis of Systems, W. Rance Cleaveland (Ed.). Springer, Berlin, 193\u2013207."},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2009.144"},{"key":"e_1_3_2_12_2","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1109\/HICSS.1997.663184","volume-title":"Proceedings of the 30th Hawaii International Conference on System Sciences: Advanced Technology Track\u2013Volume 5 (HICSS\u201997)","author":"Chang Tien-fu","year":"1997","unstructured":"Tien-fu Chang, Alejandro Danylyzsn, So Norimatsu, Jose Rivera, David Shepard, Anthony Lattanze, and James Tomayko. 1997. \u201cContinuous verification\u201d in mission critical software development. In Proceedings of the 30th Hawaii International Conference on System Sciences: Advanced Technology Track\u2013Volume 5 (HICSS\u201997). IEEE Computer Society, Los Alamitos, CA, 273."},{"key":"e_1_3_2_13_2","first-page":"222","volume-title":"Theoretical Computer Science","author":"Ciancia Vincenzo","year":"2014","unstructured":"Vincenzo Ciancia, Diego Latella, Michele Loreti, and Mieke Massink. 2014. Specifying and verifying properties of space. In Theoretical Computer Science, Josep Diaz, Ivan Lanese, and Davide Sangiorgi (Eds.). Springer, Berlin, 222\u2013235."},{"key":"e_1_3_2_14_2","unstructured":"European Commission. 2021. Web Accessibility Directive\u2014Standards and Harmonisation. Retrieved March 30 2022 from https:\/\/digital-strategy.ec.europa.eu\/en\/policies\/web-accessibility-directive-standards-and-harmonisation"},{"key":"e_1_3_2_15_2","unstructured":"California Senate Judiciary Committee et\u00a0al. 2018. California Consumer Privacy Act: AB 375 Legislative History. (2018)."},{"key":"e_1_3_2_16_2","volume-title":"Web Content Accessibility Guidelines (WCAG) 2.1","author":"Cooper Michael","year":"2018","unstructured":"Michael Cooper, Alastair Campbell, Joshue O\u2019Connor, and Andrew Kirkpatrick. 2018. Web Content Accessibility Guidelines (WCAG) 2.1. W3C Recommendation. W3C. Retrieved from https:\/\/www.w3.org\/TR\/2018\/REC-WCAG21-20180605\/"},{"key":"e_1_3_2_17_2","doi-asserted-by":"crossref","first-page":"264","DOI":"10.1007\/978-3-642-39799-8_19","volume-title":"Computer Aided Verification","author":"Donz\u00e9 Alexandre","year":"2013","unstructured":"Alexandre Donz\u00e9, Thomas Ferr\u00e8re, and Oded Maler. 2013. Efficient robust monitoring for STL. In Computer Aided Verification, Natasha Sharygina and Helmut Veith (Eds.). Springer, Berlin, 264\u2013279."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/267895.267914"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACSOS49614.2020.00038"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-09975-3"},{"key":"e_1_3_2_21_2","volume-title":"International Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ-2020) (Aeronautics ARC-E-DAA-TN77785)","author":"Giannakopoulou Dimitra","year":"2020","unstructured":"Dimitra Giannakopoulou, Anastasia Mavridou, Julian Rhein, Thomas Pressburger, Johann Schumann, and Nija Shi. 2020. Formal requirements elicitation with FRET. In International Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ-2020) (Aeronautics ARC-E-DAA-TN77785). National Aeronautics and Space Administration (NASA), Langley Research Center, Hampton VA 23681-2199, USA, 6."},{"key":"e_1_3_2_22_2","volume-title":"Selectors Level 3","author":"Glazman Daniel","year":"2018","unstructured":"Daniel Glazman, John Williams, Tantek \u00c7elik, Peter Linss, Ian Hickson, and Elika Etemad. 2018. Selectors Level 3. W3C Recommendation. W3C. Retrieved from https:\/\/www.w3.org\/TR\/2018\/REC-selectors-3-20181106\/"},{"key":"e_1_3_2_23_2","unstructured":"Ilya Grigorik. 2020. Introducing Web Vitals: Essential Metrics for a Healthy Site. Retrieved December 16 2021 from blog.chromium.org\/2020\/05\/introducing-web-vitals-essential-metrics.html"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2804939"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510171"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10209-022-00948-x"},{"key":"e_1_3_2_27_2","volume-title":"Ergonomics of Human-system Interaction\u2014Part 210: Human-centred Design for Interactive Systems","author":"Secretary ISO Central","year":"2019","unstructured":"ISO Central Secretary. 2019. Ergonomics of Human-system Interaction\u2014Part 210: Human-centred Design for Interactive Systems. Standard ISO 9241-210:2019. International Organization for Standardization, Geneva, CH."},{"key":"e_1_3_2_28_2","first-page":"1019","volume-title":"Proceedings of the 4th International Conference on Computational Linguistics and Intelligent Systems (COLINS\u201920). Volume I: Main Conference (CEUR Workshop Proceedings)","volume":"2604","author":"Ivanova Kateryna","year":"2020","unstructured":"Kateryna Ivanova, Galyna V. Kondratenko, Ievgen V. Sidenko, and Yuriy P. Kondratenko. 2020. Artificial intelligence in automated system for web-interfaces visual testing. In Proceedings of the 4th International Conference on Computational Linguistics and Intelligent Systems (COLINS\u201920). Volume I: Main Conference (CEUR Workshop Proceedings), Vasyl Lytvyn, Victoria Vysotska, Thierry Hamon, Natalia Grabar, Natalia Sharonova, Olga Cherednichenko, and Olga Kanishcheva (Eds.), Vol. 2604. CEUR-WS.org1019\u20131031."},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2022.2095833"},{"key":"e_1_3_2_30_2","series-title":"9th European Congress on Embedded Real Time Software and Systems (ERTS\u201918)","volume-title":"Proceedings of the","author":"Kaijser Henrik","year":"2018","unstructured":"Henrik Kaijser, Henrik L\u00f6nn, Peter Thorngren, Johan Ekberg, Maria Henningsson, and Mats Larsson. 2018. Towards simulation-based verification for continuous integration and delivery. In Proceedings of the9th European Congress on Embedded Real Time Software and Systems (ERTS\u201918). Hyper Articles en Ligne, Toulouse, France, 10."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC60848.2023.00081"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3466722"},{"key":"e_1_3_2_33_2","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1007\/978-3-319-19282-6_7","volume-title":"Coordination Models and Languages","author":"Lafuente Alberto Lluch","year":"2015","unstructured":"Alberto Lluch Lafuente, Michele Loreti, and Ugo Montanari. 2015. A fixpoint-based calculus for graph-shaped computational fields. In Coordination Models and Languages, Tom Holvoet and Mirko Viroli (Eds.). Springer International, Cham, 101\u2013116."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-23392-5_3"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-30206-3_12"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCAD.2020.3013053"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2945329"},{"key":"e_1_3_2_38_2","volume-title":"Accessibility for Everybody: Understanding the Section 508 Accessibility Requirements","author":"Mueller John","year":"2008","unstructured":"John Mueller. 2008. Accessibility for Everybody: Understanding the Section 508 Accessibility Requirements. Apress, London, United Kingdom."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.46298\/lmcs-18(1:4)2022"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/2699417"},{"key":"e_1_3_2_41_2","volume-title":"Automated Technology for Verification and Analysis","author":"Ni\u010dkovi\u0107 Dejan","year":"2020","unstructured":"Dejan Ni\u010dkovi\u0107 and Tomoya Yamaguchi. 2020. RTAMT: Online robustness monitors from STL. In Automated Technology for Verification and Analysis, Dang Van Hung and Oleg Sokolsky (Eds.). Springer International, Cham, 13."},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-08-052029-2.50008-5"},{"key":"e_1_3_2_43_2","unstructured":"Dan North. 2006. Introducing BDD. Retrieved March 18 2019 from https:\/\/dannorth.net\/introducing-bdd\/"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1201\/9781003145110"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3360577"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3296979.3192407"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3022671.2984010"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/276627.276647"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/2695664.2695913"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2020.1805876"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00766-017-0282-x"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3218585.3218673"},{"key":"e_1_3_2_53_2","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1007\/978-3-030-02450-5_2","volume-title":"Formal Methods and Software Engineering","author":"Snook Colin","year":"2018","unstructured":"Colin Snook, Thai Son Hoang, Dana Dghyam, Michael Butler, Tomas Fischer, Rupert Schlick, and Keming Wang. 2018. Behaviour-driven formal model development. In Formal Methods and Software Engineering, Jing Sun and Meng Sun (Eds.). Springer International, Cham, 21\u201336."},{"key":"e_1_3_2_54_2","volume-title":"WebDriver","author":"Stewart Simon","year":"2018","unstructured":"Simon Stewart and David Burns. 2018. WebDriver. W3C Working Draft. W3C. Retrieved from https:\/\/www.w3.org\/TR\/webdriver\/"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106299"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416579"},{"key":"e_1_3_2_57_2","first-page":"1","article-title":"Regulation (EU) 2016\/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95\/46\/EC (General Data Protection Regulation)","volume":"59","author":"Union European","unstructured":"European Union. 2016-05-04. Regulation (EU) 2016\/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95\/46\/EC (General Data Protection Regulation). Official J. L110 59 (2016-05-04), 1\u201388.","journal-title":"Official J. L110"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3559538"},{"key":"e_1_3_2_59_2","unstructured":"W3C Web Accessibility Initiative (WAI). 2023. Web Accessibility Evaluation Tools List. Retrieved May 5 2023 from https:\/\/www.w3.org\/WAI\/ER\/tools\/"},{"key":"e_1_3_2_60_2","volume-title":"HTML","author":"Web Hypertext Application Technology Working Group (WHATWG)","year":"2022","unstructured":"Web Hypertext Application Technology Working Group (WHATWG) 2022. HTML. Retrieved from https:\/\/html.spec.whatwg.org\/multipage\/webappapis.html#event-loopLiving Standard."},{"key":"e_1_3_2_61_2","article-title":"Harnessing the power of large language models for natural language to first-order logic translation","volume":"2305","author":"Yang Yuan","year":"2023","unstructured":"Yuan Yang, Siheng Xiong, Ali Payani, Ehsan Shareghi, and Faramarz Fekri. 2023. Harnessing the power of large language models for natural language to first-order logic translation. arXiv: 2305.15541. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:258888128","journal-title":"arXiv:"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/1622176.1622213"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00139"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468561"}],"container-title":["ACM Transactions on the Web"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3708512","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3708512","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:17:45Z","timestamp":1750295865000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3708512"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,17]]},"references-count":63,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,5,31]]}},"alternative-id":["10.1145\/3708512"],"URL":"https:\/\/doi.org\/10.1145\/3708512","relation":{},"ISSN":["1559-1131","1559-114X"],"issn-type":[{"type":"print","value":"1559-1131"},{"type":"electronic","value":"1559-114X"}],"subject":[],"published":{"date-parts":[[2025,3,17]]},"assertion":[{"value":"2023-07-27","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-10","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}