{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T12:05:08Z","timestamp":1776168308298,"version":"3.50.1"},"reference-count":52,"publisher":"Association for Computing Machinery (ACM)","issue":"2","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Web"],"published-print":{"date-parts":[[2026,5,31]]},"abstract":"<jats:p>The Website Accessibility Conformance Evaluation Methodology (WCAG-EM) directs an auditor to select samples of web pages for accessibility evaluation. However, the sampling method of WCAG-EM cannot statistically claim to represent pages outside of the selected sample. To generate an optimal representative sample, we previously proposed OPTIMAL-EM. In this article, we apply the framework to explore the relationships between two metrics in the framework: web page complexity and accessibility. We first cluster pages by their structure by employing t-distributed Stochastic Neighbour Embedding (t-SNE) and Density-Based Spatial Clustering of Applications with Noise (DBSCAN) by representing pages using two approaches: one including all HTML elements and another focusing on block-level HTML elements. We then compare the average complexity of a cluster and the variance of the complexity of a given cluster against the average accessibility of that cluster. To measure their correlations, we conducted a two-stage evaluation: an initial experiment with a website of 388 pages and a validation study with three additional random sites of 500 pages. Our experiments demonstrate that clusters characterised by lower variability in complexity tend to exhibit fewer accessibility barriers, suggesting that standardised and templated web design could be more beneficial to accessibility than more complex pages. Furthermore, the main contribution of this article is showing that by identifying and focusing on clusters with high complexity variance, auditors can target their efforts more effectively, prioritising areas more likely to present accessibility challenges. By providing a more systematic and scalable sampling method, our approach can optimise conformance evaluations of large websites by reducing the human effort involved, focusing on a more targeted assessment of more representative pages.<\/jats:p>","DOI":"10.1145\/3799797","type":"journal-article","created":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T11:40:43Z","timestamp":1772106043000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["OPTIMAL-EM: Complexity-Driven Clustering for Optimised Web Accessibility Evaluation"],"prefix":"10.1145","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1193-6632","authenticated-orcid":false,"given":"Alexander","family":"Hambley","sequence":"first","affiliation":[{"name":"The University of Manchester","place":["Manchester, United Kingdom of Great Britain and Northern Ireland"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8501-0205","authenticated-orcid":false,"given":"Yeliz","family":"Yesilada","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Middle East Technical University Northern Cyprus Campus (METU-NCC)","place":["G\u00fczelyurt, Turkey"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4768-7623","authenticated-orcid":false,"given":"Markel","family":"Vigo","sequence":"additional","affiliation":[{"name":"The University of Manchester","place":["Manchester, United Kingdom of Great Britain and Northern Ireland"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9301-5049","authenticated-orcid":false,"given":"Simon","family":"Harper","sequence":"additional","affiliation":[{"name":"Computer Science, University of ManchesterSchool of Computer Science2.44 Kilburn Building","place":["Manchester, United Kingdom of Great Britain and Northern Ireland"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,4,14]]},"reference":[{"key":"e_1_3_3_2_2","unstructured":"Hayfa.Y. Abuaddous Mohd Zalisham Jali and N. Basir. 2013. Study of the accessibility diagnosis on the public higher institutions website in Malaysia. In Proceedings of the 4th International Conference on Computing and Informatics (ICOCI\u201913). 412\u2013417. Retrieved from https:\/\/repo.uum.edu.my\/id\/eprint\/12052"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","unstructured":"Hayfa. Y. Abuaddous Mohd Zalisham Jali and Nurlida Basir. 2016. Web Accessibility Challenges. International Journal of Advanced Computer Science and Applications 7 10 (2016). DOI:10.14569\/IJACSA.2016.071023","DOI":"10.14569\/IJACSA.2016.071023"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","unstructured":"Daniel E. Berlyne. 1960. Conflict arousal and curiosity. McGraw-Hill Book Company. DOI:10.1037\/11164-000","DOI":"10.1037\/11164-000"},{"key":"e_1_3_3_5_2","unstructured":"Council of European Union. 2016. Directive (EU) 2016\/2102 of the European Parliament and of the Council of 26 October 2016 on the accessibility of the websites and mobile applications of public sector bodies. Retrieved from https:\/\/eur-lex.europa.eu\/eli\/dir\/2016\/2102\/oj"},{"key":"e_1_3_3_6_2","doi-asserted-by":"crossref","unstructured":"Radu G. Cre\u0163ulescu Daniel I. Morariu Macarie Breazu and Daniel Volovici. 2019. DBSCAN algorithm for document clustering. International Journal of Advanced Statistics and IT&C for Economics and Life Sciences 9 1 (2019) 58\u201366.","DOI":"10.2478\/ijasitels-2019-0007"},{"key":"e_1_3_3_7_2","first-page":"226","volume-title":"kdd","author":"Ester Martin","year":"1996","unstructured":"Martin Ester, Hans-Peter Kriegel, J\u00f6rg Sander, Xiaowei Xu, et\u00a0al. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, Vol. 96. 226\u2013231."},{"key":"e_1_3_3_8_2","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1007\/978-3-031-05752-6_15","volume-title":"Big Data Analytics and Computational Intelligence for Cybersecurity","author":"Omer Albasheer Fawzia","year":"2022","unstructured":"Albasheer Fawzia Omer, H. Ahmed Mohammed, M. Ahmed Awadallah, Zia Khan, Said Ul Abrar, and Mian Dawood Shah. 2022. Big data mining using K-means and DBSCAN clustering techniques. In Big Data Analytics and Computational Intelligence for Cybersecurity. Springer, 231\u2013246."},{"key":"e_1_3_3_9_2","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1109\/CIC.2015.7410989","volume-title":"2015 Computing in Cardiology Conference (CinC)","author":"Ghassemi Mohammad M.","year":"2015","unstructured":"Mohammad M. Ghassemi, Roger G. Mark, and Shamim Nemati. 2015. A visualization of evolving clinical sentiment using vector representations of clinical notes. In 2015 Computing in Cardiology Conference (CinC). IEEE, 629\u2013632."},{"key":"e_1_3_3_10_2","unstructured":"Government Digital Service (GDS). 2017. How do automated accessibility checkers compare? (February2017). Retrieved from https:\/\/alphagov.github.io\/accessibility-tool-audit\/"},{"key":"e_1_3_3_11_2","unstructured":"GOV.UK. 2018. Understanding WCAG 2.1. Retrieved from https:\/\/www.gov.uk\/service-manual\/helping-people-to-use-your-service\/understanding-wcag"},{"key":"e_1_3_3_12_2","unstructured":"Yi Maggie Guo and Dianne Hall. 2009. Website complexity: Objective versus subjective measures. MWAIS 2009 Proceedings (2009) 28."},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3493612.3520452"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3587281.3587962"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3543507.3583508"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","unstructured":"Simon Harper Caroline Jay Eleni Michailidou and Huangmao Quan. 2013. Analysing the visual complexity of web pages using document structure. Behaviour & Information Technology 32 (052013) 491\u2013502. DOI:10.1080\/0144929X.2012.726647","DOI":"10.1080\/0144929X.2012.726647"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","unstructured":"Simon Harper Anwar Moon Markel Vigo Giorgio Brajnik and Yeliz Yesilada. 2015. DOM block clustering for enhanced sampling and evaluation. 1\u201310. DOI:10.1145\/2745555.2746649","DOI":"10.1145\/2745555.2746649"},{"key":"e_1_3_3_18_2","doi-asserted-by":"crossref","unstructured":"Christopher Heaps and Stephen Handel. 1999. Similarity and features of natural textures. Journal of Experimental Psychology: Human Perception and Performance 25 2 (1999) 299.","DOI":"10.1037\/\/0096-1523.25.2.299"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","unstructured":"Abid Ismail and K. S. Kuppusamy. 2022. Web accessibility investigation and identification of major issues of higher education websites with statistical measures: A case study of college websites. Journal of King Saud University - Computer and Information Sciences 34 3 (2022) 901\u2013911. DOI:10.1016\/j.jksuci.2019.03.011","DOI":"10.1016\/j.jksuci.2019.03.011"},{"key":"e_1_3_3_20_2","volume-title":"W3C Web Content Accessibility Guidelines (WCAG) 2.0","year":"2012","unstructured":"ISO\/IEC 40500:2012 2012. W3C Web Content Accessibility Guidelines (WCAG) 2.0. Standard. International Organization for Standardization, Geneva."},{"key":"e_1_3_3_21_2","doi-asserted-by":"crossref","unstructured":"Liping Jing Michael K. Ng and Joshua Z. Huang. 2010. Knowledge-based vector space model for text clustering. Knowledge and Information Systems 25 1 (2010) 35\u201355.","DOI":"10.1007\/s10115-009-0256-5"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","unstructured":"William Kruskal and Frederick Mosteller. 1979. Representative sampling III: The current statistical literature. 47 3 (2021\/04\/13\/1979) 245\u2013265. DOI:10.2307\/1402647","DOI":"10.2307\/1402647"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/223355.223680"},{"key":"e_1_3_3_24_2","doi-asserted-by":"crossref","unstructured":"Talia Lavie and Tal Oron-Gilad. 2013. Perceptions of electronic navigation displays. Behaviour & Information Technology 32 8 (2013) 800\u2013823.","DOI":"10.1080\/0144929X.2011.639806"},{"key":"e_1_3_3_25_2","unstructured":"Michael O. Leavitt Ben Shneiderman et\u00a0al. 2006. Research-based web design & usability guidelines. Background and Methodology (2006)."},{"key":"e_1_3_3_26_2","first-page":"1","volume-title":"2022 International Congress on Human\u2013Computer Interaction, Optimization and Robotic Applications (hora)","author":"Macako\u011flu \u015eevval Seray","year":"2022","unstructured":"\u015eevval Seray Macako\u011flu and Serhat Peker. 2022. Web accessibility performance analysis using web content accessibility guidelines and automated tools: A systematic literature review. In 2022 International Congress on Human\u2013Computer Interaction, Optimization and Robotic Applications (hora). IEEE, 1\u20138."},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","unstructured":"Grace Mbipom and Simon Harper. 2011. The interplay between web aesthetics and accessibility. ASSETS\u201911: Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility 147\u2013154. DOI:10.1145\/2049536.2049564","DOI":"10.1145\/2049536.2049564"},{"key":"e_1_3_3_28_2","doi-asserted-by":"crossref","unstructured":"Binu Melit Devassy Sony George and Peter Nussbaum. 2020. Unsupervised clustering of hyperspectral paper data using t-SNE. Journal of Imaging 6 5 (2020) 29.","DOI":"10.3390\/jimaging6050029"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","unstructured":"Eleni Michailidou Sukru Eraslan Yeliz Yesilada and Simon Harper. 2021. Automated prediction of visual complexity of web pages: Tools and evaluations. International Journal of Human\u2013Computer Studies 145 (2021) 102523. DOI:10.1016\/j.ijhcs.2020.102523","DOI":"10.1016\/j.ijhcs.2020.102523"},{"key":"e_1_3_3_30_2","doi-asserted-by":"crossref","unstructured":"Morten Moshagen and Meinald T. Thielsch. 2010. Facets of visual aesthetics. International Journal of Human\u2013Computer Studies 68 10 (2010) 689\u2013709.","DOI":"10.1016\/j.ijhcs.2010.05.006"},{"key":"e_1_3_3_31_2","volume-title":"Designing Web Usability: The Practice of Simplicity","author":"Nielsen Jakob","year":"1999","unstructured":"Jakob Nielsen. 1999. Designing Web Usability: The Practice of Simplicity. New Riders Publishing."},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","unstructured":"Christopher Olston and Marc Najork. 2010. Web Crawling. Found. Trends Inf. Retr. 4 3 (March2010) 175\u2013246. DOI:10.1561\/1500000017","DOI":"10.1561\/1500000017"},{"key":"e_1_3_3_33_2","unstructured":"F. Pedregosa G. Varoquaux A. Gramfort V. Michel B. Thirion O. Grisel M. Blondel P. Prettenhofer R. Weiss V. Dubourg et\u00a0al. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12 (2011) 2825\u20132830."},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207736"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557052"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","unstructured":"Alex Rodriguez and Alessandro Laio. 2014. Clustering by fast search and find of density peaks. Science 344 6191 (2014) 1492\u20131496. Retrieved from arXiv:https:\/\/www.science.org\/doi\/pdf\/10.1126\/science.1242072. DOI:10.1126\/science.1242072","DOI":"10.1126\/science.1242072"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSDE.2010.14"},{"key":"e_1_3_3_38_2","unstructured":"Peter Tarasewich. 2002. An investigation into web site design complexity and usability metrics. Quarterly Journal of Electronic Commerce (2002)."},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/1805986.1806029"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/MERCon.2016.7480134"},{"key":"e_1_3_3_41_2","doi-asserted-by":"crossref","unstructured":"Markel Vigo and Giorgio Brajnik. 2011. Automatic web accessibility metrics: Where we are and where we can go. Interacting with Computers 23 2 (2011) 137\u2013155.","DOI":"10.1016\/j.intcom.2011.01.001"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","unstructured":"Markel Vigo Justin Brown and Vivienne Conway. 2013. Benchmarking web accessibility evaluation tools: Measuring the harm of sole reliance on automated tests. W4A 2013 - International Cross-Disciplinary Conference on Web Accessibility. DOI:10.1145\/2461121.2461124","DOI":"10.1145\/2461121.2461124"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","unstructured":"Beat Vollenwyder Glena H. Iten Florian Br\u00fchlmann Klaus Opwis and Elisa D. Mekler. 2019. Salient beliefs influencing the intention to consider Web Accessibility. Computers in Human Behavior 92 (2019) 352\u2013360. DOI:10.1016\/j.chb.2018.11.016","DOI":"10.1016\/j.chb.2018.11.016"},{"key":"e_1_3_3_44_2","unstructured":"W3C World Wide Web Consortium. 2008. Web Content Accessibility Guidelines (WCAG) 2.0. Retrieved from https:\/\/www.w3.org\/TR\/WCAG20\/"},{"key":"e_1_3_3_45_2","volume-title":"A vocabulary and associated APIs for HTML and XHTML","author":"Consortium W3C World Wide Web","year":"2014","unstructured":"W3C World Wide Web Consortium. 2014. A vocabulary and associated APIs for HTML and XHTML. Technical Report. W3C. Retrieved from https:\/\/www.w3.org\/TR\/2014\/REC-html5-20141028\/Overview.html"},{"key":"e_1_3_3_46_2","unstructured":"W3C World Wide Web Consortium. 2014. Website Accessibility Conformance Evaluation Methodology (WCAG-EM) 1.0. Retrieved from https:\/\/www.w3.org\/TR\/WCAG-EM\/"},{"key":"e_1_3_3_47_2","unstructured":"W3C World Wide Web Consortium. 2020. Web Content Accessibility Guidelines (WCAG) 2.2. Retrieved from https:\/\/www.w3.org\/TR\/WCAG22\/"},{"key":"e_1_3_3_48_2","first-page":"435","volume-title":"International Symposium on Intelligence Computation and Applications","author":"Wang Le","year":"2007","unstructured":"Le Wang, Yan Jia, and Weihong Han. 2007. Instant message clustering based on extended vector space model. In International Symposium on Intelligence Computation and Applications. Springer, 435\u2013443."},{"key":"e_1_3_3_49_2","unstructured":"WebAIM. 2024. The WebAIM Million. Retrieved from https:\/\/webaim.org\/projects\/million\/"},{"key":"e_1_3_3_50_2","unstructured":"Michal Wroblewski. 2003. A hierarchical www pages clustering algorithm based on the vector space model. Master\u2019s thesis Poznan University of Technology Poland (2003)."},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","unstructured":"Ou Wu Weiming Hu and Lei Shi. 2013. Measuring the visual complexities of web pages. ACM Trans. Web 7 1 Article 1 (mar2013) 34 pages. DOI:10.1145\/2435215.2435216","DOI":"10.1145\/2435215.2435216"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","unstructured":"Yeliz Yesilada Giorgio Brajnik Markel Vigo and Simon Harper. 2015. Exploring perceptions of web accessibility: A survey approach. Behaviour & Information Technology 34 2 (2015) 119\u2013134. Retrieved from 10.1080\/0144929X.2013.848238. DOI:10.1080\/0144929X.2013.848238","DOI":"10.1080\/0144929X.2013.848238"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","unstructured":"Yanlong Zhang Hong Zhu and Sue Greenwood. 2004. Website complexity metrics for measuring navigability. Proceedings\u2014Fourth International Conference on Quality Software QSIC 2004 172\u2013179. DOI:10.1109\/QSIC.2004.40","DOI":"10.1109\/QSIC.2004.40"}],"container-title":["ACM Transactions on the Web"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3799797","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T11:23:40Z","timestamp":1776165820000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3799797"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,14]]},"references-count":52,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,5,31]]}},"alternative-id":["10.1145\/3799797"],"URL":"https:\/\/doi.org\/10.1145\/3799797","relation":{},"ISSN":["1559-1131","1559-114X"],"issn-type":[{"value":"1559-1131","type":"print"},{"value":"1559-114X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,14]]},"assertion":[{"value":"2025-01-20","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-02-17","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-04-14","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}