{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,21]],"date-time":"2026-04-21T15:34:51Z","timestamp":1776785691023,"version":"3.51.2"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,12,9]],"date-time":"2022-12-09T00:00:00Z","timestamp":1670544000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,12,9]],"date-time":"2022-12-09T00:00:00Z","timestamp":1670544000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia","award":["UIDB\/00408\/2020"],"award-info":[{"award-number":["UIDB\/00408\/2020"]}]},{"name":"Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia","award":["UIDP\/00408\/2020"],"award-info":[{"award-number":["UIDP\/00408\/2020"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Univ Access Inf Soc"],"published-print":{"date-parts":[[2024,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Evaluating the accessibility of web resources is usually done by checking the conformance of the resource against a standard or set of guidelines (e.g., the WCAG 2.1). The result of the evaluation will indicate what guidelines are respected (or not) by the resource. While it might hint at the accessibility level of web resources, often it will be complicated to compare the level of accessibility of different resources or of different versions of the same resource from evaluation reports. Web accessibility metrics synthesize the accessibility level of a web resource into a quantifiable value. The fact that there is a wide number of accessibility metrics, makes it challenging to choose which ones to use. In this paper, we explore the relationship between web accessibility metrics. For that purpose, we investigated eleven web accessibility metrics. The metrics were computed from automated accessibility evaluations obtained using QualWeb. A set of around three million web pages were evaluated. By computing the metrics over this sample of nearly three million web pages, it was possible to identify groups of metrics that offer similar results. Our analysis shows that there are metrics that behave similarly, which, when deciding what metrics to use, assists in picking the metric that is less resource intensive or for which it might be easier to collect the inputs.<\/jats:p>","DOI":"10.1007\/s10209-022-00956-x","type":"journal-article","created":{"date-parts":[[2022,12,9]],"date-time":"2022-12-09T05:02:56Z","timestamp":1670562176000},"page":"411-434","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Large-scale study of web accessibility metrics"],"prefix":"10.1007","volume":"23","author":[{"given":"Beatriz","family":"Martins","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1370-5379","authenticated-orcid":false,"given":"Carlos","family":"Duarte","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,12,9]]},"reference":[{"key":"956_CR1","unstructured":"W3C: Introduction to Web Accessibility. (2005). Accessed in 28 of December of 2021. https:\/\/www.w3.org\/WAI\/fundamentals\/accessibility-intro\/"},{"key":"956_CR2","unstructured":"Henry, S.L.: Web Content Accessibility Guidelines (WCAG) Overview. (2021). Accessed in 28 of December of 2021. https:\/\/www.w3.org\/WAI\/standards-guidelines\/wcag\/"},{"key":"956_CR3","unstructured":"Vigo, M., Brajnik, G., Connor, J.O.: Research report on web accessibility metrics. In: Vigo, M., Brajnik, G., eds., J.O.C. (eds.) W3C WAI Symposium on Website Accessibility Metrics, First public working draft edn. W3C WAI Research and Development Working Group (RDWG) Notes. W3C Web Accessibility Initiative (WAI), ??? (2012). http:\/\/www.w3.org\/TR\/accessibility-metrics-report"},{"key":"956_CR4","doi-asserted-by":"publisher","unstructured":"Vigo, M., Brajnik, G.: Automatic web accessibility metrics: Where we are and where we can go. Interacting with Computers 23(2), 137\u2013155 (2011) https:\/\/arxiv.org\/abs\/https:\/\/academic.oup.com\/iwc\/article-pdf\/23\/2\/137\/2238626\/iwc23-0137.pdf. https:\/\/doi.org\/10.1016\/j.intcom.2011.01.001","DOI":"10.1016\/j.intcom.2011.01.001"},{"key":"956_CR5","doi-asserted-by":"publisher","unstructured":"Lopes, R., Gomes, D., Carri\u00e7o, L.: Web not for all: A large scale study of web accessibility. In: Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A). W4A \u201910. Association for Computing Machinery, New York, NY, USA (2010). https:\/\/doi.org\/10.1145\/1805986.1806001","DOI":"10.1145\/1805986.1806001"},{"key":"956_CR6","doi-asserted-by":"publisher","DOI":"10.1007\/s12528-017-9151-3","author":"R Kimmons","year":"2017","unstructured":"Kimmons, R.: Open to all? nationwide evaluation of high-priority web accessibility considerations among higher education websites. J. Comput. Higher Educ. (2017). https:\/\/doi.org\/10.1007\/s12528-017-9151-3","journal-title":"J. Comput. Higher Educ."},{"key":"956_CR7","doi-asserted-by":"publisher","first-page":"266","DOI":"10.1007\/978-3-030-79816-1_33","volume-title":"Advances in Human Factors and System Interactions","author":"G Acosta-Vargas","year":"2021","unstructured":"Acosta-Vargas, G., Acosta-Vargas, P., Jad\u00e1n-Guerrero, J., Salvador-Ullauri, L., Gonzalez, M.: Improvement of accessibility in medical and healthcare websites. In: Nunes, I.L. (ed.) Advances in Human Factors and System Interactions, pp. 266\u2013273. Springer, Cham (2021)"},{"key":"956_CR8","doi-asserted-by":"publisher","unstructured":"Snaprud, M., Sawicka, A.: Large scale web accessibility evaluation - a european perspective. In: Stephanidis, C. (ed.) Universal Access in Human-Computer Interaction. Applications and Services, pp. 150\u2013159. Springer, Berlin, Heidelberg (2007). https:\/\/doi.org\/10.1007\/978-3-540-73283-9_18","DOI":"10.1007\/978-3-540-73283-9_18"},{"key":"956_CR9","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1007\/978-3-642-40483-2_23","volume-title":"Human-Computer Interaction - INTERACT 2013","author":"D Costa","year":"2013","unstructured":"Costa, D., Fernandes, N., Neves, S., Duarte, C., Hij\u00f3n-Neira, R., Carri\u00e7o, L.: Web accessibility in africa: A study of three african domains. In: Kotz\u00e9, P., Marsden, G., Lindgaard, G., Wesson, J., Winckler, M. (eds.) Human-Computer Interaction - INTERACT 2013, pp. 331\u2013338. Springer, Berlin, Heidelberg (2013)"},{"key":"956_CR10","doi-asserted-by":"publisher","unstructured":"Sirithumgul, P., Suchato, A., Punyabukkana, P.: Quantitative evaluation for web accessibility with respect to disabled groups. In: Proceedings of the 2009 International Cross-Disciplinary Conference on Web Accessibililty (W4A). W4A \u201909, pp. 136\u2013141. Association for Computing Machinery, New York, NY, USA (2009). https:\/\/doi.org\/10.1145\/1535654.1535687","DOI":"10.1145\/1535654.1535687"},{"key":"956_CR11","doi-asserted-by":"publisher","unstructured":"Freire, A.P., Bittar, T.J., Fortes, R.P.M.: An approach based on metrics for monitoring web accessibility in brazilian municipalities web sites. In: Proceedings of the 2008 ACM Symposium on Applied Computing. SAC \u201908, pp. 2421\u20132425. Association for Computing Machinery, New York, NY, USA (2008). https:\/\/doi.org\/10.1145\/1363686.1364259","DOI":"10.1145\/1363686.1364259"},{"key":"956_CR12","doi-asserted-by":"publisher","unstructured":"Song, S., Bu, J., Shen, C., Artmeier, A., Yu, Z., Zhou, Q.: Reliability aware web accessibility experience metric. In: Proceedings of the 15th International Web for All Conference. W4A \u201918. Association for Computing Machinery, New York, NY, USA (2018). https:\/\/doi.org\/10.1145\/3192714.3192836","DOI":"10.1145\/3192714.3192836"},{"key":"956_CR13","doi-asserted-by":"publisher","unstructured":"Freire, A.P., Fortes, R.P.M., Turine, M.A.S., Paiva, D.M.B.: An evaluation of web accessibility metrics based on their attributes. In: Proceedings of the 26th Annual ACM International Conference on Design of Communication. SIGDOC \u201908, pp. 73\u201380. Association for Computing Machinery, New York, NY, USA (2008). https:\/\/doi.org\/10.1145\/1456536.1456551","DOI":"10.1145\/1456536.1456551"},{"issue":"13","key":"956_CR14","doi-asserted-by":"publisher","first-page":"1394","DOI":"10.1002\/asi.20233","volume":"56","author":"B Parmanto","year":"2005","unstructured":"Parmanto, B., Zeng, X.: Metric for web accessibility evaluation. J. Am. Soc. Inform. Sci. Technol. 56(13), 1394\u20131404 (2005)","journal-title":"J. Am. Soc. Inform. Sci. Technol."},{"issue":"1","key":"956_CR15","first-page":"81","volume":"16","author":"HY Abuaddous","year":"2017","unstructured":"Abuaddous, H.Y., Jali, M.Z., Basir, N.: Quantitative metric for ranking web accessibility barriers based on their severity. J. Inform. Commun. Technol. 16(1), 81\u2013102 (2017)","journal-title":"J. Inform. Commun. Technol."},{"key":"956_CR16","doi-asserted-by":"publisher","unstructured":"Sullivan, T., Matson, R.: Barriers to use: Usability and content accessibility on the web\u2019s most popular sites. In: Proceedings on the 2000 Conference on Universal Usability. CUU \u201900, pp. 139\u2013144. Association for Computing Machinery, New York, NY, USA (2000). https:\/\/doi.org\/10.1145\/355460.355549","DOI":"10.1145\/355460.355549"},{"issue":"4","key":"956_CR17","doi-asserted-by":"publisher","first-page":"248","DOI":"10.1080\/17483100902903291","volume":"4","author":"M Vigo","year":"2009","unstructured":"Vigo, M., Brajnik, G., Arrue, M., Abascal, J.: Tool independence for the web accessibility quantitative metric. Disabil. Rehabil. Assist. Technol. 4(4), 248\u2013263 (2009). https:\/\/doi.org\/10.1080\/17483100902903291","journal-title":"Disabil. Rehabil. Assist. Technol."},{"key":"956_CR18","doi-asserted-by":"publisher","unstructured":"Vigo, M., Arrue, M., Brajnik, G., Lomuscio, R., Abascal, J.: Quantitative metrics for measuring web accessibility. In: Proceedings of the 2007 International Cross-Disciplinary Conference on Web Accessibility (W4A), pp. 99\u2013107. Association for Computing Machinery, New York, NY, USA (2007). https:\/\/doi.org\/10.1145\/1243441.1243465","DOI":"10.1145\/1243441.1243465"},{"key":"956_CR19","doi-asserted-by":"crossref","unstructured":"Velleman, E., Strobbe, C., Koch, J., Velasco, C.A., Snaprud, M.: A unified web evaluation methodology using wcag. In: Stephanidis, C. (ed.) Universal Access in Human-Computer Interaction. Applications and Services, pp. 177\u2013184. Springer, Berlin, Heidelberg (2007)","DOI":"10.1007\/978-3-540-73283-9_21"},{"key":"956_CR20","doi-asserted-by":"publisher","unstructured":"Freire, A.P., Power, C., Petrie, H., Tanaka, E.H., Rocha, H.V., Fortes, R.P.: Web accessibility metrics: Effects of different computational approaches. In: International Conference on Universal Access in Human-Computer Interaction, pp. 664\u2013673 (2009). https:\/\/doi.org\/10.1007\/978-3-642-02713-0_70. Springer","DOI":"10.1007\/978-3-642-02713-0_70"},{"key":"956_CR21","doi-asserted-by":"publisher","unstructured":"Mart\u00ednez, A.B., Juan, A.A., \u00c1lvarez, D., del Carmen\u00a0Su\u00e1rez, M.: Wab*: A quantitative metric based on wab. In: Gaedke, M., Grossniklaus, M., D\u00edaz, O. (eds.) Web Engineering, pp. 485\u2013488. Springer, Berlin, Heidelberg (2009). https:\/\/doi.org\/10.1007\/978-3-642-02818-2_44","DOI":"10.1007\/978-3-642-02818-2_44"},{"key":"956_CR22","doi-asserted-by":"publisher","unstructured":"B\u00fchler, C., Heck, H., Perlick, O., Nietzio, A., Ulltveit-Moe, N.: Interpreting results from large scale automatic evaluation of web accessibility. In: Miesenberger, K., Klaus, J., Zagler, W.L., Karshmer, A.I. (eds.) Computers Helping People with Special Needs, pp. 184\u2013191. Springer, Berlin, Heidelberg (2006). https:\/\/doi.org\/10.1007\/11788713_28","DOI":"10.1007\/11788713_28"},{"key":"956_CR23","doi-asserted-by":"publisher","unstructured":"Hackett, S., Parmanto, B., Zeng, X.: Accessibility of internet websites through time. In: Proceedings of the 6th International ACM SIGACCESS Conference on Computers and Accessibility. Assets \u201904, pp. 32\u201339. Association for Computing Machinery, New York, NY, USA (2003). https:\/\/doi.org\/10.1145\/1028630.1028638","DOI":"10.1145\/1028630.1028638"},{"key":"956_CR24","doi-asserted-by":"publisher","unstructured":"Brajnik, G., Vigo, M.: Automatic web accessibility metrics: Where we were and where we went. In: Web Accessibility, pp. 505\u2013521 (2019). https:\/\/doi.org\/10.1007\/978-1-4471-7440-0_27","DOI":"10.1007\/978-1-4471-7440-0_27"},{"key":"956_CR25","doi-asserted-by":"publisher","unstructured":"Bailey, J., Burd, E.: Tree-map visualisation for web accessibility. In: 29th Annual International Computer Software and Applications Conference (COMPSAC\u201905), vol. 1, pp. 275\u20132802 (2005). https:\/\/doi.org\/10.1109\/COMPSAC.2005.161","DOI":"10.1109\/COMPSAC.2005.161"},{"key":"956_CR26","doi-asserted-by":"publisher","unstructured":"Bailey, J., Burd, E.: Towards more mature web maintenance practices for accessibility. In: 2007 9th IEEE International Workshop on Web Site Evolution, pp. 81\u201387 (2007). https:\/\/doi.org\/10.1109\/WSE.2007.4380248","DOI":"10.1109\/WSE.2007.4380248"},{"key":"956_CR27","doi-asserted-by":"publisher","unstructured":"Brajnik, G., Lomuscio, R.: Samba: A semi-automatic method for measuring barriers of accessibility. In: Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility. Assets \u201907, pp. 43\u201350. Association for Computing Machinery, New York, NY, USA (2007). https:\/\/doi.org\/10.1145\/1296843.1296853","DOI":"10.1145\/1296843.1296853"},{"key":"956_CR28","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1007\/11788713_24","volume-title":"Computers Helping People with Special Needs","author":"G Brajnik","year":"2006","unstructured":"Brajnik, G.: Web accessibility testing: When the method is the culprit. In: Miesenberger, K., Klaus, J., Zagler, W.L., Karshmer, A.I. (eds.) Computers Helping People with Special Needs, pp. 156\u2013163. Springer, Berlin, Heidelberg (2006)"},{"key":"956_CR29","doi-asserted-by":"publisher","unstructured":"Song, S., Wang, C., Li, L., Yu, Z., Lin, X., Bu, J.: Waem: A web accessibility evaluation metric based on partial user experience order. In: Proceedings of the 14th Web for All Conference on The Future of Accessible Work. W4A \u201917. Association for Computing Machinery, New York, NY, USA (2017). https:\/\/doi.org\/10.1145\/3058555.3058576","DOI":"10.1145\/3058555.3058576"},{"key":"956_CR30","unstructured":"Battistelli, M., Mirri, S., Muratori, L.A., Salomoni, P.: Measuring Accessibility Barriers on Large Scale Sets of Pages. (2011). Accessed in 28 of December of 2021. https:\/\/www.w3.org\/WAI\/RD\/2011\/metrics\/paper2\/"},{"key":"956_CR31","unstructured":"Vigo, M., Abascal, J., Aizpurua, A., Arrue, M.: Attaining Metric Validity and Reliability with the Web Accessibility Quantitative Metric. (2011). Accessed in 28 of December of 2021. https:\/\/www.w3.org\/WAI\/RD\/2011\/metrics\/paper6\/"},{"key":"956_CR32","doi-asserted-by":"publisher","unstructured":"Fukuda, K., Saito, S., Takagi, H., Asakawa, C.: Proposing new metrics to evaluate web usability for the blind. In: CHI \u201905 Extended Abstracts on Human Factors in Computing Systems, pp. 1387\u20131390. Association for Computing Machinery, New York, NY, USA (2005). https:\/\/doi.org\/10.1145\/1056808.1056923","DOI":"10.1145\/1056808.1056923"},{"key":"956_CR33","doi-asserted-by":"publisher","unstructured":"Lopes, R., Carri\u00e7o, L.: The impact of accessibility assessment in macro scale universal usability studies of the web. In: Proceedings of the 2008 International Cross-Disciplinary Conference on Web Accessibility (W4A), pp. 5\u201314. Association for Computing Machinery, New York, NY, USA (2008). https:\/\/doi.org\/10.1145\/1368044.1368048","DOI":"10.1145\/1368044.1368048"},{"key":"956_CR34","unstructured":"Benavidez, C.: Libro Blanco de eXaminator, (2012)"},{"key":"956_CR35","doi-asserted-by":"publisher","unstructured":"Mirri, S., Muratori, L.A., Salomoni, P.: Monitoring accessibility: Large scale evaluations at a geo political level. In: The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility. ASSETS \u201911, pp. 163\u2013170. Association for Computing Machinery, New York, NY, USA (2011). https:\/\/doi.org\/10.1145\/2049536.2049566","DOI":"10.1145\/2049536.2049566"},{"key":"956_CR36","unstructured":"Lazar, J., Goldstein, D., Taylor, A.: Ensuring Digital Accessibility Through Process and Policy. Morgan kaufmann, ??? (2015)"},{"key":"956_CR37","doi-asserted-by":"publisher","unstructured":"Fernandes, N., Costa, D., Neves, S., Duarte, C., Carri\u00e7o, L.: Evaluating the accessibility of rich internet applications. In: Proceedings of the International Cross-Disciplinary Conference on Web Accessibility. W4A \u201912. Association for Computing Machinery, New York, NY, USA (2012). https:\/\/doi.org\/10.1145\/2207016.2207019","DOI":"10.1145\/2207016.2207019"},{"key":"956_CR38","unstructured":"Statstutor: Spearman\u2019s Correlation. (2021). Accessed in 28 of December of 2021. https:\/\/www.statstutor.ac.uk\/resources\/uploaded\/spearmans.pdf"},{"key":"956_CR39","unstructured":"Wikipedia: Hierarchical Clustering. (2021). Accessed in 28 of December of 2021. https:\/\/en.wikipedia.org\/wiki\/Hierarchical_clustering"},{"key":"956_CR40","doi-asserted-by":"crossref","unstructured":"Hackett, S., Parmanto, B.: Homepage not enough when evaluating web site accessibility. Internet Research (2009)","DOI":"10.1108\/10662240910927830"},{"key":"956_CR41","doi-asserted-by":"publisher","unstructured":"Abascal, J., Arrue, M., Valencia, X.: Tools for web accessibility evaluation. In: Yesilada, Y., Harper, S. (eds.) Web Accessibility: A Foundation for Research, pp. 479\u2013503. Springer, London (2019). https:\/\/doi.org\/10.1007\/978-1-4471-7440-0_26","DOI":"10.1007\/978-1-4471-7440-0_26"}],"container-title":["Universal Access in the Information Society"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10209-022-00956-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10209-022-00956-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10209-022-00956-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,13]],"date-time":"2024-03-13T21:07:45Z","timestamp":1710364065000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10209-022-00956-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,9]]},"references-count":41,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,3]]}},"alternative-id":["956"],"URL":"https:\/\/doi.org\/10.1007\/s10209-022-00956-x","relation":{},"ISSN":["1615-5289","1615-5297"],"issn-type":[{"value":"1615-5289","type":"print"},{"value":"1615-5297","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,9]]},"assertion":[{"value":"24 November 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 December 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"On behalf of all authors, the corresponding author states that there is no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}