{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T08:24:28Z","timestamp":1781079868640,"version":"3.54.1"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,3,28]],"date-time":"2023-03-28T00:00:00Z","timestamp":1679961600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Access. Comput."],"published-print":{"date-parts":[[2023,3,31]]},"abstract":"<jats:p>Several Web accessibility evaluation tools have been put forward to reduce the burden of identifying accessibility barriers for users, especially those with disabilities. One common issue in using accessibility evaluation tools in practice is that the results provided by different tools are sometimes unclear, and often diverging. Such limitations may confuse the users who may not understand the reasons behind them, and thus hamper the possible adoption of such tools. Hence, there is a need for tools that shed light on their actual functioning, and the success criteria and techniques supported. For this purpose, we must identify what criteria should be adopted in order for such tools to be transparent and to help users better interpret their results. In this paper, we discuss such issues, provide design criteria for obtaining user-centred and transparent accessibility evaluation tools, and analyse how they have been addressed by a representative set of open, license-free, accessibility tools. We also report on the results of a survey with 138 users of such tools, aimed at capturing the perceived usefulness of previously identified transparency requirements. Finally, we performed a user study with 18 users working in the Web design or accessibility fields with the goal of receiving more feedback about the transparency of a selected subset of accessibility tools.<\/jats:p>","DOI":"10.1145\/3556979","type":"journal-article","created":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T12:07:07Z","timestamp":1660738027000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":26,"title":["The Transparency of Automatic Web Accessibility Evaluation Tools: Design Criteria, State of the Art, and User Perception"],"prefix":"10.1145","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1029-9934","authenticated-orcid":false,"given":"Marco","family":"Manca","sequence":"first","affiliation":[{"name":"CNR-ISTI, HIIS Laboratory, Via Moruzzi, Pisa, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8008-2090","authenticated-orcid":false,"given":"Vanessa","family":"Palumbo","sequence":"additional","affiliation":[{"name":"CNR-ISTI, HIIS Laboratory, Via Moruzzi, Pisa, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8355-6909","authenticated-orcid":false,"given":"Fabio","family":"Patern\u00f2","sequence":"additional","affiliation":[{"name":"CNR-ISTI, HIIS Laboratory, Via Moruzzi, Pisa, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0556-7538","authenticated-orcid":false,"given":"Carmen","family":"Santoro","sequence":"additional","affiliation":[{"name":"CNR-ISTI, HIIS Laboratory, Via Moruzzi, Pisa, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,3,28]]},"reference":[{"key":"e_1_3_2_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4471-7440-0_26"},{"key":"e_1_3_2_3_1","unstructured":"Accessibility Scanning & Monitoring by UserWay. https:\/\/userway.org\/scanner (Last accessed 30 July 2021)."},{"key":"e_1_3_2_4_1","unstructured":"Accessi.org by Adam. https:\/\/www.accessi.org\/ (Last accessed 30 July 2021)."},{"key":"e_1_3_2_5_1","unstructured":"aCe by accessiBe. https:\/\/accessibe.com\/ (Last accessed 30 July 2021)."},{"key":"e_1_3_2_6_1","unstructured":"A11y Color Contrast Accessibility Validator. https:\/\/color.a11y.com\/?wc3 (Last accessed 1 March 2022)."},{"issue":"9","key":"e_1_3_2_7_1","first-page":"18","article-title":"Towards automated web accessibility evaluation: A comparative study","volume":"9","author":"Abduganiev S. G.","year":"2017","unstructured":"S. G. Abduganiev. 2017. Towards automated web accessibility evaluation: A comparative study. Int. J. Inf. Technol. Comput. Sci. (IJITCS) 9, 9 (2017), 18\u201344.","journal-title":"Int. J. Inf. Technol. Comput. Sci. (IJITCS)"},{"key":"e_1_3_2_8_1","unstructured":"S. Abou-Zahra. 2017. Evaluation and report language (EARL). Retrieved February 2 2017 from https:\/\/www.w3.org\/TR\/EARL10-Schema\/#OutcomeValue."},{"key":"e_1_3_2_9_1","unstructured":"ACT Rules. https:\/\/act-rules.github.io\/pages\/about\/ (Last accessed 30 July 2021)."},{"key":"e_1_3_2_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-92698-6_37"},{"key":"e_1_3_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3282894.3282921"},{"key":"e_1_3_2_12_1","unstructured":"A. Beirekdar J. Vanderdonckt and M. Noirhomme-Fraiture. 2002. Kwaresmi\u2013knowledge-based web automated evaluation with reconfigurable guidelines optimization. (Springer Ed.) DSV-IS 2545 362\u2013376."},{"key":"e_1_3_2_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/11555261_25"},{"key":"e_1_3_2_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10209-004-0105-y"},{"key":"e_1_3_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2141943.2141946"},{"key":"e_1_3_2_16_1","first-page":"505","article-title":"Automatic web accessibility metrics","author":"Brajnik G.","year":"2019","unstructured":"G. Brajnik and M. Vigo. 2019. Automatic web accessibility metrics. Where We Were and Where We Went. (Springer, Ed.) Web Accessibility, 505\u2013521.","journal-title":"Where We Were and Where We Went"},{"key":"e_1_3_2_17_1","doi-asserted-by":"publisher","DOI":"10.3389\/fcomp.2021.628770"},{"key":"e_1_3_2_18_1","unstructured":"EqualWeb. https:\/\/www.equalweb.com\/ (Last accessed 30 July 2021)."},{"key":"#cr-split#-e_1_3_2_19_1.1","unstructured":"EU Commission. 2016. Directive"},{"key":"#cr-split#-e_1_3_2_19_1.2","unstructured":"(EU) 2016\/2102 of the European Parliament and of the Council. Retrieved from https:\/\/eur-lex.europa.eu: https:\/\/eur-lex.europa.eu\/eli\/dir\/2016\/2102\/oj."},{"key":"e_1_3_2_20_1","unstructured":"Accessibility Checker by EXPERTE. https:\/\/www.experte.com\/accessibility (Last accessed 30 July 2021)."},{"key":"e_1_3_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2596695.2596698"},{"key":"e_1_3_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3371300.3383346"},{"key":"e_1_3_2_23_1","unstructured":"Free Web Accessibility Checker AlumniOnline Web Services. https:\/\/www.alumnionlineservices.com\/scanner\/ (Last accessed 30 July 2021)."},{"key":"e_1_3_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1535654.1535661"},{"key":"e_1_3_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397871"},{"key":"e_1_3_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1805986.1806019"},{"key":"e_1_3_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1744161.1744168"},{"key":"e_1_3_2_28_1","volume-title":"Joint Proceedings of the ACM IUI Workshops 2022","author":"Hellmann M.","year":"2022","unstructured":"M. Hellmann, D. C. Hernandez-Bocanegra, and J. Ziegler. 2022. Development of an instrument for measuring users\u2019 perception of transparency in recommender systems. Joint Proceedings of the ACM IUI Workshops 2022. March 2022, Helsinki, Finland."},{"key":"e_1_3_2_29_1","unstructured":"IBM Equal Access Accessibility Checker. https:\/\/www.ibm.com\/able\/toolkit\/tools\/ (Last accessed 30 July 2021)."},{"key":"e_1_3_2_30_1","first-page":"117","article-title":"Using automated tools to improve web site usage by users with diverse abilities","author":"Ivory M. Y.","year":"2003","unstructured":"M. Y. Ivory, J. Mankoff, and A. Le. 2003. Using automated tools to improve web site usage by users with diverse abilities. Human-Computer Interaction Institute 2003, 117.","journal-title":"Human-Computer Interaction Institute"},{"key":"e_1_3_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/355460.355559"},{"key":"e_1_3_2_32_1","first-page":"498","volume-title":"International Conference on Universal Access in Human-Computer Interaction","author":"Lazar J.","year":"2011","unstructured":"J. Lazar and A. Olalere. 2011. Investigation of best practices for maintaining section 508 compliance in us federal web sites. International Conference on Universal Access in Human-Computer Interaction. Berlin, Springer, 498\u2013506."},{"key":"e_1_3_2_33_1","doi-asserted-by":"crossref","unstructured":"Q. V. Liao D. M. Gruen and S. Miller. 2020. Questioning the AI: Informing design practices for explainable AI user experiences CHI 2020.","DOI":"10.1145\/3313831.3376590"},{"key":"e_1_3_2_34_1","unstructured":"MAUVE++. https:\/\/mauve.isti.cnr.it\/ (Last accessed 30 July 2021)."},{"key":"e_1_3_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300738"},{"key":"e_1_3_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2049536.2049566"},{"issue":"2","key":"e_1_3_2_37_1","first-page":"218","article-title":"Reliability in automated evaluation tools for web accessibility standards compliance","volume":"7","author":"Molinero A. M.","year":"2006","unstructured":"A. M. Molinero, F. G. Kohun, and R. Morris. 2006. Reliability in automated evaluation tools for web accessibility standards compliance. Issues in Information Systems 7, 2 (2006), 218\u2013222.","journal-title":"Issues in Information Systems"},{"key":"e_1_3_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3335595.3335651"},{"key":"e_1_3_2_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-41264-1_35"},{"key":"e_1_3_2_40_1","volume-title":"Proceedings of W3C Online Symposium on Website Accessibility Metrics","author":"Nietzio A.","year":"2011","unstructured":"A. Nietzio, M. Eibegger, M. Goodwin, and M. Snaprud. 2011. Towards a score function for WCAG 2.0 benchmarking. Proceedings of W3C Online Symposium on Website Accessibility Metrics. Retrieved from https:\/\/www.w3.org\/WAI\/RD\/2011\/metrics\/paper11."},{"key":"e_1_3_2_41_1","unstructured":"M. Padure and C. Pribeanu. 2019. Exploring the differences between five accessibility evaluation tools. (2019)."},{"key":"e_1_3_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3430263.3452436"},{"key":"e_1_3_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2745395"},{"key":"e_1_3_2_44_1","doi-asserted-by":"publisher","DOI":"10.3389\/fcomp.2021.605772"},{"key":"e_1_3_2_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-73283-9_15"},{"key":"e_1_3_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207736"},{"key":"e_1_3_2_47_1","unstructured":"Puppeteer. https:\/\/github.com\/puppeteer\/puppeteer (Last accessed 30 July 2021)."},{"key":"e_1_3_2_48_1","unstructured":"QualWeb. http:\/\/qualweb.di.fc.ul.pt\/ (Last accessed 30 July 2021)."},{"key":"e_1_3_2_49_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10209-014-0399-3"},{"key":"e_1_3_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445455"},{"key":"e_1_3_2_51_1","unstructured":"TAW by CTIC Technology Centre. https:\/\/www.tawdis.net\/ (Last accessed 30 July 2021)."},{"key":"e_1_3_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461121.2461124"},{"key":"e_1_3_2_53_1","doi-asserted-by":"publisher","DOI":"10.1080\/0144929X.2013.848238"},{"key":"e_1_3_2_54_1","unstructured":"WAVE. https:\/\/wave.webaim.org\/ (Last accessed 30 July 2021)."},{"key":"e_1_3_2_55_1","unstructured":"W3C WAETL. 2021. Web Accessibility Evaluation Tools List. https:\/\/www.w3.org\/WAI\/ER\/tools\/ (last accessed 20 October 2021)."}],"container-title":["ACM Transactions on Accessible Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3556979","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3556979","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:48:52Z","timestamp":1750286932000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3556979"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,28]]},"references-count":55,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,3,31]]}},"alternative-id":["10.1145\/3556979"],"URL":"https:\/\/doi.org\/10.1145\/3556979","relation":{},"ISSN":["1936-7228","1936-7236"],"issn-type":[{"value":"1936-7228","type":"print"},{"value":"1936-7236","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,28]]},"assertion":[{"value":"2021-11-17","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-08-08","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-03-28","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}