{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,6]],"date-time":"2026-06-06T06:09:11Z","timestamp":1780726151423,"version":"3.54.1"},"reference-count":36,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2012,3,1]],"date-time":"2012-03-01T00:00:00Z","timestamp":1330560000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Access. Comput."],"published-print":{"date-parts":[[2012,3]]},"abstract":"<jats:p>The Web Content Accessibility Guidelines (WCAG) 2.0 separate testing into both \u201cMachine\u201d and \u201cHuman\u201d audits; and further classify \u201cHuman Testability\u201d into \u201cReliably Human Testable\u201d and \u201cNot Reliably Testable\u201d; it is human testability that is the focus of this paper. We wanted to investigate the likelihood that \u201cat least 80% of knowledgeable human evaluators would agree on the conclusion\u201d of an accessibility audit, and therefore understand the percentage of success criteria that could be described as reliably human testable, and those that could not. In this case, we recruited twenty-five experienced evaluators to audit four pages for WCAG 2.0 conformance. These pages were chosen to differ in layout, complexity, and accessibility support, thereby creating a small but variable sample.<\/jats:p>\n          <jats:p>\n            We found that an 80% agreement between experienced evaluators almost never occurred and that the average agreement was at the 70--75% mark, while the error rate was around 29%. Further, trained\u2014but novice\u2014evaluators performing the same audits exhibited the same agreement to that of our more experienced ones, but a reduction on validity of 6--13% ; the validity that an untrained user would attain can only be a conjecture. Expertise appears to improve (by 19%) the ability to avoid false positives. Finally, pooling the results of two independent experienced evaluators would be the best option, capturing\n            <jats:italic>at most<\/jats:italic>\n            76% of the true problems and producing\n            <jats:italic>only<\/jats:italic>\n            24% of false positives. Any other\n            <jats:italic>independent<\/jats:italic>\n            combination of audits would achieve worse results.\n          <\/jats:p>\n          <jats:p>This means that an 80% target for agreement, when audits are conducted without communication between evaluators, is not attainable, even with experienced evaluators, when working on pages similar to the ones used in this experiment; that the error rate even for experienced evaluators is relatively high and further, that untrained accessibility auditors be they developers or quality testers from other domains, would do much worse than this.<\/jats:p>","DOI":"10.1145\/2141943.2141946","type":"journal-article","created":{"date-parts":[[2012,3,27]],"date-time":"2012-03-27T15:17:31Z","timestamp":1332861451000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":37,"title":["Is accessibility conformance an elusive property? A study of validity and reliability of WCAG 2.0"],"prefix":"10.1145","volume":"4","author":[{"given":"Giorgio","family":"Brajnik","sequence":"first","affiliation":[{"name":"University of Udine, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yeliz","family":"Yesilada","sequence":"additional","affiliation":[{"name":"Middle East Technical University, Northern Cyprus Campus, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Simon","family":"Harper","sequence":"additional","affiliation":[{"name":"University of Manchester, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2012,3,30]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Web Accessibility: A Foundation for Research","author":"Abou-Zahra S.","unstructured":"Abou-Zahra , S. 2008. Web accessibility evaluation . In Web Accessibility: A Foundation for Research 1 st Ed., S. Harper and Y. Yesilada, Eds ., Human-Computer Interaction Series, Springer , Chapter 7, 79--106. Abou-Zahra, S. 2008. Web accessibility evaluation. In Web Accessibility: A Foundation for Research 1st Ed., S. Harper and Y. Yesilada, Eds., Human-Computer Interaction Series, Springer, Chapter 7, 79--106.","edition":"1"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1805986.1806000"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10209-004-0105-y"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-85200-1_9"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1878803.1878813"},{"key":"e_1_2_1_6_1","first-page":"1","article-title":"The expertise effect on web accessibility evaluation methods","volume":"26","author":"Brajnik G.","year":"2011","unstructured":"Brajnik , G. , Yesilada , Y. , and Harper , S. 2011 . The expertise effect on web accessibility evaluation methods . Hum.-Comput. Interac. 26 , 3, 1 -- 38 . Brajnik, G., Yesilada, Y., and Harper, S. 2011. The expertise effect on web accessibility evaluation methods. Hum.-Comput. Interac. 26, 3, 1--38.","journal-title":"Hum.-Comput. Interac."},{"key":"e_1_2_1_7_1","unstructured":"Canty A. 2002. Resampling methods in r: The boot package. R News (newsletter of the R Project) 2 3 2--7. http:\/\/cran.r-project.org\/doc\/Rnews\/.  Canty A. 2002. Resampling methods in r: The boot package. R News (newsletter of the R Project) 2 3 2--7. http:\/\/cran.r-project.org\/doc\/Rnews\/."},{"key":"e_1_2_1_8_1","unstructured":"Cooper M. Reid L. G. Vanderheiden G. and Caldwell B. 2010. Understanding wcag 2.0. http:\/\/www.w3.org\/TR\/2010\/NOTE-UNDERSTANDING-WCAG20-20101014\/. Section \u201cUnderstanding Levels of Conformance \u201d http:\/\/www.w3.org\/TR\/UNDERSTANDING-WCAG20\/conformance.html&num;uc-levels-head.  Cooper M. Reid L. G. Vanderheiden G. and Caldwell B. 2010. Understanding wcag 2.0. http:\/\/www.w3.org\/TR\/2010\/NOTE-UNDERSTANDING-WCAG20-20101014\/. Section \u201cUnderstanding Levels of Conformance \u201d http:\/\/www.w3.org\/TR\/UNDERSTANDING-WCAG20\/conformance.html&num;uc-levels-head."},{"key":"e_1_2_1_9_1","unstructured":"Coyne K. and Nielsen J. 2001. How to conduct usability evaluations for accessibility: methodology guidelines for testing websites and intranets with users who use assistive technology. http:\/\/www.nngroup.com\/reports\/accessibility\/testing Nielsen Norman Group.  Coyne K. and Nielsen J. 2001. How to conduct usability evaluations for accessibility: methodology guidelines for testing websites and intranets with users who use assistive technology. http:\/\/www.nngroup.com\/reports\/accessibility\/testing Nielsen Norman Group."},{"key":"e_1_2_1_10_1","volume-title":"The Web: Access and inclusion for disabled people. Tech. rep., Disability Rights Commission (DRC), UK.","author":"DRC.","year":"2004","unstructured":"DRC. 2004 . The Web: Access and inclusion for disabled people. Tech. rep., Disability Rights Commission (DRC), UK. DRC. 2004. The Web: Access and inclusion for disabled people. Tech. rep., Disability Rights Commission (DRC), UK."},{"key":"e_1_2_1_11_1","volume-title":"Web Accessibility: A Foundation for Research","author":"Edwards A. D. N.","unstructured":"Edwards , A. D. N. 2008. Assistive technologies . In Web Accessibility: A Foundation for Research 1 st Ed. S. Harper and Y. Yesilada, Eds ., Human-Computer Interaction Series, Springer , Chapter 10, 142--162. Edwards, A. D. N. 2008. Assistive technologies. In Web Accessibility: A Foundation for Research 1st Ed. S. Harper and Y. Yesilada, Eds., Human-Computer Interaction Series, Springer, Chapter 10, 142--162.","edition":"1"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327590IJHC1501_13"},{"key":"e_1_2_1_13_1","volume-title":"Just Ask: Accessibility in the User-Centered Design Process","author":"Henry S.","year":"2004","unstructured":"Henry , S. and Grossnickle , M . 2004 . Just Ask: Accessibility in the User-Centered Design Process . Georgia Tech Research Corporation , Atlanta, GA . Online book: www.UIAccess.com\/AccessUCD. Henry, S. and Grossnickle, M. 2004. Just Ask: Accessibility in the User-Centered Design Process. Georgia Tech Research Corporation, Atlanta, GA. Online book: www.UIAccess.com\/AccessUCD."},{"key":"e_1_2_1_14_1","volume-title":"Human-Computer Interaction: Ergonomics and User Interfaces","volume":"1","author":"Hertzum M.","unstructured":"Hertzum , M. and Jacobsen , N . 1999. The evaluator effect during first-time use of the cognitive walkthrough technique . In Human-Computer Interaction: Ergonomics and User Interfaces , vol. 1 , Lawrence Erlbaum Associates, 1063--1067. Hertzum, M. and Jacobsen, N. 1999. The evaluator effect during first-time use of the cognitive walkthrough technique. In Human-Computer Interaction: Ergonomics and User Interfaces, vol. 1, Lawrence Erlbaum Associates, 1063--1067."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327590IJHC1304_05"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/506443.506534"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1080\/07370020802278205"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1735223.1735255"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/286498.286737"},{"key":"e_1_2_1_20_1","first-page":"107","article-title":"End user evaluations. In Web Accessibility: A Foundation for Research 1st Ed. S. Harper and Y. Yesilada, Eds., Human-Computer Interaction Series, Springer","volume":"8","author":"Jay C.","year":"2008","unstructured":"Jay , C. , Lunn , D. , and Michailidou , E. 2008 . End user evaluations. In Web Accessibility: A Foundation for Research 1st Ed. S. Harper and Y. Yesilada, Eds., Human-Computer Interaction Series, Springer , Chapter 8 , 107 -- 126 . Jay, C., Lunn, D., and Michailidou, E. 2008. End user evaluations. In Web Accessibility: A Foundation for Research 1st Ed. S. Harper and Y. Yesilada, Eds., Human-Computer Interaction Series, Springer, Chapter 8, 107--126.","journal-title":"Chapter"},{"key":"e_1_2_1_21_1","unstructured":"Lang T. 2003. Comparing website accessibility evaluation methods and learnings from usability evaluation methods. http:\/\/www.peakusability.com.au\/about-us\/pdf\/website_accessibility.pdf (accessed 5\/08).  Lang T. 2003. Comparing website accessibility evaluation methods and learnings from usability evaluation methods. http:\/\/www.peakusability.com.au\/about-us\/pdf\/website_accessibility.pdf (accessed 5\/08)."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2008.11.002"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/142750.142834"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1125021.1125117"},{"key":"e_1_2_1_25_1","volume-title":"Usability Engineering. Morgan Kaufmann","author":"Nielsen J.","unstructured":"Nielsen , J. 1993. Usability Engineering. Morgan Kaufmann , San Francisco . Nielsen, J. 1993. Usability Engineering. Morgan Kaufmann, San Francisco."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/97243.97281"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1240624.1240688"},{"key":"e_1_2_1_28_1","volume-title":"R: A Language and Environment for Statistical Computing","author":"Development Core Team","year":"2010","unstructured":"R Development Core Team . 2010 . R: A Language and Environment for Statistical Computing . R Foundation for Statistical Computing , Vienna, Austria . R Development Core Team. 2010. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria."},{"key":"e_1_2_1_29_1","unstructured":"Rubin J. and Chisnell D. 2008. Handbook of Usability Testing 2nd Ed. Wiley.   Rubin J. and Chisnell D. 2008. Handbook of Usability Testing 2nd Ed. Wiley."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15327590ijhc0903_2"},{"key":"e_1_2_1_31_1","volume-title":"Web Accessibility: Web Standards and Regulatory Compliance. FriendsofED.","author":"Thatcher J.","year":"2006","unstructured":"Thatcher , J. , Burks , M. , Heilmann , C. , Henry , S. , Kirkpatrick , A. , Lauke , P. , Lawson , B. , Regan , B. , Rutter , R. , Urban , M. , and Waddell , C . 2006 . Web Accessibility: Web Standards and Regulatory Compliance. FriendsofED. Thatcher, J., Burks, M., Heilmann, C., Henry, S., Kirkpatrick, A., Lauke, P., Lawson, B., Regan, B., Rutter, R., Urban, M., and Waddell, C. 2006. Web Accessibility: Web Standards and Regulatory Compliance. FriendsofED."},{"key":"e_1_2_1_32_1","unstructured":"W3C\/WAI. 2005a. Preliminary review of web sites for accessibility. http:\/\/www.w3.org\/WAI\/eval\/preliminary.html.  W3C\/WAI. 2005a. Preliminary review of web sites for accessibility. http:\/\/www.w3.org\/WAI\/eval\/preliminary.html."},{"key":"e_1_2_1_33_1","unstructured":"W3C\/WAI. 2005b. Requirements for WCAG 2.0 checklists techniques and test files. http:\/\/www.w3.org\/WAI\/GL\/WCAG20\/WD-wcag2-tech-req-20050125.html.  W3C\/WAI. 2005b. Requirements for WCAG 2.0 checklists techniques and test files. http:\/\/www.w3.org\/WAI\/GL\/WCAG20\/WD-wcag2-tech-req-20050125.html."},{"key":"e_1_2_1_34_1","unstructured":"W3C\/WAI. 2005c. Selecting web accessibility evaluation tools. http:\/\/www.w3.org\/WAI\/eval\/selectingtools.  W3C\/WAI. 2005c. Selecting web accessibility evaluation tools. http:\/\/www.w3.org\/WAI\/eval\/selectingtools."},{"key":"e_1_2_1_35_1","volume-title":"Conformance evaluation of web sites for accessibility","unstructured":"W3C\/WAI. 2005d. Conformance evaluation of web sites for accessibility . World Wide Web Consortium , Web Accessibility Initiative, http:\/\/www.w3.org\/WAI\/eval\/conformance.html. W3C\/WAI. 2005d. Conformance evaluation of web sites for accessibility. World Wide Web Consortium, Web Accessibility Initiative, http:\/\/www.w3.org\/WAI\/eval\/conformance.html."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1639642.1639678"}],"container-title":["ACM Transactions on Accessible Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2141943.2141946","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2141943.2141946","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T19:08:04Z","timestamp":1750273684000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2141943.2141946"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,3]]},"references-count":36,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2012,3]]}},"alternative-id":["10.1145\/2141943.2141946"],"URL":"https:\/\/doi.org\/10.1145\/2141943.2141946","relation":{},"ISSN":["1936-7228","1936-7236"],"issn-type":[{"value":"1936-7228","type":"print"},{"value":"1936-7236","type":"electronic"}],"subject":[],"published":{"date-parts":[[2012,3]]},"assertion":[{"value":"2011-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2011-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2012-03-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}