{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T02:28:45Z","timestamp":1778120925749,"version":"3.51.4"},"reference-count":31,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T00:00:00Z","timestamp":1541030400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Zhejiang Provincial Natural Science Foundation of China","award":["LZ13F020001"],"award-info":[{"award-number":["LZ13F020001"]}]},{"name":"National Key Technology R&D Program of China","award":["2012BAI34B01, 2014BAK15B02"],"award-info":[{"award-number":["2012BAI34B01, 2014BAK15B02"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61173185, 61173186"],"award-info":[{"award-number":["61173185, 61173186"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2018,11]]},"abstract":"<jats:p>Web accessibility evaluation examines how well websites comply with accessibility guidelines which help people with disabilities to perceive, navigate and contribute to the Web. This demanding task usually requires manual assessment by experts with many years of training and experience. However, not enough experts are available to carry out the increasing number of evaluation projects while non-experts often have different opinions about the presence of accessibility barriers. Addressing these issues, we introduce a crowdsourcing system with a novel truth inference algorithm to derive reliable and accurate assessments from conflicting opinions of evaluators. Extensive evaluation on 23,901 complex tasks assessed by 50 people with and without disabilities shows that our approach outperforms state of the art approaches. In addition, we conducted surveys to identify frequent barriers that people with disabilities are facing in their daily lives and the difficulty to access Web pages when they encounter these barriers. The frequencies and severities of barriers correlate with their derived importance in our evaluation project.<\/jats:p>","DOI":"10.1145\/3274432","type":"journal-article","created":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T17:21:27Z","timestamp":1541092887000},"page":"1-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["Crowdsourcing-Based Web Accessibility Evaluation with Golden Maximum Likelihood Inference"],"prefix":"10.1145","volume":"2","author":[{"given":"Shuyi","family":"Song","sequence":"first","affiliation":[{"name":"Zhejiang University, Hangzhou, Zhejiang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiajun","family":"Bu","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andreas","family":"Artmeier","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Keyue","family":"Shi","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ye","family":"Wang","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhi","family":"Yu","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Can","family":"Wang","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2018,11]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2016.03.008"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.5555\/2892753.2892959"},{"key":"e_1_2_2_3_1","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1080\/07370024.2011.601670","article-title":"The expertise effect on web accessibility evaluation methods","volume":"26","author":"Brajnik Giorgio","year":"2011","unstructured":"Giorgio Brajnik, Yeliz Yesilada, and Simon Harper. 2011. The expertise effect on web accessibility evaluation methods. In Proceedings of Human--Computer Interaction. 26, 3, 246--283.","journal-title":"Proceedings of Human--Computer Interaction."},{"key":"e_1_2_2_4_1","volume-title":"Statistics on the population of people from different disability groups in China. Retrieved","author":"China Disabled Persons' Federation CDPF. 2012.","year":"2018","unstructured":"China Disabled Persons' Federation CDPF. 2012. Statistics on the population of people from different disability groups in China. Retrieved July 4, 2018 from http:\/\/www.cdpf.org.cn\/sjzx\/cjrgk\/201206\/t20120626_387581.shtml"},{"key":"e_1_2_2_5_1","volume-title":"Skene","author":"Dawid Alexander P.","year":"1979","unstructured":"Alexander P. Dawid, and Allan M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics. 28, 1, 20--28."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187900"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1977.tb01600.x"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998282"},{"key":"e_1_2_2_9_1","unstructured":"Hyun-Chul Kim and Zoubin Ghahramani. 2012. Bayesian classifier combination. Artificial Intelligence and Statistics. 619--627."},{"key":"e_1_2_2_10_1","volume-title":"WCAG-EM Overview: Website Accessibility Conformance Evaluation Methodology. Web Accessibility Initiative. Retrieved","author":"Henry Shawn Lawton","year":"2018","unstructured":"Shawn Lawton Henry and Shadi Abou-Zahra. First published in 2005, updated in 2016. WCAG-EM Overview: Website Accessibility Conformance Evaluation Methodology. Web Accessibility Initiative. Retrieved April 10, 2018 from http:\/\/www.w3.org\/WAI\/eval\/conformance.html."},{"key":"e_1_2_2_11_1","unstructured":"Jeff Howe. 2006. The rise of crowdsourcing. In Wired magazine. 14 6 1--4."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","unstructured":"David R. Karger Sewoong Oh and Devavrat Shah. 2011. Iterative learning for reliable crowdsourcing systems. In Advances in neural information processing systems. 1953--1961.","DOI":"10.5555\/2986459.2986677"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1147\/sj.443.0519"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3058555.3058573"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2610509"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1805986.1806001"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.5555\/1866696.1866702"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1859894"},{"key":"e_1_2_2_19_1","volume-title":"AAAI Fall symposium on Using Uncertainty within Computation. 121--128","author":"Rusmevichientong Paat","unstructured":"Paat Rusmevichientong, David M. Pennock, Steve Lawrence, and C. Lee Giles. 2001. Methods for sampling pages uniformly from the World Wide Web. AAAI Fall symposium on Using Uncertainty within Computation. 121--128."},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3192714.3192836"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3192714.3192827"},{"key":"e_1_2_2_22_1","unstructured":"Eric Velleman Colin Meerveld Christophe Strobbe Johannes Koch Carlos A. Velasco Mikael Snaprud and Annika Nietzio. 2007. D-WAB4 Unified Web Evaluation Methodology (UWEM 1.2 Core)."},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1243441.1243465"},{"key":"e_1_2_2_24_1","volume-title":"Retrieved","author":"World Wide Web Consortium","year":"2008","unstructured":"World Wide Web Consortium. 2008. Web Content Accessibility Guidelines (WCAG) 2.0. Retrieved April 10, 2018 from http:\/\/www.w3.org\/TR\/WCAG20\/"},{"key":"e_1_2_2_25_1","volume-title":"World report on disability","author":"World Health Organization","year":"2018","unstructured":"World Health Organization. 2011. World report on disability, WHO Press. Retrieved April 10, 2018 from http:\/\/www.who.int\/disabilities\/world_report\/2011\/report\/en\/index.html."},{"key":"e_1_2_2_26_1","volume-title":"Retrieved","author":"World Health Organization","year":"2016","unstructured":"World Health Organization. 2016. Disability and health. Retrieved April 10, 2018 from http:\/\/www.who.int\/mediacentre\/factsheets\/fs352\/en\/"},{"key":"e_1_2_2_27_1","volume-title":"Retrieved","author":"World Wide Web Consortium","year":"2005","unstructured":"World Wide Web Consortium. 2005. Introduction to Web accessibility. Retrieved April 10, 2018 from https:\/\/www.w3.org\/WAI\/intro\/accessibility"},{"key":"e_1_2_2_28_1","volume-title":"Retrieved","author":"World Wide Web Consortium","year":"2016","unstructured":"World Wide Web Consortium. 2016. Failure of Success Criterion 1.4.4 when resizing visually rendered text up to 200 percent causes the text, image or controls to be clipped, truncated or obscured. Retrieved April 10, 2018 from https:\/\/www.w3.org\/TR\/WCAG20-TECHS\/F69.html"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2745555.2746663"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055547"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675158"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274432","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3274432","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T23:05:00Z","timestamp":1775257500000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274432"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,11]]},"references-count":31,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2018,11]]}},"alternative-id":["10.1145\/3274432"],"URL":"https:\/\/doi.org\/10.1145\/3274432","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,11]]},"assertion":[{"value":"2018-11-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}