{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,18]],"date-time":"2026-05-18T10:46:26Z","timestamp":1779101186206,"version":"3.51.4"},"reference-count":30,"publisher":"Association for Computing Machinery (ACM)","issue":"125","license":[{"start":{"date-parts":[[2020,3,2]],"date-time":"2020-03-02T00:00:00Z","timestamp":1583107200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGACCESS Access. Comput."],"published-print":{"date-parts":[[2020,3,2]]},"abstract":"<jats:p>\n            Persons with disabilities face many barriers to full participation in society, and the rapid advancement of technology has the potential to create ever more. Building equitable and inclusive technologies for people with disabilities demands paying attention to more than accessibility, but also to how social attitudes towards disability are represented within technology. Representations perpetuated by machine learning (ML) models often inadvertently encode undesirable social biases from the data on which they are trained. This can result, for example, in text classification models producing very different predictions for\n            <jats:italic toggle=\"yes\">I am a person with mental illness<\/jats:italic>\n            , and\n            <jats:italic toggle=\"yes\">I am a tall person<\/jats:italic>\n            . In this paper, we present evidence of such biases in existing ML models, and in data used for model development. First, we demonstrate that a machine-learned model to moderate conversations classifies texts which mention disability as more \"toxic\". Similarly, a machine-learned sentiment analysis model rates texts which mention disability as more negative. Second, we demonstrate that neural text representation models that are critical to many ML applications can also contain undesirable biases towards mentions of disabilities. Third, we show that the data used to develop such models reflects topical biases in social discourse which may explain such biases in the models - for instance, gun violence, homelessness, and drug addiction are over-represented in discussions about mental illness.\n          <\/jats:p>","DOI":"10.1145\/3386296.3386305","type":"journal-article","created":{"date-parts":[[2020,3,2]],"date-time":"2020-03-02T15:51:31Z","timestamp":1583164291000},"page":"1-1","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":35,"title":["Unintended machine learning biases as social barriers for persons with disabilitiess"],"prefix":"10.1145","author":[{"given":"Ben","family":"Hutchinson","sequence":"first","affiliation":[{"name":"Google"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vinodkumar","family":"Prabhakaran","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Remi","family":"Denton","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kellie","family":"Webster","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yu","family":"Zhong","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stephen","family":"Denuyl","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,3,2]]},"reference":[{"key":"e_1_2_1_1_1","first-page":"07","volume":"02","author":"Cavender A.","year":"2019","unstructured":"A. Cavender, S. Trewin and V. Hanson, \"Accessible Writing Guide,\" [Online]. Available: https:\/\/www.sigaccess.org\/welcome-to-sigaccess\/resources\/accessible-writing-guide\/. [Accessed 02 07 2019].","journal-title":"\"Accessible Writing Guide,\" [Online]. Available: https:\/\/www.sigaccess.org\/welcome-to-sigaccess\/resources\/accessible-writing-guide\/. [Accessed"},{"key":"e_1_2_1_2_1","first-page":"07","volume":"02","year":"2019","unstructured":"\"Common Barriers to Participation Experienced by People with Disabilities,\" [Online]. Available: https:\/\/www.cdc.gov\/ncbddd\/disabilityandhealth\/disability-barriers.html. [Accessed 02 07 2019].","journal-title":"Accessed"},{"key":"e_1_2_1_3_1","first-page":"195","volume-title":"Representation and its discontents: The uneasy home of disability in literature and film,\" Handbook of Disability Studies","author":"Mitchell D. T.","year":"2001","unstructured":"D. T. Mitchell and S. L. Snyder, \"Representation and its discontents: The uneasy home of disability in literature and film,\" Handbook of Disability Studies, pp. 195 - 218, 2001."},{"key":"e_1_2_1_4_1","first-page":"07","volume":"02","year":"2019","unstructured":"\"Stereotypes About People With Disabilities,\" [Online]. Available: https:\/\/www.disabilitymuseum.org\/dhm\/edu\/essay.html?id=24. [Accessed 02 07 2019].","journal-title":"Accessed"},{"key":"e_1_2_1_5_1","volume-title":"Black looks: Race and representation","year":"2006","unstructured":"b. hooks, Black looks: Race and representation, Academic Internet Pub Inc, 2006."},{"key":"e_1_2_1_6_1","unstructured":"Jigsaw \"Perspective API \" 2017. [Online]. Available: http:\/\/www.perspectiveapi.com."},{"key":"e_1_2_1_7_1","volume-title":"a. K. K. Geyik and A. T. Kalai, \"Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting,\" in ACM Conference on Fairness, Accountability, and Transparency (ACM FAT*)","author":"De-Arteaga M.","year":"2019","unstructured":"M. De-Arteaga, A. Romanov, H. Wallach, J. Chayes, C. Borgs, A. Chouldechova, S. a. K. K. Geyik and A. T. Kalai, \"Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting,\" in ACM Conference on Fairness, Accountability, and Transparency (ACM FAT*), 2019."},{"key":"e_1_2_1_8_1","first-page":"183","volume-title":"Semantics derived automatically from language corpora contain human-like biases,\" Science","author":"Caliskan A.","year":"2017","unstructured":"A. Caliskan, J. J. Bryson and A. Narayanan, \"Semantics derived automatically from language corpora contain human-like biases,\" Science, vol. 356, pp. 183-186, 2017."},{"key":"e_1_2_1_9_1","volume-title":"On Measuring Social Biases in Sentence Encoders,\" Proceedings of the 2019 Conference of the North American","author":"May C.","year":"2019","unstructured":"C. May, A. Wang, S. Bordia, S. R. Bowman and R. Rudinger, \"On Measuring Social Biases in Sentence Encoders,\" Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1, 2019."},{"key":"e_1_2_1_10_1","volume-title":"Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes,\" Proceedings of the National Academy of Science","author":"Garg N.","year":"2017","unstructured":"N. Garg, L. Schiebinger, D. Jurafsky and J. Zou, \"Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes,\" Proceedings of the National Academy of Science, vol. 115, 2017."},{"key":"e_1_2_1_11_1","volume-title":"Man is to Computer Programmer As Woman is to Homemaker? Debiasing Word Embeddings,\" in Neural Information Processing Systems","author":"Bolukbasi T.","year":"2016","unstructured":"T. Bolukbasi, K.-W. Chang, J. Zou, V. Saligrama and A. Kalai, \"Man is to Computer Programmer As Woman is to Homemaker? Debiasing Word Embeddings,\" in Neural Information Processing Systems, 2016."},{"key":"e_1_2_1_12_1","volume-title":"The problem with bias: from allocative to representational harms in machine learning","author":"Barocas S.","year":"2017","unstructured":"S. Barocas, K. Crawford, A. Shapiro and H. Wallach, \"The problem with bias: from allocative to representational harms in machine learning. Special Interest Group for Computing,\" Information and Society (SIGCIS), 2017."},{"key":"e_1_2_1_13_1","volume-title":"Counterfactual fairness in text classification through robustness,\" in 3rd AAAI\/ACM Conference on AI, Ethics, and Society","author":"Garg S.","year":"2019","unstructured":"S. Garg, V. Perot, N. Limtiaco, A. Taly, E. H. Chi and A. Beutel, \"Counterfactual fairness in text classification through robustness,\" in 3rd AAAI\/ACM Conference on AI, Ethics, and Society, 2019."},{"key":"e_1_2_1_14_1","volume-title":"Measuring and mitigating unintended bias in text classification,\" in AAAI\/ACM Conference on AI, Ethics, and Society","author":"Dixon L.","year":"2018","unstructured":"L. Dixon, J. Li, J. Sorensen, N. Thain and L. Vasserman, \"Measuring and mitigating unintended bias in text classification,\" in AAAI\/ACM Conference on AI, Ethics, and Society, 2018."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.18574\/nyu\/9781479833641.001.0001"},{"key":"e_1_2_1_16_1","first-page":"07","volume":"02","year":"2019","unstructured":"\"Disability Inclusion,\" [Online]. Available: https:\/\/www.worldbank.org\/en\/topic\/disability. [Accessed 02 07 2019].","journal-title":"Accessed"},{"key":"e_1_2_1_17_1","volume-title":"Pervasiveness and correlates of implicit attitudes and stereotypes,\" European Review of Social Psychology","author":"Nosek B. A.","unstructured":"B. A. Nosek, F. L. Smyth, J. J. Hansen, T. Devos, N. M. Lindner, K. A. Ranganath, C. T. Smith, K. R. Olson, D. Chugh, A. G. Greenwald and M. R. Banaji, \"Pervasiveness and correlates of implicit attitudes and stereotypes,\" European Review of Social Psychology, vol. 16, no. 6, pp. 699-713, 2007."},{"key":"e_1_2_1_18_1","unstructured":"SIGACCESS \"Accessible Writing Guide. \" 2015. [Online]. Available: https:\/\/www.sigaccess.org\/welcome-to-sigaccess\/resources\/accessible-writing-guide\/. [Accessed 02 07 2019]."},{"key":"e_1_2_1_19_1","unstructured":"A.-D. League. \"Suggested Language forPeople with Disabilities \" 02 07 2005. [Online]. Available: https:\/\/www.adl.org\/sites\/default\/files\/documents\/assets\/pdf\/education-outreach\/suggested-language-for-people-with-disabilities.pdf. [Accessed 2019]."},{"key":"e_1_2_1_20_1","volume-title":"Writing about accessibility,\" Interactionn","author":"Hanson V. L.","unstructured":"V. L. Hanson, A. Cavender and S. Trewin, \"Writing about accessibility,\" Interactionn, vol. 22, no. 6, 2015."},{"key":"e_1_2_1_21_1","volume-title":"Network., \"Guidelines for WritingAbout People With Disabilities","author":"A.","year":"2018","unstructured":"A. N. Network., \"Guidelines for WritingAbout People With Disabilities,\" 2018. [Online]. Available: https:\/\/adata.org\/factsheet\/ADANN-writing. [Accessed 02 07 2019]."},{"key":"e_1_2_1_22_1","volume-title":"Perturbation Sensitivity Analysis to Detect Unintended Model Biases,\" in Conference on Empirical Methods in Natural Language Processing","author":"Prabhakaran V.","year":"2019","unstructured":"V. Prabhakaran, B. Hutchinson and M. Mitchell, \"Perturbation Sensitivity Analysis to Detect Unintended Model Biases,\" in Conference on Empirical Methods in Natural Language Processing, 2019."},{"key":"e_1_2_1_23_1","unstructured":"G. Cloud \"Google Cloud NLP API Version 1 Beta 2 \" 2018. [Online]. Available: https:\/\/cloud.google.com\/natural-language\/. [Accessed 21 05 2019]."},{"key":"e_1_2_1_24_1","volume-title":"G. S. Corrado and J. Dean, \"Efficient Estimation of Word Representations in Vector Space,\" in International Conference on Learning Representations","author":"T.","year":"2013","unstructured":"T. M. a. K. Chen, G. S. Corrado and J. Dean, \"Efficient Estimation of Word Representations in Vector Space,\" in International Conference on Learning Representations, 2013."},{"key":"e_1_2_1_25_1","volume-title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,\" in Conference of the North American","author":"Devlin J.","year":"2018","unstructured":"J. Devlin, M.-W. Chang, K. Lee and K. Toutanova, \"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,\" in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2018."},{"key":"e_1_2_1_26_1","unstructured":"J. R. Firth \"A synopsis of linguistic theory 1930-1955 \" Studies in linguistic analysis 1957."},{"key":"e_1_2_1_27_1","volume-title":"Fightin'words: Lexical feature selection and evaluation for identifying the content of political conflict,\" Political Analysis","author":"Monroe B. L.","unstructured":"B. L. Monroe, M. P. Colaresi and K. M. Quinn, \"Fightin'words: Lexical feature selection and evaluation for identifying the content of political conflict,\" Political Analysis, vol. 16, no. 4, pp. 372-403, 2008."},{"key":"e_1_2_1_28_1","doi-asserted-by":"crossref","unstructured":"D. Wasserman \"Philosophical issues in the definition and social response to disability \" Disability and Equality Law pp. 19-52 2017.","DOI":"10.4324\/9781315094861-3"},{"key":"e_1_2_1_29_1","volume-title":"Fairness in Machine Learning","author":"Barocas S.","year":"2018","unstructured":"S. Barocas, M. Hardt and A. Naranayan, \"Fairness in Machine Learning,\" 2018. [Online]. Available: http:\/\/fairmlbook.org."},{"key":"e_1_2_1_30_1","volume-title":"FairTest: Discovering unwarranted associations in data-driven applications,\" in IEEE European Symposium on Security and Privacy (EuroS&P)","author":"Tramer F.","year":"2017","unstructured":"F. Tramer, V. Atlidakis, R. Geambasu, D. Hsu, J.-P. Hubaux, M. Humbert, A. Juels and H. Lin, \"FairTest: Discovering unwarranted associations in data-driven applications,\" in IEEE European Symposium on Security and Privacy (EuroS&P), 2017."}],"container-title":["ACM SIGACCESS Accessibility and Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3386296.3386305","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3386296.3386305","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,18]],"date-time":"2025-08-18T17:33:32Z","timestamp":1755538412000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3386296.3386305"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,2]]},"references-count":30,"journal-issue":{"issue":"125","published-print":{"date-parts":[[2020,3,2]]}},"alternative-id":["10.1145\/3386296.3386305"],"URL":"https:\/\/doi.org\/10.1145\/3386296.3386305","relation":{},"ISSN":["1558-2337","1558-1187"],"issn-type":[{"value":"1558-2337","type":"print"},{"value":"1558-1187","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3,2]]},"assertion":[{"value":"2020-03-02","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}