{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,17]],"date-time":"2026-11-17T11:12:22Z","timestamp":1794913942784,"version":"4.9.1"},"reference-count":93,"publisher":"Association for Computing Machinery (ACM)","issue":"GROUP","license":[{"start":{"date-parts":[[2022,1,14]],"date-time":"2022-01-14T00:00:00Z","timestamp":1642118400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"German Federal Ministry of Education and Research","award":["16DII113"],"award-info":[{"award-number":["16DII113"]}]},{"DOI":"10.13039\/501100000193","name":"International Development Research Centre","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000193","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2022,1,14]]},"abstract":"<jats:p>Research in machine learning (ML) has argued that models trained on incomplete or biased datasets can lead to discriminatory outputs. In this commentary, we propose moving the research focus beyond bias-oriented framings by adopting a power-aware perspective to \"study up\" ML datasets. This means accounting for historical inequities, labor conditions, and epistemological standpoints inscribed in data. We draw on HCI and CSCW work to support our argument, critically analyze previous research, and point at two co-existing lines of work within our research community \\,---\\,one bias-centered, the other power-aware. We highlight the need for dialogue and cooperation in three areas: data quality, data work, and data documentation. In the first area, we argue that reducing societal problems to \"bias\" misses the context-based nature of data. In the second one, we highlight the corporate forces and market imperatives involved in the labor of data workers that subsequently shape ML datasets. Finally, we propose expanding current transparency-oriented efforts in dataset documentation to reflect the social contexts of data design and production.<\/jats:p>","DOI":"10.1145\/3492853","type":"journal-article","created":{"date-parts":[[2022,1,14]],"date-time":"2022-01-14T16:26:16Z","timestamp":1642177576000},"page":"1-14","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":135,"title":["Studying Up Machine Learning Data"],"prefix":"10.1145","volume":"6","author":[{"given":"Milagros","family":"Miceli","sequence":"first","affiliation":[{"name":"Technische Universit\u00e4t Berlin, Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Julian","family":"Posada","sequence":"additional","affiliation":[{"name":"University of Toronto, Toronto, ON, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tianling","family":"Yang","sequence":"additional","affiliation":[{"name":"Technische Universit\u00e4t Berlin, Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,1,14]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proc. ACM Hum.-Comput. Interact. 3, CSCW (Nov. 2019), 199:1--199:30. Article 199. Proc. ACM Hum.-Comput. Interact.","volume":"6","author":"Ali M.","year":"2022","unstructured":"Ali , M. , Sapiezynski , P. , Bogen , M. , Korolova , A. , Mislove , A. , and Rieke , A . Discrimination Through Optimization: How Facebook's Ad Delivery Can Lead to Biased Outcomes . Proc. ACM Hum.-Comput. Interact. 3, CSCW (Nov. 2019), 199:1--199:30. Article 199. Proc. ACM Hum.-Comput. Interact. , Vol. 6 , No. GROUP, Article 34. Publication date : January 2022 . Studying Up Machine Learning Data: Why Talk About Bias When We Mean Power? 34:11 Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., and Rieke, A. Discrimination Through Optimization: How Facebook's Ad Delivery Can Lead to Biased Outcomes. Proc. ACM Hum.-Comput. Interact. 3, CSCW (Nov. 2019), 199:1--199:30. Article 199. Proc. ACM Hum.-Comput. Interact., Vol. 6, No. GROUP, Article 34. Publication date: January 2022. Studying Up Machine Learning Data: Why Talk About Bias When We Mean Power? 34:11"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v36i1.2564"},{"key":"e_1_2_1_3_1","first-page":"139","volume-title":"Proceedings of FG-MoL 2005 : the 10th Conference on Formal Grammar and the 9th Meeting on Mathematics of Language","author":"Artstein R.","year":"2005","unstructured":"Artstein , R. , and Poesio , M . Bias decreases in proportion to the number of annotators . In Proceedings of FG-MoL 2005 : the 10th Conference on Formal Grammar and the 9th Meeting on Mathematics of Language , Edinburgh, 5- -7 August , 2005 (2005), pp. 139 -- 148 . Artstein, R., and Poesio, M. Bias decreases in proportion to the number of annotators. In Proceedings of FG-MoL 2005 : the 10th Conference on Formal Grammar and the 9th Meeting on Mathematics of Language, Edinburgh, 5--7 August, 2005 (2005), pp. 139--148."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1080\/17405904.2012.744320"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372859"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300773"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979041"},{"key":"e_1_2_1_8_1","volume-title":"Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics 6","author":"Bender E. M.","year":"2018","unstructured":"Bender , E. M. , and Friedman , B . Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics 6 ( 2018 ), 587--604. Bender, E. M., and Friedman, B. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics 6 (2018), 587--604."},{"key":"e_1_2_1_9_1","volume-title":"Race After Technology: Abolitionist Tools for the New Jim Code, 1. edition ed","author":"Benjamin R.","year":"2019","unstructured":"Benjamin , R. Race After Technology: Abolitionist Tools for the New Jim Code, 1. edition ed . Polity , Medford, MA , June 2019 . Benjamin, R. Race After Technology: Abolitionist Tools for the New Jim Code, 1. edition ed. Polity, Medford, MA, June 2019."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1515\/pjbr-2019-0004"},{"key":"e_1_2_1_11_1","volume-title":"The Values Encoded in Machine Learning Research. arXiv:2106.15590 [cs] (June","author":"Birhane A.","year":"2021","unstructured":"Birhane , A. , Kalluri , P. , Card , D. , Agnew , W. , Dotan , R. , and Bao , M . The Values Encoded in Machine Learning Research. arXiv:2106.15590 [cs] (June 2021 ). arXiv: 2106.15590. Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., and Bao, M. The Values Encoded in Machine Learning Research. arXiv:2106.15590 [cs] (June 2021). arXiv: 2106.15590."},{"key":"e_1_2_1_12_1","volume-title":"d. How an Algorithmic World Can Be Undermined","year":"2018","unstructured":"boyd , d. How an Algorithmic World Can Be Undermined , 2018 . boyd, d. How an Algorithmic World Can Be Undermined, 2018."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.5555\/3013545.3013548"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1215\/9780822375302"},{"key":"e_1_2_1_15_1","first-page":"77","volume-title":"Proceedings of the 1st Conference on Fairness, Accountability and Transparency","volume":"81","author":"Buolamwini J.","year":"2018","unstructured":"Buolamwini , J. , and Gebru , T . Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification . In Proceedings of the 1st Conference on Fairness, Accountability and Transparency ( 2018 ), vol. 81 , PMLR, pp. 77 -- 91 . Buolamwini, J., and Gebru, T. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (2018), vol. 81, PMLR, pp. 77--91."},{"key":"e_1_2_1_16_1","first-page":"3934","volume":"11","author":"Casilli A. A.","year":"2017","unstructured":"Casilli , A. A. Digital labor studies go global: Toward a digital decolonial turn. International Journal of Communication 11 ( 2017 ), 3934 -- 3954 . Casilli, A. A. Digital labor studies go global: Toward a digital decolonial turn. International Journal of Communication 11 (2017), 3934--3954.","journal-title":"International Journal of Communication"},{"key":"e_1_2_1_17_1","volume-title":"The Platformisation of Labor and Society","author":"Casilli A. A.","year":"2019","unstructured":"Casilli , A. A. , and Posada , J . The Platformisation of Labor and Society . In Society and the Internet, M. Graham and W. H. Dutton, Eds., vol. 2 ed. Oxford University Press , Oxford, 2019 . Casilli, A. A., and Posada, J. The Platformisation of Labor and Society. In Society and the Internet, M. Graham and W. H. Dutton, Eds., vol. 2 ed. Oxford University Press, Oxford, 2019."},{"key":"e_1_2_1_18_1","volume-title":"Le Micro-Travail en France. Derri\u00e8re l'automatisation de nouvelles pr\u00e9carit\u00e9s au travail ? Projet DiPLab \u00ab Digital Platform Labor \u00bb","author":"Casilli A. A.","year":"2019","unstructured":"Casilli , A. A. , Tubaro , P. , Le Ludec , C. , Coville , M. , Besenval , M. , Mouhtare , T. , and Wahal , E . Le Micro-Travail en France. Derri\u00e8re l'automatisation de nouvelles pr\u00e9carit\u00e9s au travail ? Projet DiPLab \u00ab Digital Platform Labor \u00bb , Paris , 2019 . Casilli, A. A., Tubaro, P., Le Ludec, C., Coville, M., Besenval, M., Mouhtare, T., and Wahal, E. Le Micro-Travail en France. Derri\u00e8re l'automatisation de nouvelles pr\u00e9carit\u00e9s au travail ? Projet DiPLab \u00ab Digital Platform Labor \u00bb, Paris, 2019."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359249"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174225"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2481492.2481519"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1177\/1527476418796632"},{"key":"e_1_2_1_23_1","volume-title":"Sept.","author":"Crawford K.","year":"2019","unstructured":"Crawford , K. , and Paglen , T . Excavating AI: The Politics of Images in Machine Learning Training Sets , Sept. 2019 . Crawford, K., and Paglen, T. Excavating AI: The Politics of Images in Machine Learning Training Sets, Sept. 2019."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-021-01170-0"},{"key":"e_1_2_1_25_1","first-page":"1","volume":"89","author":"Danielle K. Citron. The Scored Society","year":"2014","unstructured":"Danielle K. Citron. The Scored Society : Due Process for Automated Predictions. Washington Law Review 89 , 1 ( Mar. 2014 ), 1--33. Danielle K. Citron. The Scored Society: Due Process for Automated Predictions. Washington Law Review 89, 1 (Mar. 2014), 1--33.","journal-title":"Due Process for Automated Predictions. Washington Law Review"},{"key":"e_1_2_1_26_1","volume-title":"Mar.","author":"Davis H.","year":"2020","unstructured":"Davis , H. A Dataset is a Worldview , Mar. 2020 . Library Catalog : towardsdatascience.com. Davis, H. A Dataset is a Worldview, Mar. 2020. Library Catalog: towardsdatascience.com."},{"key":"e_1_2_1_27_1","volume-title":"A., Nicole, H., and Scheuerman, M. K. Bringing the People Back In: Contesting Benchmark Machine Learning Datasets. arXiv:2007.07399 [cs] (July","author":"Denton E.","year":"2020","unstructured":"Denton , E. , Hanna , A. , Amironesei , R. , Smart , A., Nicole, H., and Scheuerman, M. K. Bringing the People Back In: Contesting Benchmark Machine Learning Datasets. arXiv:2007.07399 [cs] (July 2020 ). arXiv: 2007.07399. Denton, E., Hanna, A., Amironesei, R., Smart, A., Nicole, H., and Scheuerman, M. K. Bringing the People Back In: Contesting Benchmark Machine Learning Datasets. arXiv:2007.07399 [cs] (July 2020). arXiv: 2007.07399."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/11805.001.0001"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3278721.3278729"},{"key":"e_1_2_1_30_1","first-page":"162","volume-title":"C. Cognitive Biases in Crowdsourcing. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (Marina Del Rey CA USA","author":"Eickhoff","year":"2018","unstructured":"Eickhoff , C. Cognitive Biases in Crowdsourcing. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (Marina Del Rey CA USA , Feb. 2018 ), ACM, pp. 162 -- 170 . Eickhoff, C. Cognitive Biases in Crowdsourcing. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (Marina Del Rey CA USA, Feb. 2018), ACM, pp. 162--170."},{"key":"e_1_2_1_31_1","volume-title":"Heteromation and its (dis)contents: The invisible division of labor between humans and machines. First Monday (May","author":"Ekbia H.","year":"2014","unstructured":"Ekbia , H. , and Nardi , B . Heteromation and its (dis)contents: The invisible division of labor between humans and machines. First Monday (May 2014 ). Ekbia, H., and Nardi, B. Heteromation and its (dis)contents: The invisible division of labor between humans and machines. First Monday (May 2014)."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2832117"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858343"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415203"},{"key":"e_1_2_1_35_1","first-page":"2952","volume-title":"M. A Design Perspective on Data. In CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA, 2017), CHI '17, Association for Computing Machinery","author":"Feinberg","unstructured":"Feinberg , M. A Design Perspective on Data. In CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA, 2017), CHI '17, Association for Computing Machinery , pp. 2952 -- 2963 . Feinberg, M. A Design Perspective on Data. In CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA, 2017), CHI '17, Association for Computing Machinery, pp. 2952--2963."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.5555\/1866696.1866709"},{"key":"e_1_2_1_37_1","volume-title":"Studying Those Who Study Us: An Anthropologist in the World of Artificial Intelligence","author":"Forsythe D. E.","year":"2001","unstructured":"Forsythe , D. E. Studying Those Who Study Us: An Anthropologist in the World of Artificial Intelligence . Stanford University Press , Stanford , 2001 . Forsythe, D. E. Studying Those Who Study Us: An Anthropologist in the World of Artificial Intelligence. Stanford University Press, Stanford, 2001."},{"key":"e_1_2_1_38_1","volume-title":"Datasheets for Datasets. arXiv:1803.09010 [cs] (Mar","author":"Gebru T.","year":"2020","unstructured":"Gebru , T. , Morgenstern , J. , Vecchione , B. , Vaughan , J. W. , Wallach , H. , Daum\u00e9 III, H. , and Crawford , K . Datasheets for Datasets. arXiv:1803.09010 [cs] (Mar . 2020 ). arXiv: 1803.09010. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daum\u00e9 III, H., and Crawford, K. Datasheets for Datasets. arXiv:1803.09010 [cs] (Mar. 2020). arXiv: 1803.09010."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1107"},{"key":"e_1_2_1_40_1","unstructured":"Ghai B. Liao Q. V. Zhang Y. and Mueller K. Measuring Social Biases of Crowd Workers using Counterfactual Queries.  Ghai B. Liao Q. V. Zhang Y. and Mueller K. Measuring Social Biases of Crowd Workers using Counterfactual Queries."},{"key":"e_1_2_1_41_1","volume-title":"Houghton Mifflin Harcourt","author":"Gray M. L.","year":"2019","unstructured":"Gray , M. L. , and Suri , S . Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass . Houghton Mifflin Harcourt , Boston , May 2019 . Gray, M. L., and Suri, S. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Houghton Mifflin Harcourt, Boston, May 2019."},{"key":"e_1_2_1_42_1","volume-title":"Social Science Research Network","author":"Green B.","year":"2020","unstructured":"Green , B. Data Science as Political Action: Grounding Data Science in a Politics of Justice. SSRN Scholarly Paper ID 3658431 , Social Science Research Network , Rochester, NY , July 2020 . Green, B. Data Science as Political Action: Grounding Data Science in a Politics of Justice. SSRN Scholarly Paper ID 3658431, Social Science Research Network, Rochester, NY, July 2020."},{"key":"e_1_2_1_43_1","volume-title":"Constructing and enforcing \"authentic\" identity online: Facebook, real names, and non-normative identities. First Monday (June","author":"Haimson O. L.","year":"2016","unstructured":"Haimson , O. L. , and Hoffmann , A. L . Constructing and enforcing \"authentic\" identity online: Facebook, real names, and non-normative identities. First Monday (June 2016 ). Haimson, O. L., and Hoffmann, A. L. Constructing and enforcing \"authentic\" identity online: Facebook, real names, and non-normative identities. First Monday (June 2016)."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173582"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372826"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998327"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174023"},{"key":"e_1_2_1_48_1","volume-title":"The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards. arXiv:1805.03677","author":"Holland S.","year":"2018","unstructured":"Holland , S. , Hosny , A. , Newman , S. , Joseph , J. , and Chmielinski , K . The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards. arXiv:1805.03677 ( 2018 ). Holland, S., Hosny, A., Newman, S., Joseph, J., and Chmielinski, K. The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards. arXiv:1805.03677 (2018)."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300637"},{"key":"e_1_2_1_50_1","volume-title":"Apr.","author":"Hunter-Syed A.","year":"2020","unstructured":"Hunter-Syed , A. , and Gebru , T . Timnit Gebru on Algorithmic Bias & Data Mining Ethics , Apr. 2020 . Hunter-Syed, A., and Gebru, T. Timnit Gebru on Algorithmic Bias & Data Mining Ethics, Apr. 2020."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470742"},{"key":"e_1_2_1_52_1","volume-title":"Don't ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583, 7815 (July","author":"Kalluri P.","year":"2020","unstructured":"Kalluri , P. Don't ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583, 7815 (July 2020 ), 169--169. Number : 7815 Publisher : Nature Publishing Group . Kalluri, P. Don't ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583, 7815 (July 2020), 169--169. Number: 7815 Publisher: Nature Publishing Group."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702520"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3375627.3375809"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.4135\/9781473909472"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3323994.3369898"},{"key":"e_1_2_1_57_1","first-page":"417","volume-title":"Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (New York, NY, USA, 2017), CSCW '17, Association for Computing Machinery","author":"Kulshrestha J.","unstructured":"Kulshrestha , J. , Eslami , M. , Messias , J. , Zafar , M. B. , Ghosh , S. , Gummadi , K. P. , and Karahalios , K . Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media . In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (New York, NY, USA, 2017), CSCW '17, Association for Computing Machinery , pp. 417 -- 432 . Kulshrestha, J., Eslami, M., Messias, J., Zafar, M. B., Ghosh, S., Gummadi, K. P., and Karahalios, K. Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (New York, NY, USA, 2017), CSCW '17, Association for Computing Machinery, pp. 417--432."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11023-020-09540-9"},{"key":"e_1_2_1_59_1","first-page":"1","volume-title":"Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA, Apr. 2020), CHI '20, Association for Computing Machinery","author":"Madaio M. A.","unstructured":"Madaio , M. A. , Stark , L. , Wortman Vaughan , J. , and Wallach , H . Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI . In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA, Apr. 2020), CHI '20, Association for Computing Machinery , pp. 1 -- 14 . Madaio, M. A., Stark, L., Wortman Vaughan, J., and Wallach, H. Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA, Apr. 2020), CHI '20, Association for Computing Machinery, pp. 1--14."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531663"},{"key":"e_1_2_1_61_1","volume-title":"Wisdom for the Crowd: Discoursive Power in Annotation Instructions for Computer Vision. arXiv:2105.10990 [cs] (May","author":"Miceli M.","year":"2021","unstructured":"Miceli , M. , and Posada , J . Wisdom for the Crowd: Discoursive Power in Annotation Instructions for Computer Vision. arXiv:2105.10990 [cs] (May 2021 ). arXiv: 2105.10990. Miceli, M., and Posada, J. Wisdom for the Crowd: Discoursive Power in Annotation Instructions for Computer Vision. arXiv:2105.10990 [cs] (May 2021). arXiv: 2105.10990."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415186"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445880"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287596"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.intcom.2011.02.001"},{"key":"e_1_2_1_66_1","first-page":"467","volume-title":"Interrogating Data Science. In Conference Companion Publication of the 2020 on Computer Supported Cooperative Work and Social Computing (Virtual Event USA","author":"Muller M.","year":"2020","unstructured":"Muller , M. , Aragon , C. , Guha , S. , Kogan , M. , Neff , G. , Seidelin , C. , Shilton , K. , and Tanweer , A . Interrogating Data Science. In Conference Companion Publication of the 2020 on Computer Supported Cooperative Work and Social Computing (Virtual Event USA , Oct. 2020 ), ACM, pp. 467 -- 473 . Muller, M., Aragon, C., Guha, S., Kogan, M., Neff, G., Seidelin, C., Shilton, K., and Tanweer, A. Interrogating Data Science. In Conference Companion Publication of the 2020 on Computer Supported Cooperative Work and Social Computing (Virtual Event USA, Oct. 2020), ACM, pp. 467--473."},{"key":"e_1_2_1_67_1","first-page":"1","volume-title":"Human-Centered Study of Data Science Work Practices. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow Scotland Uk","author":"Muller M.","year":"2019","unstructured":"Muller , M. , Feinberg , M. , George , T. , Jackson , S. J. , John , B. E. , Kery , M. B. , and Passi , S . Human-Centered Study of Data Science Work Practices. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow Scotland Uk , May 2019 ), ACM, pp. 1 -- 8 . Muller, M., Feinberg, M., George, T., Jackson, S. J., John, B. E., Kery, M. B., and Passi, S. Human-Centered Study of Data Science Work Practices. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow Scotland Uk, May 2019), ACM, pp. 1--8."},{"key":"e_1_2_1_68_1","first-page":"1","volume-title":"Creation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk, 2019), CHI '19, Association for Computing Machinery","author":"Muller M.","unstructured":"Muller , M. , Lange , I. , Wang , D. , Piorkowski , D. , Tsay , J. , Liao , Q. V. , Dugan , C. , and Erickson , T . How Data Science Workers Work with Data: Discovery, Capture, Curation, Design , Creation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk, 2019), CHI '19, Association for Computing Machinery , pp. 1 -- 15 . Muller, M., Lange, I., Wang, D., Piorkowski, D., Tsay, J., Liao, Q. V., Dugan, C., and Erickson, T. How Data Science Workers Work with Data: Discovery, Capture, Curation, Design, Creation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk, 2019), CHI '19, Association for Computing Machinery, pp. 1--15."},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445402"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.2307\/j.ctt1pwt9w5"},{"key":"e_1_2_1_71_1","volume-title":"Methodological Pitfalls, and Ethical Boundaries. SSRN Electronic Journal","author":"Olteanu A.","year":"2016","unstructured":"Olteanu , A. , Castillo , C. , Diaz , F. , and Kiciman , E . Social Data: Biases , Methodological Pitfalls, and Ethical Boundaries. SSRN Electronic Journal ( 2016 ). Olteanu, A., Castillo, C., Diaz, F., and Kiciman, E. Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries. SSRN Electronic Journal (2016)."},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287567"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998331"},{"key":"e_1_2_1_74_1","first-page":"1","volume":"2","author":"Passi S.","unstructured":"Passi , S. , and Jackson , S. J. Trust in Data Science: Collaboration, Translation , and Accountability in Corporate Data Science Projects. Proc. ACM Hum.-Comput. Interact. 2 , CSCW (Nov. 2018), 1 -- 28 . Passi, S., and Jackson, S. J. Trust in Data Science: Collaboration, Translation, and Accountability in Corporate Data Science Projects. Proc. ACM Hum.-Comput. Interact. 2, CSCW (Nov. 2018), 1--28.","journal-title":"Hum.-Comput. Interact."},{"key":"e_1_2_1_75_1","volume-title":"Data and its (dis)contents: A survey of dataset development and use in machine learning research. arXiv:2012.05345 [cs] (Dec","author":"Paullada A.","year":"2020","unstructured":"Paullada , A. , Raji , I. D. , Bender , E. M. , Denton , E. , and Hanna , A . Data and its (dis)contents: A survey of dataset development and use in machine learning research. arXiv:2012.05345 [cs] (Dec . 2020 ). arXiv: 2012.05345. Paullada, A., Raji, I. D., Bender, E. M., Denton, E., and Hanna, A. Data and its (dis)contents: A survey of dataset development and use in machine learning research. arXiv:2012.05345 [cs] (Dec. 2020). arXiv: 2012.05345."},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1177\/0162243910389594"},{"key":"e_1_2_1_77_1","first-page":"3147","volume-title":"Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (New York, NY, USA, 2015), CHI '15, Association for Computing Machinery","author":"Pine K. H.","unstructured":"Pine , K. H. , and Liboiron , M . The Politics of Measurement and Action . In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (New York, NY, USA, 2015), CHI '15, Association for Computing Machinery , pp. 3147 -- 3156 . Pine, K. H., and Liboiron, M. The Politics of Measurement and Action. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (New York, NY, USA, 2015), CHI '15, Association for Computing Machinery, pp. 3147--3156."},{"key":"e_1_2_1_78_1","volume-title":"The Future of Work Is Here: Toward a Comprehensive Approach to Artificial Intelligence and Labour. Ethics of AI in Context","author":"Posada J.","year":"2020","unstructured":"Posada , J. The Future of Work Is Here: Toward a Comprehensive Approach to Artificial Intelligence and Labour. Ethics of AI in Context ( 2020 ). Posada, J. The Future of Work Is Here: Toward a Comprehensive Approach to Artificial Intelligence and Labour. Ethics of AI in Context (2020)."},{"key":"e_1_2_1_79_1","volume-title":"Eds. AI Now Institute","author":"Posada J.","year":"2021","unstructured":"Posada , J. Unbiased : AI Needs Ethics from Below. In New AI Lexicon, N. Raval, A. Kak, and L. Strathman , Eds. AI Now Institute , New York, NY , 2021 . Posada, J. Unbiased: AI Needs Ethics from Below. In New AI Lexicon, N. Raval, A. Kak, and L. Strathman, Eds. AI Now Institute, New York, NY, 2021."},{"key":"e_1_2_1_80_1","volume-title":"Dec.","author":"Powles J.","year":"2018","unstructured":"Powles , J. , and Nissenbaum , H . The Seductive Diversion of \"Solving' Bias in Artificial Intelligence , Dec. 2018 . Powles, J., and Nissenbaum, H. The Seductive Diversion of \"Solving' Bias in Artificial Intelligence, Dec. 2018."},{"key":"e_1_2_1_81_1","volume-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Ramanath R.","year":"2013","unstructured":"Ramanath , R. , Choudhury , M. , Bali , K. , and Roy , R. S . Crowd Prefers the Middle Path: A New IAA Metric for Crowdsourcing Reveals Turker Biases in Query Segmentation . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) ( Sofia, Bulgaria , 2013 ), Association for Computational Linguistics, pp. 1713--1722. Ramanath, R., Choudhury, M., Bali, K., and Roy, R. S. Crowd Prefers the Middle Path: A New IAA Metric for Crowdsourcing Reveals Turker Biases in Query Segmentation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Sofia, Bulgaria, 2013), Association for Computational Linguistics, pp. 1713--1722."},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.2307\/j.ctvhrcz0v"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753846.1753873"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702508"},{"key":"e_1_2_1_85_1","volume-title":"June","author":"Sampat R.","year":"2015","unstructured":"Sampat , R. Protesters target Facebook's 'real name' policy , June 2015 . Sampat, R. Protesters target Facebook's 'real name' policy, June 2015."},{"key":"e_1_2_1_86_1","first-page":"W1","volume":"4","author":"Identity ACM","year":"2020","unstructured":"Scheuerman, M. K., Wade, K., Lustig, C., and Brubaker, J. R. How We've Taught Algorithms to See Identity : Constructing Race and Gender in Image Databases for Facial Analysis. Proc. ACM Hum.-Comput. Interact. 4 , CSC W1 ( 2020 ). Article 058. Scheuerman, M. K., Wade, K., Lustig, C., and Brubaker, J. R. How We've Taught Algorithms to See Identity: Constructing Race and Gender in Image Databases for Facial Analysis. Proc. ACM Hum.-Comput. Interact. 4, CSCW1 (2020). Article 058.","journal-title":"Hum.-Comput. Interact."},{"key":"e_1_2_1_87_1","volume-title":"N","author":"Seaver","year":"2014","unstructured":"Seaver , N . Studying Up : The Ethnography of Technologists , Mar. 2014 . Seaver, N. Studying Up: The Ethnography of Technologists, Mar. 2014."},{"key":"e_1_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008651105359"},{"key":"e_1_2_1_90_1","volume-title":"Micro-work, artificial intelligence and the automotive industry. Journal of Industrial and Business Economics","author":"Tubaro P.","year":"2019","unstructured":"Tubaro , P. , and Casilli , A. A . Micro-work, artificial intelligence and the automotive industry. Journal of Industrial and Business Economics ( 2019 ). Tubaro, P., and Casilli, A. A. Micro-work, artificial intelligence and the automotive industry. Journal of Industrial and Business Economics (2019)."},{"key":"e_1_2_1_91_1","first-page":"1800","volume-title":"Proceedings of the 24th International Conference on Neural Information Processing Systems (Granada, Spain, 2011), NIPS'11, Curran Associates Inc.","author":"Wauthier F. L.","unstructured":"Wauthier , F. L. , and Jordan , M. I . Bayesian Bias Mitigation for Crowdsourcing . In Proceedings of the 24th International Conference on Neural Information Processing Systems (Granada, Spain, 2011), NIPS'11, Curran Associates Inc. , pp. 1800 -- 1808 . Wauthier, F. L., and Jordan, M. I. Bayesian Bias Mitigation for Crowdsourcing. In Proceedings of the 24th International Conference on Neural Information Processing Systems (Granada, Spain, 2011), NIPS'11, Curran Associates Inc., pp. 1800-- 1808."},{"key":"e_1_2_1_92_1","volume-title":"The Gig Economy: A Critical Introduction","author":"Woodcock J.","year":"2020","unstructured":"Woodcock , J. , and Graham , M . The Gig Economy: A Critical Introduction . Polity Press , London , 2020 . Woodcock, J., and Graham, M. The Gig Economy: A Critical Introduction. Polity Press, London, 2020."},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.5555\/1866696.1866719"},{"key":"e_1_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.1145\/3392826"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3492853","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3492853","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:48:27Z","timestamp":1750178907000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3492853"}},"subtitle":["Why Talk About Bias When We Mean Power?"],"short-title":[],"issued":{"date-parts":[[2022,1,14]]},"references-count":93,"journal-issue":{"issue":"GROUP","published-print":{"date-parts":[[2022,1,14]]}},"alternative-id":["10.1145\/3492853"],"URL":"https:\/\/doi.org\/10.1145\/3492853","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,14]]},"assertion":[{"value":"2022-01-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}