{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T20:13:28Z","timestamp":1783541608998,"version":"3.55.0"},"reference-count":92,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2022,3,30]],"date-time":"2022-03-30T00:00:00Z","timestamp":1648598400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2022,3,30]]},"abstract":"<jats:p>Various tools and practices have been developed to support practitioners in identifying, assessing, and mitigating fairness-related harms caused by AI systems. However, prior research has highlighted gaps between the intended design of these tools and practices and their use within particular contexts, including gaps caused by the role that organizational factors play in shaping fairness work. In this paper, we investigate these gaps for one such practice: disaggregated evaluations of AI systems, intended to uncover performance disparities between demographic groups. By conducting semi-structured interviews and structured workshops with thirty-three AI practitioners from ten teams at three technology companies, we identify practitioners' processes, challenges, and needs for support when designing disaggregated evaluations. We find that practitioners face challenges when choosing performance metrics, identifying the most relevant direct stakeholders and demographic groups on which to focus, and collecting datasets with which to conduct disaggregated evaluations. More generally, we identify impacts on fairness work stemming from a lack of engagement with direct stakeholders or domain experts, business imperatives that prioritize customers over marginalized groups, and the drive to deploy AI systems at scale.<\/jats:p>","DOI":"10.1145\/3512899","type":"journal-article","created":{"date-parts":[[2022,4,7]],"date-time":"2022-04-07T16:54:59Z","timestamp":1649350499000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":137,"title":["Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support"],"prefix":"10.1145","volume":"6","author":[{"given":"Michael","family":"Madaio","sequence":"first","affiliation":[{"name":"Microsoft Research, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lisa","family":"Egede","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hariharan","family":"Subramonyam","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer","family":"Wortman Vaughan","sequence":"additional","affiliation":[{"name":"Microsoft Research, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hanna","family":"Wallach","sequence":"additional","affiliation":[{"name":"Microsoft Research, New York City, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,4,7]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327051HCI1523_5"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445888"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1080\/01944366908977225"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301019.3320001"},{"key":"e_1_2_2_5_1","volume-title":"Manish Raghavan, Joshua Simons, et al.","author":"Bakalar Chlo\u00e9","year":"2021","unstructured":"Chlo\u00e9 Bakalar , Renata Barreto , Miranda Bogen , Sam Corbett-Davies , Melissa Hall , Isabel Kloumann , Michelle Lam , Joaquin Qui nonero Candela , Manish Raghavan, Joshua Simons, et al. 2021 . Fairness On The Ground: Applying Algorithmic Fairness Approaches to Production Systems . arXiv preprint arXiv:2103.06172 (2021). Chlo\u00e9 Bakalar, Renata Barreto, Miranda Bogen, Sam Corbett-Davies, Melissa Hall, Isabel Kloumann, Michelle Lam, Joaquin Qui nonero Candela, Manish Raghavan, Joshua Simons, et al. 2021. Fairness On The Ground: Applying Algorithmic Fairness Approaches to Production Systems. arXiv preprint arXiv:2103.06172 (2021)."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3375691"},{"key":"e_1_2_2_7_1","volume-title":"Jennifer Wortman Vaughan, Duncan Wadsworth, and Hanna Wallach.","author":"Barocas Solon","year":"2021","unstructured":"Solon Barocas , Anhong Guo , Ece Kamar , Jacquelyn Krones , Meredith Ringel Morris , Jennifer Wortman Vaughan, Duncan Wadsworth, and Hanna Wallach. 2021 . Designing Disaggregated Evaluations of AI Systems: Choices, Considerations , and Tradeoffs . arXiv preprint arXiv:2103.06076 (2021). Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, and Hanna Wallach. 2021. Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs. arXiv preprint arXiv:2103.06076 (2021)."},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979275"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445922"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300528"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.2966\/scrip.170220.389"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1177\/1744987107081254"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372877"},{"key":"e_1_2_2_14_1","volume-title":"Overcoming Failures of Imagination in AI Infused System Development and Deployment. arXiv preprint arXiv:2011.13416","author":"Boyarskaya Margarita","year":"2020","unstructured":"Margarita Boyarskaya , Alexandra Olteanu , and Kate Crawford . 2020. Overcoming Failures of Imagination in AI Infused System Development and Deployment. arXiv preprint arXiv:2011.13416 ( 2020 ). Margarita Boyarskaya, Alexandra Olteanu, and Kate Crawford. 2020. Overcoming Failures of Imagination in AI Infused System Development and Deployment. arXiv preprint arXiv:2011.13416 (2020)."},{"key":"e_1_2_2_15_1","volume-title":"Using thematic analysis in psychology. Qualitative research in psychology","author":"Braun Virginia","year":"2006","unstructured":"Virginia Braun and Victoria Clarke . 2006. Using thematic analysis in psychology. Qualitative research in psychology , Vol. 3 , 2 ( 2006 ), 77--101. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology, Vol. 3, 2 (2006), 77--101."},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1080\/2159676X.2019.1628806"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11569-012-0141-7"},{"key":"e_1_2_2_18_1","volume-title":"Conference on fairness, accountability and transparency. PMLR, 77--91","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru . 2018 . Gender shades: Intersectional accuracy disparities in commercial gender classification . In Conference on fairness, accountability and transparency. PMLR, 77--91 . Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77--91."},{"key":"e_1_2_2_19_1","doi-asserted-by":"crossref","unstructured":"Adele E Clarke etal 2016. Anticipation work: Abduction simplification hope. Boundary objects and beyond: Working with Leigh Star Vol. 85 (2016). Adele E Clarke et al. 2016. Anticipation work: Abduction simplification hope. Boundary objects and beyond: Working with Leigh Star Vol. 85 (2016).","DOI":"10.7551\/mitpress\/10113.003.0007"},{"key":"e_1_2_2_20_1","volume-title":"Design justice: Community-led practices to build the worlds we need","author":"Costanza-Chock Sasha","unstructured":"Sasha Costanza-Chock . 2020. Design justice: Community-led practices to build the worlds we need . The MIT Press . Sasha Costanza-Chock. 2020. Design justice: Community-led practices to build the worlds we need .The MIT Press."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2017.03.079"},{"key":"e_1_2_2_22_1","unstructured":"Ezekiel Dixon-Rom\u00e1n and Luciana Parisi. 2020. Data Capitalism Sociogenic Prediction and Recursive Indeterminacies. In Public Plurality in an Era of Data Determinacy: Data Publics . Ezekiel Dixon-Rom\u00e1n and Luciana Parisi. 2020. Data Capitalism Sociogenic Prediction and Recursive Indeterminacies. In Public Plurality in an Era of Data Determinacy: Data Publics ."},{"key":"e_1_2_2_23_1","volume-title":"Designs for the pluriverse: Radical interdependence, autonomy, and the making of worlds","author":"Escobar Arturo","unstructured":"Arturo Escobar . 2018. Designs for the pluriverse: Radical interdependence, autonomy, and the making of worlds . Duke University Press . Arturo Escobar. 2018. Designs for the pluriverse: Radical interdependence, autonomy, and the making of worlds .Duke University Press."},{"key":"e_1_2_2_24_1","volume-title":"Ethical foresight analysis: What it is and why it is needed? Minds and Machines","author":"Floridi Luciano","year":"2020","unstructured":"Luciano Floridi and Andrew Strait . 2020. Ethical foresight analysis: What it is and why it is needed? Minds and Machines ( 2020 ), 1--21. Luciano Floridi and Andrew Strait. 2020. Ethical foresight analysis: What it is and why it is needed? Minds and Machines (2020), 1--21."},{"key":"e_1_2_2_25_1","volume-title":"Moving beyond user-centered design. interactions","author":"Forlizzi Jodi","year":"2018","unstructured":"Jodi Forlizzi . 2018. Moving beyond user-centered design. interactions , Vol. 25 , 5 ( 2018 ), 22--23. Jodi Forlizzi. 2018. Moving beyond user-centered design. interactions, Vol. 25, 5 (2018), 22--23."},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1561\/1100000015"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15327051hci2102_3"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951720943234"},{"key":"e_1_2_2_29_1","volume-title":"Against Scale: Provocations and Resistances to Scale Thinking. arXiv preprint arXiv:2010.08850","author":"Hanna Alex","year":"2020","unstructured":"Alex Hanna and Tina M Park . 2020 . Against Scale: Provocations and Resistances to Scale Thinking. arXiv preprint arXiv:2010.08850 (2020). Alex Hanna and Tina M Park. 2020. Against Scale: Provocations and Resistances to Scale Thinking. arXiv preprint arXiv:2010.08850 (2020)."},{"key":"e_1_2_2_30_1","volume-title":"Lana Yarosh, Bushra Anjam, Danish Contractor, and Cathy Wu.","author":"Hecht Brent","year":"2018","unstructured":"Brent Hecht , Lauren Wilcox , Jeffrey P. Bigham , Johannes Sch\u00f6ning , Ehsan Hoque , Jason Ernst , Yonatan Bisk , Luigi De Russis , Lana Yarosh, Bushra Anjam, Danish Contractor, and Cathy Wu. 2018 . It's time to do something: Mitigating the negative impacts of computing through a change to the peer review process. ACM Future of Computing Blog . Brent Hecht, Lauren Wilcox, Jeffrey P. Bigham, Johannes Sch\u00f6ning, Ehsan Hoque, Jason Ernst, Yonatan Bisk, Luigi De Russis, Lana Yarosh, Bushra Anjam, Danish Contractor, and Cathy Wu. 2018. It's time to do something: Mitigating the negative impacts of computing through a change to the peer review process. ACM Future of Computing Blog."},{"key":"e_1_2_2_31_1","volume-title":"Discourse, Violence","author":"Hoffmann Anna Lauren","year":"2020","unstructured":"Anna Lauren Hoffmann . 2020. Terms of Inclusion: Data , Discourse, Violence . New Media & Society ( 2020 ). Anna Lauren Hoffmann. 2020. Terms of Inclusion: Data, Discourse, Violence. New Media & Society (2020)."},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300830"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445918"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702564"},{"key":"e_1_2_2_35_1","doi-asserted-by":"crossref","unstructured":"Abigail Z Jacobs and Hanna Wallach. 2021. Measurement and fairness. (2021) 375--385. Abigail Z Jacobs and Hanna Wallach. 2021. Measurement and fairness. (2021) 375--385.","DOI":"10.1145\/3442188.3445901"},{"key":"e_1_2_2_36_1","volume-title":"The growing gap between emerging technologies and legal-ethical oversight","author":"Johnson Deborah G","unstructured":"Deborah G Johnson . 2011. Software agents, anticipatory ethics, and accountability . In The growing gap between emerging technologies and legal-ethical oversight . Springer , 61--76. Deborah G Johnson. 2011. Software agents, anticipatory ethics, and accountability. In The growing gap between emerging technologies and legal-ethical oversight. Springer, 61--76."},{"key":"e_1_2_2_37_1","first-page":"54","article-title":"Private accountability in the age of artificial intelligence","volume":"66","author":"Katyal Sonia K","year":"2019","unstructured":"Sonia K Katyal . 2019 . Private accountability in the age of artificial intelligence . UCLA L. Rev. , Vol. 66 (2019), 54 . Sonia K Katyal. 2019. Private accountability in the age of artificial intelligence. UCLA L. Rev., Vol. 66 (2019), 54.","journal-title":"UCLA L. Rev."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1915768117"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1098\/rsta.2018.0084"},{"key":"e_1_2_2_40_1","volume-title":"The Landscape and Gaps in Open Source Fairness Toolkits. Available at SSRN","author":"Ah Lee Michelle Seng","year":"2020","unstructured":"Michelle Seng Ah Lee and Jatinder Singh . 2020. The Landscape and Gaps in Open Source Fairness Toolkits. Available at SSRN ( 2020 ). Michelle Seng Ah Lee and Jatinder Singh. 2020. The Landscape and Gaps in Open Source Fairness Toolkits. Available at SSRN (2020)."},{"key":"e_1_2_2_41_1","volume-title":"Asking Questions of the Data: Memo Writing in the Grounded Theory Tradition. The SAGE Handbook of Grounded Theory: Paperback Edition","author":"Lempert Lora Bex","year":"2010","unstructured":"Lora Bex Lempert . 2010. Asking Questions of the Data: Memo Writing in the Grounded Theory Tradition. The SAGE Handbook of Grounded Theory: Paperback Edition ( 2010 ), 245. Lora Bex Lempert. 2010. Asking Questions of the Data: Memo Writing in the Grounded Theory Tradition. The SAGE Handbook of Grounded Theory: Paperback Edition (2010), 245."},{"key":"e_1_2_2_42_1","volume-title":"Scale: Discourse and dimensions of social life","author":"Lempert Michael","year":"2016","unstructured":"Michael Lempert and E Summerson Carr . 2016 . Scale: Discourse and dimensions of social life . University of California Press . Michael Lempert and E Summerson Carr. 2016. Scale: Discourse and dimensions of social life .University of California Press."},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3443686"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376445"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/RFID.2007.346143"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1353\/sor.2019.0022"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415186"},{"key":"e_1_2_2_48_1","volume-title":"Toward a theory of stakeholder identification and salience: Defining the principle of who and what really counts","author":"Mitchell Ronald K","year":"1997","unstructured":"Ronald K Mitchell , Bradley R Agle , and Donna J Wood . 1997. Toward a theory of stakeholder identification and salience: Defining the principle of who and what really counts . Academy of management review, Vol. 22 , 4 ( 1997 ), 853--886. Ronald K Mitchell, Bradley R Agle, and Donna J Wood. 1997. Toward a theory of stakeholder identification and salience: Defining the principle of who and what really counts. Academy of management review, Vol. 22, 4 (1997), 853--886."},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13347-020-00405-8"},{"key":"e_1_2_2_50_1","volume-title":"Ethics as a service: a pragmatic operationalisation of AI Ethics. arXiv preprint arXiv:2102.09364","author":"Morley Jessica","year":"2021","unstructured":"Jessica Morley , Anat Elhalal , Francesca Garcia , Libby Kinsey , Jakob Mokander , and Luciano Floridi . 2021. Ethics as a service: a pragmatic operationalisation of AI Ethics. arXiv preprint arXiv:2102.09364 ( 2021 ). Jessica Morley, Anat Elhalal, Francesca Garcia, Libby Kinsey, Jakob Mokander, and Luciano Floridi. 2021. Ethics as a service: a pragmatic operationalisation of AI Ethics. arXiv preprint arXiv:2102.09364 (2021)."},{"key":"e_1_2_2_51_1","volume-title":"Ranjit Singh, Madeleine Clare Elish, and Jacob Metcalf.","author":"Moss Emanuel","year":"2021","unstructured":"Emanuel Moss , Elizabeth Anne Watkins , Ranjit Singh, Madeleine Clare Elish, and Jacob Metcalf. 2021 . Assembling Accountability : Algorithmic Impact Assessment for the Public Interest . Available at SSRN 3877437 (2021). Emanuel Moss, Elizabeth Anne Watkins, Ranjit Singh, Madeleine Clare Elish, and Jacob Metcalf. 2021. Assembling Accountability: Algorithmic Impact Assessment for the Public Interest. Available at SSRN 3877437 (2021)."},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290607.3299018"},{"key":"e_1_2_2_53_1","volume-title":"Anticipatory Ethics and the Role of Uncertainty. arXiv preprint arXiv:2011.13170","author":"Nanayakkara Priyanka","year":"2020","unstructured":"Priyanka Nanayakkara , Nicholas Diakopoulos , and Jessica Hullman . 2020. Anticipatory Ethics and the Role of Uncertainty. arXiv preprint arXiv:2011.13170 ( 2020 ). Priyanka Nanayakkara, Nicholas Diakopoulos, and Jessica Hullman. 2020. Anticipatory Ethics and the Role of Uncertainty. arXiv preprint arXiv:2011.13170 (2020)."},{"key":"e_1_2_2_54_1","volume-title":"Face recognition vendor test (frvt) part 4: Morph-performance of automated face morph detection .US Department of Commerce","author":"Ngan Mei","unstructured":"Mei Ngan , Mei Ngan , Patrick Grother , Kayee Hanaoka , and Jason Kuo . 2020. Face recognition vendor test (frvt) part 4: Morph-performance of automated face morph detection .US Department of Commerce , National Institute of Standards and Technology . Mei Ngan, Mei Ngan, Patrick Grother, Kayee Hanaoka, and Jason Kuo. 2020. Face recognition vendor test (frvt) part 4: Morph-performance of automated face morph detection .US Department of Commerce, National Institute of Standards and Technology."},{"key":"e_1_2_2_55_1","volume-title":"Science","volume":"366","author":"Obermeyer Ziad","year":"2019","unstructured":"Ziad Obermeyer , Brian Powers , Christine Vogeli , and Sendhil Mullainathan . 2019 . Dissecting racial bias in an algorithm used to manage the health of populations . Science , Vol. 366 , 6464 (2019), 447--453. Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science, Vol. 366, 6464 (2019), 447--453."},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287567"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274405"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951720939605"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445103"},{"key":"e_1_2_2_60_1","volume-title":"Dakuo Wang, Michael Muller, and Felix Portnoy.","author":"Piorkowski David","year":"2021","unstructured":"David Piorkowski , Soya Park , April Yi Wang , Dakuo Wang, Michael Muller, and Felix Portnoy. 2021 . How AI Developers Overcome Communication Challenges in a Multidisciplinary Team : A Case Study . arXiv preprint arXiv:2101.06098 (2021). David Piorkowski, Soya Park, April Yi Wang, Dakuo Wang, Michael Muller, and Felix Portnoy. 2021. How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study. arXiv preprint arXiv:2101.06098 (2021)."},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-021-00298-y"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372873"},{"key":"e_1_2_2_63_1","volume-title":"Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practices. arXiv preprint arXiv:2006.12358","author":"Rakova Bogdana","year":"2020","unstructured":"Bogdana Rakova , Jingying Yang , Henriette Cramer , and Rumman Chowdhury . 2020. Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practices. arXiv preprint arXiv:2006.12358 ( 2020 ). Bogdana Rakova, Jingying Yang, Henriette Cramer, and Rumman Chowdhury. 2020. Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practices. arXiv preprint arXiv:2006.12358 (2020)."},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1177\/0003122418822335"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445604"},{"key":"e_1_2_2_66_1","volume-title":"Reconsidering Scale and Scaling in CSCW Research. In Conference Companion Publication of the 2020 on Computer Supported Cooperative Work and Social Computing. 493--501","author":"Rossitto Chiara","year":"2020","unstructured":"Chiara Rossitto , Airi Lampinen , Susanne B\u00f8dker , Ann Light , Ketie Berns , and Julie Hui . 2020 . Reconsidering Scale and Scaling in CSCW Research. In Conference Companion Publication of the 2020 on Computer Supported Cooperative Work and Social Computing. 493--501 . Chiara Rossitto, Airi Lampinen, Susanne B\u00f8dker, Ann Light, Ketie Berns, and Julie Hui. 2020. Reconsidering Scale and Scaling in CSCW Research. In Conference Companion Publication of the 2020 on Computer Supported Cooperative Work and Social Computing. 493--501."},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445896"},{"key":"e_1_2_2_68_1","doi-asserted-by":"crossref","unstructured":"Kjeld Schmidt. 2000. The critical role of workplace studies in CSCW. 141--149 pages. Kjeld Schmidt. 2000. The critical role of workplace studies in CSCW. 141--149 pages.","DOI":"10.1017\/CBO9780511628122.007"},{"key":"e_1_2_2_69_1","volume-title":"The Reflective Practitioner: How Professionals Think in Action Basic Books Inc","author":"Sch\u00f6n DA","year":"1983","unstructured":"DA Sch\u00f6n . 1983. The Reflective Practitioner: How Professionals Think in Action Basic Books Inc . New York, NY ( 1983 ). DA Sch\u00f6n. 1983. The Reflective Practitioner: How Professionals Think in Action Basic Books Inc. New York, NY (1983)."},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287598"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/1094562.1094569"},{"key":"e_1_2_2_72_1","volume-title":"Techniques and Challenges for Practicing Anticipatory Technology Ethics. iConference 2015 Proceedings","author":"Shilton Katie","year":"2015","unstructured":"Katie Shilton . 2015a. \" That's Not An Architecture Problem!\" : Techniques and Challenges for Practicing Anticipatory Technology Ethics. iConference 2015 Proceedings ( 2015 ). Katie Shilton. 2015a. \"That's Not An Architecture Problem!\": Techniques and Challenges for Practicing Anticipatory Technology Ethics. iConference 2015 Proceedings (2015)."},{"key":"e_1_2_2_73_1","volume-title":"Anticipatory ethics for a future Internet: Analyzing values during the design of an internet infrastructure. Science and engineering ethics","author":"Shilton Katie","year":"2015","unstructured":"Katie Shilton . 2015b. Anticipatory ethics for a future Internet: Analyzing values during the design of an internet infrastructure. Science and engineering ethics , Vol. 21 , 1 ( 2015 ), 1--18. Katie Shilton. 2015b. Anticipatory ethics for a future Internet: Analyzing values during the design of an internet infrastructure. Science and engineering ethics, Vol. 21, 1 (2015), 1--18."},{"key":"e_1_2_2_74_1","volume-title":"Participation is not a design fix for machine learning. arXiv preprint arXiv:2007.02423","author":"Sloane Mona","year":"2020","unstructured":"Mona Sloane , Emanuel Moss , Olaitan Awomolo , and Laura Forlano . 2020. Participation is not a design fix for machine learning. arXiv preprint arXiv:2007.02423 ( 2020 ). Mona Sloane, Emanuel Moss, Olaitan Awomolo, and Laura Forlano. 2020. Participation is not a design fix for machine learning. arXiv preprint arXiv:2007.02423 (2020)."},{"key":"e_1_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675298"},{"key":"e_1_2_2_76_1","volume-title":"Located accountabilities in technology production. Scandinavian journal of information systems","author":"Suchman Lucy","year":"2002","unstructured":"Lucy Suchman . 2002. Located accountabilities in technology production. Scandinavian journal of information systems , Vol. 14 , 2 ( 2002 ), 7. Lucy Suchman. 2002. Located accountabilities in technology production. Scandinavian journal of information systems, Vol. 14, 2 (2002), 7."},{"key":"e_1_2_2_77_1","volume-title":"Anthropological relocations and the limits of design. Annual review of anthropology","author":"Suchman Lucy","year":"2011","unstructured":"Lucy Suchman . 2011. Anthropological relocations and the limits of design. Annual review of anthropology , Vol. 40 ( 2011 ), 1--18. Lucy Suchman. 2011. Anthropological relocations and the limits of design. Annual review of anthropology, Vol. 40 (2011), 1--18."},{"key":"e_1_2_2_78_1","volume-title":"Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs. arXiv preprint arXiv:2101.09824","author":"Suresh Harini","year":"2021","unstructured":"Harini Suresh , Steven R Gomez , Kevin K Nam , and Arvind Satyanarayan . 2021. Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs. arXiv preprint arXiv:2101.09824 ( 2021 ). Harini Suresh, Steven R Gomez, Kevin K Nam, and Arvind Satyanarayan. 2021. Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs. arXiv preprint arXiv:2101.09824 (2021)."},{"key":"e_1_2_2_79_1","volume-title":"A world of standards but not a standard world: Toward a sociology of standards and standardization. Annual review of Sociology","author":"Timmermans Stefan","year":"2010","unstructured":"Stefan Timmermans and Steven Epstein . 2010. A world of standards but not a standard world: Toward a sociology of standards and standardization. Annual review of Sociology , Vol. 36 ( 2010 ), 69--89. Stefan Timmermans and Steven Epstein. 2010. A world of standards but not a standard world: Toward a sociology of standards and standardization. Annual review of Sociology, Vol. 36 (2010), 69--89."},{"key":"e_1_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300791"},{"key":"e_1_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.1215\/0961754X-1630424"},{"key":"e_1_2_2_82_1","volume-title":"Discriminating systems. AI Now","author":"West Sarah Myers","year":"2019","unstructured":"Sarah Myers West , Meredith Whittaker , and Kate Crawford . 2019. Discriminating systems. AI Now ( 2019 ). Sarah Myers West, Meredith Whittaker, and Kate Crawford. 2019. Discriminating systems. AI Now (2019)."},{"key":"e_1_2_2_83_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0164-1212(01)00130-3"},{"key":"e_1_2_2_84_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359245"},{"key":"e_1_2_2_85_1","volume-title":"Conceptualizing Care in the Everyday Work Practices of Machine Learning Developers. In Companion Publication of the 2019 on Designing Interactive Systems Conference 2019 Companion. ACM, 331--335","author":"Wolf Christine T","year":"2019","unstructured":"Christine T Wolf . 2019 . Conceptualizing Care in the Everyday Work Practices of Machine Learning Developers. In Companion Publication of the 2019 on Designing Interactive Systems Conference 2019 Companion. ACM, 331--335 . https:\/\/doi.org\/10.1145\/3301019.3323879 10.1145\/3301019.3323879 Christine T Wolf. 2019. Conceptualizing Care in the Everyday Work Practices of Machine Learning Developers. In Companion Publication of the 2019 on Designing Interactive Systems Conference 2019 Companion. ACM, 331--335. https:\/\/doi.org\/10.1145\/3301019.3323879"},{"key":"e_1_2_2_86_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10606-019-09363-y"},{"key":"e_1_2_2_87_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274453"},{"key":"e_1_2_2_88_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272973.3273005"},{"key":"e_1_2_2_89_1","volume-title":"Timelines: A World-Building Activity for Values Advocacy.","author":"Wong Richmond Y","year":"2021","unstructured":"Richmond Y Wong and Tonya Nguyen . 2021 . Timelines: A World-Building Activity for Values Advocacy. (2021). Richmond Y Wong and Tonya Nguyen. 2021. Timelines: A World-Building Activity for Values Advocacy. (2021)."},{"key":"e_1_2_2_90_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470715"},{"key":"e_1_2_2_91_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753596"},{"key":"e_1_2_2_92_1","volume-title":"Justice and the Politics of Difference","author":"Young Iris Marion","unstructured":"Iris Marion Young . 1990. Justice and the Politics of Difference . Princeton University Press . Iris Marion Young. 1990. Justice and the Politics of Difference. Princeton University Press."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3512899","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3512899","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:12:28Z","timestamp":1750191148000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3512899"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,30]]},"references-count":92,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2022,3,30]]}},"alternative-id":["10.1145\/3512899"],"URL":"https:\/\/doi.org\/10.1145\/3512899","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,30]]},"assertion":[{"value":"2022-04-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}