{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T00:40:31Z","timestamp":1755823231577,"version":"3.44.0"},"reference-count":52,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2024,11,7]],"date-time":"2024-11-07T00:00:00Z","timestamp":1730937600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100006374","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2026498"],"award-info":[{"award-number":["2026498"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2024,11,7]]},"abstract":"<jats:p>Research on data annotation for artificial intelligence (AI) has demonstrated that biases, power, and culture impact the ways that annotators apply labels to data and subsequently affect downstream AI systems. However, annotators can only apply labels that are available to them in the annotation classification scheme. Drawing on a 3-year ethnographic study of an R&amp;D collaboration between medical and AI researchers, we argue that the construction of the classification schema itself -- decisions about what kinds of data can and cannot be collected, what activities can and cannot be detected in the data, what the possible annotation classes ought to be, and the rules by which an item ought to be classified into each class -- dramatically shape the annotation process, and through it, the AI. We draw on Bowker and Star's [9] classification theory to detail how the creation of a training data codebook for a computer vision algorithm in hospital intensive care units (ICUs) evolved from its original, clinically-driven goal of classifying complex clinical activities into a narrower goal of identifying physical objects and simpler activities in the ICU. This work reinforces how trade-offs and decisions made long before annotators begin labeling data are highly consequential to the resulting AI system.<\/jats:p>","DOI":"10.1145\/3687029","type":"journal-article","created":{"date-parts":[[2024,11,8]],"date-time":"2024-11-08T15:52:40Z","timestamp":1731081160000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Constructing a Classification Scheme - and its Consequences: A Field Study of Learning to Label Data for Computer Vision in a Hospital Intensive Care Unit"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7517-4054","authenticated-orcid":false,"given":"Melissa A.","family":"Valentine","sequence":"first","affiliation":[{"name":"Stanford University, Stanford, California, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3628-0388","authenticated-orcid":false,"given":"Roger E.","family":"Bohn","sequence":"additional","affiliation":[{"name":"Clinical Excellence Research Center, Stanford University &amp; University of California San Diego, Stanford, California, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-0724-6963","authenticated-orcid":false,"given":"Amanda L.","family":"Pratt","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, California, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0940-7109","authenticated-orcid":false,"given":"Prachee","family":"Jain","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, California, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3374-1177","authenticated-orcid":false,"given":"Sara J.","family":"Singer","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, California, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8020-9434","authenticated-orcid":false,"given":"Michael S.","family":"Bernstein","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, California, USA"}]}],"member":"320","published-online":{"date-parts":[[2024,11,8]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/UKSim.2013.124"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v36i1.2564"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infoandorg.2019.100286"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951718819569"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134659"},{"volume-title":"Information Acumen: The Understanding and Use of Knowledge in Modern Business,, Lisa Bud-Frierman (Ed.)","author":"Bowker Geoffrey C.","key":"e_1_2_1_6_1","unstructured":"Geoffrey C. Bowker. 1994 a. Information Mythology and Infrastructure. In Information Acumen: The Understanding and Use of Knowledge in Modern Business,, Lisa Bud-Frierman (Ed.). Routledge, London; New York, 231--247."},{"key":"e_1_2_1_7_1","volume-title":"1994 b. Science on the run: information management and industrial geophysics at Schlumberger","author":"Bowker Geoffrey C.","year":"1920","unstructured":"Geoffrey C. Bowker. 1994 b. Science on the run: information management and industrial geophysics at Schlumberger, 1920--1940. MIT Press, Cambridge, MA; London."},{"key":"e_1_2_1_8_1","volume-title":"Bowker and Susan Leigh Star","author":"Geoffrey","year":"1994","unstructured":"Geoffrey C. Bowker and Susan Leigh Star. 1994. Knowledge and Infrastructure in International Information Management: Problems of Classification and Coding. In Information Acumen: The Understanding and Use of Knowledge in Modern Business,, Lisa Bud-Frierman (Ed.). Routledge, London; New York, 187--213."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/6352.001.0001"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-0--387--34872--8_21"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1177\/1077800414545235"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2481492.2481519"},{"key":"e_1_2_1_13_1","unstructured":"Hannah Davis. 2020. A Dataset is a Worldview. https:\/\/towardsdatascience.com\/a-dataset-is-a-worldview-5328216dd44d"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/978--3--540--88564--1_1"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1177\/0021886396321001"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1097\/CCM.0000000000002175"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1162\/daed_a_01902"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025837"},{"key":"e_1_2_1_19_1","volume-title":"Hess","author":"Forsythe Diana","year":"2001","unstructured":"Diana Forsythe and David J. Hess. 2001. Studying those who study us: an anthropologist in the world of artificial intelligence. Stanford University Press, Stanford, Calif."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372862"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13012-019-0894--2"},{"key":"e_1_2_1_22_1","volume-title":"Composing qualitative research","author":"Golden-Biddle Karen","unstructured":"Karen Golden-Biddle and Karen Locke. 2007. Composing qualitative research 2nd ed.). Sage, Thousand Oaks, Calif.","edition":"2"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1002\/job.625"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820016"},{"key":"e_1_2_1_25_1","volume-title":"Jiajun Wu, Juan Carlos Niebles, Ehsan Adeli, and Fei-Fei Li.","author":"Luo Zelun","year":"2022","unstructured":"Zelun Luo, Zane Durante, Linden Li, Wanze Xie, Ruochen Liu, Emily Jin, Zhuoyi Huang, Lun Yu Li, Jiajun Wu, Juan Carlos Niebles, Ehsan Adeli, and Fei-Fei Li. 2022. MOMA-LRG: Language-Refined Graphs for Multi-Object Multi-Actor Activity Parsing. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 5282--5298. https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2022\/file\/22c16986b2f50af520f56dc34d91e403-Paper-Datasets_and_Benchmarks.pdf"},{"key":"e_1_2_1_26_1","volume-title":"Ehsan Adeli, and Fei-Fei Li.","author":"Luo Zelun","year":"2021","unstructured":"Zelun Luo, Wanze Xie, Siddharth Kapoor, Yiyun Liang, Michael Cooper, Juan Carlos Niebles, Ehsan Adeli, and Fei-Fei Li. 2021. MOMA: Multi-Object Multi-Actor Activity Parsing. In Advances in Neural Information Processing Systems, Vol. 34. Curran Associates, Inc., 17939--17955. https:\/\/proceedings.neurips.cc\/paper\/2021\/hash\/95688ba636a4720a85b3634acfec8cdd-Abstract.html"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ccc.2016.12.005"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415186"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbx044"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300356"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445402"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1186\/cc11140"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1097\/HMR.0000000000000398"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998331"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274405"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951720939605"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702298"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445518"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2008.4562953"},{"volume-title":"The ethnographic interview. Holt, Rinehart and Winston","author":"Spradley James P.","key":"e_1_2_1_40_1","unstructured":"James P. Spradley. 1979. The ethnographic interview. Holt, Rinehart and Winston, New York."},{"volume-title":"1995 a. The Cultures of Computing","author":"Star Susan Leigh","key":"e_1_2_1_41_1","unstructured":"Susan Leigh Star (Ed.). 1995 a. The Cultures of Computing. Blackwell Publisher, Oxford, UK; Cambridge, MA, USA."},{"volume-title":"1995 b. Ecologies of Knowledge: Work and Politics in Science and Technology","author":"Star Susan Leigh","key":"e_1_2_1_42_1","unstructured":"Susan Leigh Star (Ed.). 1995 b. Ecologies of Knowledge: Work and Politics in Science and Technology. State University of New York Press, Albany."},{"volume-title":"Ecologies of Knowledge: Work and Politics in Science and Technology,","author":"Star Susan Leigh","key":"e_1_2_1_43_1","unstructured":"Susan Leigh Star. 1995 c. The Politics of Formal Representations: Wizards, Gurus, and Organizational Complexity. In Ecologies of Knowledge: Work and Politics in Science and Technology,, Susan Leigh Star (Ed.). State University of New York Press, Albany, 88--118."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/223248.223257"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951720919776"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3502121"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1097\/CCM.0b013e318186aec8"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.14778\/3275536.3275541"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-019-0087-z"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/PERCOMW.2018.8480380"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.1803.05843"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.2307\/41166066"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3687029","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3687029","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T00:44:20Z","timestamp":1755737060000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3687029"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,7]]},"references-count":52,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2024,11,7]]}},"alternative-id":["10.1145\/3687029"],"URL":"https:\/\/doi.org\/10.1145\/3687029","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2024,11,7]]},"assertion":[{"value":"2024-11-08","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}