{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,29]],"date-time":"2026-06-29T18:45:21Z","timestamp":1782758721552,"version":"3.54.5"},"publisher-location":"New York, NY, USA","reference-count":56,"publisher":"ACM","license":[{"start":{"date-parts":[[2026,6,25]],"date-time":"2026-06-25T00:00:00Z","timestamp":1782345600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"name":"Edward Holley Research Fund","award":["28201-71068-EA320"],"award-info":[{"award-number":["28201-71068-EA320"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,6,25]]},"DOI":"10.1145\/3805689.3806512","type":"proceedings-article","created":{"date-parts":[[2026,6,23]],"date-time":"2026-06-23T16:20:39Z","timestamp":1782231639000},"page":"2359-2384","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Risks and Opportunities in Human-Machine Teaming in Operationalizing Machine Learning Target Variables"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9299-7594","authenticated-orcid":false,"given":"Mengtian","family":"Guo","sequence":"first","affiliation":[{"name":"School of Information and Library Science, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6424-7374","authenticated-orcid":false,"given":"David","family":"Gotz","sequence":"additional","affiliation":[{"name":"School of Information and Library Science, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0278-2347","authenticated-orcid":false,"given":"Yue","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Information and Library Science, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,25]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"key":"e_1_3_2_1_2_1","unstructured":"Anthropic. 2025. Claude Code. https:\/\/claude.ai AI coding assistant Version 2.0."},{"key":"e_1_3_2_1_3_1","unstructured":"Anysphere. 2025. Cursor. https:\/\/cursor.sh AI-powered code editor."},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","unstructured":"Aleksander Aristovnik Damijana Ker\u017ei\u010d Dejan Rav\u0161elj Nina Toma\u017eevi\u010d Lan Umek Toyin Cotties Adetiba Adetutu Deborah Aina Oluwatoyin Ayodele Ajani Bibi Alajmi Sultan Ghaleb Aldaihani Magdalena Waleska Aldana-Segura Said Aldhafri Jogymol Alex Fahad Ahmed Al-Harbi Yusuf Alpayadin Parag Amin George Kofi Amoako Octavian Andronic Sorin Gabriel Anton Arheiam Arheiam Alex Riolexus Ario Maja Arslanagi\u0107-Kalajd\u017ei\u0107 Sofia Asonitou Roxana Pamela Balbont\u00edn Alvarado Martin Mabunda Baluku Mohammad Bashaar Joy Benatov Naima Benkari Syed Ahmad Helmi Bin Syed Hassan Isaac Mensah Boafo Roberto Burro Michael P. Cameron Silvia Cantele Maria Cheraghi Yi-Lin Chiang Andy Choi Yeung Simeon-Pierre Choukem \u00d6zkan \u00c7ikrikci Michaela Cortini Baye Dagnew Denilson da Silva Bezerra Vera Dimitrievska Beata Dobrowolska Jadranka Durovi\u0107 Todorovi\u0107 Diena Dwidienawati Falk Ebinger Arri Eisen Maha El Tantawi Mahmoud M. Emam Ibeawuchi K. Enwereuzor Adeniyi Francis Fagbamigbe Stefania Fantinelli MoezAlIslam E. Faris Ali Farooq Maria Fedorova Paulo Ferrinho Barbara Fogarty-Perry Morenike Oluwatoyin Folayan Thais Fran\u00e7a Bongani Thulani Gamede Yongtao Gan Manuel Gericota Belinka Gonz\u00e1lez-Fern\u00e1ndez Luz Mar\u00eda Gonz\u00e1lez-Robledo Paul Gorczynski Muji Gunarto Adam Gyedu Soumeyya Halayem Sarah J. Halvorson Nazir S. Hawi Shiva Heidari Azita Hekmatdoost Meeri Hellsten Meirav Hen Evelyne H\u00fcbscher Fany Inasius Takashi Inoguchi Yariv Itzkovich Ervin Iusein Telesphore Kabera Sedighe Sadat Hashemi Kamangar Sujita Kumar Kar Konstantinos Karampelas Elham Kateeb Amrita Kaur Kerefu Lawrence Joseph Aleksandar Ke\u0161eljevi\u0107 Pavol Kr\u00e1l' Hiroko Kudo P. A. P. Samantha Kumara Murodbek Laldjebaev Korn\u00e9lia Laz\u00e1nyi Florin Laz\u0103r Paul H. Lee Poliana Mihaela Leru Aurora Lopez-Fogues Rataya Luechapudiporn Philippe N. Lukanu Prosper Lutala Juan D. Machin-Mastromatteo Marwa Madi Piotr Major Maria Malliarou Niko M\u00e4nnikk\u00f6 Jo\u00e3o P. Maroco Bertil P. Marques Jo\u00e3o Matias Oliva Mej\u00eda-Rodr\u00edguez Jana Meloska Petrova Silvia Mariela M\u00e9ndez Prado Milena Mili\u0107evi\u0107 Marek Milosz Jos\u00e9 Joaqu\u00edn Mira Marta Miret Alpana Mishra Masoud Mohammadnezhad Cristina Mollica Immanuel Azaad Moonesar Nicolas J. Mouawad Elfi Mu'awanah Dilbar Mukhamedova Lillias Hamufari Natsai Mutambara Joseph Muthiani Malechwanzi Silvana G. Navarro David Musyimi Ndetei Nga Nguyen Singhanat Nomnian Alka Obadi\u0107 Ryan Michael Oducado Olawale Festus Olaniyan Izabela Ostoj Efstathia Papageorgiou Nino Paresashvili Shirona Patel Susan Kane Patton Lidia Perenc Virtudes P\u00e9rez-Jover Harm Peters Justyna Podg\u00f3rska-Bednarz Eka Sunarwidhi Prasedya Bo Pu Sumayyah Qudah Daniela Raccanello Agustine Ramie Luis Armando Ramos Palacios Mamun Ur Rashid Vijayalakshmi Reddy Iveta Reinholde Maya Roche Ana Sofia Rodrigues Danilo V. Rogayan Piotr Rzymski Fahad Saleem Roberta Sammut Grover Sandeep Oana S\u0103ndulescu Rinku Sanjeev Muhammad Saqib Pavlos Sarafis Muthupandian Saravanan Mariano Schlez Abdul-Aziz Seidu Akkaya Senkrua Abdel-Aziz Sharabati Bidhan Shrestha Aggrey Siya Ricarda Steinmayr Eveline Surbakti Rajanikanta Swain Vanphanom Sychareun Sne\u017eana \u0160\u0107epanovi\u0107 David \u0160pa\u010dek Ivana Tadi\u0107 Kathy W. Tannous Sanja Tatalovi\u0107 Vorkapi\u0107 Harold Jan Terano Mehmet S. Tosun Chinaza Uleanya Olga Ushakova Thomas Varghese Daina Vasilevska Tengiz Verulava Giada Vicentini Sornkanok Vimolmangkang Jeffrey Dawala Wilang Angelique Wildschut Nikolay N. Yagodka Guo-liang Yang Chunlin Yao Norhafezah Yusof Ana-Maria Zamfir Shehla A. Yasin Adrian P. Yba\u00f1ez \u00d6zlem Yorulmaz Yunquan Zhang Oksana Zhirosh and Al Et. 2021. Impacts of the Covid-19 Pandemic on Life of Higher Education Students: Global Survey Dataset from the First Wave. 5 (Dec. 2021). doi:10.17632\/88y3nffs82.5 Publisher: Mendeley Data.","DOI":"10.17632\/88y3nffs82.5"},{"key":"e_1_3_2_1_5_1","unstructured":"Michelle Bao Angela Zhou Samantha Zottola Brian Brubach Sarah Desmarais Aaron Horowitz Kristian Lum and Suresh Venkata-subramanian. 2022. It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks. http:\/\/arxiv.org\/abs\/2106.05498 arXiv:2106.05498 [cs]."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jml.2012.11.001"},{"key":"e_1_3_2_1_7_1","volume-title":"Proceedings of the 30th International Conference on Machine Learning. PMLR, 115\u2013123","author":"Bergstra James","year":"2013","unstructured":"James Bergstra, Daniel Yamins, and David Cox. 2013. Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures. In Proceedings of the 30th International Conference on Machine Learning. PMLR, 115\u2013123. https:\/\/proceedings.mlr.press\/v28\/bergstra13.html ISSN: 1938\u20137228."},{"key":"e_1_3_2_1_8_1","volume-title":"The Effects of Data Quality on Machine Learning Performance. arXiv preprint arXiv:2207.14529","author":"Budach Lukas","year":"2022","unstructured":"Lukas Budach, Moritz Feuerpfeil, Nina Ihde, Andrea Nathansen, Nele Noack, Hendrik Patzlaff, Felix Naumann, and Hazar Harmouch. 2022. The Effects of Data Quality on Machine Learning Performance. arXiv preprint arXiv:2207.14529 (2022). https:\/\/arxiv.org\/abs\/2207.14529"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","unstructured":"Michelle Carney Barron Webster Irene Alvarado Kyle Phillips Noura Howell Jordan Griffith Jonas Jongejan Amit Pitaru and Alexander Chen. 2020. Teachable Machine: Approachable Web-Based Tool for Exploring Machine Learning Classification. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. ACM Honolulu HI USA 1\u20138. doi:10.1145\/3334480.3382839","DOI":"10.1145\/3334480.3382839"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13681"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3501831"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/SaTML54575.2023.00050"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-13059-5_22"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581338"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1177\/0093854812453911"},{"key":"e_1_3_2_1_16_1","volume-title":"Automated Machine Learning: State-of-The-Art and Open Challenges. arXiv:1906.02287[cs, stat] (June","author":"Elshawi Radwa","year":"2019","unstructured":"Radwa Elshawi, Mohamed Maher, and Sherif Sakr. 2019. Automated Machine Learning: State-of-The-Art and Open Challenges. arXiv:1906.02287[cs, stat] (June 2019). http:\/\/arxiv.org\/abs\/1906.02287 arXiv: 1906.02287."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/240455.240464"},{"key":"e_1_3_2_1_18_1","volume-title":"Advances in Neural Information Processing Systems","volume":"28","author":"Feurer Matthias","year":"2015","unstructured":"Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and Robust Automated Machine Learning. In Advances in Neural Information Processing Systems, Vol. 28. Curran Associates, Inc. https:\/\/proceedings.neurips.cc\/paper\/2015\/hash\/11d0e6287202fced83f79975ec59a3a6-Abstract.html"},{"key":"e_1_3_2_1_19_1","volume-title":"Lineup: Visual analysis of multi-attribute rankings","author":"Gratzl Samuel","year":"2013","unstructured":"Samuel Gratzl, Alexander Lex, Nils Gehlenborg, Hanspeter Pfister, and Marc Streit. 2013. Lineup: Visual analysis of multi-attribute rankings. IEEE transactions on visualization and computer graphics 19, 12 (2013), 2277\u20132286."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3594036"},{"key":"e_1_3_2_1_21_1","volume-title":"AutoML: A survey of the state-of-the-art. Knowledge-based systems 212","author":"He Xin","year":"2021","unstructured":"Xin He, Kaiyong Zhao, and Xiaowen Chu. 2021. AutoML: A survey of the state-of-the-art. Knowledge-based systems 212 (2021), 106622."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2016.7739680"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376177"},{"key":"e_1_3_2_1_24_1","volume-title":"Browser-based Statistical Interface for Data Repositories.","author":"Honaker James","year":"2014","unstructured":"James Honaker and Vito D'Orazio. 2014. Statistical Modeling by Gesture: A graphical, Browser-based Statistical Interface for Data Repositories. (2014)."},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2012.219"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517439"},{"key":"e_1_3_2_1_27_1","volume-title":"Guo","author":"Kross Sean","year":"2021","unstructured":"Sean Kross and Philip J. Guo. 2021. Orienting, Framing, Bridging, Magic, and Counseling: How Data Scientists Navigate the Outer Loop of Client Collaborations in Industry and Academia. http:\/\/arxiv.org\/abs\/2105.05849 arXiv:2105.05849 [cs]."},{"key":"e_1_3_2_1_28_1","volume-title":"Anders Hammerich Riis, Ulrick Skipper Espelund, Jesper Bo Weile, and Jeppe Lange.","author":"Lauritsen Simon Meyer","year":"2021","unstructured":"Simon Meyer Lauritsen, Bo Thiesson, Marianne Johansson J\u00f8rgensen, Anders Hammerich Riis, Ulrick Skipper Espelund, Jesper Bo Weile, and Jeppe Lange. 2021. The Framing of machine learning risk prediction models illustrated by evaluation of sepsis in general wards. NPJ digital medicine 4, 1 (2021), 158."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2204781120"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3361118"},{"key":"e_1_3_2_1_31_1","volume-title":"Andrew Smart, and William S. Isaac","author":"Jr Donald Martin","year":"2020","unstructured":"Donald Martin Jr., Vinodkumar Prabhakaran, Jill Kuhlberg, Andrew Smart, and William S. Isaac. 2020. Participatory Problem Formulation for Fairer Machine Learning Through Community Based System Dynamics. http:\/\/arxiv.org\/abs\/2005.07572 arXiv:2005.07572 [cs, stat]."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1017\/S089006041600007X"},{"key":"e_1_3_2_1_33_1","unstructured":"Microsoft. [n. d.]. Team Data Science Process -TDSP. https:\/\/datascienceprocess.com\/member-home-page\/team-data-science-process-tdsp\/"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300356"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510209"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-010-9156-z"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.aax2342"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287567"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866038"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372828"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533158"},{"key":"e_1_3_2_1_42_1","volume-title":"Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv preprint arXiv:1908.10084","author":"Reimers N","year":"2019","unstructured":"N Reimers. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv preprint arXiv:1908.10084 (2019)."},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","unstructured":"Olawale Salaudeen Anka Reuel Ahmed Ahmed Suhana Bedi Zachary Robertson Sudharsan Sundar Ben Domingue Angelina Wang and Sanmi Koyejo. 2025. Measurement to Meaning: A Validity-Centered Framework for AI Evaluation. doi:10.48550\/arXiv.2505.10573 arXiv:2505.10573 [cs].","DOI":"10.48550\/arXiv.2505.10573"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3328519.3329134"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376229"},{"key":"e_1_3_2_1_46_1","volume-title":"Tempo: Helping Data Scientists and Domain Experts Collaboratively Specify Predictive Modeling Tasks. arXiv preprint arXiv:2502.10526","author":"Sivaraman Venkatesh","year":"2025","unstructured":"Venkatesh Sivaraman, Anika Vaishampayan, Xiaotong Li, Brian R Buck, Ziyong Ma, Richard D Boyce, and Adam Perer. 2025. Tempo: Helping Data Scientists and Domain Experts Collaboratively Specify Predictive Modeling Tasks. arXiv preprint arXiv:2502.10526 (2025)."},{"key":"e_1_3_2_1_47_1","volume-title":"Christian Drescher, Alexander Hanuschkin, Ludwig Winkler, Steven Peters, and Klaus-Robert Mueller.","author":"Studer Stefan","year":"2021","unstructured":"Stefan Studer, Thanh Binh Bui, Christian Drescher, Alexander Hanuschkin, Ludwig Winkler, Steven Peters, and Klaus-Robert Mueller. 2021. Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology. http:\/\/arxiv.org\/abs\/2003.05155 arXiv:2003.05155 [cs, stat]."},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3600211.3604678"},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1177\/0008125619867910"},{"key":"e_1_3_2_1_50_1","unstructured":"Rhema Vaithianathan Emily Putnam-Hornstein Nan Jiang Parma Nand and Tim Maloney. 2017. Developing Predictive Models to Support Child Maltreatment Hotline Screening Decisions: Allegheny County Methodology and Implementation. (2017). https:\/\/www.alleghenycountyanalytics.us\/wp-content\/uploads\/2019\/05\/Methodology-V1-from-16-ACDHS-26_PredictiveRisk_Package_050119_FINAL.pdf Methodology V1 from 16-ACDHS-26 Predictive Risk Package."},{"key":"e_1_3_2_1_51_1","volume-title":"When the Machine Meets the Expert: An Ethnography of Developing AI for Hiring. MIS quarterly 45, 3","author":"den Broek Elmira Van","year":"2021","unstructured":"Elmira Van den Broek, Anastasia Sergeeva, and Marleen Huysman. 2021. When the Machine Meets the Expert: An Ethnography of Developing AI for Hiring. MIS quarterly 45, 3 (2021)."},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3636509"},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0204920"},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3654777.3676374"},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376301"},{"key":"e_1_3_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3196709.3196729"}],"event":{"name":"FAccT '26: The 2026 ACM Conference on Fairness, Accountability, and Transparency","location":"Montreal QC Canada","acronym":"FAccT '26","sponsor":["ACM\/SIG"]},"container-title":["Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3805689.3806512","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,29]],"date-time":"2026-06-29T18:05:13Z","timestamp":1782756313000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3805689.3806512"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,25]]},"references-count":56,"alternative-id":["10.1145\/3805689.3806512","10.1145\/3805689"],"URL":"https:\/\/doi.org\/10.1145\/3805689.3806512","relation":{},"subject":[],"published":{"date-parts":[[2026,6,25]]},"assertion":[{"value":"2026-06-25","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}