{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:59:26Z","timestamp":1750309166055,"version":"3.41.0"},"reference-count":31,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2023,12,31]],"date-time":"2023-12-31T00:00:00Z","timestamp":1703980800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Queue"],"published-print":{"date-parts":[[2023,12,31]]},"abstract":"<jats:p>\n            In recent years, machine learning (ML) has relied heavily on crowdworkers both for building datasets and for addressing research questions requiring human interaction or judgment. The diversity of both the tasks performed and the uses of the resulting data render it difficult to determine when crowdworkers are best thought of as workers versus human subjects. These difficulties are compounded by conflicting policies, with some institutions and researchers regarding all ML crowdworkers as human subjects and others holding that they rarely constitute human subjects. Notably few ML papers involving crowdwork mention IRB oversight, raising the prospect of non-compliance with ethical and regulatory requirements. We investigate the appropriate designation of ML crowdsourcing studies, focusing our inquiry on natural language processing to expose unique challenges for research oversight. Crucially, under the U.S. Common Rule, these judgments hinge on determinations of\n            <jats:italic>aboutness<\/jats:italic>\n            , concerning both whom (or what) the collected data is about and whom (or what) the analysis is about. We highlight two challenges posed by ML: the same set of workers can serve multiple roles and provide many sorts of information; and ML research tends to embrace a dynamic workflow, where research questions are seldom stated ex ante and data sharing opens the door for future studies to aim questions at different targets. Our analysis exposes a potential loophole in the Common Rule, where researchers can elude research ethics oversight by splitting data collection and analysis into distinct studies. Finally, we offer several policy recommendations to address these concerns.\n          <\/jats:p>","DOI":"10.1145\/3639452","type":"journal-article","created":{"date-parts":[[2024,1,14]],"date-time":"2024-01-14T23:05:49Z","timestamp":1705273549000},"page":"101-127","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Resolving the Human-subjects Status of Machine Learning's Crowdworkers"],"prefix":"10.1145","volume":"21","author":[{"given":"Divyansh","family":"Kaushik","sequence":"first","affiliation":[{"name":"Federation of American Scientists"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zachary C.","family":"Lipton","sequence":"additional","affiliation":[{"name":"CMU"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alex John","family":"London","sequence":"additional","affiliation":[{"name":"CMU"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,1,14]]},"reference":[{"issue":"1","key":"e_1_2_1_1_1","first-page":"116","article-title":"MasakhaNER: named entity recognition for African languages","volume":"9","author":"Adelani D. F.","year":"2021","unstructured":"Adelani, D. F., Abbott, J., Neubig, G., D'souza, D., Kreutzer, J., Lignos, C., Palen-Michel, C., Buzaaba, H., Rijhwani, S., Ruder, S., et al. 2021. MasakhaNER: named entity recognition for African languages. Transactions of the Association for Computational Linguistics 9, 1,116-1,131; https:\/\/aclanthology.org\/2021.tacl-1.66.pdf.","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"e_1_2_1_2_1","unstructured":"Birmingham-Southern College. Do I need IRB approval? https:\/\/www.bsc.edu\/academics\/irb\/documents\/BSC%20IRB%20Decision%20Tree.pdf."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.408"},{"key":"e_1_2_1_4_1","unstructured":"Dodge J. Gururangan S. Card D. Schwartz R. Smith N. A. 2019. Show your work: improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the Ninth International Joint Conference on Natural Language Processing (EMNLPIJCNLP) 2 185?2 194; https:\/\/aclanthology.org\/D19-1224\/."},{"key":"e_1_2_1_5_1","volume-title":"Amazon Mechanical Turk: gold mine or coal mine? Computational Linguistics 37 (2), 413?420","author":"Fort K.","year":"2010","unstructured":"Fort, K., Adda, G., Cohen, K. B. 2011. Amazon Mechanical Turk: gold mine or coal mine? Computational Linguistics 37 (2), 413?420; https:\/\/aclanthology.org\/J11-2010.pdf."},{"key":"e_1_2_1_6_1","doi-asserted-by":"crossref","unstructured":"Geva M. Goldberg Y. Berant J. 2019. Are we modeling the task or the annotator? An investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the Ninth International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) 1 161?1 166; https:\/\/aclanthology.org\/D19-1107.pdf.","DOI":"10.18653\/v1\/D19-1107"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.654"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-2062"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-2096"},{"key":"e_1_2_1_10_1","volume-title":"human subjects, and IRBs","author":"Ipeirotis P.","year":"2009","unstructured":"Ipeirotis, P. 2009. Mechanical Turk, human subjects, and IRBs; https:\/\/www.behind-the-enemy-lines.com\/2009\/01\/mechanical-turk-human-subjects-and-irbs.html."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.517"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.12"},{"key":"e_1_2_1_13_1","doi-asserted-by":"crossref","unstructured":"Kovashka A. Russakovsky O. Fei-Fei L. Grauman K. 2016. Crowdsourcing in computer vision. Foundations and Trends in Computer Graphics and Vision 10 (3) 177?243; https:\/\/www.nowpublishers.com\/article\/Details\/CGV-0711.","DOI":"10.1561\/0600000071"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-1604"},{"volume-title":"For the Common Good: Philosophical Foundations of Research Ethics","author":"London A. J..","key":"e_1_2_1_15_1","unstructured":"London, A. J.. 2021. For the Common Good: Philosophical Foundations of Research Ethics. Oxford University Press."},{"volume-title":"Clinical trial portfolios: a critical oversight in human research ethics, drug regulation, and policy","author":"London A. J.","key":"e_1_2_1_16_1","unstructured":"London, A. J., Kimmelman, J. 2019. Clinical trial portfolios: a critical oversight in human research ethics, drug regulation, and policy. Hastings Center Report 49 (4), 31?41; https:\/\/pubmed.ncbi.nlm.nih.gov\/31429954\/."},{"key":"e_1_2_1_17_1","doi-asserted-by":"crossref","unstructured":"London A. J. Taljaard M. Weijer C. 2020. Loopholes in the research ethics system? Informed consent waivers in cluster randomized trials with individual-level intervention. Ethics & Human Research 42 (6) 21?28; https:\/\/onlinelibrary.wiley.com\/doi\/abs\/10.1002\/eahr.500071.","DOI":"10.1002\/eahr.500071"},{"key":"e_1_2_1_18_1","unstructured":"Loyola University. Do I need IRB review? https:\/\/www.luc.edu\/irb\/gettingstarted\/isirbreviewrequired\/."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1260"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.195"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-demos.17"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1133"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.295"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","volume":"1","author":"Talmor A.","year":"2019","unstructured":"Talmor, A., Herzig, J., Lourie, N., Berant, J. 2019. CommonsenseQA: a question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4,149?4,158; https:\/\/aclanthology.org\/N19-1421.pdf."},{"issue":"1","key":"e_1_2_1_25_1","article-title":"Making better use of the crowd: how crowdsourcing can advance machine learning research","volume":"18","author":"Vaughan J. W.","year":"2017","unstructured":"Vaughan, J. W. 2017. Making better use of the crowd: how crowdsourcing can advance machine learning research. Journal of Machine Learning Research 18 (1), 7,026?7,071; https:\/\/dl.acm.org\/doi\/10.5555\/3122009.3242050.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v7i1.5283"},{"key":"e_1_2_1_27_1","unstructured":"Whittier College. Do I need IRB Review? https:\/\/www.whittier.edu\/academics\/researchethics\/irb\/need."},{"key":"e_1_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Williams A. Nangia N. Bowman S. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 1 (Long Papers). 1 112?1 122; https:\/\/aclanthology.org\/N18-1101.pdf.","DOI":"10.18653\/v1\/N18-1101"},{"key":"e_1_2_1_29_1","volume-title":"Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079","author":"Xu J.","year":"2010","unstructured":"Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., Dinan, E. 2020. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079; https:\/\/arxiv.org\/abs\/2010.07079."},{"key":"e_1_2_1_30_1","unstructured":"Zaidan O. Eisner J. Piatko C. 2007. Using \"Annotator Rationales\" to improve machine learning for text categorization. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference 260?267; https:\/\/aclanthology.org\/N07-1033."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.249"}],"container-title":["Queue"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3639452","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3639452","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:53:37Z","timestamp":1750287217000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3639452"}},"subtitle":["What ethical framework should govern the interaction of ML researchers and crowdworkers?"],"short-title":[],"issued":{"date-parts":[[2023,12,31]]},"references-count":31,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,12,31]]}},"alternative-id":["10.1145\/3639452"],"URL":"https:\/\/doi.org\/10.1145\/3639452","relation":{},"ISSN":["1542-7730","1542-7749"],"issn-type":[{"type":"print","value":"1542-7730"},{"type":"electronic","value":"1542-7749"}],"subject":[],"published":{"date-parts":[[2023,12,31]]},"assertion":[{"value":"2024-01-14","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}