{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T18:00:25Z","timestamp":1776103225143,"version":"3.50.1"},"reference-count":70,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2025,5,2]],"date-time":"2025-05-02T00:00:00Z","timestamp":1746144000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nd\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2025,5,2]]},"abstract":"<jats:p>Supervised machine learning is a common approach for automated harmful content detection to support content moderation. This approach relies on data annotated by humans to train models to recognize classes of harmful content. For detection tasks, researchers or content moderation communities typically either design their own annotation tasks to generate training data for new harmful content detection models, or use off-the-shelf (OTS) pre-trained harmful content detection models. OTS model reuse can enable detection tasks in resource-constrained contexts and can help to reduce the environmental impact of training new models -- an energy-intensive process. However, given the plethora of OTS models now available for reuse, determining which OTS model to reuse for a particular task and how to use it can be challenging, especially given that many of these models have been developed for specific contexts that are not always easily transferred onto others. This work aims to provide best practices for reusing OTS models for harmful content detection tasks. By using content analysis and statistical methods to evaluate assumptions about OTS model utility and reusability, we show that model reusers cannot assume that a model claimed to detect a particular concept, will actually detect that concept. Instead, based on our findings, we offer a decision tree for how to assess whether an OTS model would be appropriate for reuse for a new harmful content detection task. This decision tree directs model reusers to critically assess concept definitions, annotation task design, and additional features specified in our content analysis codebook to identify expected model output, and consequently evaluate whether that OTS model is appropriate for reuse for a new detection task.<\/jats:p>","DOI":"10.1145\/3711099","type":"journal-article","created":{"date-parts":[[2025,5,20]],"date-time":"2025-05-20T11:36:19Z","timestamp":1747740979000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Using Off-the-Shelf Harmful Content Detection Models: Best Practices for Model Reuse"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7912-1371","authenticated-orcid":false,"given":"Angela","family":"Sch\u00f6pke-Gonzalez","sequence":"first","affiliation":[{"name":"School of Information, University of Michigan, Ann Arbor, MI, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1384-6908","authenticated-orcid":false,"given":"Siqi","family":"Wu","sequence":"additional","affiliation":[{"name":"Indiana University Bloomington, Bloomington, IN, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0157-4615","authenticated-orcid":false,"given":"Sagar","family":"Kumar","sequence":"additional","affiliation":[{"name":"Northeastern University, Boston, MA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3793-7281","authenticated-orcid":false,"given":"Libby","family":"Hemphill","sequence":"additional","affiliation":[{"name":"School of Information, University of Michigan, Ann Arbor, MI, USA"}]}],"member":"320","published-online":{"date-parts":[[2025,5,2]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Sarcasm","unstructured":"2022. Sarcasm. In Cambridge Advanced Learner's Dictionary & Thesaurus. Cambridge University Press."},{"key":"e_1_2_1_2_1","volume-title":"Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14--18, 2020, Proceedings, Part V","author":"Aluru Sai Saketh","year":"2020","unstructured":"Sai Saketh Aluru, Binny Mathew, Punyajoy Saha, and Animesh Mukherjee. 2020. A Deep Dive into Multilingual Hate Speech Classification. In Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14--18, 2020, Proceedings, Part V (Ghent, Belgium). Springer-Verlag, Berlin, Heidelberg, 423--439."},{"key":"e_1_2_1_3_1","unstructured":"Sai Saketh Aluru Punyajoy Saha and Binny Mathew. 2020. DE-LIMIT: DeEpLearning models for MultIlingual haTespeech (DELIMIT): Benchmarking multilingual models across 9 languages and 16 datasets."},{"key":"e_1_2_1_4_1","unstructured":"Amazon Web Services. 2016. Amazon Rekognition."},{"key":"e_1_2_1_5_1","unstructured":"Perspectiv API. [n. d.]. Case Studies. https:\/\/perspectiveapi.com\/case-studies\/. Accessed: 2024--6--26."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/s43681-021-00084-x"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445157"},{"key":"e_1_2_1_8_1","first-page":"3","article-title":"A unified approach to measurement error and missing data: Overview and applications","volume":"46","author":"Blackwell Matthew","year":"2017","unstructured":"Matthew Blackwell, James Honaker, and Gary King. 2017. A unified approach to measurement error and missing data: Overview and applications. Sociol. Methods Res. 46, 3 (Aug. 2017), 303--341.","journal-title":"Sociol. Methods Res."},{"key":"e_1_2_1_9_1","first-page":"4","article-title":"Online and Uncivil? Patterns and Determinants of Incivility in Newspaper Website Comments","volume":"64","author":"Coe Kevin","year":"2014","unstructured":"Kevin Coe, Kate Kenski, and Stephen A Rains. 2014. Online and Uncivil? Patterns and Determinants of Incivility in Newspaper Website Comments. J. Commun. 64, 4 (Aug. 2014), 658--679.","journal-title":"J. Commun."},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","unstructured":"Thomas Davidson Dana Warmsley Michael Macy and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Eleventh international aaai conference on web and social media.","DOI":"10.1609\/icwsm.v11i1.14955"},{"key":"e_1_2_1_11_1","unstructured":"Thomas Davidson Ingmar Weber and Jonathan Zarecki. 2019. Automated Hate Speech Detection and the Problem of Offensive Language."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3278721.3278729"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v12i1.15041"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v12i1.15038"},{"key":"e_1_2_1_15_1","unstructured":"Andr\u00e9 Ferreira. 2022. Analyzing Political Manifestos through Natural Language Processing and Dashboarding. https:\/\/towardsdatascience.com\/analyzing-political-manifestos-through-natural-language-processing-and-dashboarding-4ad1d62d6b9a. Accessed: 2024--6--26."},{"key":"e_1_2_1_16_1","unstructured":"Andr\u00e9 Ferreira. 2022. Polids. https:\/\/andrecnf-polids-appapp-naawtf.streamlit.app\/. Accessed: 2024--6--26."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1017\/pan.2020.38"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the Asian HCI Symposium 2022 (New Orleans, LA, USA) (Asian HCI '22). Association for Computing Machinery","author":"Ganeshan Loshini","year":"2023","unstructured":"Loshini Ganeshan, Masitah Ghazali, and Nur Zuraifah Syazrah Othman. 2023. An Acceptance Towards Buzzer as a Filtering Approach towards Creating Responsible Users on Social Media Postings. In Proceedings of the Asian HCI Symposium 2022 (New Orleans, LA, USA) (Asian HCI '22). Association for Computing Machinery, New York, NY, USA, 33--39."},{"key":"e_1_2_1_19_1","volume-title":"Predicting Different Types of Subtle Toxicity in Unhealthy Online Conversations. Procedia Comput. Sci. 198 (Jan","author":"Gilda Shlok","year":"2022","unstructured":"Shlok Gilda, Luiz Giovanini, Mirela Silva, and Daniela Oliveira. 2022. Predicting Different Types of Subtle Toxicity in Unhealthy Online Conversations. Procedia Comput. Sci. 198 (Jan. 2022), 360--366."},{"key":"e_1_2_1_20_1","first-page":"4","article-title":"Safety for whom? Investigating how platforms frame and perform safety and harm interventions","volume":"8","author":"Gillett Rosalie","year":"2022","unstructured":"Rosalie Gillett, Zahra Stardust, and Jean Burgess. 2022. Safety for whom? Investigating how platforms frame and perform safety and harm interventions. Soc. Media Soc. 8, 4 (Oct. 2022), 205630512211443.","journal-title":"Soc. Media Soc."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951719897945"},{"key":"e_1_2_1_22_1","unstructured":"Tesh Goyal Ian Kivlichan Rachel Rosen and Lucy Vasserman. 2022. Jigsaw Specialized Rater Pools Dataset."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3270101.3270103"},{"key":"e_1_2_1_24_1","volume-title":"Don't stop pretraining: Adapt language models to domains and tasks. arXiv [cs.CL] (April","author":"Gururangan Suchin","year":"2020","unstructured":"Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. arXiv [cs.CL] (April 2020)."},{"key":"e_1_2_1_25_1","volume-title":"Wayne Xin Zhao, and Jun Zhu","author":"Han Xu","year":"2021","unstructured":"Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. 2021. Pre-trained models: Past, present and future. AI Open 2 (Jan. 2021), 225--250."},{"key":"e_1_2_1_26_1","unstructured":"Hatebase. [n. d.]. Hatebase. https:\/\/hatebase.org\/. Accessed: 2022--8--4."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1177\/1049732305276687"},{"key":"e_1_2_1_28_1","first-page":"2","article-title":"Silenced on social media: the gatekeeping functions of shadowbans in the American Twitterverse","volume":"73","author":"Jaidka Kokil","year":"2023","unstructured":"Kokil Jaidka, Subhayan Mukerjee, and Yphtach Lelkes. 2023. Silenced on social media: the gatekeeping functions of shadowbans in the American Twitterverse. J. Commun. 73, 2 (April 2023), 163--178.","journal-title":"J. Commun."},{"key":"e_1_2_1_29_1","volume-title":"KOLD: Korean Offensive Language Dataset. (Dec.","author":"Jeong Younghoon","year":"2022","unstructured":"Younghoon Jeong, Juhyun Oh, Jongwon Lee, Jaimeen Ahn, Jihyung Moon, Sungjoon Park, and Alice Oh. 2022. KOLD: Korean Offensive Language Dataset. (Dec. 2022), 10818--10833."},{"key":"e_1_2_1_30_1","unstructured":"Jigsaw. 2017. Perspective API. www.perspectiveapi.com. Accessed: 2022--1-NA."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.296"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3628797.3628966"},{"key":"e_1_2_1_33_1","volume-title":"Confronting abusive language online: A survey from the ethical and human rights perspective. J. Artif. Intell. Res. 71 (July","author":"Kiritchenko Svetlana","year":"2021","unstructured":"Svetlana Kiritchenko, Isar Nejadgholi, and Kathleen C Fraser. 2021. Confronting abusive language online: A survey from the ethical and human rights perspective. J. Artif. Intell. Res. 71 (July 2021), 431--478."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1177\/001316447003000105"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1468--2958.2004.tb00738.x"},{"key":"e_1_2_1_36_1","doi-asserted-by":"crossref","unstructured":"Klaus Krippendorff. 2019. Conceptual Foundation. In Content Analysis: An Introduction to Its Methodology (4 ed.) Klaus Krippendorff (Ed.). SAGE 24--50.","DOI":"10.4135\/9781071878781.n3"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19--1425"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3539609"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3643829"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.5555\/2984075.2984080"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3637406"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1177\/1461444804041444"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/978--3--319--77116--8_30"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCSS.2018.2887240"},{"key":"e_1_2_1_45_1","unstructured":"Dark Data Project. 2023. About. https:\/\/darkdataproject.org\/about. Accessed: 2023--8--8."},{"key":"e_1_2_1_46_1","unstructured":"The Sentinel Project. 2018. Home. https:\/\/thesentinelproject.org\/. Accessed: 2023--8--8."},{"key":"e_1_2_1_47_1","volume-title":"Natural Language Annotation for Machine Learning","author":"Pustejovsky James","unstructured":"James Pustejovsky and Amber Stubbs. 2012. Natural Language Annotation for Machine Learning. O'Reilly Media, Inc."},{"key":"e_1_2_1_48_1","volume-title":"R: A Language and Environment for Statistical Computing","author":"Team R Core","year":"2022","unstructured":"R Core Team. 2022. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria."},{"key":"e_1_2_1_49_1","volume-title":"Behind the Screen: Content Moderation in the Shadows of Social Media","author":"Roberts Sarah T","unstructured":"Sarah T Roberts. 2019. Behind the Screen: Content Moderation in the Shadows of Social Media. Yale University Press."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3032963"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1177\/0093650220921314"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1080\/1369118X.2014.940365"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3449861"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531766"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-1101"},{"key":"e_1_2_1_56_1","volume-title":"Najmin Ahmed, and Libby Hemphill.","author":"Sch\u00f6pke-Gonzalez Angela M","year":"2022","unstructured":"Angela M Sch\u00f6pke-Gonzalez, Shubham Atreja, Han Na Shin, Najmin Ahmed, and Libby Hemphill. 2022. Why do volunteer content moderators quit? Burnout, conflict, and harmful behaviors. New Media & Society (Dec. 2022), 14614448221138529."},{"key":"e_1_2_1_57_1","unstructured":"Gudbjartur Ingi Sigurbergsson and Leon Derczynski. 2020. Offensive Language and Hate Speech Detection for Danish. In Proceedings of the Twelfth Language Resources and Evaluation Conference Nicoletta Calzolari Fr\u00e9d\u00e9ric B\u00e9chet Philippe Blache Khalid Choukri Christopher Cieri Thierry Declerck Sara Goggi Hitoshi Isahara Bente Maegaard Joseph Mariani H\u00e9l\u00e8ne Mazo Asuncion Moreno Jan Odijk and Stelios Piperidis (Eds.). European Language Resources Association Marseille France 3498--3508."},{"key":"e_1_2_1_58_1","unstructured":"Spandana Singh. 2019. Everything in Moderation: The Limitations of Automated Tools in Content Moderation. Technical Report 1. Open Technology Institute."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.5117\/CCR2020.1.005.KATH"},{"key":"e_1_2_1_60_1","first-page":"3","article-title":"Misclassification in automated content analysis causes bias in regression. Can we fix it? Yes we can! Commun","volume":"18","author":"TeBlunthuis Nathan","year":"2024","unstructured":"Nathan TeBlunthuis, Valerie Hase, and Chung-Hong Chan. 2024. Misclassification in automated content analysis causes bias in regression. Can we fix it? Yes we can! Commun. Methods Meas. 18, 3 (July 2024), 278--299.","journal-title":"Methods Meas."},{"key":"e_1_2_1_61_1","volume-title":"The Four Basic Styles of Communication","author":"University of Kentucky Violence Intervention and Prevention Center 2014.","unstructured":"University of Kentucky Violence Intervention and Prevention Center 2014. The Four Basic Styles of Communication. University of Kentucky Violence Intervention and Prevention Center."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0243300"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-3012"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N16-2013"},{"key":"e_1_2_1_65_1","volume-title":"Proceedings of the GermEval 2018 Workshop.","author":"Wiegand M","year":"2018","unstructured":"M Wiegand, M Siegel, and J Ruppenhofer. 2018. Overview of the GermEval 2018 Shared Task on the Identification of Offensive Language. In Proceedings of the GermEval 2018 Workshop."},{"key":"e_1_2_1_66_1","unstructured":"Siqi Wu Angela Sch\u00f6pke-Gonzalez Sagar Kumar Libby Hemphill and Paul Resnick. 2023. HOT Speech: Comments from Political News Posts and Videos that were Annotated for Hateful Offensive and Toxic Content."},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","unstructured":"Ellery Wulczyn Nithum Thain and Lucas Dixon. 2017. Ex Machina: Personal Attacks Seen at Scale. In Proceedings of the 26th International Conference on World Wide Web (Perth Australia) (WWW '17). International World Wide Web Conferences Steering Committee Republic and Canton of Geneva CHE 1391--1399. doi:10.1145\/3038912.3052591","DOI":"10.1145\/3038912.3052591"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1080\/13600834.2021.1905593"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394231.3397902"},{"key":"e_1_2_1_70_1","volume-title":"Measuring inter-rater reliability for nominal data - which coefficients and confidence intervals are appropriate? BMC Med. Res. Methodol. 16 (Aug","author":"Zapf Antonia","year":"2016","unstructured":"Antonia Zapf, Stefanie Castell, Lars Morawietz, and Andr\u00e9 Karch. 2016. Measuring inter-rater reliability for nominal data - which coefficients and confidence intervals are appropriate? BMC Med. Res. Methodol. 16 (Aug. 2016), 93."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3711099","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3711099","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T09:28:15Z","timestamp":1755768495000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3711099"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,2]]},"references-count":70,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,5,2]]}},"alternative-id":["10.1145\/3711099"],"URL":"https:\/\/doi.org\/10.1145\/3711099","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,2]]},"assertion":[{"value":"2025-05-02","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}