{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,4]],"date-time":"2026-08-04T15:13:59Z","timestamp":1785856439548,"version":"3.56.0"},"reference-count":95,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2023,7,22]],"date-time":"2023-07-22T00:00:00Z","timestamp":1689984000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,9,30]]},"abstract":"<jats:p>Toxic conversations during software development interactions may have serious repercussions on a Free and Open Source Software (FOSS) development project. For example, victims of toxic conversations may become afraid to express themselves, therefore get demotivated, and may eventually leave the project. Automated filtering of toxic conversations may help a FOSS community maintain healthy interactions among its members. However, off-the-shelf toxicity detectors perform poorly on a software engineering dataset, such as one curated from code review comments. To counter this challenge, we present<jats:italic>ToxiCR<\/jats:italic>, a supervised learning based toxicity identification tool for code review interactions. ToxiCR includes a choice to select one of the 10 supervised learning algorithms, an option to select text vectorization techniques, eight preprocessing steps, and a large-scale labeled dataset of 19,651 code review comments. Two out of those eight preprocessing steps are software engineering domain specific. With our rigorous evaluation of the models with various combinations of preprocessing steps and vectorization techniques, we have identified the best combination for our dataset that boosts 95.8% accuracy and an 88.9% F1-score in identifying toxic texts. ToxiCR significantly outperforms existing toxicity detectors on our dataset. We have released our dataset, pre-trained models, evaluation results, and source code publicly, which is available at<jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/github.com\/WSU-SEAL\/ToxiCR\">https:\/\/github.com\/WSU-SEAL\/ToxiCR<\/jats:ext-link>.<\/jats:p>","DOI":"10.1145\/3583562","type":"journal-article","created":{"date-parts":[[2023,2,9]],"date-time":"2023-02-09T13:48:04Z","timestamp":1675950484000},"page":"1-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":41,"title":["Automated Identification of Toxic Code Reviews Using ToxiCR"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6440-7596","authenticated-orcid":false,"given":"Jaydeb","family":"Sarker","sequence":"first","affiliation":[{"name":"Wayne State University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0869-4962","authenticated-orcid":false,"given":"Asif Kamal","family":"Turzo","sequence":"additional","affiliation":[{"name":"Wayne State University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8133-7809","authenticated-orcid":false,"given":"Ming","family":"Dong","sequence":"additional","affiliation":[{"name":"Wayne State University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3178-6232","authenticated-orcid":false,"given":"Amiangshu","family":"Bosu","sequence":"additional","affiliation":[{"name":"Wayne State University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,7,22]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"GitHub. 2018. Annotation Instructions for Toxicity with Sub-Attributes. Retrieved February 18 2023 from https:\/\/github.com\/conversationai\/conversationai.github.io\/blob\/main\/crowdsourcing_annotation_schemes\/toxicity_with_subattributes.md."},{"key":"e_1_3_2_3_2","unstructured":"Kaggle. 2018. Toxic Comment Classification Challenge. Retrieved February 18 2023 from https:\/\/www.kaggle.com\/c\/jigsaw-toxic-comment-classification-challenge."},{"key":"e_1_3_2_4_2","first-page":"265","volume-title":"Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)","author":"Abadi Mart\u00edn","year":"2016","unstructured":"Mart\u00edn Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, et\u00a0al. 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). USENIX Association, Savannah, GA, 265\u2013283."},{"key":"e_1_3_2_5_2","article-title":"DocBERT: BERT for document classification","author":"Adhikari Ashutosh","year":"2019","unstructured":"Ashutosh Adhikari, Achyudh Ram, Raphael Tang, and Jimmy Lin. 2019. DocBERT: BERT for document classification. arXiv preprint arXiv:1904.08398 (2019).","journal-title":"arXiv preprint arXiv:1904.08398"},{"key":"e_1_3_2_6_2","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1145\/3270316.3271545","volume-title":"Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts","author":"Adinolf Sonam","year":"2018","unstructured":"Sonam Adinolf and Selen Turkay. 2018. Toxic behaviors in Esports games: Player perceptions and coping strategies. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts. 365\u2013372."},{"key":"e_1_3_2_7_2","first-page":"106","volume-title":"Proceedings of the 2017 32nd IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201917)","author":"Ahmed Toufique","year":"2017","unstructured":"Toufique Ahmed, Amiangshu Bosu, Anindya Iqbal, and Shahram Rahimi. 2017. SentiCR: A customized sentiment analysis tool for code review interactions. In Proceedings of the 2017 32nd IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201917). IEEE, Los Alamitos, CA, 106\u2013111."},{"key":"e_1_3_2_8_2","unstructured":"Perspective. n.d.Using machine learning to reduce toxicity online. Retrieved February 18 2023 from https:\/\/www.perspectiveapi.com\/."},{"key":"e_1_3_2_9_2","unstructured":"GitHub. 2018. Annotation Instructions for Toxicity with Sub-Attributes. Retrieved February 18 2023 from https:\/\/github.com\/conversationai\/conversationai.github.io\/blob\/master\/crowdsourcing_annotation_schemes\/toxicity_with_subattributes.md."},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2019.105210"},{"issue":"1","key":"e_1_3_2_11_2","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1093\/ijpor\/edw022","article-title":"Toxic talk: How online incivility can undermine perceptions of media","volume":"30","author":"Anderson Ashley A.","year":"2018","unstructured":"Ashley A. Anderson, Sara K. Yeo, Dominique Brossard, Dietram A. Scheufele, and Michael A. Xenos. 2018. Toxic talk: How online incivility can undermine perceptions of media. International Journal of Public Opinion Research 30, 1 (2018), 156\u2013168.","journal-title":"International Journal of Public Opinion Research"},{"key":"e_1_3_2_12_2","unstructured":"Anonymous. 2014. Leaving Toxic Open Source Communities. Retrieved February 18 2023 from https:\/\/modelviewculture.com\/pieces\/leaving-toxic-open-source-communities."},{"key":"e_1_3_2_13_2","unstructured":"Hayden Barnes. 2020. Toxicity in Open Source. Retrieved February 18 2023 from https:\/\/boxofcables.dev\/toxicity-in-linux-and-open-source\/."},{"issue":"227","key":"e_1_3_2_14_2","first-page":"357","article-title":"Application of the logistic function to bio-assay","volume":"39","author":"Berkson Joseph","year":"1944","unstructured":"Joseph Berkson. 1944. Application of the logistic function to bio-assay. Journal of the American Statistical Association 39, 227 (1944), 357\u2013365.","journal-title":"Journal of the American Statistical Association"},{"key":"e_1_3_2_15_2","doi-asserted-by":"crossref","first-page":"2017","DOI":"10.18653\/v1\/2021.findings-emnlp.173","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2021","author":"Bhat Meghana Moorthy","year":"2021","unstructured":"Meghana Moorthy Bhat, Saghar Hosseini, Ahmed Hassan, Paul Bennett, and Weisheng Li. 2021. Say \u201cYES\u201d to positivity: Detecting toxic language in workplace communications. In Findings of the Association for Computational Linguistics: EMNLP 2021. 2017\u20132029."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00051"},{"key":"e_1_3_2_17_2","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1109\/ESEM.2013.23","volume-title":"Proceedings of the 2013 ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement","author":"Bosu Amiangshu","year":"2013","unstructured":"Amiangshu Bosu and Jeffrey C. Carver. 2013. Impact of peer code review on peer impression formation: A survey. In Proceedings of the 2013 ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE, Los Alamitos, CA, 133\u2013142."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09708-7"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00058655"},{"key":"e_1_3_2_20_2","first-page":"1877","article-title":"Language models are few-shot learners","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et\u00a0al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems33. 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_21_2","first-page":"79","volume-title":"Proceedings of the 2017 7th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW\u201917)","author":"Calefato Fabio","year":"2017","unstructured":"Fabio Calefato, Filippo Lanubile, and Nicole Novielli. 2017. EmoTxt: A toolkit for emotion recognition from text. In Proceedings of the 2017 7th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW\u201917). IEEE, Los Alamitos, CA, 79\u201380."},{"key":"e_1_3_2_22_2","article-title":"Towards developing a theory of toxicity in the context of free\/open source software & peer production communities","author":"Carillo Kevin Daniel Andr\u00e9","year":"2016","unstructured":"Kevin Daniel Andr\u00e9 Carillo, Josianne Marsan, and Bogdan Negoita. 2016. Towards developing a theory of toxicity in the context of free\/open source software & peer production communities. In Proceedings of the SIGOPEN 2016 Developmental Workshop for Openness Research (ICIS\u201916).","journal-title":"Proceedings of the SIGOPEN 2016 Developmental Workshop for Openness Research (ICIS\u201916)."},{"key":"e_1_3_2_23_2","first-page":"125","volume-title":"Proceedings of the International AAAI Conference on Web and Social Media","volume":"13","author":"Chen Hao","year":"2019","unstructured":"Hao Chen, Susan McKeever, and Sarah Jane Delany. 2019. The use of deep learning distributed representations in the identification of abusive text. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 13. 125\u2013133."},{"key":"e_1_3_2_24_2","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/W16-6103","article-title":"Modelling radiological language with bidirectional long short-term memory networks","author":"Cornegruta Savelie","year":"2016","unstructured":"Savelie Cornegruta, Robert Bakewell, Samuel Withey, and Giovanni Montana. 2016. Modelling radiological language with bidirectional long short-term memory networks. In Proceedings of the 7th International Workshop on Health Text Mining and Information Analysis. 17\u201327.","journal-title":"Proceedings of the 7th International Workshop on Health Text Mining and Information Analysis."},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00994018"},{"key":"e_1_3_2_26_2","first-page":"250","volume-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics","author":"Danescu-Niculescu-Mizil Cristian","year":"2013","unstructured":"Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application to social factors. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. 250\u2013259."},{"issue":"2","key":"e_1_3_2_27_2","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1080\/10196780410001675059","article-title":"Managing conflicts in open source communities","volume":"14","author":"Joode R. Van Wendel De","year":"2004","unstructured":"R. Van Wendel De Joode. 2004. Managing conflicts in open source communities. Electronic Markets 14, 2 (2004), 104\u2013113.","journal-title":"Electronic Markets"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1423"},{"key":"e_1_3_2_29_2","unstructured":"Maeve Duggan. 2017. Online Harassment 2017. Retrieved February 18 2023 from https:\/\/www.pewresearch.org\/internet\/2017\/07\/11\/online-harassment-2017\/."},{"key":"e_1_3_2_30_2","first-page":"174","volume-title":"Proceedings of the 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE\u201920)","author":"Egelman Carolyn D.","year":"2020","unstructured":"Carolyn D. Egelman, Emerson Murphy-Hill, Elizabeth Kammer, Margaret Morrow Hodges, Collin Green, Ciera Jaspan, and James Lin. 2020. Predicting developers\u2019 negative feelings about code review. In Proceedings of the 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE\u201920). IEEE, Los Alamitos, CA, 174\u2013185."},{"key":"e_1_3_2_31_2","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1145\/3299819.3299845","volume-title":"Proceedings of the 2018 Artificial Intelligence and Cloud Computing Conference","author":"Elnaggar Ahmed","year":"2018","unstructured":"Ahmed Elnaggar, Bernhard Waltl, Ingo Glaser, J\u00f6rg Landthaler, Elena Scepankova, and Florian Matthes. 2018. Stop illegal comments: A multi-task deep learning approach. In Proceedings of the 2018 Artificial Intelligence and Cloud Computing Conference. 41\u201347."},{"issue":"5","key":"e_1_3_2_32_2","article-title":"Deep gated recurrent and convolutional network hybrid model for univariate time series classification","volume":"10","author":"Elsayed Nelly","year":"2019","unstructured":"Nelly Elsayed, Anthony S. Maida, and Magdy Bayoumi. 2019. Deep gated recurrent and convolutional network hybrid model for univariate time series classification. International Journal of Advanced Computer Science and Applications 10, 5 (2019), 654\u2013664.","journal-title":"International Journal of Advanced Computer Science and Applications"},{"key":"e_1_3_2_33_2","unstructured":"Samir Faci. 2020. The Toxicity of Open Source. Retrieved February 18 2023 from https:\/\/www.esamir.com\/20\/12\/23\/the-toxicity-of-open-source\/."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3479497"},{"key":"e_1_3_2_35_2","first-page":"705","volume-title":"Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work and Social Computing","author":"Filippova Anna","year":"2016","unstructured":"Anna Filippova and Hichang Cho. 2016. The effects and antecedents of conflict in free and open source software development. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work and Social Computing. 705\u2013716."},{"key":"e_1_3_2_36_2","unstructured":"LibreOffice. n.d. The Document Foundation Code of Conduct. Retrieved February 18 2023 from https:\/\/www.documentfoundation.org\/foundation\/code-of-conduct\/."},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1013203451"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3200947.3208069"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2005.06.042"},{"key":"e_1_3_2_40_2","doi-asserted-by":"crossref","first-page":"21","DOI":"10.18653\/v1\/W18-5103","volume-title":"Proceedings of the 2nd Workshop on Abusive Language Online (ALW2\u201918)","author":"Gunasekara Isuru","year":"2018","unstructured":"Isuru Gunasekara and Isar Nejadgholi. 2018. A review of standard text classification practices for multi-label toxicity identification of online content. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2\u201918). 21\u201325."},{"key":"e_1_3_2_41_2","doi-asserted-by":"crossref","unstructured":"Sanuri Dananja Gunawardena Peter Devine Isabelle Beaumont Lola Garden Emerson Rex Murphy-Hill and Kelly Blincoe. 2022. Destructive criticism in software code review impacts inclusion. Proceedings of the ACM on Human-Computer Interaction 6 CSCW2 (2022) Article 292 29 pages.","DOI":"10.1145\/3555183"},{"key":"e_1_3_2_42_2","unstructured":"Laura Hanu and Unitary Team. 2020. Detoxify. Retrieved February 18 2023 from https:\/\/github.com\/unitaryai\/detoxify."},{"key":"e_1_3_2_43_2","doi-asserted-by":"crossref","first-page":"278","DOI":"10.1109\/ICDAR.1995.598994","volume-title":"Proceedings of 3rd International Conference on Document Analysis and Recognition","volume":"1","author":"Ho Tin Kam","year":"1995","unstructured":"Tin Kam Ho. 1995. Random decision forests. In Proceedings of 3rd International Conference on Document Analysis and Recognition, Vol. 1. IEEE, Los Alamitos, CA, 278\u2013282."},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_3_2_45_2","article-title":"Deceiving Google\u2019s perspective API built for detecting toxic comments","author":"Hosseini Hossein","year":"2017","unstructured":"Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017. Deceiving Google\u2019s perspective API built for detecting toxic comments. arXiv preprint arXiv:1702.08138 (2017).","journal-title":"arXiv preprint arXiv:1702.08138"},{"key":"e_1_3_2_46_2","article-title":"Bidirectional LSTM-CRF models for sequence tagging","author":"Huang Zhiheng","year":"2015","unstructured":"Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991 (2015).","journal-title":"arXiv preprint arXiv:1508.01991"},{"key":"e_1_3_2_47_2","doi-asserted-by":"crossref","first-page":"700","DOI":"10.1109\/ICSE.2019.00079","volume-title":"Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE\u201919)","author":"Imtiaz N.","year":"2019","unstructured":"N. Imtiaz, J. Middleton, J. Chakraborty, N. Robson, G. Bai, and E. Murphy-Hill. 2019. Investigating the effects of gender bias on GitHub. In Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE\u201919). 700\u2013711."},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3357384.3357891"},{"key":"e_1_3_2_49_2","first-page":"1","volume-title":"Proceedings of the 2011 44th Hawaii International Conference on System Sciences","author":"Jensen Carlos","year":"2011","unstructured":"Carlos Jensen, Scott King, and Victor Kuechler. 2011. Joining free\/open source software communities: An analysis of newbies\u2019 first interactions on project mailing lists. In Proceedings of the 2011 44th Hawaii International Conference on System Sciences. IEEE, Los Alamitos, CA, 1\u201310."},{"key":"e_1_3_2_50_2","doi-asserted-by":"crossref","first-page":"562","DOI":"10.18653\/v1\/P17-1052","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Volume 1 (Long Papers)","author":"Johnson Rie","year":"2017","unstructured":"Rie Johnson and Tong Zhang. 2017. Deep pyramid convolutional neural networks for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Volume 1 (Long Papers). 562\u2013570."},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-016-9493-x"},{"key":"e_1_3_2_52_2","unstructured":"Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. Retrieved February 18 2023 from https:\/\/dblp.org\/rec\/journals\/corr\/KingmaB14.html."},{"key":"e_1_3_2_53_2","first-page":"364","volume-title":"Proceedings of the 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA\u201917)","author":"Kowsari Kamran","year":"2017","unstructured":"Kamran Kowsari, Donald E. Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi, Matthew S. Gerber, and Laura E. Barnes. 2017. HDLTex: Hierarchical deep learning for text classification. In Proceedings of the 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA\u201917). IEEE, Los Alamitos, CA, 364\u2013371."},{"key":"e_1_3_2_54_2","first-page":"299","volume-title":"Proceedings of the 17th Symposium on Usable Privacy and Security (SOUPS\u201921)","author":"Kumar Deepak","year":"2021","unstructured":"Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021. Designing toxic content classification for a diversity of perspectives. In Proceedings of the 17th Symposium on Usable Privacy and Security (SOUPS\u201921). 299\u2013318."},{"key":"e_1_3_2_55_2","article-title":"Towards robust toxic content classification","author":"Kurita Keita","year":"2019","unstructured":"Keita Kurita, Anna Belova, and Antonios Anastasopoulos. 2019. Towards robust toxic content classification. arXiv preprint arXiv:1912.06872 (2019).","journal-title":"arXiv preprint arXiv:1912.06872"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00766-018-0293-2"},{"key":"e_1_3_2_57_2","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1145\/3180155.3180195","volume-title":"Proceedings of the 40th International Conference on Software Engineering","author":"Lin Bin","year":"2018","unstructured":"Bin Lin, Fiorella Zampetti, Gabriele Bavota, Massimiliano Di Penta, Michele Lanza, and Rocco Oliveto. 2018. Sentiment analysis for software engineering: How far can we go? In Proceedings of the 40th International Conference on Software Engineering. 94\u2013104."},{"key":"e_1_3_2_58_2","article-title":"Decoupled weight decay regularization","author":"Loshchilov Ilya","year":"2018","unstructured":"Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representations.","journal-title":"Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_2_59_2","first-page":"3111","volume-title":"Advances in Neural Information Processing Systems 26","author":"Mikolov Tomas","year":"2013","unstructured":"Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26. 3111\u20133119."},{"key":"e_1_3_2_60_2","doi-asserted-by":"crossref","unstructured":"Courtney Miller Sophie Cohen Daniel Klug Bodgan Vasilescu and Christian K\u00e4stner. 2022. \u201cDid you miss my comment or what?\u201d Understanding toxicity in open source discussions. In Proceedings of the International Conference on Software Engineering (ICSE\u201922) . IEEE Los Alamitos CA.","DOI":"10.1145\/3510003.3510111"},{"key":"e_1_3_2_61_2","article-title":"Neural character-based composition models for abuse detection","author":"Mishra Pushkar","year":"2018","unstructured":"Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. 2018. Neural character-based composition models for abuse detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP\u201918). 1.","journal-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP\u201918)."},{"key":"e_1_3_2_62_2","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1109\/MSR.2013.6624002","volume-title":"Proceedings of the 2013 10th Working Conference on Mining Software Repositories (MSR\u201913)","author":"Mukadam Murtuza","year":"2013","unstructured":"Murtuza Mukadam, Christian Bird, and Peter C. Rigby. 2013. Gerrit software code review data from Android. In Proceedings of the 2013 10th Working Conference on Mining Software Repositories (MSR\u201913). IEEE, Los Alamitos, CA, 45\u201348."},{"key":"e_1_3_2_63_2","unstructured":"Dawn Nafus James Leach and Bernhard Krieger. 2006. FLOSSPOLS Deliverable D 16 Gender: Integrated Report of Findings . FLOSSPOLS."},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2883062"},{"key":"e_1_3_2_65_2","doi-asserted-by":"crossref","first-page":"364","DOI":"10.1145\/3196398.3196403","volume-title":"Proceedings of the 2018 IEEE\/ACM 15th International Conference on Mining Software Repositories (MSR\u201918)","author":"Novielli Nicole","year":"2018","unstructured":"Nicole Novielli, Daniela Girardi, and Filippo Lanubile. 2018. A benchmark study on sentiment analysis for software engineering research. In Proceedings of the 2018 IEEE\/ACM 15th International Conference on Mining Software Repositories (MSR\u201918). IEEE, Los Alamitos, CA, 364\u2013375."},{"key":"e_1_3_2_66_2","unstructured":"OpenStack. n.d. OpenStack Code of Conduct. Retrieved February 18 2023 from https:\/\/wiki.openstack.org\/wiki\/Conduct."},{"key":"e_1_3_2_67_2","volume-title":"Proceedings of the 26th IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER\u201919)","author":"Paul Rajshakhar","year":"2019","unstructured":"Rajshakhar Paul, Amiangshu Bosu, and Kazi Zakia Sultana. 2019. Expressions of sentiments during code reviews: Male vs. female. In Proceedings of the 26th IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER\u201919). IEEE, Los Alamitos, CA."},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"e_1_3_2_70_2","unstructured":"Android Open Source Project. n.d. Code of Conduct. Retrieved February 18 2023 from https:\/\/source.android.com\/setup\/cofc."},{"key":"e_1_3_2_71_2","doi-asserted-by":"crossref","unstructured":"Huilian Sophie Qiu Bogdan Vasilescu Christian K\u00e4stner Carolyn Denomme Egelman Ciera Nicole Christopher Jaspan and Emerson Rex Murphy-Hill. 2022. Detecting interpersonal conflict in issues and code review: Cross pollinating open-and closed-source approaches. In Proceedings of the 2022 ACM\/IEEE 44th International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS\u201922) . 41\u201355.","DOI":"10.1145\/3510458.3513019"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00116251"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1177\/1094428110375002"},{"key":"e_1_3_2_74_2","volume-title":"Proceedings of the International Conference on Software Engineering, New Ideas, and Emerging Results (ICSE\u201920)","author":"Raman Naveen","year":"2020","unstructured":"Naveen Raman, Minxuan Cao, Yulia Tsvetkov, Christian K\u00e4stner, and Bogdan Vasilescu. 2020. Stress and burnout in open source: Toward finding, understanding, and mitigating unhealthy interactions. In Proceedings of the International Conference on Software Engineering, New Ideas, and Emerging Results (ICSE\u201920). ACM, New York, NY."},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1038\/323533a0"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1163"},{"key":"e_1_3_2_77_2","unstructured":"Jaydeb Sarkar Asif Turzo Ming Dong and Amiangshu Bosu. 2022. ToxiCR: Replication Rackage. Retrieved February 18 2023 from https:\/\/github.com\/WSU-SEAL\/ToxiCR."},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC51365.2020.00030"},{"key":"e_1_3_2_79_2","unstructured":"Jaydeb Sarker Asif Kamal Turzo Ming Dong and Amiangshu Bosu. 2022. WSU SEAL implementation of the STRUDEL toxicity detector. Retrieved February 18 2023 from https:\/\/github.com\/WSU-SEAL\/toxicity-detector\/tree\/master\/WSU_SEAL."},{"key":"e_1_3_2_80_2","volume-title":"Model Assisted Survey Sampling","author":"S\u00e4rndal Carl-Erik","year":"2003","unstructured":"Carl-Erik S\u00e4rndal, Bengt Swensson, and Jan Wretman. 2003. Model Assisted Survey Sampling. Springer Science & Business Media."},{"key":"e_1_3_2_81_2","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1007\/978-0-387-21579-2_9","article-title":"The boosting approach to machine learning: An overview","author":"Schapire Robert E.","year":"2003","unstructured":"Robert E. Schapire. 2003. The boosting approach to machine learning: An overview. In Nonlinear Estimation and Classification. Lecture Notes in Computer Science, Vol. 171. Springer, 149\u2013171.","journal-title":"Nonlinear Estimation and Classification."},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2015.623"},{"key":"e_1_3_2_83_2","first-page":"98","volume-title":"Proceedings of the 1st Workshop on Trolling, Aggression, and Cyberbullying (TRAC\u201918)","author":"Srivastava Saurabh","year":"2018","unstructured":"Saurabh Srivastava, Prerna Khurana, and Vartika Tewari. 2018. Identifying aggression and toxicity in comments using capsule network. In Proceedings of the 1st Workshop on Trolling, Aggression, and Cyberbullying (TRAC\u201918). 98\u2013105."},{"key":"e_1_3_2_84_2","first-page":"199","volume-title":"Proceedings of the IFIP International Conference on Open Source Systems","author":"Steinmacher Igor","year":"2014","unstructured":"Igor Steinmacher and Marco Aur\u00e9lio Gerosa. 2014. How to support newcomers onboarding to open source software projects. In Proceedings of the IFIP International Conference on Open Source Systems. 199\u2013201."},{"key":"e_1_3_2_85_2","doi-asserted-by":"crossref","first-page":"534","DOI":"10.1109\/ICSME.2017.14","volume-title":"Proceedings of the 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME\u201917)","author":"Terdchanakul Pannavat","year":"2017","unstructured":"Pannavat Terdchanakul, Hideaki Hata, Passakorn Phannachitta, and Kenichi Matsumoto. 2017. Bug or not? Bug report classification using n-gram IDF. In Proceedings of the 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME\u201917). IEEE, Los Alamitos, CA, 534\u2013538."},{"key":"e_1_3_2_86_2","doi-asserted-by":"crossref","first-page":"683","DOI":"10.1609\/icwsm.v14i1.7334","article-title":"Empirical analysis of multi-task learning for reducing identity bias in toxic comment detection","author":"Vaidya Ameya","year":"2020","unstructured":"Ameya Vaidya, Feng Mai, and Yue Ning. 2020. Empirical analysis of multi-task learning for reducing identity bias in toxic comment detection. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM\u201920).683\u2013693.","journal-title":"Proceedings of the International AAAI Conference on Web and Social Media (ICWSM\u201920)."},{"key":"e_1_3_2_87_2","doi-asserted-by":"crossref","first-page":"33","DOI":"10.18653\/v1\/W18-5105","article-title":"Challenges for toxic comment classification: An in-depth error analysis","author":"Aken Betty van","year":"2018","unstructured":"Betty van Aken, Julian Risch, Ralf Krestel, and Alexander L\u00f6ser. 2018. Challenges for toxic comment classification: An in-depth error analysis. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2\u201918).33\u201342.","journal-title":"Proceedings of the 2nd Workshop on Abusive Language Online (ALW2\u201918)."},{"key":"e_1_3_2_88_2","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_89_2","doi-asserted-by":"crossref","first-page":"1587","DOI":"10.18653\/v1\/2020.semeval-1.207","volume-title":"Proceedings of the 14th Workshop on Semantic Evaluation","author":"Wang Susan","year":"2020","unstructured":"Susan Wang and Zita Marinho. 2020. Nova-Wang at SemEval-2020 Task 12: OffensEmblert: An ensemble ofoffensive language classifiers. In Proceedings of the 14th Workshop on Semantic Evaluation. 1587\u20131597."},{"key":"e_1_3_2_90_2","first-page":"7","article-title":"Demoting racial bias in hate speech detection","author":"Xia Mengzhou","year":"2020","unstructured":"Mengzhou Xia, Anjalie Field, and Yulia Tsvetkov. 2020. Demoting racial bias in hate speech detection. In Proceedings of the 8th International Workshop on Natural Language Processing for Social Media.7\u201314.","journal-title":"Proceedings of the 8th International Workshop on Natural Language Processing for Social Media."},{"key":"e_1_3_2_91_2","article-title":"XLNet: Generalized autoregressive pretraining for language understanding","volume":"32","author":"Yang Zhilin","year":"2019","unstructured":"Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32.","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"1","key":"e_1_3_2_92_2","first-page":"13","article-title":"Toxic comment classification","volume":"3","author":"Zaheri Sara","year":"2020","unstructured":"Sara Zaheri, Jeff Leath, and David Stroud. 2020. Toxic comment classification. SMU Data Science Review 3, 1 (2020), 13.","journal-title":"SMU Data Science Review"},{"key":"e_1_3_2_93_2","doi-asserted-by":"crossref","first-page":"1425","DOI":"10.18653\/v1\/2020.semeval-1.188","volume-title":"Proceedings of the 14th Workshop on Semantic Evaluation","author":"Zampieri Marcos","year":"2020","unstructured":"Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and \u00c7a\u011fr\u0131 \u00c7\u00f6ltekin. 2020. SemEval-2020 Task 12: Multilingual offensive language identification in social media (OffensEval 2020). In Proceedings of the 14th Workshop on Semantic Evaluation. 1425\u20131447."},{"key":"e_1_3_2_94_2","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","volume":"1","author":"Zhang Justine","year":"2018","unstructured":"Justine Zhang, Jonathan P. Chang, Cristian Danescu-Niculescu-Mizil, Lucas Dixon, Yiqing Hua, Nithum Tahin, and Dario Taraborelli. 2018. Conversations gone awry: Detecting early signs of conversational failure. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Vol. 1."},{"key":"e_1_3_2_95_2","first-page":"649","article-title":"Character-level convolutional networks for text classification","volume":"28","author":"Zhang Xiang","year":"2015","unstructured":"Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems 28.649\u2013657.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_96_2","volume-title":"Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201915)","author":"Zhu Yukun","year":"2015","unstructured":"Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201915)."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3583562","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3583562","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:37:54Z","timestamp":1750178274000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3583562"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,22]]},"references-count":95,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,9,30]]}},"alternative-id":["10.1145\/3583562"],"URL":"https:\/\/doi.org\/10.1145\/3583562","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,22]]},"assertion":[{"value":"2022-02-24","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-01-17","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-07-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}