{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T04:25:17Z","timestamp":1785558317141,"version":"3.56.0"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","funder":[{"DOI":"10.13039\/100000001","name":"US National Science Foundation","doi-asserted-by":"crossref","award":["850475, 234038"],"award-info":[{"award-number":["850475, 234038"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,19]]},"abstract":"<jats:p>Toxicity on GitHub can severely impact Open Source Software (OSS) development communities. To mitigate such behavior, a better understanding of its nature and how various measurable characteristics of project contexts and participants are associated with its prevalence is necessary. To achieve this goal, we conducted a large-scale mixed-method empirical study of 2,828 GitHub-based OSS projects randomly selected based on a stratified sampling strategy. Using ToxiCR, an SE domain-specific toxicity detector, we automatically classified each comment as toxic or non-toxic. Additionally, we manually analyzed a random sample of 600 comments to validate ToxiCR's performance and gain insights into the nature of toxicity within our dataset. The results of our study suggest that profanity is the most frequent toxicity on GitHub, followed by trolling and insults. While a project's popularity is positively associated with the prevalence of toxicity, its issue resolution rate has the opposite association. Corporate-sponsored projects are less toxic, but gaming projects are seven times more toxic than non-gaming ones. OSS contributors who have authored toxic comments in the past are significantly more likely to repeat such behavior. Moreover, such individuals are more likely to become targets of toxic texts.<\/jats:p>","DOI":"10.1145\/3715744","type":"journal-article","created":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T15:16:02Z","timestamp":1750346162000},"page":"623-646","source":"Crossref","is-referenced-by-count":7,"title":["The Landscape of Toxicity: An Empirical Investigation of Toxicity on GitHub"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6440-7596","authenticated-orcid":false,"given":"Jaydeb","family":"Sarker","sequence":"first","affiliation":[{"name":"University of Nebraska at Omaha, Omaha, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0869-4962","authenticated-orcid":false,"given":"Asif Kamal","family":"Turzo","sequence":"additional","affiliation":[{"name":"Wayne State University, Detroit, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3178-6232","authenticated-orcid":false,"given":"Amiangshu","family":"Bosu","sequence":"additional","affiliation":[{"name":"Wayne State University, Detroit, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,19]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"The diversity crisis in software development","author":"Albusays Khaled","year":"2021","unstructured":"Khaled Albusays, Pernille Bjorn, Laura Dabbish, Denae Ford, Emerson Murphy-Hill, Alexander Serebrenik, and MargaretAnne Storey. 2021. The diversity crisis in software development. IEEE Software 38, 2 ( 2021 ), 19-25."},{"key":"e_1_2_1_2_1","unstructured":"Paul Allison. 2014. Prediction vs. causation in regression analysis. Statistical Horizons 703 ( 2014 )."},{"key":"e_1_2_1_3_1","unstructured":"Anonymous Author. 2014. Leaving Toxic Open Source Communities. https:\/\/modelviewculture.com\/pieces\/leaving-toxicopen-source-communities."},{"key":"e_1_2_1_4_1","unstructured":"Hayden Barnes. 2020. Toxicity in Open Source. https:\/\/boxofcables.dev\/toxicity-in-linux-and-open-source\/."},{"key":"e_1_2_1_5_1","doi-asserted-by":"crossref","unstructured":"Mohammad R Basirati Marko Otasevic Koushyar Rajavi Markus B\u00f6hm and Helmut Krcmar. 2020. Understanding the relationship of conflict and success in software development projects. Information and Software Technology 126 ( 2020 ).","DOI":"10.1016\/j.infsof.2020.106331"},{"key":"e_1_2_1_6_1","article-title":"Multinomial Logistic Regression: Usage and Application in Risk Analysis","volume":"5","author":"Bayaga Anass","year":"2010","unstructured":"Anass Bayaga. 2010. Multinomial Logistic Regression: Usage and Application in Risk Analysis. Journal of applied quantitative methods 5, 2 ( 2010 ).","journal-title":"Journal of applied quantitative methods"},{"key":"e_1_2_1_7_1","article-title":"Measuring toxicity toward women in game-based communities","volume":"1","author":"Belskie Matthew","year":"2023","unstructured":"Matthew Belskie, Hanlin Zhang, and Bradley M Hemminger. 2023. Measuring toxicity toward women in game-based communities. Journal of Electronic Gaming and Esports 1, 1 ( 2023 ).","journal-title":"Journal of Electronic Gaming and Esports"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445157"},{"key":"e_1_2_1_9_1","doi-asserted-by":"crossref","unstructured":"Nicolas B\u00e9rub\u00e9 Gita Ghiasi Maxime Sainte-Marie et al. 2020. Wiki-Gendersort: Automatic gender detection using first names in Wikipedia. ( 2020 ).","DOI":"10.31235\/osf.io\/ezw7p"},{"key":"e_1_2_1_10_1","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2021. 2017","author":"Bhat Meghana Moorthy","year":"2021","unstructured":"Meghana Moorthy Bhat, Saghar Hosseini, Ahmed Hassan, Paul Bennett, and Weisheng Li. 2021. Say 'YES'to positivity: Detecting toxic language in workplace communications. In Findings of the Association for Computational Linguistics: EMNLP 2021. 2017-2029."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2015.21"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2019.8870179"},{"key":"e_1_2_1_13_1","volume-title":"SIGOPEN 2016 ( 2016 ).","author":"Andr\u00e9 Carillo Kevin Daniel","year":"2016","unstructured":"Kevin Daniel Andr\u00e9 Carillo, Josianne Marsan, and Bogdan Negoita. 2016. Towards Developing a Theory of Toxicity in the Context of Free\/Open Source Software & Peer Production Communities. SIGOPEN 2016 ( 2016 )."},{"key":"e_1_2_1_14_1","doi-asserted-by":"crossref","first-page":"254","DOI":"10.1145\/3463274.3463805","article-title":"Towards ofensive language detection and reduction in four Software Engineering communities","author":"Cheriyan Jithin","year":"2021","unstructured":"Jithin Cheriyan, Bastin Tony Roy Savarimuthu, and Stephen Cranefield. 2021. Towards ofensive language detection and reduction in four Software Engineering communities. In Evaluation and Assessment in Software Engineering. 254-259.","journal-title":"Evaluation and Assessment in Software Engineering."},{"key":"e_1_2_1_15_1","volume-title":"Sampling techniques","author":"Cochran William G","year":"1977","unstructured":"William G Cochran. 1977. Sampling techniques. Johan Wiley & Sons Inc ( 1977 )."},{"key":"e_1_2_1_16_1","doi-asserted-by":"crossref","unstructured":"Jacob Cohen. 1960. A coeficient of agreement for nominal scales. Educational and psychological measurement 20 1 ( 1960 ) 37-46.","DOI":"10.1177\/001316446002000104"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1669-1671","author":"Cohen Sophie","year":"2021","unstructured":"Sophie Cohen. 2021. Contextualizing toxicity in open source: a qualitative study. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1669-1671."},{"key":"e_1_2_1_18_1","doi-asserted-by":"crossref","unstructured":"Daniel Alencar da Costa Shane McIntosh Christoph Treude Uira Kulesza and Ahmed E Hassan. 2018. The impact of rapid release cycles on the integration delay of fixed issues. Empirical Software Engineering 23 ( 2018 ) 835-904.","DOI":"10.1007\/s10664-017-9548-7"},{"key":"e_1_2_1_19_1","first-page":"560","volume-title":"Sampling Projects in GitHub for MSR Studies. In 18th IEEE\/ACM International Conference on Mining Software Repositories, MSR","author":"Dabic Ozren","year":"2021","unstructured":"Ozren Dabic, Emad Aghajani, and Gabriele Bavota. 2021. Sampling Projects in GitHub for MSR Studies. In 18th IEEE\/ACM International Conference on Mining Software Repositories, MSR 2021. IEEE, 560-564."},{"key":"e_1_2_1_20_1","volume-title":"Octoverse: The state of open source and rise of AI in 2023. https:\/\/github.blog\/2023-11-08-the-state-ofopen-source-and-ai\/.","author":"Daigle Kyle","year":"2023","unstructured":"Kyle Daigle. 2023. Octoverse: The state of open source and rise of AI in 2023. https:\/\/github.blog\/2023-11-08-the-state-ofopen-source-and-ai\/."},{"key":"e_1_2_1_21_1","first-page":"86","volume-title":"Proceedings of the First Italian Conference on Cybersecurity (ITASEC17)","author":"Vigna Fabio Del","year":"2017","unstructured":"Fabio Del Vigna, Andrea Cimino, Felice Dell'Orletta, Marinella Petrocchi, and Maurizio Tesconi. 2017. Hate me, hate me not: Hate speech detection on facebook. In Proceedings of the First Italian Conference on Cybersecurity (ITASEC17). 86-95."},{"key":"e_1_2_1_22_1","unstructured":"Alan Diggs. 2021. Windows is sh*t:' Linux Users and The Technical Superiority Problem. https:\/\/medium.com\/ linuxforeveryone\/windows-is-sh-t-linux-users-and-the-technical-superiority-problem-196a597aa860\/."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2019.8668024"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380414"},{"key":"e_1_2_1_25_1","article-title":"Age and gender estimation of unfiltered faces","volume":"9","author":"Eidinger Eran","year":"2014","unstructured":"Eran Eidinger, Roee Enbar, and Tal Hassner. 2014. Age and gender estimation of unfiltered faces. IEEE Transactions on information forensics and security 9, 12 ( 2014 ), 2170-2179.","journal-title":"IEEE Transactions on information forensics and security"},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the 19th International Conference on Mining Software Repositories. 309-320","author":"Ferreira Isabella","year":"2022","unstructured":"Isabella Ferreira, Bram Adams, and Jinghui Cheng. 2022. How heated is it? Understanding GitHub locked issues. In Proceedings of the 19th International Conference on Mining Software Repositories. 309-320."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 ( 2021 ), 1-35","author":"Ferreira Isabella","year":"2021","unstructured":"Isabella Ferreira, Jinghui Cheng, and Bram Adams. 2021. The\" Shut the f** k up\" Phenomenon: Characterizing Incivility in Open Source Code Review Discussions. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 ( 2021 ), 1-35."},{"key":"e_1_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Isabella Ferreira Ahlaam Rafiq and Jinghui Cheng. 2024. Incivility detection in open source code review and issue discussions. Journal of Systems and Software 209 ( 2024 ) 111935.","DOI":"10.1016\/j.jss.2023.111935"},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the ACM on Software Engineering 1, FSE ( 2024 ), 1632-1655","author":"Rahman Md Shamimur","year":"2024","unstructured":"Md Shamimur Rahman, Zadia Codabux, and Chanchal K Roy. 2024. Do Words Have Power? Understanding and Fostering Civility in Code Review Discussion. Proceedings of the ACM on Software Engineering 1, FSE ( 2024 ), 1632-1655."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377816.3381732"},{"key":"e_1_2_1_31_1","unstructured":"Jim Salter. 2021. The Perl Foundation is fragmenting over Code of Conduct enforcement. https:\/\/arstechnica.com\/gadgets\/ 2021\/08\/the-perl-foundation-is-fragmenting-over-code-of-conduct-enforcement\/."},{"key":"e_1_2_1_32_1","doi-asserted-by":"crossref","unstructured":"Luc\u00eda Santamar\u00eda and Helena Mihaljevi\u0107. 2018. Comparison and benchmark of name-to-gender inference services. PeerJ Computer Science 4 ( 2018 ) e156.","DOI":"10.7717\/peerj-cs.156"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM56168.2023.10304855"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC51365.2020.00030"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.14802294"},{"key":"e_1_2_1_36_1","volume-title":"The Varclus Procedure","author":"Sarle WS","year":"1990","unstructured":"WS Sarle. 1990. SAS\/STAT User's Guide: The Varclus Procedure. SAS Institute. Inc., Cary, NC, USA ( 1990 )."},{"key":"e_1_2_1_37_1","unstructured":"Thomas J Smith and Cornelius M McKenna. 2013. A comparison of logistic regression pseudo R2 indices. Multiple Linear Regression Viewpoints 39 2 ( 2013 ) 17-26."},{"key":"e_1_2_1_38_1","unstructured":"Laerd Statistics. 2013. Spearman's rank-order correlation. Laerd Statistics ( 2013 )."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675215"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/CHASE.2013.6614728"},{"key":"e_1_2_1_41_1","doi-asserted-by":"crossref","unstructured":"Vikram N Subramanian Ifraz Rehman Meiyappan Nagappan and Raula Gaikovina Kula. 2020. Analyzing first contributions on github: What do newcomers do? IEEE Software 39 1 ( 2020 ) 93-101.","DOI":"10.1109\/MS.2020.3041241"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3559571"},{"key":"e_1_2_1_43_1","volume-title":"Identifying Sexism and Misogyny in Pull Request Comments. In 37th IEEE\/ACM International Conference on Automated Software Engineering. 1-3.","author":"Sultana Sayma","year":"2022","unstructured":"Sayma Sultana. 2022b. Identifying Sexism and Misogyny in Pull Request Comments. In 37th IEEE\/ACM International Conference on Automated Software Engineering. 1-3."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3475716.3484189"},{"key":"e_1_2_1_45_1","volume-title":"Asif Kamal Turzo, and Amiangshu Bosu","author":"Sultana Sayma","year":"2023","unstructured":"Sayma Sultana, Asif Kamal Turzo, and Amiangshu Bosu. 2023. Code reviews in open source projects: how do gender biases afect participation and outcomes? Empirical Software Engineering 28, 4 ( 2023 ), 92."},{"key":"e_1_2_1_46_1","doi-asserted-by":"crossref","unstructured":"Patanamon Thongtanunam Shane McIntosh Ahmed E Hassan and Hajimu Iida. 2017. Review participation in modern code review: An empirical study of the android Qt and OpenStack projects. Empirical Software Engineering 22 ( 2017 ) 768-817.","DOI":"10.1007\/s10664-016-9452-6"},{"key":"e_1_2_1_47_1","doi-asserted-by":"crossref","unstructured":"Asif Kamal Turzo and Amiangshu Bosu. 2024. What makes a code review useful to opendev developers? an empirical investigation. Empirical Software Engineering 29 1 ( 2024 ) 6.","DOI":"10.1007\/s10664-023-10411-x"},{"key":"e_1_2_1_48_1","doi-asserted-by":"crossref","unstructured":"Bogdan Vasilescu Andrea Capiluppi and Alexander Serebrenik. 2014. Gender representation and online participation: A quantitative study. Interacting with Computers 26 5 ( 2014 ) 488-511.","DOI":"10.1093\/iwc\/iwt047"},{"key":"e_1_2_1_49_1","unstructured":"Steven Vaughan-Nichols. 2018. Linus Torvalds takes a break from Linux. https:\/\/www.zdnet.com\/article\/linus-torvaldstakes-a-break-from-linux\/."},{"key":"e_1_2_1_50_1","doi-asserted-by":"crossref","unstructured":"Michael R Veall and Klaus F Zimmermann. 1994. Evaluating Pseudo-R2's for binary probit models. Quality and Quantity 28 2 ( 1994 ) 151-164.","DOI":"10.1007\/BF01102759"},{"key":"e_1_2_1_51_1","unstructured":"Sawyer X. 2021. I Am Stepping Down from PSC and Core Efective Immediately. https:\/\/perl.topicbox.com\/groups\/perlcore\/T7a4f1bf9e069641f-Mebbcc218eb006f0da34c7a41"},{"key":"e_1_2_1_52_1","doi-asserted-by":"crossref","unstructured":"Li Xu Chris Gotwalt Yili Hong Caleb B King and William Q Meeker. 2020. Applications of the fractional-random-weight bootstrap. The American Statistician 74 4 ( 2020 ) 345-358.","DOI":"10.1080\/00031305.2020.1731599"},{"key":"e_1_2_1_53_1","article-title":"Toxic comment classification","volume":"3","author":"Zaheri Sara","year":"2020","unstructured":"Sara Zaheri, Jef Leath, and David Stroud. 2020. Toxic comment classification. SMU Data Science Review 3, 1 ( 2020 ), 13.","journal-title":"SMU Data Science Review"},{"key":"e_1_2_1_54_1","first-page":"500","volume-title":"Companion Proceedings of the Web Conference","author":"Zhao Zhixue","year":"2021","unstructured":"Zhixue Zhao, Ziqi Zhang, and Frank Hopfgartner. 2021. A comparative study of using pre-trained language models for toxic comment classification. In Companion Proceedings of the Web Conference 2021. 500-507."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380412"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3715744","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T15:19:28Z","timestamp":1750346368000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3715744"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,19]]},"references-count":55,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2025,6,19]]}},"alternative-id":["10.1145\/3715744"],"URL":"https:\/\/doi.org\/10.1145\/3715744","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,19]]}}}