{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T03:09:08Z","timestamp":1778728148624,"version":"3.51.4"},"reference-count":103,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T00:00:00Z","timestamp":1778630400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>Machine Learning (ML) software systems are frequently used in our day-to-day lives. Some of these systems are used in various sensitive environments to make life-changing decisions. Therefore, it is crucial to ensure that these AI\/ML systems do not make any discriminatory decisions for any specific groups or populations. In that vein, different bias detection and mitigation open source software libraries (aka API libraries) are being developed and used. In this article, we conduct a qualitative study to understand in what scenarios these open source fairness APIs are used in the wild, how they are used, and what challenges the developers of these APIs face while developing and adopting these libraries. We have analyzed 204 GitHub repositories (from a list of 1,885 candidate repositories) which used 13 APIs that are developed to address bias in ML software. We found that these APIs are used for two primary purposes (i.e., learning and solving real-world problems), targeting 17 unique use-cases. Our study suggests that developers are not well-versed in bias detection and mitigation; they face lots of troubleshooting issues, and frequently ask for opinions and resources. Our findings can be instrumental for future bias-related software engineering research, and for guiding educators in developing more state-of-the-art curricula.<\/jats:p>","DOI":"10.1145\/3765735","type":"journal-article","created":{"date-parts":[[2025,9,11]],"date-time":"2025-09-11T22:22:26Z","timestamp":1757629346000},"page":"1-47","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Applications and Challenges of Fairness APIs in Machine Learning Software"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1769-2704","authenticated-orcid":false,"given":"Ajoy","family":"Das","sequence":"first","affiliation":[{"name":"University of Calgary, Calgary, Alberta, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1376-095X","authenticated-orcid":false,"given":"Gias","family":"Uddin","sequence":"additional","affiliation":[{"name":"EECS, York University, Toronto, Ontario, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2301-6104","authenticated-orcid":false,"given":"Shaiful","family":"Chowdhury","sequence":"additional","affiliation":[{"name":"CS, University of Manitoba, Winnipeg, Manitoba, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4902-4858","authenticated-orcid":false,"given":"Mostafijur Rahman","family":"Akhond","sequence":"additional","affiliation":[{"name":"EECS, York University, Toronto, Ontario, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0204-9812","authenticated-orcid":false,"given":"Hadi","family":"Hemmati","sequence":"additional","affiliation":[{"name":"University of Calgary, Calgary, Alberta, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,5,13]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"AI Fairness 360 (AIF360): A Comprehensive Set of Fairness Metrics for Datasets and Machine Learning Models Explanations for These Metrics and Algorithms to Mitigate Bias in Datasets and Models. Retrieved February 13 2022 from https:\/\/github.com\/Trusted-AI\/AIF360"},{"key":"e_1_3_2_3_2","unstructured":"Amazon Scraps Secret AI Recruiting Tool That Showed Bias Against Women. Retrieved February 24 2022 from https:\/\/www.reuters.com\/article\/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G"},{"key":"e_1_3_2_4_2","unstructured":"The Bias and Fairness Audit Toolkit. Retrieved February 13 2022 from https:\/\/github.com\/dssg\/aequitas"},{"key":"e_1_3_2_5_2","unstructured":"Black Box Auditing and Certifying and Removing Disparate Impact. Retrieved February 13 2022 from https:\/\/github.com\/algofairness\/BlackBoxAuditing"},{"key":"e_1_3_2_6_2","unstructured":"A Curated List of Awesome Fairness in AI Resources. Retrieved February 13 2022 from https:\/\/github.com\/datamllab\/awesome-fairness-in-ai"},{"key":"e_1_3_2_7_2","unstructured":"A Curated List of Awesome Machine Learning Interpretability Resources. Retrieved February 13 2022 from https:\/\/github.com\/jphall663\/awesome-machine-learning-interpretability"},{"key":"e_1_3_2_8_2","unstructured":"EthicML Documentation: Mitigation Algorithms. Retrieved February 13 2022 from https:\/\/wearepal.ai\/EthicML\/ethicml.algorithms.html#module-ethicml.algorithms.preprocess.upsampler"},{"key":"e_1_3_2_9_2","unstructured":"Fairlearn: A Python Package to Assess and Improve Fairness of Machine Learning Models. Retrieved February 13 2022 from https:\/\/github.com\/fairlearn\/fairlearn"},{"key":"e_1_3_2_10_2","unstructured":"Fairlearn Documentation: Fairlearn.preprocessing Package. Retrieved February 13 2022 from https:\/\/fairlearn.org\/v0.7.0\/api_reference\/fairlearn.preprocessing.html#fairlearn.preprocessing.CorrelationRemover"},{"key":"e_1_3_2_11_2","unstructured":"Fairlens: An Open-Source Python Library for Automatically Discovering Bias and Measuring Fairness in Data. Retrieved December 6 2024 from https:\/\/github.com\/synthesized-io\/fairlens"},{"key":"e_1_3_2_12_2","unstructured":"Fairness Comparison: Comparing Fairness-Aware Machine Learning Techniques. Retrieved February 13 2022 from https:\/\/github.com\/algofairness\/fairness-comparison"},{"key":"e_1_3_2_13_2","unstructured":"Fairness Indicators: Tensorflow\u2019s Fairness Evaluation and Visualization Toolkit. Retrieved February 13 2022 from https:\/\/github.com\/tensorflow\/fairness-indicators"},{"key":"e_1_3_2_14_2","unstructured":"GerryFair: Auditing and Learning for Subgroup Fairness. Retrieved February 13 2022 from https:\/\/github.com\/algowatchpenn\/GerryFair"},{"key":"e_1_3_2_15_2","unstructured":"Stack Overflow Developer Survey 2021: Most Commonly Used Programming Language. Retrieved April 10 2022 from https:\/\/insights.stackoverflow.com\/survey\/2021\/?utm_source=social-share&utm_medium=social&utm_campaign=dev-survey-2021"},{"key":"e_1_3_2_16_2","unstructured":"J. Adebayo. 2022. FairML: Auditing Black-Box Predictive Models. Retrieved February 13 2022 from https:\/\/github.com\/adebayoj\/fairml"},{"key":"e_1_3_2_17_2","unstructured":"A. Agarwal A. Beygelzimer M. Dud\u00edk J. Langford and H. Wallach. 2018. A reductions approach to fair classification. arXiv:1803.02453. Retrieved from https:\/\/arxiv.org\/abs\/1803.02453"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338937"},{"key":"e_1_3_2_19_2","first-page":"1","volume-title":"Proceedings of the 2021 IEEE International Conference on Autonomous Systems (ICAS)","author":"Al Alamin M. A.","year":"2021","unstructured":"M. A. Al Alamin and G. Uddin. 2021. Quality assurance challenges for machine learning software applications during software development life cycle phases. In Proceedings of the 2021 IEEE International Conference on Autonomous Systems (ICAS), 1\u20135."},{"key":"e_1_3_2_20_2","first-page":"46","volume-title":"Proceedings of the 2021 IEEE\/ACM 18th International Conference on Mining Software Repositories (MSR)","author":"Alamin M. A. A.","year":"2021","unstructured":"M. A. A. Alamin, S. Malakar, G. Uddin, S. Afroz, T. B. Haider, and A. Iqbal. 2021. An empirical study of developer discussions on low-code software development challenges. In Proceedings of the 2021 IEEE\/ACM 18th International Conference on Mining Software Repositories (MSR), 46\u201357."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"issue":"234","key":"e_1_3_2_22_2","first-page":"1","article-title":"Learning certifiably optimal rule lists for categorical data","volume":"18","author":"Angelino E.","year":"2018","unstructured":"E. Angelino, N. Larus-Stone, D. Alabi, M. Seltzer, and C. Rudin. 2018. Learning certifiably optimal rule lists for categorical data. Journal of Machine Learning Research 18, 234 (Aug. 2018), 1\u201378.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_23_2","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1007\/978-3-642-13657-3_43","volume-title":"Advances in Knowledge Discovery and Data Mining","author":"Arun R.","year":"2010","unstructured":"R. Arun, V. Suresh, C. E. Veni Madhavan, and M. N. Narasimha Murthy. 2010. On finding the natural number of topics with latent Dirichlet allocation: Some observations. In Advances in Knowledge Discovery and Data Mining. M. J. Zaki, J. X. Yu, B. Ravindran, and V. Pudi (Eds.), Lecture Notes in Computer Science, Springer, Berlin, 391\u2013402."},{"issue":"12","key":"e_1_3_2_24_2","first-page":"5087","article-title":"BiasFinder: Metamorphic test generation to uncover bias for sentiment analysis systems","volume":"48","author":"Asyrofi M. H.","year":"2021","unstructured":"M. H. Asyrofi, I. N. B. Yusuf, H. J. Kang, F. Thung, Z. Yang, and D. Lo. 2021. BiasFinder: Metamorphic test generation to uncover bias for sentiment analysis systems. IEEE Transactions on Software Engineering 48, 12 (Feb. 2021), 5087\u20135101.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209581"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338939"},{"key":"e_1_3_2_27_2","unstructured":"N. Bantilan. 2022. Themis ML: A Library That Implements Fairness-Aware Machine Learning Algorithms. Retrieved February 13 2022 from https:\/\/github.com\/cosmicBboy\/themis-ml"},{"key":"e_1_3_2_28_2","unstructured":"N. Bantilan. 2022. Themis ML Source Code: Relabeling Algorithm Implementation. Retrieved February 13 2022 from https:\/\/github.com\/cosmicBboy\/themis-ml\/blob\/198012eb1745dc25f5707c773b17189ec7d523e8\/themis_ml\/preprocessing\/relabelling.py"},{"key":"e_1_3_2_29_2","unstructured":"R. K. E. Bellamy K. Dey M. Hind S. C. Hoffman S. Houde K. Kannan P. Lohia J. Martino S. Mehta A. Mojsilovic et al. 2018. AI fairness 360: An extensible toolkit for detecting understanding and mitigating unwanted algorithmic bias. arXiv:1810.01943. Retrieved from https:\/\/arxiv.org\/abs\/1810.01943"},{"key":"e_1_3_2_30_2","unstructured":"A. Beutel J. Chen Z. Zhao and E. H. Chi. 2017. Data decisions and theoretical implications when adversarially learning fair representations. arXiv:1707.00075. Retrieved from https:\/\/arxiv.org\/abs\/1707.00075"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460231.3473897"},{"key":"e_1_3_2_32_2","first-page":"642","volume-title":"Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event","author":"Biswas S.","year":"2020","unstructured":"S. Biswas and H. Rajan. 2020. Do the machine learning models on a crowd sourced platform exhibit bias? An empirical study on model fairness. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event. ACM, 642\u2013653."},{"key":"e_1_3_2_33_2","doi-asserted-by":"crossref","unstructured":"S. Biswas and H. Rajan. 2021. Fair preprocessing: Towards understanding compositional fairness of data transformers in machine learning pipeline. arXiv:2106.06054. Retrieved from https:\/\/arxiv.org\/abs\/2106.06054","DOI":"10.1145\/3468264.3468536"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.5555\/944919.944937"},{"key":"e_1_3_2_35_2","doi-asserted-by":"crossref","unstructured":"S. L. Blodgett S. Barocas H. Daum\u00e9 III and H. Wallach. 2020. Language (technology) is power: A critical survey of \u201cBias\u201d in NLP. arXiv:2005.14050. Retrieved from https:\/\/arxiv.org\/abs\/2005.14050","DOI":"10.18653\/v1\/2020.acl-main.485"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264838"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-010-0190-x"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aal4230"},{"key":"e_1_3_2_39_2","first-page":"3992","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","author":"Calmon F.","year":"2017","unstructured":"F. Calmon, D. Wei, B. Vinzamuri, K. Natesan Ramamurthy, and K. R. Varshney. 2017. Optimized pre-processing for discrimination prevention. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc., 3992\u20134001."},{"key":"e_1_3_2_40_2","doi-asserted-by":"crossref","unstructured":"A. Castelnovo R. Crupi G. Greco and D. Regoli. 2021. The zoo of fairness metrics in machine learning. arXiv:2106.00467. Retrieved from https:\/\/arxiv.org\/abs\/2106.00467","DOI":"10.21203\/rs.3.rs-1162350\/v1"},{"key":"e_1_3_2_41_2","doi-asserted-by":"crossref","unstructured":"L. E. Celis L. Huang V. Keswani and N. K. Vishnoi. 2020. Classification with fairness constraints: A meta-algorithm with provable guarantees. arXiv:1806.06055. Retrieved from https:\/\/arxiv.org\/abs\/1806.06055","DOI":"10.1145\/3287560.3287586"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468537"},{"key":"e_1_3_2_43_2","first-page":"654","volume-title":"Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event","author":"Chakraborty J.","year":"2020","unstructured":"J. Chakraborty, S. Majumder, Z. Yu, and T. Menzies. 2020. Fairway: A way to build fair ML software. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event. ACM, 654\u2013665."},{"key":"e_1_3_2_44_2","doi-asserted-by":"crossref","unstructured":"J. Chakraborty Q. H. Tu S. Majumder and T. Menzies. 2022. Fair-SSL: Building fair ML Software with less data. In Proceedings of the 2nd International Workshop on Equitable Data and Technology. ACM 1\u20138.","DOI":"10.1145\/3524491.3527305"},{"key":"e_1_3_2_45_2","unstructured":"J. Chen. 2018. Fair lending needs explainable models for responsible recommendation. arXiv:1809.04684. Retrieved from https:\/\/arxiv.org\/abs\/1809.04684"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1089\/big.2016.0047"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-7439(99)00047-7"},{"key":"e_1_3_2_48_2","unstructured":"C. Dwork M. Hardt T. Pitassi O. Reingold and R. Zemel. 2011. Fairness through awareness. arXiv:1104.3913. Retrieved from https:\/\/arxiv.org\/abs1104.3913"},{"key":"e_1_3_2_49_2","doi-asserted-by":"crossref","unstructured":"M. Feldman S. Friedler J. Moeller C. Scheidegger and S. Venkatasubramanian. 2015. Certifying and removing disparate impact. arXiv:1412.3756. Retrieved from https:\/\/arxiv.org\/abs\/1412.3756","DOI":"10.1145\/2783258.2783311"},{"key":"e_1_3_2_50_2","unstructured":"J. Foulds R. Islam K. N. Keya and S. Pan. 2019. An intersectional definition of fairness. arXiv:1807.08362. Retrieved from https:\/\/arxiv.org\/abs\/1807.08362"},{"key":"e_1_3_2_51_2","unstructured":"P. Gajane and M. Pechenizkiy. 2018. On formalizing fairness in prediction with machine learning. arXiv:1710.03184. Retrieved from https:\/\/arxiv.org\/abs\/1710.03184"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106277"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.3390\/app11073184"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387473"},{"key":"e_1_3_2_55_2","first-page":"3315","article-title":"Equality of opportunity in supervised learning","volume":"29","author":"Hardt M.","year":"2016","unstructured":"M. Hardt, E. Price, and N. Srebro. 2016. Equality of opportunity in supervised learning. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 29, 3315\u20133323.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372831"},{"key":"e_1_3_2_57_2","unstructured":"T. B. Hashimoto M. Srivastava H. Namkoong and P. Liang. 2018. Fairness without demographics in repeated loss minimization. arXiv:1806.08010. Retrieved from https:\/\/arxiv.org\/abs\/1806.08010"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468565"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1111\/lnc3.12432"},{"key":"e_1_3_2_60_2","first-page":"700","volume-title":"Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE)","author":"Imtiaz N.","year":"2019","unstructured":"N. Imtiaz, J. Middleton, J. Chakraborty, N. Robson, G. Bai, and E. Murphy-Hill. 2019. Investigating the effects of gender bias on GitHub. In Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE), 700\u2013711."},{"key":"e_1_3_2_61_2","first-page":"3770","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Jin X.","year":"2021","unstructured":"X. Jin, F. Barbieri, B. Kennedy, A. Mostafazadeh Davani, L. Neves, and X. Ren. 2021. On transferability of bias mitigation effects in language model fine-tuning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 3770\u20133783."},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-011-0463-8"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2012.45"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33486-3_3"},{"key":"e_1_3_2_65_2","unstructured":"M. Kearns S. Neel A. Roth and Z. S. Wu. 2018. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. arXiv:1711.05144. Retrieved from https:\/\/arxiv.org\/abs\/1711.05144"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2931051"},{"key":"e_1_3_2_67_2","first-page":"4066","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","author":"Kusner M. J.","year":"2017","unstructured":"M. J. Kusner, J. Loftus, C. Russell, and R. Silva. 2017. Counterfactual fairness. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc., 4066\u20134076."},{"key":"e_1_3_2_68_2","unstructured":"N. Lauga. 2022. TransparentAI: A Python Library That Aims to Understand How to Check if an Artificial Intelligence Is Ethical. Retrieved February 13 2022 from https:\/\/github.com\/Nathanlauga\/transparentai"},{"key":"e_1_3_2_69_2","volume-title":"Proceedings of the 4th International Conference on Learning Representations","author":"Louizos C.","year":"2017","unstructured":"C. Louizos, K. Swersky, Y. Li, M. Welling, and R. Zemel. 2017. The variational fair autoencoder. In Proceedings of the 4th International Conference on Learning Representations."},{"key":"e_1_3_2_70_2","first-page":"237","volume-title":"Proceedings of the 2016 IEEE\/ACM 38th International Conference on Software Engineering (ICSE)","author":"Manotas I.","year":"2016","unstructured":"I. Manotas, C. Bird, R. Zhang, D. Shepherd, C. Jaspan, C. Sadowski, L. Pollock, and J. Clause. 2016. An empirical study of practitioners\u2019 perspectives on green software engineering. In Proceedings of the 2016 IEEE\/ACM 38th International Conference on Software Engineering (ICSE), 237\u2013248."},{"key":"e_1_3_2_71_2","unstructured":"Mattu Jeff Larson. 2021. Machine Bias. Retrieved December 14 2021 from https:\/\/www.propublica.org\/article\/machine-bias-risk-assessments-in-criminal-sentencing"},{"key":"e_1_3_2_72_2","unstructured":"A. K. McCallum. 2022. MALLET: A Machine Learning for Language Toolkit. Retrieved February 16 2022 from http:\/\/mallet.cs.umass.edu"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/3457607"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10515-021-00313-x"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2015.83"},{"key":"e_1_3_2_76_2","first-page":"26","volume-title":"Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER)","author":"Paul R.","year":"2019","unstructured":"R. Paul, A. Bosu, and K. Z. Sultana. 2019. Expressions of sentiments during code reviews: Male vs. Female. In Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 26\u201337."},{"key":"e_1_3_2_77_2","unstructured":"K. Peng J. Chakraborty and T. Menzies. 2022. xFAIR: Better fairness via model-based rebalancing of protected attributes. arXiv:2110.01109. Retrieved from https:\/\/arxiv.org\/abs\/2110.01109"},{"key":"e_1_3_2_78_2","unstructured":"D. Pessach and E. Shmueli. 2020. Algorithmic fairness. arXiv:2001.09784. Retrieved from https:\/\/arxiv.org\/abs\/2001.09784"},{"key":"e_1_3_2_79_2","first-page":"5680","article-title":"On fairness and calibration","volume":"30","author":"Pleiss G.","year":"2017","unstructured":"G. Pleiss, M. Raghavan, F. Wu, J. Kleinberg, and K. Q. Weinberger. 2017. On fairness and calibration. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 30, 5680\u20135689.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_80_2","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1145\/2491411.2491418","volume-title":"Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC\/FSE \u201913)","author":"Rahman F.","year":"2013","unstructured":"F. Rahman, D. Posnett, I. Herraiz, and P. Devanbu. 2013. Sample size vs. bias in defect prediction. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC\/FSE \u201913). ACM, New York, NY, 147\u2013157."},{"key":"e_1_3_2_81_2","first-page":"45","volume-title":"Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks","author":"Rehurek R.","year":"2010","unstructured":"R. Rehurek and P. Sojka. 2010. Software framework for topic modelling with large corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, 45\u201350."},{"key":"e_1_3_2_82_2","doi-asserted-by":"crossref","first-page":"237802312096717","DOI":"10.1177\/2378023120967171","article-title":"Diagnosing gender bias in image recognition systems","volume":"6","author":"Schwemmer C.","year":"2020","unstructured":"C. Schwemmer, C. Knight, E. D. Bello-Pardo, S. Oklobdzija, M. Schoonvelde, and J. W. Lockhart. 2020. Diagnosing gender bias in image recognition systems. Socius: Sociological Research for a Dynamic World 6 (Jan. 2020), 2378023120967171.","journal-title":"Socius: Sociological Research for a Dynamic World"},{"key":"e_1_3_2_83_2","first-page":"271","volume-title":"Proceedings of the 2021 IEEE\/ACM 18th International Conference on Mining Software Repositories (MSR)","author":"Scoccia G. L.","year":"2021","unstructured":"G. L. Scoccia, P. Migliarini, and M. Autili. 2021. Challenges in developing desktop web apps: A study of stack overflow and GitHub. In Proceedings of the 2021 IEEE\/ACM 18th International Conference on Mining Software Repositories (MSR), 271\u2013282."},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2322358"},{"issue":"11","key":"e_1_3_2_85_2","doi-asserted-by":"crossref","first-page":"1129","DOI":"10.1109\/TSE.2017.2731308","article-title":"Authors\u2019 reply to \u201cComments on \u2018Researcher bias: The use of machine learning in software defect prediction\u2019\u201d","volume":"44","author":"Shepperd M.","year":"2018","unstructured":"M. Shepperd, T. Hall, and D. Bowes. 2018. Authors\u2019 reply to \u201cComments on \u2018Researcher bias: The use of machine learning in software defect prediction\u2019\u201d. IEEE Transactions on Software Engineering 44, 11 (Nov. 2018), 1129\u20131131.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.simpa.2022.100406"},{"key":"e_1_3_2_87_2","first-page":"2239","volume-title":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Speicher T.","year":"2018","unstructured":"T. Speicher, H. Heidari, N. Grgic-Hlaca, K. P. Gummadi, A. Singla, A. Weller, and M. B. Zafar. 2018. A unified approach to quantifying algorithmic unfairness: Measuring individual & group unfairness via inequality indices. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2239\u20132248."},{"key":"e_1_3_2_88_2","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1145\/3183519.3183547","volume-title":"Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP \u201918)","author":"Tantithamthavorn C.","year":"2018","unstructured":"C. Tantithamthavorn and A. E. Hassan. 2018. An experience report on defect modelling in practice: Pitfalls and challenges. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP \u201918). ACM, New York, NY, 286\u2013295."},{"issue":"11","key":"e_1_3_2_89_2","doi-asserted-by":"crossref","first-page":"1092","DOI":"10.1109\/TSE.2016.2553030","article-title":"Comments on \u201cResearcher bias: The use of machine learning in software defect prediction\u201d","volume":"42","author":"Tantithamthavorn C.","year":"2016","unstructured":"C. Tantithamthavorn, S. McIntosh, A. E. Hassan, and K. Matsumoto. 2016. Comments on \u201cResearcher bias: The use of machine learning in software defect prediction\u201d. IEEE Transactions on Software Engineering 42, 11 (Nov. 2016), 1092\u20131094.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_90_2","unstructured":"O. Thomas T. Kehrenberg M. Bartlett and N. Quadrianto. 2022. EthicML: A Featureful Framework for Developing Fair Algorithms. Retrieved February 13 2022 from https:\/\/github.com\/predictive-analytics-lab\/EthicML"},{"key":"e_1_3_2_91_2","first-page":"159","volume-title":"Proceedings of the 2017 32nd IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Uddin G.","year":"2017","unstructured":"G. Uddin and F. Khomh. 2017. Automatic summarization of API reviews. In Proceedings of the 2017 32nd IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 159\u2013170."},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-10021-5"},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.1145\/3194770.3194776"},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-021-00028-w"},{"key":"e_1_3_2_95_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2921343"},{"key":"e_1_3_2_96_2","article-title":"Card sorting","author":"William H.","year":"2013","unstructured":"H. William. 2013. Card sorting. In The Encyclopedia of Human-Computer Interaction. Retrieved October 7, 2025 from https:\/\/www.interaction-design.org\/literature\/book\/the-encyclopedia-of-human-computer-interaction-2nd-ed\/card-sorting","journal-title":"The Encyclopedia of Human-Computer Interaction"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3473117"},{"key":"e_1_3_2_98_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2021.106664"},{"key":"e_1_3_2_99_2","unstructured":"S. Yao and B. Huang. 2017. Beyond parity: Fairness objectives for collaborative filtering. arXiv:1705.08804. Retrieved from https:\/\/arxiv.org\/abs\/1705.08804"},{"key":"e_1_3_2_100_2","unstructured":"Z. Yu J. Chakraborty and T. Menzies. 2021. FairBalance: Improving machine learning fairness on multiple sensitive attributes with data balancing. arXiv:2107.08310. Retrieved from https:\/\/arxiv.org\/abs\/2107.08310"},{"key":"e_1_3_2_101_2","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052660"},{"key":"e_1_3_2_102_2","first-page":"325","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Zemel R.","year":"2013","unstructured":"R. Zemel, Y. L. Wu, K. Swersky, and T. Pitassi. 2013. Learning fair representations. In Proceedings of the International Conference on Machine Learning, 325\u2013333."},{"key":"e_1_3_2_103_2","doi-asserted-by":"publisher","DOI":"10.1145\/3278721.3278779"},{"key":"e_1_3_2_104_2","first-page":"644","volume-title":"Proceedings of the 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME)","author":"Zhou Y.","year":"2021","unstructured":"Y. Zhou, J. Harshit, S. Jieke, A. Muhammad Hilmi, and D. Lo. 2021. BiasHeal: On-the-fly black-box healing of bias in sentiment analysis systems. In Proceedings of the 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME), 644\u2013648."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3765735","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T02:34:02Z","timestamp":1778726042000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3765735"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,13]]},"references-count":103,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3765735"],"URL":"https:\/\/doi.org\/10.1145\/3765735","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,13]]},"assertion":[{"value":"2022-04-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-08-20","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-05-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}