{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,18]],"date-time":"2026-05-18T10:13:47Z","timestamp":1779099227259,"version":"3.51.4"},"reference-count":130,"publisher":"Association for Computing Machinery (ACM)","issue":"1","funder":[{"DOI":"10.13039\/501100012166","name":"National Key R&D Program of China","doi-asserted-by":"crossref","award":["2022ZD0160201"],"award-info":[{"award-number":["2022ZD0160201"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"crossref"}]},{"name":"HK RGC RIF","award":["R7030-22"],"award-info":[{"award-number":["R7030-22"]}]},{"name":"Huawei Flagship Research"},{"name":"HK RGC GRF","award":["Ref: 17208223 & 17204424"],"award-info":[{"award-number":["Ref: 17208223 & 17204424"]}]},{"name":"HKU-CAS Joint Laboratory for Intelligent System Software"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,1,31]]},"abstract":"<jats:p>As the adoption of LLMs becomes more widespread in software coding ecosystems, a pressing issue has emerged: does the generated code contain social bias and unfairness, such as those related to age, gender, and race? This issue concerns the integrity, fairness, and ethical foundation of software applications that depend on the code generated by these models but are underexplored in the literature. This article presents a novel bias testing framework that is specifically designed for code generation tasks. Based on this framework, we conduct an extensive empirical study on the biases in code generated by five widely studied LLMs (i.e., PALM-2-CodeChat-bison, Claude-instant-1, GPT-3.5-turbo, GPT-4-turbo, and GPT-4). Our findings reveal that biases are prevalent. For example, 13.47% to 49.10% of the codes generated by these LLMs have biased behaviors towards gender. Moreover, we study five bias mitigation prompt strategies that are commonly used in current code generation scenarios, i.e., zero-shot, one-shot, few-shot, and two Chain-of-Thought (CoT) prompts, with and without provided feedback-driven refinement. Our evaluation results illustrate that using direct prompt engineering strategies has limited effectiveness in mitigating bias, but our test execution feedback can help to reduce the ratio of code biases to a large extent (e.g., from 59.88% to 4.79% for GPT-4).<\/jats:p>","DOI":"10.1145\/3724117","type":"journal-article","created":{"date-parts":[[2025,3,18]],"date-time":"2025-03-18T14:47:51Z","timestamp":1742309271000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":23,"title":["Bias Testing and Mitigation in LLM-based Code Generation"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4275-3006","authenticated-orcid":false,"given":"Dong","family":"Huang","sequence":"first","affiliation":[{"name":"The University of Hong Kong, Hong Kong, Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0481-7264","authenticated-orcid":false,"given":"Jie","family":"M. Zhang","sequence":"additional","affiliation":[{"name":"King\u2019s College London, London, United Kingdom of Great Britain and Northern Ireland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-0270-3004","authenticated-orcid":false,"given":"Qingwen","family":"Bu","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1288-6502","authenticated-orcid":false,"given":"Xiaofei","family":"Xie","sequence":"additional","affiliation":[{"name":"School of Computing and Information Systems, Singapore Management University, Singapore, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3056-9962","authenticated-orcid":false,"given":"Junjie","family":"Chen","sequence":"additional","affiliation":[{"name":"College of Intelligence and Computing, Tianjin University, Tianjin, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7746-440X","authenticated-orcid":false,"given":"Heming","family":"Cui","sequence":"additional","affiliation":[{"name":"The University of Hong Kong, Hong Kong, Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,12,11]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"M. Chen J. Tworek H. Jun Q. Yuan H. P. de Oliveira Pinto J. Kaplan H. Edwards Y. Burda N. Joseph G. Brockman et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https:\/\/arxiv.org\/abs\/2107.03374"},{"key":"e_1_3_2_3_2","unstructured":"OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. Retrieved from https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_2_4_2","unstructured":"D. Huang G. Zeng J. Dai M. Luo H. Weng Y. Qing H. Cui Z. Guo and J. M. Zhang. 2024. Effi-Code: Unleashing code efficiency in language models. arXiv:2410.10209. Retrieved from https:\/\/arxiv.org\/abs\/2410.10209"},{"key":"e_1_3_2_5_2","unstructured":"D. Huang J. Dai H. Weng P. Wu Y. Qing H. Cui Z. Guo and J. M. Zhang. 2024. EffiLearner: Enhancing efficiency of generated code via self-optimization. arXiv:2405.15189. Retrieved from https:\/\/arxiv.org\/abs\/2405.15189"},{"key":"e_1_3_2_6_2","unstructured":"D. Huang J. M. Zhang M. Du M. Harman and H. Cui. 2024. Rethinking the influence of source code on test case generation. arXiv:2409.09464. Retrieved from https:\/\/arxiv.org\/abs\/2409.09464"},{"key":"e_1_3_2_7_2","unstructured":"J. Dai J. Lu Y. Feng D. Huang G. Zeng R. Ruan M. Cheng H. Tan and Z. Guo. 2024. MHPP: Exploring the capabilities and limitations of language models beyond basic code generation. arXiv:2405.11430. Retrieved from https:\/\/arxiv.org\/abs\/2405.11430"},{"key":"e_1_3_2_8_2","first-page":"2368","article-title":"Uncovering and quantifying social biases in code generation","author":"Liu Y.","year":"2023","unstructured":"Y. Liu, X. Chen, Y. Gao, Z. Su, F. Zhang, D. Zan, J.-G. Lou, P.-Y. Chen, and T.-Y. Ho. 2023. Uncovering and quantifying social biases in code generation. In Advances in Neural Information Processing Systems 36, 2368\u20132380.","journal-title":"Advances in Neural Information Processing Systems 36"},{"issue":"1","key":"e_1_3_2_9_2","first-page":"1","article-title":"Role of information technology in human resource management of SME: A study on the use of applicant tracking system","volume":"3","author":"Mukherjee A. N.","year":"2014","unstructured":"A. N. Mukherjee, S. Bhattacharyya, and R. Bera. 2014. Role of information technology in human resource management of SME: A study on the use of applicant tracking system. IBMRD\u2019s Journal of Management & Research 3, 1 (2014) 1\u201322.","journal-title":"IBMRD\u2019s Journal of Management & Research"},{"key":"e_1_3_2_10_2","first-page":"452","volume-title":"Proceedings of the 2009 2nd International Conference on the Applications of Digital Information and Web Technologies","author":"Ahmad N.","year":"2009","unstructured":"N. Ahmad and A. N. Abd Alla. 2009. Smart evaluation for job vacancy application system. In Proceedings of the 2009 2nd International Conference on the Applications of Digital Information and Web Technologies. IEEE, 452\u2013455."},{"key":"e_1_3_2_11_2","doi-asserted-by":"crossref","unstructured":"H. Thakur A. Jain P. Vaddamanu P. P. Liang and L.-P. Morency. 2023. Language models get a gender makeover: Mitigating gender bias with few-shot data interventions. arXiv:2306.04597. Retrieved from https:\/\/arxiv.org\/abs\/2306.04597","DOI":"10.18653\/v1\/2023.acl-short.30"},{"key":"e_1_3_2_12_2","doi-asserted-by":"crossref","unstructured":"E. L. Ungless A. Rafferty H. Nag and B. Ross. 2022. A robust bias mitigation procedure based on the stereotype content model. arXiv:2210.14552. Retrieved from https:\/\/arxiv.org\/abs\/2210.14552","DOI":"10.18653\/v1\/2022.nlpcss-1.23"},{"key":"e_1_3_2_13_2","doi-asserted-by":"crossref","unstructured":"H. Lee S. Hong J. Park T. Kim G. Kim and J.-W. Ha. 2023. KoSBI: A dataset for mitigating social bias risks towards safer large language model application. arXiv:2305.17701. Retrieved from https:\/\/arxiv.org\/abs\/2305.17701","DOI":"10.18653\/v1\/2023.acl-industry.21"},{"key":"e_1_3_2_14_2","doi-asserted-by":"crossref","unstructured":"S. Barikeri A. Lauscher I. Vuli\u0107 and G. Glava\u0161. 2021. RedditBias: A real-world resource for bias evaluation and debiasing of conversational language models. arXiv:2106.03521. Retrieved from https:\/\/arxiv.org\/abs\/2106.03521","DOI":"10.18653\/v1\/2021.acl-long.151"},{"key":"e_1_3_2_15_2","doi-asserted-by":"crossref","unstructured":"V. K. Felkner H.-C. H. Chang E. Jang and J. May. 2023. WinoQueer: A community-in-the-loop benchmark for anti-LGBTQ+ bias in large language models. arXiv:2306.15087. Retrieved from https:\/\/arxiv.org\/abs\/2306.15087","DOI":"10.18653\/v1\/2023.acl-long.507"},{"key":"e_1_3_2_16_2","unstructured":"E. Fleisig and C. Fellbaum. 2022. Mitigating gender bias in machine translation through adversarial learning. arXiv:2203.10675. Retrieved from https:\/\/arxiv.org\/abs\/2203.10675"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00134"},{"key":"e_1_3_2_18_2","volume-title":"Proceedings of the 45th International Conference on Software Engineering (ICSE \u201923)","author":"Usman Gohar S. B.","year":"2023","unstructured":"S. B. Usman Gohar, and H. Rajan. 2023. Towards understanding fairness and its composition in ensemble machine learning. In Proceedings of the 45th International Conference on Software Engineering (ICSE \u201923)."},{"key":"e_1_3_2_19_2","volume-title":"Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201920)","author":"Biswas S.","year":"2020","unstructured":"S. Biswas and H. Rajan. 2020. Do the machine learning models on a crowd sourced platform exhibit bias? An empirical study on model fairness. In Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201920)."},{"issue":"140","key":"e_1_3_2_20_2","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel C.","year":"2020","unstructured":"C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21, 140 (2020), 1\u201367. Retrieved from http:\/\/jmlr.org\/papers\/v21\/20-074.html","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_21_2","doi-asserted-by":"crossref","unstructured":"Y. Wang H. Le A. D. Gotmare N. D. Bui J. Li and S. C. Hoi. 2023. CodeT5+: Open code large language models for code understanding and generation. arXiv:2305.07922. Retrieved from https:\/\/arxiv.org\/abs\/2305.07922","DOI":"10.18653\/v1\/2023.emnlp-main.68"},{"key":"e_1_3_2_22_2","unstructured":"S. Ouyang J. M. Zhang M. Harman and M. Wang. 2023. LLM is like a box of chocolates: The non-determinism of ChatGPT in code generation. arXiv:2308.02828. Retrieved from https:\/\/arxiv.org\/abs\/2308.02828"},{"key":"e_1_3_2_23_2","doi-asserted-by":"crossref","unstructured":"S. Wang Z. Li H. Qian C. Yang Z. Wang M. Shang V. Kumar S. Tan B. Ray P. Bhatia et al. 2022. ReCode: Robustness evaluation of code generation models. arXiv:2212.10264. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:254877229","DOI":"10.18653\/v1\/2023.acl-long.773"},{"key":"e_1_3_2_24_2","first-page":"23","article-title":"Flamingo: A visual language model for few-shot learning","author":"Alayrac J.-B.","year":"2022","unstructured":"J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. 2022. Flamingo: A visual language model for few-shot learning. In Advances in Neural Information Processing Systems 35, 23716\u201323736.","journal-title":"Advances in Neural Information Processing Systems 35"},{"key":"e_1_3_2_25_2","unstructured":"G. Izacard P. Lewis M. Lomeli L. Hosseini F. Petroni T. Schick J. Dwivedi-Yu A. Joulin S. Riedel and E. Grave. 2022. Few-shot learning with retrieval augmented language models. arXiv:2208.03299. Retrieved from https:\/\/arxiv.org\/abs\/2208.03299"},{"key":"e_1_3_2_26_2","unstructured":"L. Tunstall N. Reimers U. E. S. Jo L. Bates D. Korat M. Wasserblat and O. Pereg. 2022. Efficient few-shot learning without prompts. arXiv:2209.11055. Retrieved from https:\/\/arxiv.org\/abs\/2209.11055"},{"key":"e_1_3_2_27_2","first-page":"82424","article-title":"Chain-of-thought prompting elicits reasoning in large language models","author":"Wei J.","year":"2022","unstructured":"J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35, 82424\u201383724.","journal-title":"Advances in Neural Information Processing Systems 35"},{"key":"e_1_3_2_28_2","unstructured":"A. Madaan and A. Yazdanbakhsh. 2022. Text and patterns: For effective chain of thought it takes two to tango. arXiv:2209.07686. Retrieved from https:\/\/arxiv.org\/abs\/2209.07686"},{"key":"e_1_3_2_29_2","unstructured":"X. Wang J. Wei D. Schuurmans Q. Le E. Chi S. Narang A. Chowdhery and D. Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv:2203.11171. Retrieved from https:\/\/arxiv.org\/abs\/2203.11171"},{"key":"e_1_3_2_30_2","unstructured":"Z. Chu J. Chen Q. Chen W. Yu T. He H. Wang W. Peng M. Liu B. Qin and T. Liu. 2023. A survey of chain of thought reasoning: Advances frontiers and future. arXiv:2309.15402. Retrieved from https:\/\/arxiv.org\/abs\/2309.15402"},{"key":"e_1_3_2_31_2","unstructured":"D. Huang Q. Bu and H. Cui. 2023. CodeCot and beyond: Learning to program and test like a developer. arXiv:2308.08784. Retrieved from https:\/\/arxiv.org\/abs\/2308.08784"},{"key":"e_1_3_2_32_2","unstructured":"D. Huang Q. Bu J. M. Zhang M. Luck and H. Cui. 2023. AgentCoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv:2312.13010. Retrieved from https:\/\/arxiv.org\/abs\/2312.13010"},{"key":"e_1_3_2_33_2","unstructured":"Y. Li D. H. Choi J. Chung N. Kushman J. Schrittwieser R. Leblond T. Eccles J. Keeling F. Gimeno A. D. Lago et al. 2022. Competition-level code generation with AlphaCode. arXiv:2203.07814. Retrieved from https:\/\/arxiv.org\/abs\/2203.07814"},{"key":"e_1_3_2_34_2","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR)","author":"Nijkamp E.","year":"2023","unstructured":"E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. 2023. CodeGen: An open large language model for code with multi-turn program synthesis. In Proceedings of the International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_2_35_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923)","author":"Fried D.","year":"2023","unstructured":"D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, S. Yih, L. Zettlemoyer, and M. Lewis. 2023. InCoder: A generative model for code infilling and synthesis. In Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923). OpenReview.net. Retrieved from https:\/\/openreview.net\/pdf?id=hQwb-lbM6EL"},{"key":"e_1_3_2_36_2","unstructured":"R. Li L. B. Allal Y. Zi N. Muennighoff D. Kocetkov C. Mou M. Marone C. Akiki J. Li J. Chim et al. 2023. StarCoder: May the source be with you! arXiv:2305.06161. Retrieved from https:\/\/arxiv.org\/abs\/2305.06161"},{"key":"e_1_3_2_37_2","unstructured":"L. B. Allal R. Li D. Kocetkov C. Mou C. Akiki C. M. Ferrandis N. Muennighoff M. Mishra A. Gu M. Dey et al. 2023. SantaCoder: Don\u2019t reach for the stars! arXiv:2301.03988. Retrieved from https:\/\/arxiv.org\/2301.03988"},{"key":"e_1_3_2_38_2","unstructured":"DeepSeekAI. 2023. DeepSeek Coder: Let the Code Write Itself. Retrieved from https:\/\/deepseekcoder.github.io\/"},{"key":"e_1_3_2_39_2","unstructured":"B. Rozi\u00e8re J. Gehring F. Gloeckle S. Sootla I. Gat X. Tan Y. Adi J. Liu T. Remez J. Rapin et al. 2023. Code Llama: Open foundation models for code. arXiv:2308.12950. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:261100919"},{"key":"e_1_3_2_40_2","volume-title":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 (NeurIPS \u201920), Virtual","author":"Brown T. B.","year":"2020","unstructured":"T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 (NeurIPS \u201920), Virtual. H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.). Retrieved from https:\/\/proceedings.neurips.cc\/paper\/2020\/hash\/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html"},{"key":"e_1_3_2_41_2","unstructured":"H. Touvron L. Martin K. Stone P. Albert A. Almahairi Y. Babaei N. Bashlykov S. Batra P. Bhargava S. Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288. Retrieved from https:\/\/arxiv.org\/abs\/2307.09288"},{"key":"e_1_3_2_42_2","unstructured":"OpenAI. 2023. GPT-3.5 Turbo. Retrieved from https:\/\/platform.openai.com\/docs\/models\/gpt-3-5"},{"key":"e_1_3_2_43_2","unstructured":"Z. Luo C. Xu P. Zhao Q. Sun X. Geng W. Hu C. Tao J. Ma Q. Lin and D. Jiang. 2023. WizardCoder: Empowering code large language models with Evol-Instruct. arXiv:2306.08568. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:259164815"},{"key":"e_1_3_2_44_2","unstructured":"M. I. Abdin S. A. Jacobs A. A. Awan J. Aneja A. Awadallah H. Awadalla N. Bach A. Bahree A. Bakhtiari H. S. Behl et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv:2404.14219. Retrieved from https:\/\/arxiv.org\/abs\/2404.14219"},{"key":"e_1_3_2_45_2","unstructured":"M. M. A. Haque W. U. Ahmad I. Lourentzou and C. Brown. 2022. FixEval: Execution-based evaluation of program fixes for competitive programming problems. arXiv:2206.07796. Retrieved from https:\/\/arxiv.org\/abs\/2206.07796"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00125"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00085"},{"key":"e_1_3_2_48_2","doi-asserted-by":"crossref","unstructured":"Y. Deng C. S. Xia C. Yang S. D. Zhang S. Yang and L. Zhang. 2023. Large language models are edge-case fuzzers: Testing deep learning libraries via FuzzGPT. arXiv:2304.02014. Retrieved from https:\/\/arxiv.org\/abs\/2304.02014","DOI":"10.1145\/3597926.3598067"},{"key":"e_1_3_2_49_2","volume-title":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 (NeurIPS \u201920), Virtual","author":"Rozi\u00e8re B.","year":"2020","unstructured":"B. Rozi\u00e8re, M. Lachaux, L. Chanussot, and G. Lample. 2020. Unsupervised translation of programming languages. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 (NeurIPS \u201920), Virtual. H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.). Retrieved from https:\/\/proceedings.neurips.cc\/paper\/2020\/hash\/ed23fbf18c2cd35f8c7f8de44f85c08d-Abstract.html"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-acl.143"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510124"},{"key":"e_1_3_2_52_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923)","author":"Wei J.","year":"2023","unstructured":"J. Wei, G. Durrett, and I. Dillig. 2023. TypeT5: Seq2seq type inference using static analysis. In Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923). OpenReview.net. Retrieved from https:\/\/openreview.net\/pdf?id=4TyNEhI2GdN"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-acl.18"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3559555"},{"key":"e_1_3_2_55_2","unstructured":"J. Austin A. Odena M. Nye M. Bosma H. Michalewski D. Dohan E. Jiang C. J. Cai M. Terry Q. V. Le et al. 2021. Program synthesis with large language models. arXiv:2108.07732. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:237142385"},{"key":"e_1_3_2_56_2","unstructured":"J. Liu C. S. Xia Y. Wang and L. Zhang. 2023. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In Proceedings of the 37th Conference on Neural Information Processing Systems. Retrieved from https:\/\/openreview.net\/forum?id=1qvx610Cu7"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.773"},{"key":"e_1_3_2_58_2","doi-asserted-by":"crossref","unstructured":"Q. Zheng X. Xia X. Zou Y. Dong S. Wang Y. Xue Z. Wang L. Shen A. Wang Y. Li et al. 2023. CodeGeeX: A pre-trained model for code generation with multilingual evaluations on HumanEval-X. arXiv:2303.17568. Retrieved from https:\/\/arxiv.org\/abs\/2303.17568","DOI":"10.1145\/3580305.3599790"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3267446"},{"key":"e_1_3_2_60_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923)","author":"Athiwaratkun B.","year":"2023","unstructured":"B. Athiwaratkun, S. K. Gouda, Z. Wang, X. Li, Y. Tian, M. Tan, W. U. Ahmad, S. Wang, Q. Sun, M. Shang, et al. 2023. Multi-lingual evaluation of code generation models. In Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923). OpenReview.net. Retrieved from https:\/\/openreview.net\/pdf?id=Bo7eeXm6An8"},{"key":"e_1_3_2_61_2","first-page":"18319","volume-title":"Proceedings of the International Conference on Machine Learning (ICML \u201923)","author":"Lai Y.","year":"2023","unstructured":"Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W. Yih, D. Fried, S. I. Wang, and T. Yu. 2023. DS-1000: A natural and reliable benchmark for data science code generation. In Proceedings of the International Conference on Machine Learning (ICML \u201923). A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, PMLR, 18319\u201318345. Retrieved from https:\/\/proceedings.mlr.press\/v202\/lai23b.html"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.9"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2022\/329"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510203"},{"key":"e_1_3_2_65_2","unstructured":"S. G. Patil T. Zhang X. Wang and J. E. Gonzalez. 2023. Gorilla: Large language model connected with massive APIs. arXiv:2305.15334. Retrieved from https:\/\/arxiv.org\/abs\/2305.15334"},{"key":"e_1_3_2_66_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923)","author":"Nijkamp E.","year":"2023","unstructured":"E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. 2023. CodeGen: An open large language model for code with multi-turn program synthesis. In Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923). OpenReview.net. Retrieved from https:\/\/openreview.net\/pdf?id=iaYcJKpY2B_"},{"key":"e_1_3_2_67_2","unstructured":"T. Liu C. Xu and J. J. McAuley. 2023. RepoBench: Benchmarking repository-level code auto-completion systems. arXiv:2306.03091. Retrieved from https:\/\/arxiv.org\/abs\/2306.03091"},{"key":"e_1_3_2_68_2","unstructured":"C. E. Jimenez J. Yang A. Wettig S. Yao K. Pei O. Press and K. Narasimhan. 2023. SWE-bench: Can language models resolve real-world GitHub issues? arXiv:2310.06770. Retrieved from https:\/\/arxiv.org\/abs\/2310.06770"},{"key":"e_1_3_2_69_2","first-page":"31","volume-title":"Proceedings of the International Conference on Machine Learning (ICML \u201923)","author":"Shrivastava D.","year":"2023","unstructured":"D. Shrivastava, H. Larochelle, and D. Tarlow. 2023. Repository-level prompt generation for large language models of code. In Proceedings of the International Conference on Machine Learning (ICML \u201923). A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, PMLR, 31693\u201331715. Retrieved from https:\/\/proceedings.mlr.press\/v202\/shrivastava23a.html"},{"key":"e_1_3_2_70_2","doi-asserted-by":"crossref","first-page":"2471","DOI":"10.18653\/v1\/2023.emnlp-main.151","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP \u201923)","author":"Zhang F.","year":"2023","unstructured":"F. Zhang, B. Chen, Y. Zhang, J. Keung, J. Liu, D. Zan, Y. Mao, J. Lou, and W. Chen. 2023. 2023. RepoCoder: Repository-level code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP \u201923). H. Bouamor, J. Pino, and K. Bali (Eds.), Association for Computational Linguistics, 2471\u20132484. Retrieved from https:\/\/aclanthology.org\/2023.emnlp-main.151"},{"key":"e_1_3_2_71_2","unstructured":"Y. Ding Z. Wang W. U. Ahmad M. K. Ramanathan R. Nallapati P. Bhatia D. Roth and B. Xiang. 2022. CoCoMIC: Code completion by jointly modeling in-file and cross-file context. arXiv:2212.10007. Retrieved from https:\/\/arxiv.org\/abs\/2212.10007"},{"key":"e_1_3_2_72_2","volume-title":"Proceedings of the 46th International Conference on Software Engineering (ICSE \u201924)","author":"OBrien D.","year":"2024","unstructured":"D. OBrien, S. Biswas, S. Imtiaz, R. Abdalkareem, E. Shihab, and H. Rajan. 2024. Are prompt engineering and TODO comments friends or foes? An evaluation on GitHub copilot. In Proceedings of the 46th International Conference on Software Engineering (ICSE \u201924)."},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/3583561"},{"key":"e_1_3_2_74_2","first-page":"6478","article-title":"Retiring adult: New datasets for fair machine learning","author":"Ding F.","year":"2021","unstructured":"F. Ding, M. Hardt, J. Miller, and L. Schmidt. 2021. Retiring adult: New datasets for fair machine learning. In Advances in Neural Information Processing Systems 34, 6478\u20136490.","journal-title":"Advances in Neural Information Processing Systems 34"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1002\/widm.1452"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287589"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.2021.1952897"},{"key":"e_1_3_2_78_2","unstructured":"J. Kang T. Xie X. Wu R. Maciejewski and H. Tong. 2021. InfoFair: Information-theoretic intersectional fairness. arXiv: 2105.11069. Retrieved from https:\/\/arxiv.org\/abs\/2105.11069"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/3457607"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287592"},{"key":"e_1_3_2_81_2","unstructured":"J. Komiyama and H. Shimao. 2017. Two-stage algorithm for fairness-aware machine learning. arXiv:1710.04924. Retrieved from https:\/\/arxiv.org\/abs\/1710.04924"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510361"},{"key":"e_1_3_2_83_2","article-title":"Federated fairness without access to demographics","author":"Papadaki A.","year":"2022","unstructured":"A. Papadaki, N. Martinez, M. A. Bertran, G. Sapiro, and M. R. Rodrigues. 2022. Federated fairness without access to demographics. In Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022)","journal-title":"Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022)"},{"key":"e_1_3_2_84_2","unstructured":"X. Han Z. Jiang H. Jin Z. Liu N. Zou Q. Wang and X. Hu. 2023. Retiring \u0394DP: New distribution-level metrics for demographic parity. arXiv: 2301.13443. Retrieved from https:\/\/arxiv.org\/abs\/2301.13443"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533081"},{"key":"e_1_3_2_86_2","unstructured":"C. Mougan L. State A. Ferrara S. Ruggieri and S. Staab. 2023. Beyond demographic parity: Redefining equal treatment. arXiv: 2303.08040. Retrieved from https:\/\/arxiv.org\/abs\/2303.08040"},{"key":"e_1_3_2_87_2","unstructured":"A. S. de Oliveira C. Kaplan K. Mallat and T. Chakraborty. 2023. An empirical analysis of fairness notions under differential privacy. arXiv:2302.02910. Retrieved from https:\/\/arxiv.org\/abs\/2302.02910"},{"key":"e_1_3_2_88_2","volume-title":"Addressing Interpretability Fairness & Privacy in Machine Learning through Combinatorial Optimization Methods","author":"Ferry J.","year":"2023","unstructured":"J. Ferry 2023. Addressing Interpretability Fairness & Privacy in Machine Learning through Combinatorial Optimization Methods. Ph.D. dissertation, Universit\u00e9 Paul Sabatier-Toulouse III."},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533101"},{"key":"e_1_3_2_90_2","first-page":"35894","article-title":"Fair infinitesimal jackknife: Mitigating the influence of biased training data points without refitting","author":"Sattigeri P.","year":"2022","unstructured":"P. Sattigeri, S. Ghosh, I. Padhi, P. Dognin, and K. R. Varshney. 2022. Fair infinitesimal jackknife: Mitigating the influence of biased training data points without refitting. In Advances in Neural Information Processing Systems 35, 35894\u201335906.","journal-title":"Advances in Neural Information Processing Systems 35"},{"key":"e_1_3_2_91_2","first-page":"9939","article-title":"Subgroup robustness grows on trees: An empirical baseline investigation","author":"Gardner J.","year":"2022","unstructured":"J. Gardner, Z. Popovic, and L. Schmidt. 2022. Subgroup robustness grows on trees: An empirical baseline investigation. In Advances in Neural Information Processing Systems 35, 9939\u20139954.","journal-title":"Advances in Neural Information Processing Systems 35"},{"key":"e_1_3_2_92_2","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/SaTML54575.2023.00012","volume-title":"Proceedings of the 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML)","author":"Ferry J.","year":"2023","unstructured":"J. Ferry, U. A\u00efvodji, S. Gambs, M.-J. Huguet, and M. Siala. 2023. Exploiting fairness to enhance sensitive attributes reconstruction. In Proceedings of the 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 18\u201341."},{"key":"e_1_3_2_93_2","unstructured":"A. F. Cruz and M. Hardt. 2023. Unprocessing seven years of algorithmic fairness. arXiv:2306.07261. Retrieved from https:\/\/arxiv.org\/abs\/2306.07261"},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3594008"},{"key":"e_1_3_2_95_2","unstructured":"A. F. Cruz C. Bel\u00e9m S. Jesus J. Bravo P. Saleiro and P. Bizarro. 2022. FairGBM: Gradient boosting with fairness constraints. arXiv:2209.07850. Retrieved from https:\/\/arxiv.org\/abs\/2209.07850"},{"key":"e_1_3_2_96_2","article-title":"Estimating and controlling for equalized odds via sensitive attribute predictors","author":"Bharti B.","year":"2024","unstructured":"B. Bharti, P. Yi, and J. Sulam. 2024. Estimating and controlling for equalized odds via sensitive attribute predictors. In Advances in Neural Information Processing Systems 36.","journal-title":"Advances in Neural Information Processing Systems 36"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.3233\/FAIA230104"},{"key":"e_1_3_2_98_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616257"},{"issue":"1","key":"e_1_3_2_99_2","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1057\/palgrave.fsm.4770138","article-title":"Impact of anti-discrimination laws on credit scoring","volume":"9","author":"Andreeva G.","year":"2004","unstructured":"G. Andreeva, J. Ansell, and J. Crook. 2004. Impact of anti-discrimination laws on credit scoring. Journal of Financial Services Marketing 9, 1 (2004), 22\u201333.","journal-title":"Journal of Financial Services Marketing"},{"key":"e_1_3_2_100_2","unstructured":"A. Chouldechova and A. Roth. 2018. The frontiers of fairness in machine learning. arXiv:1810.08810. Retrieved from https:\/\/arxiv.org\/abs\/1810.08810"},{"key":"e_1_3_2_101_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510202"},{"key":"e_1_3_2_102_2","doi-asserted-by":"crossref","first-page":"292","DOI":"10.1109\/EuroSP51992.2021.00028","volume-title":"Proceedings of the 2021 IEEE European Symposium on Security and Privacy (EuroS&P)","author":"Chang H.","year":"2021","unstructured":"H. Chang and R. Shokri. 2021. On the privacy risks of algorithmic fairness. In Proceedings of the 2021 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 292\u2013303."},{"key":"e_1_3_2_103_2","unstructured":"S. Corbett-Davies and S. Goel. 2018. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv:1808.00023. Retrieved from https:\/\/arxiv.org\/abs\/1808.00023"},{"key":"e_1_3_2_104_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639083"},{"key":"e_1_3_2_105_2","unstructured":"L. Salewski S. Alaniz I. Rio-Torto E. Schulz and Z. Akata. 2023. In-context impersonation reveals large language models\u2019 strengths and biases. arXiv:2305.14930. Retrieved from https:\/\/arxiv.org\/abs\/2305.14930"},{"key":"e_1_3_2_106_2","unstructured":"P. Wang L. Li L. Chen D. Zhu B. Lin Y. Cao Q. Liu T. Liu and Z. Sui. 2023. Large language models are not fair evaluators. arXiv:2305.17926. Retrieved from https:\/\/arxiv.org\/abs\/2305.17926"},{"key":"e_1_3_2_107_2","unstructured":"Y. Yu Y. Zhuang J. Zhang Y. Meng A. Ratner R. Krishna J. Shen and C. Zhang. 2023. Large language model as attributed training data generator: A tale of diversity and bias. arXiv:2306.15895. Retrieved from https:\/\/arxiv.org\/abs\/2306.15895"},{"issue":"4","key":"e_1_3_2_108_2","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1037\/apl0000363","article-title":"Bargaining while black: The role of race in salary negotiations","volume":"104","author":"Hernandez M.","year":"2019","unstructured":"M. Hernandez, D. R. Avery, S. D. Volpone, and C. R. Kaiser. 2019. Bargaining while black: The role of race in salary negotiations. The Journal of Applied Psychology 104, 4 (2019), 581\u2013592.","journal-title":"The Journal of Applied Psychology"},{"issue":"1","key":"e_1_3_2_109_2","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1007\/s12122-022-09331-4","article-title":"Gender stereotypes in job advertisements: What do they imply for the gender salary gap","volume":"43","author":"Arceo-Gomez E. O.","year":"2022","unstructured":"E. O. Arceo-Gomez, R. M. Campos-Vazquez, R. Y. Badillo, and S. Lopez-Araiza. 2022. Gender stereotypes in job advertisements: What do they imply for the gender salary gap? Journal of Labor Research 43, 1 (2022), 65\u2013102.","journal-title":"Journal of Labor Research"},{"issue":"1","key":"e_1_3_2_110_2","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1177\/0091026019845119","article-title":"How to do a salary equity study: With an illustrative example from higher education","volume":"49","author":"Taylor L. L.","year":"2020","unstructured":"L. L. Taylor, J. N. Lahey, M. I. Beck, and J. E. Froyd. 2020. How to do a salary equity study: With an illustrative example from higher education. Public Personnel Management 49, 1 (2020), 57\u201382.","journal-title":"Public Personnel Management"},{"key":"e_1_3_2_111_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.worlddev.2021.105498"},{"key":"e_1_3_2_112_2","unstructured":"Kaggle. 2023. Adult Income Dataset. Retrieved August 1 2023 from www.kaggle.com\/datasets\/wenruliu\/adult-income-dataset"},{"key":"e_1_3_2_113_2","unstructured":"Kaggle. 2023. Employee Dataset. Retrieved August 1 2023 from www.kaggle.com\/datasets\/tawfikelmetwally\/employee-dataset"},{"key":"e_1_3_2_114_2","unstructured":"Kaggle. 2023. US Health Insurance Dataset. Retrieved August 1 2023 from www.kaggle.com\/datasets\/teertha\/ushealthinsurancedataset"},{"key":"e_1_3_2_115_2","doi-asserted-by":"crossref","unstructured":"N. Mehrabi F. Morstatter N. A. Saxena K. Lerman and A. G. Galstyan. 2019. A survey on bias and fairness in machine learning. ACM Computing Surveys 54 (2019) 1\u201335. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:201666566","DOI":"10.1145\/3457607"},{"key":"e_1_3_2_116_2","volume-title":"Proceedings of the Annual Meeting of the Association for Computational Linguistics","author":"Nadeem M.","year":"2020","unstructured":"M. Nadeem, A. Bethke, and S. Reddy. 2020. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics."},{"key":"e_1_3_2_117_2","volume-title":"Proceedings of the 46th International Conference on Software Engineering (ICSE \u201924)","author":"Chen Z.","year":"2023","unstructured":"Z. Chen, J. Zhang, F. Sarro, and M. Harman. 2023. Fairness improvement with multiple protected attributes: How far are we? In Proceedings of the 46th International Conference on Software Engineering (ICSE \u201924). ACM."},{"key":"e_1_3_2_118_2","first-page":"2803","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Dutta S.","year":"2020","unstructured":"S. Dutta, D. Wei, H. Yueksel, P.-Y. Chen, S. Liu, and K. Varshney. 2020. Is there a trade-off between fairness and accuracy? A perspective using mismatched hypothesis testing. In Proceedings of the International Conference on Machine Learning. PMLR, 2803\u20132813."},{"issue":"3","key":"e_1_3_2_119_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3432931","article-title":"To \u201csee\u201d is to stereotype: Image tagging algorithms, gender recognition, and the accuracy-fairness trade-off","volume":"4","author":"Barlas P.","year":"2021","unstructured":"P. Barlas, K. Kyriakou, O. Guest, S. Kleanthous, and J. Otterbacher. 2021. To \u201csee\u201d is to stereotype: Image tagging algorithms, gender recognition, and the accuracy-fairness trade-off. Proceedings of the ACM on Human-Computer Interaction 4, CSCW3 (2021), 1\u201331.","journal-title":"Proceedings of the ACM on Human-Computer Interaction"},{"key":"e_1_3_2_120_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549093"},{"key":"e_1_3_2_121_2","doi-asserted-by":"publisher","DOI":"10.1145\/3461702.3462519"},{"key":"e_1_3_2_122_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10287-022-00425-z"},{"key":"e_1_3_2_123_2","unstructured":"D. Guo S. Ren S. Lu Z. Feng D. Tang S. Liu L. Zhou N. Duan J. Yin D. Jiang et al. 2020. GraphCodeBERT: Pre-training code representations with data flow. arXiv:2009.08366. Retrieved from https:\/\/arxiv.org\/abs\/2009.08366"},{"key":"e_1_3_2_124_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_3_2_125_2","doi-asserted-by":"crossref","unstructured":"W. U. Ahmad S. Chakraborty B. Ray and K.-W. Chang. 2021. Unified pre-training for program understanding and generation. arXiv:2103.06333. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:232185260","DOI":"10.18653\/v1\/2021.naacl-main.211"},{"key":"e_1_3_2_126_2","first-page":"311","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni K.","year":"2002","unstructured":"K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 311\u2013318. Available. Retrieved from https:\/\/aclanthology.org\/P02-1040"},{"key":"e_1_3_2_127_2","first-page":"74","volume-title":"Text Summarization Branches Out","author":"Lin C.-Y.","year":"2004","unstructured":"C.-Y. Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out. Association for Computational Linguistics, 74\u201381. Retrieved from https:\/\/aclanthology.org\/W04-1013"},{"key":"e_1_3_2_128_2","unstructured":"S. Ren D. Guo S. Lu L. Zhou S. Liu D. Tang M. Zhou A. Blanco and S. Ma. 2020. CodeBLEU: A method for automatic evaluation of code synthesis. arXiv: 2009.10297. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:221836101"},{"key":"e_1_3_2_129_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111741"},{"key":"e_1_3_2_130_2","unstructured":"D. Huang Y. Qing W. Shang H. Cui and J. M. Zhang. 2024. EffiBench: Benchmarking the efficiency of automatically generated code. arXiv:2402.02037. Retrieved from https:\/\/arxiv.org\/abs\/2402.02037"},{"key":"e_1_3_2_131_2","doi-asserted-by":"publisher","DOI":"10.1145\/3663529.3664462"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3724117","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,11]],"date-time":"2025-12-11T15:56:47Z","timestamp":1765468607000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3724117"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,11]]},"references-count":130,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,1,31]]}},"alternative-id":["10.1145\/3724117"],"URL":"https:\/\/doi.org\/10.1145\/3724117","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,11]]},"assertion":[{"value":"2024-07-07","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-25","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-12-11","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}