{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T11:01:45Z","timestamp":1785495705379,"version":"3.56.0"},"publisher-location":"New York, NY, USA","reference-count":56,"publisher":"ACM","license":[{"start":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T00:00:00Z","timestamp":1776038400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,4,13]]},"DOI":"10.1145\/3793302.3793341","type":"proceedings-article","created":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T10:42:54Z","timestamp":1785494574000},"page":"86-97","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7338-2044","authenticated-orcid":false,"given":"Ali","family":"Al-Kaswan","sequence":"first","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-5932-7022","authenticated-orcid":false,"given":"Claudio","family":"Spiess","sequence":"additional","affiliation":[{"name":"University of California, Davis, Davis, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0818-2430","authenticated-orcid":false,"given":"Prem","family":"Devanbu","sequence":"additional","affiliation":[{"name":"University of California, Davis, Davis, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4850-3312","authenticated-orcid":false,"given":"Arie","family":"van Deursen","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5093-5523","authenticated-orcid":false,"given":"Maliheh","family":"Izadi","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,7,31]]},"reference":[{"key":"e_1_3_3_2_2_2","doi-asserted-by":"crossref","unstructured":"Jean-Fran\u00e7ois Abramatic Roberto Di\u00a0Cosmo and Stefano Zacchiroli. 2018. Building the universal archive of source code. Commun. ACM 61 10 (2018) 29\u201331.","DOI":"10.1145\/3183558"},{"key":"e_1_3_3_2_3_2","doi-asserted-by":"crossref","unstructured":"Ali Al-Kaswan Sebastian Deatc Beg\u00fcm Ko\u00e7 Arie van Deursen and Maliheh Izadi. 2025. Code Red! On the Harmfulness of Applying Off-the-shelf Large Language Models to Programming Tasks. Proceedings of the ACM on Software Engineering 2 FSE (2025) 2477\u20132499.","DOI":"10.1145\/3729380"},{"key":"e_1_3_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/NLBSE59153.2023.00008"},{"key":"e_1_3_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639133"},{"key":"e_1_3_3_2_6_2","unstructured":"Elie Bakouch Loubna Ben\u00a0Allal Anton Lozhkov Nouamane Tazi Lewis Tunstall Carlos\u00a0Miguel Pati\u00f1o Edward Beeching Aymeric Roucher Aksel\u00a0Joonas Reedi Quentin Gallou\u00e9dec Kashif Rasul Nathan Habib Cl\u00e9mentine Fourrier Hynek Kydlicek Guilherme Penedo Hugo Larcher Mathieu Morlon Vaibhav Srivastav Joshua Lochner Xuan-Son Nguyen Colin Raffel Leandro von Werra and Thomas Wolf. 2025. SmolLM3: smol multilingual long-context reasoner. https:\/\/huggingface.co\/blog\/smollm3."},{"key":"e_1_3_3_2_7_2","doi-asserted-by":"publisher","unstructured":"Maciej Besta Julia Barth Eric Schreiber Ales Kubicek Afonso Catarino Robert Gerstenberger Piotr Nyczyk Patrick Iff Yueling Li Sam Houliston Tomasz Sternal Marcin Copik Grzegorz Kwa\u015bniewski J\u00fcrgen M\u00fcller \u0141ukasz Flis Hannes Eberhard Hubert Niewiadomski and Torsten Hoefler. 2025. Reasoning Language Models: A Blueprint. 10.48550\/arXiv.2501.11223 arxiv:https:\/\/arXiv.org\/abs\/2501.11223\u00a0[cs]","DOI":"10.48550\/arXiv.2501.11223"},{"key":"e_1_3_3_2_8_2","doi-asserted-by":"crossref","unstructured":"Burton\u00a0H Bloom. 1970. Space\/time trade-offs in hash coding with allowable errors. Commun. ACM 13 7 (1970) 422\u2013426.","DOI":"10.1145\/362686.362692"},{"key":"e_1_3_3_2_9_2","first-page":"2633","volume-title":"30th USENIX Security Symposium (USENIX Security 21)","author":"Carlini Nicholas","year":"2021","unstructured":"Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et\u00a0al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21). 2633\u20132650."},{"key":"e_1_3_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3726302.3730178"},{"key":"e_1_3_3_2_11_2","unstructured":"Jan Dubi\u0144ski Antoni Kowalczuk Stanis\u0142aw Pawlak Przemys\u0142aw Rokita Tomasz Trzci\u0144ski and Pawe\u0142 Morawiecki. 2023. Towards More Realistic Membership Inference Attacks on Large Diffusion Models. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2306.12983 (2023)."},{"key":"e_1_3_3_2_12_2","doi-asserted-by":"publisher","unstructured":"Sarah Fakhoury Aaditya Naik Georgios Sakkas Saikat Chakraborty and Shuvendu\u00a0K. Lahiri. 2024. LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation. IEEE Transactions on Software Engineering 50 9 (2024) 2254\u20132268. 10.1109\/TSE.2024.3428972","DOI":"10.1109\/TSE.2024.3428972"},{"key":"e_1_3_3_2_13_2","unstructured":"Alex Gu Baptiste Rozi\u00e8re Hugh Leather Armando Solar-Lezama Gabriel Synnaeve and Sida\u00a0I Wang. 2024. Cruxeval: A benchmark for code reasoning understanding and execution. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2401.03065 (2024)."},{"key":"e_1_3_3_2_14_2","unstructured":"Valentin Hartmann Anshuman Suri Vincent Bindschaedler David Evans Shruti Tople and Robert West. 2023. SoK: Memorization in General-Purpose Large Language Models. arxiv:https:\/\/arXiv.org\/abs\/2310.18362\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2310.18362"},{"key":"e_1_3_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.naacl-long.469"},{"key":"e_1_3_3_2_16_2","unstructured":"Xinyi Hou Yanjie Zhao Yue Liu Zhou Yang Kailong Wang Li Li Xiapu Luo David Lo John Grundy and Haoyu Wang. 2023. Large language models for software engineering: A systematic literature review. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2308.10620 (2023)."},{"key":"e_1_3_3_2_17_2","doi-asserted-by":"crossref","unstructured":"Hongsheng Hu Zoran Salcic Lichao Sun Gillian Dobbie Philip\u00a0S Yu and Xuyun Zhang. 2022. Membership inference attacks on machine learning: A survey. ACM Computing Surveys (CSUR) 54 11s (2022) 1\u201337.","DOI":"10.1145\/3523273"},{"key":"e_1_3_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510172"},{"key":"e_1_3_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639138"},{"key":"e_1_3_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/LLM4Code66737.2025.00018"},{"key":"e_1_3_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR59073.2023.00082"},{"key":"e_1_3_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/2610384.2628055"},{"key":"e_1_3_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR52588.2021.00066"},{"key":"e_1_3_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387491"},{"key":"e_1_3_3_2_25_2","unstructured":"Anjan Karmakar Julian\u00a0Aron Prenner Marco D\u2019Ambros and Romain Robbes. 2022. Codex hacks hackerrank: Memorization issues and a framework for code synthesis evaluation. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2212.02684 (2022)."},{"key":"e_1_3_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3650105.3652298"},{"key":"e_1_3_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/Forge66646.2025.00025"},{"key":"e_1_3_3_2_28_2","doi-asserted-by":"crossref","unstructured":"Jiaolong Kong Xiaofei Xie and Shangqing Liu. 2025. Demystifying Memorization in LLM-Based Program Repair via a General Hypothesis Testing Framework. Proceedings of the ACM on Software Engineering 2 FSE (2025) 2712\u20132734.","DOI":"10.1145\/3729390"},{"key":"e_1_3_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3135932.3135941"},{"key":"e_1_3_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.52202\/075280-0943"},{"key":"e_1_3_3_2_31_2","unstructured":"Anton Lozhkov Raymond Li Loubna\u00a0Ben Allal Federico Cassano Joel Lamy-Poirier Nouamane Tazi Ao Tang Dmytro Pykhtar Jiawei Liu Yuxiang Wei Tianyang Liu Max Tian Denis Kocetkov Arthur Zucker Younes Belkada Zijian Wang Qian Liu Dmitry Abulkhanov Indraneil Paul Zhuang Li Wen-Ding Li Megan Risdal Jia Li Jian Zhu Terry\u00a0Yue Zhuo Evgenii Zheltonozhskii Nii Osae\u00a0Osae Dade Wenhao Yu Lucas Krau\u00df Naman Jain Yixuan Su Xuanli He Manan Dey Edoardo Abati Yekun Chai Niklas Muennighoff Xiangru Tang Muhtasham Oblokulov Christopher Akiki Marc Marone Chenghao Mou Mayank Mishra Alex Gu Binyuan Hui Tri Dao Armel Zebaze Olivier Dehaene Nicolas Patry Canwen Xu Julian McAuley Han Hu Torsten Scholak Sebastien Paquet Jennifer Robinson Carolyn\u00a0Jane Anderson Nicolas Chapados Mostofa Patwary Nima Tajbakhsh Yacine Jernite Carlos\u00a0Mu\u00f1oz Ferrandis Lingming Zhang Sean Hughes Thomas Wolf Arjun Guha Leandro von Werra and Harm de Vries. 2024. StarCoder 2 and The Stack v2: The Next Generation. arxiv:https:\/\/arXiv.org\/abs\/2402.19173\u00a0[cs.SE] https:\/\/arxiv.org\/abs\/2402.19173"},{"key":"e_1_3_3_2_32_2","unstructured":"Xingjun Ma Yifeng Gao Yixu Wang Ruofan Wang Xin Wang Ye Sun Yifan Ding Hengyuan Xu Yunhao Chen Yunhan Zhao et\u00a0al. 2025. Safety at Scale: A Comprehensive Survey of Large Model Safety. CoRR (2025)."},{"key":"e_1_3_3_2_33_2","doi-asserted-by":"crossref","unstructured":"Marc Marone and Benjamin Van\u00a0Durme. 2023. Data portraits: Recording foundation model training data. Advances in Neural Information Processing Systems 36 (2023) 15121\u201315135.","DOI":"10.52202\/075280-0665"},{"key":"e_1_3_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/SaTML64287.2025.00028"},{"key":"e_1_3_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.570"},{"key":"e_1_3_3_2_36_2","volume-title":"First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022","author":"Mireshghallah Fatemehsadat","year":"2022","unstructured":"Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David Evans, and Taylor Berg-Kirkpatrick. 2022. Memorization in NLP Fine-tuning Methods. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022."},{"key":"e_1_3_3_2_37_2","unstructured":"Ahmad Mohsin Helge Janicke Adrian Wood Iqbal\u00a0H Sarker Leandros Maglaras and Naeem Janjua. 2024. Can we trust large language models generated code? a framework for in-context learning security patterns and code evaluations across diverse llms. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2406.12513 (2024)."},{"key":"e_1_3_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639187"},{"key":"e_1_3_3_2_39_2","unstructured":"Nikita Pavlichenko Iurii Nazarov Ivan Dolgov Ekaterina Garanina Karol Lasocki Julia Reshetnikova Sergei Boitsov Ivan Bondyrev Dariia Karaeva Maksim Sheptyakov Dmitry Ustalov Artem Mukhin Semyon Proshev Nikita Abramov Olga Kolomyttseva Kseniia Lysaniuk Ilia Zavidnyi Anton Semenkin Vladislav Tankov and Uladzislau Sazanovich. 2025. Mellum-4b-base."},{"key":"e_1_3_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524459.3527351"},{"key":"e_1_3_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510216"},{"key":"e_1_3_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524842.3528505"},{"key":"e_1_3_3_2_43_2","doi-asserted-by":"publisher","unstructured":"Ahmed Salem Yang Zhang Mathias Humbert Pascal Berrang Mario Fritz and Michael Backes. 2018. ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models. 10.48550\/arXiv.1806.01246arXiv:https:\/\/arXiv.org\/abs\/1806.01246 [cs].","DOI":"10.48550\/arXiv.1806.01246"},{"key":"e_1_3_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR66628.2025.00080"},{"key":"e_1_3_3_2_45_2","doi-asserted-by":"crossref","unstructured":"Avi Schwarzschild Zhili Feng Pratyush Maini Zachary Lipton and J\u00a0Zico Kolter. 2024. Rethinking llm memorization through the lens of adversarial compression. Advances in Neural Information Processing Systems 37 (2024) 56244\u201356267.","DOI":"10.52202\/079017-1790"},{"key":"e_1_3_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2017.41"},{"key":"e_1_3_3_2_47_2","doi-asserted-by":"publisher","unstructured":"Ola Shorinwa Zhiting Mei Justin Lidard Allen\u00a0Z. Ren and Anirudha Majumdar. 2025. A Survey on Uncertainty Quantification of Large Language Models: Taxonomy Open Research Challenges and Future Directions. ACM Comput. Surv. (June 2025). 10.1145\/3744238","DOI":"10.1145\/3744238"},{"key":"e_1_3_3_2_48_2","unstructured":"Claudio Spiess David Gros Kunal\u00a0Suresh Pai Michael Pradel Md\u00a0Rafiqul\u00a0Islam Rabin Amin Alipour Susmit Jha Prem Devanbu and Toufique Ahmed. 2024. Calibration and correctness of language models for code. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2402.02047 (2024)."},{"key":"e_1_3_3_2_49_2","doi-asserted-by":"crossref","unstructured":"Florian Tambon Arghavan Moradi-Dakhel Amin Nikanjam Foutse Khomh Michel\u00a0C Desmarais and Giuliano Antoniol. 2025. Bugs in large language models generated code: An empirical study. Empirical Software Engineering 30 3 (2025) 65.","DOI":"10.1007\/s10664-025-10614-4"},{"key":"e_1_3_3_2_50_2","unstructured":"Yao Wan Guanghua Wan Shijie Zhang Hongyu Zhang Yulei Sui Pan Zhou Hai Jin and Lichao Sun. 2024. Does Your Neural Code Completion Model Use My Code? A Membership Inference Approach. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2404.14296 (2024)."},{"key":"e_1_3_3_2_51_2","unstructured":"Kun Wang Guibin Zhang Zhenhong Zhou Jiahao Wu Miao Yu Shiqian Zhao Chenlong Yin Jinhu Fu Yibo Yan Hanjun Luo et\u00a0al. 2025. A comprehensive survey in llm (-agent) full stack safety: Data training and deployment. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2504.15585 (2025)."},{"key":"e_1_3_3_2_52_2","unstructured":"Chunqiu\u00a0Steven Xia Yinlin Deng Soren Dunn and Lingming Zhang. 2024. Agentless: Demystifying LLM-based Software Engineering Agents. CoRR (Jan. 2024). https:\/\/openreview.net\/forum?id=WYK8E00dm2"},{"key":"e_1_3_3_2_53_2","unstructured":"Jinwei Xu He Zhang Yanjing Yang Lanxin Yang Zeru Cheng Jun Lyu Bohan Liu Xin Zhou Alberto Bacchelli Yin\u00a0Kia Chiam et\u00a0al. 2024. One Size Does Not Fit All: Investigating Efficacy of Perplexity in Detecting LLM-Generated Code. ACM Transactions on Software Engineering and Methodology (2024)."},{"key":"e_1_3_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639074"},{"key":"e_1_3_3_2_55_2","unstructured":"Shenglai Zeng Yaxin Li Jie Ren Yiding Liu Han Xu Pengfei He Yue Xing Shuaiqiang Wang Jiliang Tang and Dawei Yin. 2023. Exploring memorization in fine-tuned language models. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2310.06714 (2023)."},{"key":"e_1_3_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i19.30185"},{"key":"e_1_3_3_2_57_2","doi-asserted-by":"crossref","unstructured":"Albert Ziegler Eirini Kalliamvakou X\u00a0Alice Li Andrew Rice Devon Rifkin Shawn Simister Ganesh Sittampalam and Edward Aftandilian. 2024. Measuring github copilot\u2019s impact on productivity. Commun. ACM 67 3 (2024) 54\u201363.","DOI":"10.1145\/3633453"}],"event":{"name":"MSR '26: 23rd International Conference on Mining Software Repositories","location":"Rio de Janeiro Brazil","acronym":"MSR '26","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 23rd International Conference on Mining Software Repositories"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3793302.3793341","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T10:49:44Z","timestamp":1785494984000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3793302.3793341"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,13]]},"references-count":56,"alternative-id":["10.1145\/3793302.3793341","10.1145\/3793302"],"URL":"https:\/\/doi.org\/10.1145\/3793302.3793341","relation":{},"subject":[],"published":{"date-parts":[[2026,4,13]]},"assertion":[{"value":"2026-07-31","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}