{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T19:43:19Z","timestamp":1782848599378,"version":"3.54.5"},"reference-count":64,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T00:00:00Z","timestamp":1782777600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>Large Language Model (LLM) alignment trains an LLM using preference data to produce outputs that better meet established quality standards (e.g., to mitigate toxic or biased responses in text generation, or to enforce coding best practices in code generation). While LLM alignment techniques are studied for non-coding tasks, we know little about their usefulness for coding tasks. Intuitively, for code generation tasks using LLMs, alignment techniques could help with coding solutions that better adhere to established software engineering standards, such as more secure code, supporting better coding practices, etc. However, it is unclear whether LLM code alignment could support both functional requirements (producing executable, correct code) and non-functional requirements (code readability, style, maintainability). It is also unknown whether alignment for a code LLM should begin with base pretrained version or the finetuned (i.e., instruction-tuned) version of the LLM. In this paper, we offer insights on the above two research questions by conducting an empirical study. We studied five state-of-the-art (SOTA) LLMs using two widely used LLM alignment techniques: Direct Preference Optimization (DPO) and BoNBoN. For each training record, we created a preference pair as accepted and rejected instances by using the SelfCodeAlign pipeline. DPO and BoNBoN are reward-free models, i.e., they eliminate the need for multiple reward scores for output preferences. We tuned each LLM using the two alignment techniques in two settings: pretrained and finetuned versions of an LLM. We evaluated functional requirements using four SOTA benchmarks (HumanEval+, MBPP+, EvalPerf, EvoEval) and non-functional requirements using the CODAL benchmark, which evaluates code quality across five dimensions derived from software engineering practices. We find that pretrained-to-aligned pathways achieve larger improvements in the aligned variant over its pretrained variant (CodeLlama-7b: +75% non-functional, Llama3-8b: +42% functional). But the pretrained variant is generally less accurate than its finetuned variant. However, finetuned-to-aligned offers smaller performance improvements or, in some cases, degradation in the aligned variant than its finetuned variant. This means that while the base pretrained version is less accurate than its base finetuned variant, alignment reduces the performance gap between pretrained and finetuned variants. Non-functional requirements improve more consistently than functional requirements via alignment. Based on these findings, we provide nine recommendations to guide alignment for code LLMs.<\/jats:p>","DOI":"10.1145\/3808123","type":"journal-article","created":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:06:14Z","timestamp":1782839174000},"page":"2606-2629","source":"Crossref","is-referenced-by-count":0,"title":["Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generation"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7225-2432","authenticated-orcid":false,"given":"Sanjeepan","family":"Sivapiran","sequence":"first","affiliation":[{"name":"York University, Toronto, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1376-095X","authenticated-orcid":false,"given":"Gias","family":"Uddin","sequence":"additional","affiliation":[{"name":"York University, Toronto, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,30]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2504.04030"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2503.15248"},{"key":"e_1_2_1_3_1","volume-title":"The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card 1, 1","author":"Anthropic AI","year":"2024","unstructured":"AI Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card 1, 1 (2024), 4."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021). doi:10.48550\/arXiv.2108.07732 10.48550\/arXiv.2108.07732","DOI":"10.48550\/arXiv.2108.07732"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2107.03374"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0026256"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","unstructured":"Gheorghe Comanici Eric Bieber Mike Schaekermann Ice Pasupat Noveen Sachdeva Inderjit Dhillon Marcel Blistein Ori Ram Dan Zhang Evan Rosen et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning multimodality long context and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025). doi:10.48550\/arXiv.2507. 06261 10.48550\/arXiv.2507.06261","DOI":"10.48550\/arXiv.2507"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2502.14560"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2308.01861"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2404.04475"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.01306"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2504.21191"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177705148"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","unstructured":"Aaron Grattafiori Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten Alex Vaughan et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024). doi:10.48550\/arXiv.2407.21783 10.48550\/arXiv.2407.21783","DOI":"10.48550\/arXiv.2407.21783"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.52202\/079017-0094"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","unstructured":"Daya Guo Dejian Yang Haowei Zhang Junxiao Song Ruoyu Zhang Runxin Xu Qihao Zhu Shirong Ma Peiyi Wang Xiao Bi et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025). doi:10.48550\/arXiv.2501.12948 10.48550\/arXiv.2501.12948","DOI":"10.48550\/arXiv.2501.12948"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","unstructured":"Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Yu Wu YK Li et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming-The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196 (2024). doi:10.48550\/arXiv.2401.14196 10.48550\/arXiv.2401.14196","DOI":"10.48550\/arXiv.2401.14196"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2403.07691"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","unstructured":"Binyuan Hui Jian Yang Zeyu Cui Jiaxi Yang Dayiheng Liu Lei Zhang Tianyu Liu Jiajun Zhang Bowen Yu Keming Lu et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186 (2024). doi:10.48550\/arXiv.2409.12186 10.48550\/arXiv.2409.12186","DOI":"10.48550\/arXiv.2409.12186"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","unstructured":"Juyong Jiang Fan Wang Jiasi Shen Sungju Kim and Sunghun Kim. 2024. A Survey on Large Language Models for Code Generation. arXiv:2406.00515 [cs.CL] doi:10.48550\/arXiv.2406.00515 10.48550\/arXiv.2406.00515","DOI":"10.48550\/arXiv.2406.00515"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2310.06770"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2410.10739"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1611835114"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2211.15533"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2309.10105"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","unstructured":"Harrison Lee Samrat Phatale Hassan Mansoor Thomas Mesnard Johan Ferret Kellie Lu Colton Bishop Ethan Hall Victor Carbune Abhinav Rastogi et al. 2023. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267 (2023). doi:10.48550\/arXiv.2309.00267 10.48550\/arXiv.2309.00267","DOI":"10.48550\/arXiv.2309.00267"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.01210"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","unstructured":"Yun Luo Zhen Yang Fandong Meng Yafu Li Jie Zhou and Yue Zhang. 2025. An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning. arXiv:2308.08747 [cs.CL] doi:10.48550\/arXiv.2308. 08747 10.48550\/arXiv.2308.08747","DOI":"10.48550\/arXiv.2308"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2306.08568"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2403.01131"},{"key":"e_1_2_1_33_1","volume-title":"Interrater reliability: the kappa statistic. Biochemia medica 22, 3","author":"McHugh Mary L","year":"2012","unstructured":"Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica 22, 3 (2012), 276-282."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2410.18585"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2404.02204"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2308.07124"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","unstructured":"OpenAI Josh Achiam et al. 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] doi:10.48550\/arXiv.2303.08774 10.48550\/arXiv.2303.08774","DOI":"10.48550\/arXiv.2303.08774"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","unstructured":"Long Ouyang Jeffrey Wu Xu Jiang Diogo Almeida Carroll Wainwright Pamela Mishkin Chong Zhang Sandhini Agarwal Katarina Slama Alex Ray et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35 (2022) 27730-27744. doi:10.48550\/arXiv.2203.02155 10.48550\/arXiv.2203.02155","DOI":"10.48550\/arXiv.2203.02155"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2404.19733"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.18290"},{"key":"e_1_2_1_41_1","volume-title":"International conference on learning representations.","author":"Ramasesh Vinay Venkatesh","year":"2021","unstructured":"Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. 2021. Effect of scale on catastrophic forgetting in neural networks. In International conference on learning representations."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","unstructured":"Weijieying Ren Xinlong Li Lei Wang Tianxiang Zhao and Wei Qin. 2024. Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning. arXiv:2402.18865 [cs.LG] doi:10.48550\/arXiv.2402.18865 10.48550\/arXiv.2402.18865","DOI":"10.48550\/arXiv.2402.18865"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2308.12950"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1707.06347"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2401.15963"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2503.19206"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2009.01325"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2407.05040"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2507.07548"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.07922"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2407.16216"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","unstructured":"Jason Wei Yi Tay Rishi Bommasani Colin Raffel Barret Zoph Sebastian Borgeaud Dani Yogatama Maarten Bosma Denny Zhou Donald Metzler et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022). doi:10.48550\/arXiv.2206.07682 10.48550\/arXiv.2206.07682","DOI":"10.48550\/arXiv.2206.07682"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2410.24198"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2312.02120"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3736407"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2403.19114"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2404.10719"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2403.07506"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.10958"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2302.00288"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2307.03601"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-acl.52"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","unstructured":"Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu Yonghao Zhuang Zi Lin Zhuohan Li Dacheng Li Eric Xing et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36 (2023) 46595-46623. doi:10.48550\/arXiv.2306.05685 10.48550\/arXiv.2306.05685","DOI":"10.48550\/arXiv.2306.05685"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3808123","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:43:32Z","timestamp":1782845012000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3808123"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,30]]},"references-count":64,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3808123"],"URL":"https:\/\/doi.org\/10.1145\/3808123","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,30]]}}}