{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T16:34:45Z","timestamp":1777566885260,"version":"3.51.4"},"reference-count":43,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2025,8,14]],"date-time":"2025-08-14T00:00:00Z","timestamp":1755129600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Key Project of Research and Development Plan of Hunan Province","award":["2025JK2146"],"award-info":[{"award-number":["2025JK2146"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Symmetry"],"abstract":"<jats:p>Code generation is a critical task in software engineering, enabling the automation of transforming natural language descriptions into executable code. Recent advancements in large language models (LLMs) have demonstrated their potential to significantly enhance code generation capabilities by leveraging complex reasoning processes. However, the large size of these models poses challenges for deployment in resource-constrained environments, as they require substantial computational resources and memory. The challenge lies in transferring the sophisticated problem-solving strategies of LLMs to smaller, more efficient models without sacrificing performance, while maintaining symmetry between the reasoning steps and final code generation. This task is further complicated by the need to preserve high code generation accuracy while reducing the resource demands of deployment. Although distillation methods have been proposed, efficiently transferring both the reasoning process and final code generation remains an underexplored area. In this work, we propose a novel distillation framework that extracts intermediate reasoning steps, such as pseudocode, from LLMs and transfers them to smaller models. Our approach enables smaller models to replicate the problem-solving strategies of larger models through a multi-task learning framework, which includes both pseudocode and code generation tasks, thus maintaining the symmetry between reasoning and output. We conducted comprehensive experiments on the CodeSearchNet dataset, comparing our distillation framework across four student models (Tranx, CodeT5, NatGen, and SPT-Code) distilled from four large language models (CodeLlama-7B, CodeQwen-7B, DeepSeek, and GPT-4). Results show that our approach consistently improves code generation performance, with the best case (CodeT5 distilled from GPT-4) achieving up to 74% improvement in Top-1 accuracy over the baseline.<\/jats:p>","DOI":"10.3390\/sym17081325","type":"journal-article","created":{"date-parts":[[2025,8,14]],"date-time":"2025-08-14T14:51:46Z","timestamp":1755183106000},"page":"1325","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Symmetry-Aware Code Generation: Distilling Pseudocode Reasoning for Lightweight Deployment of Large Language Models"],"prefix":"10.3390","volume":"17","author":[{"given":"Yonglin","family":"Li","sequence":"first","affiliation":[{"name":"Hunan Huishiwei Intelligent Technology Co., Ltd., Changsha 410013, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shanzhi","family":"Gu","sequence":"additional","affiliation":[{"name":"Hunan Huishiwei Intelligent Technology Co., Ltd., Changsha 410013, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7239-1819","authenticated-orcid":false,"given":"Mingyang","family":"Geng","sequence":"additional","affiliation":[{"name":"College of Computer Science, National University of Defense Technology, Changsha 410073, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,8,14]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Gu, Q. (2023, January 3\u20139). Llm-based code generation method for golang compiler testing. Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, San Francisco, CA, USA.","DOI":"10.1145\/3611643.3617850"},{"key":"ref_2","first-page":"1","article-title":"Self-planning code generation with large language models","volume":"33","author":"Jiang","year":"2024","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"ref_3","first-page":"1","article-title":"Acecoder: An effective prompting technique specialized in code generation","volume":"33","author":"Li","year":"2024","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Zamfirescu-Pereira, J., Jun, E., Terry, M., Yang, Q., and Hartmann, B. (May, January 26). Beyond Code Generation: LLM-supported Exploration of the Program Design Space. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.","DOI":"10.1145\/3706598.3714154"},{"key":"ref_5","unstructured":"Zhang, H., Cheng, W., Wu, Y., and Hu, W. (November, January 27). A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinement. Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering, Sacramento, CA, USA."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Geng, M., Wang, S., Dong, D., Wang, H., Li, G., Jin, Z., Mao, X., and Liao, X. (2024, January 14\u201320). Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning. Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering, Lisbon, Portugal.","DOI":"10.1145\/3597503.3608134"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3672459","article-title":"Self-collaboration code generation via chatgpt","volume":"33","author":"Dong","year":"2024","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"ref_8","unstructured":"Hong, S., Zheng, X., Chen, J., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S.K.S., Lin, Z., and Zhou, L. (2023). Metagpt: Meta programming for multi-agent collaborative framework. arXiv."},{"key":"ref_9","unstructured":"Sun, Z., Lyu, C., Li, B., Wan, Y., Zhang, H., Li, G., and Jin, Z. (2024). Enhancing code generation performance of smaller models by distilling the reasoning ability of llms. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Chen, H., Saha, A., Hoi, S., and Joty, S. (2023). Personalised distillation: Empowering open-sourced llms with adaptive learning for code generation. arXiv.","DOI":"10.18653\/v1\/2023.emnlp-main.417"},{"key":"ref_11","unstructured":"Malmsten, E. (2023). Distil-CodeGPT, Distilling Code-Generation Models for Local Use. [Ph.D. Thesis, Delft University of Technology]."},{"key":"ref_12","unstructured":"Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., and Huang, F. (2023). Qwen technical report. arXiv."},{"key":"ref_13","unstructured":"Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X.E., Adi, Y., Liu, J., Sauvestre, R., and Remez, T. (2023). Code llama: Open foundation models for code. arXiv."},{"key":"ref_14","unstructured":"Li, J., Zhu, H., Liu, H., Shi, X., Zong, H., Dong, Y., Zhang, K., Jiang, S., Jin, Z., and Li, G. (2025). aiXcoder-7B-v2: Training LLMs to Fully Utilize the Long Context in Repository-level Code Completion. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Jiang, S., Li, J., Zong, H., Liu, H., Zhu, H., Hu, S., Li, E., Ding, J., Han, Y., and Ning, W. (2024). aixcoder-7b: A lightweight and effective large language model for code completion. arXiv.","DOI":"10.32388\/ATAHD0"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Hsieh, C.Y., Li, C.L., Yeh, C.K., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.Y., and Pfister, T. (2023). Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv.","DOI":"10.18653\/v1\/2023.findings-acl.507"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Li, L.H., Hessel, J., Yu, Y., Ren, X., Chang, K.W., and Choi, Y. (2023). Symbolic chain-of-thought distillation: Small models can also \u201cthink\u201d step-by-step. arXiv.","DOI":"10.18653\/v1\/2023.acl-long.150"},{"key":"ref_18","unstructured":"Husain, H., Wu, H.H., Gazit, T., Allamanis, M., and Brockschmidt, M. (2019). Codesearchnet challenge: Evaluating the state of semantic code search. arXiv."},{"key":"ref_19","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_20","first-page":"70757","article-title":"Towards revealing the mystery behind chain of thought: A theoretical perspective","volume":"36","author":"Feng","year":"2023","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_21","unstructured":"Xia, Y., Wang, R., Liu, X., Li, M., Yu, T., Chen, X., McAuley, J., and Li, S. (2024). Beyond chain-of-thought: A survey of chain-of-x paradigms for llms. arXiv."},{"key":"ref_22","first-page":"333","article-title":"Chain of preference optimization: Improving chain-of-thought reasoning in llms","volume":"37","author":"Zhang","year":"2024","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_23","unstructured":"Yang, R., Wu, T., Wang, J., Hu, P., Wu, Y.C., Wong, N., and Yang, Y. (2024). Llm-neo: Parameter efficient knowledge distillation for large language models. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Pangakis, N., and Wolken, S. (2024). Knowledge distillation in automated annotation: Supervised text classification with llm-generated training labels. arXiv.","DOI":"10.18653\/v1\/2024.nlpcss-1.9"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Li, L., Zhang, Y., and Chen, L. (2023, January 21\u201325). Prompt distillation for efficient llm-based recommendation. Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, Birmingham, UK.","DOI":"10.1145\/3583780.3615017"},{"key":"ref_26","unstructured":"Hu, S., Zou, G., Yang, S., Lin, S., Gan, Y., Zhang, B., and Chen, Y. (March, January 27). Large language model meets graph neural network in knowledge distillation. Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, PA, USA."},{"key":"ref_27","first-page":"1","article-title":"Efficient and effective role player: A compact knowledge-grounded persona-based dialogue model enhanced by llm distillation","volume":"43","author":"Hu","year":"2025","journal-title":"ACM Trans. Inf. Syst."},{"key":"ref_28","first-page":"98297","article-title":"Ddk: Distilling domain knowledge for efficient large language models","volume":"37","author":"Liu","year":"2024","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_29","unstructured":"Fang, L., Yu, X., Cai, J., Chen, Y., Wu, S., Liu, Z., Yang, Z., Lu, H., Gong, X., and Liu, Y. (2025). Knowledge distillation and dataset distillation of large language models: Emerging trends, challenges, and future directions. arXiv."},{"key":"ref_30","unstructured":"Song, Y., Zhang, J., Tian, Z., Yang, Y., Huang, M., and Li, D. (2024). LLM-based privacy data augmentation guided by knowledge distillation with a distribution tutor for medical text classification. arXiv."},{"key":"ref_31","unstructured":"Xu, X., Li, M., Tao, C., Shen, T., Cheng, R., Li, J., Xu, C., Tao, D., and Zhou, T. (2024). A survey on knowledge distillation of large language models. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"McDonald, D., Papadopoulos, R., and Benningfield, L. (2024). Reducing LLM hallucination using knowledge distillation: A case study with mistral large and MMLU benchmark. TechRxiv.","DOI":"10.36227\/techrxiv.171665607.76504195\/v1"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1758","DOI":"10.1109\/TKDE.2023.3312255","article-title":"Out-of-distribution generalization with causal feature separation","volume":"36","author":"Wang","year":"2023","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Wang, H., Yang, W., Yang, L., Wu, A., Xu, L., Ren, J., Wu, F., and Kuang, K. (2022, January 14\u201318). Estimating individualized causal effect with confounded instruments. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA.","DOI":"10.1145\/3534678.3539335"},{"key":"ref_35","unstructured":"Wang, H., Li, H., Zou, H., Chi, H., Lan, L., Huang, W., and Yang, W. (2025, January 24\u201328). Effective and efficient time-varying counterfactual prediction with state-space models. Proceedings of the Thirteenth International Conference on Learning Representations, Singapore."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Wang, S., Geng, M., Lin, B., Sun, Z., Wen, M., Liu, Y., Li, L., Bissyand\u00e9, T.F., and Mao, X. (2023, January 3\u20139). Natural language to code: How far are we?. Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, San Francisco, CA, USA.","DOI":"10.1145\/3611643.3616323"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002, January 7\u201312). Bleu: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.","DOI":"10.3115\/1073083.1073135"},{"key":"ref_38","unstructured":"Ren, S., Guo, D., Lu, S., Zhou, L., Liu, S., Tang, D., Sundaresan, N., Zhou, M., Blanco, A., and Ma, S. (2020). Codebleu: A method for automatic evaluation of code synthesis. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Yin, P., and Neubig, G. (2018). TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation. arXiv.","DOI":"10.18653\/v1\/D18-2002"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Ni, J., Abrego, G.H., Constant, N., Ma, J., Hall, K.B., Cer, D., and Yang, Y. (2021). Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv.","DOI":"10.18653\/v1\/2022.findings-acl.146"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Wang, Y., Wang, W., Joty, S., and Hoi, S.C. (2021). Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv.","DOI":"10.18653\/v1\/2021.emnlp-main.685"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Chakraborty, S., Ahmed, T., Ding, Y., Devanbu, P.T., and Ray, B. (2022, January 14\u201318). Natgen: Generative pre-training by \u201cnaturalizing\u201d source code. Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Singapore.","DOI":"10.1145\/3540250.3549162"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Niu, C., Li, C., Ng, V., Ge, J., Huang, L., and Luo, B. (2022, January 21\u201329). Spt-code: Sequence-to-sequence pre-training for learning source code representations. Proceedings of the 44th International Conference on Software Engineering, Pittsburgh, PA, USA.","DOI":"10.1145\/3510003.3510096"}],"container-title":["Symmetry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-8994\/17\/8\/1325\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:27:16Z","timestamp":1760034436000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-8994\/17\/8\/1325"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,14]]},"references-count":43,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2025,8]]}},"alternative-id":["sym17081325"],"URL":"https:\/\/doi.org\/10.3390\/sym17081325","relation":{},"ISSN":["2073-8994"],"issn-type":[{"value":"2073-8994","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,14]]}}}