{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T02:12:59Z","timestamp":1775873579300,"version":"3.50.1"},"reference-count":67,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T00:00:00Z","timestamp":1720742400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"NSF","award":["CNS-1941898"],"award-info":[{"award-number":["CNS-1941898"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2024,7,12]]},"abstract":"<jats:p>Software developers often repeat the same code changes within a project or across different projects. These repetitive changes are known as \u201ccode change patterns\u201d (CPATs). Automating CPATs is crucial to expedite the software development process. While current Transformation by Example (TBE) techniques can automate CPATs, they are limited by the quality and quantity of the provided input examples. Thus, they miss transforming code variations that do not have the exact syntax, data-, or control-flow of the provided input examples, despite being semantically similar. Large Language Models (LLMs), pre-trained on extensive source code datasets, offer a potential solution. Harnessing the capability of LLMs to generate semantically equivalent, yet previously unseen variants of the original CPAT could significantly increase the effectiveness of TBE systems.<\/jats:p>\n                  <jats:p>\n                    In this paper, we first discover best practices for harnessing LLMs to generate code variants that meet three criteria: correctness (semantic equivalence to the original CPAT), usefulness (reflecting what developers typically write), and applicability (aligning with the primary intent of the original CPAT). We then implement these practices in our tool PyCraft, which synergistically combines static code analysis, dynamic analysis, and LLM capabilities. By employing chain-of-thought reasoning, PyCraft generates variations of input examples\n                    <jats:italic toggle=\"yes\">and<\/jats:italic>\n                    comprehensive test cases that identify correct variations with an F-measure of 96.6%. Our algorithm uses feedback iteration to expand the original input examples by an average factor of 58x. Using these richly generated examples, we inferred transformation rules and then automated these changes, resulting in an increase of up to 39x, with an average increase of 14x in target codes compared to a previous state-of-the-art tool that relies solely on static analysis. We submitted patches generated by PyCraft to a range of projects, notably esteemed ones like\n                    <jats:italic toggle=\"yes\">microsoft\/DeepSpeed<\/jats:italic>\n                    and\n                    <jats:italic toggle=\"yes\">IBM\/inFairness.<\/jats:italic>\n                    Their developers accepted and merged 83% the 86 CPAT instances submitted through 44 pull requests. This confirms the usefulness of these changes.\n                  <\/jats:p>","DOI":"10.1145\/3643755","type":"journal-article","created":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T10:22:09Z","timestamp":1720779729000},"page":"631-653","source":"Crossref","is-referenced-by-count":17,"title":["Unprecedented Code Change Automation: The Fusion of LLMs and Transformation by Example"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5471-7179","authenticated-orcid":false,"given":"Malinda","family":"Dilhara","sequence":"first","affiliation":[{"name":"University of Colorado, Boulder, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-7048-4406","authenticated-orcid":false,"given":"Abhiram","family":"Bellur","sequence":"additional","affiliation":[{"name":"University of Colorado, Boulder, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9022-3563","authenticated-orcid":false,"given":"Timofey","family":"Bryksin","sequence":"additional","affiliation":[{"name":"JetBrains Research, Limassol, Cyprus"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5046-2017","authenticated-orcid":false,"given":"Danny","family":"Dig","sequence":"additional","affiliation":[{"name":"JetBrains Research - University of Colorado, Boulder, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,7,12]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"Google AI. 2023. Google Bard: An Early Experiment with Generative AI. https:\/\/ai.google\/static\/documents\/google-about-bard.pdf"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3276954.3276960"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635901"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2008.44"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3360585"},{"key":"e_1_3_1_7_2","article-title":"DeepCoder: Learning to Write Programs","author":"Balog Matej","year":"2016","unstructured":"MatejBalog, Alexander L.Gaunt, MarcBrockschmidt, SebastianNowozin, and DanielTarlow. 2016. DeepCoder: Learning to Write Programs. ArXiv (2016).","journal-title":"ArXiv"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635898"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196445"},{"key":"e_1_3_1_10_2","article-title":"Language models are few-shot learners","author":"Brown Tom B","year":"2020","unstructured":"Tom BBrown, BenjaminMann, NickRyder, MelanieSubbiah, JaredKaplan, PrafullaDhariwal, ArvindNeelakantan, PranavShyam, GirishSastry, AmandaAskell, etal. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020).","journal-title":"arXiv preprint arXiv:2005.14165"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1177\/0049124113500475"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3128234"},{"key":"e_1_3_1_13_2","article-title":"Advanced C++ Programming Styles and Idioms. Addison-W esley","author":"Coplien James","year":"1992","unstructured":"JamesCoplien. 1992. Advanced C++ Programming Styles and Idioms. Addison-W esley. Reading, MA (1992).","journal-title":"Reading, MA"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/2000799.2000805"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00091"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3453478"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510225"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568295"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00078"},{"key":"e_1_3_1_20_2","unstructured":"ZhiyuFan and XiangGao. [n. d.]. Automated Repair of Programs from Large Language Models. ([n. d.])."},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330571"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387905.3388608"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","unstructured":"YuFeng RubenMartins Jacob VanGeffen IsilDillig and SwaratChaudhuri. 2017. Component-Based Synthesis of Table Consolidation and Transformation Tasks from Examples (PLDI). https:\/\/doi.org\/10.1145\/3062341.3062351 10.1145\/3062341.3062351","DOI":"10.1145\/3062341.3062351"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00109"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3485538"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387904.3389285"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-10096-0"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME52107.2021.00061"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/1062455.1062512"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/2902362"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/WCRE.2013.6671308"},{"key":"e_1_3_1_33_2","unstructured":"AndersHovmaller. 2023. mutmut - python mutation tester. https:\/\/github.com\/boxed\/mutmut."},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510115"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00072"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.2988396"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/1993316.1993537"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2013.6606596"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3360569"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568317"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00089"},{"key":"e_1_3_1_43_2","unstructured":"OpenAI. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 (2023)."},{"key":"e_1_3_1_44_2","unstructured":"OpenAI. 2023. OpenAI Codex. https:\/\/openai.com\/blog\/openai-codex."},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME46990.2020.00098"},{"key":"e_1_3_1_46_2","article-title":"Synchromesh: Reliable Code Generation from Pre-trained Language Models","author":"Poesia Gabriel","year":"2021","unstructured":"GabrielPoesia, AlexPolozov, VuLe, AshishTiwari, GustavoSoares, ChristopherMeek, and SumitGulwani. 2021. Synchromesh: Reliable Code Generation from Pre-trained Language Models. In International Conference on Learning Representations. https:\/\/arxiv.org\/abs\/2201.11227","journal-title":"International Conference on Learning Representations"},{"key":"e_1_3_1_47_2","unstructured":"DorinPomian AbhiramBellur MalindaDilhara ZarinaKurbatova EgorBogomolov TimofeyBryksin and DannyDig. 2024. Together We Go Further: LLMs and IDE Static Analysis for Extract Method Refactoring. (2024). arXiv:2401.15298 [cs.SE]"},{"key":"e_1_3_1_48_2","doi-asserted-by":"crossref","unstructured":"DorinPomian AbhiramBellur MalindaDilhara ZarinaKurbatova EgorBogomolov AndreySokolov TimofeyBryksin and DannyDig. 2024. EM-Assist: Safe Automated ExtractMethod Refactoring with LLMs. (2024). arXiv:2405.20551 [cs.SE]","DOI":"10.1145\/3663529.3663803"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3276517"},{"key":"e_1_3_1_50_2","unstructured":"PyCraft-Authors. 2023. PyCraft. https:\/\/pycrafttool.github.io Accessed: 2024-02-26."},{"key":"e_1_3_1_51_2","article-title":"Language models are unsupervised multitask learners","volume":"8","author":"Radford Alec","year":"2019","unstructured":"AlecRadford, JeffreyWu, RewonChild, DavidLuan, DarioAmodei, IlyaSutskever, etal. 2019. Language models are unsupervised multitask learners. OpenAI blog 8 (2019).","journal-title":"OpenAI blog"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00117"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.44"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884877"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","unstructured":"TattiyaSakulniwat Raula GaikovinaKula ChaiyongRagkhitwetsagul MorakotChoetkiertikul ThanwadeeSunetnanta DongWang TakashiIshio and KenichiMatsumoto. 2019. Visualizing the Usage of Pythonic Idioms Over Time: A Case Study of the with open Idiom. https:\/\/doi.org\/10.1109\/IWESEP49350.2019.00016 10.1109\/IWESEP49350.2019.00016","DOI":"10.1109\/IWESEP49350.2019.00016"},{"key":"e_1_3_1_56_2","volume-title":"Proceedings of the USENIX Annual Technical Conference (USENIX ATC)","author":"Serrano Luis","year":"2020","unstructured":"LuisSerrano, Vu AnhNguyen, FerdianThung, LianghaoJiang, DavidLo, JuliaLawall, and GillesMuller. 2020. SPINFER: Inferring Semantic Patches for the Linux Kernel. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC). USENIX Association. https:\/\/www.usenix.org\/conference\/atc20\/presentation\/serrano"},{"key":"e_1_3_1_57_2","article-title":"Prompting Is All Your Need: Automated Android Bug Replay with Large Language Models","author":"Sidong Feng","year":"2024","unstructured":"FengSidong and ChenChunyang. 2024. Prompting Is All Your Need: Automated Android Bug Replay with Large Language Models. In Proceedings of the 46th International Conference on Software Engineering (ICSE).","journal-title":"Proceedings of the 46th International Conference on Software Engineering (ICSE)"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.14778\/2977797.2977807"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3474624.3474650"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00033"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3314221.3314589"},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678615"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","unstructured":"YueWang WeishiWang ShafiqJoty and Steven C.H.Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. https:\/\/doi.org\/10.18653\/v1\/2021.emnlp-main.685 10.18653\/v1\/2021.emnlp-main.685","DOI":"10.18653\/v1\/2021.emnlp-main.685"},{"key":"e_1_3_1_64_2","unstructured":"JasonWei XuezhiWang DaleSchuurmans MaartenBosma brianichter FeiXia EdChi Quoc VLe and DennyZhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models."},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1108\/QROM-08-2016-1408"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC.2019.00052"},{"key":"e_1_3_1_67_2","article-title":"Repairing bugs in python assignments using large language models","author":"Zhang Jialu","year":"2022","unstructured":"JialuZhang, JosaCambronero, SumitGulwani, VuLe, RuzicaPiskac, GustavoSoares, and GustVerbruggen. 2022. Repairing bugs in python assignments using large language models. arXiv preprint arXiv:2209.14876 (2022).","journal-title":"arXiv preprint arXiv:2209.14876"},{"key":"e_1_3_1_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549143"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643755","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3643755","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3643755","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T08:07:40Z","timestamp":1770192460000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643755"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,12]]},"references-count":67,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2024,7,12]]}},"alternative-id":["10.1145\/3643755"],"URL":"https:\/\/doi.org\/10.1145\/3643755","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,12]]}}}