{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T10:07:21Z","timestamp":1789034841913,"version":"build-2803163510"},"publisher-location":"New York, NY, USA","reference-count":42,"publisher":"ACM","funder":[{"name":"NSF","award":["22-38045"],"award-info":[{"award-number":["22-38045"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,6,23]]},"DOI":"10.1145\/3696630.3728605","type":"proceedings-article","created":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T19:08:09Z","timestamp":1753729689000},"page":"1178-1182","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-1891-2564","authenticated-orcid":false,"given":"Changshu","family":"Liu","sequence":"first","affiliation":[{"name":"University of Illinois at Urbana-Champaign, Urbana, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0668-8526","authenticated-orcid":false,"given":"Reyhan","family":"Jabbarvand","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign, Urbana, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,7,28]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"2025. Matplot library for visualization. https:\/\/matplotlib.org\/."},{"key":"e_1_3_2_1_2_1","unstructured":"2025. Network Analysis in Python using NetworkX. https:\/\/networkx.org\/."},{"key":"e_1_3_2_1_3_1","unstructured":"2025. PyGraphVis library for visualization. https:\/\/pygraphviz.github.io\/."},{"key":"e_1_3_2_1_4_1","unstructured":"2025. Python AST module. https:\/\/docs.python.org\/3\/library\/ast.html."},{"key":"e_1_3_2_1_5_1","volume-title":"Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.","author":"Achiam Josh","year":"2023","unstructured":"Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)."},{"key":"e_1_3_2_1_6_1","volume-title":"Saikat Chakraborty, and Kai-Wei Chang.","author":"Ahmad Wasi Uddin","year":"2021","unstructured":"Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai-Wei Chang. 2021. Avatar: A parallel corpus for java-python program translation. arXiv preprint arXiv:2108.11590 (2021)."},{"key":"e_1_3_2_1_7_1","unstructured":"Xiao Bi Deli Chen Guanting Chen Shanhuang Chen Damai Dai Chengqi Deng Honghui Ding Kai Dong Qiushi Du Zhe Fu et al. 2024. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. arXiv preprint arXiv:2401.02954 (2024)."},{"key":"e_1_3_2_1_8_1","volume-title":"Yuanzhi Li, Scott Lundberg, et al.","author":"Bubeck S\u00e9bastien","year":"2023","unstructured":"S\u00e9bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023)."},{"key":"e_1_3_2_1_9_1","volume-title":"2025 IEEE\/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 140\u2013152","author":"Chen Junkai","year":"2024","unstructured":"Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2024. Reasoning Runtime Behavior of a Program with LLM: How Far Are We?. In 2025 IEEE\/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 140\u2013152."},{"key":"e_1_3_2_1_10_1","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs.LG]"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3650212.3680369"},{"key":"e_1_3_2_1_12_1","unstructured":"Rohan Deshpande Jerry Chen and Isabelle Lee. 2021. RecT: A Recursive Transformer Architecture for Generalizable Mathematical Reasoning.. In NeSy. 165\u2013175."},{"key":"e_1_3_2_1_13_1","volume-title":"SemCoder: Training Code Language Models with Comprehensive Semantics. arXiv preprint arXiv:2406.01006","author":"Ding Yangruibo","year":"2024","unstructured":"Yangruibo Ding, Jinjun Peng, Marcus J Min, Gail Kaiser, Junfeng Yang, and Baishakhi Ray. 2024. SemCoder: Training Code Language Models with Comprehensive Semantics. arXiv preprint arXiv:2406.01006 (2024)."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639219"},{"key":"e_1_3_2_1_15_1","volume-title":"CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution. arXiv preprint arXiv:2401.03065","author":"Gu Alex","year":"2024","unstructured":"Alex Gu, Baptiste Rozi\u00e8re, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024. CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution. arXiv preprint arXiv:2401.03065 (2024)."},{"key":"e_1_3_2_1_16_1","volume-title":"Yi R Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, and Heng Ji.","author":"Huang Kung-Hsiang","year":"2023","unstructured":"Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi R Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, and Heng Ji. 2023. Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning. arXiv preprint arXiv:2312.10160 (2023)."},{"key":"e_1_3_2_1_17_1","volume-title":"Automated bug generation in the era of large language models. arXiv preprint arXiv:2310.02407","author":"Ibrahimzada Ali Reza","year":"2023","unstructured":"Ali Reza Ibrahimzada, Yang Chen, Ryan Rong, and Reyhaneh Jabbarvand. 2023. Automated bug generation in the era of large language models. arXiv preprint arXiv:2310.02407 (2023)."},{"key":"e_1_3_2_1_18_1","volume-title":"Rangeet Pan, Saurabh Sinha, and Reyhaneh Jabbarvand.","author":"Ibrahimzada Ali Reza","year":"2024","unstructured":"Ali Reza Ibrahimzada, Kaiyao Ke, Mrigank Pawagi, Muhammad Salman Abid, Rangeet Pan, Saurabh Sinha, and Reyhaneh Jabbarvand. 2024. Repository-level compositional code translation and validation. arXiv preprint arXiv:2410.24117 (2024)."},{"key":"e_1_3_2_1_19_1","volume-title":"Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05398","author":"Imani Shima","year":"2023","unstructured":"Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05398 (2023)."},{"key":"e_1_3_2_1_20_1","volume-title":"What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? arXiv preprint arXiv:2411.07681","author":"Kang Katie","year":"2024","unstructured":"Katie Kang, Amrith Setlur, Dibya Ghosh, Jacob Steinhardt, Claire Tomlin, Sergey Levine, and Aviral Kumar. 2024. What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? arXiv preprint arXiv:2411.07681 (2024)."},{"key":"e_1_3_2_1_21_1","volume-title":"Code Simulation Challenges for Large Language Models. arXiv preprint arXiv:2401.09074","author":"Malfa Emanuele La","year":"2024","unstructured":"Emanuele La Malfa, Christoph Weinhuber, Orazio Torre, Fangru Lin, Anthony Cohn, Nigel Shadbolt, and Michael Wooldridge. 2024. Code Simulation Challenges for Large Language Models. arXiv preprint arXiv:2401.09074 (2024)."},{"key":"e_1_3_2_1_22_1","volume-title":"Ali Reza Ibrahimzada, and Reyhaneh Jabbarvand.","author":"Liu Changshu","year":"2024","unstructured":"Changshu Liu, Shizhuo Dylan Zhang, Ali Reza Ibrahimzada, and Reyhaneh Jabbarvand. 2024. Codemind: A framework to challenge large language models for code reasoning. arXiv preprint arXiv:2402.09664 (2024)."},{"key":"e_1_3_2_1_23_1","volume-title":"Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al.","author":"Lozhkov Anton","year":"2024","unstructured":"Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. StarCoder 2 and The Stack v2: The Next Generation. arXiv preprint arXiv:2402.19173 (2024)."},{"key":"e_1_3_2_1_24_1","volume-title":"Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583","author":"Luo Haipeng","year":"2023","unstructured":"Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583 (2023)."},{"key":"e_1_3_2_1_25_1","volume-title":"The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Python. arXiv preprint arXiv:2305.15507","author":"Miceli-Barone Antonio Valerio","year":"2023","unstructured":"Antonio Valerio Miceli-Barone, Fazl Barez, Ioannis Konstas, and Shay B Cohen. 2023. The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Python. arXiv preprint arXiv:2305.15507 (2023)."},{"key":"e_1_3_2_1_26_1","volume-title":"Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain. arXiv preprint arXiv:2310.14053","author":"Min Marcus J","year":"2023","unstructured":"Marcus J Min, Yangruibo Ding, Luca Buratti, Saurabh Pujar, Gail Kaiser, Suman Jana, and Baishakhi Ray. 2023. Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain. arXiv preprint arXiv:2310.14053 (2023)."},{"key":"e_1_3_2_1_27_1","volume-title":"NExT: Teaching Large Language Models to Reason about Code Execution. arXiv preprint arXiv:2404.14662","author":"Ni Ansong","year":"2024","unstructured":"Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. 2024. NExT: Teaching Large Language Models to Reason about Code Execution. arXiv preprint arXiv:2404.14662 (2024)."},{"key":"e_1_3_2_1_28_1","unstructured":"OpenAI. 2023. GPT-4 Technical Report. https:\/\/arxiv.org\/abs\/2303.08774 (2023)."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639226"},{"key":"e_1_3_2_1_30_1","unstructured":"Machel Reid Nikolay Savinov Denis Teplyashin Dmitry Lepikhin Timothy Lillicrap Jean-baptiste Alayrac Radu Soricut Angeliki Lazaridou Orhan Firat Julian Schrittwieser et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 (2024)."},{"key":"e_1_3_2_1_31_1","volume-title":"Yossi Adi, Jingyu Liu, Tal Remez, J\u00e9r\u00e9my Rapin, et al.","author":"Roziere Baptiste","year":"2023","unstructured":"Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J\u00e9r\u00e9my Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"crossref","unstructured":"Charles Spearman. 1961. The proof and measurement of association between two things. (1961).","DOI":"10.1037\/11491-005"},{"key":"e_1_3_2_1_33_1","unstructured":"Gemini Team Rohan Anil Sebastian Borgeaud Yonghui Wu Jean-Baptiste Alayrac Jiahui Yu Radu Soricut Johan Schalkwyk Andrew M Dai Anja Hauth et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)."},{"key":"e_1_3_2_1_34_1","volume-title":"Large Language Models Still Can't Plan (A Benchmark for LLMs on Planning and Reasoning about Change). arXiv preprint arXiv:2206.10498","author":"Valmeekam Karthik","year":"2022","unstructured":"Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2022. Large Language Models Still Can't Plan (A Benchmark for LLMs on Planning and Reasoning about Change). arXiv preprint arXiv:2206.10498 (2022)."},{"key":"e_1_3_2_1_35_1","volume-title":"Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. arXiv preprint arXiv:2310.03731","author":"Wang Ke","year":"2023","unstructured":"Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. 2023. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. arXiv preprint arXiv:2310.03731 (2023)."},{"key":"e_1_3_2_1_36_1","unstructured":"ExeRScope Tool Artifact Website. 2025. https:\/\/github.com\/Intelligent-CAT-Lab\/ExeRScope."},{"key":"e_1_3_2_1_37_1","volume-title":"Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:2307.02477","author":"Wu Zhaofeng","year":"2023","unstructured":"Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Aky\u00fcrek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. 2023. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:2307.02477 (2023)."},{"key":"e_1_3_2_1_38_1","unstructured":"An Yang Baosong Yang Binyuan Hui Bo Zheng Bowen Yu Chang Zhou Chengpeng Li Chengyuan Li Dayiheng Liu Fei Huang et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671 (2024)."},{"key":"e_1_3_2_1_39_1","unstructured":"Terry Yin. 2025. Lizard. https:\/\/github.com\/terryyin\/lizard."},{"key":"e_1_3_2_1_40_1","volume-title":"Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursion. arXiv preprint arXiv:2401.12947","author":"Zhang Dylan","year":"2024","unstructured":"Dylan Zhang, Curt Tigges, Zory Zhang, Stella Biderman, Maxim Raginsky, and Talia Ringer. 2024. Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursion. arXiv preprint arXiv:2401.12947 (2024)."},{"key":"e_1_3_2_1_41_1","volume-title":"Can LLM Graph Reasoning Generalize beyond Pattern Memorization? arXiv preprint arXiv:2406.15992","author":"Zhang Yizhuo","year":"2024","unstructured":"Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xiaochuang Han, Tianxing He, and Yulia Tsvetkov. 2024. Can LLM Graph Reasoning Generalize beyond Pattern Memorization? arXiv preprint arXiv:2406.15992 (2024)."},{"key":"e_1_3_2_1_42_1","unstructured":"Qihao Zhu Daya Guo Zhihong Shao Dejian Yang Peiyi Wang Runxin Xu Y Wu Yukun Li Huazuo Gao Shirong Ma et al. 2024. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence. arXiv preprint arXiv:2406.11931 (2024)."}],"event":{"name":"FSE Companion '25: 33rd ACM International Conference on the Foundations of Software Engineering","location":"Clarion Hotel Trondheim Trondheim Norway","acronym":"FSE Companion '25","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3696630.3728605","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T19:09:09Z","timestamp":1753729749000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3696630.3728605"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,23]]},"references-count":42,"alternative-id":["10.1145\/3696630.3728605","10.1145\/3696630"],"URL":"https:\/\/doi.org\/10.1145\/3696630.3728605","relation":{},"subject":[],"published":{"date-parts":[[2025,6,23]]},"assertion":[{"value":"2025-07-28","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}