{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T11:34:23Z","timestamp":1791459263202,"version":"4.3.4"},"publisher-location":"New York, NY, USA","reference-count":44,"publisher":"ACM","license":[{"start":{"date-parts":[[2026,7,5]],"date-time":"2026-07-05T00:00:00Z","timestamp":1783209600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,7,5]]},"DOI":"10.1145\/3805760.3814905","type":"proceedings-article","created":{"date-parts":[[2026,7,5]],"date-time":"2026-07-05T08:46:13Z","timestamp":1783241173000},"page":"151-159","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["An Empirical Study of Reasoning Steps in Thinking Code LLMs"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-1937-3733","authenticated-orcid":false,"given":"Haoran","family":"Xue","sequence":"first","affiliation":[{"name":"York University, Department of Electrical Engineering and Computer Science, North York, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1376-095X","authenticated-orcid":false,"given":"Gias","family":"Uddin","sequence":"additional","affiliation":[{"name":"York University, Department of Electrical Engineering and Computer Science, North York, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0617-2877","authenticated-orcid":false,"given":"Song","family":"Wang","sequence":"additional","affiliation":[{"name":"York University, Department of Electrical Engineering and Computer Science, North York, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,7,5]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Anthropic. 2025. Claude 3.7 Sonnet and Claude Code. https:\/\/www.anthropic. com\/news\/claude-3-7-sonnet"},{"key":"e_1_3_2_1_2_1","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732 [cs.PL] https:\/\/arxiv.org\/abs\/2108.07732"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3643757"},{"key":"e_1_3_2_1_4_1","volume-title":"ACORN: Aspect-wise Commonsense Reasoning Explanation Evaluation. arXiv:2405.04818 [cs.CL] https:\/\/arxiv.org\/abs\/2405.04818","author":"Brassard Ana","year":"2024","unstructured":"Ana Brassard, Benjamin Heinzerling, Keito Kudo, Keisuke Sakaguchi, and Kentaro Inui. 2024. ACORN: Aspect-wise Commonsense Reasoning Explanation Evaluation. arXiv:2405.04818 [cs.CL] https:\/\/arxiv.org\/abs\/2405.04818"},{"key":"e_1_3_2_1_5_1","unstructured":"Ira Ceka Saurabh Pujar Irene Manotas Gail Kaiser Baishakhi Ray and Shyam Ramji. 2025. How Does LLM Reasoning Work for Code? A Survey and a Call to Action. arXiv:2506.13932 [cs.SE] https:\/\/arxiv.org\/abs\/2506.13932"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde Jared Kaplan Harri Edwards Yura Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. doi:10.48550\/arXiv.2107.03374 10.48550\/arXiv.2107.03374","DOI":"10.48550\/arXiv.2107.03374"},{"key":"e_1_3_2_1_7_1","unstructured":"DeepSeek-AI and Daya Guo et al. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 [cs.CL] https: \/\/arxiv.org\/abs\/2501.12948"},{"key":"e_1_3_2_1_8_1","unstructured":"DeepSeek Inc. 2025. Reasoning Model (deepseek-reasoner). https:\/\/api-docs. deepseek.com\/guides\/reasoning_model"},{"key":"e_1_3_2_1_9_1","unstructured":"Xueying Du Mingwei Liu Kaixin Wang Hanlin Wang Junwei Liu Yixuan Chen Jiayi Feng Chaofeng Sha Xin Peng and Yiling Lou. 2023. ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation. arXiv:2308.01861 [cs.CL] https:\/\/arxiv.org\/abs\/2308.01861"},{"key":"e_1_3_2_1_10_1","unstructured":"DeepSeek-AI et al. 2025. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL] https:\/\/arxiv.org\/abs\/2412.19437"},{"key":"e_1_3_2_1_11_1","volume-title":"Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805 [cs.CL] https:\/\/arxiv.org\/abs\/2312.11805","author":"Gemini Team","year":"2025","unstructured":"Gemini Team et al. 2025. Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805 [cs.CL] https:\/\/arxiv.org\/abs\/2312.11805"},{"key":"e_1_3_2_1_12_1","volume-title":"Thoughtology: Let's think about LLM Reasoning. arXiv:2504.07128 [cs.CL] https:\/\/arxiv.org\/abs\/2504.07128","author":"Marjanovi\u0107 Sara Vera","year":"2025","unstructured":"Sara Vera Marjanovi\u0107 et al. 2025. DeepSeek-R1 Thoughtology: Let's think about LLM Reasoning. arXiv:2504.07128 [cs.CL] https:\/\/arxiv.org\/abs\/2504.07128"},{"key":"e_1_3_2_1_13_1","unstructured":"Terry Yue Zhuo et al. 2024. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. arXiv:2406.15877 [cs.SE] https:\/\/arxiv.org\/abs\/2406.15877"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00128"},{"key":"e_1_3_2_1_15_1","unstructured":"Mohamed Amine Ferrag Norbert Tihanyi and Merouane Debbah. 2025. From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review. arXiv:2504.19678 [cs.AI] https:\/\/arxiv.org\/abs\/2504.19678"},{"key":"e_1_3_2_1_16_1","unstructured":"Google AI. 2025. Gemini API Models Documentation: Gemini 2.5 Flash. https: \/\/ai.google.dev\/gemini-api\/docs\/models#gemini-2.5-flash"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3729379"},{"key":"e_1_3_2_1_18_1","unstructured":"Naman Jain King Han Alex Gu Wen-Ding Li Fanjia Yan Tianjun Zhang Sida Wang Armando Solar-Lezama Koushik Sen and Ion Stoica. 2024. Live- CodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv:2403.07974 [cs.SE] https:\/\/arxiv.org\/abs\/2403.07974"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3747588"},{"key":"e_1_3_2_1_20_1","volume-title":"Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770","author":"Jimenez Carlos E","year":"2023","unstructured":"Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 (2023)."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.5555\/3600270.3601883"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcm.2016.02.012"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"crossref","unstructured":"Jinu Lee and Julia Hockenmaier. 2025. Evaluating Step-by-step Reasoning Traces: A Survey. arXiv:2502.12289 [cs.CL] https:\/\/arxiv.org\/abs\/2502.12289","DOI":"10.18653\/v1\/2025.findings-emnlp.94"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","unstructured":"Vera Liao and Jennifer Vaughan. 2023. AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap. doi:10.48550\/arXiv.2306.01941 10.48550\/arXiv.2306.01941","DOI":"10.48550\/arXiv.2306.01941"},{"key":"e_1_3_2_1_25_1","unstructured":"Hunter Lightman Vineet Kosaraju Yura Burda Harri Edwards Bowen Baker Teddy Lee Jan Leike John Schulman Ilya Sutskever and Karl Cobbe. 2023. Let's Verify Step by Step. arXiv:2305.20050 [cs.LG] https:\/\/arxiv.org\/abs\/2305.20050"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.emnlp-demo.46"},{"key":"e_1_3_2_1_27_1","unstructured":"Iman Mirzadeh Keivan Alizadeh Hooman Shahrokhi Oncel Tuzel Samy Bengio and Mehrdad Farajtabar. 2024. GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. arXiv:2410.05229 [cs.LG] https:\/\/arxiv.org\/abs\/2410.05229"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","unstructured":"Birger Moell Fredrik Sand and Sanian Akbar. 2025. Medical Reasoning in LLMs: An In-Depth Analysis of DeepSeek R1. doi:10.48550\/arXiv.2504.00016 10.48550\/arXiv.2504.00016","DOI":"10.48550\/arXiv.2504.00016"},{"key":"e_1_3_2_1_29_1","first-page":"2332","volume-title":"Proceedings of the ACM on Software Engineering 1, FSE","author":"Mu Fangwen","year":"2024","unstructured":"Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. 2024. Clarifygpt: A framework for enhancing llm-based code generation via requirements clarification. Proceedings of the ACM on Software Engineering 1, FSE (2024), 2332-2354."},{"key":"e_1_3_2_1_30_1","unstructured":"OpenAI : and Aaron Jaech et al. 2024. OpenAI o1 System Card. arXiv:2412.16720 [cs.AI] https:\/\/arxiv.org\/abs\/2412.16720"},{"key":"e_1_3_2_1_31_1","unstructured":"OpenAI. 2025. OpenAI o3-mini."},{"key":"e_1_3_2_1_32_1","unstructured":"Pengcheng Qiu Chaoyi Wu Shuyu Liu Weike Zhao Zhuoxia Chen Hongfei Gu Chuanjin Peng Ya Zhang Yanfeng Wang and Weidi Xie. 2025. Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases. arXiv:2503.04691 [cs.CL] https:\/\/arxiv.org\/abs\/2503.04691"},{"key":"e_1_3_2_1_33_1","unstructured":"Melanie Sclar Yejin Choi Yulia Tsvetkov and Alane Suhr. 2024. Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. arXiv:2310.11324 [cs.CL] https:\/\/arxiv.org\/abs\/2310.11324"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-"},{"key":"e_1_3_2_1_35_1","volume-title":"International Journal of economics, commerce and management 2, 11","author":"Singh Ajay S","year":"2014","unstructured":"Ajay S Singh and Micah B Masuku. 2014. Sampling techniques & determination of sample size in applied statistics research: An overview. International Journal of economics, commerce and management 2, 11 (2014), 1-22."},{"key":"e_1_3_2_1_36_1","volume-title":"Bowman","author":"Turpin Miles","year":"2023","unstructured":"Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language models don't always say what they think: unfaithful explanations in chain-of-thought prompting (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, Article 3275, 14 pages."},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/MedAI59581.2023.00044"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","unstructured":"Yuxiang Wei Olivier Duchenne Jade Copet Quentin Carbonneaux Lingming Zhang Daniel Fried Gabriel Synnaeve Rishabh Singh and Sida Wang. 2025. SWE- RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution. doi:10.48550\/arXiv.2502.18449 10.48550\/arXiv.2502.18449","DOI":"10.48550\/arXiv.2502.18449"},{"key":"e_1_3_2_1_39_1","unstructured":"Tianhao Wu Janice Lan Weizhe Yuan Jiantao Jiao Jason Weston and Sainbayar Sukhbaatar. 2024. Thinking LLMs: General Instruction Following with Thought Generation. arXiv:2410.10630 [cs.CL] https:\/\/arxiv.org\/abs\/2410.10630"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00129"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","unstructured":"Shijie Xia Xuefeng Li Yixin Liu Tongshuang Wu and Pengfei Liu. 2025. Evaluating mathematical reasoning beyond accuracy. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence (AAAI'25\/IAAI'25\/EAAI'25). AAAI Press Article 3089 8 pages. doi:10.1609\/aaai.v39i26.34987 10.1609\/aaai.v39i26.34987","DOI":"10.1609\/aaai.v39i26.34987"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","unstructured":"Binquan Zhang Li Zhang Zhiwen Luo Yuxin Du Fang Liu Song Wang and Lin Shi. 2025. Are They All Good? Evaluating the Quality of CoTs in LLM-based Code Generation. doi:10.48550\/arXiv.2507.06980 10.48550\/arXiv.2507.06980","DOI":"10.48550\/arXiv.2507.06980"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.acl-long.50"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-024-10602-0"}],"event":{"name":"AIware '26: 3rd ACM International Conference on AI-Powered Software","location":"Montreal QC Canada","acronym":"AIware '26","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 3rd ACM International Conference on AI-Powered Software"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3805760.3814905","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,5]],"date-time":"2026-07-05T08:50:55Z","timestamp":1783241455000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3805760.3814905"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,5]]},"references-count":44,"alternative-id":["10.1145\/3805760.3814905","10.1145\/3805760"],"URL":"https:\/\/doi.org\/10.1145\/3805760.3814905","relation":{},"subject":[],"published":{"date-parts":[[2026,7,5]]},"assertion":[{"value":"2026-07-05","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}