{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T21:18:43Z","timestamp":1783804723306,"version":"3.55.0"},"reference-count":95,"publisher":"Association for Computing Machinery (ACM)","issue":"7","license":[{"start":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T00:00:00Z","timestamp":1783728000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"name":"Fonds de Recherche du Qu\u00e9bec (FRQ), the Canadian Institute for Advanced Research","award":["DEEL project CRDPJ 537462-18"],"award-info":[{"award-number":["DEEL project CRDPJ 537462-18"]}]},{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100015773","name":"Consortium for Research and Innovation in Aerospace in Qu\u00e9bec","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100015773","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Thales Canada Inc., Bell Textron Canada Limited, CAE\u00a0Inc.,\u00a0and Bombardier Inc."}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,7,31]]},"abstract":"<jats:p>Large Language Models (LLMs) excel in code-related tasks like code generation, but benchmark evaluations often overlook task characteristics such as difficulty. Moreover, benchmarks are usually built using tasks described with a single prompt, despite the formulation of prompts having a profound impact on the outcome. This article introduces a generalist approach, TaskEval, a framework using diverse prompts and Item Response Theory (IRT) to efficiently assess LLMs\u2019 capabilities and benchmark task characteristics, improving the understanding of their performance.<\/jats:p>\n                  <jats:p>\n                    Using two code generation benchmarks,\n                    <jats:italic toggle=\"yes\">HumanEval<\/jats:italic>\n                    + and\n                    <jats:italic toggle=\"yes\">ClassEval<\/jats:italic>\n                    , as well as eight code generation LLMs, we show that\n                    <jats:italic toggle=\"yes\">TaskEval<\/jats:italic>\n                    is capable of characterising the properties of tasks. Using topic analysis, we identify and analyse the tasks of 17 and 21 topics within the benchmarks. We also cross-analyse tasks\u2019 characteristics with programming constructs (e.g. variable assignment, conditions) used by LLMs, emphasising some patterns with tasks\u2019 difficulty. Finally, we conduct a comparison between the difficulty assessment of tasks by human annotators and LLMs. Orthogonal to current benchmarking evaluation efforts,\n                    <jats:italic toggle=\"yes\">TaskEval<\/jats:italic>\n                    can assist researchers and practitioners in fostering better assessments of LLMs. The tasks\u2019 characteristics can be used to identify shortcomings within existing benchmarks or improve the evaluation of LLMs.\n                  <\/jats:p>","DOI":"10.1145\/3773285","type":"journal-article","created":{"date-parts":[[2025,10,28]],"date-time":"2025-10-28T16:12:30Z","timestamp":1761667950000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5593-9400","authenticated-orcid":false,"given":"Florian","family":"Tambon","sequence":"first","affiliation":[{"name":"SnT, University of Luxembourg, Luxembourg City, Luxembourg"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0440-6839","authenticated-orcid":false,"given":"Amin","family":"Nikanjam","sequence":"additional","affiliation":[{"name":"Huawei Distributed Scheduling and Data Engine Lab, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-8988-3483","authenticated-orcid":false,"given":"Cyrine","family":"Zid","sequence":"additional","affiliation":[{"name":"Polytechnique Montr\u00e9al, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5704-4173","authenticated-orcid":false,"given":"Foutse","family":"Khomh","sequence":"additional","affiliation":[{"name":"Polytechnique Montr\u00e9al, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7892-9202","authenticated-orcid":false,"given":"Giuliano","family":"Antoniol","sequence":"additional","affiliation":[{"name":"Polytechnique Montr\u00e9al, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,7,11]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"OpenAI. GPT-3.5 Turbo. 2023. Retrieved from https:\/\/platform.openai.com\/docs\/models\/gpt-3-5-turbo"},{"key":"e_1_3_2_3_2","unstructured":"OpenAI. GPT-4. 2023. Retrieved from https:\/\/platform.openai.com\/docs\/models\/gpt-4-turbo-and-gpt-4"},{"key":"e_1_3_2_4_2","unstructured":"HuggingFace. Big Code Leaderboard. 2024. Retrieved from https:\/\/huggingface.co\/spaces\/bigcode\/bigcode-models-leaderboard"},{"key":"e_1_3_2_5_2","unstructured":"Terry Yue Zhuo Minh Chien Vu Jenny Chim Han Hu Wenhao Yu Ratnadira Widyasari Imam Nur Bani Yusuf Haolan Zhan Junda He Indraneil Paul et al. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. 2024. arXiv:2406.15877 Retrieved from https:\/\/arxiv.org\/abs\/2406.15877"},{"key":"e_1_3_2_6_2","unstructured":"Xueying Du Mingwei Liu Kaixin Wang Hanlin Wang Junwei Liu Yixuan Chen Jiayi Feng Chaofeng Sha Xin Peng and Yiling Lou. 2024. ClassEval Leaderboard. 2024. ClassEval Leaderboard. Retrieved from https:\/\/fudanselab-classeval.github.io\/leaderboard.html"},{"key":"e_1_3_2_7_2","unstructured":"Jiawei Liu Chunqiu Steven Xia Yuyao Wang and Lingming Zhang. 2024. EvalPlus Leaderboard. 2024. EvalPlus Leaderboard. Retrieved from https:\/\/evalplus.github.io\/leaderboard.html"},{"key":"e_1_3_2_8_2","unstructured":"HuggingFace. Models on HuggingFace. 2024. Retrieved from https:\/\/huggingface.co\/models"},{"key":"e_1_3_2_9_2","unstructured":"Florian Tambon. Rep-Package. 2024. Retrieved from https:\/\/github.com\/FlowSs\/TaskEval"},{"key":"e_1_3_2_10_2","unstructured":"Alex Young Bei Chen Chao Li Chengen Huang Ge Zhang Guanwei Zhang Guoyin Wang Heng Li Jiangcheng Zhu Jianqun Chen et al. 2025. Yi: Open foundation models by 01.AI. arXiv:2403.04652. Retrieved from https:\/\/arxiv.org\/abs\/2403.04652"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00014"},{"key":"e_1_3_2_12_2","unstructured":"Usman Anwar Abulhair Saparov Javier Rando Daniel Paleka Miles Turpin Peter Hase Ekdeep Singh Lubana Erik Jenner Stephen Casper Oliver Sourbut et al. 2024. Foundational challenges in assuring alignment and safety of large language models. arXiv:2404.09932. Retrieved from https:\/\/arxiv.org\/abs\/2404.09932"},{"key":"e_1_3_2_13_2","unstructured":"Ben Athiwaratkun Sanjay Krishna Gouda Zijian Wang Xiaopeng Li Yuchen Tian Ming Tan Wasi Uddin Ahmad Shiqi Wang Qing Sun Mingyue Shang et al. 2022. Multi-lingual evaluation of code generation models. arXiv:2210.14868. Retrieved from https:\/\/arxiv.org\/abs\/10.48550\/arXiv.2210.14868"},{"key":"e_1_3_2_14_2","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et al. 2021. Program synthesis with large language models. arXiv:2108.07732. Retrieved from https:\/\/arxiv.org\/abs\/2108.07732"},{"key":"e_1_3_2_15_2","volume-title":"The Basics of Item Response Theory","author":"Baker Frank B.","year":"2001","unstructured":"Frank B. Baker. 2001. The Basics of Item Response Theory. ERIC."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-short.15"},{"key":"e_1_3_2_17_2","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https:\/\/arxiv.org\/abs\/2107.03374"},{"issue":"1","key":"e_1_3_2_18_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3143561","article-title":"Metamorphic testing: A review of challenges and opportunities","volume":"51","author":"Chen Tsong Yueh","year":"2018","unstructured":"Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, T. H. Tse, and Zhi Quan Zhou. 2018. Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys 51, 1 (2018), 1\u201327.","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_2_19_2","unstructured":"Xinyun Chen Maxwell Lin Nathanael Sch\u00e4rli and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv:2304.05128. Retrieved from https:\/\/arxiv.org\/abs\/2304.05128"},{"key":"e_1_3_2_20_2","first-page":"1013","volume-title":"Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics","author":"Chen Yu","year":"2019","unstructured":"Yu Chen, Telmo Silva Filho, Ricardo B. Prudencio, Tom Diethe, and Peter Flach. 2019. \\(\\beta^{3}\\) -IRT: A new item response model and its applications. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 1013\u20131021."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3661167.3661221"},{"key":"e_1_3_2_22_2","unstructured":"DeepSeek-AI. 2024. DeepSeek-V3 technical report. arXiv:2412.19437. Retrieved from https:\/\/arxiv.org\/abs\/2412.19437"},{"key":"e_1_3_2_23_2","unstructured":"Yihe Deng Weitong Zhang Zixiang Chen and Quanquan Gu. 2023. Rephrase and respond: Let large language models ask better questions for themselves. arXiv:2311.04205. Retrieved from https:\/\/arxiv.org\/abs\/2311.04205"},{"key":"e_1_3_2_24_2","unstructured":"Xueying Du Mingwei Liu Kaixin Wang Hanlin Wang Junwei Liu Yixuan Chen Jiayi Feng Chaofeng Sha Xin Peng and Yiling Lou. 2023. ClassEval: A manually-crafted benchmark for evaluating LLMS on class-level code generation. arXiv:2308.01861. Retrieved from https:\/\/arxiv.org\/abs\/2308.01861"},{"key":"e_1_3_2_25_2","unstructured":"Chrisantha Fernando Dylan Banarse Henryk Michalewski Simon Osindero and Tim Rockt\u00e4schel. 2023. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv:2309.16797. Retrieved from https:\/\/arxiv.org\/abs\/2309.16797"},{"key":"e_1_3_2_26_2","unstructured":"Manuel Ferreira-Junior Jessica T. S. Reinaldo Telmo M. Silva Filho Eufrasio A. Lima Neto and Ricardo B. C. Prudencio. 2023. \\(\\beta^{4}\\) -IRT: A new \\(\\beta^{3}\\) -IRT with enhanced discrimination estimation. arXiv:2303.17731. Retrieved from https:\/\/arxiv.org\/abs\/2303.17731"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1037\/a0024338"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-023-42227-6"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1177\/0013164493053001007"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-emnlp.679"},{"key":"e_1_3_2_31_2","first-page":"22","article-title":"Planning poker or how to avoid analysis paralysis while release planning","volume":"3","author":"Grenning James","year":"2002","unstructured":"James Grenning. 2002. Planning poker or how to avoid analysis paralysis while release planning. Hawthorn Woods: Renaissance Software Consulting 3 (2002), 22\u201323.","journal-title":"Hawthorn Woods: Renaissance Software Consulting"},{"key":"e_1_3_2_32_2","unstructured":"Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv:2203.05794. Retrieved from https:\/\/arxiv.org\/abs\/2203.05794"},{"key":"e_1_3_2_33_2","unstructured":"Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Y. Wu Y. K. Li et al. 2024. DeepSeek-Coder: When the large language model meets programming\u2014The rise of code intelligence. arXiv:2401.14196. Retrieved from https:\/\/arxiv.org\/abs\/2401.14196"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3580688"},{"key":"e_1_3_2_35_2","first-page":"507","volume-title":"Proceedings of the 6th Conference on Machine Translation","author":"Hanna Michael","year":"2021","unstructured":"Michael Hanna and Ond\u0159ej Bojar. 2021. A fine-grained analysis of BERTScore. In Proceedings of the 6th Conference on Machine Translation, 507\u2013517."},{"key":"e_1_3_2_36_2","unstructured":"Dan Hendrycks Steven Basart Saurav Kadavath Mantas Mazeika Akul Arora Ethan Guo Collin Burns Samir Puranik Horace He Dawn Song et al. 2021. Measuring coding challenge competence with apps. arXiv:2105.09938. Retrieved from https:\/\/arxiv.org\/abs\/2105.09938"},{"key":"e_1_3_2_37_2","first-page":"65","article-title":"A simple sequentially rejective multiple test procedure","author":"Holm Sture","year":"1979","unstructured":"Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6 (1979), 65\u201370.","journal-title":"Scandinavian Journal of Statistics"},{"key":"e_1_3_2_38_2","unstructured":"HuggingFace. 2024. Hugging Face\u2014huggingface.co. Retrieved May 27 2024 from https:\/\/huggingface.co\/"},{"key":"e_1_3_2_39_2","unstructured":"Naman Jain King Han Alex Gu Wen-Ding Li Fanjia Yan Tianjun Zhang Sida Wang Armando Solar-Lezama Koushik Sen and Ion Stoica. 2024. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. arXiv:2403.07974. Retrieved from https:\/\/arxiv.org\/abs\/2403.07974"},{"key":"e_1_3_2_40_2","unstructured":"Juyong Jiang Fan Wang Jiasi Shen Sungju Kim and Sunghun Kim. 2024. A survey on large language models for code generation. arXiv:2406.00515. Retrieved from https:\/\/arxiv.org\/abs\/2406.00515"},{"key":"e_1_3_2_41_2","volume-title":"Proceedings of the 12th International Conference on Learning Representations","author":"Jimenez Carlos E.","year":"2024","unstructured":"Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can language models resolve real-world GitHub issues? In Proceedings of the 12th International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=VTF8yNQM66"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3613892"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/30.1-2.81"},{"key":"e_1_3_2_44_2","doi-asserted-by":"crossref","unstructured":"Jan H. Klemmer Stefan Albert Horstmann Nikhil Patnaik Cordelia Ludden Cordell Burton Jr Carson Powers Fabio Massacci Akond Rahman Daniel Votipka Heather Richter Lipford et al. 2024. Using AI assistants in software development: A qualitative study on security practices and concerns. arXiv:2405.06371. Retrieved from https:\/\/arxiv.org\/abs\/2405.06371","DOI":"10.1145\/3658644.3690283"},{"key":"e_1_3_2_45_2","unstructured":"Bonan Kou Shengmai Chen Zhijie Wang Lei Ma and Tianyi Zhang. 2023. Is model attention aligned with human attention? An empirical study on large language models for code generation. arXiv:2306.01220. Retrieved from https:\/\/arxiv.org\/abs\/2306.01220"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1062"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.emnlp-main.764"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00085"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.abq1158"},{"key":"e_1_3_2_50_2","unstructured":"Changshu Liu Shizhuo Zhang Ali Reza Ibrahimzada and Reyhaneh Jabbarvand. n.d. Can large language models reason about code? arXiv:2402.09664v4. Retrieved from https:\/\/arxiv.org\/abs\/2402.09664v4"},{"key":"e_1_3_2_51_2","unstructured":"Fang Liu Yang Liu Lin Shi Houkun Huang Ruifeng Wang Zhen Yang Li Zhang Zhongqi Li and Yuchi Ma. 2024. Exploring and evaluating hallucinations in LLM-powered code generation. arXiv:2404.00971. Retrieved from https:\/\/arxiv.org\/abs\/2404.00971"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.52202\/075280-0943"},{"key":"e_1_3_2_53_2","unstructured":"Anton Lozhkov Raymond Li Loubna Ben Allal Federico Cassano Joel Lamy-Poirier Nouamane Tazi Ao Tang Dmytro Pykhtar et al. 2024. StarCoder 2 and the Stack v2: The next generation. arXiv:2402.19173. Retrieved from https:\/\/arxiv.org\/abs\/2402.19173"},{"key":"e_1_3_2_54_2","unstructured":"Shuai Lu Daya Guo Shuo Ren Junjie Huang Alexey Svyatkovskiy Ambrosio Blanco Colin B. Clement Dawn Drain et al. 2021. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. arXiv:2102.04664. Retrieved from https:\/\/arxiv.org\/abs\/2102.04664"},{"key":"e_1_3_2_55_2","unstructured":"Vahid Majdinasab Michael Joshua Bishop Shawn Rasheed Arghavan Moradidakhel Amjed Tahir and Foutse Khomh. 2023. Assessing the security of GitHub Copilot generated code\u2014A targeted replication study. arXiv:2311.11177. Retrieved from https:\/\/arxiv.org\/abs\/2311.11177"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00181"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.21105\/joss.00205"},{"key":"e_1_3_2_58_2","unstructured":"Leland McInnes John Healy and James Melville. 2018. UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv:1802.03426. Retrieved from https:\/\/arxiv.org\/abs\/1802.03426"},{"key":"e_1_3_2_59_2","unstructured":"Timothy R. McIntosh Teo Susnjak Nalin Arachchilage Tong Liu Paul Watters and Malka N. Halgamuge. 2024. Inadequacies of large language model benchmarks in the era of generative artificial intelligence. arXiv:2402.09880. Retrieved from https:\/\/arxiv.org\/abs\/2402.09880"},{"key":"e_1_3_2_60_2","doi-asserted-by":"crossref","unstructured":"Moran Mizrahi Guy Kaplan Dan Malkin Rotem Dror Dafna Shahaf and Gabriel Stanovsky. 2024. State of what art? A call for multi-prompt LLM evaluation. arXiv:2401.00595. Retrieved from https:\/\/arxiv.org\/abs\/2401.00595","DOI":"10.1162\/tacl_a_00681"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111734"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1080\/02664769723918"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524842.3528470"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3697010"},{"key":"e_1_3_2_65_2","unstructured":"Shuo Ren Daya Guo Shuai Lu Long Zhou Shujie Liu Duyu Tang Neel Sundaresan Ming Zhou Ambrosio Blanco and Shuai Ma. 2020. CodeBLEU: A method for automatic evaluation of code synthesis. arXiv:2009.10297. Retrieved from https:\/\/arxiv.org\/abs\/2009.10297"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.346"},{"key":"e_1_3_2_67_2","unstructured":"Baptiste Roziere Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat Xiaoqing Ellen Tan Yossi Adi Jingyu Liu Tal Remez J\u00e9r\u00e9my Rapin et al. 2023. Code llama: Open foundation models for code. arXiv:2308.12950. Retrieved from https:\/\/arxiv.org\/abs\/2308.12950"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC61105.2024.00096"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1213\/ANE.0000000000002864"},{"key":"e_1_3_2_70_2","doi-asserted-by":"crossref","unstructured":"Fritz W. Scholz and Michael A. Stephens. 1987. K-sample Anderson\u2013Darling tests. Journal of the American Statistical Association 82 399 (1987) 918\u2013924.","DOI":"10.1080\/01621459.1987.10478517"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.eval4nlp-1.3"},{"key":"e_1_3_2_72_2","unstructured":"Xinyue Shen Zeyuan Chen Michael Backes and Yang Zhang. 2023. In ChatGPT we trust? Measuring and characterizing the reliability of ChatGPT. arXiv:2304.08979. Retrieved from https:\/\/arxiv.org\/abs\/2304.08979"},{"key":"e_1_3_2_73_2","unstructured":"Mohammed Latif Siddiq Simantika Dristi Joy Saha and Joanna C. S. Santos. 2024. Quality assessment of prompts used in code generation. arXiv:2404.10155. Retrieved from https:\/\/arxiv.org\/abs\/2404.10155"},{"key":"e_1_3_2_74_2","doi-asserted-by":"crossref","unstructured":"Florian Tambon Arghavan Moradi Dakhel Amin Nikanjam Foutse Khomh Michel C. Desmarais and Giuliano Antoniol. 2024. Bugs in large language models generated code. arXiv:2403.08937. Retrieved from https:\/\/arxiv.org\/abs\/2403.08937","DOI":"10.1007\/s10664-025-10614-4"},{"key":"e_1_3_2_75_2","unstructured":"Gemma Team Thomas Mesnard Cassidy Hardin Robert Dadashi Surya Bhupatiraju Shreya Pathak Laurent Sifre Morgane Rivi\u00e8re Mihir Sanjay Kale Juliette Love et al. 2024. Gemma: Open models based on Gemini research and technology. arXiv:2403.08295. Retrieved from https:\/\/arxiv.org\/abs\/2403.08295"},{"key":"e_1_3_2_76_2","doi-asserted-by":"crossref","unstructured":"Rebeka T\u00f3th Tamas Bisztray and L\u00e1szl\u00f3 Erdodi. 2024. LLMs in web-development: Evaluating LLM-generated PHP code unveiling vulnerabilities and limitations. arXiv:2404.14459. Retrieved from https:\/\/arxiv.org\/abs\/2404.14459","DOI":"10.1007\/978-3-031-68738-9_34"},{"key":"e_1_3_2_77_2","unstructured":"Clara Vania Phu Mon Htut William Huang Dhara Mungra Richard Yuanzhe Pang Jason Phang Haokun Liu Kyunghyun Cho and Samuel R. Bowman. 2021. Comparing test sets with item response theory. arXiv:2106.00840. Retrieved from https:\/\/arxiv.org\/abs\/2106.00840"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511618765.006"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-3984.1987.tb00274.x"},{"key":"e_1_3_2_80_2","unstructured":"Shiqi Wang Zheng Li Haifeng Qian Chenghao Yang Zijian Wang Mingyue Shang Varun Kumar Samson Tan Baishakhi Ray Parminder Bhatia et al. 2022. ReCode: Robustness evaluation of code generation models. arXiv:2212.10264. Retrieved from https:\/\/arxiv.org\/abs\/2212.10264"},{"key":"e_1_3_2_81_2","unstructured":"Xiaoyin Wang and Dakai Zhu. 2024. Validating LLM-generated programs with metamorphic prompt testing. arXiv:2406.06864. Retrieved from https:\/\/arxiv.org\/abs\/2406.06864"},{"key":"e_1_3_2_82_2","unstructured":"Yizhong Wang Yeganeh Kordi Swaroop Mishra Alisa Liu Noah A. Smith Daniel Khashabi and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language models with self-generated instructions. arXiv:2212.10560. Retrieved from https:\/\/arxiv.org\/abs\/2212.10560"},{"key":"e_1_3_2_83_2","doi-asserted-by":"crossref","unstructured":"Yue Wang Weishi Wang Shafiq Joty and Steven C. H. Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv:2109.00859. Retrieved from https:\/\/arxiv.org\/abs\/2109.00859","DOI":"10.18653\/v1\/2021.emnlp-main.685"},{"key":"e_1_3_2_84_2","doi-asserted-by":"crossref","unstructured":"Lucas Weber Elia Bruni and Dieuwke Hupkes. 2023. Mind the instructions: A holistic evaluation of consistency and interactions in prompt-based learning. arXiv:2310.13486. Retrieved from https:\/\/arxiv.org\/abs\/2310.13486","DOI":"10.18653\/v1\/2023.conll-1.20"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.52202\/068431-1800"},{"key":"e_1_3_2_86_2","unstructured":"Yuxiang Wei Zhe Wang Jiawei Liu Yifeng Ding and Lingming Zhang. 2023. Magicoder: Source code is all you need. arXiv:2312.02120. Retrieved from https:\/\/arxiv.org\/abs\/2312.02120"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4612-4380-9_16"},{"key":"e_1_3_2_88_2","unstructured":"Chunqiu Steven Xia Yinlin Deng and Lingming Zhang. 2024. Top leaderboard ranking = top coding proficiency always? EvoEval: Evolving coding benchmarks via LLM. arXiv:2403.19114. Retrieved from https:\/\/arxiv.org\/abs\/2403.19114"},{"key":"e_1_3_2_89_2","unstructured":"An Yang Baosong Yang Binyuan Hui Bo Zheng Bowen Yu Chang Zhou Chengpeng Li Chengyuan Li Dayiheng Liu Fei Huang et al. 2024. Qwen2 technical report. arXiv:2407.10671. Retrieved from https:\/\/arxiv.org\/abs\/2407.10671"},{"key":"e_1_3_2_90_2","unstructured":"Burak Yeti\u015ftiren I\u015f\u0131k \u00d6zsoy Miray Ayerdem and Eray T\u00fcz\u00fcn. 2023. Evaluating the code quality of ai-assisted code generation tools: An empirical study on GitHub Copilot Amazon CodeWhisperer and ChatGPT. arXiv:2304.10778. Retrieved from https:\/\/arxiv.org\/abs\/2304.10778"},{"key":"e_1_3_2_91_2","doi-asserted-by":"crossref","unstructured":"Hao Yu Bo Shen Dezhi Ran Jiaxin Zhang Qi Zhang Yuchi Ma Guangtai Liang Ying Li Tao Xie and Qianxiang Wang. 2023. CoderEval: A benchmark of pragmatic code generation with generative pre-trained models. arXiv:2302.00288v1. Retrieved from https:\/\/arxiv.org\/abs\/2302.00288v1","DOI":"10.1145\/3597503.3623316"},{"key":"e_1_3_2_92_2","article-title":"An application of item response theory to psychological test development","volume":"29","author":"Zanon Cristian","year":"2016","unstructured":"Cristian Zanon, Claudio S. Hutz, Hanwook Henry Yoo, and Ronald K. Hambleton. 2016. An application of item response theory to psychological test development. Psicologia: Reflex\u00e3o e Cr\u00edtica 29 (2016), Article 18.","journal-title":"Psicologia: Reflex\u00e3o e Cr\u00edtica"},{"key":"e_1_3_2_93_2","unstructured":"Tianyi Zhang Varsha Kishore Felix Wu Kilian Q. Weinberger and Yoav Artzi. 2019. BERTScore: Evaluating text generation with BERT. arXiv:1904.09675. Retrieved from https:\/\/arxiv.org\/abs\/1904.09675"},{"key":"e_1_3_2_94_2","unstructured":"Yan Zhuang Qi Liu Yuting Ning Weizhe Huang Rui Lv Zhenya Huang Guanhao Zhao Zheng Zhang Qingyang Mao Shijin Wang et al. 2023. Efficiently measuring the cognitive ability of LLMs: An adaptive testing perspective. arXiv:2306.10512. Retrieved from https:\/\/arxiv.org\/abs\/2306.10512"},{"key":"e_1_3_2_95_2","unstructured":"Yan Zhuang Qi Liu Yuting Ning Weizhe Huang Zachary A. Pardos Patrick C. Kyllonen Jiyun Zu Qingyang Mao Rui Lv Zhenya Huang et al. 2024. From static benchmarks to adaptive testing: Psychometrics in AI evaluation. arXiv:2306.10512. Retrieved from https:\/\/arxiv.org\/abs\/2306.10512"},{"key":"e_1_3_2_96_2","first-page":"1","volume-title":"Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering","author":"Zid Cyrine","year":"2024","unstructured":"Cyrine Zid, Fiorella Zampetti, Giuliano Antoniol, and Massimiliano Di Penta. 2024. A study on the pythonic functional constructs\u2019 understandability. In Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering, 1\u201313."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3773285","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T15:04:21Z","timestamp":1783782261000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3773285"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,11]]},"references-count":95,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2026,7,31]]}},"alternative-id":["10.1145\/3773285"],"URL":"https:\/\/doi.org\/10.1145\/3773285","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,11]]},"assertion":[{"value":"2025-03-14","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-21","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-07-11","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}