{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,10]],"date-time":"2026-05-10T05:40:55Z","timestamp":1778391655108,"version":"3.51.4"},"reference-count":51,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2026,2,10]],"date-time":"2026-02-10T00:00:00Z","timestamp":1770681600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>The recent use of large language models (LLMs) in code generation and code summarization tasks has been widely adopted by the software engineering community. New LLMs are emerging regularly with improved functionalities, efficiency, and expanding data that allow models to learn more effectively. The lack of guidelines for selecting the right LLMs for coding tasks makes the selection a subjective choice by developers rather than a choice built on code complexity, code correctness, and linguistic similarity analysis. This research investigates the use of machine learning classification and ranking methods to select the best-suited open-source LLMs for code generation and code summarization tasks. This work conducts a comparison experiment on four open-source LLMs (Mistral, CodeLlama, Gemma 2, and Phi-3) and uses the MBPP coding question dataset to analyze code-generated outputs in terms of code complexity, maintainability, cyclomatic complexity, code structure, and LLM perplexity by collecting these as a set of features. An SVM classification problem is conducted on the highest correlated feature pairs, where the models are evaluated through performance metrics, including accuracy, area under the ROC curve (AUC), precision, recall, and F1 scores. The RankNet ranking methodology is used to evaluate code summarization model capabilities by measuring ROUGE and BERTScore accuracies between LLM code-generated summaries and the coding questions used from the dataset. The study results show a maximum accuracy of 49% for the code generation experiment, with the highest AUC score reaching 86% among the top four correlated feature pairs. The highest precision score reached is 90%, and the recall score reached up to 92%. Code summarization experiment results show Gemma 2 scored a 1.93 RankNet win probability score, and represented the highest ranking reached among other models. The phi3 model was the second-highest ranking with a 1.66 score. The research highlights the potential of machine learning to select LLMs based on coding metrics and paves the way for advancements in terms of accuracy, dataset diversity, and exploring other machine learning algorithms for other researchers.<\/jats:p>","DOI":"10.3390\/computers15020119","type":"journal-article","created":{"date-parts":[[2026,2,10]],"date-time":"2026-02-10T17:10:52Z","timestamp":1770743452000},"page":"119","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Evaluating LLMs for Source Code Generation and Summarization Using Machine Learning Classification and Ranking"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-6564-5616","authenticated-orcid":false,"given":"Hussain","family":"Mahfoodh","sequence":"first","affiliation":[{"name":"Independent Researcher, Manama, Bahrain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6527-5070","authenticated-orcid":false,"given":"Mustafa","family":"Hammad","sequence":"additional","affiliation":[{"name":"Department of Software Engineering, Mutah University, Mutah 61710, Jordan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2232-9387","authenticated-orcid":false,"given":"Bassam A. Y.","family":"Alqaralleh","sequence":"additional","affiliation":[{"name":"College of Business Administration, American University of the Middle East, Egaila 54200, Kuwait"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4900-497X","authenticated-orcid":false,"given":"Aymen I.","family":"Zreikat","sequence":"additional","affiliation":[{"name":"College of Engineering and Technology, American University of the Middle East, Egaila 54200, Kuwait"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2026,2,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Xu, J., Zhang, H., Yang, Y., Yang, L., Cheng, Z., Lyu, J., Liu, B., Zhou, X., Bacchelli, A., and Chiam, Y.K. (2024). One Size Does Not Fit All: Investigating Efficacy of Perplexity in Detecting LLM-Generated Code. ACM Trans. Softw. Eng. Methodol.","DOI":"10.1145\/3748506"},{"key":"ref_2","unstructured":"Jiang, J., Wang, F., Shen, J., Kim, S., and Kim, S. (2024). A survey on large language models for code generation. arXiv."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Shao, Y., Huang, Y., Shen, J., Ma, L., Su, T., and Wan, C. (2024). Are LLMs Correctly Integrated into Software Systems?. arXiv.","DOI":"10.1109\/ICSE55347.2025.00204"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Haque, M.A. (2025). Llms: A game-changer for software engineers?. Benchcouncil Trans. Benchmarks Stand. Eval., 5.","DOI":"10.1016\/j.tbench.2025.100204"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Das, J.K., Mondal, S., and Roy, C.K. (2025). Why Do Developers Engage with ChatGPT in Issue-Tracker? Investigating Usage and Reliance on ChatGPT-Generated Code. Proceedings of the 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE.","DOI":"10.1109\/SANER64311.2025.00015"},{"key":"ref_6","unstructured":"Tian, H., Lu, W., Li, T.O., Tang, X., Cheung, S.C., Klein, J., and Bissyand\u00e9, T.F. (2023). Is ChatGPT the ultimate programming assistant\u2013how far is it?. arXiv."},{"key":"ref_7","first-page":"21558","article-title":"Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation","volume":"36","author":"Liu","year":"2023","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_8","first-page":"1","article-title":"Refining chatgpt-generated code: Characterizing and mitigating code quality issues","volume":"33","author":"Liu","year":"2024","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Sepidband, M., Taherkhani, H., Wang, S., and Hemmati, H. (2025). Enhancing LLM-Based Code Generation with Complexity Metrics: A Feedback-Driven Approach. arXiv.","DOI":"10.1109\/COMPSAC65507.2025.00178"},{"key":"ref_10","unstructured":"Khan, M.F.A., Ramsdell, M., Falor, E., and Karimi, H. (2023). Assessing the promise and pitfalls of chatgpt for automated code generation. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Hu, W., Duan, J., Wei, C., Zhang, L., Zhang, Y., and Xu, K. (2025). Dynacode: A dynamic complexity-aware code benchmark for evaluating large language models in code generation. arXiv.","DOI":"10.18653\/v1\/2025.findings-acl.1133"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Biswas, S., Singh, G., Maiti, B., Ezugwu, A.E.S., Saleem, K., Smerat, A., Abualigah, L., and Bera, U.K. (2025). Integrating Differential Evolution into Gazelle Optimization for advanced global optimization and engineering applications. Comput. Methods Appl. Mech. Eng., 434.","DOI":"10.1016\/j.cma.2024.117588"},{"key":"ref_13","unstructured":"Mahmud, T., Duan, B., Pasareanu, C., and Yang, G. (2025). Enhancing llm code generation with ensembles: A similarity-based selection approach. arXiv."},{"key":"ref_14","unstructured":"Huynh, N., and Lin, B. (2025). Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applications. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Mostafa, A., Nahid, R.A., and Mulder, S. (2025). How Different Tokenization Algorithms Impact LLMs and Transformer Models for Binary Code Analysis. arXiv.","DOI":"10.14722\/bar.2025.23013"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Xu, Z., and Sheng, V.S. (2024, January 20\u201327). Detecting AI-generated code assignments using perplexity of large language models. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.","DOI":"10.1609\/aaai.v38i21.30361"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Wang, Q., Parnin, C., and Orso, A. (2015, January 13\u201317). Evaluating the usefulness of ir-based fault localization techniques. Proceedings of the 2015 International Symposium on Software Testing and Analysis, Baltimore, MD, USA.","DOI":"10.1145\/2771783.2771797"},{"key":"ref_18","unstructured":"Bo, L., Ji, W., Sun, X., Zhang, T., Wu, X., and Wei, Y. (November, January 27). Chatbr: Automated assessment and improvement of bug report quality using chatgpt. Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering, Sacramento, CA, USA."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Acharya, J., and Ginde, G. (2025). Can We Enhance Bug Report Quality Using LLMs?: An Empirical Study of LLM-Based Bug Report Generation. arXiv.","DOI":"10.1145\/3756681.3756995"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Sureka, A., and Indukuri, K.V. (2010, January 22\u201323). Linguistic analysis of bug report titles with respect to the dimension of bug importance. Proceedings of the Third Annual ACM Bangalore Conference, Bangalore, India.","DOI":"10.1145\/1754288.1754297"},{"key":"ref_21","unstructured":"Kang, S., Milliken, L., and Yoo, S. (2024). Identifying inaccurate descriptions in llm-generated code comments via test execution. arXiv."},{"key":"ref_22","unstructured":"Kumar, A., Haiduc, S., Das, P.P., and Chakrabarti, P.P. (2024). LLMs as Evaluators: A Novel Approach to Evaluate Bug Report Summarization. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Colavito, G. (2025). Foundation Models for Automatic Issue Labeling. Proceedings of the 2025 IEEE\/ACM 47th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), IEEE.","DOI":"10.1109\/ICSE-Companion66252.2025.00038"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Shivashankar, K., Haugerud, K.M., and Martini, A. (2025). Enhancing Task Prioritization in Software Development Issues Tracking system. J. Softw. Evol. Process, 37.","DOI":"10.1002\/smr.70068"},{"key":"ref_25","unstructured":"Chen, Z., Nava-Camal, V., Suleiman, A., Tang, Y., Hou, D., and Shang, W. (2025). An Empirical Study on the Capability of LLMs in Decomposing Bug Reports. arXiv."},{"key":"ref_26","unstructured":"Ciepielowski, L. (2024). An LLM-Based Tool for Knowledge Retrieval from (Heterogeneous) Issue Tracking Systems. [Ph.D. Thesis, Universitat Hamburg]."},{"key":"ref_27","unstructured":"Plein, L., and Bissyand\u00e9, T.F. (2023). Can LLMs demystify bug reports?. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Sun, W., Miao, Y., Li, Y., Zhang, H., Fang, C., Liu, Y., Deng, G., Liu, Y., and Chen, Z. (2024). Source code summarization in the era of large language models. arXiv.","DOI":"10.1109\/ICSE55347.2025.00034"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"3205","DOI":"10.1109\/TSE.2025.3595283","article-title":"Can large language models serve as evaluators for code summarization?","volume":"51","author":"Wu","year":"2025","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_30","first-page":"221","article-title":"How to Choose the Best AI LLM: A Guide to Navigating the Diversity of Models","volume":"10","author":"Apollinaire","year":"2024","journal-title":"J. Inf. Syst. Eng. Manag."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Cheng, J., Liu, F., Wu, C., and Zhang, L. (2025, January 20\u201322). Adaptivellm: A framework for selecting optimal cost-efficient llm for code-generation based on cot length. Proceedings of the 16th International Conference on Internetware, Trondheim, Norway.","DOI":"10.1145\/3755881.3755925"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Arora, C., Sayeed, A.I., Licorish, S., Wang, F., and Treude, C. Optimizing LLMs for Code Generation: Which Hyperparameter Settings Yield the Best Results?. Proceedings of the 2024 31st Asia-Pacific Software Engineering Conference (APSEC).","DOI":"10.1109\/APSEC65559.2024.00039"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Fan, Z., Ruan, H., Mechtaev, S., and Roychoudhury, A. (2024, January 16\u201320). Oracle-guided Program Selection from Large Language Models. Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, Vienna, Austria.","DOI":"10.1145\/3650212.3680308"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Su, J., Deng, L., Wen, C., Qin, S., and Tian, C. (2024). Cfstra: Enhancing configurable program analysis through llm-driven strategy selection based on code features. Proceedings of the International Symposium on Theoretical Aspects of Software Engineering, Springer.","DOI":"10.1007\/978-3-031-64626-3_22"},{"key":"ref_35","unstructured":"Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., and Le, Q. (2021). Program synthesis with large language models. arXiv."},{"key":"ref_36","unstructured":"Polozov, A. (2025, November 23). Sanitized-Mbpp, Available online: https:\/\/github.com\/google-research\/google-research\/blob\/master\/mbpp\/sanitized-mbpp.json."},{"key":"ref_37","unstructured":"Mahfoodh, H. (2025, November 23). Hussain Mahfodh Github Page, Available online: https:\/\/github.com\/hmahfoodh\/AI_ML_Data_Engineering\/tree\/main\/LLM."},{"key":"ref_38","unstructured":"Ollama Team (2026, January 28). Mistral Model Library Entry. Available online: https:\/\/ollama.com\/library\/mistral."},{"key":"ref_39","unstructured":"Ollama Team (2026, January 28). Gemma 2 Model Library Entry. Available online: https:\/\/ollama.com\/library\/gemma2."},{"key":"ref_40","unstructured":"Ollama Team (2026, January 28). Phi-3 Model Library Entry. Available online: https:\/\/ollama.com\/library\/phi3."},{"key":"ref_41","unstructured":"Ollama Team (2026, January 28). CodeLlama Model Library Entry. Available online: https:\/\/ollama.com\/library\/codellama."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"19","DOI":"10.12785\/ijcnt\/040103","article-title":"Comparison of software complexity metrics","volume":"4","author":"Khan","year":"2016","journal-title":"Int. J. Comput. Netw. Technol."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"3880","DOI":"10.1007\/s10489-021-02588-9","article-title":"Optimal distributed parallel algorithms for deep learning framework tensorflow","volume":"52","author":"Xie","year":"2022","journal-title":"Appl. Intell."},{"key":"ref_44","unstructured":"Microsoft (2025, November 08). CodeGPT-small-py on Hugging Face Model Hub. Available online: https:\/\/huggingface.co\/microsoft\/CodeGPT-small-py."},{"key":"ref_45","unstructured":"Saidi, I., Alshammari, A., Alosaimi, A., and Alqahtani, F. (2020). The Impact of Design on the Performance of Large Language Models. Inf. Softw. Technol., 123."},{"key":"ref_46","unstructured":"Hugging Face (2025, November 08). distilgpt2 on Hugging Face Model Hub. Available online: https:\/\/huggingface.co\/distilbert\/distilgpt2."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Mavroudis, V. (2024). LangChain. Preprints.","DOI":"10.20944\/preprints202411.0566.v1"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Zhao, G., Liu, Y., Zhang, W., and Wang, Y. (2018, January 13\u201315). TFIDF based feature words extraction and topic modeling for short text. Proceedings of the 2018 2nd International Conference on Management Engineering, Software Engineering and Service Sciences, Wuhan, China.","DOI":"10.1145\/3180374.3181354"},{"key":"ref_49","unstructured":"Fu, Z., Song, W., Wang, Y., Wu, X., Zheng, Y., Zhang, Y., Xu, D., Wei, X., Xu, T., and Zhao, X. (2025). Sliding Window Attention Training for Efficient Large Language Models. arXiv."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Wang, Z., Liang, Z., Shao, Z., Ma, Y., Dai, H., Chen, B., Mao, L., Lei, C., Ding, Y., and Li, H. (2025). InfoGain-RAG: Boosting Retrieval-Augmented Generation via Document Information Gain-based Reranking and Filtering. arXiv.","DOI":"10.18653\/v1\/2025.emnlp-main.365"},{"key":"ref_51","unstructured":"Han, I., Jayaram, R., Karbasi, A., Mirrokni, V., Woodruff, D.P., and Zandieh, A. (2023). Hyperattention: Long-context attention in near-linear time. arXiv."}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/15\/2\/119\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,10]],"date-time":"2026-02-10T17:14:34Z","timestamp":1770743674000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/15\/2\/119"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,10]]},"references-count":51,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2026,2]]}},"alternative-id":["computers15020119"],"URL":"https:\/\/doi.org\/10.3390\/computers15020119","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,10]]}}}