{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,17]],"date-time":"2026-10-17T06:32:51Z","timestamp":1792218771057,"version":"4.4.0"},"reference-count":38,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2021,2,1]],"date-time":"2021-02-01T00:00:00Z","timestamp":1612137600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["KAKENHI; Grant no. 19K12252"],"award-info":[{"award-number":["KAKENHI; Grant no. 19K12252"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Symmetry"],"abstract":"<jats:p>Programming is a vital skill in computer science and engineering-related disciplines. However, developing source code is an error-prone task. Logical errors in code are particularly hard to identify for both students and professionals, and a single error is unexpected to end-users. At present, conventional compilers have difficulty identifying many of the errors (especially logical errors) that can occur in code. To mitigate this problem, we propose a language model for evaluating source codes using a bidirectional long short-term memory (BiLSTM) neural network. We trained the BiLSTM model with a large number of source codes with tuning various hyperparameters. We then used the model to evaluate incorrect code and assessed the model\u2019s performance in three principal areas: source code error detection, suggestions for incorrect code repair, and erroneous code classification. Experimental results showed that the proposed BiLSTM model achieved 50.88% correctness in identifying errors and providing suggestions. Moreover, the model achieved an F-score of approximately 97%, outperforming other state-of-the-art models (recurrent neural networks (RNNs) and long short-term memory (LSTM)).<\/jats:p>","DOI":"10.3390\/sym13020247","type":"journal-article","created":{"date-parts":[[2021,2,1]],"date-time":"2021-02-01T11:40:48Z","timestamp":1612179648000},"page":"247","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":97,"title":["A Bidirectional LSTM Language Model for Code Evaluation and Repair"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9368-7638","authenticated-orcid":false,"given":"Md. Mostafizer","family":"Rahman","sequence":"first","affiliation":[{"name":"Department of Computer and Information Systems, School of Computer Science and Engineering, The University of Aizu, Aizu-Wakamatsu, Fukushima 965-8580, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yutaka","family":"Watanobe","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Systems, School of Computer Science and Engineering, The University of Aizu, Aizu-Wakamatsu, Fukushima 965-8580, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Keita","family":"Nakamura","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Systems, School of Computer Science and Engineering, The University of Aizu, Aizu-Wakamatsu, Fukushima 965-8580, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,2,1]]},"reference":[{"key":"ref_1","first-page":"42","article-title":"Understanding Computer Programming as a Literacy","volume":"1","author":"Vee","year":"2013","journal-title":"LICS Lit. Compos. Stud."},{"key":"ref_2","first-page":"131","article-title":"Competitive learning in informatics: The UVa online judge experience","volume":"2","author":"Revilla","year":"2008","journal-title":"Olymp. Inform."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1109\/TLT.2017.2723389","article-title":"Jutge.org: Characteristics and Experiences","volume":"11","author":"Petit","year":"2018","journal-title":"IEEE Trans. Learn. Technol."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Bez, J.L., Tonin, N.A., and Rodegheri, P.R. (2014, January 22\u201324). URI Online Judge Academic: A tool for algorithms and programming classes. Proceedings of the 2014 9th International Conference on Computer Science Education, Vancouver, BC, Canada.","DOI":"10.1109\/ICCSE.2014.6926445"},{"key":"ref_5","unstructured":"Watanobe, Y. (2020, October 20). Aizu Online Judge. Available online: https:\/\/onlinejudge.u-aizu.ac.jp."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"81154","DOI":"10.1109\/ACCESS.2020.2990980","article-title":"Building a comprehensive automated programming assessment system","volume":"8","year":"2020","journal-title":"IEEE Access"},{"key":"ref_7","first-page":"10","article-title":"A Practical Approach for Detecting Logical Error in Object Oriented Environment","volume":"7","author":"Ghassan","year":"2017","journal-title":"World Comput. Sci. Inf. Technol. J."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Al-Ashwal, D., Al-Sewari, E.Z., and Al-Shargabi, A.A. (2018, January 28\u201330). A CASE Tool for JAVA Programs Logical Errors Detection: Static and Dynamic Testing. Proceedings of the 2018 International Arab Conference on Information Technology (ACIT), Werdanye, Lebanon.","DOI":"10.1109\/ACIT.2018.8672669"},{"key":"ref_9","unstructured":"Seo, H., Sadowski, C., Elbaum, S., Aftandilian, E., and Bowdidge, R. (June, January 31). Programmers\u2019 build errors: A case study (at google). Proceedings of the 36th International Conference on Software Engineering (ICSE\u201914), Hyderabad, India."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1007\/s13748-016-0092-2","article-title":"Data mining for software engineering and humans in the loop","volume":"5","author":"Minku","year":"2016","journal-title":"Progress Artif. Intell."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3105906","article-title":"Automatic software repair: A bibliography","volume":"51","author":"Monperrus","year":"2018","journal-title":"ACM Comput. Surv."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1016\/j.jss.2007.03.004","article-title":"Examining the significance of high-level programming features in source code author classification","volume":"81","author":"Frantzeskou","year":"2008","journal-title":"J. Syst. Softw."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Mou, L., Li, G., Zhang, L., Wang, T., and Jin, Z. (2016, January 12\u201317). Convolutional neural networks over tree structures for programming language processing. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.","DOI":"10.1609\/aaai.v30i1.10139"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"White, M., Tufano, M., Vendome, C., and Poshyvanyk, D. (2016, January 3\u20137). Deep learning code fragments for code clone detection. Proceedings of the 31st IEEE\/ACM International Conference on Automated Software Engineering, Singapore.","DOI":"10.1145\/2970276.2970326"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Wei, H.H., and Li, M. (2017, January 19\u201325). Supervised deep features for software functional clone detection by exploiting lexical and syntactical information in source code. Proceedings of the 26th International Joint Conference on Artificial Intelligence, Melbourne, VIC, Australia.","DOI":"10.24963\/ijcai.2017\/423"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1007\/s10664-011-9173-9","article-title":"Evaluating defect prediction approaches: A benchmark and an extensive comparison","volume":"17","author":"Lanza","year":"2012","journal-title":"Empir. Softw. Eng."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Pu, Y., Narasimhan, K., Solar-Lezama, A., and Barzilay, R. (2016, January 20). Sk_p: A neural program corrector for mooc. Proceedings of the 2016 ACM SIGPLAN International Conference on Systems, Programming, Languages and Applications: Software for Humanity, Amsterdam, The Netherlands.","DOI":"10.1145\/2984043.2989222"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Gupta, R., Pal, S., Kanade, A., and Shevade, S.K. (2017, January 4\u20139). DeepFix: Fixing common c language errors by deep learning. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17), San Francisco, CA, USA.","DOI":"10.1609\/aaai.v31i1.10742"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Raychev, V., Vechev, M., and Yahav, E. (2014, January 9\u201311). Code completion with statistical language models. Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI\u201914, Edinburgh, UK.","DOI":"10.1145\/2594291.2594321"},{"key":"ref_20","first-page":"7426461","article-title":"A Neural Network Based Intelligent Support Model for Program Code Completion","volume":"2020","author":"Rahman","year":"2020","journal-title":"Sci. Program."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Pennington, J., Socher, R., and Manning, C.D. (2014, January 25\u201329). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.","DOI":"10.3115\/v1\/D14-1162"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Teshima, Y., and Watanobe, Y. (2018, January 7\u201310). Bug detection based on LSTM networks and solution codes. Proceedings of the 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki, Japan.","DOI":"10.1109\/SMC.2018.00599"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Rahman, M.M., Watanobe, Y., and Nakamura, K. (2020). Source Code Assessment and Classification Based on Estimated Error Probability Using Attentive LSTM Language Model and Its Application in Programming Education. Appl. Sci., 10.","DOI":"10.3390\/app10082973"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Terada, K., and Watanobe, Y. (2019, January 9\u201310). Code Completion for Programming Education based on Recurrent Neural Network. Proceedings of the 2019 IEEE 11th International Workshop on Computational Intelligence and Applications (IWCIA), Hiroshima, Japan.","DOI":"10.1109\/IWCIA47330.2019.8955090"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Rahman, M.M., Watanobe, Y., and Nakamura, K. (2020, January 21\u201323). Evaluation of Source Codes Using Bidirectional LSTM Neural Network. Proceedings of the 2020 3rd IEEE International Conference on Knowledge Innovation and Invention (ICKII), Kaohsiung, Taiwan.","DOI":"10.1109\/ICKII50300.2020.9318916"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Le, P., and Zuidema, W.H. (2016, January 11). Quantifying the vanishing gradient and long distance dependency problem in recursive neural networks and recursive LSTMs. Proceedings of the 1st Workshop on Representation Learning for NLP, Berlin, Germany.","DOI":"10.18653\/v1\/W16-1610"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"2673","DOI":"10.1109\/78.650093","article-title":"Bidirectional recurrent neural networks","volume":"45","author":"Schuster","year":"1997","journal-title":"IEEE Trans. Signal Process."},{"key":"ref_28","unstructured":"(2020, December 11). Intelligent Coding Environment (ICE). Available online: https:\/\/onlinejudge.u-aizu.ac.jp\/services\/ice\/."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Song, D., Lee, M., and Oh, H. (2019, January 20\u201325). Automatic and Scalable Detection of Logical Errors in Functional Programming Assignments. Proceedings of the ACM Programming Languages OOPSLA, Athens, Greece.","DOI":"10.1145\/3360614"},{"key":"ref_30","first-page":"37","article-title":"A Survey of Machine Learning for Big Code and Naturalness","volume":"51","author":"Allamanis","year":"2018","journal-title":"ACM Comput. Surv."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Allamanis, M., Barr, E.T., Bird, C., and Sutton, C. (September, January 30). Suggesting accurate method and class names. Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC\/FSE 2015). Association for Computing Machinery, Bergamo, Italy.","DOI":"10.1145\/2786805.2786849"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Wang, S., Liu, T., and Tan, L. (2016, January 14\u201322). Automatically learning semantic features for defect prediction. Proceedings of the 38th International Conference on Software Engineering (ICSE\u201916), Austin, TX, USA.","DOI":"10.1145\/2884781.2884804"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Pradel, M., and Sen, K. (2018, January 4\u20139). DeepBugs: A learning approach to name-based bug detection. Proceedings of the ACM Programming Languages, Boston, MA, USA.","DOI":"10.1145\/3276517"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Song, X., Chen, C., Cui, B., and Fu, J. (2020). Malicious JavaScript Detection Based on Bidirectional LSTM Model. Appl. Sci., 10.","DOI":"10.3390\/app10103440"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"602","DOI":"10.1016\/j.neunet.2005.06.042","article-title":"Framewise phoneme classification with bidirectional LSTM and other neural network architectures","volume":"18","author":"Graves","year":"2005","journal-title":"Neural Netw."},{"key":"ref_36","unstructured":"Aizu Online Judge (2020, June 25). Developers Site (API). Available online: http:\/\/developers.u-aizu.ac.jp\/index."},{"key":"ref_37","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_38","unstructured":"Kingma, D.P., and Ba, J. (2015, January 7\u20139). Adam: A method for stochastic optimization. Proceedings of the 3rd International Conference for Learning Representations (ICLR), San Diego, CA, USA."}],"container-title":["Symmetry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-8994\/13\/2\/247\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:18:23Z","timestamp":1760159903000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-8994\/13\/2\/247"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,1]]},"references-count":38,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2021,2]]}},"alternative-id":["sym13020247"],"URL":"https:\/\/doi.org\/10.3390\/sym13020247","relation":{},"ISSN":["2073-8994"],"issn-type":[{"value":"2073-8994","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,1]]}}}