{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T06:43:24Z","timestamp":1740120204571,"version":"3.37.3"},"reference-count":43,"publisher":"World Scientific Pub Co Pte Ltd","issue":"10","funder":[{"DOI":"10.13039\/100014717","name":"National Outstanding Youth Science Fund Project of National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62150410441"],"award-info":[{"award-number":["62150410441"]}],"id":[{"id":"10.13039\/100014717","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Int. J. Soft. Eng. Knowl. Eng."],"published-print":{"date-parts":[[2022,10]]},"abstract":"<jats:p> Studies have confirmed the robust performance of machine learning classifiers for various source code modeling tasks. In general, machine learning approaches are incapable of handling imbalanced datasets, since they are sensitive to the choice of diverse classes. Therefore, these approaches may lean towards the classes with a large percentage of observations. In this work, we investigate and explore the impact of balanced and imbalanced learning on source code suggestion task otherwise known as code completion, covering a large number of imbalanced classes. We further explore the impact of vocabulary size on modeling performance. First, we provide the essentials to formulate the problem of source code suggestion as a classification task and investigate the level of imbalanced classes. Second, we train the four most adapted neural language models as a baseline to assess the modeling performance. Third, we impose two diverse class balancing techniques, TomekLinks and AllKNN, to balance the datasets and evaluate their impact on the modeling performance. Finally, we trained these models with a weighted imbalanced learning approach and compared the performance with balanced learning approaches. Additionally, we train models by varying the vocabulary size to study their impact. In total, we trained 230 models on 10 real-world software projects and extensively evaluated these models with widely used performance metrics such as Precision, Recall, FScore, mean reciprocal rank (MRR), and Receiver operating characteristics (ROC). Additionally, we employed ANOVA statistical analysis to study the statistical significance and differences between these approaches. This study has demonstrated that the modeling performance decreases during balanced model training, whereas the weighted imbalance training produces comparable results and is more efficient in terms of time cost. Additionally, this study exhibits that a large size of vocabulary does not necessarily improve the modeling performance when out-of-vocabulary predictions are disregarded. <\/jats:p>","DOI":"10.1142\/s0218194022500589","type":"journal-article","created":{"date-parts":[[2022,9,20]],"date-time":"2022-09-20T04:20:01Z","timestamp":1663647601000},"page":"1499-1526","source":"Crossref","is-referenced-by-count":0,"title":["Exploring the Impact of Balanced and Imbalanced Learning in Source Code Suggestion"],"prefix":"10.1142","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7318-8137","authenticated-orcid":false,"given":"Yasir","family":"Hussain","sequence":"first","affiliation":[{"name":"College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics (NUAA), Nanjing, Jiangsu 211106, P.\u00a0R.\u00a0China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhiqiu","family":"Huang","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics (NUAA), Nanjing, Jiangsu 211106, P.\u00a0R.\u00a0China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yu","family":"Zhou","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics (NUAA), Nanjing, Jiangsu 211106, P.\u00a0R.\u00a0China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Izhar Ahmed","family":"Khan","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics (NUAA), Nanjing, Jiangsu 211106, P.\u00a0R.\u00a0China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"219","published-online":{"date-parts":[[2022,10,31]]},"reference":[{"key":"S0218194022500589BIB001","doi-asserted-by":"publisher","DOI":"10.1016\/S0893-6080(05)80125-X"},{"key":"S0218194022500589BIB002","first-page":"12","volume-title":"Eleventh Annual Conf. Int. Speech Communication Association","author":"Mikolov T.","year":"2010"},{"key":"S0218194022500589BIB003","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"S0218194022500589BIB006","first-page":"7339","volume-title":"Thirty-Second AAAI Conf. Artificial Intelligence","author":"Sethi A.","year":"2018"},{"key":"S0218194022500589BIB007","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09730-9"},{"key":"S0218194022500589BIB008","doi-asserted-by":"crossref","first-page":"2073","DOI":"10.18653\/v1\/P16-1195","volume-title":"Proc. 54th Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers)","author":"Iyer S.","year":"2016"},{"key":"S0218194022500589BIB009","first-page":"2091","volume-title":"Int. Conf. Machine Learning","author":"Allamanis M.","year":"2016"},{"key":"S0218194022500589BIB010","first-page":"2309","volume-title":"The Web Conf. Proc. World Wide Web Conf.","author":"Ye W.","year":"2020"},{"key":"S0218194022500589BIB011","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2015.38"},{"key":"S0218194022500589BIB012","doi-asserted-by":"publisher","DOI":"10.1145\/2666356.2594321"},{"issue":"12","key":"S0218194022500589BIB013","doi-asserted-by":"crossref","first-page":"604","DOI":"10.1049\/el.2020.0500","volume":"56","author":"Hussain Y.","year":"2020","journal-title":"Electron. Lett."},{"key":"S0218194022500589BIB014","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2020.106309"},{"key":"S0218194022500589BIB015","doi-asserted-by":"publisher","DOI":"10.1142\/S0218194020500230"},{"key":"S0218194022500589BIB016","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3053111"},{"key":"S0218194022500589BIB017","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.infsof.2018.07.006","volume":"104","author":"Mi Q.","year":"2018","journal-title":"Inf. Softw. Technol."},{"key":"S0218194022500589BIB018","doi-asserted-by":"publisher","DOI":"10.3233\/IDA-2002-6504"},{"issue":"18","key":"S0218194022500589BIB019","doi-asserted-by":"crossref","first-page":"3549","DOI":"10.1080\/014311699211192","volume":"20","author":"Foody G. M.","year":"1999","journal-title":"Int. J. Remote Sens."},{"key":"S0218194022500589BIB020","first-page":"359","volume-title":"Proc. 4th Indian Int. Conf. Artificial Intelligence","author":"Prati R. C.","year":"2009"},{"key":"S0218194022500589BIB021","doi-asserted-by":"publisher","DOI":"10.1142\/S0219622006002258"},{"key":"S0218194022500589BIB022","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2013.6638947"},{"key":"S0218194022500589BIB023","first-page":"1097","volume-title":"Advances in Neural Information Processing Systems","author":"Krizhevsky A.","year":"2012"},{"issue":"11","key":"S0218194022500589BIB024","first-page":"769","volume":"6","author":"Tomek I.","year":"1976","journal-title":"IEEE Trans. Syst. Man Cybern."},{"issue":"11","key":"S0218194022500589BIB025","doi-asserted-by":"crossref","first-page":"1239","DOI":"10.1109\/T-C.1972.223484","volume":"100","author":"Das S. R.","year":"1972","journal-title":"IEEE Trans. Comput."},{"issue":"6","key":"S0218194022500589BIB026","first-page":"448","volume":"6","author":"Tomek I.","year":"1976","journal-title":"IEEE Trans. Syst. Man Cybern."},{"key":"S0218194022500589BIB027","first-page":"323","volume-title":"IEEE 25th Int. Conf. Software Analysis, Evolution and Reengineering","author":"Nguyen A. T.","year":"2018"},{"key":"S0218194022500589BIB028","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2012.6227135"},{"key":"S0218194022500589BIB029","first-page":"311","volume-title":"IEEE 25th Int. Conf. Software Analysis, Evolution and Reengineering","author":"Santos E. A.","year":"2018"},{"key":"S0218194022500589BIB030","first-page":"40","volume-title":"Proc. ACM Program. Lang.","volume":"3","author":"Alon U.","year":"2019"},{"issue":"1","key":"S0218194022500589BIB031","first-page":"25","volume":"30","author":"Kotsiantis S.","year":"2006","journal-title":"GESTS Int. Trans. Comput. Sci. Eng."},{"volume-title":"Pattern Recognition and Neural Networks","year":"2007","author":"Ripley B. D.","key":"S0218194022500589BIB032"},{"key":"S0218194022500589BIB033","doi-asserted-by":"crossref","first-page":"620","DOI":"10.1016\/j.renene.2018.10.047","volume":"133","author":"Stetco A.","year":"2019","journal-title":"Renew. Energy"},{"issue":"2","key":"S0218194022500589BIB034","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1007\/s13748-019-00172-4","volume":"8","author":"Kuncheva L. I.","year":"2019","journal-title":"Prog. Artif. Intell."},{"key":"S0218194022500589BIB035","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2008.239"},{"key":"S0218194022500589BIB036","volume":"1","author":"Elhassan T.","year":"2016","journal-title":"Global. J. Technol. Optim. S"},{"key":"S0218194022500589BIB037","doi-asserted-by":"publisher","DOI":"10.1111\/j.0824-7935.2004.t01-1-00228.x"},{"key":"S0218194022500589BIB038","first-page":"1","volume-title":"ICML-KDD 2003 Workshop: Learning from Imbalanced Datasets","author":"Drumnond C.","year":"2003"},{"key":"S0218194022500589BIB039","doi-asserted-by":"publisher","DOI":"10.1613\/jair.953"},{"key":"S0218194022500589BIB040","doi-asserted-by":"crossref","first-page":"352","DOI":"10.1016\/j.neucom.2019.06.100","volume":"366","author":"Kov\u00e1cs G.","year":"2019","journal-title":"Neurocomputing"},{"key":"S0218194022500589BIB041","first-page":"1322","volume-title":"IEEE Int. Joint Conf. Neural Networks (IEEE World Congress on Computational Intelligence)","author":"He H.","year":"2008"},{"key":"S0218194022500589BIB042","doi-asserted-by":"publisher","DOI":"10.1145\/1007730.1007736"},{"key":"S0218194022500589BIB045","first-page":"200","volume-title":"IEEE\/ACM 26th Int. Conf. Program Comprehension","author":"Hu X.","year":"2018"},{"key":"S0218194022500589BIB046","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2019.07.087"},{"key":"S0218194022500589BIB048","first-page":"1287","volume-title":"Thirtieth AAAI Conf. Artificial Intelligence","author":"Mou L.","year":"2016"}],"container-title":["International Journal of Software Engineering and Knowledge Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S0218194022500589","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,24]],"date-time":"2022-11-24T03:13:43Z","timestamp":1669259623000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/10.1142\/S0218194022500589"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10]]},"references-count":43,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2022,10]]}},"alternative-id":["10.1142\/S0218194022500589"],"URL":"https:\/\/doi.org\/10.1142\/s0218194022500589","relation":{},"ISSN":["0218-1940","1793-6403"],"issn-type":[{"type":"print","value":"0218-1940"},{"type":"electronic","value":"1793-6403"}],"subject":[],"published":{"date-parts":[[2022,10]]}}}