{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T15:38:09Z","timestamp":1783697889766,"version":"3.55.0"},"reference-count":34,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2023,1,20]],"date-time":"2023-01-20T00:00:00Z","timestamp":1674172800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>This article is dedicated to solving the problem of an insufficient degree of automation of artificial neural network training. Despite the availability of a large number of libraries for training neural networks, machine learning engineers often have to manually control the training process to detect overfitting or underfitting. This article considers the task of automatically estimating neural network training results through an analysis of learning curves. Such analysis allows one to determine one of three possible states of the training process: overfitting, underfitting, and optimal training. We propose several algorithms for extracting feature descriptions from learning curves using mathematical statistics. Further state classification is performed using classical machine learning models. The proposed automatic estimation model serves to improve the degree of automation of neural network training and interpretation of its results, while also taking a step toward constructing self-training models. In most cases when the training process of neural networks leads to overfitting, the developed model determines its onset ahead of the early stopping method by 3\u20135 epochs.<\/jats:p>","DOI":"10.3390\/computers12020026","type":"journal-article","created":{"date-parts":[[2023,1,20]],"date-time":"2023-01-20T02:43:29Z","timestamp":1674182609000},"page":"26","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["Automatic Evaluation of Neural Network Training Results"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0269-3205","authenticated-orcid":false,"given":"Roman","family":"Barinov","sequence":"first","affiliation":[{"name":"Department of Computing Systems and Technologies, Nizhny Novgorod State Technical University n.a. R.E. Alekseev, st. Minina, 24, 603155 Nizhny Novgorod, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3644-5234","authenticated-orcid":false,"given":"Vasiliy","family":"Gai","sequence":"additional","affiliation":[{"name":"Department of Computing Systems and Technologies, Nizhny Novgorod State Technical University n.a. R.E. Alekseev, st. Minina, 24, 603155 Nizhny Novgorod, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5564-045X","authenticated-orcid":false,"given":"George","family":"Kuznetsov","sequence":"additional","affiliation":[{"name":"Department of Computing Systems and Technologies, Nizhny Novgorod State Technical University n.a. R.E. Alekseev, st. Minina, 24, 603155 Nizhny Novgorod, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4683-3249","authenticated-orcid":false,"given":"Vladimir","family":"Golubenko","sequence":"additional","affiliation":[{"name":"Department of Computing Systems and Technologies, Nizhny Novgorod State Technical University n.a. R.E. Alekseev, st. Minina, 24, 603155 Nizhny Novgorod, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"183396","DOI":"10.1109\/ACCESS.2019.2959620","article-title":"3D ROC Histogram: A New ROC Analysis Tool Incorporating Information on Instances","volume":"7","author":"Guo","year":"2019","journal-title":"IEEE Access"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Jalaeian Zaferani, E., Teshnehlab, M., Khodadadian, A., Heitzinger, C., Vali, M., Noii, N., and Wick, T. (2022). Hyper-Parameter Optimization of Stacked Asymmetric Auto-Encoders for Automatic Personality Traits Perception. Sensors, 22.","DOI":"10.3390\/s22166206"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Yotov, K., Hadzhikolev, E., Hadzhikoleva, S., and Cheresharov, S. (2023). Finding the Optimal Topology of an Approximating Neural Network. Mathematics, 11.","DOI":"10.3390\/math11010217"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"3037","DOI":"10.1109\/TPAMI.2021.3051276","article-title":"AutoML for Multi-Label Classification: Overview and Empirical Evaluation","volume":"43","author":"Wever","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"106622","DOI":"10.1016\/j.knosys.2020.106622","article-title":"AutoML: A survey of the state-of-the-art","volume":"212","author":"Xin","year":"2021","journal-title":"Knowl. Based Syst."},{"key":"ref_6","unstructured":"Kotthoff, L., Thornton, C., Hoos, H., Hutter, F., and Leyton-Brown, K. (2017). Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA. J. Mach. Learn. Res."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Hutter, F., Kotthoff, L., and Vanschoren, J. (2019). Automated Machine Learning: Methods, Systems, Challenges, Springer Nature.","DOI":"10.1007\/978-3-030-05318-5"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"3067","DOI":"10.1109\/TPAMI.2021.3062900","article-title":"Adaptation Strategies for Automated Machine Learning on Evolving Data","volume":"43","author":"Celik","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Alsharef, A., Kumar, K., and Iwendi, C. (2022). Time Series Data Modeling Using Advanced Machine Learning and AutoML. Sustainability, 14.","DOI":"10.3390\/su142215292"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"521","DOI":"10.3390\/make1010032","article-title":"Evaluation of Regression Models: Model Assessment, Model Selection and Generalization Error","volume":"1","author":"Dehmer","year":"2019","journal-title":"Mach. Learn. Knowl. Extr."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Chetoui, M., Akhloufi, M., Yousefi, B., and Bouattane, E. (2021). Explainable COVID-19 Detection on Chest X-rays Using an End-to-End Deep Convolutional Neural Network Architecture. Big Data Cogn. Comput., 5.","DOI":"10.3390\/bdcc5040073"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Algehyne, E., Jibril, M., Algehainy, N., Alamri, O., and Alzahrani, A. (2022). Fuzzy Neural Network Expert System with an Improved Gini Index Random Forest-Based Feature Importance Measure Algorithm for Early Diagnosis of Breast Cancer in Saudi Arabia. Big Data Cogn. Comput., 6.","DOI":"10.3390\/bdcc6010013"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Dora, S., and Kasabov, N. (2021). Spiking Neural Networks for Computational Intelligence: An Overview. Big Data Cogn. Comput., 5.","DOI":"10.3390\/bdcc5040067"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Frank, M., Drikakis, D., and Charissis, V. (2020). Machine-Learning Methods for Computational Science and Engineering. Computation, 8.","DOI":"10.3390\/computation8010015"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Huang, Y.-C., Hung, K.-C., and Lin, J.-C. (2022). Automated Machine Learning System for Defect Detection on Cylindrical Metal Surfaces. Sensors, 22.","DOI":"10.3390\/s22249783"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1722","DOI":"10.1109\/TKDE.2019.2911585","article-title":"Evaluating Overfit and Underfit in Models of Network Community Structure","volume":"32","author":"Ghasemian","year":"2020","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"52588","DOI":"10.1109\/ACCESS.2020.2981072","article-title":"Basic Enhancement Strategies When Using Bayesian Optimization for Hyperparameter Tuning of Deep Neural Networks","volume":"8","author":"Cho","year":"2020","journal-title":"IEEE Access"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Nallakaruppan, M., Ramalingam, S., Somayaji, S., and Prathiba, S. (2022). Comparative Analysis of Deep Learning Models Used in Impact Analysis of Coronavirus Chest X-ray Imaging. Biomedicines, 10.","DOI":"10.3390\/biomedicines10112791"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Gu, Y., Wylie, B.K., Boyte, S.P., Picotte, J., Howard, D.M., Smith, K., and Nelson, K.J. (2016). An Optimal Sample Data Usage Strategy to Minimize Overfitting and Underfitting Effects in Regression Tree Models Based on Remotely-Sensed Data. Remote Sens., 8.","DOI":"10.3390\/rs8110943"},{"key":"ref_20","unstructured":"Domhan, T., Springenberg, J.T., and Hutter, F. (2015, January 25). Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence Buenos Aires, Buenos Aires, Argentina."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1109\/TMI.2020.3046692","article-title":"Analyzing Overfitting Under Class Imbalance in Neural Networks for Image Segmentation","volume":"40","author":"Li","year":"2011","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"62830","DOI":"10.1109\/ACCESS.2020.2983774","article-title":"Sequence-Dropout Block for Reducing Overfitting Problem in Image Classification","volume":"8","author":"Qian","year":"2020","journal-title":"IEEE Access"},{"key":"ref_23","first-page":"53","article-title":"Early Stopping\u2014But When?","volume":"Volume 7700","author":"Montavon","year":"2012","journal-title":"Neural Networks: Tricks of the Trade. Lecture Notes in Computer Science"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"4844","DOI":"10.1109\/TNNLS.2017.2769128","article-title":"Cross-Modal Metric Learning for AUC Optimization","volume":"29","author":"Huo","year":"2018","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"9:1","DOI":"10.1147\/JRD.2017.2709578","article-title":"An effective algorithm for hyperparameter optimization of neural networks","volume":"61","author":"Diaz","year":"2017","journal-title":"IBM J. Res. Dev."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Kim, D., Seo, S.B., Yoo, N.H., and Shin, G. (2022). A Study on Sample Size Sensitivity of Factory Manufacturing Dataset for CNN-Based Defective Product Classification. Computation, 10.","DOI":"10.3390\/computation10080142"},{"key":"ref_27","first-page":"61","article-title":"Elements of the study of image detection","volume":"81","author":"Utrobin","year":"2010","journal-title":"Trans. NNSTU N. A. R. E. Alekseev"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Wang, C., Baratchi, M., B\u00e4ck, T., Hoos, H.H., Limmer, S., and Olhofer, M. (2022). Towards Time-Series Feature Engineering in Automated Machine Learning for Multi-Step-Ahead Forecasting. Eng. Proc., 18.","DOI":"10.3390\/engproc2022018017"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Leite, D., Martins, A., Rativa, D., De Oliveira, J.F.L., and Maciel, A.M.A. (2022). An Automated Machine Learning Approach for Real-Time Fault Detection and Diagnosis. Sensors, 22.","DOI":"10.3390\/s22166138"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"74763","DOI":"10.1109\/ACCESS.2021.3080316","article-title":"Radius-SMOTE: A New Oversampling Technique of Minority Samples Based on Radius Distance for Learning From Imbalanced Data","volume":"9","author":"Pradipta","year":"2021","journal-title":"IEEE Access"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"47491","DOI":"10.1109\/ACCESS.2021.3068316","article-title":"Effects of Data Augmentation Method Borderline-SMOTE on Emotion Recognition of EEG Signals Based on Convolutional Neural Network","volume":"9","author":"Chen","year":"2021","journal-title":"IEEE Access"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Dablain, D., Krawczyk, B., and Chawla, N.V. (2022). DeepSMOTE: Fusing Deep Learning and SMOTE for Imbalanced Data. IEEE Trans. Neural Netw. Learn. Syst.","DOI":"10.1109\/TNNLS.2021.3136503"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"GhoshRoy, D., Alvi, P.A., and Santosh, K. (2023). Explainable AI to Predict Male Fertility Using Extreme Gradient Boosting Algorithm with SMOTE. Electronics, 12.","DOI":"10.3390\/electronics12010015"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Kumar, P., Ali, I., Kim, D.-G., Byun, S.-J., Kim, D.-G., Pu, Y.-G., and Lee, K.-Y. (2022). A Study on the Design Procedure of Re-Configurable Convolutional Neural Network Engine for FPGA-Based Applications. Electronics, 11.","DOI":"10.3390\/electronics11233883"}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/12\/2\/26\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:11:55Z","timestamp":1760119915000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/12\/2\/26"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,20]]},"references-count":34,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["computers12020026"],"URL":"https:\/\/doi.org\/10.3390\/computers12020026","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,20]]}}}