{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,10]],"date-time":"2026-01-10T19:57:46Z","timestamp":1768075066052,"version":"3.49.0"},"reference-count":42,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2024,7,1]],"date-time":"2024-07-01T00:00:00Z","timestamp":1719792000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"the Ministry of Health and Welfare, Republic of Korea","award":["HI21C0977"],"award-info":[{"award-number":["HI21C0977"]}]},{"name":"the Ministry of Health and Welfare, Republic of Korea","award":["HI20C1234"],"award-info":[{"award-number":["HI20C1234"]}]},{"name":"the Ministry of Health and Welfare, Republic of Korea","award":["DBSD1-06"],"award-info":[{"award-number":["DBSD1-06"]}]},{"name":"the Ministry of Health and Welfare, Republic of Korea","award":["NRF-2022R1I1A307278"],"award-info":[{"award-number":["NRF-2022R1I1A307278"]}]},{"name":"the Ministry of Health and Welfare, Republic of Korea","award":["RS-2022-00166898"],"award-info":[{"award-number":["RS-2022-00166898"]}]},{"name":"the Korea government (MSIT and Daegu Metropolitan City)","award":["HI21C0977"],"award-info":[{"award-number":["HI21C0977"]}]},{"name":"the Korea government (MSIT and Daegu Metropolitan City)","award":["HI20C1234"],"award-info":[{"award-number":["HI20C1234"]}]},{"name":"the Korea government (MSIT and Daegu Metropolitan City)","award":["DBSD1-06"],"award-info":[{"award-number":["DBSD1-06"]}]},{"name":"the Korea government (MSIT and Daegu Metropolitan City)","award":["NRF-2022R1I1A307278"],"award-info":[{"award-number":["NRF-2022R1I1A307278"]}]},{"name":"the Korea government (MSIT and Daegu Metropolitan City)","award":["RS-2022-00166898"],"award-info":[{"award-number":["RS-2022-00166898"]}]},{"name":"the National Research Foundation of Korea","award":["HI21C0977"],"award-info":[{"award-number":["HI21C0977"]}]},{"name":"the National Research Foundation of Korea","award":["HI20C1234"],"award-info":[{"award-number":["HI20C1234"]}]},{"name":"the National Research Foundation of Korea","award":["DBSD1-06"],"award-info":[{"award-number":["DBSD1-06"]}]},{"name":"the National Research Foundation of Korea","award":["NRF-2022R1I1A307278"],"award-info":[{"award-number":["NRF-2022R1I1A307278"]}]},{"name":"the National Research Foundation of Korea","award":["RS-2022-00166898"],"award-info":[{"award-number":["RS-2022-00166898"]}]},{"name":"Development of camera-based non-contact medical device with universal design","award":["HI21C0977"],"award-info":[{"award-number":["HI21C0977"]}]},{"name":"Development of camera-based non-contact medical device with universal design","award":["HI20C1234"],"award-info":[{"award-number":["HI20C1234"]}]},{"name":"Development of camera-based non-contact medical device with universal design","award":["DBSD1-06"],"award-info":[{"award-number":["DBSD1-06"]}]},{"name":"Development of camera-based non-contact medical device with universal design","award":["NRF-2022R1I1A307278"],"award-info":[{"award-number":["NRF-2022R1I1A307278"]}]},{"name":"Development of camera-based non-contact medical device with universal design","award":["RS-2022-00166898"],"award-info":[{"award-number":["RS-2022-00166898"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Facial expressions are often considered the primary indicators of emotions. However, it is challenging to detect genuine emotions because they can be controlled. Many studies on emotion recognition have been conducted actively in recent years. In this study, we designed a convolutional neural network (CNN) model and proposed an algorithm that combines the analysis of bio-signals with facial expression templates to effectively predict emotional states. We utilized the EfficientNet-B0 architecture for network design and validation, known for achieving maximum performance with minimal parameters. The accuracy for emotion recognition using facial expression images alone was 74%, while the accuracy for emotion recognition combining biological signals reached 88.2%. These results demonstrate that integrating these two types of data leads to significantly improved accuracy. By combining the image and bio-signals captured in facial expressions, our model offers a more comprehensive and accurate understanding of emotional states.<\/jats:p>","DOI":"10.3390\/a17070285","type":"journal-article","created":{"date-parts":[[2024,7,1]],"date-time":"2024-07-01T08:17:29Z","timestamp":1719821849000},"page":"285","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["The Novel EfficientNet Architecture-Based System and Algorithm to Predict Complex Human Emotions"],"prefix":"10.3390","volume":"17","author":[{"given":"Mavlonbek","family":"Khomidov","sequence":"first","affiliation":[{"name":"Department of Computer Engineering, Keimyung University, Daegu 42601, Republic of Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6819-8816","authenticated-orcid":false,"given":"Jong-Ha","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Biomedical Engineering, Keimyung University, Daegu 42601, Republic of Korea"}]}],"member":"1968","published-online":{"date-parts":[[2024,7,1]]},"reference":[{"key":"ref_1","unstructured":"Cha, W.-Y., Shin, D.-K., and Shin, D.-I. (2017, January 10\u201313). Analysis and Comparison of The Emotion Recognition by Multiple Bio-Signal. Proceedings of the Korean Information Science Society Conference, Seoul, Republic of Korea."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Kortelainen, J., Tiinanen, S., Huang, X., Li, X., Laukka, S., Pietik\u00e4inen, M., and Sepp\u00e4nen, T. (September, January 28). Multimodal emotion recognition by combining physiological signals and facial expressions: A preliminary study. Proceedings of the 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Diego, CA, USA.","DOI":"10.1109\/EMBC.2012.6347175"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1016\/j.neunet.2005.03.004","article-title":"Emotion recognition through facial expression analysis based on a neurofuzzy network","volume":"18","author":"Ioannou","year":"2005","journal-title":"Neural Netw."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"8375","DOI":"10.1109\/ACCESS.2016.2628407","article-title":"Facial Emotion Recognition Based on Biorthogonal Wavelet Entropy, Fuzzy Support Vector Machine, and Stratified Cross Validation","volume":"4","author":"Zhang","year":"2016","journal-title":"IEEE Access"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"967","DOI":"10.1037\/0022-3514.74.4.967","article-title":"Independence and bipolarity in the structure of current affect","volume":"74","author":"Barrett","year":"1998","journal-title":"J. Pers. Soc. Psychol."},{"key":"ref_6","unstructured":"Tacconi, D., Mayora, O., Lukowicz, P., Arnrich, B., Setz, C., Troster, G., and Haring, C. (February, January 30). Activity and emotion recognition to support early diagnosis of psychiatric diseases. Proceedings of the Second International Conference on Pervasive Computing Technologies for Healthcare, Tampere, Finland."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Visvizi, A., and Lytras, M. (2019). Emotion Recognition to Improve e-Healthcare Systems in Smart Cities. Research & Innovation Forum 2019: Technology, Innovation, Education, and their Social Impact 1, Springer. Springer Proceedings in Complexity.","DOI":"10.1007\/978-3-030-30809-4"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Wioleta, S. (2013, January 6\u20138). Using physiological signals for emotion recognition. Proceedings of the 6th International Conference on Human System Interactions (HSI), Gdansk, Poland.","DOI":"10.1109\/HSI.2013.6577880"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Shin, J., Maeng, J., and Kim, D.-H. (2018, January 24\u201326). Inner Emotion Recognition Using Multi Bio-Signals. Proceedings of the 2018 IEEE International Conference on Consumer Electronics\u2014Asia (ICCE-Asia), Jeju, Republic of Korea.","DOI":"10.1109\/ICCE-ASIA.2018.8552152"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1717","DOI":"10.1007\/s00500-020-05501-7","article-title":"Generalisation and robustness investigation for facial and speech emotion recognition using bio-inspired spiking neural networks","volume":"25","author":"Ye","year":"2021","journal-title":"Soft Comput."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1016\/j.ijpsycho.2012.08.012","article-title":"Heart rate variability is associated with emotion recognition: Direct evidence for a relationship between the autonomic nervous system and social cognition","volume":"86","author":"Quintana","year":"2012","journal-title":"Int. J. Psychophysiol."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Roveda, J.M., Fink, W., Chen, K., and Wu, W.-T. (2016, January 5\u201312). Psychological health monitoring for pilots and astronauts by tracking sleep-stress-emotion changes. Proceedings of the IEEE Aerospace Conference, Big Sky, MT, USA.","DOI":"10.1109\/AERO.2016.7500908"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Randazzo, V., Ferretti, J., and Pasero, E. (2021). Anytime ECG Monitoring through the Use of a Low-Cost, User-Friendly, Wearable Device. Sensors, 21.","DOI":"10.3390\/s21186036"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Jing, C., Liu, G., and Hao, M. (2009, January 25\u201326). The Research on Emotion Recognition from ECG Signal. Proceedings of the International Conference on Information Technology and Computer Science, Kiev, Ukraine.","DOI":"10.1109\/ITCS.2009.108"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Ravindran, A.S., Nakagome, S., Wickramasuriya, D.S., Contreras-Vidal, J.L., and Faghih, R.T. (2019, January 20\u201322). Emotion Recognition by Point Process Characterization of Heartbeat Dynamics. Proceedings of the IEEE Healthcare Innovations and Point of Care Technologies, Bethesda, MD, USA.","DOI":"10.1109\/HI-POCT45284.2019.8962886"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Khomidov, M., Lee, D., Kim, C.-H., and Lee, J.-H. (2024). The Real-Time Image Sequences-Based Stress Assessment Vision System for Mental Health. Electronics, 13.","DOI":"10.3390\/electronics13112180"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., and Lee, D.-H. (2013, January 3\u20137). Challenges in representation learning: A report on three machine learning contests. Proceedings of the International Conference on Neural Information Processing, Daegu, Republic of Korea.","DOI":"10.1007\/978-3-642-42051-1_16"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Liu, K., Zhang, M., and Pan, Z. (2016, January 28\u201330). Facial Expression Recognition with CNN Ensemble. Proceedings of the 2016 International Conference on Cyberworlds (CW), Chongqing, China.","DOI":"10.1109\/CW.2016.34"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"26756","DOI":"10.1109\/ACCESS.2022.3156598","article-title":"Ad-Corre: Adaptive Correlation-Based Loss for Facial Expression Recognition in the Wild","volume":"10","author":"Fard","year":"2022","journal-title":"IEEE Access"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Chollet, F. (2017, January 21\u201326). Xception: Deep Learning with Depthwise Separable Convolutions. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.195"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Vulpe-Grigora\u015fi, A., and Grigore, O. (2021, January 25\u201327). Convolutional Neural Network Hyperparameters optimization for Facial Emotion Recognition. Proceedings of the 2021 12th International Symposium on Advanced Topics in Electrical Engineering (ATEE), Bucharest, Romania.","DOI":"10.1109\/ATEE52255.2021.9425073"},{"key":"ref_23","first-page":"1777","article-title":"A novel facial emotion recognition model using segmentation VGG-19 architecture","volume":"15","author":"Vignesh","year":"2023","journal-title":"Int. J. Inf. Technol."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Pham, L., Vu, T.H., and Tran, T.A. (2021, January 10\u201315). Facial Expression Recognition Using Residual Masking Network. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.","DOI":"10.1109\/ICPR48806.2021.9411919"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Guo, H.-W., Huang, Y.-S., Lin, C.-H., Chien, J.-C., Haraikawa, K., and Shieh, J.-S. (November, January 31). Heart Rate Variability Signal Features for Emotion Recognition by Using Principal Component Analysis and Support Vectors Machine. Proceedings of the 2016 IEEE 16th International Conference on Bioinformatics and Bioengineering (BIBE), Taichung, Taiwan.","DOI":"10.1109\/BIBE.2016.40"},{"key":"ref_26","unstructured":"Vapnik, V. (1998). Statistical Learning Theory, Wiley."},{"key":"ref_27","unstructured":"De Marsico, M., di Baja, G., and Fred, A. (2018). Emotion Recognition Using Neighborhood Components Analysis and ECG\/HRV-Based Features. Pattern Recognition Applications and Methods: 6th International Conference, ICPRAM 2017, Porto, Portugal, 24\u201326 February 2017, Springer. Lecture Notes in Computer Science."},{"key":"ref_28","first-page":"1","article-title":"Neighbourhood Components Analysis","volume":"17","author":"Goldberger","year":"2004","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Lee, M., Lee, Y.K., Lim, M.-T., and Kang, T.-K. (2020). Emotion Recognition Using Convolutional Neural Network with Selected Statistical Photoplethysmogram Features. Appl. Sci., 10.","DOI":"10.3390\/app10103501"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1016\/j.inffus.2021.07.007","article-title":"Emotion recognition based on convolutional neural networks and heterogeneous bio-signal data sources","volume":"77","author":"Ngai","year":"2022","journal-title":"Inf. Fusion"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"100372","DOI":"10.1016\/j.imu.2020.100372","article-title":"Development of a Real-Time Emotion Recognition System Using Facial Expressions and EEG based on machine learning and deep neural network methods","volume":"20","author":"Hassouneh","year":"2020","journal-title":"Inform. Med. Unlocked"},{"key":"ref_32","unstructured":"Godin, C., Prost-Boucle, F., Campagne, A., Charbonnier, S., Bonnet, S., and Vidal, A. (2015, January 11\u201313). Selection of the Most Relevant Physiological Features for Classifying Emotion. Proceedings of the 2nd International Conference on Physiological Computing Systems, Loire Valley, France."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Dzedzickis, A., Kaklauskas, A., and Bucinskas, V. (2020). Human Emotion Recognition: Review of Sensors and Methods. Sensors, 20.","DOI":"10.3390\/s20030592"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Singson, L.N.B., Sanchez, M.T.U.R., and Villaverde, J.F. (2021, January 20\u201322). Emotion Recognition Using Short-Term Analysis of Heart Rate Variability and ResNet Architecture. Proceedings of the 13th International Conference on Computer and Automation Engineering (ICCAE), Melbourne, Australia.","DOI":"10.1109\/ICCAE51876.2021.9426094"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"11896","DOI":"10.1109\/ACCESS.2020.2964794","article-title":"Non-Contact Emotion Recognition Combining Heart Rate and Facial Expression for Interactive Gaming Environments","volume":"8","author":"Du","year":"2020","journal-title":"IEEE Access"},{"key":"ref_36","unstructured":"Tan, M., and Le, Q.V. (2019, January 9\u201315). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Pal, K.K., and Sudeep, K.S. (2016, January 20\u201321). Preprocessing for image classification by convolutional neural networks. Proceedings of the IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT), Bangalore, India.","DOI":"10.1109\/RTEICT.2016.7808140"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"159081","DOI":"10.1109\/ACCESS.2019.2949741","article-title":"A Face Emotion Recognition Method Using Convolutional Neural Network and Image Edge Computing","volume":"7","author":"Zhang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Zhao, L., Yang, L., Shi, H., Xia, Y., Li, F., and Liu, C. (2017, January 20\u201322). Evaluation of consistency of HRV indices change among different emotions. Proceedings of the Chinese Automation Congress (CAC), Jinan, China.","DOI":"10.1109\/CAC.2017.8243625"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"523","DOI":"10.1016\/j.procs.2017.10.038","article-title":"Enhancing CNN with Preprocessing Stage in Automatic Emotion Recognition","volume":"116","author":"Pitaloka","year":"2017","journal-title":"Procedia Comput. Sci."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Li, X., Song, D., Zhang, P., Yu, G., Hou, Y., and Hu, B. (2016, January 15\u201318). Emotion recognition from multi-channel EEG data through Convolutional Recurrent Neural Network. Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Shenzhen, China.","DOI":"10.1109\/BIBM.2016.7822545"},{"key":"ref_42","unstructured":"Khaireddin, Y., and Chen, Z. (2021). Facial Emotion Recognition: State of the Art Performance on FER2013. arXiv."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/7\/285\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:08:38Z","timestamp":1760108918000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/7\/285"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,1]]},"references-count":42,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2024,7]]}},"alternative-id":["a17070285"],"URL":"https:\/\/doi.org\/10.3390\/a17070285","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,1]]}}}