{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,31]],"date-time":"2026-10-31T20:08:11Z","timestamp":1793477291053,"version":"4.5.0"},"reference-count":76,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,6,1]],"date-time":"2023-06-01T00:00:00Z","timestamp":1685577600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Psychol."],"abstract":"<jats:p>Objective and accurate classification of fear levels is a socially important task that contributes to developing treatments for Anxiety Disorder, Obsessive\u2013compulsive Disorder, Post-Traumatic Stress Disorder (PTSD), and Phobia. This study examines a deep learning model to automatically estimate human fear levels with high accuracy using multichannel EEG signals and multimodal peripheral physiological signals in the DEAP dataset. The Multi-Input CNN-LSTM classification model combining Convolutional Neural Network (CNN) and Long Sort-Term Memory (LSTM) estimated four fear levels with an accuracy of 98.79% and an F1 score of 99.01% in a 10-fold cross-validation. This study contributes to the following; (1) to present the possibility of recognizing fear emotion with high accuracy using a deep learning model from physiological signals without arbitrary feature extraction or feature selection, (2) to investigate effective deep learning model structures for high-accuracy fear recognition and to propose Multi-Input CNN-LSTM, and (3) to examine the model\u2019s tolerance to individual differences in physiological signals and the possibility of improving accuracy through additional learning.<\/jats:p>","DOI":"10.3389\/fpsyg.2023.1141801","type":"journal-article","created":{"date-parts":[[2023,6,1]],"date-time":"2023-06-01T05:03:43Z","timestamp":1685595823000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":21,"title":["Multi-Input CNN-LSTM deep learning model for fear level classification based on EEG and peripheral physiological signals"],"prefix":"10.3389","volume":"14","author":[{"given":"Nagisa","family":"Masuda","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ikuko Eguchi","family":"Yairi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,6,1]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1109\/TAFFC.2015.2392932","article-title":"DECAF: MEG-based multimodal database for decoding affective physiological responses","volume":"6","author":"Abadi","year":"2015","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref2","doi-asserted-by":"publisher","first-page":"54","DOI":"10.3390\/bioengineering10010054","article-title":"A novel baseline removal paradigm for subject-independent features in emotion classification using EEG","volume":"10","author":"Ahmed","year":"2023","journal-title":"Bioengineering"},{"key":"ref3","doi-asserted-by":"crossref","first-page":"2623","DOI":"10.1145\/3292500.3330701","article-title":"Optuna: a next-generation hyperparameter optimization framework","volume-title":"Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery and data mining (KDD)","author":"Akiba","year":"2019"},{"key":"ref4","doi-asserted-by":"publisher","first-page":"10","DOI":"10.14569\/IJACSA.2017.081046","article-title":"Emotion recognition based on EEG using LSTM recurrent neural network","volume":"8","author":"Alhagry","year":"2017","journal-title":"Int. J. Adv. Comput. Sci. Appl."},{"key":"ref5","doi-asserted-by":"publisher","first-page":"175","DOI":"10.2307\/2685209","article-title":"An introduction to kernel and nearest-neighbor nonparametric regression","volume":"46","author":"Altman","year":"1992","journal-title":"Am. Stat."},{"key":"ref7","doi-asserted-by":"publisher","first-page":"537","DOI":"10.1111\/j.2044-829f5.1972.tb01304.x","article-title":"Selective attention and performance in dangerous environments","volume":"63","author":"Baddeley","year":"1972","journal-title":"Br. J. Psychol."},{"key":"ref8","doi-asserted-by":"publisher","first-page":"1738","DOI":"10.3390\/s19071738","article-title":"Fear level classification based on emotional dimensions and machine learning techniques","volume":"19","author":"B\u0103lan","year":"2019","journal-title":"Sensors"},{"key":"ref9","doi-asserted-by":"publisher","first-page":"496","DOI":"10.3390\/s20020496","article-title":"An investigation of various machine and deep learning techniques applied in automatic fear level detection and acrophobia virtual therapy","volume":"20","author":"B\u0103lan","year":"2020","journal-title":"Sensors"},{"key":"ref10","first-page":"527","article-title":"Pleasure, pain, desire, and dread: hidden core processes of emotion","volume-title":"Well-being: Foundations of Hedonic Psychology","author":"Berridge","year":"1999"},{"key":"ref11","doi-asserted-by":"publisher","first-page":"382","DOI":"10.1016\/j.chb.2017.02.064","article-title":"Shopping with a robotic companion","volume":"77","author":"Bertacchini","year":"2018","journal-title":"Comput. Hum. Behav."},{"key":"ref12","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1016\/0005-7916(94)90063-9","article-title":"Measuring emotion: the self-assessment manikin and the semantic differential","volume":"25","author":"Bradley","year":"1994","journal-title":"J. Behav. Ther. Exp. Psychiatry"},{"key":"ref13","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"ref14","doi-asserted-by":"publisher","first-page":"401","DOI":"10.3390\/s18020401","article-title":"A brief review of facial emotion recognition based on visual information","volume":"18","author":"Byoung","year":"2018","journal-title":"Sensors"},{"key":"ref16","doi-asserted-by":"publisher","first-page":"1052","DOI":"10.1109\/TSMCA.2011.2116000","article-title":"Emotion assessment from physiological signals for adaptation of game difficulty","volume":"41","author":"Chanel","year":"2011","journal-title":"IEEE Trans. Syst. Man Cybern. A Syst. Hum."},{"key":"ref17","doi-asserted-by":"publisher","first-page":"321","DOI":"10.48550\/arXiv.1106.1813","article-title":"SMOTE: synthetic minority over-sampling technique","volume":"16","author":"Chawla","year":"2002","journal-title":"J. Artif. Intell. Res."},{"key":"ref18","doi-asserted-by":"publisher","first-page":"17507","DOI":"10.3390\/s150717507","article-title":"Evaluation of fear using nonintrusive measurement of multimodal sensors","volume":"15","author":"Choi","year":"2015","journal-title":"Sensors"},{"key":"ref19","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1007\/BF00994018","article-title":"Support-vector networks","volume":"20","author":"Cortes","year":"1995","journal-title":"Mach. Learn."},{"key":"ref21","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1177\/1534582305276837","article-title":"Brain lateralization of emotional processing: historical roots and a future incorporating \u201cdominance\u201d","volume":"4","author":"Demaree","year":"2005","journal-title":"Behav. Cogn. Neurosci. Rev."},{"key":"ref22","doi-asserted-by":"publisher","first-page":"306","DOI":"10.5772\/56344","article-title":"Sensor feature selection and combination for stress identification using combinatorial fusion","volume":"10","author":"Deng","year":"2013","journal-title":"Int. J. Adv. Robot. Syst."},{"key":"ref23","article-title":"Adam: a method for stochastic optimization","author":"Diederik","year":"2015"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1111\/j.1467-9450.2007.00586.x","article-title":"Speech anxiety and rapid emotional reactions to angry and happy facial expressions","volume":"48","author":"Dimberg","year":"2007","journal-title":"Scand. J. Psychol."},{"key":"ref26","doi-asserted-by":"publisher","first-page":"325","DOI":"10.1017\/S0048577299980344","article-title":"Facial flushing during provocation in women","volume":"36","author":"Drummond","year":"1999","journal-title":"Psychophysiology"},{"key":"ref27","doi-asserted-by":"publisher","first-page":"592","DOI":"10.3390\/s20030592","article-title":"Human emotion recognition: review of sensors and methods","volume":"20","author":"Dzedzickis","year":"2020","journal-title":"Sensors"},{"key":"ref28","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1109\/ICBME.2015.7404123","article-title":"Machine learning-based signal processing using physiological signals for stress detection","volume-title":"Proceedings of 22nd Iranian conference on biomedical engineering (ICBME)","author":"Ghaderi","year":"2015"},{"key":"ref29","doi-asserted-by":"publisher","first-page":"5422","DOI":"10.1109\/JSEN.2023.3237383","article-title":"Automatic eye-blink and muscular artifact detection and removal from EEG signals using k-nearest neighbour classifier and long short-term memory networks","volume":"23","author":"Ghosh","year":"2023","journal-title":"IEEE Sensors J."},{"key":"ref30","doi-asserted-by":"publisher","first-page":"602","DOI":"10.1016\/j.neunet.2005.06.042","article-title":"Framewise phoneme classification with bidirectional LSTM and other neural network architectures","volume":"18","author":"Graves","year":"2005","journal-title":"Neural Netw."},{"key":"ref31","doi-asserted-by":"publisher","first-page":"1334","DOI":"10.1016\/j.jnca.2006.09.007","article-title":"Bi-modal emotion recognition from expressive face and body gestures","volume":"30","author":"Gunes","year":"2007","journal-title":"J. Netw. Comput. Appl."},{"key":"ref32","doi-asserted-by":"crossref","first-page":"886","DOI":"10.1037\/0022-3514.81.5.886","article-title":"Cardiovascular responses of embarrassment and effects of emotional suppression in a social setting","volume":"81","author":"Harris","year":"2001","journal-title":"J. Pers. Soc. Psychol."},{"key":"ref33","first-page":"770","article-title":"Deep residual learning for image recognition","author":"He","year":"2016"},{"key":"ref34","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1109\/TITS.2005.848368","article-title":"Detecting stress during real-world driving tasks using physiological sensors","volume":"6","author":"Healey","year":"2005","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref35","author":"Healey","year":"2010"},{"key":"ref36","doi-asserted-by":"crossref","first-page":"32","DOI":"10.4018\/978-1-59140-863-5.ch003","article-title":"Combinatorial fusion analysis: methods and practice of combining multiple scoring systems","volume-title":"Advanced data mining technologies in bioinformatics","author":"Hsu","year":"2006"},{"key":"ref37","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1109\/IISR.2018.8535774","article-title":"Research on the characteristics of acrophobia in virtual altitude environment","volume-title":"Proceedings of the 2018 IEEE international conference on intelligence and safety for robotics (ISR)","author":"Hu","year":"2018"},{"key":"ref38","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1016\/0301-0511(90)90079-C","article-title":"Psychophysiological response patterns to positive and negative film stimuli","volume":"31","author":"Hubert","year":"1990","journal-title":"Biol. Psychol."},{"key":"ref39","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1093\/mind\/os-IX.34.188","article-title":"What is an emotion?","volume":"9","author":"James","year":"1884","journal-title":"Mind"},{"key":"ref40","year":"2015"},{"key":"ref41","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/T-AFFC.2011.15","article-title":"DEAP: a database for emotion analysis using physiological signals","volume":"3","author":"Koelstra","year":"2012","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref42","first-page":"1","article-title":"The effects virtual reality has on physiological responses as compared to two-dimensional video","volume":"1","author":"Kometer","year":"2010","journal-title":"J. Adv. Stud. Sci."},{"key":"ref43","volume-title":"International affective picture system (IAPS): affective ratings of pictures and instruction manual","author":"Lang","year":"2008"},{"key":"ref44","first-page":"2682","article-title":"Understanding the disharmony between dropout and batch normalization by variance shift","volume-title":"Proceedings of the IEEE \/ CVF computer vision and pattern recognition conference (CVPR)","author":"Li","year":"2019"},{"key":"ref45","doi-asserted-by":"publisher","first-page":"506","DOI":"10.1080\/10447310902963944","article-title":"Dynamic difficulty adjustment in computer games through real-time anxiety-based affective feedback","volume":"25","author":"Liu","year":"2009","journal-title":"Int. J. Hum. Comput. Interact."},{"key":"ref46","first-page":"3730","article-title":"Deep learning face attributes in the wild","volume-title":"Proceedings of the IEEE international conference on computer vision","author":"Liu","year":"2015"},{"key":"ref47","first-page":"448","article-title":"Batch normalization: accelerating deep network training by reducing internal covariate shift","volume-title":"Proceedings of the 32nd international conference on international conference on machine learning (ICML)","author":"Loffe","year":"2015"},{"key":"ref48","volume-title":"Emotion detection and recognition market by technology, software tool, service, application area, end user, and region - global forecast to 2021","year":"2017"},{"key":"ref49","doi-asserted-by":"publisher","first-page":"1587","DOI":"10.3390\/s21051587","article-title":"Fear recognition for women using a reduced set of physiological signals","volume":"21","author":"Miranda","year":"2021","journal-title":"Sensors"},{"key":"ref50","first-page":"807","article-title":"Rectified linear units improve restricted boltzmann machines","volume-title":"Proceedings of the 27th international conference on machine learning (ICML)","author":"Nair","year":"2010"},{"key":"ref51","first-page":"371","article-title":"A Chatbot for psychiatric counseling in mental healthcare service based on emotional dialogue analysis and sentence generation","volume-title":"Proceedings of 18th IEEE international conference on Mobile data management (MDM)","author":"Oh","year":"2017"},{"key":"ref52","doi-asserted-by":"publisher","first-page":"68755","DOI":"10.1109\/ACCESS.2022.3186318","article-title":"Stress level estimation based on physiological signals for virtual reality applications","volume":"10","author":"Orozco-Mora","year":"2022","journal-title":"IEEE Access"},{"key":"ref53","doi-asserted-by":"publisher","first-page":"201","DOI":"10.5897\/IJMMS.9000091","article-title":"Emotion recognition based on the asymmetric left and right activation","volume":"3","author":"Park","year":"2011","journal-title":"Int. J. Med. Med Sci."},{"key":"ref54","doi-asserted-by":"publisher","first-page":"2825","DOI":"10.5555\/1953048.2078195","article-title":"Scikit-learn: machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref55","doi-asserted-by":"publisher","first-page":"4519","DOI":"10.3390\/s21134519","article-title":"Machine learning methods for fear classification based on physiological features","volume":"21","author":"Petrescu","year":"2021","journal-title":"Sensors"},{"key":"ref56","first-page":"90","article-title":"Affective wearables","volume-title":"Proceedings of the 1st IEEE international symposium on wearable computers (ISWC)","author":"Picard","year":"1997"},{"key":"ref57","doi-asserted-by":"crossref","first-page":"1175","DOI":"10.1109\/34.954607","article-title":"Toward machine emotional intelligence: analysis of affective physiological state","volume":"23","author":"Picard","year":"2001","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref58","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1300\/J075v08n02_03","article-title":"Health consequences of stress","volume":"8","author":"Quick","year":"1987","journal-title":"J. Organ. Behav. Manage."},{"key":"ref59","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1016\/0092-6566(77)90037-X","article-title":"Evidence for a three-factor theory of emotions","volume":"11","author":"Russell","year":"1977","journal-title":"J. Res. Pers."},{"key":"ref60","doi-asserted-by":"publisher","first-page":"995","DOI":"10.1016\/j.future.2017.11.013","article-title":"Raspberry Pi assisted face recognition framework for enhanced law-enforcement services in smart cities","volume":"108","author":"Sajjad","year":"2020","journal-title":"Future Gener. Comput. Syst."},{"key":"ref61","doi-asserted-by":"publisher","first-page":"1039","DOI":"10.3390\/electronics8091039","article-title":"Anxiety level recognition for virtual reality therapy system using physiological signals","volume":"8","author":"\u0160alkevicius","year":"2019","journal-title":"Electronics"},{"key":"ref62","first-page":"1","article-title":"Towards emotion recognition from electroencephalographic signals","volume-title":"Proceedings of the 3rd international conference on affective computing and intelligent interaction and workshops, Amsterdam (ACII)","author":"Schaaff","year":"2009"},{"key":"ref63","first-page":"92","article-title":"Evaluation of pooling operations in convolutional architectures for object recognition","author":"Scherer","year":"2010"},{"key":"ref64","first-page":"400","article-title":"Introducing WESAD, a multimodal dataset for wearable stress and affect detection","volume-title":"Proceedings of the 20th ACM international conference on multimodal interaction (ICMI)","author":"Schmidt","year":"2018"},{"key":"ref65","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1145\/3129340","article-title":"Speech emotion recognition: two decades in a nutshell, benchmarks and ongoing trends","volume":"61","author":"Schuller","year":"2018","journal-title":"Commun. J. Ass. Comp. Mach."},{"key":"ref66","doi-asserted-by":"publisher","first-page":"2074","DOI":"10.3390\/s18072074","article-title":"A review of emotion recognition using physiological signals","volume":"18","author":"Shu","year":"2018","journal-title":"Sensors"},{"key":"ref67","volume-title":"Speech emotion recognition","author":"Sidorova","year":"2007"},{"key":"ref68","article-title":"Very deep convolutional networks for large-scale image recognition","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)","author":"Simonyan","year":"2014"},{"key":"ref69","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1109\/T-AFFC.2011.25","article-title":"A multimodal database for affect recognition and implicit tagging","volume":"3","author":"Soleymani","year":"2012","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref70","doi-asserted-by":"publisher","first-page":"236","DOI":"10.1016\/S0167-2789(01)00386\u20134","article-title":"Synchronization likelihood: an unbiased measure of generalized synchronization in multivariate data sets","volume":"163","author":"Stam","year":"2002","journal-title":"Phys. D Nonlinear Phenomena"},{"key":"ref71","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1109\/TAFFC.2016.2625250","article-title":"ASCERTAIN: emotion and personality recognition using commercial sensors","volume":"9","author":"Subramanian","year":"2018","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref73","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1088\/1741-2552\/abcdbd","article-title":"Using convolutional neural networks to decode EEG-based functional brain network with different severity of acrophobia","volume":"18","author":"Wang","year":"2020","journal-title":"J. Neural Eng."},{"key":"ref74","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1475-925X-5-51","article-title":"Data mining: practical machine learning tools and techniques","volume":"5","author":"Witten","year":"2005","journal-title":"Biomed. Eng. Online"},{"key":"ref75","volume-title":"The practice of behavior therapy","author":"Wolpe","year":"1969"},{"key":"ref76","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1016\/j.procs.2017.12.003","article-title":"An emotion recognition model based on facial recognition in virtual learning environment","volume":"125","author":"Yang","year":"2018","journal-title":"Proc. Comput. Sci."},{"key":"ref77","first-page":"4303","article-title":"Robots learn social skills: end-to-end learning of co-speech gesture generation for humanoid robots","volume-title":"The proceedings of IEEE international conference on robotics and automation (ICRA)","author":"Yoon","year":"2019"},{"key":"ref78","doi-asserted-by":"crossref","first-page":"197","DOI":"10.4108\/icst.mobicase.2014.257786","article-title":"Convolutional neural networks for human activity recognition using mobile sensors","volume-title":"Proceedings of the 6th international conference on mobile computing, applications and services (MobiCASE)","author":"Zeng","year":"2014"},{"key":"ref79","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.inffus.2020.01.011","article-title":"Emotion recognition using multi-modal data and machine learning techniques: a tutorial and review","volume":"59","author":"Zhang","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref80","doi-asserted-by":"publisher","first-page":"1110","DOI":"10.1109\/TCYB.2018.2797176","article-title":"EmotionMeter: a multimodal framework for recognizing human emotions","volume":"49","author":"Zheng","year":"2019","journal-title":"IEEE Trans. Cybernetics"},{"key":"ref81","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1155\/2018\/5681652","article-title":"Dynamic difficulty adjustment (DDA) in computer games: a review","volume":"2018","author":"Zohaib","year":"2018","journal-title":"Adv. Hum. Comput. Interact."}],"container-title":["Frontiers in Psychology"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpsyg.2023.1141801\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,21]],"date-time":"2024-10-21T16:27:36Z","timestamp":1729528056000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpsyg.2023.1141801\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,1]]},"references-count":76,"alternative-id":["10.3389\/fpsyg.2023.1141801"],"URL":"https:\/\/doi.org\/10.3389\/fpsyg.2023.1141801","relation":{},"ISSN":["1664-1078"],"issn-type":[{"value":"1664-1078","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,1]]},"article-number":"1141801"}}