{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,24]],"date-time":"2025-08-24T01:19:42Z","timestamp":1755998382396},"reference-count":80,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2020,10,20]],"date-time":"2020-10-20T00:00:00Z","timestamp":1603152000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,4,19]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Human affect recognition (HAR) using images of facial expression and electrocardiogram (ECG) signal plays an important role in predicting human intention. This system improves the performance of the system in applications like the security system, learning technologies and health care systems. The primary goal of our work is to recognize individual affect states automatically using the multilayered binary structured support vector machine (MBSVM), which efficiently classify the input into one of the four affect classes, relax, happy, sad and angry. The classification is performed efficiently by designing an efficient support vector machine (SVM) classifier in multilayer mode operation. The classifier is trained using the 8-fold cross-validation method, which improves the learning of the classifier, thus increasing its efficiency. The classification and recognition accuracy is enhanced and also overcomes the drawback of \u2018facial mimicry\u2019 by using hybrid features that are extracted from both facial images (visual elements) and physiological signal ECG (signal features). The reliability of the input database is improved by acquiring the face images and ECG signals experimentally and by inducing emotions through image stimuli. The performance of the affect recognition system is evaluated using the confusion matrix, obtaining the classification accuracy of 96.88%.<\/jats:p>","DOI":"10.1093\/comjnl\/bxaa125","type":"journal-article","created":{"date-parts":[[2020,9,12]],"date-time":"2020-09-12T19:09:54Z","timestamp":1599937794000},"page":"897-917","source":"Crossref","is-referenced-by-count":24,"title":["Smart Affect Recognition System for Real-Time Biometric Surveillance Using Hybrid Features and Multilayered Binary Structured Support Vector Machine"],"prefix":"10.1093","volume":"65","author":[{"given":"Thamba Meshach","family":"W","sequence":"first","affiliation":[{"name":"Department of Computer Science and Engineering, Prathyusha Engineering College, Aranvoyalkuppam, Thiruvallur District, Chennai 602025, Tamilnadu, India"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hemajothi","family":"S","sequence":"additional","affiliation":[{"name":"Department of Electronics and Communication Engineering, Prince Shri Venkateshwara Padmavathy Engineering College, Medavakkam-Mambakkam Road, Ponmar, Chennai 600127, Tamilnadu, India"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mary Anita","family":"E A","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Christ University, Mysore Rd, Kumbalgodu, Bangalore, Karnataka 560074, India"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2020,10,20]]},"reference":[{"key":"2022041811431255600_ref1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1371\/journal.pone.0110763","article-title":"When age matters: differences in facial mimicry and autonomic responses to peers\u2019 emotions in teenagers and adults","volume":"9","author":"Ardizzi","year":"2014","journal-title":"PLOS ONE"},{"key":"2022041811431255600_ref2","doi-asserted-by":"crossref","first-page":"410","DOI":"10.1109\/CSPA.2011.5759912","volume-title":"IEEE 7th International Colloquium on Signal Processing and its Application, Malaysia, 4\u20136 March 2011","author":"Jerritta","year":"2011"},{"key":"2022041811431255600_ref3","first-page":"61","article-title":"An inter-domain study for arousal recognition from physiological signals","volume":"42","author":"Gjoreski","year":"2018","journal-title":"Informatica"},{"key":"2022041811431255600_ref4","first-page":"497","volume-title":"In Proc. of the 2009 Int. Conf. on Information Technology and Computer Science","author":"Cai","year":"2009"},{"key":"2022041811431255600_ref5","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1016\/j.eswa.2007.09.002","article-title":"Facial expression recognition \u2013 a real-time approach","volume":"36","author":"Geetha","year":"2009","journal-title":"Expert Syst. Appl."},{"key":"2022041811431255600_ref6","first-page":"4106","volume-title":"Int. Conf. on Robotics and Automation","author":"Alazrai","year":"2012"},{"key":"2022041811431255600_ref7","author":"Lu","year":"2003"},{"key":"2022041811431255600_ref8","first-page":"1337","article-title":"Emotion detection in human beings using ECG signals","volume":"4","author":"Baby Shalini","year":"2013","journal-title":"IJETT"},{"key":"2022041811431255600_ref9","doi-asserted-by":"crossref","first-page":"2528","DOI":"10.1109\/TMM.2016.2598092","article-title":"A deep neural network-driven feature learning method for multi-view facial expression recognition","volume":"18","author":"Zhang","year":"2016","journal-title":"IEEE Trans Multimedia"},{"key":"2022041811431255600_ref10","volume-title":"Emotions in social psychology: essential readings","author":"Parrott","year":"2001"},{"key":"2022041811431255600_ref11","doi-asserted-by":"crossref","first-page":"1063","DOI":"10.1037\/0022-3514.54.6.1063","article-title":"Development and validation of brief measures of positive and negative affect: the panas scales","volume":"54","author":"Watson","year":"1982","journal-title":"J. Pers. Soc. Psychol"},{"key":"2022041811431255600_ref12","first-page":"529","article-title":"A psychoevolutionary theory of emotions","volume":"21","author":"Plutchik","year":"1982","journal-title":"Soc. Sci. Med."},{"key":"2022041811431255600_ref13","first-page":"1354","article-title":"Driver Frustration Detection From Audio and Video","volume-title":"Proc. Int. Joint Conf. on Artificial Intelligence","author":"Abdic","year":"2016"},{"key":"2022041811431255600_ref14","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1109\/TPAMI.2008.52","article-title":"A survey of affect recognition methods: audio, visual, and spontaneous expressions","volume":"31","author":"Zeng","year":"2009","journal-title":"Trans. Pattern Anal. Mach. Intell."},{"key":"2022041811431255600_ref15","doi-asserted-by":"crossref","first-page":"2168","DOI":"10.1109\/ICCVW.2011.6130516","volume-title":"IEEE Int. Conf. on Computer Vision Workshops (ICCV Workshops)","author":"Jeni","year":"2011"},{"key":"2022041811431255600_ref16","first-page":"21","article-title":"FER based on local binary patterns and local fisher discriminant analysis","volume":"8","author":"Zhang","year":"2012","journal-title":"WSEAS Trans. Signal Process."},{"key":"2022041811431255600_ref17","volume-title":"IEEE Int. Conf. and Workshop Automatic Face and Gesture Recognition (FG)","author":"Liu","year":"2013"},{"key":"2022041811431255600_ref18","doi-asserted-by":"crossref","first-page":"2106","DOI":"10.1109\/ICCVW.2011.6130508","volume-title":"IEEE Int. Conf. Computer Vision Workshops (ICCV Workshops), Barcelona","author":"Dhall","year":"2011"},{"key":"2022041811431255600_ref19","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1109\/TAFFC.2016.2625250","article-title":"Ascertain: emotion and personality recognition using commercial sensors","volume":"9","author":"Subramanian","year":"2018","journal-title":"IEEE T. Affect. Comput."},{"key":"2022041811431255600_ref20","volume-title":"IEEE T. Affect. Comput.","author":"Miranda Correa","year":"2018"},{"key":"2022041811431255600_ref21","first-page":"127","article-title":"Facial feature detection using Haar classifier","volume":"21","author":"Wilson","year":"2006","journal-title":"Comput. Sci. Colleges"},{"key":"2022041811431255600_ref22","first-page":"265","article-title":"Understanding color models: a review","volume":"2","author":"Ibraheem","year":"2012","journal-title":"ARPN J. Sci. Technol."},{"key":"2022041811431255600_ref23","author":"Hassan","year":"2010"},{"key":"2022041811431255600_ref24","doi-asserted-by":"crossref","first-page":"1215","DOI":"10.1109\/TPAMI.2003.1233896","article-title":"Face detection and tracking in a video by propagating detection probabilities","volume":"25","author":"Verma","year":"2003","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"2022041811431255600_ref25","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1016\/0005-7916(94)90063-9","article-title":"Measuring emotion: the self-assessment manikin and the semantic differential","volume":"25","author":"Bradley","year":"1994","journal-title":"J. Behav. Ther. Exp. Psychiatry"},{"key":"2022041811431255600_ref26","first-page":"71","volume-title":"IEEE Int. Symposium on Robot and Human Interactive Communication, Munich","author":"Abdat","year":"2008"},{"key":"2022041811431255600_ref27","first-page":"9","article-title":"Facial expression analysis using active shape model","volume":"8","author":"Shbib","year":"2015","journal-title":"Int. J. Signal Process. Image Process. Pattern Recognit."},{"key":"2022041811431255600_ref28","volume-title":"Int. Conf. Computational Science, Subject: Computer Vision and Pattern Recognition","author":"Bhattacharjee","year":"2005"},{"key":"2022041811431255600_ref29","first-page":"1","volume-title":"Int. Conf. Intelligent Human Computer Interaction (IHCI), Kharagpur","author":"Happy","year":"2012"},{"key":"2022041811431255600_ref30","first-page":"569","article-title":"Automatic facial expression recognition on a single 3D face by exploring shape deformation","author":"Gong","year":"2009","journal-title":"ACM Multimedia"},{"key":"2022041811431255600_ref31","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.knosys.2016.07.011","article-title":"Facial expression recognition with automatic segmentation of face regions using a fuzzy based classification approach","volume":"110","author":"Hernandez-Matamoros","year":"2016","journal-title":"Knowl.-Based Syst."},{"key":"2022041811431255600_ref32","first-page":"1321","article-title":"Kernel locality preserving symmetrical weighted fisher discriminant analysis based subspace approach for expression recognition","volume":"19","author":"Hegde","year":"2016","journal-title":"Eng. Sci. Technol."},{"key":"2022041811431255600_ref33","article-title":"Facial expression recognition using a combination of multiple facial features and support vector machine","volume":"22","author":"Hung-Hsu","year":"2017","journal-title":"Appl. Soft Comput"},{"key":"2022041811431255600_ref34","first-page":"595","author":"Ghandi","year":"2009"},{"key":"2022041811431255600_ref35","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1109\/T-AFFC.2011.28","article-title":"ECG pattern analysis for emotion detection","volume":"3","author":"Agrafioti","year":"2012","journal-title":"IEEE Trans. Affect. Comput."},{"key":"2022041811431255600_ref36","doi-asserted-by":"crossref","first-page":"7714","DOI":"10.3390\/s130607714","article-title":"Geometric feature-based FER in image sequences using multiclass AdaBoost and support vector machines","volume":"13","author":"Ghimire","year":"2013","journal-title":"Sensors"},{"key":"2022041811431255600_ref37","author":"Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology","year":"1997"},{"key":"2022041811431255600_ref38","volume-title":"The Nature of Statistical Learning Theory","author":"Vapnik","year":"1999","edition":"2nd"},{"key":"2022041811431255600_ref39","doi-asserted-by":"crossref","first-page":"696","DOI":"10.1109\/TNN.2006.872343","article-title":"Binary Tree of SVM: a new fast multiclass training and classification algorithm","volume":"17","author":"Fei","year":"2006","journal-title":"IEEE Trans. Neural. Netw. Learn. Syst."},{"key":"2022041811431255600_ref40","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1016\/j.ipm.2009.03.002","article-title":"A systematic analysis of performance measures for classification tasks","volume":"45","author":"Sokolova","year":"2009","journal-title":"Information processing and management"},{"key":"2022041811431255600_ref41","doi-asserted-by":"crossref","first-page":"1","DOI":"10.5121\/ijdkp.2015.5201","article-title":"A review on evaluation metrics for data classification evaluations","volume":"5","author":"Hossin","year":"2015","journal-title":"Int. J. Data Min. Knowl. Manag. Process (IJDKP)"},{"key":"2022041811431255600_ref42","article-title":"A survey on human face expression recognition techniques","author":"Michael Revina","year":"2018","journal-title":"J. King Saud Univ. Sci"},{"key":"2022041811431255600_ref43","doi-asserted-by":"crossref","first-page":"2310","DOI":"10.1152\/jappl.1993.75.5.2310","article-title":"Important influence of respiration on human RR interval power spectra is largely ignored","volume":"75","author":"Brown","year":"1993","journal-title":"J. Appl. Physiol."},{"key":"2022041811431255600_ref44","doi-asserted-by":"crossref","first-page":"537","DOI":"10.1161\/01.CIR.81.2.537","article-title":"Continuous 24-h assessment of the neural regulation of systemic arterial pressure and RR variabilities in ambulant subjects","volume":"81","author":"Furlan","year":"1990","journal-title":"Circulation"},{"key":"2022041811431255600_ref45","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1111\/j.1542-474X.2005.10101.x","article-title":"Heart rate variability: measurement and clinical utility","volume":"10","author":"Kleiger","year":"2005","journal-title":"Ann. Noninvas. Electro."},{"key":"2022041811431255600_ref46","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1037\/1528-3542.6.1.150","article-title":"Differential subjective and psychophysiological responses to socially and nonsocially generated emotional stimuli","volume":"6","author":"Britton","year":"2006","journal-title":"Emotion"},{"key":"2022041811431255600_ref47","doi-asserted-by":"crossref","first-page":"527","DOI":"10.1007\/s40846-017-0238-0","article-title":"Differences of heart rate variability between happiness and sadness emotion states: a pilot study","volume":"37","author":"Shi","year":"2017","journal-title":"J. Med. Biol. Eng."},{"key":"2022041811431255600_ref48","volume-title":"Human emotion recognition using heart rate variability analysis with spectral bands based on respiration. Paper presented at the 2015 37th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (EMBC)","author":"Valderas","year":"2015"},{"key":"2022041811431255600_ref49","doi-asserted-by":"crossref","DOI":"10.1088\/1361-6579\/ab1887","article-title":"Heart rate variability monitoring for emotion and disorders of emotion","volume":"40","author":"Zhu","year":"2019","journal-title":"Physiol. Meas."},{"key":"2022041811431255600_ref50","author":"Khan, F.","year":"2018"},{"key":"2022041811431255600_ref51","volume-title":"International Affective Picture System (IAPS): Technical Manual and Affective Ratings","author":"Lang","year":"1999"},{"key":"2022041811431255600_ref52","volume-title":"Proc. of the 2018 IEEE Smart World, Ubiquitous Intelligence Computing, Advanced Trusted Computing, Scalable Computing Communications, Cloud Big Data Computing, Internet of People and Smart City Innovation (SmartWorld\/SCALCOM\/UIC\/ATC\/CBDCom\/IOP\/SCI)","author":"Zhao","year":"2018"},{"key":"2022041811431255600_ref53","volume-title":"Out of the Lab and into the Fray: Towards Modeling Emotion in Everyday Life, Pervasive Computing, Lecture Notes in Computer Science, 6030","author":"Healey","year":"2010"},{"key":"2022041811431255600_ref54","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1159\/000119004","article-title":"The Trier Social Stress Test \u2014 a tool for investigating psychobiological stress responses in a laboratory setting","volume":"28","author":"Kirschbaum","year":"1993","journal-title":"Neuropsychobiology"},{"key":"2022041811431255600_ref55","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/T-AFFC.2011.15","article-title":"A database for emotion analysis; using physiological signals","volume":"3","author":"Koelstra","year":"2012","journal-title":"IEEE Trans. Affect. Comput"},{"key":"2022041811431255600_ref56","first-page":"493","volume-title":"Proc. of the ACM Int. Conf. on Ubiquitous Computing, UbiComp (Conf.), 2015","author":"Hovsepian","year":"2015"},{"key":"2022041811431255600_ref57","first-page":"14","volume-title":"IEEE Int. Conf. on Pervasive Computing and Communication Workshops (PerCom Workshops), Sydney, Australia","author":"Zenonos","year":"2016"},{"key":"2022041811431255600_ref58","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1016\/j.bandc.2007.02.007","article-title":"The effects of stress and stress hormones on human cognition: Implications for the field of brain and cognition","volume":"65","author":"Lupien","year":"2007","journal-title":"Brain Cogn."},{"key":"2022041811431255600_ref59","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1109\/TITB.2011.2169804","article-title":"Development and evaluation of an ambulatory stress monitor based on wearable sensors","volume":"16","author":"Choi","year":"2012","journal-title":"IEEE Trans. Inf. Technol. Biomed."},{"key":"2022041811431255600_ref60","first-page":"589","volume-title":"Proc. of Measuring Behavior","author":"Uyl","year":"2005"},{"key":"2022041811431255600_ref61","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1007\/s12559-013-9239-7","article-title":"Classification of music-induced emotions based on information fusion of forehead biosignals and electrocardiogram","volume":"6","author":"Naji","year":"2014","journal-title":"Cogn. Comput."},{"key":"2022041811431255600_ref62","first-page":"31","volume-title":"Proc. of the 4th IASTED Int. Conf. on Telehealth and Assistive Technologies, Telehealth","author":"Dai","year":"2008"},{"key":"2022041811431255600_ref63","volume-title":"Emotion Recognition Using Bio-sensors: First Steps Towards an Automatic System, Affective Dialogue Systems, ADS 2004, Lecture Notes in Computer Science, 3068","author":"Haag","year":"2004"},{"key":"2022041811431255600_ref64","doi-asserted-by":"crossref","first-page":"2067","DOI":"10.1109\/TPAMI.2008.26","article-title":"Emotion recognition based on physiological changes in music listening","volume":"30","author":"Kim","year":"2008","journal-title":"Trans. Pattern Anal. Mach. Intell."},{"key":"2022041811431255600_ref65","first-page":"1","article-title":"Using noninvasive wearable computers to recognize human emotions from physiological signals","volume":"11","author":"Lisetti","year":"2004","journal-title":"EURASIP J. Adv. Signal Process"},{"key":"2022041811431255600_ref66","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1109\/CSIE.2009.130","volume-title":"WRI World Congress on Computer Science and Information Engineering","author":"Wan-Hui","year":"2009"},{"key":"2022041811431255600_ref67","author":"Maaoui","year":"2010"},{"key":"2022041811431255600_ref68","doi-asserted-by":"crossref","DOI":"10.5772\/4754","volume-title":"Bimodal Emotion Recognition Using Speech and Physiological Changes","author":"Kim","year":"2007"},{"key":"2022041811431255600_ref69","doi-asserted-by":"crossref","first-page":"991","DOI":"10.1109\/TRO.2007.904899","article-title":"Affective state estimation for human\u2013robot interaction","volume":"23","author":"Kulic","year":"2007","journal-title":"IEEE Trans. Robot"},{"key":"2022041811431255600_ref70","doi-asserted-by":"crossref","first-page":"1175","DOI":"10.1109\/34.954607","article-title":"Toward machine emotional intelligence: analysis of affective physiological state","volume":"23","author":"Picard","year":"2001","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"2022041811431255600_ref71","doi-asserted-by":"crossref","first-page":"1672","DOI":"10.1155\/S1110865704406192","article-title":"Using noninvasive wearable computers to recognize human emotions from physiological signals","volume":"2004","author":"Lisetti","year":"2004","journal-title":"EURASIP J. Adv. Signal Process."},{"key":"2022041811431255600_ref72","first-page":"818","volume-title":"Lecture Notes in Computer Science, 3610","author":"Yoo","year":"2005"},{"key":"2022041811431255600_ref73","first-page":"437","volume-title":"Lecture Notes in Computer Science, 282","author":"Li","year":"2006"},{"key":"2022041811431255600_ref74","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1007\/s10044-006-0025-y","article-title":"An empirical study of machine learning techniques for affect recognition in human robot interaction","volume":"9","author":"Rani","year":"2006","journal-title":"Pattern Anal. Appl."},{"key":"2022041811431255600_ref75","doi-asserted-by":"crossref","first-page":"662","DOI":"10.1016\/j.ijhsc.2008.04.003","article-title":"Physiology-based affect recognition for computer-assisted intervention of children with autism spectrum disorder","volume":"66","author":"Liu","year":"2008","journal-title":"Int. J. Hum.-Comput. Stud."},{"key":"2022041811431255600_ref76","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1016\/j.bspc.2010.12.001","article-title":"An integrated system based on physiological signals for the assessment of affective states in patients with anxiety disorders","volume":"6","author":"Katsis","year":"2011","journal-title":"Biomed Signal Process Control"},{"key":"2022041811431255600_ref77","first-page":"538","volume-title":"Int. Conf. on Communications and Signal Processing (ICCSP), April 3\u20135 2014","author":"Piparsaniyan","year":"2014"},{"key":"2022041811431255600_ref78","doi-asserted-by":"crossref","first-page":"3383","DOI":"10.1016\/j.eswa.2013.11.041","article-title":"A neural-AdaBoost based facial expression recognition system","volume":"41","author":"Owusu","year":"2014","journal-title":"Expert Syst. Appl"},{"key":"2022041811431255600_ref79","first-page":"2265","volume-title":"22nd Signal Processing and Communications Applications Conf. (SIU), Trabzon","author":"Abdulrahman","year":"2014"},{"key":"2022041811431255600_ref80","doi-asserted-by":"crossref","first-page":"498","DOI":"10.4028\/www.scientific.net\/AMM.631-632.498","article-title":"Recognition of facial expression via kernel PCA network","volume":"631","author":"Hu","year":"2014","journal-title":"Appl. Mech. Mater."}],"container-title":["The Computer Journal"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/comjnl\/article-pdf\/65\/4\/897\/43377412\/bxaa125.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/comjnl\/article-pdf\/65\/4\/897\/43377412\/bxaa125.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,18]],"date-time":"2022-04-18T11:44:44Z","timestamp":1650282284000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/comjnl\/article\/65\/4\/897\/5930867"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,20]]},"references-count":80,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2020,10,20]]},"published-print":{"date-parts":[[2022,4,19]]}},"URL":"https:\/\/doi.org\/10.1093\/comjnl\/bxaa125","relation":{},"ISSN":["0010-4620","1460-2067"],"issn-type":[{"value":"0010-4620","type":"print"},{"value":"1460-2067","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,4]]},"published":{"date-parts":[[2020,10,20]]}}}