{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,4]],"date-time":"2025-12-04T10:00:52Z","timestamp":1764842452405},"reference-count":42,"publisher":"Oxford University Press (OUP)","issue":"7","license":[{"start":{"date-parts":[[2021,4,19]],"date-time":"2021-04-19T00:00:00Z","timestamp":1618790400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,7,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Affective computing is a key research topic in artificial intelligence which is applied to psychology and machines. It consists of the estimation and measurement of human emotions. A person\u2019s body language is one of the most significant sources of information during job interview, and it reflects a deep psychological state that is often missing from other data sources. In our work, we combine two tasks of pose estimation and emotion classification for emotional body gesture recognition to propose a deep multi-stage architecture that is able to deal with both tasks. Our deep pose decoding method detects and tracks the candidate\u2019s skeleton in a video using a combination of depthwise convolutional network and detection-based method for 2D pose reconstruction. Moreover, we propose a representation technique based on the superposition of skeletons to generate for each video sequence a single image synthesizing the different poses of the subject. We call this image: \u2018history pose image\u2019, and it is used as input to the convolutional neural network model based on the Visual Geometry Group architecture. We demonstrate the effectiveness of our method in comparison with other methods in the state of the art on the standard Common Object in Context keypoint dataset and Face and Body gesture video database.<\/jats:p>","DOI":"10.1093\/comjnl\/bxab011","type":"journal-article","created":{"date-parts":[[2021,2,27]],"date-time":"2021-02-27T04:10:00Z","timestamp":1614399000000},"page":"1702-1716","source":"Crossref","is-referenced-by-count":4,"title":["Deep Multi-Stage Approach For Emotional Body Gesture Recognition In Job Interview"],"prefix":"10.1093","volume":"65","author":[{"given":"Intissar","family":"Khalifa","sequence":"first","affiliation":[{"name":"Research Team on Intelligent Machines , National Engineering School of Gabes, , Street Omar Ibn El Khattab, Zrig Eddakhlania 6029, Gabes, Tunisia"},{"name":"University of Gabes , National Engineering School of Gabes, , Street Omar Ibn El Khattab, Zrig Eddakhlania 6029, Gabes, Tunisia"},{"name":"Department of Informatics , Systems and Communication, , Viale Sarca, 336, 20126, Milan, Italy"},{"name":"University of Milano Bicocca , Systems and Communication, , Viale Sarca, 336, 20126, Milan, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ridha","family":"Ejbali","sequence":"additional","affiliation":[{"name":"Research Team on Intelligent Machines , National Engineering School of Gabes, , Street Omar Ibn El Khattab, Zrig Eddakhlania 6029, Gabes, Tunisia"},{"name":"University of Gabes , National Engineering School of Gabes, , Street Omar Ibn El Khattab, Zrig Eddakhlania 6029, Gabes, Tunisia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Raimondo","family":"Schettini","sequence":"additional","affiliation":[{"name":"Department of Informatics , Systems and Communication, , Viale Sarca, 336, 20126, Milan, Italy"},{"name":"University of Milano Bicocca , Systems and Communication, , Viale Sarca, 336, 20126, Milan, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mourad","family":"Zaied","sequence":"additional","affiliation":[{"name":"Research Team on Intelligent Machines , National Engineering School of Gabes, , Street Omar Ibn El Khattab, Zrig Eddakhlania 6029, Gabes, Tunisia"},{"name":"University of Gabes , National Engineering School of Gabes, , Street Omar Ibn El Khattab, Zrig Eddakhlania 6029, Gabes, Tunisia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2021,4,19]]},"reference":[{"key":"2022071813385330100_ref1","first-page":"53","article-title":"Communication without words","volume":"2","author":"Mehrabian","year":"1968","journal-title":"Psychol. Today"},{"key":"2022071813385330100_ref2","first-page":"42","article-title":"Kinesics, haptics, and proxemics: Aspects of non-verbal communication","volume":"20","author":"Hans","year":"2015","journal-title":"IOSR J. Humanities and Social Science"},{"key":"2022071813385330100_ref3","volume-title":"Gesture Generation by Imitation from Human Behavior to Computer Character Animation","author":"Kipp","year":"2004"},{"key":"2022071813385330100_ref4","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1515\/semi.1969.1.1.49","article-title":"The repertoire of nonverbal behavior: Categories, origins, and coding","volume":"1","author":"Ekman and Friesen","year":"2009","journal-title":"Semiotica"},{"key":"2022071813385330100_ref5","doi-asserted-by":"crossref","DOI":"10.7208\/chicago\/9780226514642.001.0001","volume-title":"Gesture and Thought","author":"McNeill","year":"2005"},{"key":"2022071813385330100_ref6","doi-asserted-by":"crossref","first-page":"1018","DOI":"10.1109\/TMM.2014.2307169","article-title":"Hire me: Computational inference of hirability in employment interviews based on nonverbal behavior","volume":"16","author":"Nguyen","year":"2014","journal-title":"IEEE Trans. Multimedia"},{"key":"2022071813385330100_ref7","first-page":"1","article-title":"Adaptive real-time emotion recognition from body movements","volume":"5","author":"Wang","year":"2015","journal-title":"ACM. Trans. Intell. Syst. Technol."},{"key":"2022071813385330100_ref8","article-title":"Survey on emotional body gesture recognition","author":"Noroozi","year":"2018","journal-title":"J. IEEE Trans. Affect. Comput."},{"key":"2022071813385330100_ref9","doi-asserted-by":"crossref","DOI":"10.1093\/acprof:oso\/9780195374346.001.0001","volume-title":"Emotions and the Body","author":"De Gelder","year":"2016"},{"key":"2022071813385330100_ref10","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1177\/1754073917749880","article-title":"Bodily communication of emotion: Evidence for extra facial behavioral expressions and available coding systems","volume":"11","author":"Witkower","year":"2018","journal-title":"J. Emot. Rev."},{"key":"2022071813385330100_ref11","first-page":"909","volume-title":"Automatic Face and Gesture Recognition and Workshops (FG2011)","author":"Baltrusaitis","year":"2011"},{"key":"2022071813385330100_ref12","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1109\/TAFFC.2015.2396531","article-title":"Liris-accede: Avideo database for affective content analysis","volume":"6","author":"Baveye","year":"2015","journal-title":"IEEE Trans. Affect. Comput."},{"key":"2022071813385330100_ref13","volume-title":"The 18th Int. Conf. Pattern Recognition (ICPR)","author":"Gunes","year":"2006"},{"key":"2022071813385330100_ref14","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1016\/j.imavis.2012.06.014","article-title":"Recognizing expressions from face and body gesture by temporal normalized motion and appearance features","volume":"3","author":"Chen","year":"2013","journal-title":"J. Image Vision Comput."},{"key":"2022071813385330100_ref15","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1109\/TSMCB.2008.927269","article-title":"Automatic temporal segment detection and affect recognition from face and body display","volume":"39","author":"Gunes","year":"2009","journal-title":"IEEE Trans. Syst. Man Cybern. Cybern."},{"key":"2022071813385330100_ref16","doi-asserted-by":"crossref","first-page":"1334","DOI":"10.1016\/j.jnca.2006.09.007","article-title":"Bi-modal emotion recognition from expressive face and body gestures","volume":"30","author":"Gunes","year":"2007","journal-title":"J. Netw. Comput. Appl."},{"key":"2022071813385330100_ref17","first-page":"84","article-title":"ImageNet classification with deep convolutional neural networks","volume":"60","author":"Krizhevsky","year":"2017","journal-title":"Adv. Neural Inf. Proces. Syst., Commun. ACM"},{"key":"2022071813385330100_ref18","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1016\/j.neunet.2015.09.009","article-title":"Multimodal emotional state recognition using sequence dependent deep hierarchical features","volume":"72","author":"Barros","year":"2015","journal-title":"J. Neural Netw."},{"key":"2022071813385330100_ref19","first-page":"104","article-title":"Emotion-modulated attention improves expression recognition: A deep learning model","volume":"253","author":"Barros","year":"2017","journal-title":"J. Neuro Computing"},{"key":"2022071813385330100_ref20","first-page":"27","volume-title":"Int. Conf. Machine Learning and Machine Intelligence","author":"Thai Ly","year":"2018"},{"key":"2022071813385330100_ref21","volume-title":"IEEE Int. Conf. Workshops Automatic Face and Gesture Recognition (FG)","author":"Marcos-Ramiro","year":"2013"},{"key":"2022071813385330100_ref22","volume-title":"IEEE Int. Conf. Intelligent Human-Machine Systems and Cybernetics","author":"Liang","year":"2014"},{"key":"2022071813385330100_ref23","volume-title":"The 10th Int. Conf. Machine Vision (ICMV)","author":"Khalifa","year":"2017"},{"key":"2022071813385330100_ref24","article-title":"Pose-conditioned spatio-temporal attention for human action recognition","volume-title":"Computer Science, Computer Vision and Pattern Recognition","author":"Baradel","year":"2017"},{"key":"2022071813385330100_ref25","first-page":"1653","volume-title":"IEEE Conf. Computer Vision and Pattern Recognition","author":"Toshev","year":"2014"},{"key":"2022071813385330100_ref26","first-page":"274","volume-title":"The 19th Int. Conf. Parallel and Distributed Computing, Applications and Technologies (PDCAT)","author":"Khalifa","year":"2018"},{"key":"2022071813385330100_ref27","article-title":"MobileNets: Efficient convolutional neural networks for mobile vision applications","volume-title":"Computer Science, Computer Vision and Pattern Recognition","author":"Howard","year":"2017"},{"key":"2022071813385330100_ref28","article-title":"Human pose estimation with CNNs and LSTMs","author":"Coskun","year":"2016"},{"key":"2022071813385330100_ref29","volume-title":"European Conf. Computer Vision (ECCV)","author":"Bulat","year":"2016"},{"key":"2022071813385330100_ref30","volume-title":"Keypoint detection task","author":"COCO","year":"2018"},{"key":"2022071813385330100_ref31","volume-title":"Computer Science, Computer Vision and Pattern Recognition","author":"Simonyan","year":"2014"},{"key":"2022071813385330100_ref32","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90","article-title":"Deep residual learning for image recognition","author":"He","year":"2016"},{"key":"2022071813385330100_ref33","article-title":"Depthwise separable convolutions for neural machine translation","volume-title":"Computer Science, Computation and Language","author":"Kaiser","year":"2017"},{"key":"2022071813385330100_ref34","first-page":"1037","article-title":"Human activity recognition using motion history algorithm","volume":"5","author":"Hassan","year":"2014","journal-title":"Int. J. Sci. Eng. Res."},{"key":"2022071813385330100_ref35","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1007\/s00138-010-0298-4","article-title":"Motion history image: Its variants and applications","volume":"23","author":"Ahad","year":"2012","journal-title":"J. Mach. Vision Appl."},{"key":"2022071813385330100_ref36","volume-title":"Large scale visual recognition challenge 2014 (ILSVRC 2014)","author":"IMAGENET"},{"key":"2022071813385330100_ref37","volume-title":"Int. Conf. Computational Statistics","author":"Bottou","year":"2010"},{"key":"2022071813385330100_ref38","article-title":"The effectiveness of data augmentation in image classification using deep learning","volume-title":"Computer Science, Computer Vision and Pattern Recognition","author":"Perez","year":"2017"},{"key":"2022071813385330100_ref39","volume-title":"TensorFlow 1 Detection Model Zoo","author":"Shi"},{"key":"2022071813385330100_ref40","volume-title":"The FABO database","author":"Gunes"},{"key":"2022071813385330100_ref41","author":"About Keras"},{"key":"2022071813385330100_ref42","author":"Optimizers"}],"container-title":["The Computer Journal"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/comjnl\/article-pdf\/65\/7\/1702\/44921933\/bxab011.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/comjnl\/article-pdf\/65\/7\/1702\/44921933\/bxab011.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,7,18]],"date-time":"2022-07-18T13:40:13Z","timestamp":1658151613000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/comjnl\/article\/65\/7\/1702\/6236092"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,19]]},"references-count":42,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2021,4,19]]},"published-print":{"date-parts":[[2022,7,15]]}},"URL":"https:\/\/doi.org\/10.1093\/comjnl\/bxab011","relation":{},"ISSN":["0010-4620","1460-2067"],"issn-type":[{"value":"0010-4620","type":"print"},{"value":"1460-2067","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,7,15]]},"published":{"date-parts":[[2021,4,19]]}}}