{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T11:41:12Z","timestamp":1783597272990,"version":"3.55.0"},"reference-count":23,"publisher":"World Scientific Pub Co Pte Ltd","issue":"04","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Int. J. Human. Robot."],"published-print":{"date-parts":[[2019,8]]},"abstract":"<jats:p> Facial expression recognition has been widely used in human computer interaction (HCI) systems. Over the years, researchers have proposed different feature descriptors, implemented different classification methods, and carried out a number of experiments on various datasets for automatic facial expression recognition. However, most of them used 2D static images or 2D video sequences for the recognition task. The main limitations of 2D-based analysis are problems associated with variations in pose and illumination, which reduce the recognition accuracy. Therefore, an alternative way is to incorporate depth information acquired by 3D sensor, because it is invariant in both pose and illumination. In this paper, we present a two-stream convolutional neural network (CNN)-based facial expression recognition system and test it on our own RGB-D facial expression dataset collected by Microsoft Kinect for XBOX in unspontaneous scenarios since Kinect is an inexpensive and portable device to capture both RGB and depth information. Our fully annotated dataset includes seven expressions (i.e., neutral, sadness, disgust, fear, happiness, anger, and surprise) for 15 subjects (9 males and 6 females) aged from 20 to 25. The two individual CNNs are identical in architecture but do not share parameters. To combine the detection results produced by these two CNNs, we propose the late fusion approach. The experimental results demonstrate that the proposed two-stream network using RGB-D images is superior to that of using only RGB images or depth images. <\/jats:p>","DOI":"10.1142\/s0219843619410020","type":"journal-article","created":{"date-parts":[[2019,6,6]],"date-time":"2019-06-06T06:07:36Z","timestamp":1559801256000},"page":"1941002","source":"Crossref","is-referenced-by-count":39,"title":["CNN-Based Facial Expression Recognition from Annotated RGB-D Images for Human\u2013Robot Interaction"],"prefix":"10.1142","volume":"16","author":[{"given":"Jing","family":"Li","sequence":"first","affiliation":[{"name":"School of Information Engineering, Nanchang University, Nanchang 330031, P. R. China"},{"name":"Metallurgical Equipment and Control Technology of Ministry of Education, Wuhan University of Science and Technology, Wuhan 430081, P. R. China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yang","family":"Mi","sequence":"additional","affiliation":[{"name":"School of Information Engineering, Nanchang University, Nanchang 330031, P. R. China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gongfa","family":"Li","sequence":"additional","affiliation":[{"name":"Metallurgical Equipment and Control Technology of Ministry of Education, Wuhan University of Science and Technology, Wuhan 430081, P. R. China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9524-7609","authenticated-orcid":false,"given":"Zhaojie","family":"Ju","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Robotics, Shenyang Institute of Automation Chinese, Academy of Sciences, P. R. China"},{"name":"School of Computing, University of Portsmouth, Portsmouth, PO1 3HE, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"219","published-online":{"date-parts":[[2019,9,27]]},"reference":[{"key":"S0219843619410020BIB002","first-page":"504","volume-title":"Int. Conf. Human-robot Interaction","author":"Baxter P.","year":"2014"},{"key":"S0219843619410020BIB003","first-page":"1069","volume-title":"2016 Int. Conf. Adv. Robotics Mechatronics (ICARM)","author":"Tadeusz S.","year":"2016"},{"key":"S0219843619410020BIB004","first-page":"63","author":"Uddin M. Z.","year":"2017","journal-title":"Comput. Electrical Eng."},{"key":"S0219843619410020BIB005","first-page":"3422","volume-title":"IEEE Conf. Comput. Vision Pattern Recognit. (CVPR)","author":"Wang Z.","year":"2013"},{"key":"S0219843619410020BIB006","doi-asserted-by":"publisher","DOI":"10.1037\/h0030377"},{"key":"S0219843619410020BIB007","volume-title":"Proc. Fourth Int. Joint Conf. Pattern Recognit.","author":"Suwa M.","year":"1978"},{"key":"S0219843619410020BIB008","doi-asserted-by":"crossref","first-page":"390","DOI":"10.1109\/AFGR.1998.670980","volume-title":"IEEE Int. Conf. Automatic Face and Gesture Recognit.","author":"Lien J. J.","year":"1998"},{"key":"S0219843619410020BIB009","first-page":"511","volume-title":"IEEE Conf. Comput. Vision and Pattern Recognit. (CVPR)","author":"Viola P.","year":"2001"},{"key":"S0219843619410020BIB010","first-page":"109","volume-title":"European Conf. Comput. Vision (ECCV)","author":"Chen D.","year":"2014"},{"key":"S0219843619410020BIB011","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2014.2386334"},{"key":"S0219843619410020BIB012","volume-title":"Connectionism in Perspective","author":"Lecun Y.","year":"1989"},{"key":"S0219843619410020BIB013","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2012.06.005"},{"key":"S0219843619410020BIB014","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2010.2060716"},{"key":"S0219843619410020BIB015","first-page":"392","volume-title":"IEEE Int. Conf. Machine Learning and Applications (ICMLA)","author":"Ijjina E. P.","year":"2014"},{"issue":"2","key":"S0219843619410020BIB016","first-page":"1097","volume":"60","author":"Krizhevsky A.","year":"2012","journal-title":"Commun. Acm."},{"key":"S0219843619410020BIB017","first-page":"1294","volume-title":"IEEE Int. Conf. Robotics Biomimetics (ICRB)","author":"Liew C. F.","year":"2013"},{"key":"S0219843619410020BIB018","first-page":"227","volume-title":"Int. Symp. System Integration (ISSI)","author":"Wang X.","year":"2013"},{"key":"S0219843619410020BIB019","author":"Simonyan K.","year":"2014","journal-title":"Comput. Sci."},{"issue":"2","key":"S0219843619410020BIB020","doi-asserted-by":"crossref","first-page":"169","DOI":"10.4103\/0256-4602.95389","volume":"29","author":"Uddin","year":"2012","journal-title":"IETE Tech. Rev."},{"issue":"5","key":"S0219843619410020BIB021","doi-asserted-by":"crossref","first-page":"1318","DOI":"10.1109\/TCYB.2013.2265378","volume":"43","author":"Han J.","year":"2013","journal-title":"IEEE Trans. Cybern."},{"key":"S0219843619410020BIB022","first-page":"511","volume-title":"IEEE Conf. Computer Vision Pattern Recognition (CVPR)","author":"Lucey P.","year":"2010"},{"issue":"1","key":"S0219843619410020BIB023","first-page":"1350014(1\u201330)","volume":"10","author":"Li Y.","year":"2013","journal-title":"Int. J. Hum. Robot."},{"issue":"3","key":"S0219843619410020BIB024","first-page":"1850006(1\u201331)","volume":"15","author":"Sandra C.","year":"2018","journal-title":"Int. J. Hum. Robot."}],"container-title":["International Journal of Humanoid Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S0219843619410020","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,9,27]],"date-time":"2019-09-27T05:33:04Z","timestamp":1569562384000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/abs\/10.1142\/S0219843619410020"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,8]]},"references-count":23,"journal-issue":{"issue":"04","published-print":{"date-parts":[[2019,8]]}},"alternative-id":["10.1142\/S0219843619410020"],"URL":"https:\/\/doi.org\/10.1142\/s0219843619410020","relation":{},"ISSN":["0219-8436","1793-6942"],"issn-type":[{"value":"0219-8436","type":"print"},{"value":"1793-6942","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,8]]}}}