{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,12]],"date-time":"2025-11-12T14:09:54Z","timestamp":1762956594426,"version":"3.41.2"},"reference-count":55,"publisher":"Wiley","issue":"1","license":[{"start":{"date-parts":[[2021,3,18]],"date-time":"2021-03-18T00:00:00Z","timestamp":1616025600000},"content-version":"vor","delay-in-days":76,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Communication University China"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Journal of Sensors"],"published-print":{"date-parts":[[2021,1]]},"abstract":"<jats:p>The interest in the facial expression recognition (FER) is increasing day by day due to its practical and potential applications, such as human physiological interaction diagnosis and mental diseases detection. This area has received much attention from the research community in recent years and achieved remarkable results; however, a significant improvement is required in spatial problems. This research work presents a novel framework and proposes an effective and robust solution for FER under an unconstrained environment. Face detection is performed using the supervision of facial attributes. Faceness\u2010Net is used for deep facial part responses for the detection of faces under severe unconstrained variations. In order to improve the generalization problems and avoid insufficient data regime, Deep Convolutional Graphical Adversarial Network (DC\u2010GAN) is utilized. Due to the challenging environmental factors faced in the wild, a large number of noises disrupt feature extraction, thus making it hard to capture ground truth. We leverage different multimodal sensors with a camera that aids in data acquisition, by extracting the features more accurately and improve the overall performance of FER. These intelligent sensors are used to tackle the significant challenges like illumination variance, subject dependence, and head pose. Dual\u2010enhanced capsule network is used which is able to handle the spatial problem. The traditional capsule networks are unable to sufficiently extract the features, as the distance varies greatly between facial features. Therefore, the proposed network is capable of spatial transformation due to action unit aware mechanism and thus forward most desiring features for dynamic routing between capsules. Squashing function is used for the classification function. We have elaborated the effectiveness of our method by validating the results on four popular and versatile databases that outperform all state\u2010of\u2010the\u2010art methods.<\/jats:p>","DOI":"10.1155\/2021\/8893661","type":"journal-article","created":{"date-parts":[[2021,3,18]],"date-time":"2021-03-18T21:20:07Z","timestamp":1616102407000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Empirical Investigation of Multimodal Sensors in Novel Deep Facial Expression Recognition In\u2010the\u2010Wild"],"prefix":"10.1155","volume":"2021","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1382-1353","authenticated-orcid":false,"given":"Asad","family":"Ullah","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jing","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"M. Shahid","family":"Anwar","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Taeg Keun","family":"Whangbo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6406-9483","authenticated-orcid":false,"given":"Yaping","family":"Zhu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"311","published-online":{"date-parts":[[2021,3,18]]},"reference":[{"key":"e_1_2_10_1_2","doi-asserted-by":"publisher","DOI":"10.3390\/s19081863"},{"key":"e_1_2_10_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3158369"},{"key":"e_1_2_10_3_2","doi-asserted-by":"crossref","unstructured":"VailA. K. Baltru\u0161aitisT. PennantL. LiebsonE. BakerJ. andMorencyL. P. Visual attention in schizophrenia: eye contact and gaze aversion during clinical interactions 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII) 2017 San Antonio TX 490\u2013497 https:\/\/doi.org\/10.1109\/ACII.2017.8273644 2-s2.0-85047473036.","DOI":"10.1109\/ACII.2017.8273644"},{"key":"e_1_2_10_4_2","doi-asserted-by":"publisher","DOI":"10.3390\/electronics8121487"},{"key":"e_1_2_10_5_2","doi-asserted-by":"publisher","DOI":"10.1037\/h0030377"},{"key":"e_1_2_10_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2015.08.011"},{"key":"e_1_2_10_7_2","doi-asserted-by":"crossref","unstructured":"UllahA. WangJ. AnwarM. S. AhmadU. SaeedU. andWangJ. Feature extraction based on canonical correlation analysis using FMEDA and DPA for facial expression recognition with RNN 2018 14th IEEE International Conference on Signal Processing (ICSP) 2018 Beijing China 418\u2013423 https:\/\/doi.org\/10.1109\/ICSP.2018.8652507 2-s2.0-85063284370.","DOI":"10.1109\/ICSP.2018.8652507"},{"key":"e_1_2_10_8_2","unstructured":"CaiJ. MengZ. KhanA. S. LiZ. O\u2019ReillyJ. andTongY. Identity-free facial expression recognition using conditional generative adversarial network 2019 https:\/\/arxiv.org\/abs\/1903.08051."},{"key":"e_1_2_10_9_2","doi-asserted-by":"crossref","unstructured":"YangH. ZhangZ. andYinL. Identity-adaptive facial expression recognition through expression regeneration using conditional generative adversarial networks 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) 2018 Xi\u2032an China 294\u2013301 https:\/\/doi.org\/10.1109\/FG.2018.00050 2-s2.0-85049396051.","DOI":"10.1109\/FG.2018.00050"},{"key":"e_1_2_10_10_2","doi-asserted-by":"crossref","unstructured":"Van QuangN. ChunJ. andTokuyamaT. Capsulenet for micro-expression recognition 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019) 2019 Lille France 1\u20137 https:\/\/doi.org\/10.1109\/FG.2019.8756544 2-s2.0-85070449521.","DOI":"10.1109\/FG.2019.8756544"},{"key":"e_1_2_10_11_2","unstructured":"Organization WHO; others Depression: key facts 2018."},{"key":"e_1_2_10_12_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jad.2008.06.026"},{"key":"e_1_2_10_13_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2019.04.002"},{"key":"e_1_2_10_14_2","doi-asserted-by":"crossref","unstructured":"FanY. LamJ. C. andLiV. O. Multi-region ensemble convolutional neural network for facial expression recognition 27th International Conference on Artificial Neural Networks 2018 Rhodes Greece 84\u201394 https:\/\/doi.org\/10.1007\/978-3-030-01418-6_9 2-s2.0-85054879043.","DOI":"10.1007\/978-3-030-01418-6_9"},{"key":"e_1_2_10_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2018.2868382"},{"key":"e_1_2_10_16_2","doi-asserted-by":"crossref","unstructured":"KlaserA. Marsza\u0142ekM. andSchmidC. A Spatio-Temporal Descriptor Based on 3d-Gradients 2008.","DOI":"10.5244\/C.22.99"},{"key":"e_1_2_10_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2009.2024578"},{"key":"e_1_2_10_18_2","doi-asserted-by":"crossref","unstructured":"JiangB. ValstarM. F. andPanticM. Action unit detection using sparse appearance descriptors in space-time video volumes 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG) 2011 Santa Barbara CA USA 314\u2013321 https:\/\/doi.org\/10.1109\/FG.2011.5771416 2-s2.0-79958720856.","DOI":"10.1109\/FG.2011.5771416"},{"key":"e_1_2_10_19_2","doi-asserted-by":"crossref","unstructured":"LiuM. LiS. ShanS. WangR. andChenX. Deeply learning deformable facial action parts model for dynamic expression analysis 12th Asian Conference on Computer Vision November 2014 Singapore Singapore 143\u2013157 Springerhttps:\/\/doi.org\/10.1007\/978-3-319-16817-3_10 2-s2.0-84983591528.","DOI":"10.1007\/978-3-319-16817-3_10"},{"key":"e_1_2_10_20_2","doi-asserted-by":"crossref","unstructured":"KimB. K. LeeH. RohJ. andLeeS. Y. Hierarchical committee of deep cnns with exponentially-weighted decision fusion for static facial expression recognition Proceedings of the 2015 ACM on International Conference on Multimodal Interaction November 2015 Seattle WA USA 427\u2013434 https:\/\/doi.org\/10.1145\/2818346.2830590 2-s2.0-84959291810.","DOI":"10.1145\/2818346.2830590"},{"key":"e_1_2_10_21_2","doi-asserted-by":"crossref","unstructured":"NgH. W. NguyenV. D. VonikakisV. andWinklerS. Deep learning for emotion recognition on small datasets using transfer learning Proceedings of the 2015 ACM on international conference on multimodal interaction November 2015 Seattle WA USA 443\u2013449 https:\/\/doi.org\/10.1145\/2818346.2830593 2-s2.0-84959295186.","DOI":"10.1145\/2818346.2830593"},{"key":"e_1_2_10_22_2","doi-asserted-by":"crossref","unstructured":"YuZ.andZhangC. Image based static facial expression recognition with multiple deep network learning Proceedings of the 2015 ACM on International Conference on Multimodal Interaction November 2015 Seattle WA USA 435\u2013442 https:\/\/doi.org\/10.1145\/2818346.2830593 2-s2.0-84959295186.","DOI":"10.1145\/2818346.2830595"},{"key":"e_1_2_10_23_2","doi-asserted-by":"crossref","unstructured":"LiS. DengW. andDuJ. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition July 2017 Honolulu HI USA 2852\u20132861.","DOI":"10.1109\/CVPR.2017.277"},{"key":"e_1_2_10_24_2","doi-asserted-by":"crossref","unstructured":"MengZ. LiuP. CaiJ. HanS. andTongY. Identity-aware convolutional neural network for facial expression recognition 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) 2017 Washington DC USA 558\u2013565 https:\/\/doi.org\/10.1109\/FG.2017.140 2-s2.0-85026295599.","DOI":"10.1109\/FG.2017.140"},{"key":"e_1_2_10_25_2","doi-asserted-by":"crossref","unstructured":"CaiJ. MengZ. KhanA. S. LiZ. O\u2019ReillyJ. andTongY. Island loss for learning discriminative features in facial expression recognition 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) 2018 Xi\u2032an China 302\u2013309 https:\/\/doi.org\/10.1109\/FG.2018.00051 2-s2.0-85049411197.","DOI":"10.1109\/FG.2018.00051"},{"key":"e_1_2_10_26_2","doi-asserted-by":"crossref","unstructured":"YangH. CiftciU. andYinL. Facial expression recognition by de-expression residue learning Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition June 2018 Salt Lake City UT USA 2168\u20132177.","DOI":"10.1109\/CVPR.2018.00231"},{"key":"e_1_2_10_27_2","unstructured":"AliK. IslerI. andHughesC. Facial expression recognition using human to animated-character expression translation 2019 https:\/\/arxiv.org\/abs\/1910.05595."},{"key":"e_1_2_10_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2017.2712788"},{"key":"e_1_2_10_29_2","doi-asserted-by":"crossref","unstructured":"LiuM. WangR. LiS. ShanS. HuangZ. andChenX. Combining multiple kernel methods on Riemannian manifold for emotion recognition in the wild Proceedings of the 16th International Conference on multimodal interaction November 2014 Istanbul Turkey 494\u2013501.","DOI":"10.1145\/2663204.2666274"},{"key":"e_1_2_10_30_2","doi-asserted-by":"crossref","unstructured":"LiuM. ShanS. WangR. andChenX. Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition June 2014 Columbus OH USA 1749\u20131756.","DOI":"10.1109\/CVPR.2014.226"},{"key":"e_1_2_10_31_2","doi-asserted-by":"publisher","DOI":"10.3390\/s20030592"},{"key":"e_1_2_10_32_2","first-page":"22","article-title":"Embedded emotion recognition: autonomous multimodal affective internet of things","volume":"2208","author":"Miranda J. A.","year":"2018","journal-title":"Proceedings of the cyber-physical systems workshop"},{"key":"e_1_2_10_33_2","doi-asserted-by":"crossref","unstructured":"ViolaP.andJonesM. Rapid object detection using a boosted cascade of simple features Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 2001 Kauai HI USA https:\/\/doi.org\/10.1109\/CVPR.2001.990517.","DOI":"10.1109\/CVPR.2001.990517"},{"key":"e_1_2_10_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2017.2738644"},{"key":"e_1_2_10_35_2","unstructured":"RadfordA. MetzL. andChintalaS. Unsupervised representation learning with deep convolutional generative adversarial networks 2015 https:\/\/arxiv.org\/abs\/1511.06434."},{"key":"e_1_2_10_36_2","doi-asserted-by":"crossref","unstructured":"ValstarM. PanticM. andPatrasI. Motion history for facial action detection in video 1 2004 IEEE International Conference on Systems Man and Cybernetics (IEEE Cat. No.04CH37583) 2004 The Hague Netherlands 635\u2013640 https:\/\/doi.org\/10.1109\/ICSMC.2004.1398371 2-s2.0-15744398669.","DOI":"10.1109\/ICSMC.2004.1398371"},{"key":"e_1_2_10_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2018.2791608"},{"key":"e_1_2_10_38_2","doi-asserted-by":"crossref","unstructured":"LuceyP. CohnJ. F. KanadeT. andSaragihJ. The extended cohn-kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops 2010 San Francisco CA USA 94\u2013101 https:\/\/doi.org\/10.1109\/CVPRW.2010.5543262 2-s2.0-77956509035.","DOI":"10.1109\/CVPRW.2010.5543262"},{"key":"e_1_2_10_39_2","doi-asserted-by":"crossref","unstructured":"PanticM. ValstarM. RademakerR. andMaatL. Web-based database for facial expression analysis 2005 IEEE International Conference on Multimedia and Expo 2005 Amsterdam Netherlands https:\/\/doi.org\/10.1109\/ICME.2005.1521424 2-s2.0-33644991895.","DOI":"10.1109\/ICME.2005.1521424"},{"key":"e_1_2_10_40_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2011.07.002"},{"key":"e_1_2_10_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2007.1110"},{"key":"e_1_2_10_42_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2013.03.003"},{"key":"e_1_2_10_43_2","unstructured":"JungH. LeeS. YimJ. ParkS. andKimJ. Joint fine-tuning in deep neural networks for facial expression recognition Proceedings of the IEEE international conference on computer vision August 2018 Beijing China."},{"key":"e_1_2_10_44_2","doi-asserted-by":"crossref","unstructured":"UllahA. WangJ. AnwarM. S. AhmadU. WangJ. andSaeedU. Nonlinear manifold feature extraction based on spectral supervised canonical correlation analysis for facial expression recognition with RRNN 2018 11th International Congress on Image and Signal Processing BioMedical Engineering and Informatics (CISP-BMEI) 2018 Beijing China 1\u20136 https:\/\/doi.org\/10.1109\/CISP-BMEI.2018.8633244 2-s2.0-85062834415.","DOI":"10.1109\/CISP-BMEI.2018.8633244"},{"key":"e_1_2_10_45_2","doi-asserted-by":"crossref","unstructured":"WuW. YinY. WangY. WangX. andXuD. Facial expression recognition for different pose faces based on special landmark detection 2018 24th International Conference on Pattern Recognition (ICPR) 2018 Beijing China 1524\u20131529 https:\/\/doi.org\/10.1109\/ICPR.2018.8545725 2-s2.0-85059747742.","DOI":"10.1109\/ICPR.2018.8545725"},{"key":"e_1_2_10_46_2","doi-asserted-by":"crossref","unstructured":"ZhongL. LiuQ. YangP. LiuB. HuangJ. andMetaxasD. N. Learning active facial patches for expression analysis 2012 IEEE Conference on Computer Vision and Pattern Recognition 2012 Providence RI USA 2562\u20132569 https:\/\/doi.org\/10.1109\/CVPR.2012.6247974 2-s2.0-84866697202.","DOI":"10.1109\/CVPR.2012.6247974"},{"key":"e_1_2_10_47_2","doi-asserted-by":"crossref","unstructured":"GuoY. ZhaoG. andPietik\u00e4inenM. Dynamic facial expression recognition using longitudinal facial expression atlases 7573 Computer Vision \u2013 ECCV 2012: 12th European Conference on Computer Vision 2012 Florence Italy 631\u2013644 https:\/\/doi.org\/10.1007\/978-3-642-33709-3_45 2-s2.0-84867849446.","DOI":"10.1007\/978-3-642-33709-3_45"},{"key":"e_1_2_10_48_2","doi-asserted-by":"crossref","unstructured":"DingH. ZhouS. K. andChellappaR. Facenet2expnet: regularizing a deep face recognition net for expression recognition 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) 2017 Washington DC 118\u2013126 https:\/\/doi.org\/10.1109\/FG.2017.23 2-s2.0-85026315336.","DOI":"10.1109\/FG.2017.23"},{"key":"e_1_2_10_49_2","doi-asserted-by":"crossref","unstructured":"ZhaoX. LiangX. LiuL. LiT. HanY. VasconcelosN. andYanS. Peak-piloted deep network for facial expression recognition Computer Vision \u2013 ECCV 2016: 14th European Conference 2016 Amsterdam The Netherlands 425\u2013442 https:\/\/doi.org\/10.1007\/978-3-319-46475-6_27 2-s2.0-84990862149.","DOI":"10.1007\/978-3-319-46475-6_27"},{"key":"e_1_2_10_50_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11063-018-9967-5"},{"key":"e_1_2_10_51_2","unstructured":"AzadiS. FengJ. JegelkaS. andDarrellT. Auxiliary image regularization for deep cnns with noisy labels 2015 https:\/\/arxiv.org\/abs\/1511.07069."},{"key":"e_1_2_10_52_2","unstructured":"GoldbergerJ.andBen-ReuvenE. Training Deep Neural-Networks Using a Noise Adaptation Layer 2016."},{"key":"e_1_2_10_53_2","doi-asserted-by":"publisher","DOI":"10.2307\/2346806"},{"key":"e_1_2_10_54_2","doi-asserted-by":"crossref","unstructured":"ZengJ. ShanS. andChenX. Facial expression recognition with inconsistently annotated datasets Proceedings of the European conference on computer vision (ECCV) 2018 United Kingdom 222\u2013237.","DOI":"10.1007\/978-3-030-01261-8_14"},{"key":"e_1_2_10_55_2","doi-asserted-by":"publisher","DOI":"10.1155\/2021\/6673992"}],"container-title":["Journal of Sensors"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/downloads.hindawi.com\/journals\/js\/2021\/8893661.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/downloads.hindawi.com\/journals\/js\/2021\/8893661.xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1155\/2021\/8893661","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,6]],"date-time":"2024-08-06T01:21:58Z","timestamp":1722907318000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1155\/2021\/8893661"}},"subtitle":[],"editor":[{"given":"Giuseppe","family":"Quero","sequence":"additional","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2021,1]]},"references-count":55,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,1]]}},"alternative-id":["10.1155\/2021\/8893661"],"URL":"https:\/\/doi.org\/10.1155\/2021\/8893661","archive":["Portico"],"relation":{},"ISSN":["1687-725X","1687-7268"],"issn-type":[{"type":"print","value":"1687-725X"},{"type":"electronic","value":"1687-7268"}],"subject":[],"published":{"date-parts":[[2021,1]]},"assertion":[{"value":"2020-05-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-03-05","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-03-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"8893661"}}