{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,18]],"date-time":"2026-05-18T11:24:54Z","timestamp":1779103494366,"version":"3.51.4"},"reference-count":44,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2020,2,6]],"date-time":"2020-02-06T00:00:00Z","timestamp":1580947200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Convolutional Neural Networks (CNNs) have become one of the state-of-the-art methods for various computer vision and pattern recognition tasks including facial affective computing. Although impressive results have been obtained in facial affective computing using CNNs, the computational complexity of CNNs has also increased significantly. This means high performance hardware is typically indispensable. Most existing CNNs are thus not generalizable enough for mobile devices, where the storage, memory and computational power are limited. In this paper, we focus on the design and implementation of CNNs on mobile devices for real-time facial affective computing tasks. We propose a light-weight CNN architecture which well balances the performance and computational complexity. The experimental results show that the proposed architecture achieves high performance while retaining the low computational complexity compared with state-of-the-art methods. We demonstrate the feasibility of a CNN architecture in terms of speed, memory and storage consumption for mobile devices by implementing a real-time facial affective computing application on an actual mobile device.<\/jats:p>","DOI":"10.3390\/s20030870","type":"journal-article","created":{"date-parts":[[2020,2,7]],"date-time":"2020-02-07T03:13:27Z","timestamp":1581045207000},"page":"870","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":31,"title":["Real-Time Facial Affective Computing on Mobile Devices"],"prefix":"10.3390","volume":"20","author":[{"given":"Yuanyuan","family":"Guo","sequence":"first","affiliation":[{"name":"Institute of Smart Healthcare Systems, Qingdao Academy of Intelligent Industries, Qingdao 266109, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yifan","family":"Xia","sequence":"additional","affiliation":[{"name":"School of Creative Technologies, University of Portsmouth, Portsmouth PO1 2DJ, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jing","family":"Wang","sequence":"additional","affiliation":[{"name":"Institute of Smart Healthcare Systems, Qingdao Academy of Intelligent Industries, Qingdao 266109, China"},{"name":"Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7655-9228","authenticated-orcid":false,"given":"Hui","family":"Yu","sequence":"additional","affiliation":[{"name":"School of Creative Technologies, University of Portsmouth, Portsmouth PO1 2DJ, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7621-1988","authenticated-orcid":false,"given":"Rung-Ching","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Information Management, Chaoyang University of Technology, Taichung 41349, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,2,6]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Tao, J., and Tan, T. (2005, January 22\u201324). Affective computing: A review. Proceedings of the International Conference on Affective Computing and Intelligent Interaction, Beijing, China.","DOI":"10.1007\/11573548"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"124","DOI":"10.1037\/h0030377","article-title":"Constants across cultures in the face and emotion","volume":"17","author":"Ekman","year":"1971","journal-title":"J. Pers. Soc. Psychol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1161","DOI":"10.1037\/h0077714","article-title":"A circumplex model of affect","volume":"39","author":"Russell","year":"1980","journal-title":"J. Pers. Soc. Psychol"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Wang, Y., Yu, H., Dong, J., Jian, M., and Liu, H. (2017, January 17\u201320). Cascade support vector regression-based facial expression-aware face frontalization. Proceedings of the IEEE International Conference on Image Processing (ICIP), Beijing, China.","DOI":"10.1109\/ICIP.2017.8296799"},{"key":"ref_5","unstructured":"Wang, Y., Yu, H., Dong, J., Stevens, B., and Liu, H. (2016, January 20\u201324). Facial expression-aware face frontalization. Proceedings of the Asian Conference on Computer Vision, Taipei, Taiwan."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"386","DOI":"10.1109\/THMS.2014.2313912","article-title":"Regression-based facial expression optimization","volume":"44","author":"Yu","year":"2014","journal-title":"IEEE Trans. Hum.-Mach. Syst."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Leo, M., Carcagn\u00ec, P., Distante, C., Spagnolo, P., Mazzeo, P., Rosato, A., Petrocchi, S., Pellegrino, C., Levante, A., and De Lum\u00e8, F. (2018). Computational Assessment of Facial Expression Production in ASD Children. Sensors, 18.","DOI":"10.3390\/s18113993"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Goulart, C., Valad\u00e3o, C., Delisle-Rodriguez, D., Funayama, D., Favarato, A., Baldo, G., Binotte, V., Caldeira, E., and Bastos-Filho, T. (2019). Visual and Thermal Image Processing for Facial Specific Landmark Detection to Infer Emotions in a Child-Robot Interaction. Sensors, 19.","DOI":"10.3390\/s19132844"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"7711","DOI":"10.3390\/s140507711","article-title":"Muecas: A multi-sensor robotic head for affective human robot interaction and imitation","volume":"14","author":"Cid","year":"2014","journal-title":"Sensors"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Varghese, E.B., and Thampi, S.M. (2018, January 24\u201326). A Deep Learning Approach to Predict Crowd Behavior Based on Emotion. Proceedings of the International Conference on Smart Multimedia, Toulon, France.","DOI":"10.1007\/978-3-030-04375-9_25"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"30240","DOI":"10.3390\/s151229794","article-title":"Tracking multiple video targets with an improved GM-PHD tracker","volume":"15","author":"Zhou","year":"2015","journal-title":"Sensors"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"803","DOI":"10.1016\/j.imavis.2008.08.005","article-title":"Facial expression recognition based on local binary patterns: A comprehensive study","volume":"27","author":"Shan","year":"2009","journal-title":"Image Vis. Comput."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Yu, H., and Liu, H. (2015, January 24\u201326). Combining appearance and geometric features for facial expression recognition. Proceedings of the Sixth International Conference on Graphic and Image Processing (ICGIP 2014), International Society for Optics and Photonics, Beijing, China.","DOI":"10.1117\/12.2179066"},{"key":"ref_14","unstructured":"Wang, Y., Ai, H., Wu, B., and Huang, C. (2004, January 23\u201326). Real time facial expression recognition with adaboost. Proceedings of the 17th International Conference on Pattern Recognition (ICPR 2004), Cambridge, UK."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"2216","DOI":"10.1109\/TCE.2009.5373791","article-title":"An enhanced independent component-based human facial expression recognition from video","volume":"55","author":"Uddin","year":"2009","journal-title":"IEEE Trans. Consum. Electron."},{"key":"ref_16","unstructured":"Edwards, G.J., Taylor, C.J., and Cootes, T.F. (1998, January 14\u201316). Interpreting face images using active appearance models. Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1006\/cviu.1995.1004","article-title":"Active shape models-their training and application","volume":"61","author":"Cootes","year":"1995","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Choi, H.C., and Oh, S.Y. (2006, January 18\u201321). Realtime facial expression recognition using active appearance model and multilayer perceptron. Proceedings of the 2006 SICE-ICASE International Joint Conference, Busan, Korea.","DOI":"10.1109\/SICE.2006.315594"},{"key":"ref_19","unstructured":"Chen, J., Chen, Z., Chi, Z., and Fu, H. (2014, January 22\u201323). Facial expression recognition based on facial components detection and hog features. Proceedings of the International Workshops on Electrical and Computer Engineering Subfields, Istanbul, Turkey."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Orrite, C., Ga\u00f1\u00e1n, A., and Rogez, G. (2009, January 10\u201312). Hog-based decision tree for facial expression classification. Proceedings of the Iberian Conference on Pattern Recognition and Image Analysis, Povoa de Varzim, Portugal.","DOI":"10.1007\/978-3-642-02172-5_24"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"2767","DOI":"10.1016\/j.ijleo.2012.08.040","article-title":"Facial expression recognition based on fusion feature of PCA and LBP with SVM","volume":"124","author":"Luo","year":"2013","journal-title":"Opt. Int. J. Light Electron. Opt."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Barroso, E., Santos, G., and Proen\u00e7a, H. (2013, January 16\u201319). Facial expressions: Discriminability of facial regions and relationship to biometrics recognition. Proceedings of the 2013 IEEE Symposium on Computational Intelligence in Biometrics and Identity Management (CIBIM), Singapore.","DOI":"10.1109\/CIBIM.2013.6607918"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1186\/s40064-015-1427-3","article-title":"Facial expression recognition and histograms of oriented gradients: a comprehensive study","volume":"4","author":"Leo","year":"2015","journal-title":"SpringerPlus"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1016\/j.patcog.2011.05.006","article-title":"Facial expression recognition using radial encoding of local Gabor features and classifier synthesis","volume":"45","author":"Gu","year":"2012","journal-title":"Pattern Recognit."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Yang, H., Ciftci, U., and Yin, L. (2018, January 18\u201322). Facial expression recognition by de-expression residue learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00231"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1127","DOI":"10.1109\/JAS.2019.1911684","article-title":"Accurate and robust eye center localization via fully convolutional networks","volume":"6","author":"Xia","year":"2019","journal-title":"IEEE\/CAA J. Autom. Sin."},{"key":"ref_27","first-page":"1","article-title":"The new era of artificial intelligence","volume":"1","author":"Zheng","year":"2019","journal-title":"Chin. J. Intell. Sci. Technol."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1109\/TCSS.2017.2766884","article-title":"Identification and analysis of driver postures for in-vehicle driving activities and secondary tasks recognition","volume":"5","author":"Xing","year":"2017","journal-title":"IEEE Trans. Comput. Soc. Syst."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Hu, B., and Wang, J. (2020). Deep Learning Based Hand Gesture Recognition and UAV Flight Controls. Int. J. Autom. Comput., 27.","DOI":"10.1007\/s11633-019-1194-7"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Sharma, A., Balouchian, P., and Foroosh, H. (2018). A Novel Multi-purpose Deep Architecture for Facial Attribute and Emotion Understanding. Iberoamerican Congress on Pattern Recognition, Springer.","DOI":"10.1007\/978-3-030-13469-3_72"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Zeng, J., Shan, S., and Chen, X. (2018, January 8\u201314). Facial expression recognition with inconsistently annotated datasets. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01261-8_14"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1007\/s11633-019-1176-9","article-title":"Expression Analysis Based on Face Regions in Read-world Conditions","volume":"17","author":"Lian","year":"2020","journal-title":"Int. J. Autom. Comput."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/TAFFC.2017.2740923","article-title":"Affectnet: A database for facial expression, valence, and arousal computing in the wild","volume":"10","author":"Mollahosseini","year":"2017","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_34","unstructured":"Siqueira, H. (2018, January 29\u201330). An Adaptive Neural Approach Based on Ensemble and Multitask Learning for Affect Recognition. Proceedings of the International PhD Conference on Safe and Social Robotics, Madrid, Spain."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Suk, M., and Prabhakaran, B. (2014, January 23\u201328). Real-time mobile facial expression recognition system-a case study. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Columbus, OH, USA.","DOI":"10.1109\/CVPRW.2014.25"},{"key":"ref_36","unstructured":"(2020, February 06). FaceReader. Available online: https:\/\/www.noldus.com\/facereader."},{"key":"ref_37","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_38","unstructured":"Ioffe, S., and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv."},{"key":"ref_39","unstructured":"Lin, M., Chen, Q., and Yan, S. (2013). Network in network. arXiv."},{"key":"ref_40","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (July, January 26). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1109\/T-AFFC.2011.9","article-title":"Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space","volume":"2","author":"Nicolaou","year":"2011","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Valstar, M., Gratch, J., Schuller, B., Ringeval, F., Lalanne, D., Torres Torres, M., Scherer, S., Stratou, G., Cowie, R., and Pantic, M. (2016, January 16). Avec 2016: Depression, mood, and emotion recognition workshop and challenge. Proceedings of the 6th International Workshop on Audio\/Visual Emotion Challenge, Amsterdam, The Netherlands.","DOI":"10.1145\/2964284.2980532"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1023\/B:VISI.0000013087.49260.fb","article-title":"Robust real-time face detection","volume":"57","author":"Viola","year":"2004","journal-title":"Int. J. Comput. Vis."},{"key":"ref_44","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2020, February 06). Imagenet classification with deep convolutional neural networks. Available online: http:\/\/papers.nips.cc\/paper\/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/3\/870\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T08:55:25Z","timestamp":1760172925000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/3\/870"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,2,6]]},"references-count":44,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2020,2]]}},"alternative-id":["s20030870"],"URL":"https:\/\/doi.org\/10.3390\/s20030870","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,2,6]]}}}