{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T15:45:36Z","timestamp":1781797536834,"version":"3.54.5"},"reference-count":39,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2021,1,15]],"date-time":"2021-01-15T00:00:00Z","timestamp":1610668800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Facial Action Units (AUs) correspond to the deformation\/contraction of individual facial muscles or their combinations. As such, each AU affects just a small portion of the face, with deformations that are asymmetric in many cases. Generating and analyzing AUs in 3D is particularly relevant for the potential applications it can enable. In this paper, we propose a solution for 3D AU detection and synthesis by developing on a newly defined 3D Morphable Model (3DMM) of the face. Differently from most of the 3DMMs existing in the literature, which mainly model global variations of the face and show limitations in adapting to local and asymmetric deformations, the proposed solution is specifically devised to cope with such difficult morphings. During a training phase, the deformation coefficients are learned that enable the 3DMM to deform to 3D target scans showing neutral and facial expression of the same individual, thus decoupling expression from identity deformations. Then, such deformation coefficients are used, on the one hand, to train an AU classifier, on the other, they can be applied to a 3D neutral scan to generate AU deformations in a subject-independent manner. The proposed approach for AU detection is validated on the Bosphorus dataset, reporting competitive results with respect to the state-of-the-art, even in a challenging cross-dataset setting. We further show the learned coefficients are general enough to synthesize realistic 3D face instances with AUs activation.<\/jats:p>","DOI":"10.3390\/s21020589","type":"journal-article","created":{"date-parts":[[2021,1,20]],"date-time":"2021-01-20T03:34:25Z","timestamp":1611113665000},"page":"589","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Action Unit Detection by Learning the Deformation Coefficients of a 3D Morphable Model"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6464-6052","authenticated-orcid":false,"given":"Luigi","family":"Ariano","sequence":"first","affiliation":[{"name":"Media Integration and Communication Center, University of Florence, 50134 Firenze, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9465-6753","authenticated-orcid":false,"given":"Claudio","family":"Ferrari","sequence":"additional","affiliation":[{"name":"Media Integration and Communication Center, University of Florence, 50134 Firenze, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1219-4386","authenticated-orcid":false,"given":"Stefano","family":"Berretti","sequence":"additional","affiliation":[{"name":"Media Integration and Communication Center, University of Florence, 50134 Firenze, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1052-8322","authenticated-orcid":false,"given":"Alberto","family":"Del Bimbo","sequence":"additional","affiliation":[{"name":"Media Integration and Communication Center, University of Florence, 50134 Firenze, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,1,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Ekman, P., and Friesen, W. (1978). Facial Action Coding System: A Technique for the Measurement of Facial Movement, Consulting Psychologists Press.","DOI":"10.1037\/t27734-000"},{"key":"ref_2","first-page":"1","article-title":"Representation, Analysis and Recognition of 3D Humans: A Survey","volume":"14","author":"Berretti","year":"2018","journal-title":"ACM Trans. Multimed. Comput. Commun. Appl."},{"key":"ref_3","unstructured":"Ferrari, C., Berretti, S., Pala, P., and Del Bimbo, A. (2020). A Sparse and Locally Coherent Morphable Face Model for Dense Semantic Correspondence Across Heterogeneous 3D Faces. arXiv."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Ferrari, C., Berretti, S., Pala, P., and Del Bimbo, A. (2018, January 8\u201314). Rendering realistic subject-dependent expression images by learning 3DMM deformation coefficients. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-11012-3_34"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2666","DOI":"10.1109\/TMM.2017.2707341","article-title":"A Dictionary Learning-Based 3D Morphable Shape Model","volume":"19","author":"Ferrari","year":"2017","journal-title":"IEEE Trans. Multimed."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Blanz, V., and Vetter, T. (1999, January 8\u201313). A morphable model for the synthesis of 3D faces. Proceedings of the ACM Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, USA.","DOI":"10.1145\/311535.311556"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Paysan, P., Knothe, R., Amberg, B., Romdhani, S., and Vetter, T. (2009, January 2\u20134). A 3D Face Model for Pose and Illumination Invariant Face Recognition. Proceedings of the IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Genoa, Italy.","DOI":"10.1109\/AVSS.2009.58"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"413","DOI":"10.1109\/TVCG.2013.249","article-title":"FaceWarehouse: A 3D Facial Expression Database for Visual Computing","volume":"20","author":"Cao","year":"2014","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_9","first-page":"194:1","article-title":"Learning a model of facial shape and expression from 4D scans","volume":"36","author":"Li","year":"2017","journal-title":"ACM TRans. Graph. (Proc. SIGGRAPH Asia)"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Brunton, A., Bolkart, T., and Wuhrer, S. (2014, January 6\u201312). Multilinear wavelets: A statistical shape space for human faces. Proceedings of the European Conf. on Computer Vision (ECCV), Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10590-1_20"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1860","DOI":"10.1109\/TPAMI.2017.2739743","article-title":"Gaussian Process Morphable Models","volume":"40","author":"Gerig","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"179:1","DOI":"10.1145\/2508363.2508417","article-title":"Sparse Localized Deformation Components","volume":"32","author":"Neumann","year":"2013","journal-title":"ACM Trans. Graph."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Sengupta, S., Kanazawa, A., Castillo, C.D., and Jacobs, D.W. (2018, January 18\u201322). SfSNet: Learning Shape, Refectance and Illuminance of Faces in the Wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Regognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00659"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Bagautdinov, T., Wu, C., Saragih, J., Fua, P., and Sheikh, Y. (2018, January 18\u201322). Modeling Facial Geometry Using Compositional VAEs. Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00408"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Ranjan, A., Bolkart, T., Sanyal, S., and Black, M. (2018, January 8\u201314). Generating 3D Faces Using Convolutional Mesh Autoencoders. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01219-9_43"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Jiang, Z.H., Wu, Q., Chen, K., and Zhang, J. (2019, January 16\u201320). Disentangled Representation Learning for 3D Face Shape. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.01223"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Liu, F., Tran, L., and Liu, X. (2019, January 27\u201328). 3D Face Modeling From Diverse Raw Scan Data. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.","DOI":"10.1109\/ICCV.2019.00950"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Charles, R.Q., Su, H., Kaichun, M., and Guibas, L.J. (2017, January 21\u201326). PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.16"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Caleanu, C.D. (2013, January 23\u201325). Face expression recognition: A brief overview of the last decade. Proceedings of the IEEE International Symposium on Applied Computational Intelligence, Timisoara, Romania.","DOI":"10.1109\/SACI.2013.6608958"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Taha, B., and Hatzinakos, D. (2019, January 5\u20138). Emotion Recognition from 2D Facial Expressions. Proceedings of the IEEE Canadian Conference of Electrical and Computer Engineering (CCECE), Edmonton, AB, Canada.","DOI":"10.1109\/CCECE.2019.8861751"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"683","DOI":"10.1016\/j.imavis.2012.06.005","article-title":"Static and Dynamic 3D Facial Expression Recognition: A Comprehensive Survey","volume":"30","author":"Sandbach","year":"2012","journal-title":"Image Vis. Comput."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Huang, Y., Wang, Y., and Tan, T. (2006, January 4\u20137). Combining Statistics of Geometrical and Correlative Features for 3D Face Recognition. Proceedings of the British Machine Vision Conf. (BMVC), Edinburgh, UK.","DOI":"10.5244\/C.20.90"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2816","DOI":"10.1109\/TMM.2017.2713408","article-title":"Multimodal 2D+3D Facial Expression Recognition With Deep Fusion Convolutional Neural Network","volume":"19","author":"Li","year":"2017","journal-title":"IEEE Trans. Multimed."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"2900","DOI":"10.1109\/TCSVT.2020.2984241","article-title":"Learned 3D Shape Representations Using Fused Geometrically Augmented Images: Application to Facial Expression and Action Unit Detection","volume":"30","author":"Taha","year":"2020","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"498","DOI":"10.1109\/TIFS.2008.924598","article-title":"Bilinear Models for 3-D Face and Facial Expression Recognition","volume":"3","author":"Mpiperis","year":"2008","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Gong, B., Wang, Y., Liu, J., and Tang, X. (2019, January 19\u201323). Automatic Facial Expression Recognition on a Single 3D Face by Exploring Shape Deformation. Proceedings of the ACM International Conference on Multimedia, 2009, MM\u201909, Beijing, China.","DOI":"10.1145\/1631272.1631358"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1109\/TCSVT.2012.2203210","article-title":"A Deformable 3-D Facial Expression Model for Dynamic Human Emotional State Recognition","volume":"23","author":"Tie","year":"2013","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1438","DOI":"10.1109\/TMM.2016.2557063","article-title":"Muscular Movement Model-Based Automatic 3D\/4D Facial Expression Recognition","volume":"18","author":"Zhen","year":"2016","journal-title":"IEEE Trans. Multimed."},{"key":"ref_29","unstructured":"Hinduja, S., and Canavan, S. (2020). Facial Action Unit Detection using 3D Facial Landmarks. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Ferrari, C., Lisanti, G., Berretti, S., and Del Bimbo, A. (2015, January 19\u201322). Dictionary Learning based 3D Morphable Model Construction for Face Recognition with Varying Expression and Pose. Proceedings of the 2015 International Conference on 3D Vision, Lyon, France.","DOI":"10.1109\/3DV.2015.63"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1111\/j.1467-9868.2005.00503.x","article-title":"Regularization and variable selection via the elastic net","volume":"67","author":"Zou","year":"2005","journal-title":"J. R. Stat. Soc. Ser. (Stat. Methodol.)"},{"key":"ref_32","first-page":"19","article-title":"Online learning for matrix factorization and sparse coding","volume":"11","author":"Mairal","year":"2010","journal-title":"J. Mach. Learn. Res."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Fan, Z., Hu, X., Chen, C., and Peng, S. (2018, January 8\u201314). Dense Semantic and Topological Correspondence of 3D Faces without Landmarks. Proceedings of the European Conf. on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01270-0_32"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1584","DOI":"10.1109\/TPAMI.2017.2725279","article-title":"Dense 3D Face Correspondence","volume":"40","author":"Gilani","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_35","unstructured":"Yin, L., Wei, X., Sun, Y., Wang, J., and Rosato, M.J. (2006, January 10\u201312). A 3D facial expression database for facial behavior research. Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition (FGR), Southampton, UK."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Sandbach, G., Zafeiriou, S., and Pantic, M. (2012, January 3\u20137). Binary pattern analysis for 3D facial action unit detection. Proceedings of the British Machine Vision Conf. (BMVC), Surrey, UK.","DOI":"10.5244\/C.26.119"},{"key":"ref_37","unstructured":"Sandbach, G., Zafeiriou, S., and Pantic, M. (October, January 30). Local normal binary patterns for 3D facial action unit detection. Proceedings of the IEEE International Conference on Image Processing (ICIP), Orlando, FL, USA."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Bayramoglu, N., Zhao, G., and Pietik\u00e4inen, M. (2013, January 4\u20137). CS-3DLBP and geometry based person independent 3D facial action unit detection. Proceedings of the International Conference on Biometrics (ICB), Madrid, Spain.","DOI":"10.1109\/ICB.2013.6612977"},{"key":"ref_39","unstructured":"Phillips, P.J., Flynn, P.J., Scruggs, T., Bowyer, K.W., Chang, J., Hoffman, K., Marques, J., Min, J., and Worek, W. (2005, January 21\u201323). Overview of the Face Recognition Grand Challenge. Proceedings of the IEEE Workshop on Face Recognition Grand Challenge Experiments, San Diego, CA, USA."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/2\/589\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:11:36Z","timestamp":1760159496000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/2\/589"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,1,15]]},"references-count":39,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2021,1]]}},"alternative-id":["s21020589"],"URL":"https:\/\/doi.org\/10.3390\/s21020589","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,1,15]]}}}