{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,2]],"date-time":"2026-01-02T07:45:52Z","timestamp":1767339952514,"version":"3.41.0"},"reference-count":48,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2018,11,28]],"date-time":"2018-11-28T00:00:00Z","timestamp":1543363200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["340164"],"award-info":[{"award-number":["340164"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2019,1,31]]},"abstract":"<jats:p>Making machines understand human expressions enables various useful applications in human-machine interaction. In this article, we present a novel facial expression recognition approach with 3D Mesh Convolutional Neural Networks (3DMCNN) and a visual analytics-guided 3DMCNN design and optimization scheme. From an RGBD camera, we first reconstruct a 3D face model of a subject with facial expressions and then compute the geometric properties of the surface. Instead of using regular Convolutional Neural Networks (CNNs) to learn intensities of the facial images, we convolve the geometric properties on the surface of the 3D model using 3DMCNN. We design a geodesic distance-based convolution method to overcome the difficulties raised from the irregular sampling of the face surface mesh. We further present interactive visual analytics for the purpose of designing and modifying the networks to analyze the learned features and cluster similar nodes in 3DMCNN. By removing low-activity nodes in the network, the performance of the network is greatly improved. We compare our method with the regular CNN-based method by interactively visualizing each layer of the networks and analyze the effectiveness of our method by studying representative cases. Testing on public datasets, our method achieves a higher recognition accuracy than traditional image-based CNN and other 3D CNNs. The proposed framework, including 3DMCNN and interactive visual analytics of the CNN, can be extended to other applications.<\/jats:p>","DOI":"10.1145\/3200572","type":"journal-article","created":{"date-parts":[[2018,11,28]],"date-time":"2018-11-28T19:16:01Z","timestamp":1543432561000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Learning Facial Expressions with 3D Mesh Convolutional Neural Network"],"prefix":"10.1145","volume":"10","author":[{"given":"Hai","family":"Jin","sequence":"first","affiliation":[{"name":"Wayne State University, Woodward Ave, Detroit, MI"}]},{"given":"Yuanfeng","family":"Lian","sequence":"additional","affiliation":[{"name":"Wayne State University, Woodward Ave, Detroit, MI"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3981-2933","authenticated-orcid":false,"given":"Jing","family":"Hua","sequence":"additional","affiliation":[{"name":"Wayne State University, Woodward Ave, Detroit, MI"}]}],"member":"320","published-online":{"date-parts":[[2018,11,28]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2009.05.007"},{"key":"e_1_2_1_2_1","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition Workshop","volume":"5","author":"Bartlett Marian Stewart","unstructured":"Marian Stewart Bartlett , Gwen Littlewort , Ian Fasel , and Javier R. Movellan . 2003. Real time face detection and facial expression recognition: Development and applications to human computer interaction . In IEEE Conference on Computer Vision and Pattern Recognition Workshop , Vol. 5 . 53--53. Marian Stewart Bartlett, Gwen Littlewort, Ian Fasel, and Javier R. Movellan. 2003. Real time face detection and facial expression recognition: Development and applications to human computer interaction. In IEEE Conference on Computer Vision and Pattern Recognition Workshop, Vol. 5. 53--53."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.598228"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/311535.311556"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2003.1227983"},{"volume-title":"IEEE Conference on Computer Vision and Pattern Recognition. 1704--1711","author":"Michael","key":"e_1_2_1_6_1","unstructured":"Michael M. Bronstein and Iasonas Kokkinos. 2010. Scale-invariant heat kernel signatures for non-rigid shape recognition . In IEEE Conference on Computer Vision and Pattern Recognition. 1704--1711 . Michael M. Bronstein and Iasonas Kokkinos. 2010. Scale-invariant heat kernel signatures for non-rigid shape recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 1704--1711."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2013.249"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.449"},{"key":"e_1_2_1_9_1","volume-title":"Huang","author":"Cohen Ira","year":"2000","unstructured":"Ira Cohen , Ashutosh Garg , and Thomas S . Huang . 2000 . Emotion recognition from facial expressions using multilevel HMM. In Neural Information Processing Systems , Vol. 2 . Ira Cohen, Ashutosh Garg, and Thomas S. Huang. 2000. Emotion recognition from facial expressions using multilevel HMM. In Neural Information Processing Systems, Vol. 2."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.927467"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2016.2515606"},{"key":"e_1_2_1_12_1","volume-title":"British Machine Vision Conference","volume":"1","author":"Cristinacce David","unstructured":"David Cristinacce and Timothy F. Cootes . 2006. Feature detection and tracking with constrained local models . In British Machine Vision Conference , Vol. 1 . 3. David Cristinacce and Timothy F. Cootes. 2006. Feature detection and tracking with constrained local models. In British Machine Vision Conference, Vol. 1. 3."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1080\/02699939208411068"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-012-0549-0"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2007.70733"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2008.134"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2005.03.004"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cagd.2016.11.001"},{"key":"e_1_2_1_19_1","volume-title":"ActiVis: Visual exploration of industry-scale deep neural network models. arXiv Preprint arXiv:1704.01942","author":"Kahng Minsuk","year":"2017","unstructured":"Minsuk Kahng , Pierre Andrews , Aditya Kalro , and Duen Horng Chau . 2017. ActiVis: Visual exploration of industry-scale deep neural network models. arXiv Preprint arXiv:1704.01942 ( 2017 ). Minsuk Kahng, Pierre Andrews, Aditya Kalro, and Duen Horng Chau. 2017. ActiVis: Visual exploration of industry-scale deep neural network models. arXiv Preprint arXiv:1704.01942 (2017)."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2007.02.003"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12193-015-0209-0"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/FUZZY.2009.5277231"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSMC.1997.633250"},{"key":"e_1_2_1_24_1","volume-title":"Hinton","author":"Krizhevsky Alex","year":"2012","unstructured":"Alex Krizhevsky , Ilya Sutskever , and Geoffrey E . Hinton . 2012 . Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems . 1097--1105. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. 1097--1105."},{"volume-title":"IEEE Conference on Computer Vision and Pattern Recognition. 1--7.","author":"Lei Zhen","key":"e_1_2_1_25_1","unstructured":"Zhen Lei , Qinqun Bai , Ran He , and S. Z. Li . 2008. Face shape recovery from a single image using CCA mapping between tensor spaces . In IEEE Conference on Computer Vision and Pattern Recognition. 1--7. Zhen Lei, Qinqun Bai, Ran He, and S. Z. Li. 2008. Face shape recovery from a single image using CCA mapping between tensor spaces. In IEEE Conference on Computer Vision and Pattern Recognition. 1--7."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598831"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2016.07.026"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2010.5543262"},{"key":"e_1_2_1_29_1","first-page":"52","article-title":"Discrete differential-geometry operators for triangulated 2-manifolds","volume":"3","author":"Meyer Mark","year":"2002","unstructured":"Mark Meyer , Mathieu Desbrun , Peter Schr\u00f6der , and Alan H. Barr . 2002 . Discrete differential-geometry operators for triangulated 2-manifolds . Visualization and Mathematics 3 , 2 (2002), 52 -- 58 . Mark Meyer, Mathieu Desbrun, Peter Schr\u00f6der, and Alan H. Barr. 2002. Discrete differential-geometry operators for triangulated 2-manifolds. Visualization and Mathematics 3, 2 (2002), 52--58.","journal-title":"Visualization and Mathematics"},{"volume-title":"IEEE Winter Conference on Applications of Computer Vision (WACV\u201916)","author":"Mollahosseini Ali","key":"e_1_2_1_30_1","unstructured":"Ali Mollahosseini , David Chan , and Mohammad H. Mahoor . 2016. Going deeper in facial expression recognition using deep neural networks . In IEEE Winter Conference on Applications of Computer Vision (WACV\u201916) . 1--10. Ali Mollahosseini, David Chan, and Mohammad H. Mahoor. 2016. Going deeper in facial expression recognition using deep neural networks. In IEEE Winter Conference on Applications of Computer Vision (WACV\u201916). 1--10."},{"volume-title":"IEEE Conference on Computer Vision and Pattern Recognition. 343--352","author":"Newcombe Richard A.","key":"e_1_2_1_31_1","unstructured":"Richard A. Newcombe , Dieter Fox , and Steven M. Seitz . 2015. DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time . In IEEE Conference on Computer Vision and Pattern Recognition. 343--352 . Richard A. Newcombe, Dieter Fox, and Steven M. Seitz. 2015. DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time. In IEEE Conference on Computer Vision and Pattern Recognition. 343--352."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744358"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2012.01.006"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298682"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2005.12.021"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2008.08.005"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46466-4_14"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.114"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2009.01515.x"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2816795.2818056"},{"volume-title":"IEEE Conference on Computer Vision and Pattern Recognition. 586--591","author":"Matthew","key":"e_1_2_1_41_1","unstructured":"Matthew A. Turk and Alex P. Pentland. 1991. Face recognition using eigenfaces . In IEEE Conference on Computer Vision and Pattern Recognition. 586--591 . Matthew A. Turk and Alex P. Pentland. 1991. Face recognition using eigenfaces. In IEEE Conference on Computer Vision and Pattern Recognition. 586--591."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2006.85"},{"key":"e_1_2_1_43_1","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition. 1912--1920","author":"Wu Zhirong","year":"2015","unstructured":"Zhirong Wu , Shuran Song , Aditya Khosla , Fisher Yu , Linguang Zhang , Xiaoou Tang , and Jianxiong Xiao . 2015 . 3D shapenets: A deep representation for volumetric shapes . In IEEE Conference on Computer Vision and Pattern Recognition. 1912--1920 . Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3D shapenets: A deep representation for volumetric shapes. In IEEE Conference on Computer Vision and Pattern Recognition. 1912--1920."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2004.1261097"},{"key":"e_1_2_1_45_1","volume-title":"Rosato","author":"Yin Lijun","year":"2006","unstructured":"Lijun Yin , Xiaozhou Wei , Yi Sun , Jun Wang , and Matthew J . Rosato . 2006 . A 3D facial expression database for facial behavior research. In Automatic Face and Gesture Recognition . 211--216. Lijun Yin, Xiaozhou Wei, Yi Sun, Jun Wang, and Matthew J. Rosato. 2006. A 3D facial expression database for facial behavior research. In Automatic Face and Gesture Recognition. 211--216."},{"volume-title":"European Conference on Computer Vision. 818--833","author":"Matthew","key":"e_1_2_1_46_1","unstructured":"Matthew D. Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks . In European Conference on Computer Vision. 818--833 . Matthew D. Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In European Conference on Computer Vision. 818--833."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.414"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2007.1110"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3200572","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3200572","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3200572","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:26:28Z","timestamp":1750213588000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3200572"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,11,28]]},"references-count":48,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,1,31]]}},"alternative-id":["10.1145\/3200572"],"URL":"https:\/\/doi.org\/10.1145\/3200572","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"type":"print","value":"2157-6904"},{"type":"electronic","value":"2157-6912"}],"subject":[],"published":{"date-parts":[[2018,11,28]]},"assertion":[{"value":"2017-08-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-11-28","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}