{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T23:07:44Z","timestamp":1789081664460,"version":"build-2803163510"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,7,19]],"date-time":"2024-07-19T00:00:00Z","timestamp":1721347200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,7,19]],"date-time":"2024-07-19T00:00:00Z","timestamp":1721347200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001809","name":"Chinese National Natural Science Foundation Projects","doi-asserted-by":"crossref","award":["62176256, U23B2054"],"award-info":[{"award-number":["62176256, U23B2054"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100012401","name":"Beijing Science and Technology Planning Project","doi-asserted-by":"publisher","award":["Z231100005923033"],"award-info":[{"award-number":["Z231100005923033"]}],"id":[{"id":"10.13039\/501100012401","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004826","name":"Natural Science Foundation of Beijing Municipality","doi-asserted-by":"publisher","award":["L221013"],"award-info":[{"award-number":["L221013"]}],"id":[{"id":"10.13039\/501100004826","id-type":"DOI","asserted-by":"publisher"}]},{"name":"the Youth Innovation Promotion Association CAS","award":["Y2021131"],"award-info":[{"award-number":["Y2021131"]}]},{"name":"InnoHK program","award":["N\/A"],"award-info":[{"award-number":["N\/A"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Vis. Intell."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Facial mesh tracking enables the production of topologically consistent 3D facial meshes from stereo video input captured by calibrated cameras. This technology is an integral part of many digital human applications, such as personalized avatar creation, audio-driven 3D facial animation, and talking face video generation. Currently, most facial mesh tracking methods are built on computer graphics techniques, which involve complex procedures and often necessitate human annotation within pipelines. As a result, these approaches are difficult to implement and hard to generalize across various scenarios. We propose a backpropagation-based solution that formulates facial mesh tracking as a differentiable optimization problem called the BPMT. Our solution leverages visual clues extracted from the stereo input to estimate vertex-wise geometry and texture information. The BPMT is composed of two steps: automatic face analysis and mesh tracking. In the first step, a range of visual clues are automatically extracted from the input, including facial point clouds, multi-view 2D landmarks, 3D landmarks in the world coordinate system, motion fields, and image masks. The second step can be viewed as a differentiable optimization problem, with constraints comprising stereo video input and facial clues. The primary objective is to achieve topologically consistent 3D facial meshes across frames. Additionally, the parameters to be optimized encompass the positions of free-form deformed vertices and a shared texture UV map. Furthermore, the 3D morphable model (3DMM) is introduced as a form of regularization to enhance the convergence of the optimization process. Leveraging the fully developed backpropagation software, we progressively register the facial meshes to the recorded object, generating high-quality 3D faces with consistent topologies. The BPMT requires no manual labeling within the pipeline, making it suitable for producing large-scale stereo facial data. Moreover, our method exhibits a high degree of flexibility and extensibility, positioning it as a promising platform for future research in the community.<\/jats:p>","DOI":"10.1007\/s44267-024-00054-x","type":"journal-article","created":{"date-parts":[[2024,7,19]],"date-time":"2024-07-19T10:01:37Z","timestamp":1721383297000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Formulating facial mesh tracking as a differentiable optimization problem: a backpropagation-based solution"],"prefix":"10.1007","volume":"2","author":[{"given":"Siran","family":"Peng","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4636-9677","authenticated-orcid":false,"given":"Xiangyu","family":"Zhu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-1282-7310","authenticated-orcid":false,"given":"Dong","family":"Yi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8761-5563","authenticated-orcid":false,"given":"Chen","family":"Qian","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0791-189X","authenticated-orcid":false,"given":"Zhen","family":"Lei","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,7,19]]},"reference":[{"issue":"1","key":"54_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2638549","volume":"34","author":"G. Fyffe","year":"2015","unstructured":"Fyffe, G., Jones, A., Alexander, O., Ichikari, R., & Debevec, P. (2015). Driving high-resolution facial scans with video performance capture. ACM Transactions on Graphics, 34(1), 1\u201314.","journal-title":"ACM Transactions on Graphics"},{"issue":"4","key":"54_CR2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3197517.3201401","volume":"37","author":"S. Lombardi","year":"2018","unstructured":"Lombardi, S., Saragih, J., Simon, T., & Sheikh, Y. (2018). Deep appearance models for face rendering. ACM Transactions on Graphics, 37(4), 1\u201313.","journal-title":"ACM Transactions on Graphics"},{"issue":"4","key":"54_CR3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3306346.3323030","volume":"38","author":"S.-E. Wei","year":"2019","unstructured":"Wei, S.-E., Saragih, J., Simon, T., Harley, A. W., Lombardi, S., Perdoch, M., et al. (2019). VR facial animation via multiview image translation. ACM Transactions on Graphics, 38(4), 1\u201316.","journal-title":"ACM Transactions on Graphics"},{"key":"54_CR4","first-page":"4574","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"W. Zielonka","year":"2023","unstructured":"Zielonka, W., Bolkart, T., & Thies, J. (2023). Instant volumetric head avatars. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp. 4574\u20134584). Piscataway: IEEE."},{"issue":"4","key":"54_CR5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3072959.3073658","volume":"36","author":"T. Karras","year":"2017","unstructured":"Karras, T., Aila, T., Laine, S., Herva, A., & Lehtinen, J. (2017). Audio-driven facial animation by joint end-to-end learning of pose and emotion. ACM Transactions on Graphics, 36(4), 1\u201312.","journal-title":"ACM Transactions on Graphics"},{"key":"54_CR6","first-page":"1173","volume-title":"Proceedings of the IEEE\/CVF international conference on computer vision","author":"A. Richard","year":"2021","unstructured":"Richard, A., Zollh\u00f6fer, M., Wen, Y., de la Torre, F., & Sheikh, Y. (2021). Meshtalk: 3D face animation from speech using cross-modality disentanglement. In Proceedings of the IEEE\/CVF international conference on computer vision (pp. 1173\u20131182). Piscataway: IEEE."},{"key":"54_CR7","first-page":"18770","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Y. Fan","year":"2022","unstructured":"Fan, Y., Lin, Z., Saito, J., Wang, W., & Komura, T. (2022). Faceformer: speech-driven 3D facial animation with transformers. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp. 18770\u201318780). Piscataway: IEEE."},{"issue":"4","key":"54_CR8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3072959.3073640","volume":"36","author":"S. Suwajanakorn","year":"2017","unstructured":"Suwajanakorn, S., Seitz, S. M., & Kemelmacher-Shlizerman, I. (2017). Synthesizing obama: learning lip sync from audio. ACM Transactions on Graphics, 36(4), 1\u201313.","journal-title":"ACM Transactions on Graphics"},{"key":"54_CR9","unstructured":"Yi, R., Ye, Z., Zhang, J., Bao, H., & Liu, Y. (2020). Audio-driven talking face video generation with natural head pose. arXiv preprint. arXiv:2002.10137."},{"key":"54_CR10","first-page":"3867","volume-title":"Proceedings of the IEEE\/CVF international conference on computer vision","author":"C. Zhang","year":"2021","unstructured":"Zhang, C., Zhao, Y., Huang, Y., Zeng, M., Ni, S., Budagavi, M., et al. (2021). Facial: synthesizing dynamic talking face with implicit attribute learning. In Proceedings of the IEEE\/CVF international conference on computer vision (pp. 3867\u20133876). Piscataway: IEEE."},{"issue":"4","key":"54_CR11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2010324.1964970","volume":"30","author":"T. Beeler","year":"2011","unstructured":"Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., et al. (2011). High-quality passive facial performance capture using anchor frames. ACM Transactions on Graphics, 30(4), 1\u201310.","journal-title":"ACM Transactions on Graphics"},{"issue":"2","key":"54_CR12","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1111\/cgf.13127","volume":"36","author":"G. Fyffe","year":"2017","unstructured":"Fyffe, G., Nagano, K., Huynh, L., Saito, S., & Debevec, P. (2017). Multi-view stereo on consistent face topology. Computer Graphics Forum, 36(2), 295\u2013309.","journal-title":"Computer Graphics Forum"},{"issue":"6","key":"54_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3272127.3275101","volume":"37","author":"C. Wu","year":"2018","unstructured":"Wu, C., Shiratori, T., & Sheikh, Y. (2018). Deep incremental learning for efficient high-fidelity face tracking. ACM Transactions on Graphics, 37(6), 1\u201312.","journal-title":"ACM Transactions on Graphics"},{"key":"54_CR14","unstructured":"Wuu, C., Zheng, N., Ardisson, S., Bali, R., Belko, D., Brockmeyer, E., et\u00a0al. (2022). Multiface: a dataset for neural face rendering. arXiv preprint. arXiv:2207.11243."},{"key":"54_CR15","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1145\/15922.15903","volume-title":"Proceedings of the 13th annual conference on computer graphics and interactive techniques","author":"T. W. Sederberg","year":"1986","unstructured":"Sederberg, T. W., & Parry, S. R. (1986). Free-form deformation of solid geometric models. In Proceedings of the 13th annual conference on computer graphics and interactive techniques (pp. 151\u2013160). New York: ACM."},{"issue":"6","key":"54_CR16","first-page":"1","volume":"36","author":"T. Li","year":"2017","unstructured":"Li, T., Bolkart, T., Black, M. J., Li, H., & Romero, J. (2017). Learning a model of facial shape and expression from 4D scans. ACM Transactions on Graphics, 36(6), 1\u201317.","journal-title":"ACM Transactions on Graphics"},{"issue":"3","key":"54_CR17","doi-asserted-by":"publisher","first-page":"475","DOI":"10.1109\/TPAMI.2005.63","volume":"27","author":"S. Vedula","year":"2005","unstructured":"Vedula, S., Rander, P., Collins, R., & Kanade, T. (2005). Three-dimensional scene flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(3), 475\u2013480.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"54_CR18","first-page":"1506","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition","author":"T. Basha","year":"2010","unstructured":"Basha, T., Moses, Y., & Kiryati, N. (2010). Multi-view scene flow estimation: a view centered variational approach. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1506\u20131513). Piscataway: IEEE."},{"key":"54_CR19","first-page":"568","volume-title":"Proceedings of the 11th European conference on computer vision","author":"L. Valgaerts","year":"2010","unstructured":"Valgaerts, L., Bruhn, A., Zimmer, H., Weickert, J., Stoll, C., & Theobalt, C., (2010). Joint estimation of motion, structure and geometry from stereo sequences. In K. Daniilidis, P. Maragos, & N. Paragios (Eds.), Proceedings of the 11th European conference on computer vision (pp. 568\u2013581). Cham: Springer."},{"issue":"4","key":"54_CR20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1778765.1778778","volume":"29","author":"D. Bradley","year":"2010","unstructured":"Bradley, D., Heidrich, W., Popa, T., & Sheffer, A. (2010). High resolution passive facial performance capture. ACM Transactions on Graphics, 29(4), 1\u201310.","journal-title":"ACM Transactions on Graphics"},{"issue":"6","key":"54_CR21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2366145.2366206","volume":"31","author":"L. Valgaerts","year":"2012","unstructured":"Valgaerts, L., Wu, C., Bruhn, A., Seidel, H.-P., & Theobalt, C. (2012). Lightweight binocular facial performance capture under uncontrolled lighting. ACM Transactions on Graphics, 31(6), 1\u201311.","journal-title":"ACM Transactions on Graphics"},{"key":"54_CR22","first-page":"17","volume-title":"Proceedings of the 2nd international conference on 3D imaging, modeling, processing, visualization and transmission","author":"M. Klaudiny","year":"2012","unstructured":"Klaudiny, M., & Hilton, A. (2012). High-detail 3D capture and non-sequential alignment of facial performance. In Proceedings of the 2nd international conference on 3D imaging, modeling, processing, visualization and transmission (pp. 17\u201324). Piscataway: IEEE."},{"key":"54_CR23","first-page":"202","volume-title":"Proceedings of the 5th IEEE international conference on automatic face gesture recognition","author":"V. Blanz","year":"2002","unstructured":"Blanz, V., Romdhani, S., & Vetter, T. (2002). Face identification across different poses and illuminations with a 3D morphable model. In Proceedings of the 5th IEEE international conference on automatic face gesture recognition (pp. 202\u2013207). Piscataway: IEEE."},{"key":"54_CR24","first-page":"146","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition","author":"X. Zhu","year":"2016","unstructured":"Zhu, X., Lei, Z., Liu, X., Shi, H., & Li, S.\u00a0Z. (2016). Face alignment across large poses: a 3D solution. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 146\u2013155). Piscataway: IEEE."},{"key":"54_CR25","first-page":"3935","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition","author":"A. T. Tran","year":"2018","unstructured":"Tran, A. T., Hassner, T., Masi, I., Paz, E., Nirkin, Y., & Medioni, G. G. (2018). Extreme 3D face reconstruction: seeing through occlusions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3935\u20133944). Piscataway: IEEE."},{"key":"54_CR26","first-page":"152","volume-title":"Proceedings of the 16th European conference on computer vision","author":"J. Guo","year":"2020","unstructured":"Guo, J., Zhu, X., Yang, Y., Yang, F., Lei, Z., & Li, S.\u00a0Z. (2020). Towards fast, accurate and stable 3D dense face alignment. In A. Vedaldi, H. Bischof, T. Brox, et al. (Eds.), Proceedings of the 16th European conference on computer vision (pp. 152\u2013168). Cham: Springer."},{"issue":"4","key":"54_CR27","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3450626.3459936","volume":"40","author":"Y. Feng","year":"2021","unstructured":"Feng, Y., Feng, H., Black, M. J., & Bolkart, T. (2021). Learning an animatable detailed 3D face model from in-the-wild images. ACM Transactions on Graphics, 40(4), 1\u201313.","journal-title":"ACM Transactions on Graphics"},{"key":"54_CR28","first-page":"187","volume-title":"Proceedings of the 26th annual conference on computer graphics and interactive techniques","author":"V. Blanz","year":"1999","unstructured":"Blanz, V., & Vetter, T. (1999). A morphable model for the synthesis of 3D faces. In Proceedings of the 26th annual conference on computer graphics and interactive techniques (pp. 187\u2013194). New York: ACM."},{"key":"54_CR29","first-page":"296","volume-title":"Proceedings of the 6th IEEE international conference on advanced video and signal based surveillance","author":"P. Paysan","year":"2009","unstructured":"Paysan, P., Knothe, R., Amberg, B., Romdhani, S., & Vetter, T. (2009). A 3D face model for pose and illumination invariant face recognition. In Proceedings of the 6th IEEE international conference on advanced video and signal based surveillance (pp. 296\u2013301). Piscataway: IEEE."},{"key":"54_CR30","first-page":"343","volume-title":"Proceedings of the 16th European conference on computer vision","author":"X. Zhu","year":"2020","unstructured":"Zhu, X., Yang, F., Huang, D., Yu, C., Wang, H., Guo, J., et al. (2020). Beyond 3DMM space: towards fine-grained 3D face reconstruction. In A. Vedaldi, H. Bischof, T. Brox, et al. (Eds.), Proceedings of the 16th European conference on computer vision (pp. 343\u2013358). Cham: Springer."},{"key":"54_CR31","first-page":"2737","volume-title":"Proceedings of the IEEE international conference on systems, man, and cybernetics","author":"H. Jung","year":"2021","unstructured":"Jung, H., Oh, M.-S., & Lee, S.-W. (2021). Learning free-form deformation for 3D face reconstruction from in-the-wild images. In Proceedings of the IEEE international conference on systems, man, and cybernetics (pp. 2737\u20132742). Piscataway: IEEE."},{"issue":"12","key":"54_CR32","doi-asserted-by":"publisher","first-page":"3174","DOI":"10.1007\/s11263-021-01521-4","volume":"129","author":"H. Jin","year":"2021","unstructured":"Jin, H., Liao, S., & Shao, L. (2021). Pixel-in-pixel net: towards efficient facial landmark detection in the wild. International Journal of Computer Vision, 129(12), 3174\u20133194.","journal-title":"International Journal of Computer Vision"},{"key":"54_CR33","first-page":"593","volume-title":"Proceedings of IEEE conference on computer vision and pattern recognition","author":"J. Shi","year":"1994","unstructured":"Shi, J., & Tomasi (1994). Good features to track. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 593\u2013600). Piscataway: IEEE."},{"key":"54_CR34","unstructured":"Kingma, D. P., & Ba, J. (2017). Adam: a method for stochastic optimization. arXiv preprint. arXiv:1412.6980."}],"container-title":["Visual Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44267-024-00054-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s44267-024-00054-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44267-024-00054-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,19]],"date-time":"2024-07-19T10:18:11Z","timestamp":1721384291000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s44267-024-00054-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,19]]},"references-count":34,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["54"],"URL":"https:\/\/doi.org\/10.1007\/s44267-024-00054-x","relation":{},"ISSN":["2731-9008"],"issn-type":[{"value":"2731-9008","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,19]]},"assertion":[{"value":"7 November 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 July 2024","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 July 2024","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 July 2024","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors affirm that they have no competing interests to declare.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"21"}}