{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,10]],"date-time":"2026-03-10T08:08:34Z","timestamp":1773130114351,"version":"3.50.1"},"reference-count":87,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2024,8,9]],"date-time":"2024-08-09T00:00:00Z","timestamp":1723161600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2024,10,31]]},"abstract":"<jats:p>We present a neural network-based simulation super-resolution framework that can efficiently and realistically enhance a facial performance produced by a low-cost, real-time physics-based simulation to a level of detail that closely approximates that of a reference-quality off-line simulator with much higher resolution (27\u00d7 element count in our examples) and accurate physical modeling. Our approach is rooted in our ability to construct a training set of paired frames, from the low- and high-resolution simulators respectively, that are in semantic correspondence with each other. We use face animation as an exemplar of such a simulation domain, where creating this semantic congruence is achieved by simply dialing in the same muscle actuation controls and skeletal pose in the two simulators. Our proposed neural network super-resolution framework generalizes from this training set to unseen expressions, compensates for modeling discrepancies between the two simulations due to limited resolution or cost-cutting approximations in the real-time variant, and does not require any semantic descriptors or parameters to be provided as input, other than the result of the real-time simulation. We evaluate the efficacy of our pipeline on a variety of expressive performances and provide comparisons and ablation experiments for plausible variations and alternatives to our proposed scheme. Our code is available at https:\/\/github.com\/hjoonpark\/3d-sim-super- res.git.<\/jats:p>","DOI":"10.1145\/3670687","type":"journal-article","created":{"date-parts":[[2024,7,11]],"date-time":"2024-07-11T08:57:57Z","timestamp":1720688277000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Near-realtime Facial Animation by Deep 3D Simulation Super-Resolution"],"prefix":"10.1145","volume":"43","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3796-9777","authenticated-orcid":false,"given":"Hyojoon","family":"Park","sequence":"first","affiliation":[{"name":"University of Wisconsin-Madison, Madison, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6508-7256","authenticated-orcid":false,"given":"Sangeetha","family":"Grama Srinivasan","sequence":"additional","affiliation":[{"name":"University of Wisconsin-Madison, Madison, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2956-2050","authenticated-orcid":false,"given":"Matthew","family":"Cong","sequence":"additional","affiliation":[{"name":"NVIDIA, Santa Clara, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8932-5519","authenticated-orcid":false,"given":"Doyub","family":"Kim","sequence":"additional","affiliation":[{"name":"NVIDIA, Santa Clara, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4482-8363","authenticated-orcid":false,"given":"Byungsoo","family":"Kim","sequence":"additional","affiliation":[{"name":"NVIDIA, Z\u00fcrich, Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1959-6396","authenticated-orcid":false,"given":"Jonathan","family":"Swartz","sequence":"additional","affiliation":[{"name":"NVIDIA, Santa Clara, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9926-780X","authenticated-orcid":false,"given":"Ken","family":"Museth","sequence":"additional","affiliation":[{"name":"NVIDIA, Santa Clara, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5608-3085","authenticated-orcid":false,"given":"Eftychios","family":"Sifakis","sequence":"additional","affiliation":[{"name":"University of Wisconsin-Madison, Madison, USA and NVIDIA, Santa Clara, USA"}]}],"member":"320","published-online":{"date-parts":[[2024,8,9]]},"reference":[{"key":"e_1_3_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/VISUAL.2001.964489"},{"key":"e_1_3_2_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2003.1175093"},{"key":"e_1_3_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2508363.2508415"},{"key":"e_1_3_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1409060.1409118"},{"key":"e_1_3_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2614028.2615425"},{"key":"e_1_3_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392397"},{"key":"e_1_3_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01106"},{"key":"e_1_3_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1073204.1073300"},{"key":"e_1_3_2_10_1","first-page":"1","volume-title":"Eurographics (State of the Art Reports)","author":"Bender Jan","year":"2013","unstructured":"Jan Bender, Matthias M\u00fcller, Miguel A. Otaduy, and Matthias Teschner. 2013. Position-based methods for the simulation of solid objects in computer graphics. In Eurographics (State of the Art Reports), Jan Bender, Matthias M\u00fcller, Miguel A. Otaduy, and Matthias Teschner (Eds.). EUROGRAPHICS, 1\u201322."},{"key":"e_1_3_2_11_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12802"},{"key":"e_1_3_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276439"},{"key":"e_1_3_2_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33863-2_8"},{"key":"e_1_3_2_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2014.2337258"},{"key":"e_1_3_2_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2016.2601059"},{"key":"e_1_3_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601097.2601116"},{"key":"e_1_3_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/383259.383266"},{"key":"e_1_3_2_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58526-6_36"},{"key":"e_1_3_2_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00574"},{"key":"e_1_3_2_20_1","doi-asserted-by":"crossref","unstructured":"Peter Yichen Chen Maurizio M. Chiaramonte Eitan Grinspun and Kevin Carlberg. 2023. Model reduction for the material point method via an implicit neural representation of the deformation map. Journal of Computational Physics 478 (1 April 2023) 111908. https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0021999123000037","DOI":"10.1016\/j.jcp.2023.111908"},{"key":"e_1_3_2_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00852"},{"key":"e_1_3_2_22_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0077576"},{"key":"e_1_3_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2786784.2786786"},{"key":"e_1_3_2_24_1","first-page":"119","volume-title":"Proceedings of the ACM SIGGRAPH\/Eurographics Symposium on Computer Animation (SCA\u201916)","author":"Cong Matthew","year":"2016","unstructured":"Matthew Cong, Kiran S. Bhat, and Ronald Fedkiw. 2016. Art-directed muscle simulation for high-end facial animation. In Proceedings of the ACM SIGGRAPH\/Eurographics Symposium on Computer Animation (SCA\u201916). Eurographics Association, Goslar, DEU, 119\u2013127."},{"key":"e_1_3_2_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAES.2016.140952"},{"key":"e_1_3_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/237170.237269"},{"key":"e_1_3_2_27_1","first-page":"1","volume-title":"SC20: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis","year":"2020","unstructured":"Soheil Esmaeilzadeh, Kamyar Azizzadenesheli, Karthik Kashinath, Mustafa Mustafa, Hamdi A. Tchelepi, Philip Marcus, Mr Prabhat, and Anima Anandkumar. 2020. MeshfreeFlowNet: A physics-constrained deep continuous space-time super-resolution framework. In SC20: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1\u201315."},{"key":"e_1_3_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1073204.1073227"},{"key":"e_1_3_2_29_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13645"},{"key":"e_1_3_2_30_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467-8659.00524"},{"key":"e_1_3_2_31_1","unstructured":"Irina Higgins Loic Matthey Arka Pal Christopher Burgess Xavier Glorot Matthew Botvinick Shakir Mohamed and Alexander Lerchner. 2016. beta-VAE: Learning basic visual concepts with a constrained variational framework. (2016)."},{"key":"e_1_3_2_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00604"},{"key":"e_1_3_2_33_1","volume-title":"Proceedings of the 4th Eurographics Symposium on Geometry Processing","volume":"7","author":"Kazhdan Michael","year":"2006","unstructured":"Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006. Poisson surface reconstruction. In Proceedings of the 4th Eurographics Symposium on Geometry Processing, Vol. 7."},{"key":"e_1_3_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487228.2487237"},{"key":"e_1_3_2_35_1","volume-title":"Geometric, Variational Integrators for Computer Animation","author":"Kharevych Liliya","year":"2006","unstructured":"Liliya Kharevych, W. Wei, Yiying Tong, Eva Kanso, Jerrold E. Marsden, Peter Schr\u00f6der, and Matthieu Desbrun. 2006. Geometric, Variational Integrators for Computer Animation. Eurographics Association."},{"key":"e_1_3_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1360612.1360649"},{"key":"e_1_3_2_37_1","doi-asserted-by":"publisher","unstructured":"Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. DOI:10.48550\/ARXIV.1412.6980","DOI":"10.48550\/ARXIV.1412.6980"},{"key":"e_1_3_2_38_1","doi-asserted-by":"publisher","DOI":"10.1002\/nme.167"},{"key":"e_1_3_2_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/IJCB52358.2021.9484379"},{"key":"e_1_3_2_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00730"},{"key":"e_1_3_2_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/3DV.2014.67"},{"issue":"6","key":"e_1_3_2_42_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2508363.2508406","article-title":"Fast simulation of mass-spring systems","volume":"32","author":"Liu Tiantian","year":"2013","unstructured":"Tiantian Liu, Adam W. Bargteil, James F. O\u2019Brien, and Ladislav Kavan. 2013. Fast simulation of mass-spring systems. ACM Transactions on Graphics (TOG) 32, 6 (2013), 1\u20137.","journal-title":"ACM Transactions on Graphics (TOG)"},{"key":"e_1_3_2_43_1","doi-asserted-by":"publisher","DOI":"10.1002\/fld.1650200824"},{"key":"e_1_3_2_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00013"},{"key":"e_1_3_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2994258.2994272"},{"key":"e_1_3_2_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00459"},{"key":"e_1_3_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3503250"},{"key":"e_1_3_2_48_1","first-page":"103","volume-title":"Proceedings of the International Meshing Roundtable","author":"Molino Neil","year":"2003","unstructured":"Neil Molino, Robert Bridson, Joseph Teran, and Ronald Fedkiw. 2003. A crystalline, red green strategy for meshing highly deformable objects with tetrahedra. In Proceedings of the International Meshing Roundtable. Citeseer, 103\u2013114."},{"key":"e_1_3_2_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jvcir.2007.01.005"},{"key":"e_1_3_2_50_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2009.01511.x"},{"key":"e_1_3_2_51_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00138-014-0623-4"},{"key":"e_1_3_2_52_1","doi-asserted-by":"publisher","DOI":"10.5555\/1281920.1281944"},{"key":"e_1_3_2_53_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.gmod.2004.06.003"},{"key":"e_1_3_2_54_1","doi-asserted-by":"publisher","DOI":"10.1007\/11744047_30"},{"key":"e_1_3_2_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00025"},{"key":"e_1_3_2_56_1","first-page":"II\u2013382","volume-title":"Proceedings of the IEEE International Conference on Image Processing 2005","volume":"2","author":"Peng Shiqi","year":"2005","unstructured":"Shiqi Peng, Gang Pan, and Zhaohui Wu. 2005. Learning-based super-resolution of 3D face model. In Proceedings of the IEEE International Conference on Image Processing 2005. Vol. 2, IEEE, II\u2013382."},{"key":"e_1_3_2_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01151"},{"key":"e_1_3_2_58_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58529-7_44"},{"key":"e_1_3_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130800.3130883"},{"key":"e_1_3_2_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00239"},{"key":"e_1_3_2_61_1","doi-asserted-by":"crossref","unstructured":"S. Shen Y. Yang T. Shao H. Wang C. Jiang L. Lan and K. Zhou. 2021. High-order differentiable autoencoder for nonlinear model reduction. ACM Transactions on Graphics 40 4 (2021).","DOI":"10.1145\/3450626.3459754"},{"key":"e_1_3_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/1186822.1073208"},{"key":"e_1_3_2_63_1","first-page":"7462","article-title":"Implicit neural representations with periodic activation functions","author":"Sitzmann Vincent","year":"2020","unstructured":"Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. 2020. Implicit neural representations with periodic activation functions. In Proceedings of the 34th International Conference on Neural Information Processing Systems. 7462\u20137473.","journal-title":"Proceedings of the 34th International Conference on Neural Information Processing Systems"},{"key":"e_1_3_2_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459883"},{"key":"e_1_3_2_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/CADCG.2009.5246818"},{"key":"e_1_3_2_66_1","doi-asserted-by":"publisher","DOI":"10.1137\/080732936"},{"key":"e_1_3_2_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2012.132"},{"key":"e_1_3_2_68_1","first-page":"7537","article-title":"Fourier features let networks learn high frequency functions in low dimensional domains","author":"Tancik Matthew","year":"2020","unstructured":"Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. 2020. Fourier features let networks learn high frequency functions in low dimensional domains. In Proceedings of the 34th International Conference on Neural Information Processing Systems. 7537\u20137547.","journal-title":"Proceedings of the 34th International Conference on Neural Information Processing Systems"},{"key":"e_1_3_2_69_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.14199"},{"key":"e_1_3_2_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766904"},{"key":"e_1_3_2_71_1","doi-asserted-by":"publisher","DOI":"10.5555\/846276.846285"},{"key":"e_1_3_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/1073368.1073394"},{"key":"e_1_3_2_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/571647.571650"},{"key":"e_1_3_2_74_1","article-title":"Attention is all you need","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems.","journal-title":"Proceedings of the 31st International Conference on Neural Information Processing Systems"},{"key":"e_1_3_2_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766952"},{"key":"e_1_3_2_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3326362"},{"issue":"4","key":"e_1_3_2_77_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3197517.3201304","article-title":"tempoGAN: A temporally coherent, volumetric GAN for super-resolution fluid flow","volume":"37","author":"Xie You","year":"2018","unstructured":"You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. 2018. tempoGAN: A temporally coherent, volumetric GAN for super-resolution fluid flow. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1\u201315.","journal-title":"ACM Transactions on Graphics (TOG)"},{"key":"e_1_3_2_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530125"},{"key":"e_1_3_2_79_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2019.2919431"},{"key":"e_1_3_2_80_1","article-title":"Meta-PU: An arbitrary-scale upsampling network for point cloud","author":"Ye Shuquan","year":"2021","unstructured":"Shuquan Ye, Dongdong Chen, Songfang Han, Ziyu Wan, and Jing Liao. 2021. Meta-PU: An arbitrary-scale upsampling network for point cloud. IEEE Transactions on Visualization and Computer Graphics 28, 9 (2021), 3206\u20133218.","journal-title":"IEEE Transactions on Visualization and Computer Graphics"},{"key":"e_1_3_2_81_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00611"},{"key":"e_1_3_2_82_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01234-2_24"},{"key":"e_1_3_2_83_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00295"},{"key":"e_1_3_2_84_1","doi-asserted-by":"publisher","DOI":"10.3390\/app10031047"},{"key":"e_1_3_2_85_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.142642"},{"key":"e_1_3_2_86_1","doi-asserted-by":"publisher","DOI":"10.3390\/a15040124"},{"key":"e_1_3_2_87_1","doi-asserted-by":"publisher","DOI":"10.1145\/3610548.3618207"},{"key":"e_1_3_2_88_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2012.79"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3670687","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3670687","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:58:20Z","timestamp":1750294700000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3670687"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,9]]},"references-count":87,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,10,31]]}},"alternative-id":["10.1145\/3670687"],"URL":"https:\/\/doi.org\/10.1145\/3670687","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,9]]},"assertion":[{"value":"2023-10-14","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-22","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-08-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}