{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"bioRxiv"}],"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T06:04:24Z","timestamp":1782281064146,"version":"3.54.5"},"posted":{"date-parts":[[2025,2,16]]},"group-title":"Neuroscience","reference-count":50,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2025,2,16]],"date-time":"2025-02-16T00:00:00Z","timestamp":1739664000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2025,2,16]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:p>\n                  Speech movements are highly complex and require precise tuning of both spatial and timing of oral articulators to support intelligible communication. These properties also make measurement of speech movements challenging, often requiring extensive physical sensors placed around the mouth and face that are not easily tolerated by certain populations such as young children. Recent progress in machine learning-based markerless facial landmark tracking technology demonstrated its potential to provide lip tracking without the need for physical sensors, but whether such technology can provide submillimeter precision and accuracy in 3D remains unknown. Moreover, it is also unclear whether such technology can be applied to track speech movements in young children. Here, we developed a novel approach that integrates Shape Preserving Facial Landmarks with Graph Attention Networks (SPIGA), a facial landmark detector, and CoTracker, a transformer-based neural network model that jointly tracks dense points across a video sequence. We further examined and validated this novel approach by assessing its tracking precision and accuracy. The findings revealed that our approach that integrates SPIGA and CoTracker was more precise (\u2248 0.15 mm in standard deviation) than SPIGA alone (\u2248 0.35 mm). In addition, its 3D tracking performance was comparable to electromagnetic articulography (\u2248 0.29 mm RMSE against simultaneously recorded articulograph data). Importantly, the approach performed similarly well across adults and young children (\n                  <jats:italic>i<\/jats:italic>\n                  .\n                  <jats:italic>e<\/jats:italic>\n                  ., 3- and 4-year-olds). Because our framework is built upon open-source pretrained models that are fully trained, it promotes accessibility and open science while saving computing resources. Furthermore, given that this framework combines a landmark detection model (SPIGA) with a tracker model (CoTracker) to improve precision\/accuracy, our novel approach serves as a proof-of-concept for enhancing the performance of a wide variety of commonly used markerless tracking applications in biology and neuroscience.\n                <\/jats:p>\n                <jats:sec>\n                  <jats:title>Author summary<\/jats:title>\n                  <jats:p>\n                    In this work, we examined whether machine learning based markerless tracking is feasible for tracking 3D lip movements in adults and young children. We developed a novel approach that integrates a landmark detection model (SPIGA) with a tracker model (CoTracker). Our combined CoTracker-based approach demonstrated submillimeter precision and accuracy desired for speech kinematic recording. In addition, our approach does not involve training and validation for each population (\n                    <jats:italic>e<\/jats:italic>\n                    .\n                    <jats:italic>g<\/jats:italic>\n                    ., young children vs. adults), saving time and computing resources. We foresee that the proposed general framework of fusing a landmark detection model with a tracker model can be generalized for a wide variety of tracking applications in biology and neuroscience that require high precision and accuracy, including studying cell behaviors, animal motions, and other types of human movements.\n                  <\/jats:p>\n                <\/jats:sec>","DOI":"10.1101\/2025.02.13.638009","type":"posted-content","created":{"date-parts":[[2025,2,16]],"date-time":"2025-02-16T15:40:20Z","timestamp":1739720420000},"source":"Crossref","is-referenced-by-count":0,"title":["3D markerless tracking of speech movements with submillimeter accuracy"],"prefix":"10.64898","author":[{"given":"Austin","family":"Lovell","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5869-0241","authenticated-orcid":false,"given":"Arielle","family":"Borovsky","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4375-0680","authenticated-orcid":false,"given":"Raymond A.","family":"Yeh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3600-7090","authenticated-orcid":false,"given":"Kwang S.","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"54368","reference":[{"key":"2025021808400655000_2025.02.13.638009v1.1","doi-asserted-by":"publisher","DOI":"10.3389\/fnhum.2014.00911"},{"key":"2025021808400655000_2025.02.13.638009v1.2","doi-asserted-by":"publisher","DOI":"10.1044\/2018-S-17-0222"},{"key":"2025021808400655000_2025.02.13.638009v1.3","doi-asserted-by":"publisher","DOI":"10.1044\/jslhr.4301.239"},{"key":"2025021808400655000_2025.02.13.638009v1.4","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcomdis.2009.03.008"},{"key":"2025021808400655000_2025.02.13.638009v1.5","doi-asserted-by":"publisher","DOI":"10.1044\/2022-22-00290"},{"key":"2025021808400655000_2025.02.13.638009v1.6","doi-asserted-by":"publisher","DOI":"10.1044\/jslhr.4302.521"},{"key":"2025021808400655000_2025.02.13.638009v1.7","doi-asserted-by":"publisher","DOI":"10.1186\/s11689-015-9123-8"},{"key":"2025021808400655000_2025.02.13.638009v1.8","doi-asserted-by":"publisher","DOI":"10.1044\/1092-4388(2003\/017)"},{"key":"2025021808400655000_2025.02.13.638009v1.9","doi-asserted-by":"publisher","DOI":"10.1044\/2017-S-17-0075"},{"issue":"1","key":"2025021808400655000_2025.02.13.638009v1.10","first-page":"1","article-title":"Speech in ALS: Longitudinal Changes in Lips and Jaw Movements and Vowel Acoustics","volume":"21","year":"2013","journal-title":"Journal of Medical Speech-Language Pathology"},{"key":"2025021808400655000_2025.02.13.638009v1.11","doi-asserted-by":"publisher","DOI":"10.1044\/cicsd_30_S_5"},{"key":"2025021808400655000_2025.02.13.638009v1.12","doi-asserted-by":"publisher","DOI":"10.1044\/jshr.2604.616"},{"key":"2025021808400655000_2025.02.13.638009v1.13","doi-asserted-by":"publisher","DOI":"10.1044\/jshr.2602.283"},{"key":"2025021808400655000_2025.02.13.638009v1.14","unstructured":"Westbury J R. X-ray microbeam speech production database user\u2019s manual (Version 1.0). Madison, WI: University of Wisconsin; 1994. 2"},{"key":"2025021808400655000_2025.02.13.638009v1.15","doi-asserted-by":"publisher","DOI":"10.1121\/1.414969"},{"key":"2025021808400655000_2025.02.13.638009v1.16","doi-asserted-by":"publisher","DOI":"10.1044\/1092-4388(2005\/037)"},{"key":"2025021808400655000_2025.02.13.638009v1.17","doi-asserted-by":"publisher","DOI":"10.1044\/1092-4388(2008\/07-0218)"},{"key":"2025021808400655000_2025.02.13.638009v1.18","doi-asserted-by":"publisher","DOI":"10.1044\/2018-L-18-0055"},{"key":"2025021808400655000_2025.02.13.638009v1.19","doi-asserted-by":"publisher","DOI":"10.1044\/2014-S-13-0007"},{"key":"2025021808400655000_2025.02.13.638009v1.20","doi-asserted-by":"publisher","DOI":"10.3390\/s24103022"},{"key":"2025021808400655000_2025.02.13.638009v1.21","doi-asserted-by":"publisher","DOI":"10.1186\/s12984-023-01186-9"},{"key":"2025021808400655000_2025.02.13.638009v1.22","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-022-01443-0"},{"key":"2025021808400655000_2025.02.13.638009v1.23","doi-asserted-by":"publisher","DOI":"10.3390\/s22031133"},{"key":"2025021808400655000_2025.02.13.638009v1.24","doi-asserted-by":"crossref","unstructured":"Zhou Z , Li H , Liu H , Wang N , Yu G , Ji R. Star loss: Reducing semantic ambiguity in facial landmark detection. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition; 2023. p. 15475\u201315484. 2","DOI":"10.1109\/CVPR52729.2023.01485"},{"key":"2025021808400655000_2025.02.13.638009v1.25","doi-asserted-by":"crossref","unstructured":"Gao Z , Patras I. Self-Supervised Facial Representation Learning with Facial Region Awareness. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2024. p. 2081\u20132092. 2","DOI":"10.1109\/CVPR52733.2024.00203"},{"key":"2025021808400655000_2025.02.13.638009v1.26","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-018-1097-z"},{"key":"2025021808400655000_2025.02.13.638009v1.27","unstructured":"Prados-Torreblanca A , Buenaposada JM , Baumela L. Shape Preserving Facial Landmarks with Graph Attention Networks; 2022. Available from: https:\/\/arxiv.org\/abs\/2210.07233. 2, 3"},{"key":"2025021808400655000_2025.02.13.638009v1.28","doi-asserted-by":"crossref","unstructured":"Burgos-Artizzu XP , Perona P , Dollar P. Robust Face Landmark Estimation under Occlusion. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV); 2013. 2","DOI":"10.1109\/ICCV.2013.191"},{"key":"2025021808400655000_2025.02.13.638009v1.29","doi-asserted-by":"crossref","unstructured":"Sagonas C , Tzimiropoulos G , Zafeiriou S , Pantic M. Faces inthe-Wild Challenge : The first facial landmark localization Challenge; 2013. Available from: https:\/\/api.semanticscholar.org\/CorpusID:7495339. 2","DOI":"10.1109\/ICCVW.2013.59"},{"key":"2025021808400655000_2025.02.13.638009v1.30","unstructured":"Karaev N , Rocco I , Graham B , Neverova N , Vedaldi A , Rupprecht C. CoTracker: It is Better to Track Together; 2023. Available from: http:\/\/arxiv.org\/abs\/2307.07635. 3, 14"},{"key":"2025021808400655000_2025.02.13.638009v1.31","doi-asserted-by":"publisher","DOI":"10.1044\/2016-S-15-0223"},{"key":"2025021808400655000_2025.02.13.638009v1.32","unstructured":"Bradski G. The OpenCV Library. Dr Dobb\u2019s Journal of Software Tools. 2000;. 3, 4, 5, 13"},{"key":"2025021808400655000_2025.02.13.638009v1.33","unstructured":"ffmpeg developers. ffmpeg tool (Version 4.4.2); 2024. Available from: http:\/\/ffmpeg.org\/. 3, 4, 14"},{"key":"2025021808400655000_2025.02.13.638009v1.34","doi-asserted-by":"crossref","unstructured":"Harley AW , Fang Z , Fragkiadaki K. Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories; 2022. Available from: https:\/\/arxiv.org\/abs\/2204.04153. 4","DOI":"10.1007\/978-3-031-20047-2_4"},{"key":"2025021808400655000_2025.02.13.638009v1.35","unstructured":"Doersch C , Gupta A , Markeeva L , Recasens A , Smaira L , Aytar Y , et al. TAP-Vid: A Benchmark for Tracking Any Point in a Video; 2023. Available from: https:\/\/arxiv.org\/abs\/2211.03726. 4"},{"key":"2025021808400655000_2025.02.13.638009v1.36","doi-asserted-by":"publisher","DOI":"10.3390\/s20216324"},{"key":"2025021808400655000_2025.02.13.638009v1.37","doi-asserted-by":"publisher","DOI":"10.1044\/jslhr.4101.18"},{"key":"2025021808400655000_2025.02.13.638009v1.38","unstructured":"Ravi N , Gabeur V , Hu YT , Hu R , Ryali C , Ma T , et al. SAM 2: Segment Anything in Images and Videos. arXiv preprint arXiv:240800714. 2024;. 10, 14"},{"key":"2025021808400655000_2025.02.13.638009v1.39","unstructured":"Rombach R , Blattmann A , Lorenz D , Esser P , Ommer B. High-Resolution Image Synthesis With Latent Diffusion Models. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2022. p. 10684\u201310695. 10, 14, 15"},{"issue":"8476","key":"2025021808400655000_2025.02.13.638009v1.40","first-page":"307","article-title":"Statistical methods for assessing agreement between two methods of clinical measurement","volume":"1","year":"1986","journal-title":"Lancet (London, England)"},{"key":"2025021808400655000_2025.02.13.638009v1.41","doi-asserted-by":"publisher","DOI":"10.1044\/2020_JSLHR-20-00276"},{"key":"2025021808400655000_2025.02.13.638009v1.42","doi-asserted-by":"publisher","DOI":"10.1016\/j.specom.2018.10.001"},{"key":"2025021808400655000_2025.02.13.638009v1.43","unstructured":"Dawson JI , Stout CE , Eyer JA . SPELT-3: structured photographic expressive language test. Janelle Publications DeKalb, IL; 2003. 13"},{"key":"2025021808400655000_2025.02.13.638009v1.44","unstructured":"Van Rossum G , Drake FL . Python 3 Reference Manual; 2009. 13"},{"key":"2025021808400655000_2025.02.13.638009v1.45","doi-asserted-by":"crossref","unstructured":"Hartley RI , Zisserman A. Multiple View Geometry in Computer Vision. 2nd ed. Cambridge University Press, ISBN: 0521540518; 2004. 14","DOI":"10.1017\/CBO9780511811685"},{"key":"2025021808400655000_2025.02.13.638009v1.46","unstructured":"R Core Team. R: A Language and Environment for Statistical Computing; 2024. Available from: https:\/\/www.R-project.org\/. 15"},{"key":"2025021808400655000_2025.02.13.638009v1.47","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v067.i01"},{"key":"2025021808400655000_2025.02.13.638009v1.48","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v082.i13"},{"key":"2025021808400655000_2025.02.13.638009v1.49","unstructured":"Lenth RV . emmeans: Estimated Marginal Means, aka Least-Squares Means; 2025. Available from: https:\/\/rvlenth.github.io\/emmeans\/. 15"},{"key":"2025021808400655000_2025.02.13.638009v1.50","unstructured":"MathWorks. MATLAB version: 9.13.0 (R2022b); 2022. Available from: https:\/\/www.mathworks.com. 16"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1101\/2025.02.13.638009","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T05:14:23Z","timestamp":1782278063000},"score":1,"resource":{"primary":{"URL":"http:\/\/biorxiv.org\/lookup\/doi\/10.1101\/2025.02.13.638009"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,16]]},"references-count":50,"URL":"https:\/\/doi.org\/10.1101\/2025.02.13.638009","relation":{"is-preprint-of":[{"id-type":"doi","id":"10.1109\/TNSRE.2026.3688903","asserted-by":"subject"}]},"subject":[],"published":{"date-parts":[[2025,2,16]]},"subtype":"preprint"}}