{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T16:03:24Z","timestamp":1788278604587,"version":"build-2803163510"},"reference-count":40,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2022,2,2]],"date-time":"2022-02-02T00:00:00Z","timestamp":1643760000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Automatic feature extraction from images of speech articulators is currently achieved by detecting edges. Here, we investigate the use of pose estimation deep neural nets with transfer learning to perform markerless estimation of speech articulator keypoints using only a few hundred hand-labelled images as training input. Midsagittal ultrasound images of the tongue, jaw, and hyoid and camera images of the lips were hand-labelled with keypoints, trained using DeepLabCut and evaluated on unseen speakers and systems. Tongue surface contours interpolated from estimated and hand-labelled keypoints produced an average mean sum of distances (MSD) of 0.93, s.d. 0.46 mm, compared with 0.96, s.d. 0.39 mm, for two human labellers, and 2.3, s.d. 1.5 mm, for the best performing edge detection algorithm. A pilot set of simultaneous electromagnetic articulography (EMA) and ultrasound recordings demonstrated partial correlation among three physical sensor positions and the corresponding estimated keypoints and requires further investigation. The accuracy of the estimating lip aperture from a camera video was high, with a mean MSD of 0.70, s.d. 0.56 mm compared with 0.57, s.d. 0.48 mm for two human labellers. DeepLabCut was found to be a fast, accurate and fully automatic method of providing unique kinematic data for tongue, hyoid, jaw, and lips.<\/jats:p>","DOI":"10.3390\/s22031133","type":"journal-article","created":{"date-parts":[[2022,2,6]],"date-time":"2022-02-06T20:38:40Z","timestamp":1644179920000},"page":"1133","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":39,"title":["Beyond the Edge: Markerless Pose Estimation of Speech Articulators from Ultrasound and Camera Images Using DeepLabCut"],"prefix":"10.3390","volume":"22","author":[{"given":"Alan","family":"Wrench","sequence":"first","affiliation":[{"name":"Clinical Audiology, Speech and Language Research Centre, Queen Margaret University, Musselburgh EH21 6UU, UK"},{"name":"Articulate Instruments Ltd., Musselburgh EH21 6UU, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathan","family":"Balch-Tomes","sequence":"additional","affiliation":[{"name":"Articulate Instruments Ltd., Musselburgh EH21 6UU, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,2,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"137","DOI":"10.21248\/zaspil.42.2005.276","article-title":"On the complex nature of speech kinematics","volume":"42","author":"Fuchs","year":"2005","journal-title":"ZAS Pap. Linguist."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1007\/BF00133570","article-title":"Snakes: Active contour models","volume":"1","author":"Kass","year":"1988","journal-title":"Int. J. Comput. Vis."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"545","DOI":"10.1080\/02699200500113616","article-title":"Automatic contour tracking in ultrasound images","volume":"19","author":"Li","year":"2005","journal-title":"Clin. Linguist. Phon."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Csap\u00f3, T.G., and Lulich, S.M. (2015, January 6\u201310). Error analysis of extracted tongue contours from 2D ultrasound images. Proceedings of the Sixteenth Annual Conference of the International Speech Communication Association, Dresden, Germany.","DOI":"10.21437\/Interspeech.2015-486"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1016\/j.media.2017.12.003","article-title":"Multi-hypothesis tracking of the tongue surface in ultrasound video recordings of normal and impaired speech","volume":"44","author":"Laporte","year":"2018","journal-title":"Med. Image Anal."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Fasel, I., and Berry, J. (2010, January 23\u201326). Deep belief networks for real-time extraction of tongue contours from ultrasound during speech. Proceedings of the 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey.","DOI":"10.1109\/ICPR.2010.369"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Fabre, D., Hueber, T., Bocquelet, F., and Badin, P. (2015, January 6\u201310). Tongue tracking in ultrasound images using eigentongue decomposition and artificial neural networks. Proceedings of the 16th Annual Conference of the International Speech Communication Association (Interspeech 2015), Dresden, Germany.","DOI":"10.21437\/Interspeech.2015-521"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"EL154","DOI":"10.1121\/1.4951024","article-title":"A comparative study on the contour tracking algorithms in ultrasound tongue images with automatic re-initialization","volume":"139","author":"Xu","year":"2016","journal-title":"J. Acoust. Soc. Am."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Mozaffari, M.H., Yamane, N., and Lee, W. (2020, January 16\u201319). Deep Learning for Automatic Tracking of Tongue Surface in Real-time Ultrasound Videos, Landmarks instead of Contours. Proceedings of the 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Seoul, Korea.","DOI":"10.1109\/BIBM49941.2020.9313262"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Aslan, E., and Akgul, Y.S. (2019, January 10\u201313). Tongue Contour Tracking in Ultrasound Images with Spatiotemporal LSTM Networks. Proceedings of the German Conference on Pattern Recognition, Dortmund, Germany.","DOI":"10.1007\/978-3-030-33676-9_36"},{"key":"ref_11","unstructured":"Zhu, J., Styler, W., and Calloway, I. (arXiv, 2019). A CNN-based tool for automatic tongue contour tracking in ultrasound images, arXiv."},{"key":"ref_12","unstructured":"Chen, W., Tiede, M., and Whalen, D.H. (2021, February 05). DeepEdge: Automatic Ultrasound Tongue Contouring Combining a Deep Neural Network and an Edge Detection Algorithm. Available online: https:\/\/issp2020.yale.edu\/S05\/chen_05_16_161_poster.pdf."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5\u20139). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1503","DOI":"10.1016\/j.media.2012.07.001","article-title":"Tongue contour tracking in dynamic ultrasound via higher-order MRFs and efficient fusion moves","volume":"16","author":"Tang","year":"2012","journal-title":"Med. Image Anal."},{"key":"ref_15","unstructured":"Jaumard-Hakoun, A., Xu, K., Roussel-Ragot, P., Dreyfus, G., and Denby, B. (arXiv, 2016). Tongue contour extraction from ultrasound images based on deep neural network, arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1192","DOI":"10.1109\/83.605417","article-title":"Lipreading from color video","volume":"6","author":"Chiou","year":"1997","journal-title":"IEEE Trans. Image Process."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Luettin, J., Thacker, N.A., and Beet, S.W. (1996, January 3\u20136). Speechreading using shape and intensity information. Proceedings of the Fourth International Conference on Spoken Language Processing, ICSLP\u201996, Philadelphia, PA, USA.","DOI":"10.1109\/ICSLP.1996.607024"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Kaucic, R., Dalton, B., and Blake, A. (1996, January 15\u201318). Real-time lip tracking for audio-visual speech recognition applications. Proceedings of the European Conference on Computer Vision, Cambridge, UK.","DOI":"10.1007\/3-540-61123-1_154"},{"key":"ref_19","unstructured":"Lallouache, M.T. (1991). Un Poste\u201c Visage-Parole\u201d Couleur: Acquisition et Traitement Automatique des Contours des L\u00e8vres. [Ph.D. Thesis, INPG]."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"King, H., and Ferragne, E. (2020). Labiodentals \/r\/ here to stay: Deep learning shows us why. Anglophonia Fr. J. Engl. Linguist., 30.","DOI":"10.4000\/anglophonia.3424"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.conb.2019.10.008","article-title":"Deep learning tools for the measurement of animal behavior in neuroscience","volume":"60","author":"Mathis","year":"2020","journal-title":"Curr. Opin. Neurobiol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1281","DOI":"10.1038\/s41593-018-0209-y","article-title":"DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning","volume":"21","author":"Mathis","year":"2018","journal-title":"Nat. Neurosci."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"e47994","DOI":"10.7554\/eLife.47994","article-title":"DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning","volume":"8","author":"Graving","year":"2019","journal-title":"eLife"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Pereira, T.D., Tabris, N., Li, J., Ravindranath, S., Papadoyannis, E.S., Wang, Z.Y., Turner, D.M., McKenzie-Smith, G., Kocher, S.D., and Falkner, A.L. (2020). SLEAP: Multi-animal pose tracking. bioRxiv.","DOI":"10.1101\/2020.08.31.276246"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Mathis, A., Biasi, T., Schneider, S., Yuksekgonul, M., Rogers, B., Bethge, M., and Mathis, M.W. (2021, January 5\u20139). Pretraining boosts out-of-domain robustness for pose estimation. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision, Virtual Conference.","DOI":"10.1109\/WACV48630.2021.00190"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13640-018-0324-4","article-title":"A review of image-based automatic facial landmark identification techniques","volume":"2018","author":"Johnston","year":"2018","journal-title":"EURASIP J. Image Video Process."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Andriluka, M., Pishchulin, L., Gehler, P., and Schiele, B. (2014, January 24\u201327). 2D Human Pose Estimation: New Benchmark and State of the Art Analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.471"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L. (2018, January 18\u201322). MobileNetV2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00474"},{"key":"ref_29","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_30","unstructured":"Tan, M., and Le, Q. (2019, January 9\u201315). Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the International Conference on Machine Learning PMLR, Long Beach, CA, USA."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., and Schiele, B. (2016). DeeperCut: A Deeper, Stronger, and Faster Multi-person Pose Estimation Model. Computer Vision\u2014ECCV 2016, Springer International Publishing.","DOI":"10.1007\/978-3-319-46466-4_3"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.S., Sanger, J., Zhang, J., Eshky, A., Wrench, A., Richmond, K., and Renals, S. (2021, January 19\u201322). TaL: A synchronised multi-speaker corpus of ultrasound tongue imaging, audio, and lip videos. Proceedings of the 2021 IEEE Spoken Language Technology Workshop (SLT), Shenzhen, China.","DOI":"10.1109\/SLT48900.2021.9383619"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Eshky, A., Ribeiro, M.S., Cleland, J., Richmond, K., Roxburgh, Z., Scobbie, J., and Wrench, A. (arXiv, 2019). UltraSuite: A repository of ultrasound and acoustic data from child speech therapy sessions, arXiv.","DOI":"10.21437\/Interspeech.2018-1736"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"592","DOI":"10.1121\/10.0005585","article-title":"Distance vs time. Acoustic and articulatory consequences of reduced vowel duration in Polish","volume":"150","author":"Strycharczuk","year":"2021","journal-title":"J. Acoust. Soc. Am."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1016\/j.specom.2017.08.002","article-title":"Automatic animation of an articulatory tongue model from ultrasound images of the vocal tract","volume":"93","author":"Fabre","year":"2017","journal-title":"Speech Commun."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"2152","DOI":"10.1038\/s41596-019-0176-0","article-title":"Using DeepLabCut for 3D markerless pose estimation across species and behaviors","volume":"14","author":"Nath","year":"2019","journal-title":"Nat. Protoc."},{"key":"ref_37","unstructured":"(2021, July 21). GetContours V3.5. Available online: https:\/\/github.com\/mktiede\/GetContours."},{"key":"ref_38","unstructured":"Scobbie, J.M., Lawson, E., Cowen, S., Cleland, J., and Wrench, A.A. (2021, November 28). A Common Co-Ordinate System for Mid-Sagittal Articulatory Measurement. QMU CASL Working Papers WP-20. Available online: https:\/\/eresearch.qmu.ac.uk\/handle\/20.500.12289\/3597."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-020-58103-6","article-title":"Automatic vocal tract landmark localization from midsagittal MRI data","volume":"10","author":"Eslami","year":"2020","journal-title":"Sci. Rep."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Lim, Y., Toutios, A., Bliesener, Y., Tian, Y., Lingala, S.G., Vaz, C., Sorensen, T., Oh, M., Harper, S., and Chen, W. (arXiv, 2021). A multispeaker dataset of raw and reconstructed speech production real-time MRI video and 3D volumetric images, arXiv.","DOI":"10.1038\/s41597-021-00976-x"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/3\/1133\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:12:52Z","timestamp":1760134372000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/3\/1133"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,2]]},"references-count":40,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,2]]}},"alternative-id":["s22031133"],"URL":"https:\/\/doi.org\/10.3390\/s22031133","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,2]]}}}