{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,25]],"date-time":"2026-06-25T10:47:40Z","timestamp":1782384460298,"version":"3.54.5"},"reference-count":47,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2021,1,12]],"date-time":"2021-01-12T00:00:00Z","timestamp":1610409600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This study was designed to develop and verify a fully automated cephalometry landmark identification system, based on multi-stage convolutional neural networks (CNNs) architecture, using a combination dataset. In this research, we trained and tested multi-stage CNNs with 430 lateral and 430 MIP lateral cephalograms synthesized by cone-beam computed tomography (CBCT) to make a combination dataset. Fifteen landmarks were manually and respectively identified by experienced examiner, at the preprocessing phase. The intra-examiner reliability was high (ICC = 0.99) in manual identification. The results of prediction of the system for average mean radial error (MRE) and standard deviation (SD) were 1.03 mm and 1.29 mm, respectively. In conclusion, different types of image data might be the one of factors that affect the prediction accuracy of a fully-automated landmark identification system, based on multi-stage CNNs.<\/jats:p>","DOI":"10.3390\/s21020505","type":"journal-article","created":{"date-parts":[[2021,1,12]],"date-time":"2021-01-12T20:11:31Z","timestamp":1610482291000},"page":"505","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":39,"title":["Automatic Cephalometric Landmark Identification System Based on the Multi-Stage Convolutional Neural Networks with CBCT Combination Images"],"prefix":"10.3390","volume":"21","author":[{"given":"Min-Jung","family":"Kim","sequence":"first","affiliation":[{"name":"Department of Orthodontics, Graduate School, Kyung Hee University, Seoul 02447, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yi","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Orthodontics, Peking University School of Stomatology, Beijing 100081, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5508-289X","authenticated-orcid":false,"given":"Song Hee","family":"Oh","sequence":"additional","affiliation":[{"name":"Department of Oral and Maxillofacial Radiology, Graduate School, Kyung Hee University, Seoul 02447, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4872-7095","authenticated-orcid":false,"given":"Hyo-Won","family":"Ahn","sequence":"additional","affiliation":[{"name":"Department of Orthodontics, Graduate School, Kyung Hee University, Seoul 02447, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Seong-Hun","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Orthodontics, Graduate School, Kyung Hee University, Seoul 02447, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1392-7505","authenticated-orcid":false,"given":"Gerald","family":"Nelson","sequence":"additional","affiliation":[{"name":"Division of Orthodontics, Department of Orofacial Science, University of California San Francisco, San Francisco, CA 94143, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,1,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"795","DOI":"10.1016\/j.jcms.2014.06.017","article-title":"Experimental and clinical assessment of three-dimensional cephalometry: A systematic review","volume":"42","author":"Smektala","year":"2014","journal-title":"J. Craniomaxillofac. Surg."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"26","DOI":"10.2319\/032210-166.1","article-title":"Accuracy and reliability of craniometric measurements on lateral cephalometry and 3D measurements on CBCT scans","volume":"81","author":"Gribel","year":"2011","journal-title":"Angle Orthod."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"293","DOI":"10.4041\/kjod.2020.50.5.293","article-title":"Three-dimensional surgical accuracy between virtually planned and actual surgical movements of the maxilla in two-jaw orthognathic surgery","volume":"50","author":"Hong","year":"2020","journal-title":"Korean J. Orthod."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"20140282","DOI":"10.1259\/dmfr.20140282","article-title":"CBCT in orthodontics: Assessment of treatment outcomes and indications for its use","volume":"44","author":"Kapila","year":"2015","journal-title":"Dentomaxillofac. Radiol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"41","DOI":"10.4041\/kjod.2019.49.1.41","article-title":"A comparative study of the reproducibility of landmark identification on posteroanterior and anteroposterior cephalograms generated from cone-beam computed tomography scans","volume":"49","author":"Na","year":"2019","journal-title":"Korean J. Orthod."},{"key":"ref_6","unstructured":"Clendenen, L.H. (2010). Intra- and Inter-Operator Reliability of Cephalometric Landmark Identification: A Comparison Amongst CBCT Ray-Sum, Maximum Intensity Projection Techniques, and Conventional Digital Cephalometric Radiographs, University of Missouri\u2014Kansas City."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"570","DOI":"10.3348\/kjr.2017.18.4.570","article-title":"Kim, N. Deep Learning in Medical Imaging: General Overview","volume":"18","author":"Lee","year":"2017","journal-title":"Korean J. Radiol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"594","DOI":"10.1109\/TBME.2012.2207898","article-title":"GPU accelerated generation of digitally reconstructed radiographs for 2-D\/3-D image registration","volume":"59","author":"Dorgham","year":"2012","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1016\/j.ajodo.2015.10.021","article-title":"Common 3-dimensional coordinate system for assessment of directional changes","volume":"149","author":"Ruellas","year":"2016","journal-title":"Am. J. Orthod. Dentofacial. Orthop."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"282","DOI":"10.4041\/kjod.2015.45.6.282","article-title":"Comparison of cone-beam computed tomography cephalometric measurements using a midsagittal projection and conventional two-dimensional cephalometric measurements","volume":"45","author":"Jung","year":"2015","journal-title":"Korean J. Orthod."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"3","DOI":"10.4041\/kjod.2020.50.1.3","article-title":"Comparison of three midsagittal planes for three-dimensional cone beam computed tomography head reorientation","volume":"50","author":"Lee","year":"2020","journal-title":"Korean J. Orthod."},{"key":"ref_12","first-page":"269","article-title":"The use of cone-beam computed tomography in an orthodontic department in between research and daily clinic","volume":"9","author":"Cattaneo","year":"2008","journal-title":"World J. Orthod."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"282","DOI":"10.1016\/0010-4809(86)90023-6","article-title":"Knowledge based landmarking of cephalograms","volume":"19","author":"Venetsamopolus","year":"1986","journal-title":"Comput. Biomed. Res."},{"key":"ref_14","unstructured":"Ibragimov, B., Likar, B., Pernu\u0161, F., and Toma\u017e, V. (2014). Automatic Cephalometric X-Ray Landmark Detection by Applying Game Theory and Random Forests. Proc. ISBI Int. Symp. Biomedical. Imaging."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"145","DOI":"10.2319\/120506-491.1","article-title":"Automatic Cephalometric Analysis: A Systematic Review","volume":"78","author":"Leonardi","year":"2008","journal-title":"Angle Orthod."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"33581","DOI":"10.1038\/srep33581","article-title":"Fully Automatic System for Accurate Localisation and Analysis of Cephalometric Landmarks in Lateral Cephalograms","volume":"6","author":"Lindner","year":"2016","journal-title":"Sci. Rep."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1797502","DOI":"10.1155\/2018\/1797502","article-title":"Automatic Analysis of Lateral Cephalograms Based on Multiresolution Decision Tree Regression Voting","volume":"2018","author":"Wang","year":"2018","journal-title":"J. Healthc. Eng."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1016\/j.neunet.2014.09.003","article-title":"Deep learning in neural networks: An overview","volume":"61","author":"Schmidhuber","year":"2015","journal-title":"Neural. Netw."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"S36","DOI":"10.1016\/j.metabol.2017.01.011","article-title":"Artificial intelligence in medicine","volume":"69S","author":"Hamet","year":"2017","journal-title":"Metabolism"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"5859727","DOI":"10.1155\/2017\/5859727","article-title":"Deep Learning- and Transfer Learning-Based Super Resolution Reconstruction from Single Medical Image","volume":"2017","author":"Zhang","year":"2017","journal-title":"J. Healthc. Eng."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"014501","DOI":"10.1117\/1.JMI.4.1.014501","article-title":"Fully automated quantitative cephalometry using convolutional neural networks","volume":"4","author":"Ibragimov","year":"2017","journal-title":"J. Med Imaging"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1097\/SCS.0000000000004901","article-title":"Personal Computer-Based Cephalometric Landmark Detection With Deep Learning, Using Cephalograms on the Internet","volume":"30","author":"Nishimoto","year":"2019","journal-title":"J. Craniofac. Surg."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"903","DOI":"10.2319\/022019-127.1","article-title":"Automated identification of cephalometric landmarks: Part 1\u2014Comparisons between the latest deep-learning methods YOLOV3 and SSD","volume":"89","author":"Park","year":"2019","journal-title":"Angle Orthod."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Kim, M.J., Liu, Y., Oh, S.H., Ahn, H.W., Kim, S.H., and Nelson, G. (Korean J. Orthod., 2020). Evaluation of multi-stage convolutional neural network based fully automatic landmark identification system using CBCT synthesized posteroanterior cephalometric images, Korean J. Orthod., in press.","DOI":"10.4041\/kjod.2021.51.2.77"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"157","DOI":"10.4041\/kjod.2020.50.3.157","article-title":"Assessment of bone density changes following two-jaw surgery using multidetector computed tomography: A pilot study","volume":"50","author":"Lee","year":"2020","journal-title":"Korean J. Orthod."},{"key":"ref_27","first-page":"06348","article-title":"How Much Data Is Needed to Train A Medical Image Deep Learning System to Achieve Necessary High Accuracy","volume":"1511","author":"Cho","year":"2015","journal-title":"arXiv"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1177\/0272989X0102100106","article-title":"Prognostic Modeling with Logistic Regression Analysis: In Search of a Sensible Strategy in Small Data Sets","volume":"21","author":"Steyerberg","year":"2001","journal-title":"Med Decis. Mak."},{"key":"ref_29","first-page":"359","article-title":"The biocreative strategy. Part 2: The Tweemac analysis","volume":"52","author":"Kim","year":"2018","journal-title":"J. Clin. Orthod."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"69","DOI":"10.2319\/022019-129.1","article-title":"Automated Identification of Cephalometric Landmarks: Part 2-Might It Be Better Than human?","volume":"90","author":"Hwang","year":"2019","journal-title":"Angle Orthod."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1299","DOI":"10.1109\/TMI.2016.2535302","article-title":"Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?","volume":"35","author":"Tajbakhsh","year":"2016","journal-title":"IEEE Trans. Med. Imaging."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"226","DOI":"10.1007\/s10916-018-1088-1","article-title":"Medical Image Analysis using Convolutional Neural Networks: A Review","volume":"42","author":"Anwar","year":"2018","journal-title":"J. Med. Syst."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"103226","DOI":"10.1016\/j.jdent.2019.103226","article-title":"Convolutional neural networks for dental image diagnostics: A scoping review","volume":"91","author":"Schwendicke","year":"2019","journal-title":"J. Dent."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1186\/s40537-019-0276-2","article-title":"Deep convolutional neural network based medical image classification for disease diagnosis","volume":"6","author":"Yadav","year":"2019","journal-title":"J. Big. Data"},{"key":"ref_35","unstructured":"Takahashiy, R., Takashi, M., and Kuniaki (2017). Multi-Stage Convolutional Neural Networks for Robustness to Scale Transformation. Int. Symp. Nonlinear Theory Appl."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1090","DOI":"10.1109\/TCSVT.2018.2822773","article-title":"A Novel Weight-Shared Multi-Stage CNN for Scale Robustness","volume":"29","author":"Takahashi","year":"2019","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1016\/j.neucom.2016.10.049","article-title":"Learning Pooling for Convolutional Neural Network","volume":"224","author":"Sun","year":"2017","journal-title":"Neurocomputing"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1109\/42.41482","article-title":"Three-dimensional display in nuclear medicine","volume":"8","author":"Wallis","year":"1989","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_39","first-page":"534","article-title":"Three-dimensional display in nuclear medicine and radiology","volume":"32","author":"Wallis","year":"1991","journal-title":"J. Nucl. Med."},{"key":"ref_40","first-page":"31","article-title":"3D cephalometric analysis obtained from computed tomography. Review of the literature","volume":"2","author":"Rossini","year":"2011","journal-title":"Ann. Stomatol."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1590\/1678-7757201302326","article-title":"Comparison of manual, digital and lateral CBCT cephalometric analyses","volume":"21","author":"Navarro","year":"2013","journal-title":"J. Appl. Oral Sci."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"3","DOI":"10.2319\/032910-173.1","article-title":"From 2D to 3D: An algorithm to derive normal values for 3-dimensional computerized assessment","volume":"81","author":"Gribel","year":"2011","journal-title":"Angle Orthod."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"201","DOI":"10.5624\/isd.2012.42.4.201","article-title":"Comparison of conventional lateral cephalograms with corresponding CBCT radiographs","volume":"42","author":"Park","year":"2012","journal-title":"Imaging Sci. Dent."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1007\/s00056-019-00203-8","article-title":"Artificial intelligence in orthodontics","volume":"81","author":"Kunz","year":"2020","journal-title":"J. Orofac. Orthop."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"105513","DOI":"10.1016\/j.cmpb.2020.105513","article-title":"Web-based fully automated cephalometric analysis by deep learning","volume":"194","author":"Kim","year":"2020","journal-title":"Comput. Methods Programs Biomed."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Lee, J.H., Yu, H.J., Kim, M.J., Kim, J.W., and Choi, J.E. (2020). Automated cephalometric landmark detection with confidence regions using Bayesian convolutional neural networks. BMC Oral Health, 20.","DOI":"10.1186\/s12903-020-01256-7"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"823","DOI":"10.2319\/021920-116.1","article-title":"How much deep learning is enough for automatic identification to be reliable?: A cephalometric example","volume":"90","author":"Moon","year":"2020","journal-title":"Angle Orthod."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/2\/505\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:10:16Z","timestamp":1760159416000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/2\/505"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,1,12]]},"references-count":47,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2021,1]]}},"alternative-id":["s21020505"],"URL":"https:\/\/doi.org\/10.3390\/s21020505","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,1,12]]}}}