{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,4]],"date-time":"2026-11-04T08:19:15Z","timestamp":1793780355589,"version":"4.6.1"},"reference-count":33,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2022,3,3]],"date-time":"2022-03-03T00:00:00Z","timestamp":1646265600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["2020R1F1A1072484"],"award-info":[{"award-number":["2020R1F1A1072484"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JPM"],"abstract":"<jats:p>Detection of cephalometric landmarks has contributed to the analysis of malocclusion during orthodontic diagnosis. Many recent studies involving deep learning have focused on head-to-head comparisons of accuracy in landmark identification between artificial intelligence (AI) and humans. However, a human\u2013AI collaboration for the identification of cephalometric landmarks has not been evaluated. We selected 1193 cephalograms and used them to train the deep anatomical context feature learning (DACFL) model. The number of target landmarks was 41. To evaluate the effect of human\u2013AI collaboration on landmark detection, 10 images were extracted randomly from 100 test images. The experiment included 20 dental students as beginners in landmark localization. The outcomes were determined by measuring the mean radial error (MRE), successful detection rate (SDR), and successful classification rate (SCR). On the dataset, the DACFL model exhibited an average MRE of 1.87 \u00b1 2.04 mm and an average SDR of 73.17% within a 2 mm threshold. Compared with the beginner group, beginner\u2013AI collaboration improved the SDR by 5.33% within a 2 mm threshold and also improved the SCR by 8.38%. Thus, the beginner\u2013AI collaboration was effective in the detection of cephalometric landmarks. Further studies should be performed to demonstrate the benefits of an orthodontist\u2013AI collaboration.<\/jats:p>","DOI":"10.3390\/jpm12030387","type":"journal-article","created":{"date-parts":[[2022,3,2]],"date-time":"2022-03-02T22:53:25Z","timestamp":1646261605000},"page":"387","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":51,"title":["Effectiveness of Human\u2013Artificial Intelligence Collaboration in Cephalometric Landmark Detection"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7232-9196","authenticated-orcid":false,"given":"Van Nhat Thang","family":"Le","sequence":"first","affiliation":[{"name":"Department of Pediatric Dentistry and Institute of Oral Bioscience, School of Dentistry, Jeonbuk National University, Jeonju 54896, Korea"},{"name":"Research Institute of Clinical Medicine, Jeonbuk National University, Jeonju 54907, Korea"},{"name":"Biomedical Research Institute, Jeonbuk National University Hospital, Jeonju 54907, Korea"},{"name":"Faculty of Odonto-Stomatology, Hue University of Medicine and Pharmacy, Hue University, Hue 49120, Vietnam"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Junhyeok","family":"Kang","sequence":"additional","affiliation":[{"name":"Division of Computer Science and Engineering, Jeonbuk National University, Jeonju 54907, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Il-Seok","family":"Oh","sequence":"additional","affiliation":[{"name":"Division of Computer Science and Engineering, Jeonbuk National University, Jeonju 54907, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jae-Gon","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Pediatric Dentistry and Institute of Oral Bioscience, School of Dentistry, Jeonbuk National University, Jeonju 54896, Korea"},{"name":"Research Institute of Clinical Medicine, Jeonbuk National University, Jeonju 54907, Korea"},{"name":"Biomedical Research Institute, Jeonbuk National University Hospital, Jeonju 54907, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yeon-Mi","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Pediatric Dentistry and Institute of Oral Bioscience, School of Dentistry, Jeonbuk National University, Jeonju 54896, Korea"},{"name":"Research Institute of Clinical Medicine, Jeonbuk National University, Jeonju 54907, Korea"},{"name":"Biomedical Research Institute, Jeonbuk National University Hospital, Jeonju 54907, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9942-2400","authenticated-orcid":false,"given":"Dae-Woo","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Pediatric Dentistry and Institute of Oral Bioscience, School of Dentistry, Jeonbuk National University, Jeonju 54896, Korea"},{"name":"Research Institute of Clinical Medicine, Jeonbuk National University, Jeonju 54907, Korea"},{"name":"Biomedical Research Institute, Jeonbuk National University Hospital, Jeonju 54907, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,3]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"145","DOI":"10.2319\/120506-491.1","article-title":"Automatic cephalometric analysis","volume":"78","author":"Leonardi","year":"2008","journal-title":"Angle Orthod."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"806","DOI":"10.1109\/JBHI.2020.3002582","article-title":"Deep Anatomical Context Feature Learning for Cephalometric Landmark Detection","volume":"25","author":"Oh","year":"2021","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"101904","DOI":"10.1016\/j.media.2020.101904","article-title":"Cascaded convolutional networks for automatic cephalometric landmark detection","volume":"68","author":"Zeng","year":"2021","journal-title":"Med. Image Anal."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Song, Y., Qiao, X., Iwamoto, Y., and Chen, Y.-W. (2020). Automatic Cephalometric Landmark Detection on X-ray Images Using a Deep-Learning Method. Appl. Sci., 10.","DOI":"10.3390\/app10072547"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"112633","DOI":"10.1109\/ACCESS.2020.3002939","article-title":"CephaNN: A Multi-Head Attention Network for Cephalometric Landmark Detection","volume":"8","author":"Qian","year":"2020","journal-title":"IEEE Access"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"014501","DOI":"10.1117\/1.JMI.4.1.014501","article-title":"Fully automated quantitative cephalometry using convolutional neural networks","volume":"4","author":"Arik","year":"2017","journal-title":"J. Med. Imaging"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"4753","DOI":"10.1109\/TIP.2017.2721106","article-title":"Detecting Anatomical Landmarks From Limited Medical Imaging Data Using Two-Stage Task-Oriented Deep Neural Networks","volume":"26","author":"Zhang","year":"2017","journal-title":"IEEE Trans. Image Process."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1016\/j.media.2017.09.003","article-title":"Integrating geometric configuration and appearance information into a unified framework for anatomical landmark localization","volume":"43","author":"Urschler","year":"2018","journal-title":"Med. Image Anal."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Lee, J.-H., Yu, H.-J., Kim, M.-j., Kim, J.-W., and Choi, J. (2020). Automated cephalometric landmark detection with confidence regions using Bayesian convolutional neural networks. BMC Oral Health, 20.","DOI":"10.1186\/s12903-020-01256-7"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"903","DOI":"10.2319\/022019-127.1","article-title":"Automated identification of cephalometric landmarks: Part 1\u2014Comparisons between the latest deep-learning methods YOLOV3 and SSD","volume":"89","author":"Park","year":"2019","journal-title":"Angle Orthod."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"69","DOI":"10.2319\/022019-129.1","article-title":"Automated Identification of Cephalometric Landmarks: Part 2-Might It Be Better Than human?","volume":"90","author":"Hwang","year":"2019","journal-title":"Angle Orthod."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"329","DOI":"10.2319\/021220-100.1","article-title":"Evaluation of automated cephalometric analysis based on the latest deep learning method","volume":"91","author":"Hwang","year":"2021","journal-title":"Angle Orthod."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1007\/s00056-019-00203-8","article-title":"Artificial intelligence in orthodontics: Evaluation of a fully automated cephalometric analysis using a customized convolutional neural network","volume":"81","author":"Kunz","year":"2020","journal-title":"J. Orofac. Orthop."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1097\/SCS.0000000000004901","article-title":"Personal Computer-Based Cephalometric Landmark Detection With Deep Learning, Using Cephalograms on the Internet","volume":"30","author":"Nishimoto","year":"2019","journal-title":"J. Craniofacial Surg."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1016\/j.media.2019.03.007","article-title":"Integrating spatial configuration into heatmap regression based CNNs for landmark localization","volume":"54","author":"Payer","year":"2019","journal-title":"Med. Image Anal."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"132738","DOI":"10.1109\/ACCESS.2019.2940623","article-title":"Locating Anatomical Landmarks on 2D Lateral Cephalograms Through Adversarial Encoder-Decoder Networks","volume":"7","author":"Dai","year":"2019","journal-title":"IEEE Access"},{"key":"ref_17","unstructured":"Oktay, O., Schlemper, J., Folgoc, L.L., Lee, M.J., Heinrich, M.P., Misawa, K., Mori, K., McDonagh, S.G., Hammerla, N.Y., and Kainz, B. (2018). Attention U-Net: Learning Where to Look for the Pancreas. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5\u20139). U-Net: Convolutional Networks for Biomedical Image Segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1890","DOI":"10.1109\/TMI.2015.2412951","article-title":"Evaluation and Comparison of Anatomical Landmark Detection Methods for Cephalometric X-Ray Images: A Grand Challenge","volume":"34","author":"Wang","year":"2015","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1016\/j.media.2016.02.004","article-title":"A benchmark for comparison of dental radiography analysis algorithms","volume":"31","author":"Wang","year":"2016","journal-title":"Med. Image Anal."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1093\/ejo\/8.3.141","article-title":"The effect of projection errors on cephalometric length measurements","volume":"8","author":"Ahlqvist","year":"1986","journal-title":"Eur. J. Orthod."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1093\/ejo\/8.3.149","article-title":"Sources of error in measurements from cephalometric radiographs","volume":"8","author":"Houston","year":"1986","journal-title":"Eur. J. Orthod."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"823","DOI":"10.2319\/021920-116.1","article-title":"How much deep learning is enough for automatic identification to be reliable?","volume":"90","author":"Moon","year":"2020","journal-title":"Angle Orthod."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"358","DOI":"10.1016\/0010-4809(90)90027-A","article-title":"Landmarking of cephalograms using a microcomputer system","volume":"23","author":"Tong","year":"1990","journal-title":"Comput. Biomed. Res. Int. J."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1016\/0002-9416(71)90028-5","article-title":"The reliability of head film measurements. 1. Landmark identification","volume":"60","author":"Baumrind","year":"1971","journal-title":"Am. J. Orthod."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1093\/ejo\/4.3.185","article-title":"Precision in cephalometric landmark identification","volume":"4","author":"Stabrun","year":"1982","journal-title":"Eur. J. Orthod."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1093\/ejo\/16.2.110","article-title":"Validity of cephalometric landmarks. An experimental study on human skulls","volume":"16","author":"Tng","year":"1994","journal-title":"Eur. J. Orthod."},{"key":"ref_28","first-page":"98","article-title":"Cephalometric landmark identification error","volume":"10","author":"Vincent","year":"1987","journal-title":"Aust. Orthod. J."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1797502","DOI":"10.1155\/2018\/1797502","article-title":"Automatic Analysis of Lateral Cephalograms Based on Multiresolution Decision Tree Regression Voting","volume":"2018","author":"Wang","year":"2018","journal-title":"J. Healthc. Eng."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1179\/bjo.1.3.95","article-title":"The clinical significance of tracing error in cephalometry","volume":"1","author":"Gravely","year":"1974","journal-title":"Br. J. Orthod."},{"key":"ref_31","first-page":"204","article-title":"Effect of training and experience on cephalometric measurement errors on surgical patients","volume":"12","author":"Lau","year":"1997","journal-title":"Int. J. Adult Orthod. Orthognath. Surg."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"359","DOI":"10.3109\/00016356909040415","article-title":"Variability in tracings of lateral head plates for diagnostic orthodontic purposes. Amethodologic study","volume":"27","author":"Kvam","year":"1969","journal-title":"Acta Odontol. Scand."},{"key":"ref_33","first-page":"139","article-title":"Frontal cephalometric landmarking: Humans vs artificial neural networks","volume":"23","author":"Muraev","year":"2020","journal-title":"Int. J. Comput. Dent."}],"container-title":["Journal of Personalized Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4426\/12\/3\/387\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:31:07Z","timestamp":1760135467000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4426\/12\/3\/387"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,3]]},"references-count":33,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["jpm12030387"],"URL":"https:\/\/doi.org\/10.3390\/jpm12030387","relation":{},"ISSN":["2075-4426"],"issn-type":[{"value":"2075-4426","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,3]]}}}