{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T13:49:50Z","timestamp":1791985790864,"version":"4.4.0"},"reference-count":37,"publisher":"BMJ","issue":"11","funder":[{"DOI":"10.13039\/100001818","name":"Research to Prevent Blindness, Inc","doi-asserted-by":"crossref","award":["N\/A"],"award-info":[{"award-number":["N\/A"]}],"id":[{"id":"10.13039\/100001818","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["Br J Ophthalmol"],"published-print":{"date-parts":[[2023,11]]},"abstract":"<jats:sec>\n                    <jats:title>Purpose<\/jats:title>\n                    <jats:p>To describe an artificial intelligence platform that detects thyroid eye disease (TED).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Design<\/jats:title>\n                    <jats:p>Development of a deep learning model.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>1944 photographs from a clinical database were used to train a deep learning model. 344 additional images (\u2018test set\u2019) were used to calculate performance metrics. Receiver operating characteristic, precision\u2013recall curves and heatmaps were generated. From the test set, 50 images were randomly selected (\u2018survey set\u2019) and used to compare model performance with ophthalmologist performance. 222 images obtained from a separate clinical database were used to assess model recall and to quantitate model performance with respect to disease stage and grade.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The model achieved test set accuracy of 89.2%, specificity 86.9%, recall 93.4%, precision 79.7% and an F1 score of 86.0%. Heatmaps demonstrated that the model identified pixels corresponding to clinical features of TED. On the survey set, the ensemble model achieved accuracy, specificity, recall, precision and F1 score of 86%, 84%, 89%, 77% and 82%, respectively. 27 ophthalmologists achieved mean performance of 75%, 82%, 63%, 72% and 66%, respectively. On the second test set, the model achieved recall of 91.9%, with higher recall for moderate to severe (98.2%, n=55) and active disease (98.3%, n=60), as compared with mild (86.8%, n=68) or stable disease (85.7%, n=63).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>The deep learning classifier is a novel approach to identify TED and is a first step in the development of tools to improve diagnostic accuracy and lower barriers to specialist evaluation.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1136\/bjo-2022-321833","type":"journal-article","created":{"date-parts":[[2022,9,8]],"date-time":"2022-09-08T13:20:30Z","timestamp":1662643230000},"page":"1722-1729","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":43,"title":["Ensemble neural network model for detecting thyroid eye disease using external photographs"],"prefix":"10.1136","volume":"107","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6111-5368","authenticated-orcid":false,"given":"Justin","family":"Karlin","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lisa","family":"Gai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nathan","family":"LaPierre","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7016-3687","authenticated-orcid":false,"given":"Kayla","family":"Danesh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2919-2002","authenticated-orcid":false,"given":"Justin","family":"Farajzadeh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bea","family":"Palileo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kodi","family":"Taraszka","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jie","family":"Zheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eleazar","family":"Eskin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Rootman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2022,9,8]]},"reference":[{"key":"2023102002150975000_107.11.1722.1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMra0905750"},{"key":"2023102002150975000_107.11.1722.2","doi-asserted-by":"publisher","DOI":"10.1089\/105072502761016476"},{"key":"2023102002150975000_107.11.1722.3","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1159\/000477803","article-title":"Does Graves' orbitopathy ever disappear? answers to an old question","volume":"6","author":"Sabini","year":"2017","journal-title":"Eur Thyroid J"},{"key":"2023102002150975000_107.11.1722.4","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1080\/01676830.2017.1280057","article-title":"Management of thyroid eye disease in the United Kingdom: a multi-centre thyroid eye disease audit","volume":"36","author":"Mellington","year":"2017","journal-title":"Orbit"},{"key":"2023102002150975000_107.11.1722.5","doi-asserted-by":"publisher","DOI":"10.1530\/EJE-09-0383"},{"key":"2023102002150975000_107.11.1722.6","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1089\/thy.2013.0240","article-title":"Spontaneous improvement of untreated mild Graves' ophthalmopathy: Rundle's curve revisited","volume":"24","author":"Menconi","year":"2014","journal-title":"Thyroid"},{"key":"2023102002150975000_107.11.1722.7","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1007\/s40618-017-0608-z","article-title":"Does early response to intravenous glucocorticoids predict the final outcome in patients with moderate-to-severe and active Graves' orbitopathy?","volume":"40","author":"Bartalena","year":"2017","journal-title":"J Endocrinol Invest"},{"key":"2023102002150975000_107.11.1722.8","doi-asserted-by":"crossref","first-page":"1748","DOI":"10.1056\/NEJMoa1614949","article-title":"Teprotumumab for thyroid-associated ophthalmopathy","volume":"376","author":"Smith","year":"2017","journal-title":"N Engl J Med"},{"key":"2023102002150975000_107.11.1722.9","doi-asserted-by":"crossref","unstructured":"Winn BJ , Kersten RC . Teprotumumab: interpreting the clinical trials in the context of thyroid eye disease pathogenesis and current therapies. Ophthalmology.doi:10.1016\/j.ophtha.2021.04.024","DOI":"10.1016\/j.ophtha.2021.04.024"},{"key":"2023102002150975000_107.11.1722.10","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-bioeng-071516-044442"},{"key":"2023102002150975000_107.11.1722.11","doi-asserted-by":"crossref","unstructured":"Spanhol FA , Oliveira LS , Petitjean C . Breast cancer histopathological image classification using convolutional neural networks. In: IJCNN, ed. International Joint Conference on Neural Networks. 2016, 2016.","DOI":"10.1109\/IJCNN.2016.7727519"},{"key":"2023102002150975000_107.11.1722.12","doi-asserted-by":"crossref","first-page":"4373","DOI":"10.3390\/s20164373","article-title":"Breast cancer histopathology image classification using an ensemble of deep learning models","volume":"20","author":"Hameed","year":"2020","journal-title":"Sensors"},{"key":"2023102002150975000_107.11.1722.13","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.2017162326"},{"key":"2023102002150975000_107.11.1722.14","doi-asserted-by":"publisher","DOI":"10.1093\/annonc\/mdy166"},{"key":"2023102002150975000_107.11.1722.15","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejca.2019.04.001"},{"key":"2023102002150975000_107.11.1722.16","doi-asserted-by":"publisher","DOI":"10.1038\/nature21056"},{"key":"2023102002150975000_107.11.1722.17","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-020-0842-3"},{"key":"2023102002150975000_107.11.1722.18","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2018.01.023"},{"key":"2023102002150975000_107.11.1722.19","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2019.07.024"},{"key":"2023102002150975000_107.11.1722.20","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2016.17216"},{"key":"2023102002150975000_107.11.1722.21","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2017.18152"},{"key":"2023102002150975000_107.11.1722.22","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1038\/s41433-018-0269-y","article-title":"Fundus photograph-based deep learning algorithms in detecting diabetic retinopathy","volume":"33","author":"Raman","year":"2019","journal-title":"Eye"},{"key":"2023102002150975000_107.11.1722.23","article-title":"Early detection of diabetic retinopathy based on deep learning and ultra-wide-field fundus images","volume":"11","author":"Oh","year":"2021","journal-title":"Sci Rep"},{"key":"2023102002150975000_107.11.1722.24","doi-asserted-by":"crossref","first-page":"322","DOI":"10.1016\/j.oret.2016.12.009","article-title":"Deep learning is effective for classifying normal versus age-related macular degeneration OCT images","volume":"1","author":"Lee","year":"2017","journal-title":"Ophthalmol Retina"},{"key":"2023102002150975000_107.11.1722.25","doi-asserted-by":"crossref","first-page":"1410","DOI":"10.1016\/j.ophtha.2018.02.037","article-title":"A deep learning algorithm for prediction of age-related eye disease study severity scale for age-related macular degeneration from color fundus photography","volume":"125","author":"Grassmann","year":"2018","journal-title":"Ophthalmology"},{"key":"2023102002150975000_107.11.1722.26","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2018.11.015"},{"key":"2023102002150975000_107.11.1722.27","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1016\/j.ajo.2021.05.007","article-title":"PeriorbitAI: artificial intelligence automation of eyelid and periorbital measurements","volume":"230","author":"Van Brummen","year":"2021","journal-title":"Am J Ophthalmol"},{"key":"2023102002150975000_107.11.1722.28","doi-asserted-by":"publisher","DOI":"10.1530\/EJE-07-0666"},{"key":"2023102002150975000_107.11.1722.29","first-page":"1755","article-title":"Dlib-ml: a machine learning toolkit","volume":"10","author":"King","year":"2009","journal-title":"Journal of Machine Learning Research"},{"key":"2023102002150975000_107.11.1722.30","unstructured":"Viola P , Jones M . Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2001."},{"key":"2023102002150975000_107.11.1722.31","doi-asserted-by":"crossref","unstructured":"Caruana R , Lawrence S , Giles L . Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. In Proceedings of the 13th International Conference on Neural Information Processing Systems. MIT Press, 2000: 381\u20137.","DOI":"10.1109\/IJCNN.2000.857823"},{"key":"2023102002150975000_107.11.1722.32","doi-asserted-by":"crossref","unstructured":"He K , Zhang X , Ren S . Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016: 770\u20138.","DOI":"10.1109\/CVPR.2016.90"},{"key":"2023102002150975000_107.11.1722.33","first-page":"6980","article-title":"Adam: a method for stochastic optimization","volume":"1412","author":"Kingma","year":"2017","journal-title":"arXiv"},{"key":"2023102002150975000_107.11.1722.34","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1007\/s11263-019-01228-7","article-title":"Grad-CAM: visual explanations from deep networks via gradient-based localization","volume":"128","author":"Selvaraju","year":"2020","journal-title":"Int J Comput Vis"},{"key":"2023102002150975000_107.11.1722.35","doi-asserted-by":"crossref","unstructured":"Wu C , Zou Y . Application of transfer learning in the recognition of TAO. In: 13th International Conference on Computer Science & Education (ICCSE). IEEE, 2018.","DOI":"10.1109\/ICCSE.2018.8468803"},{"key":"2023102002150975000_107.11.1722.36","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1002683"},{"key":"2023102002150975000_107.11.1722.37","doi-asserted-by":"crossref","first-page":"514e","DOI":"10.1097\/PRS.0000000000006555","article-title":"A simple standardized three-dimensional anthropometry for the periocular region in a European population","volume":"145","author":"Guo","year":"2020","journal-title":"Plast Reconstr Surg"}],"container-title":["British Journal of Ophthalmology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bjo-2022-321833","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,20]],"date-time":"2023-10-20T05:19:34Z","timestamp":1697779174000},"score":1,"resource":{"primary":{"URL":"https:\/\/bjo.bmj.com\/lookup\/doi\/10.1136\/bjo-2022-321833"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,8]]},"references-count":37,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2023,10,20]]},"published-print":{"date-parts":[[2023,11]]}},"alternative-id":["10.1136\/bjo-2022-321833"],"URL":"https:\/\/doi.org\/10.1136\/bjo-2022-321833","relation":{},"ISSN":["0007-1161","1468-2079"],"issn-type":[{"value":"0007-1161","type":"print"},{"value":"1468-2079","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,8]]}}}