{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,18]],"date-time":"2026-09-18T03:33:12Z","timestamp":1789702392653,"version":"4.0.1"},"reference-count":43,"publisher":"Wiley","issue":"4","license":[{"start":{"date-parts":[[2020,8,7]],"date-time":"2020-08-07T00:00:00Z","timestamp":1596758400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["P30\u2010EY06360"],"award-info":[{"award-number":["P30\u2010EY06360"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100007623","name":"School of Medicine, Emory University","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100007623","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001349","name":"National Medical Research Council","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001349","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Annals of Neurology"],"published-print":{"date-parts":[[2020,10]]},"abstract":"<jats:sec><jats:title>Objective<\/jats:title><jats:p>To compare the diagnostic performance of an artificial intelligence deep learning system with that of expert neuro\u2010ophthalmologists in classifying optic disc appearance.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>The deep learning system was previously trained and validated on 14,341 ocular fundus photographs from 19 international centers. The performance of the system was evaluated on 800 new fundus photographs (400 normal optic discs, 201 papilledema [disc edema from elevated intracranial pressure], 199 other optic disc abnormalities) and compared with that of 2 expert neuro\u2010ophthalmologists who independently reviewed the same randomly presented images without clinical information. Area under the receiver operating characteristic curve, accuracy, sensitivity, and specificity were calculated.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The system correctly classified 678 of 800 (84.7%) photographs, compared with 675 of 800 (84.4%) for Expert 1 and 641 of 800 (80.1%) for Expert 2. The system yielded areas under the receiver operating characteristic curve of 0.97 (95% confidence interval [CI] =\u20090.96\u20130.98), 0.96 (95% CI = 0.94\u20130.97), and 0.89 (95% CI = 0.87\u20130.92) for the detection of normal discs, papilledema, and other disc abnormalities, respectively. The accuracy, sensitivity, and specificity of the system's classification of optic discs were similar to or better than the 2 experts. Intergrader agreement at the eye level was 0.71 (95% CI = 0.67\u20130.76) between Expert 1 and Expert 2, 0.72 (95% CI = 0.68\u20130.76) between the system and Expert 1, and 0.65 (95% CI = 0.61\u20130.70) between the system and Expert 2.<\/jats:p><\/jats:sec><jats:sec><jats:title>Interpretation<\/jats:title><jats:p>The performance of this deep learning system at classifying optic disc abnormalities was at least as good as 2 expert neuro\u2010ophthalmologists. Future prospective studies are needed to validate this system as a diagnostic aid in relevant clinical settings. <jats:bold>ANN NEUROL 2020;88:785\u2013795<\/jats:bold><\/jats:p><\/jats:sec>","DOI":"10.1002\/ana.25839","type":"journal-article","created":{"date-parts":[[2020,7,4]],"date-time":"2020-07-04T06:34:55Z","timestamp":1593844495000},"page":"785-795","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":84,"title":["Optic Disc Classification by Deep Learning versus Expert Neuro\u2010Ophthalmologists"],"prefix":"10.1002","volume":"88","author":[{"given":"Val\u00e9rie","family":"Biousse","sequence":"first","affiliation":[{"name":"Department of Ophthalmology Emory University School of Medicine  Atlanta Georgia USA"},{"name":"Department of Neurology Emory University School of Medicine  Atlanta Georgia USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nancy J.","family":"Newman","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology Emory University School of Medicine  Atlanta Georgia USA"},{"name":"Department of Neurology Emory University School of Medicine  Atlanta Georgia USA"},{"name":"Department of Neurological Surgery Emory University School of Medicine  Atlanta Georgia USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Raymond P.","family":"Najjar","sequence":"additional","affiliation":[{"name":"Singapore Eye Research Institute  Singapore"},{"name":"Duke\u2010NUS Medical School  Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Caroline","family":"Vasseneix","sequence":"additional","affiliation":[{"name":"Singapore Eye Research Institute  Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xinxing","family":"Xu","sequence":"additional","affiliation":[{"name":"Institute of High\u2010Performance Computing, Agency for Science, Technology, and Research  Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel S.","family":"Ting","sequence":"additional","affiliation":[{"name":"Singapore Eye Research Institute  Singapore"},{"name":"Duke\u2010NUS Medical School  Singapore"},{"name":"Singapore National Eye Center  Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"L\u00e9onard B.","family":"Milea","sequence":"additional","affiliation":[{"name":"University of California, Berkeley  Berkeley California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeong\u2010Min","family":"Hwang","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology Seoul National University College of Medicine, Seoul National University Bundang Hospital  Seoul South Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3179-3122","authenticated-orcid":false,"given":"Dong Hyun","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology Seoul National University College of Medicine, Seoul National University Bundang Hospital  Seoul South Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hee Kyung","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology Seoul National University College of Medicine, Seoul National University Bundang Hospital  Seoul South Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Steffen","family":"Hamann","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Rigshospitalet University of Copenhagen  Glostrup Denmark"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John J.","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology Mayo Clinic  Rochester Minnesota USA"},{"name":"Department of Neurology Mayo Clinic  Rochester Minnesota USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yong","family":"Liu","sequence":"additional","affiliation":[{"name":"Institute of High\u2010Performance Computing, Agency for Science, Technology, and Research  Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tien Yin","family":"Wong","sequence":"additional","affiliation":[{"name":"Singapore Eye Research Institute  Singapore"},{"name":"Duke\u2010NUS Medical School  Singapore"},{"name":"Singapore National Eye Center  Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6388-8639","authenticated-orcid":false,"given":"Dan","family":"Milea","sequence":"additional","affiliation":[{"name":"Singapore Eye Research Institute  Singapore"},{"name":"Duke\u2010NUS Medical School  Singapore"},{"name":"Singapore National Eye Center  Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"name":"for the BONSAI (Brain and Optic Nerve Study with Artificial Intelligence) Study Group","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2020,8,7]]},"reference":[{"key":"e_1_2_8_2_1","doi-asserted-by":"publisher","DOI":"10.1212\/WNL.0000000000004868"},{"key":"e_1_2_8_3_1","article-title":"Diagnostic error of neuro\u2010ophthalmologic conditions\u2014state of the science","author":"Stunkel L","journal-title":"J Neuroophthalmol"},{"key":"e_1_2_8_4_1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMc1009733"},{"key":"e_1_2_8_5_1","doi-asserted-by":"publisher","DOI":"10.1212\/CPJ.0000000000000115"},{"key":"e_1_2_8_6_1","doi-asserted-by":"publisher","DOI":"10.3389\/fneur.2015.00233"},{"key":"e_1_2_8_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.annemergmed.2013.01.010"},{"key":"e_1_2_8_8_1","doi-asserted-by":"publisher","DOI":"10.1080\/01658107.2017.1419368"},{"key":"e_1_2_8_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jaapos.2011.07.010"},{"key":"e_1_2_8_10_1","doi-asserted-by":"publisher","DOI":"10.1097\/PEC.0000000000001203"},{"key":"e_1_2_8_11_1","doi-asserted-by":"publisher","DOI":"10.1212\/WNL.0000000000005480"},{"key":"e_1_2_8_12_1","doi-asserted-by":"publisher","DOI":"10.1097\/NRL.0000000000000259"},{"key":"e_1_2_8_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2017.05.026"},{"key":"e_1_2_8_14_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamaophthalmol.2016.0316"},{"key":"e_1_2_8_15_1","doi-asserted-by":"publisher","DOI":"10.1089\/tmj.2014.0115"},{"key":"e_1_2_8_16_1","doi-asserted-by":"publisher","DOI":"10.1089\/tmj.2018.0209"},{"key":"e_1_2_8_17_1","doi-asserted-by":"publisher","DOI":"10.2196\/18810"},{"key":"e_1_2_8_18_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamaneurol.2020.1452"},{"key":"e_1_2_8_19_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-020-0824-5"},{"key":"e_1_2_8_20_1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMoa1917130"},{"key":"e_1_2_8_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1474-4422(16)30237-X"},{"key":"e_1_2_8_22_1","unstructured":"MileaL NajjarRP. Classif\u2010Eye: a semi\u2010automated image classification application.2020. GitHub repository. Available at:https:\/\/github.com\/milealeonard\/Classif-Eye\/. Accessed April 13 2020."},{"key":"e_1_2_8_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02295996"},{"key":"e_1_2_8_24_1","doi-asserted-by":"publisher","DOI":"10.1093\/biostatistics\/kxn025"},{"key":"e_1_2_8_25_1","doi-asserted-by":"publisher","DOI":"10.11613\/BM.2012.031"},{"key":"e_1_2_8_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/0895-4356(91)90128-V"},{"key":"e_1_2_8_27_1","doi-asserted-by":"publisher","DOI":"10.1097\/WNO.0b013e31821a8b0b"},{"key":"e_1_2_8_28_1","doi-asserted-by":"publisher","DOI":"10.1212\/WNL.0000000000004895"},{"key":"e_1_2_8_29_1","first-page":"47","article-title":"Papilledema: epidemiology, etiology, and clinical management","volume":"7","author":"Rigi M","year":"2015","journal-title":"Eye Brain"},{"key":"e_1_2_8_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.emc.2016.06.003"},{"key":"e_1_2_8_31_1","doi-asserted-by":"publisher","DOI":"10.1038\/eye.2017.274"},{"key":"e_1_2_8_32_1","first-page":"1","article-title":"Neuroimaging for the evaluation of chronic headaches: an evidence\u2010based analysis","volume":"10","author":"Medical Advisory Secretariat","year":"2010","journal-title":"Ont Health Technol Assess Ser"},{"key":"e_1_2_8_33_1","unstructured":"American College of Radiology. ACR appropriateness criteria. Revised 2019. Available at:https:\/\/acsearch.acr.org\/docs\/69482\/Narrative\/. Accessed April 13 2020."},{"key":"e_1_2_8_34_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1553-2712.2011.01147.x"},{"key":"e_1_2_8_35_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature14539"},{"key":"e_1_2_8_36_1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMra1814259"},{"key":"e_1_2_8_37_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-018-0300-7"},{"key":"e_1_2_8_38_1","doi-asserted-by":"publisher","DOI":"10.1016\/S2589-7500(19)30123-2"},{"key":"e_1_2_8_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.preteyeres.2019.04.003"},{"key":"e_1_2_8_40_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-018-0048-y"},{"key":"e_1_2_8_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(19)33051-X"},{"key":"e_1_2_8_42_1","doi-asserted-by":"publisher","DOI":"10.1097\/WCO.0000000000000773"},{"key":"e_1_2_8_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2011.09.013"},{"key":"e_1_2_8_44_1","article-title":"Referral patterns in neuro\u2010ophthalmology","author":"Stunkel L","year":"2011","journal-title":"J Neuroophthalmol"}],"container-title":["Annals of Neurology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fana.25839","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/ana.25839","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/ana.25839","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/ana.25839","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,2]],"date-time":"2023-09-02T18:08:00Z","timestamp":1693678080000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/ana.25839"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,7]]},"references-count":43,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,10]]}},"alternative-id":["10.1002\/ana.25839"],"URL":"https:\/\/doi.org\/10.1002\/ana.25839","archive":["Portico"],"relation":{},"ISSN":["0364-5134","1531-8249"],"issn-type":[{"value":"0364-5134","type":"print"},{"value":"1531-8249","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,7]]},"assertion":[{"value":"2020-04-27","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-07-02","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-08-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}