{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,17]],"date-time":"2026-10-17T20:54:04Z","timestamp":1792270444021,"version":"4.4.0"},"reference-count":35,"publisher":"Public Library of Science (PLoS)","issue":"5","license":[{"start":{"date-parts":[[2021,5,14]],"date-time":"2021-05-14T00:00:00Z","timestamp":1620950400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001691","name":"KAKENHI","doi-asserted-by":"crossref","award":["18H02957"],"award-info":[{"award-number":["18H02957"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:sec id=\"sec001\"><jats:title>Purpose<\/jats:title><jats:p>Abnormalities of the running pattern of choroidal vessel have been reported in eyes with pachychoroid diseases. However, it is difficult for clinicians to judge the running pattern with high reproducibility. Thus, the purpose of this study was to compare the degree of concordance of the running pattern of the choroidal vessels between that determined by artificial intelligence (AI) to that determined by experienced clinicians.<\/jats:p><\/jats:sec><jats:sec id=\"sec002\"><jats:title>Methods<\/jats:title><jats:p>The running pattern of the choroidal vessels in en face images of Haller\u2019s layer of 413 normal and pachychoroid diseased eyes was classified as symmetrical or asymmetrical by human raters and by three supervised machine learning models; the support vector machine (SVM), Xception, and random forest models. The data from the human raters were used as the supervised data. The accuracy rates of the human raters and the certainty of AI\u2019s answers were compared using confidence scores (CSs).<\/jats:p><\/jats:sec><jats:sec id=\"sec003\"><jats:title>Results<\/jats:title><jats:p>The choroidal vascular running pattern could be determined by each AI model with an area under the curve better than 0.94. The random forest method was able to discriminate with the highest accuracy among the three AIs. In the CS analyses, the percentage of certainty was highest (66.4%) and that of uncertainty was lowest (6.1%) in the agreement group. On the other hand, the rate of uncertainty was highest (27.3%) in the disagreement group.<\/jats:p><\/jats:sec><jats:sec id=\"sec004\"><jats:title>Conclusion<\/jats:title><jats:p>AI algorithm can automatically classify with ambiguous criteria the presence or absence of a symmetrical blood vessel running pattern of the choroid. The classification was as good as that of supervised humans in accuracy and reproducibility.<\/jats:p><\/jats:sec>","DOI":"10.1371\/journal.pone.0251553","type":"journal-article","created":{"date-parts":[[2021,5,14]],"date-time":"2021-05-14T17:33:43Z","timestamp":1621013623000},"page":"e0251553","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":7,"title":["Artificial intelligence for classifying uncertain images by humans in determining choroidal vascular running pattern and comparisons with automated classification between artificial intelligence"],"prefix":"10.1371","volume":"16","author":[{"given":"Shozo","family":"Sonoda","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hideki","family":"Shiihara","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hiroto","family":"Terasaki","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Naoko","family":"Kakiuchi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ryoh","family":"Funatsu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Masatoshi","family":"Tomita","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuki","family":"Shinohara","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eisuke","family":"Uchino","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Takuma","family":"Udagawa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guangzhou","family":"An","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Masahiro","family":"Akiba","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hideo","family":"Yokota","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0287-3801","authenticated-orcid":true,"given":"Taiji","family":"Sakamoto","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2021,5,14]]},"reference":[{"issue":"22","key":"pone.0251553.ref001","doi-asserted-by":"crossref","first-page":"2199","DOI":"10.1001\/jama.2017.14585","article-title":"Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer","volume":"318","author":"B Ehteshami Bejnordi","year":"2017","journal-title":"JAMA"},{"issue":"9","key":"pone.0251553.ref002","doi-asserted-by":"crossref","first-page":"1342","DOI":"10.1038\/s41591-018-0107-6","article-title":"Clinically applicable deep learning for diagnosis and referral in retinal disease","volume":"24","author":"J De Fauw","year":"2018","journal-title":"Nat Med"},{"key":"pone.0251553.ref003","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.preteyeres.2018.07.004","article-title":"Artificial intelligence in retina","volume":"67","author":"U Schmidt-Erfurth","year":"2018","journal-title":"Prog Retin Eye Res"},{"issue":"1","key":"pone.0251553.ref004","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1111\/aos.13613","article-title":"Validation of automated screening for referable diabetic retinopathy with the IDx-DR device in the Hoorn Diabetes Care System","volume":"96","author":"AA van der Heijden","year":"2018","journal-title":"Acta Ophthalmol"},{"issue":"2","key":"pone.0251553.ref005","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1136\/bjophthalmol-2018-313173","article-title":"Artificial intelligence and deep learning in ophthalmology","volume":"103","author":"DSW Ting","year":"2019","journal-title":"Br J Ophthalmol"},{"key":"pone.0251553.ref006","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/978-3-030-33128-3_1","article-title":"Deep Learning in Medical Image Analysis","volume":"1213","author":"HP Chan","year":"2020","journal-title":"Adv Exp Med Biol"},{"issue":"22","key":"pone.0251553.ref007","doi-asserted-by":"crossref","first-page":"2402","DOI":"10.1001\/jama.2016.17216","article-title":"Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs","volume":"316","author":"V Gulshan","year":"2016","journal-title":"JAMA"},{"key":"pone.0251553.ref008","doi-asserted-by":"crossref","first-page":"7820971","DOI":"10.1155\/2019\/7820971","article-title":"Severity Classification of Conjunctival Hyperaemia by Deep Neural Network Ensembles","volume":"2019","author":"H Masumoto","year":"2019","journal-title":"J Ophthalmol"},{"issue":"1","key":"pone.0251553.ref009","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1038\/s41433-018-0158-4","article-title":"Pachychoroid disease","volume":"33","author":"CMG Cheung","year":"2019","journal-title":"Eye (Lond)"},{"issue":"4","key":"pone.0251553.ref010","first-page":"111","article-title":"Pachychoroid diseases of the macula","volume":"3","author":"R Gallego-Pinazo","year":"2014","journal-title":"Med Hypothesis Discov Innov Ophthalmol"},{"key":"pone.0251553.ref011","doi-asserted-by":"crossref","first-page":"16204","DOI":"10.1038\/srep16204","article-title":"Pachychoroid neovasculopathy and age-related macular degeneration","volume":"5","author":"M Miyake","year":"2015","journal-title":"Sci Rep"},{"issue":"Pt 2","key":"pone.0251553.ref012","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1038\/eye.1990.39","article-title":"In vivo choroidal circulation and its watershed zones","volume":"4","author":"SS Hayreh","year":"1990","journal-title":"Eye (Lond)"},{"issue":"5","key":"pone.0251553.ref013","doi-asserted-by":"crossref","first-page":"681","DOI":"10.1001\/archopht.1987.01060050099046","article-title":"Angioarchitecture of the human choroid","volume":"105","author":"S Yoneya","year":"1987","journal-title":"Archives of Ophthalmology"},{"issue":"3","key":"pone.0251553.ref014","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1016\/j.ophtha.2003.06.009","article-title":"Asymmetry of choroidal venous vascular patterns in the human eye","volume":"111","author":"K Mori","year":"2004","journal-title":"Ophthalmology"},{"issue":"9","key":"pone.0251553.ref015","doi-asserted-by":"crossref","first-page":"1879","DOI":"10.1007\/s00417-019-04399-8","article-title":"Running pattern of choroidal vessel in en face OCT images determined by machine learning-based quantitative method","volume":"257","author":"H Shiihara","year":"2019","journal-title":"Graefes Arch Clin Exp Ophthalmol"},{"issue":"2","key":"pone.0251553.ref016","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1016\/j.oret.2017.05.013","article-title":"Dilatation of Asymmetric Vortex Vein in Central Serous Chorioretinopathy","volume":"2","author":"T Hiroe","year":"2017","journal-title":"Ophthalmology Retin"},{"issue":"6","key":"pone.0251553.ref017","doi-asserted-by":"crossref","first-page":"1211","DOI":"10.1097\/IAE.0000000000001678","article-title":"Classification of haller vessel arrangements in acute and chronic central serous chorioretinopathy imaged with en face optical coherence tomography","volume":"38","author":"MC Savastano","year":"2017","journal-title":"Retina"},{"issue":"6","key":"pone.0251553.ref018","doi-asserted-by":"crossref","first-page":"643","DOI":"10.1007\/s10384-018-0625-2","article-title":"Automated segmentation of en face choroidal images obtained by optical coherent tomography by machine learning","volume":"62","author":"H Shiihara","year":"2018","journal-title":"Jpn J Ophthalmol"},{"key":"pone.0251553.ref019","doi-asserted-by":"crossref","unstructured":"Boser BE, Guyon IM, Vapnik VN, editors. A training algorithm for optimal margin classifiers. Proceedings of the fifth annual workshop on Computational learning theory; 1992. 144\u2013152","DOI":"10.1145\/130385.130401"},{"key":"pone.0251553.ref020","doi-asserted-by":"crossref","unstructured":"Byun H, Lee S-W, editors. Applications of support vector machines for pattern recognition: A survey. International Workshop on Support Vector Machines; 2002: Springer. 213\u2013236","DOI":"10.1007\/3-540-45665-1_17"},{"issue":"2","key":"pone.0251553.ref021","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1016\/j.eswa.2005.09.024","article-title":"A GA-based feature selection and parameters optimizationfor support vector machines","volume":"31","author":"C-L Huang","year":"2006","journal-title":"Expert Systems with applications"},{"key":"pone.0251553.ref022","doi-asserted-by":"crossref","unstructured":"Chollet F, editor Xception: Deep learning with depthwise separable convolutions. Proceedings of the IEEE conference on computer vision and pattern recognition; 2017. 1251\u20131258","DOI":"10.1109\/CVPR.2017.195"},{"issue":"1","key":"pone.0251553.ref023","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"L Breiman","year":"2001","journal-title":"Machine learning"},{"key":"pone.0251553.ref024","doi-asserted-by":"crossref","unstructured":"Thomson B, Yu K, Ga\u0161i\u0107 M, Keizer S, Mairesse F, Schatzmann J, et al., editors. Evaluating semantic-level confidence scores with multiple hypotheses. Ninth Annual Conference of the International Speech Communication Association; 2008.","DOI":"10.21437\/Interspeech.2008-350"},{"key":"pone.0251553.ref025","doi-asserted-by":"crossref","unstructured":"Blatz J, Fitzgerald E, Foster G, Gandrabur S, Goutte C, Kulesza A, et al., editors. Confidence estimation for machine translation. Coling 2004: Proceedings of the 20th international conference on computational linguistics; 2004; 315\u2013321.","DOI":"10.3115\/1220355.1220401"},{"issue":"2","key":"pone.0251553.ref026","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1007\/BF00127636","article-title":"Physiological anatomy of the choroidal vascular bed","volume":"6","author":"SS Hayreh","year":"1983","journal-title":"Int Ophthalmol"},{"key":"pone.0251553.ref027","first-page":"702","article-title":"A New and Improved Method for Automated Screening of Age-Related Macular Degeneration Using Ensemble Deep Neural Networks","volume":"2018","author":"A Govindaiah","year":"2018","journal-title":"Conf Proc IEEE Eng Med Biol Soc"},{"key":"pone.0251553.ref028","unstructured":"Athiwaratkun B, Kang K. Feature representation in convolutional neural networks. arXiv preprint arXiv:150702313. 2015."},{"key":"pone.0251553.ref029","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1016\/j.compbiomed.2018.10.031","article-title":"Exudate detection in fundus images using deeply-learnable features","volume":"104","author":"P Khojasteh","year":"2019","journal-title":"Comput Biol Med"},{"issue":"6","key":"pone.0251553.ref030","doi-asserted-by":"crossref","first-page":"1626","DOI":"10.3390\/s20061626","article-title":"Stochastic Selection of Activation Layers for Convolutional Neural Networks","volume":"20","author":"L Nanni","year":"2020","journal-title":"Sensors"},{"key":"pone.0251553.ref031","doi-asserted-by":"crossref","unstructured":"Tao S, editor Deep neural network ensembles. International Conference on Machine Learning, Optimization, and Data Science; 2019: Springer.","DOI":"10.1007\/978-3-030-37599-7_1"},{"key":"pone.0251553.ref032","doi-asserted-by":"crossref","unstructured":"Yang J, Wang F. Auto-Ensemble: An Adaptive Learning Rate Scheduling based Deep Learning Model Ensembling. arXiv preprint arXiv:200311266. 2020.","DOI":"10.1109\/ACCESS.2020.3041525"},{"issue":"3","key":"pone.0251553.ref033","first-page":"18","article-title":"Classification and regression by randomForest","volume":"2","author":"A Liaw","year":"2002","journal-title":"R news"},{"issue":"6","key":"pone.0251553.ref034","doi-asserted-by":"crossref","first-page":"e015719","DOI":"10.1371\/journal.pone.0157190","article-title":"Structural changes of inner and outer choroid in central serous chorioretinopathy determined by optical coherence tomography","volume":"11","author":"S Sonoda","year":"2016","journal-title":"Plos One"},{"issue":"2","key":"pone.0251553.ref035","doi-asserted-by":"crossref","first-page":"e0212055","DOI":"10.1371\/journal.pone.0212055","article-title":"The association of choroidal structure and its response to anti-VEGF treatment with the short-time outcome in pachychoroid neovasculopathy","volume":"14","author":"K Azuma","year":"2019","journal-title":"Plos One"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0251553","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,27]],"date-time":"2022-12-27T12:00:22Z","timestamp":1672142422000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0251553"}},"subtitle":[],"editor":[{"given":"Alfred S","family":"Lewin","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2021,5,14]]},"references-count":35,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2021,5,14]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0251553","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,14]]}}}