{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T16:45:20Z","timestamp":1790700320458,"version":"4.1.0"},"reference-count":45,"publisher":"IOP Publishing","issue":"4","license":[{"start":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T00:00:00Z","timestamp":1753660800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/page\/copyright"},{"start":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T00:00:00Z","timestamp":1753660800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Biomed. Phys. Eng. Express"],"published-print":{"date-parts":[[2025,7,31]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Retinal diseases such as age-related macular degeneration and diabetic retinopathy will lead to irreversible blindness without timely diagnosis and treatment. Optical coherence tomography (OCT) and optical coherence tomography angiography (OCTA) images provide complementary views of the retina, and the integration of the two imaging modalities can improve the accuracy of retinal disease classification. We propose a multi-modal classification model consisting of two branches to automatically diagnose retinal diseases, in which OCT and OCTA images are efficiently integrated to improve both the accuracy and efficiency of disease diagnosis. A bright line cropping is used to remove the useless black edge region while preserving the lesion features and reducing the calculation load. To solve the insufficient data issue, data enhancement and loose matching methods are adopted to increase the data amount. A two-step training method is used to train our proposed model, alleviating the limited training images. Our model is tested on an external test set instead of a training set, making the classification results more rigorous. The intermediate fusion and two-step training methods are adopted in our multiple classification model, achieving 0.9667, 0.9418, 0.8569, 0.9422, and 0.8921 in average accuracy, precision, recall, specificity, and F1-Score, respectively. Our multi-modal model outperforms the single-modal model, the early, and late fusion multi-modal model in accuracy. Our model offers doctors less human error, lower cost, more uniform, and effective mass screening, thus providing a solution to improve deep learning performance in terms of a relatively fewer number of training data and even more imbalanced classes.<\/jats:p>","DOI":"10.1088\/2057-1976\/adeb92","type":"journal-article","created":{"date-parts":[[2025,7,3]],"date-time":"2025-07-03T18:48:06Z","timestamp":1751568486000},"page":"045030","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Multi-modal classification of retinal disease based on convolutional neural network"],"prefix":"10.1088","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-8039-9067","authenticated-orcid":true,"given":"Hongyi","family":"Pan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4217-5389","authenticated-orcid":true,"given":"Jingpeng","family":"Miao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jie","family":"Yu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jingmin","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaobing","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jihong","family":"Feng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2025,7,28]]},"reference":[{"key":"bpexadeb92bib1","doi-asserted-by":"publisher","first-page":"347","DOI":"10.1016\/j.survophthal.2012.01.004","article-title":"Prevalence of diabetic retinopathy in various ethnic groups: a worldwide perspective","volume":"57","author":"Sivaprasad","year":"2012","journal-title":"Surv Ophthalmol"},{"key":"bpexadeb92bib2","doi-asserted-by":"publisher","first-page":"1580","DOI":"10.1016\/j.ophtha.2021.04.027","article-title":"Global prevalence of diabetic retinopathy and projection of burden through 2045 systematic review and meta-analysis","volume":"128","author":"Teo","year":"2021","journal-title":"Ophthalmology"},{"key":"bpexadeb92bib3","doi-asserted-by":"publisher","first-page":"c981","DOI":"10.1136\/bmj.c981","article-title":"Age related macular degeneration","volume":"340","author":"Chakravarthy","year":"2010","journal-title":"Bmj-Brit. Med. J."},{"key":"bpexadeb92bib4","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1007\/s10916-007-9113-9","article-title":"Automated identification of diabetic retinopathy stages using digital fundus images","volume":"32","author":"Nayak","year":"2008","journal-title":"J. Med. Syst."},{"key":"bpexadeb92bib5","doi-asserted-by":"publisher","first-page":"1178","DOI":"10.1126\/science.1957169","article-title":"Optical coherence tomography","volume":"254","author":"Huang","year":"1991","journal-title":"Science"},{"key":"bpexadeb92bib6","doi-asserted-by":"publisher","first-page":"149","DOI":"10.1038\/s41433-020-01233-y","article-title":"Optical coherence tomography angiography in diabetic retinopathy: an updated review","volume":"35","author":"Sun","year":"2021","journal-title":"Eye."},{"key":"bpexadeb92bib7","doi-asserted-by":"publisher","first-page":"871","DOI":"10.1159\/000518620","article-title":"Standardization of optical coherence tomography angiography imaging biomarkers in diabetic retinal disease","volume":"64","author":"Vujosevic","year":"2021","journal-title":"Ophthalmic Res."},{"key":"bpexadeb92bib8","doi-asserted-by":"publisher","first-page":"220","DOI":"10.1097\/IAE.0000000000001628","article-title":"Biomarkers of neovascular activity in age-related macular degeneration using optical coherence tomography angiography","volume":"38","author":"Al-Sheikh","year":"2018","journal-title":"Retina-J. Ret. Vit. Dis."},{"key":"bpexadeb92bib9","doi-asserted-by":"publisher","first-page":"911","DOI":"10.1364\/BOE.7.000911","article-title":"Comparison of amplitude-decorrelation, speckle-variance and phase-variance oct angiography methods for imaging the human retina and choroid","volume":"7","author":"Gorczynska","year":"2016","journal-title":"Biomed. Opt. Express"},{"key":"bpexadeb92bib10","doi-asserted-by":"publisher","DOI":"10.1016\/j.bspc.2023.104604","article-title":"Transformer and convolutional based dual branch network for retinal vessel segmentation in octa images","volume":"83","author":"Liu","year":"2023","journal-title":"Biomed Signal Proces"},{"key":"bpexadeb92bib11","doi-asserted-by":"publisher","DOI":"10.3389\/fpubh.2022.1047608","article-title":"Ultra-widefield color fundus photography combined with high-speed ultra-widefield swept-source optical coherence tomography angiography for non-invasive detection of lesions in diabetic retinopathy","volume":"10","author":"Li","year":"2022","journal-title":"Frontiers in Public Health"},{"key":"bpexadeb92bib12","doi-asserted-by":"publisher","first-page":"192","DOI":"10.1186\/s12886-021-01933-3","article-title":"Ultra-wide field swept-source optical coherence tomography angiography in patients with diabetes without clinically detectable retinopathy","volume":"21","author":"Yang","year":"2021","journal-title":"Bmc Ophthalmol."},{"key":"bpexadeb92bib13","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1142\/S1793545822500092","article-title":"Computer-aided diagnosis of retinopathy based on vision transformer","volume":"15","author":"Jiang","year":"2022","journal-title":"J. Innov. Opt. Heal. Sci"},{"key":"bpexadeb92bib14","doi-asserted-by":"publisher","first-page":"102608","DOI":"10.1016\/j.media.2022.102608","article-title":"Focused attention in transformers for interpretable classification of retinal images","volume":"82","author":"Playout","year":"2022","journal-title":"Med. Image Anal."},{"key":"bpexadeb92bib15","doi-asserted-by":"publisher","first-page":"163","DOI":"10.3390\/electronics10020163","article-title":"Development of decision support software for deep learning-based automated retinal disease screening using relatively limited fundus photograph data","volume":"10","author":"Lee","year":"2021","journal-title":"Electronics"},{"key":"bpexadeb92bib16","doi-asserted-by":"crossref","DOI":"10.1109\/ICICML57342.2022.10009714","article-title":"Sae-wamd: a self-attention enhanced convolution neural network for fine-grained classification of wet age-related macular degeneration using oct images","author":"Haihong","year":"2022"},{"key":"bpexadeb92bib17","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1007\/s10792-024-03115-8","article-title":"A new intelligent system based deep learning to detect dme and amd in oct images","volume":"44","author":"Gueddena","year":"2024","journal-title":"Int Ophthalmol."},{"key":"bpexadeb92bib18","doi-asserted-by":"publisher","DOI":"10.1155\/2023\/1305583","article-title":"Classification of diabetic retinopathy severity in fundus images using the vision transformer and residual attention","volume":"2023","author":"Gu","year":"2023","journal-title":"Comput. Intel. Neurosc."},{"key":"bpexadeb92bib19","doi-asserted-by":"publisher","first-page":"1122","DOI":"10.1016\/j.cell.2018.02.010","article-title":"Identifying medical diagnoses and treatable diseases by image-based deep learning","volume":"172","author":"Kermany","year":"2018","journal-title":"Cell."},{"key":"bpexadeb92bib20","doi-asserted-by":"publisher","first-page":"9435","DOI":"10.3390\/app12199435","article-title":"Towards explainable deep neural networks for the automatic detection of diabetic retinopathy","volume":"12","author":"Alghamdi","year":"2022","journal-title":"Appl. Sci-Basel."},{"key":"bpexadeb92bib21","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1109\/MSP.2017.2738401","article-title":"Deep multimodal learning a survey on recent advances and trends","volume":"34","author":"Ramachandram","year":"2017","journal-title":"Ieee Signal Proc Mag."},{"key":"bpexadeb92bib22","doi-asserted-by":"publisher","first-page":"2324","DOI":"10.1002\/mp.15541","article-title":"Automated diagnosis of age-related macular degeneration using multi-modal vertical plane feature fusion via deep learning","volume":"49","author":"Chen","year":"2022","journal-title":"Med. Phys."},{"key":"bpexadeb92bib23","doi-asserted-by":"publisher","DOI":"10.3389\/fnins.2022.939472","article-title":"A multimodal classification architecture for the severity diagnosis of glaucoma based on deep learning","volume":"16","author":"Yi","year":"2022","journal-title":"Front Neurosci-Switz."},{"key":"bpexadeb92bib24","doi-asserted-by":"publisher","first-page":"2392","DOI":"10.1109\/TMI.2021.3077484","article-title":"Deep relation transformer for diagnosing glaucoma with optical coherence tomography and visual field function","volume":"40","author":"Song","year":"2021","journal-title":"Ieee T. Med. Imaging."},{"key":"bpexadeb92bib25","doi-asserted-by":"publisher","first-page":"841","DOI":"10.1080\/0952813X.2019.1653383","article-title":"Multi-modal classifier fusion with feature cooperation for glaucoma diagnosis","volume":"31","author":"Benzebouchi","year":"2019","journal-title":"J. Exp. Theor. Artif. In."},{"key":"bpexadeb92bib26","doi-asserted-by":"crossref","DOI":"10.1109\/ISBI.2018.8363717","article-title":"Medical image segmentation based on multi-modal convolutional neural network: study on image fusion schemes","author":"Guo","year":"2018"},{"key":"bpexadeb92bib27","doi-asserted-by":"publisher","first-page":"569","DOI":"10.1093\/bib\/bbab569","article-title":"Multimodal deep learning for biomedical data fusion: a review","volume":"23","author":"Stahlschmidt","year":"2022","journal-title":"Brief. Bioinform."},{"key":"bpexadeb92bib28","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1167\/tvst.11.2.39","article-title":"A deep learning algorithm for classifying diabetic retinopathy using optical coherence tomography angiography","volume":"11","author":"Ryu","year":"2022","journal-title":"Transl. Vis. Sci. Techn."},{"key":"bpexadeb92bib29","doi-asserted-by":"publisher","first-page":"1591","DOI":"10.1109\/TMI.2021.3059956","article-title":"Multi-modal retinal image classification with modality-specific attention network","volume":"40","author":"He","year":"2021","journal-title":"Ieee T. Med. Imaging."},{"key":"bpexadeb92bib30","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2023.107396","article-title":"De-janet: a unified network based on dual encoder and joint attention for alzheimer\u2019s disease classification using multi-modal data","volume":"165","author":"Dai","year":"2023","journal-title":"Comput. Biol. Med."},{"key":"bpexadeb92bib31","doi-asserted-by":"publisher","first-page":"507","DOI":"10.1016\/j.bbe.2023.06.001","article-title":"Transformer-based cross-modal multi-contrast network for ophthalmic diseases diagnosis","volume":"43","author":"Yu","year":"2023","journal-title":"Biocybern Biomed Eng."},{"key":"bpexadeb92bib32","doi-asserted-by":"publisher","first-page":"E512","DOI":"10.1111\/aos.14928","article-title":"Multimodal deep learning with feature level fusion for identification of choroidal neovascularization activity in age-related macular degeneration","volume":"100","author":"Jin","year":"2022","journal-title":"Acta Ophthalmol."},{"key":"bpexadeb92bib33","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2024.103092","article-title":"Octa-500: a retinal dataset for optical coherence tomography angiography study","volume":"93","author":"Li","year":"2024","journal-title":"Med. Image Anal."},{"key":"bpexadeb92bib34","doi-asserted-by":"publisher","first-page":"3343","DOI":"10.1109\/TMI.2020.2992244","article-title":"Image projection network: 3d to 2d image segmentation in octa images","volume":"39","author":"Li","year":"2020","journal-title":"Ieee T. Med. Imaging."},{"key":"bpexadeb92bib35","doi-asserted-by":"publisher","first-page":"2212","DOI":"10.1167\/iovs.17-23498","article-title":"Evaluation of automatically quantified foveal avascular zone metrics for diagnosis of diabetic retinopathy using optical coherence tomography angiography","volume":"59","author":"Lu","year":"2018","journal-title":"Invest. Ophth. Vis. Sci."},{"key":"bpexadeb92bib36","doi-asserted-by":"publisher","first-page":"1435","DOI":"10.1016\/j.ophtha.2014.01.034","article-title":"Quantitative optical coherence tomography angiography of choroidal neovascularization in age-related macular degeneration","volume":"121","author":"Jia","year":"2014","journal-title":"Ophthalmology"},{"key":"bpexadeb92bib37","doi-asserted-by":"publisher","first-page":"816","DOI":"10.1364\/BOE.7.000816","article-title":"Projection-resolved optical coherence tomographic angiography","volume":"7","author":"Zhang","year":"2016","journal-title":"Biomed. Opt. Express"},{"key":"bpexadeb92bib38","doi-asserted-by":"publisher","DOI":"10.1016\/j.bspc.2024.107146","article-title":"A lightweight model for the retinal disease classification using optical coherence tomography","volume":"101","author":"Pan","year":"2025","journal-title":"Biomed Signal Proces."},{"key":"bpexadeb92bib39","doi-asserted-by":"publisher","first-page":"1126","DOI":"10.1038\/eye.2014.147","article-title":"Decreased photoreceptor inner segment\/outer segment junction reflectivity in patients with idiopathic epimacular membrane","volume":"28","author":"Toprak","year":"2014","journal-title":"Eye."},{"key":"bpexadeb92bib40","article-title":"Efficientnetv2: smaller models and faster training","author":"Tan","year":"2021"},{"key":"bpexadeb92bib41","doi-asserted-by":"publisher","first-page":"4713","DOI":"10.1364\/BOE.495999","article-title":"Optimizing the octa layer fusion option for deep learning classification of diabetic retinopathy","volume":"14","author":"Ebrahimi","year":"2023","journal-title":"Biomed. Opt. Express"},{"key":"bpexadeb92bib42","doi-asserted-by":"publisher","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A survey on transfer learning","volume":"22","author":"Pan","year":"2010","journal-title":"Ieee T. Knowl. Data. En."},{"key":"bpexadeb92bib43","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1109\/JPROC.2020.3004555","article-title":"A comprehensive survey on transfer learning","volume":"109","author":"Zhuang","year":"2021","journal-title":"P. Ieee."},{"key":"bpexadeb92bib44","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"Imagenet large scale visual recognition challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vision"},{"key":"bpexadeb92bib45","doi-asserted-by":"publisher","first-page":"4111","DOI":"10.1109\/JBHI.2022.3171523","article-title":"Learning two-stream cnn for multi-modal age-related macular degeneration categorization","volume":"26","author":"Wang","year":"2022","journal-title":"Ieee J. Biomed Health."}],"container-title":["Biomedical Physics &amp; Engineering Express"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/adeb92","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/adeb92\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/adeb92","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/adeb92\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/adeb92\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/adeb92\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/adeb92\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/adeb92\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,29]],"date-time":"2025-07-29T04:47:00Z","timestamp":1753764420000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/adeb92"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,28]]},"references-count":45,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2025,7,28]]},"published-print":{"date-parts":[[2025,7,31]]}},"URL":"https:\/\/doi.org\/10.1088\/2057-1976\/adeb92","relation":{},"ISSN":["2057-1976"],"issn-type":[{"value":"2057-1976","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,28]]},"assertion":[{"value":"Multi-modal classification of retinal disease based on convolutional neural network","name":"article_title","label":"Article Title"},{"value":"Biomedical Physics & Engineering Express","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2025 IOP Publishing Ltd. All rights, including for text and data mining, AI training, and similar technologies, are reserved.","name":"copyright_information","label":"Copyright Information"},{"value":"2025-03-28","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2025-07-03","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2025-07-28","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}