{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,12]],"date-time":"2026-10-12T12:33:46Z","timestamp":1791808426134,"version":"4.3.4"},"reference-count":46,"publisher":"MDPI AG","issue":"14","license":[{"start":{"date-parts":[[2025,7,19]],"date-time":"2025-07-19T00:00:00Z","timestamp":1752883200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Union NextGenerationEU National Recovery and Resilience Plan (Piano Nazionale di Ripresa e Resilienza, PNRR)\u2014Project: \u201cSoBigData.it\u2014Strengthening the Italian RI for Social Mining and Big Data Analytics\u201d","award":["Prot. IR0000013-Avviso n. 3264 del 28\/12\/2021"],"award-info":[{"award-number":["Prot. IR0000013-Avviso n. 3264 del 28\/12\/2021"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Ocular diseases can significantly affect vision and overall quality of life, with diagnosis often being time-consuming and dependent on expert interpretation. While previous computer-aided diagnostic systems have focused primarily on medical imaging, this paper proposes VisionTrack, a multi-modal AI system for predicting multiple retinal diseases, including Diabetic Retinopathy (DR), Age-related Macular Degeneration (AMD), Diabetic Macular Edema (DME), drusen, Central Serous Retinopathy (CSR), and Macular Hole (MH), as well as normal cases. The proposed framework integrates a Convolutional Neural Network (CNN) for image-based feature extraction, a Graph Neural Network (GNN) to model complex relationships among clinical risk factors, and a Large Language Model (LLM) to process patient medical reports. By leveraging diverse data sources, VisionTrack improves prediction accuracy and offers a more comprehensive assessment of retinal health. Experimental results demonstrate the effectiveness of this hybrid system, highlighting its potential for early detection, risk assessment, and personalized ophthalmic care. Experiments were conducted using two publicly available datasets, RetinalOCT and RFMID, which provide diverse retinal imaging modalities: OCT images and fundus images, respectively. The proposed multi-modal AI system demonstrated strong performance in multi-label disease prediction. On the RetinalOCT dataset, the model achieved an accuracy of 0.980, F1-score of 0.979, recall of 0.978, and precision of 0.979. Similarly, on the RFMID dataset, it reached an accuracy of 0.989, F1-score of 0.881, recall of 0.866, and precision of 0.897. These results confirm the robustness, reliability, and generalization capability of the proposed approach across different imaging modalities.<\/jats:p>","DOI":"10.3390\/s25144492","type":"journal-article","created":{"date-parts":[[2025,7,21]],"date-time":"2025-07-21T08:47:14Z","timestamp":1753087634000},"page":"4492","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":32,"title":["Multi-Modal AI for Multi-Label Retinal Disease Prediction Using OCT and Fundus Images: A Hybrid Approach"],"prefix":"10.3390","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5915-9986","authenticated-orcid":false,"given":"Amina","family":"Zedadra","sequence":"first","affiliation":[{"name":"LabSTIC Laboratory, University 8 May 1945 Guelma, Algeria, BP 401, Guelma 24000, Algeria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahmoud Yassine","family":"Salah-Salah","sequence":"additional","affiliation":[{"name":"Medical-Surgical Ophthalmology Office, 284 CNEP Apartments, University Road, Block 37, No. 1, Guelma 24006, Algeria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7991-0677","authenticated-orcid":false,"given":"Ouarda","family":"Zedadra","sequence":"additional","affiliation":[{"name":"LabSTIC Laboratory, University 8 May 1945 Guelma, Algeria, BP 401, Guelma 24000, Algeria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1469-9484","authenticated-orcid":false,"given":"Antonio","family":"Guerrieri","sequence":"additional","affiliation":[{"name":"ICAR-CNR\u2014Institute for High Performance Computing and Networking, National Research Council of Italy, Via P. Bucci 8\/9C, 87036 Rende, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,7,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2029","DOI":"10.1007\/s11517-025-03314-0","article-title":"A Novel Deep Learning Framework for Retinal Disease Detection Leveraging Contextual and Local Feature Cues from Retinal Images","volume":"63","author":"Khan","year":"2025","journal-title":"Med. Biol. Eng. Comput."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2215","DOI":"10.1109\/TMI.2024.3362964","article-title":"Embedding Tasks into the Latent Space: Cross-Space Consistency for Multi-Dimensional Analysis in Echocardiography","volume":"43","author":"Zhang","year":"2024","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"101668","DOI":"10.1016\/j.media.2020.101668","article-title":"Contrast Agent-Free Synthesis and Segmentation of Ischemic Heart Disease Images Using Progressive Sequential Causal GANs","volume":"62","author":"Xu","year":"2020","journal-title":"Med. Image Anal."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Karule, P.T., Bhele, S.B., Palsodkar, P., Agarkar, P.T., Hajare, H.R., and Patil, P.R. (2024, January 7\u20138). Detection of Multi-Class Multi-Label Ophthalmological Diseases in Retinal Fundus Images Using Machine Learning. Proceedings of the 2024 International Conference on Innovations and Challenges in Emerging Technologies (ICICET), Nagpur, India.","DOI":"10.1109\/ICICET59348.2024.10616291"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Li, T., and Sheng, B. (2024, January 3\u20136). MSCE-LT: Multi-Label Supervised Contrastive Enhancement for Long-Tailed Retinal Diseases Recognition. Proceedings of the 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Lisbon, Portugal.","DOI":"10.1109\/BIBM62325.2024.10821772"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Chandran, A., Arathy, S., Gowri, R., Valsaraj, S., and Jeena, R.S. (2024, January 24\u201328). Real Time Diagnosis of Ocular Diseases using AI. Proceedings of the 2024 15th International Conference on Computing Communication and Networking Technologies (ICCCNT), Kamand, India.","DOI":"10.1109\/ICCCNT61001.2024.10724911"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1007\/s10462-024-10736-z","article-title":"A Comprehensive Review of Artificial Intelligence Models for Screening Major Retinal Diseases","volume":"57","author":"Hassan","year":"2024","journal-title":"Artif. Intell. Rev."},{"key":"ref_8","unstructured":"El-Sharkawy, M., Ahmed, A., Nassar, M., and Farag, A. (2024). A Clinically Explainable AI-Based Grading System for Age-Related Macular Degeneration Using Optical Coherence Tomography. Sci. Rep., 14."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Abd El-Khalek, A.A., Balaha, H.M., Sewelam, A., Ghazal, M., Khalil, A.T., Abo-Elsoud, M.E.A., and El-Baz, A. (2024). A Comprehensive Review of AI Diagnosis Strategies for Age-Related Macular Degeneration (AMD). Bioengineering, 11.","DOI":"10.3390\/bioengineering11070711"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Yang, J., Liu, X., Zhao, W., and Chen, Q. (2024). Dry Age-Related Macular Degeneration Classification from Optical Coherence Tomography Images Based on Ensemble Deep Learning Architecture. Front. Med., 11.","DOI":"10.3389\/fmed.2024.1438768"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"3026500","DOI":"10.1155\/2024\/3026500","article-title":"ViT-AMD: A New Deep Learning Model for Age-Related Macular Degeneration Diagnosis From Fundus Images","volume":"2024","author":"Le","year":"2024","journal-title":"Int. J. Intell. Syst."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1038\/s41746-024-01018-7","article-title":"Translating Color Fundus Photography to Indocyanine Green Angiography Using Deep Learning for Age-Related Macular Degeneration Screening","volume":"7","author":"Chen","year":"2024","journal-title":"NPJ Digit. Med."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Akram, M., Adnan, M., Ali, S.F., Ahmad, J., Yousef, A., Alshalali, T.A.N., and Shaikh, Z.A. (2025). Uncertainty-aware diabetic retinopathy detection using deep learning enhanced by Bayesian approaches. Sci. Rep., 15.","DOI":"10.1038\/s41598-024-84478-x"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Liu, C., Wang, W., Lian, J., and Jiao, W. (2025). Lesion classification and diabetic retinopathy grading by integrating softmax and pooling operators into vision transformer. Front. Public Health, 12.","DOI":"10.3389\/fpubh.2024.1442114"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"3301618","DOI":"10.1109\/ACCESS.2023.3301618","article-title":"Diabetic Retinopathy Detection and Grading: A Transfer Learning Approach Using Simultaneous Parameter Optimization and Feature-Weighted ECOC Ensemble","volume":"11","author":"Wong","year":"2023","journal-title":"IEEE Access"},{"key":"ref_16","first-page":"10099069","article-title":"An Enhanced Segmentation and Deep Learning Architecture for Early Diabetic Retinopathy Detection","volume":"13","author":"Maaliw","year":"2023","journal-title":"IEEE CCWC"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Zedadra, A., Zedadra, O., Salah-Salah, M.Y., and Guerrieri, A. (2025). Graph-Aware Multimodal Deep Learning for Classification of Diabetic Retinopathy Images. IEEE Access.","DOI":"10.1109\/ACCESS.2025.3564529"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"51435","DOI":"10.1109\/ACCESS.2023.3279393","article-title":"Classification of Diabetic Retinopathy Disease Levels by Extracting Topological Features Using Graph Neural Networks","volume":"11","author":"Sumod","year":"2023","journal-title":"IEEE Access"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"171","DOI":"10.31449\/inf.v48i2.5078","article-title":"DRG-Net: Diabetic Retinopathy Grading Network Using Graph Learning with Extreme Gradient Boosting Classifier","volume":"48","author":"Poranki","year":"2024","journal-title":"Informatica"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Zhang, G., Sun, B., Zhang, Z., Pan, J., Yang, W., and Liu, Y. (2022). Multi-Model Domain Adaptation for Diabetic Retinopathy Classification. Front. Physiol., 13.","DOI":"10.3389\/fphys.2022.918929"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"98391","DOI":"10.1109\/ACCESS.2023.3312709","article-title":"Grading of Diabetic Retinopathy Images Based on Graph Neural Network","volume":"11","author":"Feng","year":"2023","journal-title":"IEEE Access"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"100303","DOI":"10.1016\/j.health.2024.100303","article-title":"An Advanced Deep Neural Network for Fundus Image Analysis and Enhancing Diabetic Retinopathy Detection","volume":"5","author":"Shamrat","year":"2024","journal-title":"Healthc. Anal."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"9","DOI":"10.14445\/23488549\/IJECE-V10I8P102","article-title":"Leveraging Semi-Supervised Graph Learning for Enhanced Diabetic Retinopathy Detection","volume":"10","author":"Dhinakaran","year":"2023","journal-title":"SSRG Int. J. Electron. Commun. Eng."},{"key":"ref_24","first-page":"101210","article-title":"Enhanced Diabetic Macular Edema Detection in Multicolor Imaging through a Multi-Feature Decomposition Fusion Attention Network","volume":"18","author":"Fu","year":"2025","journal-title":"J. Radiat. Res. Appl. Sci."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Zhang, P., Conze, P.H., Lamard, M., Cochener, B., Quellec, G., and El Habib Daho, M. (2025). Deep Learning-Based Detection of Referable Diabetic Retinopathy and Macular Edema Using Ultra-Widefield Fundus Imaging. Proceedings of the MICCAI Challenge on Ultra-Widefield Fundus Imaging for Diabetic Retinopathy, Springer.","DOI":"10.1007\/978-3-031-89388-9_9"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1110","DOI":"10.1097\/IAE.0000000000002992","article-title":"Detection of Morphologic Patterns of Diabetic Macular Edema Using a Deep Learning Approach Based on Optical Coherence Tomography Images","volume":"41","author":"Wu","year":"2021","journal-title":"Retina"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Saidi, L., Jomaa, H., Zainab, H., Zgolli, H., Mabrouk, S., Sidib\u00e9, D., Tabia, H., and Khlifa, N. (2023). Automatic Detection of AMD and DME Retinal Pathologies Using Deep Learning. Int. J. Biomed. Imaging, 2023.","DOI":"10.1155\/2023\/9966107"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"e18773","DOI":"10.1016\/j.heliyon.2023.e18773","article-title":"Generating OCT B-Scan DME Images Using Optimized Generative Adversarial Networks (GANs)","volume":"9","author":"Tripathi","year":"2023","journal-title":"Heliyon"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Thanikachalam, V., Kabilan, K., and Erramchetty, S. (2024). Optimized Deep CNN for Detection and Classification of Diabetic Retinopathy and Diabetic Macular Edema. BMC Med. Imaging, 24.","DOI":"10.1186\/s12880-024-01406-1"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Nazir, T., Nawaz, M., and Rashid, J. (2021). Detection of Diabetic Eye Disease from Retinal Images Using a Deep Learning-Based CenterNet Model. Sensors, 21.","DOI":"10.3390\/s21165283"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1100","DOI":"10.1016\/j.procs.2024.09.529","article-title":"Fully-Automatic End-to-End Approaches for 3D Drusen Segmentation in Optical Coherence Tomography Images","volume":"246","author":"Goyanes","year":"2024","journal-title":"Procedia Comput. Sci."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Ilyasova, N.Y., Demin, N.S., and Kuritsyn, N.S. (2024). Recognition of Drusen Subtypes on OCT Data for the Diagnosis of Age-Related Macular Degeneration. J. Biomed. Photonics Eng., 10.","DOI":"10.18287\/JBPE24.10.030307"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"51809","DOI":"10.1007\/s11042-023-17169-w","article-title":"Exudate and Drusen Classification in Retinal Images Using Bagged Colour Vector Angles and Inter Colour Local Binary Patterns","volume":"83","author":"Omar","year":"2024","journal-title":"Multimed. Tools Appl."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"108056","DOI":"10.1016\/j.optlaseng.2024.108056","article-title":"Residual Attention Network for Distinction Between Visible Optic Disc Drusen and Healthy Optic Discs","volume":"176","author":"Nowomiejska","year":"2024","journal-title":"Opt. Lasers Eng."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"168731","DOI":"10.1109\/ACCESS.2021.3108395","article-title":"Recent Developments in Detection of Central Serous Retinopathy through Imaging and Artificial Intelligence Techniques\u2014A Review","volume":"9","author":"Hassan","year":"2021","journal-title":"IEEE Access"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Zhen, Y., Chen, H., Zhang, X., Liu, M., Meng, X., Zhang, J., and Pu, J. (2020). Assessment of Central Serous Chorioretinopathy (CSC) Depicted on Color Fundus Photographs Using Deep Learning. arXiv.","DOI":"10.1097\/IAE.0000000000002621"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Yoon, J., Han, J., Park, J.I., Hwang, J.S., Han, J.M., Sohn, J., Park, K.H., and Hwang, D.D.J. (2020). Optical Coherence Tomography-Based Deep-Learning Model for Detecting Central Serous Chorioretinopathy. Sci. Rep., 10.","DOI":"10.1038\/s41598-020-75816-w"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"21369","DOI":"10.1007\/s11042-023-16206-y","article-title":"Detection of Central Serous Retinopathy Using Deep Learning Through Retinal Images","volume":"83","author":"Hassan","year":"2024","journal-title":"Multimed. Tools Appl."},{"key":"ref_39","first-page":"2358","article-title":"AI-Driven Macular Hole Detection in OCT Scans: Efficient Training, Robust Performance, and Global Significance","volume":"65","author":"Bolanos","year":"2024","journal-title":"Investig. Ophthalmol. Vis. Sci."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"6904217","DOI":"10.1155\/2021\/6904217","article-title":"Macular Hole Detection Using a New Hybrid Method: Using Multilevel Thresholding and Derivation on Optical Coherence Tomographic Images","volume":"2021","author":"Shahalinejad","year":"2021","journal-title":"Comput. Intell. Neurosci."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Ko, J., Han, J., Yoon, J., Park, J.I., Hwang, J.S., Han, J.M., Park, K.H., and Hwang, D.D.J. (2022). Assessing Central Serous Chorioretinopathy with Deep Learning and Multiple Optical Coherence Tomography Images. Sci. Rep., 12.","DOI":"10.1038\/s41598-022-05051-y"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Pachade, S., Porwal, P., Thulkar, D., Kokare, M., Deshmukh, G., Sahasrabuddhe, V., and Meriaudeau, F. (2021). Retinal Fundus Multi-Disease Image Dataset (RFMiD): A Dataset for Multi-Disease Detection Research. Data, 6.","DOI":"10.3390\/data6020014"},{"key":"ref_43","unstructured":"Naren, O.S. (2025, June 11). Retinal OCT Image Classification-C8 Dataset; Kaggle. Available online: https:\/\/www.kaggle.com\/datasets\/obulisainaren\/retinal-oct-c8\/versions\/2."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"49813","DOI":"10.1007\/s11042-023-17530-z","article-title":"Multiclass Multilabel Ophthalmological Fundus Image Classification Based on Optimised Deep Feature Space Evolutionary Model","volume":"83","author":"Bali","year":"2024","journal-title":"Multimed. Tools Appl."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"255","DOI":"10.53106\/199115992024023501021","article-title":"Retinal OCT Image Classification Based on CNN-RNN Unified Neural Networks","volume":"35","author":"Jiang","year":"2024","journal-title":"J. Comput."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Tan, Z., Zhang, Q., Lan, G., Xu, J., Ou, C., An, L., Qin, J., and Huang, Y. (2024). OCT Retinopathy Classification via a Semi-Supervised Pseudo-Label Sub-Domain Adaptation and Fine-Tuning Method. Mathematics, 12.","DOI":"10.3390\/math12020347"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/25\/14\/4492\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:12:31Z","timestamp":1760033551000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/25\/14\/4492"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,19]]},"references-count":46,"journal-issue":{"issue":"14","published-online":{"date-parts":[[2025,7]]}},"alternative-id":["s25144492"],"URL":"https:\/\/doi.org\/10.3390\/s25144492","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,19]]}}}