{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,24]],"date-time":"2026-08-24T23:23:16Z","timestamp":1787613796998,"version":"build-2736575974"},"reference-count":56,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T00:00:00Z","timestamp":1781136000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,8,25]],"date-time":"2026-08-25T00:00:00Z","timestamp":1787616000000},"content-version":"vor","delay-in-days":75,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Scientific Research Deanship at University of Ha\u2019il - Saudi Arabia","award":["RG-24 140"],"award-info":[{"award-number":["RG-24 140"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Retinal lesion segmentation is one of the critical tasks to analyze retinal diseases. Many researchers have proposed deep-learning models to extract lesions from the retinal scans. However, these models often rely on image features that might not be clinically meaningful for identifying the retinal lesions. Additionally, these models need pixel-level ground truths, which are challenging to procure in the real world. To overcome these issues, we present a novel language-assisted multimodal convolutional transformer pipeline that aligns image features with the text features, where the text features are extracted from the prompts that contain clinically meaningful information about the retinal lesions, and the image features are generated from the retinal scans. This alignment between image and text features is established with one-time training using the proposed loss function. Afterward, the proposed network can robustly extract retinal lesions across different datasets at the inference stage. Moreover, since the proposed network infers learning from the text prompts, it does not require additional training rounds using pixel-level ground truth annotations to adapt to new datasets like the state-of-the-art methods. The proposed network is thoroughly tested on six public datasets, and it outperforms the state-of-the-art by achieving up to 7.77% improvements in terms of intersection-over-union.<\/jats:p>","DOI":"10.1038\/s41598-026-55337-8","type":"journal-article","created":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T05:43:55Z","timestamp":1781156635000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Language-assisted multimodal convolutional transformer pipeline for retinal lesions segmentation"],"prefix":"10.1038","volume":"16","author":[{"given":"Wilayat","family":"Khan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohammad","family":"Alsaffar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Muhammad Faisal","family":"Abrar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Taimur","family":"Hassan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,6,11]]},"reference":[{"key":"55337_CR1","doi-asserted-by":"crossref","unstructured":"Niu, Y., Gu, L., Zhao, Y. & Lu, F. Explainable diabetic retinopathy detection and retinal image generation. IEEE J. Biomed. Health Inform. 26(1), 44\u201355 (2022).","DOI":"10.1109\/JBHI.2021.3110593"},{"key":"55337_CR2","doi-asserted-by":"crossref","unstructured":"Mani, P., Ramachandran, N., Naveen, P. & Ramesh, P. V. Automated segmentation of retinal layers in optical coherence tomography images using Xception70 feature extraction. Appl. Soft Comput. 167, 112414 (2024).","DOI":"10.1016\/j.asoc.2024.112414"},{"key":"55337_CR3","doi-asserted-by":"crossref","unstructured":"Li, M., Zhang, W., Zhao, H., Xu, Y., Xu, J. & Li, H., A spatiotemporal convolution recurrent neural network for pixel-level peripapillary atrophy prediction using sequential fundus images. Appl. Soft Comput. 155, 111431 (2024).","DOI":"10.1016\/j.asoc.2024.111431"},{"key":"55337_CR4","doi-asserted-by":"crossref","unstructured":"Reddy, P. C. & Gurrala, K. K. Joint DR-DME grading classification using optimal feature selection-based deep graph correlation network. Appl. Soft Comput. 149, 110981 (2023).","DOI":"10.1016\/j.asoc.2023.110981"},{"key":"55337_CR5","doi-asserted-by":"crossref","unstructured":"Morales, S., Engan, K., Naranjo, V. & Colomer, A. Retinal disease screening through local binary patterns. IEEE J. Biomed. Health Inform. 21(1), 184\u2013192 (2017).","DOI":"10.1109\/JBHI.2015.2490798"},{"key":"55337_CR6","doi-asserted-by":"crossref","unstructured":"Sch\u00fcrer-Waldheim, S. Robust fovea detection in retinal OCT imaging using deep learning. IEEE J. Biomed. Health Inform. 26(8), 3927\u20133937 (2022).","DOI":"10.1109\/JBHI.2022.3166068"},{"key":"55337_CR7","doi-asserted-by":"crossref","unstructured":"Shao, H. .-C. Retina-TransNet: A gradient-guided few-shot retinal vessel segmentation net. IEEE J. Biomed. Health Inform. 27(10), 4902\u20134913 (2023).","DOI":"10.1109\/JBHI.2023.3298710"},{"key":"55337_CR8","doi-asserted-by":"crossref","unstructured":"Xu, R., Liu, T., Ye, X., Liu, F., Lin, L., Li, L., Tanaka, S. & Chen, Y. W., Joint extraction of retinal vessels and centerlines based on deep semantics and multi-scaled cross-task aggregation. IEEE J. Biomed. Health Inform. 25(7), 2722\u20132732 (2021).","DOI":"10.1109\/JBHI.2020.3044957"},{"key":"55337_CR9","doi-asserted-by":"crossref","unstructured":"Hassan, T., Akram, M. U., Werghi, N. & Nazir, N. RAG-FW: A hybrid convolutional framework for the automated extraction of retinal lesions and lesion-influenced grading of human retinal pathology. IEEE J. Biomed. Health Inform. 25(1), 108\u2013120 (2020).","DOI":"10.1109\/JBHI.2020.2982914"},{"key":"55337_CR10","doi-asserted-by":"crossref","unstructured":"Hassan, T. Li, Z., Akram, M. U., Hussain, I., Khalaf, K., & Werghi, N., Angular contrastive distillation driven self-supervised scanner independent screening and grading of retinopathy. Inf. Fus. 92, 404\u2013419 (2023).","DOI":"10.1016\/j.inffus.2022.12.006"},{"key":"55337_CR11","doi-asserted-by":"crossref","unstructured":"Helmy, Y.\u00a0M. & Allah, H. R.\u00a0A. Grading of age-related macular degeneration: Comparison between color fundus photography, fluorescein angiography, and spectral domain optical coherence tomography. J. Ophthalmol. 2013, 85915 (2013).","DOI":"10.1155\/2013\/385915"},{"key":"55337_CR12","unstructured":"Choudhry, N. & Sodhi, S.\u00a0K. Peripheral OCT imaging in practice. Retina Today (2021)."},{"key":"55337_CR13","doi-asserted-by":"crossref","unstructured":"Fang, L., Wang, C., Li, S., Rabbani, H., Chen, X. & Liu, Z., Attention to lesion: Lesion-aware convolutional neural network for retinal optical coherence tomography image classification. In IEEE Transactions on Medical Imaging (2019).","DOI":"10.1109\/TMI.2019.2898414"},{"key":"55337_CR14","unstructured":"Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G. & Sutskever, I., Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020 (2021)."},{"key":"55337_CR15","doi-asserted-by":"crossref","unstructured":"Hassan, T. Raja, H., Belwafi, K., Akcay, S., Jleli, M., Samet, B., Werghi, N., Yousaf, J. & Ghazal, M., A vision language correlation framework for screening disabled retina. IEEE J. Biomed. Health Inform. (2024).","DOI":"10.1109\/JBHI.2024.3462653"},{"key":"55337_CR16","doi-asserted-by":"crossref","unstructured":"Hassan, T., Akram, M.\u00a0U., Shaukat, A., Khawaja, S.\u00a0G. & Hassan, B. Structure Tensor Graph Searches Based Fully Automated Grading and 3D Profiling of Maculopathy From Retinal OCT Images. (IEEE Access, 2018).","DOI":"10.1109\/ACCESS.2018.2862626"},{"key":"55337_CR17","doi-asserted-by":"crossref","unstructured":"Chiu, S. J. Kernel regression based segmentation of optical coherence tomography images with diabetic macular edema. Biomed. Opt. Exp. (2015).","DOI":"10.1364\/BOE.6.001172"},{"key":"55337_CR18","doi-asserted-by":"crossref","unstructured":"Haider, A., Arsalan, M., Park, C., Sultan, H. & Park, K.\u00a0R. Exploring deep feature-blending capabilities to assist glaucoma screening. Appl. Soft Comput. (2023).","DOI":"10.1016\/j.asoc.2022.109918"},{"key":"55337_CR19","doi-asserted-by":"crossref","unstructured":"Fang, L. et al. Automatic segmentation of nine retinal layer boundaries in oct images of non-exudative and patients using deep learning and graph search. Biomed. Opt. Exp. 8(5) (2017).","DOI":"10.1364\/BOE.8.002732"},{"issue":"8","key":"55337_CR20","doi-asserted-by":"publisher","first-page":"4340","DOI":"10.1364\/BOE.10.004340","volume":"10","author":"P Zang","year":"2019","unstructured":"Zang, P. et al. Automated segmentation of peripapillary retinal boundaries in oct combining a convolutional neural network and a multi-weights graph search. Biomed. Opt. Exp. 10(8), 4340\u20134352 (2019).","journal-title":"Biomed. Opt. Exp."},{"key":"55337_CR21","doi-asserted-by":"crossref","unstructured":"Liu, W., Sun, Y. & Ji, Q. MDAN-UNet: Multi-scale and dual attention enhanced nested u-net architecture for segmentation of optical coherence tomography images. Algorithms (2020).","DOI":"10.3390\/a13030060"},{"key":"55337_CR22","doi-asserted-by":"crossref","unstructured":"Sun, Y., Zhang, H. & Yao, X. Automatic diagnosis of macular diseases from OCT volume based on its two-dimensional feature map and convolutional neural network with attention mechanism. J. Biomed. Opt. (2020).","DOI":"10.1117\/1.JBO.25.9.096004"},{"key":"55337_CR23","doi-asserted-by":"crossref","unstructured":"Li, J. et al. Multi-scale GCN-assisted two-stage network for joint segmentation of retinal layers and disc in peripapillary OCT images. Biomed. Opt. Exp. (2021).","DOI":"10.1364\/BOE.417212"},{"key":"55337_CR24","doi-asserted-by":"crossref","unstructured":"Butola, A. et al. Deep learning architecture \u201cLightOCT\u201d for diagnostic decision support using optical coherence tomography images of biological samples. Biomed. Opt. Exp. (2020).","DOI":"10.1364\/BOE.395487"},{"key":"55337_CR25","doi-asserted-by":"crossref","unstructured":"Das, V., Dandapat, S. & Bora, P.\u00a0K. Unsupervised Super-Resolution of OCT Images Using Generative Adversarial Network for Improved Age-Related Macular Degeneration Diagnosis. (IEEE Sensors, 2020).","DOI":"10.1109\/JSEN.2020.2985131"},{"key":"55337_CR26","doi-asserted-by":"crossref","unstructured":"Wang, J. Weakly supervised anomaly segmentation in retinal OCT images using an adversarial learning approach. Biomed. Opt. Exp. (2021).","DOI":"10.1364\/BOE.426803"},{"key":"55337_CR27","doi-asserted-by":"crossref","unstructured":"Ma, X. et al. MS-CAM: Multi-scale class activation maps for weakly-supervised segmentation of geographic atrophy lesions in SD-OCT images. IEEE J. Biomed. Health Inform. 24(12) (2021).","DOI":"10.1109\/JBHI.2020.2999588"},{"key":"55337_CR28","doi-asserted-by":"crossref","unstructured":"Lopez-Varela, E. Evolutionary multi-target neural network architectures for flow void analysis in optical coherence tomography angiography. Appl. Soft Comput. (2024).","DOI":"10.1016\/j.asoc.2024.111304"},{"key":"55337_CR29","unstructured":"Kermany, D. Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning (Elsevier Cell, 2018)."},{"key":"55337_CR30","doi-asserted-by":"crossref","unstructured":"Huang, A. S., Hirabayashi, K. & Barna, L. Assessment of a large language model\u2019s responses to questions and cases about glaucoma and retina management. JAMA Ophthalmol. (2024).","DOI":"10.1001\/jamaophthalmol.2023.6917"},{"key":"55337_CR31","unstructured":"ChatGPT 4.0. https:\/\/chat.openai.com (2024)."},{"key":"55337_CR32","doi-asserted-by":"crossref","unstructured":"Kleinig, O. How to use large language models in ophthalmology: From prompt engineering to protecting confidentiality. Eye (2023).","DOI":"10.1038\/s41433-023-02772-w"},{"key":"55337_CR33","unstructured":"Microsoft. Bing AI. (2024)."},{"key":"55337_CR34","doi-asserted-by":"crossref","unstructured":"Ong, J. Applying large language model artificial intelligence for retina International Classification of Diseases (ICD) coding. JMAI (2023).","DOI":"10.21037\/jmai-23-106"},{"key":"55337_CR35","unstructured":"ChatGPT 3.5. https:\/\/chat.openai.com (2024)."},{"key":"55337_CR36","doi-asserted-by":"crossref","unstructured":"Kianian, R., Sun, D., Crowell, E.\u00a0L. & Tsui, E. The use of large language models to generate education materials about uveitis. Ophthalmol. Retina (2024).","DOI":"10.1016\/j.oret.2023.09.008"},{"key":"55337_CR37","unstructured":"Google. Google Bard (Gemini). https:\/\/gemini.google.com\/?hl=en (2024)."},{"key":"55337_CR38","unstructured":"Desideri, L. F. Application and accuracy of artificial intelligence-derived large language models in patients with age related macular degeneration. Int. J. Retina Vitreous (2023)."},{"key":"55337_CR39","doi-asserted-by":"crossref","unstructured":"Jin, K., Yuan, L., Wu, H., Grzybowski, A. & Ye, J. Exploring large language model for next generation of artificial intelligence in ophthalmology. Front. Med. (2023).","DOI":"10.3389\/fmed.2023.1291404"},{"key":"55337_CR40","unstructured":"Dosovitskiy, A. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR, 2021)."},{"key":"55337_CR41","doi-asserted-by":"crossref","unstructured":"Howard, A. Searching for MobileNetV3. (ICCV, 2019).","DOI":"10.1109\/ICCV.2019.00140"},{"key":"55337_CR42","unstructured":"Hendrycks, D. & Gimpel, K. Gaussian Error Linear Units (GELUs). arXiv: 1606.08415 (2023)."},{"key":"55337_CR43","first-page":"12077","volume":"34","author":"E Xie","year":"2021","unstructured":"Xie, E. et al. SegFormer: Simple and efficient design for semantic segmentation with transformers. Adv. Neural Inf. Process. Syst. 34, 12077\u201312090 (2021).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"55337_CR44","doi-asserted-by":"crossref","unstructured":"Zheng, S. Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers (CVPR, 2021).","DOI":"10.1109\/CVPR46437.2021.00681"},{"key":"55337_CR45","doi-asserted-by":"crossref","unstructured":"Hassan, T., Akram, M.\u00a0U., Masood, M.\u00a0F. & Yasin, U. BIOMISA retinal image database for macular and ocular syndromes. In International Conference on Image Analysis and Recognition (ICIAR) (2018).","DOI":"10.1007\/978-3-319-93000-8_79"},{"key":"55337_CR46","doi-asserted-by":"crossref","unstructured":"Farsiu, S. Quantitative classification of eyes with and without intermediate age-related macular degeneration using optical coherence tomography. Ophthalmology (2014).","DOI":"10.1016\/j.ophtha.2013.07.013"},{"key":"55337_CR47","doi-asserted-by":"crossref","unstructured":"Srinivasan, P. P. et al. Fully automated detection of diabetic macular edema and dry age-related macular degeneration from optical coherence tomography images. Biomed. Opt. Exp. (2014).","DOI":"10.1364\/BOE.5.003568"},{"issue":"4","key":"55337_CR48","doi-asserted-by":"publisher","first-page":"1024","DOI":"10.1109\/TMI.2017.2780115","volume":"37","author":"R Rasti","year":"2018","unstructured":"Rasti, R., Rabbani, H., Mehri, A. & Hajizadeh, F. Macular OCT classification using a multi-scale convolutional neural network ensemble. IEEE Trans. Med. Imaging 37(4), 1024\u20131034 (2018).","journal-title":"IEEE Trans. Med. Imaging"},{"key":"55337_CR49","volume-title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","author":"J Devlin","year":"2019","unstructured":"Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (ACL Human Language Technologies, 2019)."},{"key":"55337_CR50","unstructured":"Mikolov, T., Chen, K., Corrado, G. & Dean, J. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781 (2013)."},{"key":"55337_CR51","unstructured":"Yang, Z. et al. XLNet: Generalized autoregressive pretraining for language understanding. In Neural Information Processing Systems (NeurIPS) (2019)."},{"key":"55337_CR52","unstructured":"Poslavskaya, E. & Korolev, A. Encoding categorical data: Is there yet anything \u2019hotter\u2019 than one-hot encoding? arXiv:2312.16930 (2023)."},{"key":"55337_CR53","doi-asserted-by":"crossref","unstructured":"Philippi, D., Rothaus, K. & Castelli, M. A vision transformer architecture for the automated segmentation of retinal lesions in spectral domain optical coherence tomography images. Sci. Rep. 13(517), 1\u201314(2023).","DOI":"10.1038\/s41598-023-27616-1"},{"key":"55337_CR54","doi-asserted-by":"crossref","unstructured":"Jiang, Q., Fan, Y., Li, M., Fang, S., Zhu, W., Xiang, D., Peng, T., Chen, X., Xu, X. and Shi. F. HyFormer: A hybrid transformer-CNN architecture for retinal OCT image segmentation. Biomed. Opt. Exp. 15(11), 6156\u20136170 (2024).","DOI":"10.1364\/BOE.538959"},{"key":"55337_CR55","doi-asserted-by":"crossref","unstructured":"Hao, J., Li, H., Lu, S., Li, Z. & Zhang, W. General retinal layer segmentation in OCT images via reinforcement constraint. In Computerized Medical Imaging and Graphics (2025).","DOI":"10.1016\/j.compmedimag.2024.102480"},{"key":"55337_CR56","doi-asserted-by":"crossref","unstructured":"Sarki, R., Ahmed, K., Wang, H. & Zhang, Y. Automatic Detection of Diabetic Eye Disease Through Deep Learning Using Fundus Images: A Survey. (IEEE Access, 2020).","DOI":"10.1109\/ACCESS.2020.3015258"}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-55337-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-55337-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-55337-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,24]],"date-time":"2026-08-24T23:03:22Z","timestamp":1787612602000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-55337-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,11]]},"references-count":56,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["55337"],"URL":"https:\/\/doi.org\/10.1038\/s41598-026-55337-8","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,11]]},"assertion":[{"value":"26 December 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 May 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 June 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","label":"Competing interests","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"26561"}}