{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"bioRxiv"}],"indexed":{"date-parts":[[2026,1,16]],"date-time":"2026-01-16T01:32:47Z","timestamp":1768527167720,"version":"3.49.0"},"posted":{"date-parts":[[2025,8,11]]},"group-title":"Bioengineering","reference-count":47,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2025,8,11]],"date-time":"2025-08-11T00:00:00Z","timestamp":1754870400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2025,8,11]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:p>Translating the intricate anatomical signatures of retinal disease from OCT B-scans into clear, accurate clinical narratives demands AI models that seamlessly fuse visual features with domain expertise. We curated a multimodal dataset of 40,000 OCT B-scans from public repositories and private clinical cohorts, each paired with expert validated summaries spanning six conditions: diabetic macular edema, diabetic retinopathy, geographic atrophy, drusen, choroidal neovascularization, and healthy retina. We introduce LO-VLM, a compact (247M parameter) vision\u2013language model (VLM) that infuses anatomical guidance into both encoder and decoder for free form summary generation and multiclass disease classification. Benchmarking against state-of-the-art RetinaVLM, LLaVA-Med, and a ViT vision only model demonstrates superior performance. In a blinded evaluation by three board certified retina specialists scored the generated summaries, LO-VLM narratives achieved mean = 8.5 (standard deviation = 1.15) out of 10, compared to mean = 5.5 (standard deviation = 1.13) for RetinaVLM (p &lt; 0.0001). In quantitative evaluations, LO-VLM achieved an SBERT similarity of 0.803 and a BERTScore F1 of 0.715, representing improvements of 8.2% and 28.8% over specialized VLM baselines. For disease classification, LO-VLM reached 96% accuracy (F1 = 96%), outperforming ViT by 13% and exceeding medical VLM benchmarks by over 62%. By reconciling interpretability with computational efficiency, LO-VLM establishes a new paradigm for efficient AI models in OCT interpretation.<\/jats:p>","DOI":"10.1101\/2025.08.07.669187","type":"posted-content","created":{"date-parts":[[2025,8,11]],"date-time":"2025-08-11T21:30:27Z","timestamp":1754947827000},"source":"Crossref","is-referenced-by-count":0,"title":["Compact Vision\u2013Language Models Enable Efficient and Interpretable Automated OCT Analysis Through Layer Specific Multimodal Learning"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-0309-3407","authenticated-orcid":false,"given":"Tania","family":"Haghighi","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sina","family":"Gholami","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jared Todd","family":"Sokol","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jennifer I.","family":"Lim","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Theodore","family":"Leng","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Atalie C.","family":"Thompson","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hamed","family":"Tabkhi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Minhaj Nur","family":"Alam","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"54368","reference":[{"issue":"1","key":"2025081319050573000_2025.08.07.669187v1.1","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1038\/s44172-025-00341-5","article-title":"Distributed training of foundation models for ophthalmic diagnosis","volume":"4","year":"2025","journal-title":"Communications Engineering"},{"key":"2025081319050573000_2025.08.07.669187v1.2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06555-x"},{"key":"2025081319050573000_2025.08.07.669187v1.3","article-title":"Visionunite: A vision-language foundation model for ophthalmology enhanced with clinical knowledge","year":"2024","journal-title":"arXiv preprint"},{"key":"2025081319050573000_2025.08.07.669187v1.4","doi-asserted-by":"crossref","unstructured":"Chengliang Wang , Xinrun Chen , Haojian Ning , and Shiying Li . Sam-octa: A fine-tuning strategy for applying foundation model octa image segmentation tasks. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1771\u20131775. IEEE, 2024.","DOI":"10.1109\/ICASSP48485.2024.10446904"},{"key":"2025081319050573000_2025.08.07.669187v1.5","doi-asserted-by":"publisher","DOI":"10.1126\/science.1957169"},{"key":"2025081319050573000_2025.08.07.669187v1.6","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2014.07.054"},{"key":"2025081319050573000_2025.08.07.669187v1.7","doi-asserted-by":"publisher","DOI":"10.1016\/j.preteyeres.2007.07.005"},{"key":"2025081319050573000_2025.08.07.669187v1.8","doi-asserted-by":"publisher","DOI":"10.1016\/S0161-6420(96)30512-5"},{"issue":"6","key":"2025081319050573000_2025.08.07.669187v1.9","doi-asserted-by":"crossref","first-page":"3453","DOI":"10.3390\/app13063453","article-title":"Vision transformer approach for classification of alzheimer\u2019s disease using 18f-florbetaben brain images","volume":"13","year":"2023","journal-title":"Applied Sciences"},{"key":"2025081319050573000_2025.08.07.669187v1.10","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-018-0107-6"},{"key":"2025081319050573000_2025.08.07.669187v1.11","doi-asserted-by":"publisher","DOI":"10.1016\/j.cell.2018.02.010"},{"key":"2025081319050573000_2025.08.07.669187v1.12","doi-asserted-by":"crossref","unstructured":"Jahida Subhedar and Anurag Mahajan . A review on recent work on oct image classification for disease detection. In 2022 OPJU International Technology Conference on Emerging Technologies for Sustainable Development (OTCON), pages 1\u20136. IEEE, 2023.","DOI":"10.1109\/OTCON56053.2023.10114003"},{"key":"2025081319050573000_2025.08.07.669187v1.13","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2017.10.031"},{"issue":"4","key":"2025081319050573000_2025.08.07.669187v1.14","doi-asserted-by":"crossref","first-page":"2318","DOI":"10.1167\/iovs.16-20541","article-title":"Automated staging of age-related macular degeneration using optical coherence tomography","volume":"58","year":"2017","journal-title":"Investigative ophthalmology & visual science"},{"issue":"8","key":"2025081319050573000_2025.08.07.669187v1.15","doi-asserted-by":"crossref","first-page":"3627","DOI":"10.1364\/BOE.8.003627","article-title":"Relaynet: retinal layer and fluid segmentation of macular optical coherence tomography using fully convolutional networks","volume":"8","year":"2017","journal-title":"Biomedical optics express"},{"issue":"2","key":"2025081319050573000_2025.08.07.669187v1.16","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1167\/tvst.9.2.61","article-title":"Deepretina: layer segmentation of retina in oct images using deep learning","volume":"9","year":"2020","journal-title":"Translational vision science & technology"},{"key":"2025081319050573000_2025.08.07.669187v1.17","doi-asserted-by":"crossref","unstructured":"Stefanos Apostolopoulos , Sandro De Zanet , Carlos Ciller , Sebastian Wolf , and Raphael Sznitman . Pathological oct retinal layer segmentation using branch residual u-shape networks. In International conference on medical image computing and Computer-Assisted intervention, pages 294\u2013301. Springer, 2017.","DOI":"10.1007\/978-3-319-66179-7_34"},{"key":"2025081319050573000_2025.08.07.669187v1.18","article-title":"Topology guaranteed segmentation of the human retina from oct using convolutional neural networks","year":"2018","journal-title":"arXiv preprint"},{"key":"2025081319050573000_2025.08.07.669187v1.19","article-title":"On the automatic generation of medical imaging reports","year":"2017","journal-title":"arXiv preprint"},{"key":"2025081319050573000_2025.08.07.669187v1.20","unstructured":"Mingjie Li , Wenjia Cai , Karin Verspoor , Shirui Pan , Xiaodan Liang , and Xiaojun Chang . Cross-modal clinical graph transformer for ophthalmic report generation. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pages 20656\u201320665, 2022."},{"key":"2025081319050573000_2025.08.07.669187v1.21","doi-asserted-by":"crossref","unstructured":"Jia-Hong Huang , C-H Huck Yang , Fangyu Liu , Meng Tian , Yi-Chieh Liu , Ting-Wei Wu , I Lin, Kang Wang , Hiromasa Morikawa , Hernghua Chang , et al. Deepopht: medical report generation for retinal images via deep models and visual explanation. In Proceedings of the IEEE\/CVF winter conference on applications of computer vision, pages 2442\u20132452, 2021.","DOI":"10.1109\/WACV48630.2021.00249"},{"key":"2025081319050573000_2025.08.07.669187v1.22","article-title":"Medblip: Fine-tuning blip for medical image captioning","year":"2025","journal-title":"arXiv preprint"},{"key":"2025081319050573000_2025.08.07.669187v1.23","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-03302-1"},{"key":"2025081319050573000_2025.08.07.669187v1.24","article-title":"Specialized curricula for training vision-language models in retinal image analysis","year":"2024","journal-title":"arXiv preprint"},{"key":"2025081319050573000_2025.08.07.669187v1.25","article-title":"Paligemma 2: A family of versatile vlms for transfer","year":"2024","journal-title":"arXiv preprint"},{"key":"2025081319050573000_2025.08.07.669187v1.26","article-title":"Sentence-bert: Sentence embeddings using siamese bert-networks","year":"2019","journal-title":"arXiv preprint"},{"key":"2025081319050573000_2025.08.07.669187v1.27","unstructured":"Abhinand Chakraborty . Medembed: Large biomedical embeddings for semantic search. https:\/\/huggingface.co\/abhinand\/MedEmbed-large-v0.1, 2024. Accessed: 2025-07-17."},{"key":"2025081319050573000_2025.08.07.669187v1.28","article-title":"Bertscore: Evaluating text generation with bert","year":"2019","journal-title":"arXiv preprint"},{"key":"2025081319050573000_2025.08.07.669187v1.29","doi-asserted-by":"crossref","unstructured":"Kishore Papineni , Salim Roukos , Todd Ward , and Wei-Jing Zhu . Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311\u2013318, 2002.","DOI":"10.3115\/1073083.1073135"},{"key":"2025081319050573000_2025.08.07.669187v1.30","doi-asserted-by":"crossref","unstructured":"Boxing Chen and Colin Cherry . A systematic comparison of smoothing techniques for sentence-level bleu. In Proceedings of the ninth workshop on statistical machine translation, pages 362\u2013367, 2014.","DOI":"10.3115\/v1\/W14-3346"},{"key":"2025081319050573000_2025.08.07.669187v1.31","unstructured":"Chin-Yew Lin . Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74\u201381, 2004."},{"key":"2025081319050573000_2025.08.07.669187v1.32","doi-asserted-by":"crossref","unstructured":"Ramakrishna Vedantam , C Lawrence Zitnick , and Devi Parikh . Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566\u20134575, 2015.","DOI":"10.1109\/CVPR.2015.7299087"},{"key":"2025081319050573000_2025.08.07.669187v1.33","doi-asserted-by":"crossref","unstructured":"Tania Haghighi , Sina Gholami , Jared Todd Sokol , Enaika Kishnani , Adnan Ahsaniyan , Holakou Rahmanian , Fares Hedayati , Theodore Leng , and Minhaj Nur Alam . Eye-llama, an in-domain large language model for ophthalmology. bioRxiv, 2024.","DOI":"10.2139\/ssrn.4850454"},{"key":"2025081319050573000_2025.08.07.669187v1.34","unstructured":"Kaiming He , Xinlei Chen , Saining Xie , Yanghao Li , Piotr Doll\u00e1r , and Ross Girshick . Masked autoencoders are scalable vision learners. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pages 16000\u201316009, 2022."},{"key":"2025081319050573000_2025.08.07.669187v1.35","unstructured":"Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . Bert: Pre-training of deep bidi-rectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171\u20134186, 2019."},{"issue":"2","key":"2025081319050573000_2025.08.07.669187v1.36","first-page":"3","article-title":"Lora: Low-rank adaptation of large language models","volume":"1","year":"2022","journal-title":"ICLR"},{"key":"2025081319050573000_2025.08.07.669187v1.37","doi-asserted-by":"crossref","first-page":"103357","DOI":"10.1016\/j.media.2024.103357","article-title":"A foundation languageimage model of the retina (flair): Encoding expert knowledge in text supervision","volume":"99","year":"2025","journal-title":"Medical Image Analysis"},{"key":"2025081319050573000_2025.08.07.669187v1.38","article-title":"Eyeclip: A visual-language foundation model for multi-modal ophthalmic image analysis","year":"2024","journal-title":"arXiv preprint"},{"key":"2025081319050573000_2025.08.07.669187v1.39","article-title":"Mm-retinal v2: Transfer an elite knowledge spark into fundus vision-language pretraining","year":"2025","journal-title":"arXiv preprint"},{"key":"2025081319050573000_2025.08.07.669187v1.40","doi-asserted-by":"crossref","unstructured":"Teja Krishna Cherukuri , Nagur Shareef Shaik , Jyostna Devi Bodapati , and Dong Hye Ye . Gcs-m3vlt: guided context self-attention based multi-modal medical vision language transformer for retinal image captioning. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1\u20135. IEEE, 2025.","DOI":"10.1109\/ICASSP49660.2025.10890310"},{"key":"2025081319050573000_2025.08.07.669187v1.41","first-page":"28541","article-title":"Llava-med: Training a large language-and-vision assistant for biomedicine in one day","volume":"36","year":"2023","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2025081319050573000_2025.08.07.669187v1.42","doi-asserted-by":"crossref","first-page":"106532","DOI":"10.1016\/j.compeleceng.2019.106532","article-title":"Octid: Optical coherence tomography image database","volume":"81","year":"2020","journal-title":"Computers & Electrical Engineering"},{"issue":"10","key":"2025081319050573000_2025.08.07.669187v1.43","doi-asserted-by":"crossref","first-page":"3568","DOI":"10.1364\/BOE.5.003568","article-title":"Fully automated detection of diabetic macular edema and dry age-related macular degeneration from optical coherence tomography images","volume":"5","year":"2014","journal-title":"Biomedical optics express"},{"issue":"1","key":"2025081319050573000_2025.08.07.669187v1.44","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1038\/s41597-024-03182-7","article-title":"Octdl: Optical coherence tomography dataset for image-based deep learning methods","volume":"11","year":"2024","journal-title":"Scientific data"},{"key":"2025081319050573000_2025.08.07.669187v1.45","unstructured":"OpenAI. Gpt-4o system card, August 2024. Accessed: 2025-07-17."},{"issue":"1","key":"2025081319050573000_2025.08.07.669187v1.46","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1007\/s10103-024-04089-w","article-title":"Automated classification of choroidal neovascularization, diabetic macular edema, and drusen from retinal oct images using vision transformers: a comparative study","volume":"39","year":"2024","journal-title":"Lasers in Medical Science"},{"key":"2025081319050573000_2025.08.07.669187v1.47","article-title":"Deep inside convolutional networks: Visualising image classification models and saliency maps","year":"2013","journal-title":"arXiv preprint"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1101\/2025.08.07.669187","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T09:19:30Z","timestamp":1768468770000},"score":1,"resource":{"primary":{"URL":"http:\/\/biorxiv.org\/lookup\/doi\/10.1101\/2025.08.07.669187"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,11]]},"references-count":47,"URL":"https:\/\/doi.org\/10.1101\/2025.08.07.669187","relation":{},"subject":[],"published":{"date-parts":[[2025,8,11]]},"subtype":"preprint"}}