{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,25]],"date-time":"2026-07-25T19:51:09Z","timestamp":1785009069272,"version":"3.55.0"},"reference-count":35,"publisher":"BMJ","issue":"10","funder":[{"name":"Start-up Fund for RAPs under the Strategic Hiring Scheme","award":["P0048623"],"award-info":[{"award-number":["P0048623"]}]},{"name":"Global STEM Professorship Scheme from HKSAR","award":["P0046113"],"award-info":[{"award-number":["P0046113"]}]}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["Br J Ophthalmol"],"accepted":{"date-parts":[[2024,3,3]]},"published-print":{"date-parts":[[2024,10]]},"abstract":"<jats:sec><jats:title>Background<\/jats:title><jats:p>Indocyanine green angiography (ICGA) is vital for diagnosing chorioretinal diseases, but its interpretation and patient communication require extensive expertise and time-consuming efforts. We aim to develop a bilingual ICGA report generation and question-answering (QA) system.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>Our dataset comprised 213\u2009129 ICGA images from 2919 participants. The system comprised two stages: image\u2013text alignment for report generation by a multimodal transformer architecture, and large language model (LLM)-based QA with ICGA text reports and human-input questions. Performance was assessed using both qualitative metrics (including Bilingual Evaluation Understudy (BLEU), Consensus-based Image Description Evaluation (CIDEr), Recall-Oriented Understudy for Gisting Evaluation-Longest Common Subsequence (ROUGE-L), Semantic Propositional Image Caption Evaluation (SPICE), accuracy, sensitivity, specificity, precision and F1 score) and subjective evaluation by three experienced ophthalmologists using 5-point scales (5 refers to high quality).<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>We produced 8757 ICGA reports covering 39 disease-related conditions after bilingual translation (66.7% English, 33.3% Chinese). The ICGA-GPT model\u2019s report generation performance was evaluated with BLEU scores (1\u20134) of 0.48, 0.44, 0.40 and 0.37; CIDEr of 0.82; ROUGE of 0.41 and SPICE of 0.18. For disease-based metrics, the average specificity, accuracy, precision, sensitivity and F1 score were 0.98, 0.94, 0.70, 0.68 and 0.64, respectively. Assessing the quality of 50 images (100 reports), three ophthalmologists achieved substantial agreement (kappa=0.723 for completeness, kappa=0.738 for accuracy), yielding scores from 3.20 to 3.55. In an interactive QA scenario involving 100 generated answers, the ophthalmologists provided scores of 4.24, 4.22 and 4.10, displaying good consistency (kappa=0.779).<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>This pioneering study introduces the ICGA-GPT model for report generation and interactive QA for the first time, underscoring the potential of LLMs in assisting with automated ICGA image interpretation.<\/jats:p><\/jats:sec>","DOI":"10.1136\/bjo-2023-324446","type":"journal-article","created":{"date-parts":[[2024,3,20]],"date-time":"2024-03-20T17:20:19Z","timestamp":1710955219000},"page":"1450-1456","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":38,"title":["ICGA-GPT: report generation and question answering for indocyanine green angiography images"],"prefix":"10.1136","volume":"108","author":[{"given":"Xiaolan","family":"Chen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weiyi","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ziwei","family":"Zhao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pusheng","family":"Xu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0914-7864","authenticated-orcid":false,"given":"Yingfeng","family":"Zheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6094-137X","authenticated-orcid":false,"given":"Danli","family":"Shi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mingguang","family":"He","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2024,3,20]]},"reference":[{"key":"2024101503100683000_108.10.1450.1","doi-asserted-by":"crossref","DOI":"10.1038\/s41746-024-01018-7","article-title":"Translating color fundus photography to Indocyanine green angiography using deep-learning for age-related macular degeneration screening","volume":"7","author":"Chen","year":"2024","journal-title":"NPJ Digit Med"},{"key":"2024101503100683000_108.10.1450.2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajo.2011.01.043"},{"key":"2024101503100683000_108.10.1450.3","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1007\/s00417-019-04493-x","article-title":"Utility of a public-available artificial intelligence in diagnosis of Polypoidal Choroidal Vasculopathy","volume":"258","author":"Yang","year":"2020","journal-title":"Graefes Arch Clin Exp Ophthalmol"},{"key":"2024101503100683000_108.10.1450.4","doi-asserted-by":"crossref","first-page":"53","DOI":"10.2147\/OPTH.S385827","article-title":"Polypoidal Choroidal Vasculopathy: an update on diagnosis and treatment","volume":"17","author":"Sen","year":"2023","journal-title":"Clin Ophthalmol"},{"key":"2024101503100683000_108.10.1450.5","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-021-01614-0"},{"key":"2024101503100683000_108.10.1450.6","unstructured":"GPT-4V(Ision) system card. n.d. Available: https:\/\/openaicom\/research\/gpt-4v-system-card"},{"key":"2024101503100683000_108.10.1450.7","doi-asserted-by":"crossref","unstructured":"Tu T , Azizi S , Driess D , et al . Towards generalist BIOMEDICAL AI. NEJM AI 2024;1. doi:10.1056\/AIoa2300138","DOI":"10.1056\/AIoa2300138"},{"key":"2024101503100683000_108.10.1450.8","doi-asserted-by":"crossref","first-page":"1045","DOI":"10.1001\/jamaophthalmol.2023.4650","article-title":"Deep learning performance of Ultra-Widefield fundus imaging for screening retinal lesions in rural locales","volume":"141","author":"Cui","year":"2023","journal-title":"JAMA Ophthalmol"},{"key":"2024101503100683000_108.10.1450.9","doi-asserted-by":"crossref","unstructured":"Li M , Cai W , Verspoor K , et al . Cross-modal clinical graph transformer for ophthalmic report generation. 2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR); New Orleans, LA, USA.2022 doi:10.1109\/CVPR52688.2022.02000","DOI":"10.1109\/CVPR52688.2022.02000"},{"key":"2024101503100683000_108.10.1450.10","doi-asserted-by":"crossref","unstructured":"Chen Z , Shen Y , Song Y , et al . Cross-modal memory networks for radiology report generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1; Online.Stroudsburg, PA, USA, 2022 doi:10.18653\/v1\/2021.acl-long.459","DOI":"10.18653\/v1\/2021.acl-long.459"},{"key":"2024101503100683000_108.10.1450.11","unstructured":"Zhao WX , Zhou K , Li J , et al . A survey of large language models. 2023."},{"key":"2024101503100683000_108.10.1450.12","unstructured":"Touvron H , Martin L , Stone K , et al . Llama 2: open foundation and fine-tuned chat models. 2023."},{"key":"2024101503100683000_108.10.1450.13","doi-asserted-by":"crossref","unstructured":"Xu P , Chen X , Zhao Z , et al . Evaluation of a Digital Ophthalmologist App built by Gpt4-V(Ision). [Preprint] 2023-11-29. doi:10.1101\/2023.11.27.23299056","DOI":"10.1101\/2023.11.27.23299056"},{"key":"2024101503100683000_108.10.1450.14","doi-asserted-by":"crossref","DOI":"10.3389\/fcvm.2022.823436","article-title":"A deep learning system for fully automated retinal vessel measurement in high throughput image analysis","volume":"9","author":"Shi","year":"2022","journal-title":"Front Cardiovasc Med"},{"key":"2024101503100683000_108.10.1450.15","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1038\/bjc.2014.639","article-title":"Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement","volume":"112","author":"Collins","year":"2015","journal-title":"Br J Cancer"},{"key":"2024101503100683000_108.10.1450.16","unstructured":"Li J , Li D , Xiong C , et al . Bootstrapping language-image pre-training for unified vision-language understanding and generation. 2022."},{"key":"2024101503100683000_108.10.1450.17","unstructured":"Dosovitskiy A , Beyer L , Kolesnikov A , et al . An image is worth 16X16 words: transformers for image recognition at scale. 2021."},{"key":"2024101503100683000_108.10.1450.18","unstructured":"Devlin J , Chang M-W , Lee K , et al . BERT: pre-training of deep Bidirectional transformers for language understanding. 2019."},{"key":"2024101503100683000_108.10.1450.19","doi-asserted-by":"crossref","first-page":"862","DOI":"10.1016\/j.oret.2023.05.022","article-title":"Appropriateness and Readability of Chatgpt-4 generated responses for surgical treatment of retinal diseases","volume":"7","author":"Momenaei","year":"2023","journal-title":"Ophthalmol Retina"},{"key":"2024101503100683000_108.10.1450.20","doi-asserted-by":"crossref","unstructured":"Papineni K , Roukos S , Ward T , et al . BLEU: a method for automatic evaluation of machine translation. the 40th Annual Meeting; 2001 2001; Philadelphia, Pennsylvania: Association for Computational Linguistics, doi:10.3115\/1073083.1073135","DOI":"10.3115\/1073083.1073135"},{"key":"2024101503100683000_108.10.1450.21","doi-asserted-by":"crossref","unstructured":"Vedantam R , Zitnick CL , Parikh D . CIDEr: consensus-based image description evaluation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Boston, MA, USA.2015 doi:10.1109\/CVPR.2015.7299087","DOI":"10.1109\/CVPR.2015.7299087"},{"key":"2024101503100683000_108.10.1450.22","unstructured":"Lin C-Y . n.d. ROUGE: a package for automatic evaluation of summaries. Text Summarization Branches Out;2004:74\u201381."},{"key":"2024101503100683000_108.10.1450.23","doi-asserted-by":"crossref","unstructured":"Anderson P , Fernando B , Johnson M , et al . Semantic Propositional image Caption evaluation. 2016. doi:10.1007\/978-3-319-46454-1","DOI":"10.1007\/978-3-319-46454-1_24"},{"key":"2024101503100683000_108.10.1450.24","doi-asserted-by":"publisher","DOI":"10.1097\/JTO.0b013e318200f983"},{"key":"2024101503100683000_108.10.1450.25","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models Encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"2024101503100683000_108.10.1450.26","doi-asserted-by":"publisher","DOI":"10.1155\/2021\/9981202"},{"key":"2024101503100683000_108.10.1450.27","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2023.102802"},{"key":"2024101503100683000_108.10.1450.28","doi-asserted-by":"crossref","unstructured":"Wu T-W , Huang J-H , Lin J , et al . Expert-defined Keywords improve Interpretability of retinal image Captioning. 2023 IEEE\/CVF Winter Conference on Applications of Computer Vision (WACV); Waikoloa, HI, USA: IEEE, Waikoloa, HI, USA. doi:10.1109\/WACV56688.2023.00190","DOI":"10.1109\/WACV56688.2023.00190"},{"key":"2024101503100683000_108.10.1450.29","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2023.104281"},{"key":"2024101503100683000_108.10.1450.30","doi-asserted-by":"crossref","unstructured":"Mosbach M , Pimentel T , Ravfogel S , et al . Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation. Findings of the Association for Computational Linguistics; Toronto, Canada.Stroudsburg, PA, USA, 2023 doi:10.18653\/v1\/2023.findings-acl.779","DOI":"10.18653\/v1\/2023.findings-acl.779"},{"key":"2024101503100683000_108.10.1450.31","unstructured":"Si C , Gan Z , Yang Z , et al . Prompting GPT-3 to be reliable. 2023."},{"key":"2024101503100683000_108.10.1450.32","doi-asserted-by":"crossref","unstructured":"Wang S , Zhao Z , Ouyang X , et al . Chatcad: interactive computer-aided diagnosis on medical image using large language models. 2023.","DOI":"10.1038\/s44172-024-00271-8"},{"key":"2024101503100683000_108.10.1450.33","doi-asserted-by":"crossref","DOI":"10.3389\/frai.2023.1169595","article-title":"Chatgpt in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations","volume":"6","author":"Dave","year":"2023","journal-title":"Front Artif Intell"},{"key":"2024101503100683000_108.10.1450.34","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2023.13674","article-title":"Integration of artificial intelligence decision aids to reduce workload and enhance efficiency in thyroid nodule management","volume":"6","author":"Tong","year":"2023","journal-title":"JAMA Netw Open"},{"key":"2024101503100683000_108.10.1450.35","doi-asserted-by":"crossref","DOI":"10.1148\/radiol.222976","article-title":"Effect of human-AI interaction on detection of malignant lung nodules on chest radiographs","volume":"307","author":"Lee","year":"2023","journal-title":"Radiology"}],"container-title":["British Journal of Ophthalmology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bjo-2023-324446","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,15]],"date-time":"2024-10-15T10:12:23Z","timestamp":1728987143000},"score":1,"resource":{"primary":{"URL":"https:\/\/bjo.bmj.com\/lookup\/doi\/10.1136\/bjo-2023-324446"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,20]]},"references-count":35,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2024,9,20]]},"published-print":{"date-parts":[[2024,10]]}},"alternative-id":["10.1136\/bjo-2023-324446"],"URL":"https:\/\/doi.org\/10.1136\/bjo-2023-324446","relation":{},"ISSN":["0007-1161","1468-2079"],"issn-type":[{"value":"0007-1161","type":"print"},{"value":"1468-2079","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,20]]}}}