{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T01:24:29Z","timestamp":1792027469599,"version":"4.4.0"},"reference-count":46,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Large language models have shown remarkable efficacy in various medical research and clinical applications. However, their skills in medical image recognition and subsequent report generation or question answering (QA) remain limited.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>We aim to finetune a multimodal, transformer-based model for generating medical reports from slit lamp images and develop a QA system using Llama2. We term this entire process slit lamp\u2013GPT.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Our research used a dataset of 25,051 slit lamp images from 3409 participants, paired with their corresponding physician-created medical reports. We used these data, split into training, validation, and test sets, to finetune the Bootstrapping Language-Image Pre-training framework toward report generation. The generated text reports and human-posed questions were then input into Llama2 for subsequent QA. We evaluated performance using qualitative metrics (including BLEU [bilingual evaluation understudy], CIDEr [consensus-based image description evaluation], ROUGE-L [Recall-Oriented Understudy for Gisting Evaluation\u2014Longest Common Subsequence], SPICE [Semantic Propositional Image Caption Evaluation], accuracy, sensitivity, specificity, precision, and F1-score) and the subjective assessments of two experienced ophthalmologists on a 1-3 scale (1 referring to high quality).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We identified 50 conditions related to diseases or postoperative complications through keyword matching in initial reports. The refined slit lamp\u2013GPT model demonstrated BLEU scores (1-4) of 0.67, 0.66, 0.65, and 0.65, respectively, with a CIDEr score of 3.24, a ROUGE (Recall-Oriented Understudy for Gisting Evaluation) score of 0.61, and a Semantic Propositional Image Caption Evaluation score of 0.37. The most frequently identified conditions were cataracts (22.95%), age-related cataracts (22.03%), and conjunctival concretion (13.13%). Disease classification metrics demonstrated an overall accuracy of 0.82 and an F1-score of 0.64, with high accuracies (\u22650.9) observed for intraocular lens, conjunctivitis, and chronic conjunctivitis, and high F1-scores (\u22650.9) observed for cataract and age-related cataract. For both report generation and QA components, the two evaluating ophthalmologists reached substantial agreement, with \u03ba scores between 0.71 and 0.84. In assessing 100 generated reports, they awarded scores of 1.36 for both completeness and correctness; 64% (64\/100) were considered \u201centirely good,\u201d and 93% (93\/100) were \u201cacceptable.\u201d In the evaluation of 300 generated answers to questions, the scores were 1.33 for completeness, 1.14 for correctness, and 1.15 for possible harm, with 66.3% (199\/300) rated as \u201centirely good\u201d and 91.3% (274\/300) as \u201cacceptable.\u201d<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>This study introduces the slit lamp\u2013GPT model for report generation and subsequent QA, highlighting the potential of large language models to assist ophthalmologists and patients.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/54047","type":"journal-article","created":{"date-parts":[[2024,12,30]],"date-time":"2024-12-30T16:30:28Z","timestamp":1735576228000},"page":"e54047","source":"Crossref","is-referenced-by-count":11,"title":["Slit Lamp Report Generation and Question Answering: Development and Validation of a Multimodal Transformer Model with Large Language Model Integration"],"prefix":"10.2196","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-4551-348X","authenticated-orcid":false,"given":"Ziwei","family":"Zhao","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-2780-9121","authenticated-orcid":false,"given":"Weiyi","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1581-5045","authenticated-orcid":false,"given":"Xiaolan","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8134-2402","authenticated-orcid":false,"given":"Fan","family":"Song","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5517-986X","authenticated-orcid":false,"given":"James","family":"Gunasegaram","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4894-0937","authenticated-orcid":false,"given":"Wenyong","family":"Huang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6094-137X","authenticated-orcid":false,"given":"Danli","family":"Shi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6912-2810","authenticated-orcid":false,"given":"Mingguang","family":"He","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-4492-229X","authenticated-orcid":false,"given":"Na","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2024,12,30]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1136\/bjo.32.4.232"},{"issue":"110","key":"ref2","first-page":"48","volume":"33","author":"Magyezi, J","year":"2020","journal-title":"Community Eye Health"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-023-30521-2"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1002\/jmri.26878"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejrad.2021.109717"},{"key":"ref6","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-020-0842-3"},{"key":"ref7","unstructured":"Introducing ChatGPT2024-10-19https:\/\/openai.com\/blog\/chatgpt"},{"key":"ref8","first-page":"1","author":"Touvron, H","year":"2023","journal-title":"arXiv:2307.09288"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.1016\/S2589-7500(23)00021-3"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-023-02448-8"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.1093\/pubmed\/fdad028"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1016\/j.isci.2024.110021"},{"key":"ref13","first-page":"1","author":"Dai, H","year":"2023","journal-title":"arXiv:2302.13007"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-020-79470-0"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.1007\/s00330-021-08334-6"},{"key":"ref16","doi-asserted-by":"crossref","unstructured":"LeeMHSiewiorekDPSmailagicABernardinoABadiaSBBIA human-AI collaborative approach for clinical decision making on rehabilitation assessment2021Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems2021 May 07Yokohama, Japan114","DOI":"10.1145\/3411764.3445472"},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.223312"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.1080\/0142159X.2019.1595557"},{"key":"ref19","first-page":"1","author":"Singhal, K","year":"2023","journal-title":"arXiv:2305.09617"},{"key":"ref20","unstructured":"KrizhevskyASutskeverIHintonGEImageNet classification with deep convolutional neural networksNeurIPS Proceedings201226th Annual Conference on Neural Information Processing Systems 20122012-12-3Lake Tahoe, Nevada, United States19"},{"key":"ref21","unstructured":"VaswaniAShazeerNParmarNUszkoreitJJonesLGomezANKaiserLPolosukhinIAttention is all you needNeurIPS Proceedings2017the 31st International Conference on Neural Information Processing Systems2017 December 04Red Hook, NY, United States60006010"},{"key":"ref22","doi-asserted-by":"publisher","DOI":"10.1016\/j.xops.2022.100147"},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.1097\/j.jcrs.0000000000000790"},{"key":"ref24","doi-asserted-by":"publisher","DOI":"10.1136\/bjophthalmol-2021-318866"},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-023-49635-8"},{"key":"ref26","first-page":"1","author":"Wu, C","year":"2023","journal-title":"arXiv:2310.09909"},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.1136\/bjo-2023-325054"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01101-z"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.1136\/bjo-2023-324446"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.1167\/iovs.14-15943"},{"key":"ref31","unstructured":"LiJLiDXiongCHoiSBlip: bootstrapping language-image pre-training for unified vision-language understanding and generation2022Proceedings of the 39th International Conference on Machine Learning2022 January 28Baltimore, Maryland, USA"},{"key":"ref32","first-page":"1","author":"Devlin, J","year":"2018","journal-title":"arXiv:1810.04805. Presented October 11, 2018"},{"key":"ref33","doi-asserted-by":"publisher","DOI":"10.1016\/j.oret.2023.05.022"},{"key":"ref34","doi-asserted-by":"crossref","unstructured":"PapineniKRoukosSWardTZhuWJBLEU: a method for automatic evaluation of machine translation2002Proceedings of the 40th Annual Meeting on Association for Computational Linguistics2002 July 06United States","DOI":"10.3115\/1073083.1073135"},{"key":"ref35","doi-asserted-by":"crossref","unstructured":"VedantamRLawrenceZCParikhDCIDEr: Consensus-based image description evaluation2015IEEE Conference on Computer Vision and Pattern Recognition (CVPR)2015 June 07-12Boston, MA, USA","DOI":"10.1109\/CVPR.2015.7299087"},{"key":"ref36","unstructured":"LinCYRouge: A package for automatic evaluation summaries2004Proceedings of the Workshop on Text Summarization Branches Out2004 July 25\u201326Barcelona, Spain7481"},{"key":"ref37","unstructured":"AndersonPFernandoBJohnsonMGouldSSPICE: semantic propositional image caption evaluation201614th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V2016 October 11-14Amsterdam, The Netherlands"},{"key":"ref38","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198"},{"key":"ref39","unstructured":"ThirunavukarasuAJChatGPT cannot pass FRCOphth examinations: implications for ophthalmology and large language model artificial intelligenceEye News20232024-10-24https:\/\/www.eyenews.uk.com\/media\/31505\/eye-am23-onex-arun-proof-2.pdf"},{"key":"ref40","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2013.629"},{"issue":"8","key":"ref41","first-page":"1031","volume":"62","author":"Shimizu, E","year":"2021","journal-title":"Invest Ophthalmol Vis Sci"},{"key":"ref42","doi-asserted-by":"publisher","DOI":"10.1097\/01.APO.0000656988.16221.04"},{"key":"ref43","doi-asserted-by":"publisher","DOI":"10.1177\/01410768231173123"},{"key":"ref44","doi-asserted-by":"publisher","DOI":"10.2196\/60063"},{"key":"ref45","doi-asserted-by":"publisher","DOI":"10.1145\/364128"},{"key":"ref46","first-page":"1","author":"Chen, X","year":"2024","journal-title":"arXiv:2405.07468"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2024\/1\/e54047","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:40:25Z","timestamp":1782841225000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2024\/1\/e54047"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,30]]},"references-count":46,"URL":"https:\/\/doi.org\/10.2196\/54047","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.54047","asserted-by":"object"}]},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,30]]}}}