{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T05:36:28Z","timestamp":1780637788622,"version":"3.54.1"},"reference-count":197,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T00:00:00Z","timestamp":1731974400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:p>Medical vision-language models (VLMs) combine computer vision (CV) and natural language processing (NLP) to analyze visual and textual medical data. Our paper reviews recent advancements in developing VLMs specialized for healthcare, focusing on publicly available models designed for medical report generation and visual question answering (VQA). We provide background on NLP and CV, explaining how techniques from both fields are integrated into VLMs, with visual and language data often fused using Transformer-based architectures to enable effective learning from multimodal data. Key areas we address include the exploration of 18 public medical vision-language datasets, in-depth analyses of the architectures and pre-training strategies of 16 recent noteworthy medical VLMs, and comprehensive discussion on evaluation metrics for assessing VLMs' performance in medical report generation and VQA. We also highlight current challenges facing medical VLM development, including limited data availability, concerns with data privacy, and lack of proper evaluation metrics, among others, while also proposing future directions to address these obstacles. Overall, our review summarizes the recent progress in developing VLMs to harness multimodal medical data for improved healthcare applications.<\/jats:p>","DOI":"10.3389\/frai.2024.1430984","type":"journal-article","created":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T06:18:16Z","timestamp":1731997096000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":159,"title":["Vision-language models for medical report generation and visual question answering: a review"],"prefix":"10.3389","volume":"7","author":[{"given":"Iryna","family":"Hartsock","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ghulam","family":"Rasool","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2024,11,19]]},"reference":[{"key":"B1","article-title":"\u201cOverview of the VQA-Med task at ImageCLEF 2020: visual question answering and generation in the medical domain,\u201d","volume-title":"CLEF 2020 Working Notes, CEUR Workshop Proceedings","author":"Abacha","year":"2020"},{"key":"B2","article-title":"\u201cVQA-Med: overview of the medical visual question answering task at imageclef 2019,\u201d","volume-title":"Conference and Labs of the Evaluation Forum","author":"Abacha","year":"2019"},{"key":"B3","doi-asserted-by":"publisher","first-page":"1773","DOI":"10.1038\/s41591-022-01981-2","article-title":"Multimodal biomedical AI","volume":"28","author":"Acosta","year":"2022","journal-title":"Nat. Med"},{"key":"B4","doi-asserted-by":"publisher","first-page":"7433","DOI":"10.48550\/arXiv.2205.01138","article-title":"Transformers in time-series analysis: a tutorial","volume":"42","author":"Ahmed","year":"2023","journal-title":"Circ. Syst. Sign. Process"},{"key":"B5","doi-asserted-by":"publisher","first-page":"23716","DOI":"10.48550\/arXiv.2204.14198","article-title":"Flamingo: a visual language model for few-shot learning","volume":"35","author":"Alayrac","year":"2022","journal-title":"Adv. Neural Inform. Process. Syst"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2001.11190","article-title":"2018 robotic scene segmentation challenge","author":"Allan","year":"2020","journal-title":"arXiv Preprint arXiv:2001.11190"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1902.06426","article-title":"2017 robotic instrument segmentation challenge","author":"Allan","year":"2019","journal-title":"arXiv Preprint arXiv:1902.06426"},{"key":"B8","first-page":"2425","article-title":"\u201cVQA: visual question answering,\u201d","volume-title":"IEEE International Conference on Computer Vision (ICCV)","author":"Antol","year":"2015"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2308.12966","article-title":"Qwen-VL: a versatile vision-language model for understanding, localization, text reading, and beyond","author":"Bai","year":"","journal-title":"arXiv Preprint arXiv:2308.12966"},{"key":"B10","first-page":"397","article-title":"\u201cCAT-ViL: co-attention gated vision-language embedding for visual question localized-answering in robotic surgery,\u201d","volume-title":"Medical Image Computing and Computer Assisted Intervention\u2014MICCAI","author":"Bai","year":""},{"key":"B11","doi-asserted-by":"publisher","first-page":"80","DOI":"10.1016\/j.neucom.2018.05.080","article-title":"A survey on automatic image caption generation","volume":"311","author":"Bai","year":"2018","journal-title":"Neurocomputing"},{"key":"B12","doi-asserted-by":"publisher","first-page":"e188","DOI":"10.7861\/fhj.2021-0095","article-title":"Artificial intelligence in healthcare: transforming the practice of medicine","volume":"8","author":"Bajwa","year":"2021","journal-title":"Fut. Healthc. J"},{"key":"B13","doi-asserted-by":"crossref","DOI":"10.1017\/9781108955652","volume-title":"Deep Learning in Science","author":"Baldi","year":"2021"},{"key":"B14","first-page":"65","article-title":"\u201cMETEOR: an automatic metric for MT evaluation with improved correlation with human judgments,\u201d","author":"Banerjee","year":"2005","journal-title":"ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2301.04558","article-title":"Learning to exploit temporal structure for biomedical vision-language processing","author":"Bannur","year":"2023","journal-title":"arXiv Preprint arXiv:2301.04558"},{"key":"B16","doi-asserted-by":"publisher","first-page":"81656","DOI":"10.48550\/arXiv.2302.00220","article-title":"Efficient scopeformer: toward scalable and rich feature extraction for intracranial hemorrhage detection","volume":"11","author":"Barhoumi","year":"2023","journal-title":"IEEE Access"},{"key":"B17","doi-asserted-by":"publisher","first-page":"380","DOI":"10.3390\/bioengineering10030380","article-title":"Vision\u2014language model for visual question answering in medical imagery","volume":"10","author":"Bazi","year":"2023","journal-title":"Bioengineering"},{"key":"B18","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1142\/9789811215636_0027","article-title":"Clinical concept embeddings learned from massive sources of multimodal medical data","volume":"25","author":"Beam","year":"2020","journal-title":"Pacif. Symp. Biocomput"},{"key":"B19","doi-asserted-by":"crossref","first-page":"370","DOI":"10.18653\/v1\/W19-5039","article-title":"\u201cOverview of the MEDIQA 2019 shared task on textual inference, question entailment and question answering,\u201d","author":"Ben Abacha","year":"2019","journal-title":"BioNLP Workshop and Shared Task"},{"key":"B20","doi-asserted-by":"publisher","first-page":"1441","DOI":"10.1038\/s41597-022-01441-z","article-title":"Reflacx, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rays","volume":"9","author":"Bigolin Lanfredi","year":"2022","journal-title":"Sci. Data"},{"key":"B21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/978-3-031-20059-5","article-title":"Making the most of text semantics to improve biomedical vision\u2013language processing","volume":"5","author":"Boecking","year":"2022","journal-title":"Comput. Vis"},{"key":"B22","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1162\/tacl_a_00051","article-title":"Enriching word vectors with subword information","volume":"5","author":"Bojanowski","year":"2017","journal-title":"Trans. Assoc. Comput. Linguist"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2108.07258","article-title":"On the opportunities and risks of foundation models","author":"Bommasani","year":"2022","journal-title":"arXiv Preprint arXiv:2108.07258"},{"key":"B24","doi-asserted-by":"publisher","first-page":"1877","DOI":"10.48550\/arXiv.2005.14165","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv. Neural Inform. Process. Syst"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2401.15275","article-title":"Dynamic transformer architecture for continual learning of multimodal tasks","author":"Cai","year":"2024","journal-title":"arXiv Preprint arXiv:2401.15275"},{"key":"B26","first-page":"213","article-title":"\u201cEnd-to-end object detection with transformers,\u201d","volume-title":"European Conference on Computer Vision","author":"Carion","year":"2020"},{"key":"B27","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1007\/s11633-022-1369-5","article-title":"VLP: a survey on vision-language pre-training","volume":"20","author":"Chen","year":"2023","journal-title":"Machine Intell. Res"},{"key":"B28","doi-asserted-by":"publisher","first-page":"850","DOI":"10.1038\/s41591-024-02857-3","article-title":"Towards a general-purpose foundation model for computational pathology","volume":"30","author":"Chen","year":"2024","journal-title":"Nat. Med"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2002.05709","article-title":"A simple framework for contrastive learning of visual representations","author":"Chen","year":"","journal-title":"arXiv Preprint arXiv:2002.05709"},{"key":"B30","first-page":"104","article-title":"\u201cUNITER: universal image-tExt representation learning,\u201d","volume-title":"European Conference on Computer Vision","author":"Chen","year":""},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2212.07143","article-title":"Reproducible scaling laws for contrastive language-image learning","author":"Cherti","year":"2022","journal-title":"arXiv Preprint arXiv:2212.07143"},{"key":"B32","unstructured":"Chiang\n              W.-L.\n            \n            \n              Li\n              Z.\n            \n            \n              Lin\n              Z.\n            \n            \n              Sheng\n              Y.\n            \n            \n              Wu\n              Z.\n            \n            \n              Zhang\n              H.\n            \n          \n          Vicuna: an Open-source Chatbot Impressing GPT-4 With 90%* ChatGPT Quality\n          \n          2023"},{"key":"B33","first-page":"1931","article-title":"\u201cUnifying vision-and-language tasks via text generation,\u201d","volume-title":"International Conference on Machine Learning, Vol. 139","author":"Cho","year":"2021"},{"key":"B34","first-page":"1724","article-title":"\u201cLearning phrase representations using rnn encoder\u2014decoder for statistical machine translation,\u201d","author":"Cho","year":"2014","journal-title":"Conference on Empirical Methods in Natural Language Processing"},{"key":"B35","doi-asserted-by":"publisher","first-page":"1","DOI":"10.48550\/arXiv.2204.02311","article-title":"PaLM: scaling language modeling with pathways","volume":"24","author":"Chowdhery","year":"2022","journal-title":"J. Machine Learn. Res"},{"key":"B36","doi-asserted-by":"publisher","first-page":"1183","DOI":"10.1007\/s10278-024-01183-x","article-title":"Summary of the National Cancer Institute 2023 virtual workshop on medical image de-identification\u2014part 2: pathology whole slide image de-identification, de-facing, the role of AI in image de-identification, and the NCI MIDI datasets and pipeline","volume":"24","author":"Clunie","year":"2024","journal-title":"J. Imag. Informat. Med"},{"key":"B37","doi-asserted-by":"publisher","first-page":"101964","DOI":"10.1016\/j.artmed.2020.101964","article-title":"Reinforcement learning for intelligent healthcare applications: a survey","volume":"109","author":"Coronato","year":"2020","journal-title":"Artif. Intell. Med"},{"key":"B38","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.06500","article-title":"InstructBLIP: towards general-purpose vision-language models with instruction tuning","author":"Dai","year":"2023","journal-title":"arXiv Preprint arXiv:2305.06500"},{"key":"B39","doi-asserted-by":"publisher","first-page":"304","DOI":"10.1093\/jamia\/ocv080","article-title":"Preparing a collection of radiology examinations for distribution and retrieval","volume":"23","author":"Demner-Fushman","year":"2015","journal-title":"J. Am. Med. Informat. Assoc"},{"key":"B40","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1109\/CVPR.2009.5206848","article-title":"\u201cImageNet: a large-scale hierarchical image database,\u201d","author":"Deng","year":"2009","journal-title":"2009 IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B41","first-page":"4171","article-title":"\u201cBERT: pre-training of deep bidirectional transformers for language understanding,\u201d","volume-title":"Conference of the North American Chapter of the Association for Computational Linguistics, Vol. 1","author":"Devlin","year":"2019"},{"key":"B42","article-title":"\u201cAn image is worth 16x16 words: transformers for image recognition at scale,\u201d","volume-title":"International Conference on Learning Representations","author":"Dosovitskiy","year":"2021"},{"key":"B43","first-page":"18145","article-title":"\u201cAn empirical study of training end-to-end vision-and-language transformers,\u201d","volume-title":"IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Dou","year":"2022"},{"key":"B44","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/11787006_1","article-title":"\u201cDifferential privacy,\u201d","volume-title":"Automata, Languages and Programming","author":"Dwork","year":"2006"},{"key":"B45","doi-asserted-by":"publisher","first-page":"1181","DOI":"10.18653\/v1\/2023.findings-eacl.88","article-title":"PubmedCLIP: how much does clip benefit visual question answering in the medical domain?","volume":"88","author":"Eslami","year":"2023","journal-title":"Find. Assoc. Comput. Linguist"},{"key":"B46","doi-asserted-by":"crossref","first-page":"12868","DOI":"10.1109\/CVPR46437.2021.01268","article-title":"\u201cTaming transformers for high-resolution image synthesis,\u201d","volume-title":"2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Esser","year":"2021"},{"key":"B47","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2210.09263","article-title":"Vision-language pre-training: basics, recent advances, and future trends","author":"Gan","year":"2022","journal-title":"arXiv Preprint arXiv:2210.09263"},{"key":"B48","volume-title":"Deep Learning","author":"Goodfellow","year":"2016"},{"key":"B49","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2307.12980","article-title":"A systematic survey of prompt engineering on vision-language foundation models","author":"Gu","year":"2023","journal-title":"arXiv Preprint arXiv:2307.12980"},{"key":"B50","doi-asserted-by":"publisher","DOI":"10.1145\/3458754","article-title":"Domain-specific language model pretraining for biomedical natural language processing","author":"Gu","year":"2021","journal-title":"ACM Trans. Comput. Healthc"},{"key":"B51","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2304.08247","article-title":"MedAlpaca\u2014an open-source collection of medical conversational AI models and training data","author":"Han","year":"2023","journal-title":"arXiv Preprint arXiv:2304.08247"},{"key":"B52","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.cmcl-1.10","article-title":"Probabilistic predictions of people perusing: evaluating metrics of language model performance for psycholinguistic modeling","author":"Hao","year":"2020","journal-title":"arXiv Preprint arXiv:2009.03954"},{"key":"B53","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2312.04913","article-title":"SA-Attack: improving adversarial transferability of vision-language pre-training models via self-augmentation","author":"He","year":"","journal-title":"arXiv Preprint arXiv:2312.04913"},{"key":"B54","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2310.05694","article-title":"A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics","author":"He","year":"","journal-title":"arXiv Preprint arXiv:2310.05694"},{"key":"B55","first-page":"770","article-title":"\u201cDeep residual learning for image recognition,\u201d","author":"He","year":"2016","journal-title":"IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B56","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2003.10286","article-title":"PathVQA: 30000+ questions for medical visual question answering","author":"He","year":"2020","journal-title":"arXiv Preprint arXiv:2003.10286"},{"key":"B57","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput"},{"key":"B58","article-title":"\u201cLoRA: low-rank adaptation of large language models,\u201d","volume-title":"International Conference on Learning Representations","author":"Hu","year":"2022"},{"key":"B59","doi-asserted-by":"publisher","first-page":"8704","DOI":"10.1109\/TPAMI.2019.2918284","article-title":"Convolutional networks with dense connectivity","volume":"44","author":"Huang","year":"2022","journal-title":"IEEE Trans. Pat. Anal. Machine Intell"},{"key":"B60","article-title":"\u201cWhat makes multimodal learning better than single (provably),\u201d","volume-title":"Advances in Neural Information Processing Systems","author":"Huang","year":"2021"},{"key":"B61","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1007\/978-3-030-85251-1_23","article-title":"Overview of the imageclef 2021: multimedia retrieval in medical, nature, internet and social media applications","volume":"23","author":"Ionescu","year":"2021","journal-title":"Exp. IR Meets Multilingual. Multimodal. Interact"},{"key":"B62","doi-asserted-by":"publisher","first-page":"590","DOI":"10.1609\/aaai.v33i01.3301590","article-title":"CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison","volume":"33","author":"Irvin","year":"2019","journal-title":"AAAI Conf. Artif. Intell"},{"key":"B63","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.17579","article-title":"Multimodal image-text matching improves retrieval-based chest x-ray report generation","author":"Jeong","year":"2023","journal-title":"arXiv Preprint arXiv:2303.17579"},{"key":"B64","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1016\/B978-0-12-803467-5.00010-1","article-title":"5\u2014computer vision applications","volume":"10","author":"Ji","year":"2020","journal-title":"Comput. Vis. Pat. Recogn"},{"key":"B65","first-page":"4904","article-title":"\u201cScaling up visual and vision-language representation learning with noisy text supervision,\u201d","author":"Jia","year":"2021","journal-title":"International Conference on Machine Learning, Vol. 139"},{"key":"B66","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2310.06825","article-title":"Mistral 7B","author":"Jiang","year":"2023","journal-title":"arXiv Preprint arXiv:2310.06825"},{"key":"B67","doi-asserted-by":"publisher","first-page":"6421","DOI":"10.3390\/app11146421","article-title":"What disease does this patient have? a large-scale open domain question answering dataset from medical exams","volume":"11","author":"Jin","year":"2021","journal-title":"Appl. Sci"},{"key":"B68","first-page":"2567","article-title":"PubMedQA: a dataset for biomedical research question answering","author":"Jin","year":"2019","journal-title":"Conference on Empirical Methods in Natural Language Processing"},{"key":"B69","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1038\/s41597-019-0322-0","article-title":"MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports","volume":"6","author":"Johnson","year":"","journal-title":"Sci. Data"},{"key":"B70","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1901.07042","article-title":"MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs","author":"Johnson","year":"","journal-title":"arXiv Preprint arXiv:1901.07042"},{"key":"B71","doi-asserted-by":"publisher","first-page":"701","DOI":"10.1007\/978-3-031-16443-9_67","article-title":"Explaining chest x-ray pathologies in natural language","volume":"13435","author":"Kayser","year":"2022","journal-title":"Int. Conf. Med. Image Comput. Computer-Assist. Interv"},{"key":"B72","doi-asserted-by":"crossref","first-page":"752","DOI":"10.1109\/ISCC58397.2023.10218203","article-title":"\u201cThe importance of robust features in mitigating catastrophic forgetting,\u201d","volume-title":"2023 IEEE Symposium on Computers and Communications (ISCC)","author":"Khan","year":"2023"},{"key":"B73","doi-asserted-by":"publisher","first-page":"14588","DOI":"10.48550\/arXiv.2404.14588","article-title":"Brain-inspired continual learning: robust feature distillation and re-consolidation for class incremental learning","volume":"2024","author":"Khan","year":"2024","journal-title":"IEEE Access"},{"key":"B74","doi-asserted-by":"publisher","first-page":"1564","DOI":"10.48550\/arXiv.1805.07932","article-title":"Bilinear attention networks","volume":"31","author":"Kim","year":"2018","journal-title":"Adv. Neural Inform. Process. Syst"},{"key":"B75","article-title":"\u201cADAM: a method for stochastic optimization,\u201d","volume-title":"International Conference on Learning Representations","author":"Kingma","year":"2014"},{"key":"B76","volume-title":"Future-Proofing Medical Imaging with Privacy-Preserving Federated Learning and Uncertainty Quantification: A Review","author":"Koutsoubis","year":""},{"key":"B77","volume-title":"Privacy Preserving Federated Learning in Medical Imaging with Uncertainty Estimation","author":"Koutsoubis","year":""},{"key":"B78","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2208.02131","article-title":"Masked vision and language modeling for multi-modal representation learning","author":"Kwon","year":"2023","journal-title":"arXiv Preprint arXiv:2208.02131"},{"key":"B79","unstructured":"Lambert\n              N.\n            \n            \n              Castricato\n              L.\n            \n            \n              von Werra\n              L.\n            \n            \n              Havrilla\n              A.\n            \n          \n          Hugging Face\n          Illustrating Reinforcement Learning From Human Feedback (RLHF)\n          \n          2022"},{"key":"B80","doi-asserted-by":"publisher","first-page":"180251","DOI":"10.1038\/sdata.2018.251","article-title":"A dataset of clinically generated visual questions and answers about radiology images","volume":"5","author":"Lau","year":"2018","journal-title":"Sci. Data"},{"key":"B81","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"B82","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2302.12172","article-title":"UniXGen: a unified vision-language model for multi-view chest x-ray generation and report generation","author":"Lee","year":"2023","journal-title":"arXiv Preprint arXiv:2302.12172"},{"key":"B83","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2104.08691","article-title":"The power of scale for parameter-efficient prompt tuning","author":"Lester","year":"2021","journal-title":"arXiv Preprint arXiv:2104.08691"},{"key":"B84","doi-asserted-by":"publisher","first-page":"9459","DOI":"10.48550\/arXiv.2005.11401","article-title":"Retrieval-augmented generation for knowledge-intensive NLP tasks","volume":"33","author":"Lewis","year":"2020","journal-title":"Neural Inform. Process. Syst"},{"key":"B85","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2306.00890","article-title":"LLaVA-Med: training a large language-and-vision assistant for biomedicine in one day","author":"Li","year":"","journal-title":"arXiv Preprint arXiv:2306.00890"},{"key":"B86","doi-asserted-by":"publisher","first-page":"7651","DOI":"10.48550\/arXiv.2107.07651","article-title":"Align before fuse: vision and language representation learning with momentum distillation","volume":"2021","author":"Li","year":"2021","journal-title":"Adv. Neural Inform.. Process. Syst"},{"key":"B87","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1908.03557","article-title":"VisualBERT: a simple and performant baseline for vision and language","author":"Li","year":"2019","journal-title":"arXiv Preprint arXiv:1908.03557"},{"key":"B88","first-page":"20624","article-title":"\u201cCross-modal clinical graph transformer for ophthalmic report generation,\u201d","volume-title":"IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Li","year":"2022"},{"key":"B89","first-page":"374","article-title":"\u201cMasked vision and language pre-training with unimodal and multimodal contrastive losses for medical visual question answering,\u201d","author":"Li","year":"","journal-title":"Medical Image Computing and Computer Assisted Intervention (MICCAI)"},{"key":"B90","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2101.00190","article-title":"Prefix-Tuning: optimizing continuous prompts for generation","author":"Li","year":"2021","journal-title":"arXiv Preprint arXiv:2101.00190"},{"key":"B91","doi-asserted-by":"publisher","first-page":"40895","DOI":"10.7759\/cureus.40895","article-title":"Chatdoctor: A medical chat model fine-tuned on a large language model meta-AI (llama) using medical domain knowledge","volume":"15","author":"Li","year":"","journal-title":"Cureus"},{"key":"B92","first-page":"74","article-title":"\u201cROUGE: a package for automatic evaluation of summaries,\u201d","volume-title":"Text Summarization Branches Out","author":"Lin","year":"2004"},{"key":"B93","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.07240","article-title":"PMC-CLIP: contrastive language-image pre-training using biomedical documents","author":"Lin","year":"","journal-title":"arXiv Preprint arXiv:2303.07240"},{"key":"B94","doi-asserted-by":"publisher","first-page":"102611","DOI":"10.1016\/j.artmed.2023.102611","article-title":"Medical visual question answering: a survey","volume":"143","author":"Lin","year":"","journal-title":"Artif. Intell. Med"},{"key":"B95","first-page":"1650","article-title":"SLAKE: a semantically-labeled knowledge-enhanced dataset for medical visual question answering","author":"Liu","year":"","journal-title":"IEEE 18th International Symposium on Biomedical Imaging (ISBI)"},{"key":"B96","doi-asserted-by":"publisher","first-page":"1532","DOI":"10.1109\/TMI.2022.3232411","article-title":"Medical visual question answering via conditional reasoning and contrastive learning","volume":"42","author":"Liu","year":"","journal-title":"IEEE Trans. Med. Imag"},{"key":"B97","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2311.14199","article-title":"A systematic review of deep learning-based research on radiology report generation","author":"Liu","year":"","journal-title":"arXiv Preprint arXiv:2311.14199"},{"key":"B98","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2212.10505","article-title":"DePlot: one-shot visual language reasoning by plot-to-table translation","author":"Liu","year":"2022","journal-title":"arXiv Preprint arXiv:2212.10505"},{"key":"B99","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2304.08485","article-title":"Visual instruction tuning","author":"Liu","year":"","journal-title":"arXiv Preprint arXiv:2304.08485"},{"key":"B100","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.00253","article-title":"A survey on hallucination in large vision-language models","author":"Liu","year":"2024","journal-title":"arXiv Preprint arXiv:2402.00253"},{"key":"B101","first-page":"9992","article-title":"\u201cSwin transformer: hierarchical vision transformer using shifted windows,\u201d","volume-title":"International Conference on Computer Vision (ICCV)","author":"Liu","year":""},{"key":"B102","first-page":"4969","article-title":"\u201cS2ORC: the semantic scholar open research corpus,\u201d","volume-title":"Annual Meeting of the Association for Computational Linguistics","author":"Lo","year":"2020"},{"key":"B103","first-page":"13","article-title":"ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks","author":"Lu","year":"2019","journal-title":"Advances in Neural Information Processing Systems"},{"key":"B104","doi-asserted-by":"publisher","first-page":"863","DOI":"10.1038\/s41591-024-02856-4","article-title":"A visual-language foundation model for computational pathology","volume":"30","author":"Lu","year":"","journal-title":"Nat. Med"},{"key":"B105","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1038\/s41586-024-07618-3","article-title":"A multimodal generative ai copilot for human pathology","volume":"24","author":"Lu","year":"","journal-title":"Nature"},{"key":"B106","doi-asserted-by":"publisher","first-page":"988","DOI":"10.1007\/s10278-020-00349-7","article-title":"Framework for extracting critical findings in radiology reports","volume":"33","author":"Mabotuwana","year":"2020","journal-title":"J. Digit. Imag"},{"key":"B107","doi-asserted-by":"publisher","first-page":"106791","DOI":"10.1016\/j.compbiomed.2023.106791","article-title":"MedViT: a robust vision transformer for generalized medical image classification","volume":"157","author":"Manzari","year":"2023","journal-title":"Comput. Biol. Med"},{"key":"B108","first-page":"52","article-title":"\u201cStacked convolutional auto-encoders for hierarchical feature extraction,\u201d","volume-title":"International Conference on Artificial Neural Networks, Vol. 6791","author":"Masci","year":"2011"},{"key":"B109","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1301.3781","article-title":"Efficient estimation of word representations in vector space","author":"Mikolov","year":"","journal-title":"arXiv Preprint arXiv:1301.3781"},{"key":"B110","doi-asserted-by":"publisher","first-page":"3111","DOI":"10.48550\/arXiv.1310.4546","article-title":"Distributed representations of words and phrases and their compositionality","volume":"26","author":"Mikolov","year":"","journal-title":"Adv. Neural Inform. Process. Syst"},{"key":"B111","first-page":"01","article-title":"\u201cVT-ADL: a vision transformer network for image anomaly detection and localization,\u201d","volume-title":"IEEE International Symposium on Industrial Electronics (ISIE)","author":"Mishra","year":"2021"},{"key":"B112","first-page":"5288","article-title":"\u201cImproving factual completeness and consistency of image-to-text radiology report generation,\u201d","author":"Miura","year":"2021","journal-title":"North American Chapter of the Association for Computational Linguistics"},{"key":"B113","doi-asserted-by":"publisher","first-page":"1814","DOI":"10.1109\/ACCESS.2022.3232719","article-title":"Vision transformer and language model based radiology report generation","volume":"11","author":"Mohsan","year":"2023","journal-title":"IEEE Access"},{"key":"B114","doi-asserted-by":"publisher","first-page":"101878","DOI":"10.1016\/j.artmed.2020.101878","article-title":"Deep learning in generating radiology reports: a survey","volume":"106","author":"Monshi","year":"2020","journal-title":"Artif. Intell. Med"},{"key":"B115","doi-asserted-by":"publisher","first-page":"6070","DOI":"10.1109\/JBHI.2022.3207502","article-title":"Multi-modal understanding and generation for medical images and text via vision-language pre-training","volume":"26","author":"Moon","year":"2022","journal-title":"IEEE J. Biomed. Health Informat"},{"key":"B116","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2307.15189","article-title":"Med-Flamingo: a multimodal medical few-shot learner","author":"Moor","year":"2023","journal-title":"arXiv Preprint arXiv:2307.15189"},{"key":"B117","doi-asserted-by":"publisher","first-page":"544","DOI":"10.1136\/amiajnl-2011-000464","article-title":"Natural language processing: an introduction","volume":"18","author":"Nadkarni","year":"2011","journal-title":"J. Am. Med. Informat. Assoc"},{"key":"B118","doi-asserted-by":"publisher","first-page":"258","DOI":"10.1038\/s41746-020-0258-y","article-title":"Protected health information filter (philter): accurately and securely de-identifying free-text clinical notes","volume":"3","author":"Norgeot","year":"2020","journal-title":"NPJ Digit. Med"},{"key":"B119","doi-asserted-by":"publisher","first-page":"27730","DOI":"10.48550\/arXiv.2203.02155","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang","year":"2022","journal-title":"Adv. Neural Inform. Process. Syst"},{"key":"B120","first-page":"311","article-title":"\u201cBLEU: a method for automatic evaluation of machine translation,\u201d","author":"Papineni","year":"2002","journal-title":"Annual Meeting of the Association for Computational Linguistics"},{"key":"B121","doi-asserted-by":"publisher","first-page":"180","DOI":"10.1007\/978-3-030-01364-6_20","article-title":"Radiology objects in context (ROCO): a multimodal image dataset","volume":"11043","author":"Pelka","year":"2018","journal-title":"Intravasc. Imag. Comput. Assist. Stent. Large-Scale Annot. Biomed. Data Expert Label Synth"},{"key":"B122","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2311.18681","article-title":"Radialog: a large vision-language model for radiology report generation and conversational assistance","author":"Pellegrini","year":"2023","journal-title":"arXiv Preprint arXiv:2311.18681"},{"key":"B123","doi-asserted-by":"publisher","first-page":"188","DOI":"10.48550\/arXiv.1712.05898","article-title":"NegBio: a high-performance tool for negation and uncertainty detection in radiology reports","volume":"2018","author":"Peng","year":"2017","journal-title":"AMIA Sum. Transl. Sci. Proc"},{"key":"B124","doi-asserted-by":"publisher","first-page":"1532","DOI":"10.3115\/v1\/D14-1162","article-title":"Glove: global vectors for word representation","volume":"14","author":"Pennington","year":"2014","journal-title":"Empir. Methods Natur. Lang. Process"},{"key":"B125","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2103.00020","article-title":"Learning transferable visual models from natural language supervision","author":"Radford","year":"2021","journal-title":"arXiv Preprint arXiv:2103.00020"},{"key":"B126","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1007\/978-981-15-6198-6_18","article-title":"Study of various methods for tokenization","volume":"18","author":"Rai","year":"2021","journal-title":"Appl. Internet Things"},{"key":"B127","doi-asserted-by":"publisher","first-page":"456","DOI":"10.48550\/arXiv.2210.06340","article-title":"Improving radiology report generation systems by removing hallucinated references to non-existent priors","volume":"193","author":"Ramesh","year":"2022","journal-title":"Machine Learn. Res"},{"key":"B128","first-page":"12159","article-title":"\u201cVision transformers for dense prediction,\u201d","volume-title":"IEEE\/CVF International Conference on Computer Vision (ICCV)","author":"Ranftl","year":"2021"},{"key":"B129","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1007\/s11831-023-09884-2","article-title":"Self-supervised learning: a succinct review","volume":"30","author":"Rani","year":"2023","journal-title":"Archiv. Comput. Methods Eng"},{"key":"B130","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.03660","article-title":"Retrieval augmented chest X-ray report generation using openAI GPT models","author":"Ranjit","year":"2023","journal-title":"arXiv Preprint arXiv:2305.03660"},{"key":"B131","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1186\/s13012-024-01357-9","article-title":"Generative AI in healthcare: an implementation science informed translational path on application, integration and governance","volume":"19","author":"Reddy","year":"2024","journal-title":"Implement. Sci"},{"key":"B132","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.18243","article-title":"Learning or self-aligning? rethinking instruction fine-tuning","author":"Ren","year":"2024","journal-title":"arXiv Preprint arXiv:2402.18243"},{"key":"B133","first-page":"658","article-title":"\u201cGeneralized intersection over union: a metric and a loss for bounding box regression,\u201d","author":"Rezatofighi","year":"2019","journal-title":"IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"B134","doi-asserted-by":"crossref","first-page":"400","DOI":"10.1214\/aoms\/1177729586","article-title":"A stochastic approximation method","volume":"22","author":"Robbins","year":"1951","journal-title":"Ann. Math. Stat"},{"key":"B135","first-page":"1586","article-title":"\u201cLessons from natural language inference in the clinical domain,\u201d","volume-title":"Conference on Empirical Methods in Natural Language Processing","author":"Romanov","year":"2018"},{"key":"B136","first-page":"1294","article-title":"\u201cOverview of imageclefmedical 2022\u2014caption prediction and concept detection,\u201d","author":"R\u00fcckert","year":"2022","journal-title":"CEUR Workshop Proceedings, Vol. 3180"},{"key":"B137","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1912.05911","article-title":"Recurrent neural networks (RNNS): a gentle introduction and overview","author":"Schmidt","year":"2019","journal-title":"arXiv Preprint arXiv:1912.05911"},{"key":"B138","first-page":"33","article-title":"\u201cSurgical-VQA: visual question answering in surgical scenes using transformer,\u201d","author":"Seenivasan","year":"2022","journal-title":"Medical Image Computing and Computer Assisted Intervention\u2014MICCAI"},{"key":"B139","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2312.01435","article-title":"Automatic report generation for histopathology images using pre-trained vision transformers and BERT","author":"Sengupta","year":"2023","journal-title":"arXiv Preprint arXiv:2312.01435"},{"key":"B140","doi-asserted-by":"crossref","first-page":"1715","DOI":"10.18653\/v1\/P16-1162","article-title":"\u201cNeural machine translation of rare words with subword units,\u201d","volume-title":"54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Sennrich","year":"2016"},{"key":"B141","doi-asserted-by":"publisher","first-page":"119773","DOI":"10.1016\/j.eswa.2023.119773","article-title":"Evolution of visual data captioning methods, datasets, and evaluation metrics: a comprehensive survey","volume":"221","author":"Sharma","year":"2023","journal-title":"Expert Syst. Appl"},{"key":"B142","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2312.06224","article-title":"Medical vision language pretraining: a survey","author":"Shrestha","year":"2023","journal-title":"arXiv Preprint arXiv:2312.06224"},{"key":"B143","unstructured":"Shu\n              C.\n            \n            \n              Chen\n              B.\n            \n            \n              Liu\n              F.\n            \n            \n              Fu\n              Z.\n            \n            \n              Shareghi\n              E.\n            \n            \n              Collier\n              N.\n            \n          \n          Visual MED-ALPACA: a Parameter-Efficient Biomedical LLM With Visual Capabilities\n          \n          2023"},{"key":"B144","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"B145","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2004.09167","article-title":"CheXbert: combining automatic labelers and expert annotations for accurate radiology report labeling using bert","author":"Smit","year":"2020","journal-title":"arXiv Preprint arXiv:2004.09167"},{"key":"B146","doi-asserted-by":"publisher","first-page":"1526","DOI":"10.1007\/s11263-022-01611-x","article-title":"Curriculum learning: a survey","volume":"130","author":"Soviany","year":"2021","journal-title":"Int. J. Comput. Vis"},{"key":"B147","doi-asserted-by":"publisher","first-page":"1611","DOI":"10.48550\/arXiv.2108.03437","article-title":"Secure neuroimaging analysis using federated learning with homomorphic encryption","volume":"22","author":"Stripelis","year":"2021","journal-title":"SPIE Med. Imag"},{"key":"B148","doi-asserted-by":"crossref","first-page":"2112","DOI":"10.18653\/v1\/2020.findings-emnlp.191","article-title":"\u201cMediCAT: a dataset of medical images, captions, and textual references,\u201d","volume-title":"Findings of the Association for Computational Linguistics: EMNLP","author":"Subramanian","year":"2020"},{"key":"B149","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2309.14525","article-title":"Aligning large multimodal models with factually augmented RLHF","author":"Sun","year":"2023","journal-title":"arXiv Preprint arXiv:2309.14525"},{"key":"B150","doi-asserted-by":"crossref","first-page":"1054","DOI":"10.1109\/TNN.1998.712192","article-title":"Reinforcement learning: an introduction","volume":"9","author":"Sutton","year":"1998","journal-title":"IEEE Trans. Neural Netw"},{"key":"B151","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1905.11946","article-title":"EfficientNet: tethinking model scaling for convolutional neural networks","author":"Tan","year":"2020","journal-title":"arXiv Preprint arXiv:1905.11946"},{"key":"B152","first-page":"714","article-title":"\u201cRepsNet: combining vision with language for automated medical reports,\u201d","author":"Tanwani","year":"2022","journal-title":"Medical Image Computing and Computer Assisted Intervention (MICCAI)"},{"key":"B153","doi-asserted-by":"publisher","first-page":"415","DOI":"10.1177\/107769905303000401","article-title":"\u201cCloze procedure\u201d: a new tool for measuring readability","volume":"30","author":"Taylor","year":"1953","journal-title":"J. Mass Commun. Quart"},{"key":"B154","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2306.07971","article-title":"XrayGPT: chest radiographs summarization using medical vision-language models","author":"Thawkar","year":"2023","journal-title":"arXiv Preprint arXiv:2306.07971"},{"key":"B155","doi-asserted-by":"publisher","first-page":"1113","DOI":"10.1186\/s12938-023-01113-y","article-title":"A survey on automatic generation of medical imaging reports based on deep learning","volume":"22","author":"Ting","year":"2023","journal-title":"BioMed. Eng. OnL"},{"key":"B156","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2302.13971","article-title":"LLaMA: open and efficient foundation language models","author":"Touvron","year":"","journal-title":"arXiv Preprint arXiv:2302.13971"},{"key":"B157","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2307.09288","article-title":"LLaMA 2: open foundation and fine-tuned chat models","author":"Touvron","year":"","journal-title":"arXiv Preprint arXiv:2307.09288"},{"key":"B158","doi-asserted-by":"publisher","first-page":"51634","DOI":"10.3390\/s24051634","article-title":"Building flexible, scalable, and machine learning-ready multimodal oncology datasets","volume":"24","author":"Tripathi","year":"","journal-title":"Sensors"},{"key":"B159","volume-title":"HoneyBee: A Scalable Modular Framework for Creating Multimodal Oncology Datasets with Foundational Embedding Models","author":"Tripathi","year":""},{"key":"B160","first-page":"12","article-title":"\u201cDetecting pneumonia using vision transformer and comparing with other techniques,\u201d","author":"Tyagi","year":"2021","journal-title":"International Conference on Electronics, Communication and Aerospace Technology (ICECA)"},{"key":"B161","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1807.03748","article-title":"Representation learning with contrastive predictive coding","author":"van den Oord","year":"2019","journal-title":"arXiv Preprint arXiv:1807.03748"},{"key":"B162","doi-asserted-by":"publisher","first-page":"5998","DOI":"10.48550\/arXiv.1706.03762","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. Neural Inform. Process. Syst"},{"key":"B163","doi-asserted-by":"crossref","first-page":"1495","DOI":"10.1007\/978-1-4419-9863-7_158","author":"Verspoor","year":"2013","journal-title":"Encyclopedia of Systems Biology, Chapter Natural Language Processing"},{"key":"B164","doi-asserted-by":"crossref","first-page":"9154","DOI":"10.1609\/aaai.v34i05.6451","article-title":"\u201cNeural machine translation with byte-level subwords,\u201d","author":"Wang","year":"2020","journal-title":"AAAI Conference on Artificial Intelligence"},{"key":"B165","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2205.14100","article-title":"GIT: a generative image-to-text transformer for vision and language","author":"Wang","year":"","journal-title":"arXiv Preprint arXiv:2205.14100"},{"key":"B166","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2302.00487","article-title":"A comprehensive survey of continual learning: theory, method and application","author":"Wang","year":"","journal-title":"arXiv Preprint arXiv:2302.00487"},{"key":"B167","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2212.10560","article-title":"Self-instruct: aligning language models with self-generated instructions","author":"Wang","year":"","journal-title":"arXiv Preprint arXiv:2212.10560"},{"key":"B168","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2210.10163","article-title":"MedCLIP: contrastive learning from unpaired medical images and text","author":"Wang","year":"","journal-title":"arXiv Preprint arXiv:2210.10163"},{"key":"B169","article-title":"\u201cSimVLM: simple visual language model pretraining with weak supervision,\u201d","author":"Wang","year":"","journal-title":"International Conference on Learning Representations (ICLR)"},{"key":"B170","doi-asserted-by":"publisher","first-page":"100255","DOI":"10.1016\/j.labinv.2023.100255","article-title":"Revolutionizing digital pathology with the power of generative artificial intelligence and foundation models","volume":"103","author":"Waqas","year":"2023","journal-title":"Lab. Invest"},{"key":"B171","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1093\/bjrai\/ubae014","article-title":"Digital pathology and multimodal learning on oncology data","volume":"1","author":"Waqas","year":"","journal-title":"Artif. Intell"},{"key":"B172","doi-asserted-by":"publisher","first-page":"1408843","DOI":"10.3389\/frai.2024.1408843","article-title":"Multimodal data integration for oncology in the era of deep neural networks: a review","volume":"7","author":"Waqas","year":"","journal-title":"Front. Artif. Intell"},{"key":"B173","volume-title":"Embedding-based Multimodal Learning on Pan-Squamous Cell Carcinomas for Improved Survival Outcomes","author":"Waqas","year":""},{"key":"B174","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1609.08144","article-title":"Google's neural machine translation system: bridging the gap between human and machine translation","author":"Wu","year":"2016","journal-title":"arXiv Preprint arXiv:1609.08144"},{"key":"B175","first-page":"5987","article-title":"\u201cAggregated residual transformations for deep neural networks,\u201d","author":"Xie","year":"2016","journal-title":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"B176","first-page":"9643","article-title":"\u201cSimMIM: a simple framework for masked image modeling,\u201d","author":"Xie","year":"2022","journal-title":"IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"B177","doi-asserted-by":"publisher","first-page":"105939","DOI":"10.1016\/j.compbiomed.2022.105939","article-title":"An improved transformer network for skin cancer classification","volume":"149","author":"Xin","year":"2022","journal-title":"Comput. Biol. Med"},{"key":"B178","doi-asserted-by":"crossref","first-page":"12350","DOI":"10.1109\/ICRA48506.2021.9561569","article-title":"\u201cLearning domain adaptation with model calibration for surgical report generation in robotic surgery,\u201d","author":"Xu","year":"2021","journal-title":"2021 IEEE International Conference on Robotics and Automation (ICRA)"},{"key":"B179","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1007\/s13244-018-0639-9","article-title":"Convolutional neural networks: an overview and application in radiology","volume":"9","author":"Yamashita","year":"2018","journal-title":"Insight. Imag"},{"key":"B180","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1038\/s41746-022-00742-2","article-title":"A large language model for electronic health records","volume":"5","author":"Yang","year":"2022","journal-title":"NPJ Digit. Med"},{"key":"B181","doi-asserted-by":"publisher","first-page":"100802","DOI":"10.1016\/j.patter.2023.100802","article-title":"Evaluating progress in automatic chest x-ray radiology report generation","volume":"4","author":"Yu","year":"2023","journal-title":"Patterns"},{"key":"B182","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.00534","article-title":"RAMM: retrieval-augmented biomedical visual question answering with multi-modal pre-training","author":"Yuan","year":"2023","journal-title":"arXiv Preprint arXiv:2303.00534"},{"key":"B183","first-page":"6713","article-title":"\u201cFrom recognition to cognition: visual commonsense reasoning,\u201d","author":"Zellers","year":"2019","journal-title":"IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"B184","doi-asserted-by":"crossref","first-page":"9241","DOI":"10.18653\/v1\/2020.emnlp-main.743","article-title":"\u201cMedDialog: large-scale medical dialogue datasets,\u201d","author":"Zeng","year":"2020","journal-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP)"},{"key":"B185","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2309.10313","article-title":"Investigating the catastrophic forgetting in multimodal large language models","author":"Zhai","year":"2023","journal-title":"arXiv Preprint arXiv:2309.10313"},{"key":"B186","first-page":"2345","article-title":"\u201cMedical visual question answering via conditional reasoning,\u201d","author":"Zhan","year":"2020","journal-title":"The 28th ACM International Conference on Multimedia"},{"key":"B187","doi-asserted-by":"publisher","first-page":"106775","DOI":"10.1016\/j.knosys.2021.106775","article-title":"A survey on federated learning","volume":"216","author":"Zhang","year":"2021","journal-title":"Knowl. Based Syst"},{"key":"B188","first-page":"4158","article-title":"\u201cGrounding referring expressions in images by variational context,\u201d","author":"Zhang","year":"2018","journal-title":"IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B189","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.00915","article-title":"Large-scale domain-specific pretraining for biomedical vision-language processing","author":"Zhang","year":"","journal-title":"arXiv Preprint arXiv:2303.00915"},{"key":"B190","article-title":"\u201cBERTscore: evaluating text generation with BERT,\u201d","author":"Zhang","year":"2020","journal-title":"International Conference on Learning Representations"},{"key":"B191","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2304.09042","article-title":"Adapter learning in pretrained feature extractor for continual learning of diseases","author":"Zhang","year":"","journal-title":"arXiv Preprint arXiv:2304.09042"},{"key":"B192","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1038\/s41597-019-0055-0","article-title":"Biowordvec, improving biomedical word embeddings with subword information and mesh","volume":"6","author":"Zhang","year":"2019","journal-title":"Sci. Data"},{"key":"B193","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.10868","article-title":"Retrieving multimodal information for augmented generation: a survey","author":"Zhao","year":"2023","journal-title":"arXiv Preprint arXiv:2303.10868"},{"key":"B194","first-page":"10386","article-title":"\u201cDeep supervised cross-modal retrieval,\u201d","author":"Zhen","year":"2019","journal-title":"IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"B195","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.19270","article-title":"Learning without forgetting for vision-language models","author":"Zhou","year":"","journal-title":"arXiv Preprint arXiv:2305.19270"},{"key":"B196","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2311.05112","article-title":"A survey of large language models in medicine: progress, application, and challenge","author":"Zhou","year":"","journal-title":"arXiv Preprint arXiv:2311.05112"},{"key":"B197","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1909.08593","article-title":"Fine-tuning language models from human preferences","author":"Ziegler","year":"2020","journal-title":"arXiv Preprint arXiv:1909.08593"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2024.1430984\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T06:19:03Z","timestamp":1731997143000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2024.1430984\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,19]]},"references-count":197,"alternative-id":["10.3389\/frai.2024.1430984"],"URL":"https:\/\/doi.org\/10.3389\/frai.2024.1430984","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,19]]},"article-number":"1430984"}}