{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T20:13:44Z","timestamp":1784319224824,"version":"3.55.0"},"reference-count":49,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2024,4,13]],"date-time":"2024-04-13T00:00:00Z","timestamp":1712966400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key R&D Program of China","doi-asserted-by":"publisher","award":["2022ZD0160702"],"award-info":[{"award-number":["2022ZD0160702"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003399","name":"Science and Technology Commission of Shanghai Municipality","doi-asserted-by":"publisher","award":["22511106101"],"award-info":[{"award-number":["22511106101"]}],"id":[{"id":"10.13039\/501100003399","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003399","name":"Science and Technology Commission of Shanghai Municipality","doi-asserted-by":"publisher","award":["18DZ2270700"],"award-info":[{"award-number":["18DZ2270700"]}],"id":[{"id":"10.13039\/501100003399","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003399","name":"Science and Technology Commission of Shanghai Municipality","doi-asserted-by":"publisher","award":["21DZ1100100"],"award-info":[{"award-number":["21DZ1100100"]}],"id":[{"id":"10.13039\/501100003399","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100013314","name":"Higher Education Discipline Innovation Project 111","doi-asserted-by":"crossref","award":["BP0719010"],"award-info":[{"award-number":["BP0719010"]}],"id":[{"id":"10.13039\/501100013314","id-type":"DOI","asserted-by":"crossref"}]},{"name":"State Key Laboratory of UHD Video and Audio Production and Presentation."}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Recently, large language models (LLMs) have showcased remarkable capabilities in natural language understanding. While demonstrating proficiency in everyday conversations and question-answering (QA) situations, these models frequently struggle in domains that require precision, such as medical applications, due to their lack of domain-specific knowledge. In this article, we describe the procedure for building a powerful, open-source language model specifically designed for medicine applications, termed as PMC-LLaMA.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and methods<\/jats:title>\n                  <jats:p>We adapt a general-purpose LLM toward the medical domain, involving data-centric knowledge injection through the integration of 4.8M biomedical academic papers and 30K medical textbooks, as well as comprehensive domain-specific instruction fine-tuning, encompassing medical QA, rationale for reasoning, and conversational dialogues with 202M tokens.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>While evaluating various public medical QA benchmarks and manual rating, our lightweight PMC-LLaMA, which consists of only 13B parameters, exhibits superior performance, even surpassing ChatGPT. All models, codes, and datasets for instruction tuning will be released to the research community.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Our contributions are 3-fold: (1) we build up an open-source LLM toward the medical domain. We believe the proposed PMC-LLaMA model can promote further development of foundation models in medicine, serving as a medical trainable basic generative language backbone; (2) we conduct thorough ablation studies to demonstrate the effectiveness of each proposed component, demonstrating how different training data and model scales affect medical LLMs; (3) we contribute a large-scale, comprehensive dataset for instruction tuning.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>In this article, we systematically investigate the process of building up an open-source medical-specific LLM, PMC-LLaMA.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae045","type":"journal-article","created":{"date-parts":[[2024,4,13]],"date-time":"2024-04-13T17:29:11Z","timestamp":1713029351000},"page":"1833-1843","source":"Crossref","is-referenced-by-count":277,"title":["PMC-LLaMA: toward building open-source language models for medicine"],"prefix":"10.1093","volume":"31","author":[{"given":"Chaoyi","family":"Wu","sequence":"first","affiliation":[{"name":"Cooperative Medianet Innovation Center (CMIC), Shanghai Jiao Tong University , Shanghai, 200240, China"},{"name":"Shanghai AI Laboratory , Shanghai, 200232, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weixiong","family":"Lin","sequence":"additional","affiliation":[{"name":"Cooperative Medianet Innovation Center (CMIC), Shanghai Jiao Tong University , Shanghai, 200240, China"},{"name":"Shanghai AI Laboratory , Shanghai, 200232, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaoman","family":"Zhang","sequence":"additional","affiliation":[{"name":"Cooperative Medianet Innovation Center (CMIC), Shanghai Jiao Tong University , Shanghai, 200240, China"},{"name":"Shanghai AI Laboratory , Shanghai, 200232, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ya","family":"Zhang","sequence":"additional","affiliation":[{"name":"Cooperative Medianet Innovation Center (CMIC), Shanghai Jiao Tong University , Shanghai, 200240, China"},{"name":"Shanghai AI Laboratory , Shanghai, 200232, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weidi","family":"Xie","sequence":"additional","affiliation":[{"name":"Cooperative Medianet Innovation Center (CMIC), Shanghai Jiao Tong University , Shanghai, 200240, China"},{"name":"Shanghai AI Laboratory , Shanghai, 200232, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yanfeng","family":"Wang","sequence":"additional","affiliation":[{"name":"Cooperative Medianet Innovation Center (CMIC), Shanghai Jiao Tong University , Shanghai, 200240, China"},{"name":"Shanghai AI Laboratory , Shanghai, 200232, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,4,13]]},"reference":[{"key":"2024082207521347500_ocae045-B1","author":"OpenAI","year":"2023"},{"key":"2024082207521347500_ocae045-B2","author":"OpenAI"},{"key":"2024082207521347500_ocae045-B3","author":"Nori","year":"2023"},{"issue":"7972","key":"2024082207521347500_ocae045-B4","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"2024082207521347500_ocae045-B5","author":"Touvron","year":"2023"},{"key":"2024082207521347500_ocae045-B6","author":"Touvron","year":"2023"},{"key":"2024082207521347500_ocae045-B7","author":"Han","year":"2023"},{"key":"2024082207521347500_ocae045-B8","author":"Yunxiang","year":"2023"},{"key":"2024082207521347500_ocae045-B9","author":"Anil","year":"2023"},{"key":"2024082207521347500_ocae045-B10"},{"key":"2024082207521347500_ocae045-B11","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang","year":"2022","journal-title":"NIPS"},{"key":"2024082207521347500_ocae045-B12","author":"Peng","year":"2023"},{"key":"2024082207521347500_ocae045-B13","author":"Chung","year":"2022"},{"key":"2024082207521347500_ocae045-B14","author":"Iyer","year":"2022"},{"key":"2024082207521347500_ocae045-B15","volume-title":"Stanford Alpaca: An Instruction-Following LLaMA Model","author":"Taori","year":"2023"},{"key":"2024082207521347500_ocae045-B16","author":"Chiang","year":"2023"},{"issue":"4","key":"2024082207521347500_ocae045-B17","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2024082207521347500_ocae045-B18","author":"Zhang","year":"2023"},{"issue":"6","key":"2024082207521347500_ocae045-B19","doi-asserted-by":"crossref","first-page":"bbac409","DOI":"10.1093\/bib\/bbac409","article-title":"BioGPT: generative pre-trained transformer for biomedical text generation and mining","volume":"23","author":"Luo","year":"2022","journal-title":"Brief Bioinform"},{"key":"2024082207521347500_ocae045-B20","doi-asserted-by":"publisher","first-page":"4969","DOI":"10.18653\/v1\/2020.acl-main.447","author":"Lo","year":"2020"},{"key":"2024082207521347500_ocae045-B21","author":"Together Computer","year":"2023"},{"issue":"14","key":"2024082207521347500_ocae045-B22","doi-asserted-by":"crossref","first-page":"6421","DOI":"10.3390\/app11146421","article-title":"What disease does this patient have? A large-scale open domain question answering dataset from medical exams","volume":"11","author":"Jin","year":"2021","journal-title":"Appl Sci"},{"key":"2024082207521347500_ocae045-B23","first-page":"2567","author":"Jin"},{"key":"2024082207521347500_ocae045-B24","first-page":"248","author":"Pal","year":"2022"},{"issue":"1","key":"2024082207521347500_ocae045-B25","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1055\/s-0038-1637976","article-title":"The unified medical language system","volume":"2","author":"Lindberg","year":"1993","journal-title":"Yearb Med Inform"},{"key":"2024082207521347500_ocae045-B26","author":"Loshchilov"},{"key":"2024082207521347500_ocae045-B27","author":"Chen","year":"2016"},{"key":"2024082207521347500_ocae045-B28","volume-title":"LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models","author":"Diao","year":"2023"},{"issue":"2","key":"2024082207521347500_ocae045-B29","doi-asserted-by":"crossref","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","year":"2022","journal-title":"PLoS Digit Health"},{"key":"2024082207521347500_ocae045-B30","author":"Peng"},{"issue":"4","key":"2024082207521347500_ocae045-B31","doi-asserted-by":"crossref","first-page":"681","DOI":"10.1007\/s11023-020-09548-1","article-title":"GPT-3: its nature, scope, limits, and consequences","volume":"30","author":"Floridi","year":"2020","journal-title":"Minds Mach"},{"key":"2024082207521347500_ocae045-B32","author":"Venigalla","year":"2022"},{"key":"2024082207521347500_ocae045-B33","author":"Christophe","year":"2023"},{"key":"2024082207521347500_ocae045-B34","author":"Mangrulkar","year":"2022"},{"key":"2024082207521347500_ocae045-B35","author":"Hu"},{"key":"2024082207521347500_ocae045-B36","first-page":"61","author":"Liu","year":"2022"},{"key":"2024082207521347500_ocae045-B37","author":"Kaplan","year":"2020"},{"key":"2024082207521347500_ocae045-B38","author":"Yang","year":"2022"},{"key":"2024082207521347500_ocae045-B39","author":"Chen","year":"2023"},{"key":"2024082207521347500_ocae045-B40","author":"Moor"},{"key":"2024082207521347500_ocae045-B41","author":"Li"},{"key":"2024082207521347500_ocae045-B42","author":"Wu","year":"2023"},{"key":"2024082207521347500_ocae045-B43","author":"Tu"},{"key":"2024082207521347500_ocae045-B44"},{"issue":"2","key":"2024082207521347500_ocae045-B45","doi-asserted-by":"crossref","first-page":"AIoa2300068","DOI":"10.1056\/AIoa2300068","article-title":"Almanac\u2014retrieval-augmented language models for clinical medicine","volume":"1","author":"Zakka","year":"2024","journal-title":"NEJM AI"},{"key":"2024082207521347500_ocae045-B46","author":"Liu","year":"2023"},{"key":"2024082207521347500_ocae045-B47","first-page":"68539","author":"Schick"},{"key":"2024082207521347500_ocae045-B48","first-page":"71995","author":"Yang"},{"key":"2024082207521347500_ocae045-B49","author":"Sumers","year":"2023"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/1833\/58868261\/ocae045.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/1833\/58868261\/ocae045.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,22]],"date-time":"2024-08-22T12:20:45Z","timestamp":1724329245000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/9\/1833\/7645318"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,13]]},"references-count":49,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,4,13]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae045","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,4,13]]}}}