{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,19]],"date-time":"2026-01-19T06:21:40Z","timestamp":1768803700821,"version":"3.49.0"},"reference-count":49,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2024,5,23]],"date-time":"2024-05-23T00:00:00Z","timestamp":1716422400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/501100011068","name":"Fondation Saint-Luc","doi-asserted-by":"publisher","award":["467E"],"award-info":[{"award-number":["467E"]}],"id":[{"id":"10.13039\/501100011068","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To optimize the training strategy of large language models for medical applications, focusing on creating clinically relevant systems that efficiently integrate into healthcare settings, while ensuring high standards of accuracy and reliability.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We curated a comprehensive collection of high-quality, domain-specific data and used it to train several models, each with different subsets of this data. These models were rigorously evaluated against standard medical benchmarks, such as the USMLE, to measure their performance. Furthermore, for a thorough effectiveness assessment, they were compared with other state-of-the-art medical models of comparable size.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The models trained with a mix of high-quality, domain-specific, and general data showed superior performance over those trained on larger, less clinically relevant datasets (P\u2009&amp;lt;\u2009.001). Our 7-billion-parameter model Med5 scores 60.5% on MedQA, outperforming the previous best of 49.3% from comparable models, and becomes the first of its size to achieve a passing score on the USMLE. Additionally, this model retained its proficiency in general domain tasks, comparable to state-of-the-art general domain models of similar size.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Our findings underscore the importance of integrating high-quality, domain-specific data in training large language models for medical purposes. The balanced approach between specialized and general data significantly enhances the model\u2019s clinical relevance and performance.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>This study sets a new standard in medical language models, proving that a strategically trained, smaller model can outperform larger ones in clinical relevance and general proficiency, highlighting the importance of data quality and expert curation in generative artificial intelligence for healthcare applications.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae120","type":"journal-article","created":{"date-parts":[[2024,5,23]],"date-time":"2024-05-23T17:58:18Z","timestamp":1716487098000},"page":"1875-1883","source":"Crossref","is-referenced-by-count":10,"title":["Impact of high-quality, mixed-domain data on the performance of medical language models"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-8180-4737","authenticated-orcid":false,"given":"Maxime","family":"Griot","sequence":"first","affiliation":[{"name":"Institute of NeuroScience, Universit\u00e9 catholique de Louvain , Brussels, 1200, Belgium"},{"name":"Louvain Research Institute in Management and Organizations, Universit\u00e9 catholique de Louvain , Louvain-la-Neuve, 1348, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0717-3804","authenticated-orcid":false,"given":"Coralie","family":"Hemptinne","sequence":"additional","affiliation":[{"name":"Ophthalmology, Cliniques Universitaires Saint-Luc , Brussels, 1200, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3275-3333","authenticated-orcid":false,"given":"Jean","family":"Vanderdonckt","sequence":"additional","affiliation":[{"name":"Louvain Research Institute in Management and Organizations, Universit\u00e9 catholique de Louvain , Louvain-la-Neuve, 1348, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Demet","family":"Yuksel","sequence":"additional","affiliation":[{"name":"Institute of NeuroScience, Universit\u00e9 catholique de Louvain , Brussels, 1200, Belgium"},{"name":"Medical Information Department, Cliniques Universitaires Saint-Luc , Brussels, 1200, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2024,5,23]]},"reference":[{"issue":"8","key":"2024082207522184700_ocae120-B1","doi-asserted-by":"crossref","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"2024082207522184700_ocae120-B2","doi-asserted-by":"crossref","first-page":"1169595","DOI":"10.3389\/frai.2023.1169595","article-title":"ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations","volume":"6","author":"Dave","year":"2023","journal-title":"Front Artif Intell"},{"key":"2024082207522184700_ocae120-B3","author":"OpenAI","year":"2022"},{"key":"2024082207522184700_ocae120-B4","author":"OpenAI","year":"2023"},{"key":"2024082207522184700_ocae120-B5","author":"Singhal","year":"2023"},{"issue":"1","key":"2024082207522184700_ocae120-B6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3325\/cmj.2023.64.1","article-title":"Opportunities and risks of ChatGPT in medicine, science, and academic publishing: a modern Promethean dilemma","volume":"64","author":"Homolak","year":"2023","journal-title":"Croat Med J"},{"issue":"2","key":"2024082207522184700_ocae120-B7","doi-asserted-by":"crossref","first-page":"204","DOI":"10.1016\/j.cjca.2021.09.004","article-title":"Opening the black box: the promise and limitations of explainable machine learning in cardiology","volume":"38","author":"Petch","year":"2022","journal-title":"Can J Cardiol"},{"issue":"1","key":"2024082207522184700_ocae120-B8","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1038\/s41746-023-00873-0","article-title":"The imperative for regulatory oversight of large language models (or generative AI) in healthcare","volume":"6","author":"Mesk\u00f3","year":"2023","journal-title":"NPJ Digit Med"},{"key":"2024082207522184700_ocae120-B9","author":"Touvron","year":"2023"},{"key":"2024082207522184700_ocae120-B10","author":"Jiang","year":"2023"},{"key":"2024082207522184700_ocae120-B11","author":"Chen","year":"2023"},{"key":"2024082207522184700_ocae120-B12","author":"Wu","year":"2024"},{"key":"2024082207522184700_ocae120-B13","author":"Han","year":"2023"},{"key":"2024082207522184700_ocae120-B14"},{"key":"2024082207522184700_ocae120-B15","author":"Gunasekar","year":"2023"},{"issue":"7972","key":"2024082207522184700_ocae120-B16","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"2024082207522184700_ocae120-B17","author":"Rozi\u00e8re","year":"2023"},{"key":"2024082207522184700_ocae120-B18","first-page":"1384","author":"Madaan","year":"2022"},{"key":"2024082207522184700_ocae120-B19","author":"National Board of Medical Examiners","year":"2023"},{"key":"2024082207522184700_ocae120-B20","author":"American Society of Hematology","year":"2023"},{"key":"2024082207522184700_ocae120-B21","year":"2023"},{"key":"2024082207522184700_ocae120-B22","author":"National Center for Biotechnology Information","year":"2011"},{"key":"2024082207522184700_ocae120-B23","author":"Wolters Kluwer"},{"issue":"1","key":"2024082207522184700_ocae120-B24","doi-asserted-by":"crossref","first-page":"38","DOI":"10.3163\/1536-5050.101.1.007","article-title":"The value of library and information services in patient care: results of a multisite study","volume":"101","author":"Marshall","year":"2013","journal-title":"J Med Libr Assoc"},{"key":"2024082207522184700_ocae120-B25","first-page":"248","author":"Pal","year":"2022"},{"issue":"14","key":"2024082207522184700_ocae120-B26","first-page":"6421","volume":"11","author":"Jin","year":"2021"},{"key":"2024082207522184700_ocae120-B27","first-page":"2567","author":"Jin","year":"2019"},{"key":"2024082207522184700_ocae120-B28","author":"Taori","year":"2023"},{"key":"2024082207522184700_ocae120-B29","author":"Lian","year":"2023"},{"key":"2024082207522184700_ocae120-B30","author":"Hartford","year":"2023"},{"key":"2024082207522184700_ocae120-B31","first-page":"727","author":"Chang","year":"2021"},{"key":"2024082207522184700_ocae120-B32","author":"Mukherjee","year":"2023"},{"key":"2024082207522184700_ocae120-B33","author":"Mitra","year":"2023"},{"key":"2024082207522184700_ocae120-B34","author":"Touvron","year":"2023"},{"key":"2024082207522184700_ocae120-B35","author":"Team","year":"2023"},{"key":"2024082207522184700_ocae120-B36","author":"Jain","year":"2024."},{"key":"2024082207522184700_ocae120-B37","author":"Ronanki","year":"2023"},{"key":"2024082207522184700_ocae120-B38","unstructured":"Gao L, Tow J, Biderman S, et al \u00a0A framework for few-shot language model evaluation. Accessed May 13, 2021. https:\/\/doi.org\/10.5281\/zenodo.5371628"},{"issue":"9","key":"2024082207522184700_ocae120-B39","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1145\/3474381","article-title":"WinoGrande: an adversarial Winograd schema challenge at scale","volume":"64","author":"Sakaguchi","year":"2021","journal-title":"Commun ACM"},{"key":"2024082207522184700_ocae120-B40","first-page":"4791","author":"Zellers","year":"2019"},{"issue":"5","key":"2024082207522184700_ocae120-B41","doi-asserted-by":"crossref","first-page":"7432","DOI":"10.1609\/aaai.v34i05.6239","article-title":"PIQA: reasoning about physical commonsense in natural language","volume":"34","author":"Bisk","year":"2020","journal-title":"AAAI"},{"key":"2024082207522184700_ocae120-B42","article-title":"Can a suit of armor conduct electricity? A new dataset for open book question answering","author":"Mihaylov","year":"2018","journal-title":", Brussels, Belgium"},{"key":"2024082207522184700_ocae120-B43","first-page":"2924","author":"Clark"},{"key":"2024082207522184700_ocae120-B44","author":"Chollet"},{"key":"2024082207522184700_ocae120-B45","year":"2023"},{"key":"2024082207522184700_ocae120-B46","author":"Williams"},{"key":"2024082207522184700_ocae120-B47","author":"Nori","year":"2023"},{"key":"2024082207522184700_ocae120-B48","first-page":"9459","author":"Lewis","year":"2020"},{"key":"2024082207522184700_ocae120-B49","first-page":"2387","author":"Semnani"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/1875\/58868289\/ocae120.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/1875\/58868289\/ocae120.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,22]],"date-time":"2024-08-22T12:27:05Z","timestamp":1724329625000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/9\/1875\/7680487"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,23]]},"references-count":49,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,5,23]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae120","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,5,23]]}}}