{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T06:32:58Z","timestamp":1783405978253,"version":"3.54.6"},"reference-count":52,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2024,4,19]],"date-time":"2024-04-19T00:00:00Z","timestamp":1713484800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>The objective of this study is to systematically examine the efficacy of both proprietary (GPT-3.5, GPT-4) and open-source large language models (LLMs) (LLAMA 7B, 13B, 70B) in the context of matching patients to clinical trials in healthcare.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and methods<\/jats:title>\n                  <jats:p>The study employs a multifaceted evaluation framework, incorporating extensive automated and human-centric assessments along with a detailed error analysis for each model, and assesses LLMs\u2019 capabilities in analyzing patient eligibility against clinical trial\u2019s inclusion and exclusion criteria. To improve the adaptability of open-source LLMs, a specialized synthetic dataset was created using GPT-4, facilitating effective fine-tuning under constrained data conditions.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The findings indicate that open-source LLMs, when fine-tuned on this limited and synthetic dataset, achieve performance parity with their proprietary counterparts, such as GPT-3.5.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>This study highlights the recent success of LLMs in the high-stakes domain of healthcare, specifically in patient-trial matching. The research demonstrates the potential of open-source models to match the performance of proprietary models when fine-tuned appropriately, addressing challenges like cost, privacy, and reproducibility concerns associated with closed-source proprietary LLMs.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>The study underscores the opportunity for open-source LLMs in patient-trial matching. To encourage further research and applications in this field, the annotated evaluation dataset and the fine-tuned LLM, Trial-LLAMA, are released for public use.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae073","type":"journal-article","created":{"date-parts":[[2024,4,20]],"date-time":"2024-04-20T00:56:45Z","timestamp":1713574605000},"page":"1953-1963","source":"Crossref","is-referenced-by-count":63,"title":["Distilling large language models for matching patients to clinical trials"],"prefix":"10.1093","volume":"31","author":[{"given":"Mauro","family":"Nievas","sequence":"first","affiliation":[{"name":"Triomics Research, Triomics, Inc. , San Francisco, CA 94105, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aditya","family":"Basu","sequence":"additional","affiliation":[{"name":"Triomics Research, Triomics, Inc. , Bengaluru, Karnataka 560102, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yanshan","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Health Information Management, University of Pittsburgh , Pittsburgh, PA 15260, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hrituraj","family":"Singh","sequence":"additional","affiliation":[{"name":"Triomics Research, Triomics, Inc. , San Francisco, CA 94105, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,4,19]]},"reference":[{"issue":"5","key":"2024082207511425800_ocae073-B1","doi-asserted-by":"crossref","first-page":"572","DOI":"10.1177\/2168479017751403","article-title":"Assessing study start-up practices, performance, and perceptions among sponsors and contract research organizations","volume":"52","author":"Lamberti","year":"2018","journal-title":"Ther Innov Regul Sci"},{"issue":"5","key":"2024082207511425800_ocae073-B2","first-page":"24","article-title":"Enrollment performance: weighing the \u201cfacts\u201d","volume":"21","author":"Getz","year":"2012","journal-title":"Appl Clin Trials"},{"issue":"2","key":"2024082207511425800_ocae073-B3","doi-asserted-by":"crossref","first-page":"djv324","DOI":"10.1093\/jnci\/djv324","article-title":"Predicting low accrual in the National Cancer Institute\u2019s Cooperative Group Clinical Trials","volume":"108","author":"Bennette","year":"2016","journal-title":"J Natl Cancer Inst"},{"issue":"10","key":"2024082207511425800_ocae073-B4","doi-asserted-by":"crossref","first-page":"1261","DOI":"10.2217\/fon-2015-0043","article-title":"Opportunities and challenges in leveraging electronic health record data in oncology","volume":"12","author":"Berger","year":"2016","journal-title":"Future Oncol"},{"issue":"10","key":"2024082207511425800_ocae073-B5","doi-asserted-by":"crossref","first-page":"990","DOI":"10.1016\/j.jclinepi.2006.12.013","article-title":"A systematic review highlights threats to validity in studies of barriers to cancer trial participation","volume":"60","author":"Fayter","year":"2007","journal-title":"J Clin Epidemiol"},{"issue":"6","key":"2024082207511425800_ocae073-B6","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1016\/j.cct.2009.06.009","article-title":"Recruiting minorities where they receive care: institutional barriers to cancer clinical trials recruitment in a safety-net hospital","volume":"30","author":"Joseph","year":"2009","journal-title":"Contemp Clin Trials"},{"issue":"3","key":"2024082207511425800_ocae073-B7","doi-asserted-by":"crossref","first-page":"137","DOI":"10.4103\/2229-3485.184820","article-title":"Challenges in recruitment and retention of clinical trial subjects","volume":"7","author":"Kadam","year":"2016","journal-title":"Perspect Clin Res"},{"key":"2024082207511425800_ocae073-B8","volume-title":"Patient Recruitment Shortcomings Laid at Feet of Poor Provider, Researcher Engagement","author":"Association of Clinical Research Professionals","year":"2017"},{"issue":"3","key":"2024082207511425800_ocae073-B9","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1016\/j.cct.2010.03.005","article-title":"Automated matching software for clinical trials eligibility: measuring efficiency and flexibility","volume":"31","author":"Penberthy","year":"2010","journal-title":"Contemp Clin Trials"},{"issue":"6","key":"2024082207511425800_ocae073-B10","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1200\/JOP.2012.000646","article-title":"Effort required in eligibility screening for clinical trials","volume":"8","author":"Penberthy","year":"2012","journal-title":"J Oncol Pract"},{"key":"2024082207511425800_ocae073-B11","volume-title":"Generating Medical Logic Modules for Clinical Trial Eligibility","author":"Parker","year":"2005"},{"issue":"11","key":"2024082207511425800_ocae073-B12","doi-asserted-by":"crossref","first-page":"1181","DOI":"10.1093\/jamia\/ocz139","article-title":"Cohort selection for clinical trials using deep learning models","volume":"26","author":"Segura-Bedmar","year":"2019","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"2024082207511425800_ocae073-B13","doi-asserted-by":"crossref","first-page":"294","DOI":"10.1093\/jamia\/ocy178","article-title":"Criteria2Query: a natural language interface to clinical databases for cohort definition","volume":"26","author":"Yuan","year":"2019","journal-title":"J Am Med Inform Assoc"},{"issue":"7972","key":"2024082207511425800_ocae073-B14","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"2024082207511425800_ocae073-B15","author":"Nori","year":"2023"},{"key":"2024082207511425800_ocae073-B16","author":"Jin","year":"2023"},{"issue":"7","key":"2024082207511425800_ocae073-B17","doi-asserted-by":"crossref","first-page":"1161","DOI":"10.1093\/jamia\/ocac051","article-title":"Combining human and machine intelligence for clinical trial eligibility querying","volume":"29","author":"Fang","year":"2022","journal-title":"J Am Med Inform Assoc"},{"key":"2024082207511425800_ocae073-B18","first-page":"669","author":"Koopman","year":"2016"},{"key":"2024082207511425800_ocae073-B19","first-page":"2325","author":"Pradeep","year":"2022"},{"key":"2024082207511425800_ocae073-B20","author":"Roberts","year":"2021"},{"key":"2024082207511425800_ocae073-B21","volume-title":"Proceedings of the Thirty-First Text REtrieval Conference, TREC 2022, Online, November 15-19, 2022","author":"Roberts","year":"2022"},{"key":"2024082207511425800_ocae073-B22","first-page":"4249","author":"Rybinski","year":"2023"},{"issue":"e1","key":"2024082207511425800_ocae073-B23","doi-asserted-by":"crossref","first-page":"e141","DOI":"10.1093\/jamia\/ocu050","article-title":"Case-based reasoning using electronic health records efficiently identifies eligible patients for clinical trials","volume":"22","author":"Miotto","year":"2015","journal-title":"J Am Med Inform Assoc"},{"issue":"6","key":"2024082207511425800_ocae073-B24","doi-asserted-by":"crossref","first-page":"869","DOI":"10.1197\/jamia.M3119","article-title":"Electronic screening improves efficiency in clinical trial recruitment","volume":"16","author":"Thadani","year":"2009","journal-title":"J Am Med Inform Assoc"},{"issue":"Suppl 1","key":"2024082207511425800_ocae073-B25","doi-asserted-by":"crossref","first-page":"i116","DOI":"10.1136\/amiajnl-2011-000321","article-title":"ELiXR: an approach to eligibility criteria extraction and representation","volume":"18","author":"Weng","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2024082207511425800_ocae073-B26","year":"2023"},{"key":"2024082207511425800_ocae073-B27","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"2024082207511425800_ocae073-B28","first-page":"1","author":"Chowdhery"},{"key":"2024082207511425800_ocae073-B29","author":"Touvron","year":"2023"},{"key":"2024082207511425800_ocae073-B30","author":"Tian"},{"issue":"1","key":"2024082207511425800_ocae073-B31","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1038\/s41746-023-00896-7","article-title":"Evaluating large language models on medical evidence summarization","volume":"6","author":"Tang","year":"2023","journal-title":"NPJ Digit Med"},{"key":"2024082207511425800_ocae073-B32","author":"Van Veen","year":"2023"},{"issue":"6","key":"2024082207511425800_ocae073-B33","doi-asserted-by":"crossref","first-page":"3197","DOI":"10.1007\/s11845-023-03377-8","article-title":"GPT-4: a new era of artificial intelligence in medicine","volume":"192","author":"Waisberg","year":"2023","journal-title":"Ir J Med Sci"},{"key":"2024082207511425800_ocae073-B34","author":"Yang"},{"key":"2024082207511425800_ocae073-B35","author":"Gao","year":"2023"},{"key":"2024082207511425800_ocae073-B36","author":"Agrawal","year":"2022"},{"key":"2024082207511425800_ocae073-B37","author":"Alazraki","year":"2023"},{"key":"2024082207511425800_ocae073-B38","author":"Chung","year":"2022"},{"key":"2024082207511425800_ocae073-B39","first-page":"3470","author":"Mishra","year":"2022"},{"key":"2024082207511425800_ocae073-B40","author":"Zhang","year":"2023"},{"key":"2024082207511425800_ocae073-B41","first-page":"22021","author":"Fleming"},{"key":"2024082207511425800_ocae073-B42","author":"Zhou","year":"2023"},{"key":"2024082207511425800_ocae073-B43","author":"Chiang","year":"2023"},{"key":"2024082207511425800_ocae073-B44","author":"Peng","year":"2023"},{"key":"2024082207511425800_ocae073-B45","volume-title":"Alpaca: A Strong, Replicable Instruction-Following Model","author":"Taori","year":"2023"},{"key":"2024082207511425800_ocae073-B46","author":"Tunstall","year":"2023"},{"key":"2024082207511425800_ocae073-B47","author":"Wang","year":"2022"},{"key":"2024082207511425800_ocae073-B48","author":"Hoffmann","year":"2022"},{"key":"2024082207511425800_ocae073-B49","first-page":"preprint: not peer reviewed","article-title":"How abilities in large language models are affected by supervised fine-tuning data composition","author":"Dong","year":"2023"},{"key":"2024082207511425800_ocae073-B50","author":"Bai","year":"2022"},{"key":"2024082207511425800_ocae073-B51","author":"Rafailov"},{"key":"2024082207511425800_ocae073-B52","author":"Jiang","year":"2023"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/1953\/58868048\/ocae073.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/1953\/58868048\/ocae073.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,22]],"date-time":"2024-08-22T10:45:09Z","timestamp":1724323509000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/9\/1953\/7654018"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,19]]},"references-count":52,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,4,19]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae073","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,4,19]]}}}