{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,11]],"date-time":"2026-02-11T04:04:35Z","timestamp":1770782675050,"version":"3.50.0"},"reference-count":51,"publisher":"Oxford University Press (OUP)","issue":"7","license":[{"start":{"date-parts":[[2025,6,3]],"date-time":"2025-06-03T00:00:00Z","timestamp":1748908800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01LM011934"],"award-info":[{"award-number":["R01LM011934"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,7,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Unlocking clinical information embedded in clinical notes has been hindered to a significant degree by domain-specific and context-sensitive language. Identification of note sections and structural document elements has been shown to improve information extraction and dependent downstream clinical natural language processing (NLP) tasks and applications. This study investigates the viability of a dynamic example selection prompting method to section classification using lightweight, open-source large language models (LLMs) as a practical solution for real-world healthcare clinical NLP systems.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We develop a dynamic few-shot prompting approach to classifying sections where section samples are first embedded using a transformer-based model and deposited in a vector store. During inference, the embedded samples with the most similar contextual embeddings to a given input section text are retrieved from the vector store and inserted into the LLM prompt. We evaluate this technique on two datasets comprising two section schemas, including varying levels of context. We compare the performance to baseline zero-shot and randomly selected few-shot scenarios.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The dynamic few-shot prompting experiments yielded the highest F1 scores in each of the classification tasks and datasets for all seven of the LLMs included in the evaluation, averaging a macro F1 increase of 39.3% and 21.1% in our primary section classification task over the zero-shot and static few-shot baselines, respectively.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion and Conclusion<\/jats:title>\n                  <jats:p>Our results showcase substantial performance improvements imparted by dynamically selecting examples for few-shot LLM prompting, and further improvement by including section context, demonstrating compelling potential for clinical applications.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf084","type":"journal-article","created":{"date-parts":[[2025,6,3]],"date-time":"2025-06-03T16:56:53Z","timestamp":1748969813000},"page":"1164-1173","source":"Crossref","is-referenced-by-count":2,"title":["Dynamic few-shot prompting for clinical note section classification using lightweight, open-source large language models"],"prefix":"10.1093","volume":"32","author":[{"given":"Kurt","family":"Miller","sequence":"first","affiliation":[{"name":"Bioinformatics and Computational Biology Program, University of Minnesota , Rochester, MN,","place":["United States"]},{"name":"Center for Digital Health, Mayo Clinic , Rochester, MN,","place":["United States"]}]},{"given":"Steven","family":"Bedrick","sequence":"additional","affiliation":[{"name":"Department of Medical Informatics & Clinical Epidemiology, Oregon Health and Science University , Portland, OR,","place":["United States"]}]},{"given":"Qiuhao","family":"Lu","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston , Houston, TX,","place":["United States"]}]},{"given":"Andrew","family":"Wen","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston , Houston, TX,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4114-5148","authenticated-orcid":false,"given":"William","family":"Hersh","sequence":"additional","affiliation":[{"name":"Department of Medical Informatics & Clinical Epidemiology, Oregon Health and Science University , Portland, OR,","place":["United States"]}]},{"given":"Kirk","family":"Roberts","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston , Houston, TX,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2570-3741","authenticated-orcid":false,"given":"Hongfang","family":"Liu","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston , Houston, TX,","place":["United States"]}]}],"member":"286","published-online":{"date-parts":[[2025,6,3]]},"reference":[{"key":"2025062707350022600_ocaf084-B1","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1007\/s11606-015-3512-2","article-title":"A systematic review of conceptual frameworks of medical complexity and new model development","volume":"31","author":"Zullig","year":"2016","journal-title":"J Gen Internal Med"},{"key":"2025062707350022600_ocaf084-B2","doi-asserted-by":"publisher","first-page":"406","DOI":"10.1136\/bmjqs-2013-002194","article-title":"Association of note quality and quality of care: a cross-sectional study","volume":"23","author":"Edwards","year":"2014","journal-title":"BMJ Qual Saf"},{"key":"2025062707350022600_ocaf084-B3","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1186\/s12911-020-1072-9","article-title":"Assessment of the impact of EHR heterogeneity for clinical research through a case study of silent brain infarction","volume":"20","author":"Fu","year":"2020","journal-title":"BMC Med Inf Decis Making"},{"key":"2025062707350022600_ocaf084-B4","doi-asserted-by":"publisher","first-page":"e2115334","DOI":"10.1001\/jamanetworkopen.2021.15334","article-title":"Length and redundancy of outpatient progress notes across a decade at an academic medical center","volume":"4","author":"Rule","year":"2021","journal-title":"JAMA Netw Open"},{"key":"2025062707350022600_ocaf084-B5","doi-asserted-by":"crossref","first-page":"103938","DOI":"10.1016\/j.jbi.2021.103938","article-title":"Estimating redundancy in clinical text","volume":"124","author":"Searle","year":"2021","journal-title":"J Biomed Inf"},{"key":"2025062707350022600_ocaf084-B6","doi-asserted-by":"publisher","first-page":"958539","DOI":"10.3389\/fdgth.2022.958539","article-title":"Quality assessment of functional status documentation in EHRs across different healthcare institutions","volume":"4","author":"Fu","year":"2022","journal-title":"Front Digital Health"},{"key":"2025062707350022600_ocaf084-B7","first-page":"1155","article-title":"Contextual variation of clinical notes induced by EHR migration","volume":"2023","author":"Miller","year":"2023","journal-title":"AMIA Annu Symp Proc"},{"key":"2025062707350022600_ocaf084-B8","doi-asserted-by":"publisher","first-page":"7656","DOI":"10.1038\/s41598-024-56324-7","article-title":"Information heterogeneity between progress notes by physicians and nurses for inpatients with digestive system diseases","volume":"14","author":"Mashima","year":"2024","journal-title":"Sci Rep"},{"key":"2025062707350022600_ocaf084-B9","doi-asserted-by":"publisher","first-page":"808","DOI":"10.1136\/amiajnl-2013-002381","article-title":"A comprehensive study of named entity recognition in Chinese clinical text","volume":"21","author":"Lei","year":"2014","journal-title":"J Am Med Inf Assoc"},{"key":"2025062707350022600_ocaf084-B10","first-page":"660","article-title":"Evaluation of clinical text segmentation to facilitate cohort retrieval","volume":"2017","author":"Edinger","year":"2017","journal-title":"AMIA Annu Symp Proc"},{"key":"2025062707350022600_ocaf084-B11","doi-asserted-by":"publisher","first-page":"20552076211057662","DOI":"10.1177\/20552076211057662","article-title":"Automatic extraction of 12 cardiovascular concepts from German discharge letters using pre-trained language models","volume":"7","author":"Richter-Pechanski","year":"2021","journal-title":"Digital Health"},{"key":"2025062707350022600_ocaf084-B12","doi-asserted-by":"publisher","first-page":"528","DOI":"10.1136\/jamia.2010.003855","article-title":"Integrating existing natural language processing tools for medication extraction from discharge summaries","volume":"17","author":"Doan","year":"2010","journal-title":"J Am Med Inf Assoc"},{"key":"2025062707350022600_ocaf084-B13","author":"A Supervised Abbreviation Resolution System for Medical Text","year":"2013"},{"key":"2025062707350022600_ocaf084-B14","doi-asserted-by":"publisher","first-page":"552","DOI":"10.1136\/jamia.2001.0080552","article-title":"The HL7 clinical document architecture","volume":"8","author":"Dolin","year":"2001","journal-title":"J Am Med Inf Assoc"},{"key":"2025062707350022600_ocaf084-B15","first-page":"1099","article-title":"Document clustering of clinical narratives: a systematic study of clinical sublanguages","volume":"2011","author":"Patterson","year":"2011","journal-title":"AMIA Annu Symp Proc"},{"key":"2025062707350022600_ocaf084-B16","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1145\/2512089.2512101","article-title":"Document sublanguage clustering to detect medical specialty in cross-institutional clinical texts","volume":"2013","author":"Doing-Harris","year":"2013","journal-title":"Proc ACM Int Workshop Data Text Min Biomed Inform"},{"key":"2025062707350022600_ocaf084-B17","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1093\/jamia\/ocx138","article-title":"Clinical documentation variations and NLP system portability: a case study in asthma birth cohorts across institutions","volume":"25","author":"Sohn","year":"2018","journal-title":"J Am Med Inf Assoc"},{"key":"2025062707350022600_ocaf084-B18","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1186\/s12874-019-0792-y","article-title":"Current approaches to identify sections within clinical narratives from electronic health records: a systematic review","volume":"19","author":"Pomares-Quimbaya","year":"2019","journal-title":"BMC Med Res Methodol"},{"key":"2025062707350022600_ocaf084-B19","doi-asserted-by":"publisher","first-page":"2114","DOI":"10.1093\/jamia\/ocae074","article-title":"Large language models for biomedicine: foundations, opportunities, challenges, and best practices","volume":"31","author":"Sahoo","year":"2024","journal-title":"J Am Med Inf Assoc"},{"key":"2025062707350022600_ocaf084-B20","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1093\/jamia\/ocad190","article-title":"Improving model transferability for clinical note section classification models using continued pretraining","volume":"31","author":"Zhou","year":"2023","journal-title":"J Am Med Inf Assoc"},{"key":"2025062707350022600_ocaf084-B21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3560815","article-title":"Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing","volume":"55","author":"Liu","year":"2023","journal-title":"ACM Comput. Surv"},{"key":"2025062707350022600_ocaf084-B22","doi-asserted-by":"publisher","first-page":"e55318","DOI":"10.2196\/55318","article-title":"An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing: algorithm development and validation study","volume":"12","author":"Sivarajkumar","year":"2024","journal-title":"JMIR Med Inf"},{"key":"2025062707350022600_ocaf084-B23","author":"Agrawal","year":"2022"},{"key":"2025062707350022600_ocaf084-B24","author":"Brown","year":"2020"},{"key":"2025062707350022600_ocaf084-B25","doi-asserted-by":"publisher","first-page":"16453","DOI":"10.1109\/TNNLS.2023.3294633","article-title":"Clinical prompt learning with frozen language models","volume":"35","author":"Taylor","year":"2024","journal-title":"IEEE Trans Neural Networks Learn Syst"},{"key":"2025062707350022600_ocaf084-B26","author":"Hegselmann","year":"2022"},{"key":"2025062707350022600_ocaf084-B27","author":"Cui","year":"2024"},{"key":"2025062707350022600_ocaf084-B28","author":"Gero","year":"2023"},{"key":"2025062707350022600_ocaf084-B29","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1038\/s41746-023-00970-0","article-title":"Large language models to identify social determinants of health in electronic health records","volume":"7","author":"Guevara","year":"2024","journal-title":"npj Digital Med"},{"key":"2025062707350022600_ocaf084-B30","doi-asserted-by":"crossref","first-page":"104458","DOI":"10.1016\/j.jbi.2023.104458","article-title":"Few-shot learning for medical text: a review of advances, trends, and opportunities","volume":"144","author":"Ge","year":"2023","journal-title":"J Biomed Inf"},{"key":"2025062707350022600_ocaf084-B31","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"2025062707350022600_ocaf084-B32","author":"What Makes Good In-Context Examples for GPT-3?","year":"2022"},{"key":"2025062707350022600_ocaf084-B33","author":"D'Abramo","year":"2024"},{"key":"2025062707350022600_ocaf084-B34","doi-asserted-by":"crossref","first-page":"1134","DOI":"10.1038\/s41591-024-02855-5","article-title":"Adapted large language models can outperform medical experts in clinical text summarization","volume":"30","author":"Van Veen","year":"2024","journal-title":"Nat Med"},{"key":"2025062707350022600_ocaf084-B35","author":"SummQA at MEDIQA-Chat 2023: In-Context Learning with GPT-4 for Medical Summarization","year":"2023"},{"key":"2025062707350022600_ocaf084-B36","author":"Berkowitz"},{"key":"2025062707350022600_ocaf084-B37","doi-asserted-by":"publisher","first-page":"1929","DOI":"10.1093\/jamia\/ocae095","article-title":"RT: a retrieving and Chain-of-Thought framework for few-shot medical named entity recognition","volume":"31","author":"Li","year":"2024","journal-title":"J Am Med Inf Assoc"},{"key":"2025062707350022600_ocaf084-B38","author":"KnowLab_AIMed at MEDIQA-CORR 2024: Chain-of-Though (CoT) prompting strategies for medical error detection and correction","year":"2024"},{"key":"2025062707350022600_ocaf084-B39","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"2025062707350022600_ocaf084-B40","first-page":"5484","article-title":"Hierarchical annotation for building a suite of clinical natural language processing tasks: progress note understanding","volume":"2022","author":"Gao","year":"2022","journal-title":"LREC Int Conf Lang Resour Eval"},{"key":"2025062707350022600_ocaf084-B41","author":"Gao"},{"key":"2025062707350022600_ocaf084-B42","doi-asserted-by":"publisher","first-page":"e21929","DOI":"10.2196\/21929","article-title":"The fast health interoperability resources (FHIR) standard: systematic literature review of implementations, applications, challenges and opportunities","volume":"9","author":"Ayaz","year":"2021","journal-title":"JMIR Med Inf"},{"key":"2025062707350022600_ocaf084-B43","author":"Touvron","year":"2023"},{"key":"2025062707350022600_ocaf084-B44","author":"Dubey","year":"2024"},{"key":"2025062707350022600_ocaf084-B45","author":"Jiang","year":"2023"},{"key":"2025062707350022600_ocaf084-B46","author":"Bleys","year":"2023"},{"key":"2025062707350022600_ocaf084-B47","author":"Mistral-7B-v0.3 repository on HuggingFace"},{"key":"2025062707350022600_ocaf084-B48","author":"Team","year":"2024"},{"key":"2025062707350022600_ocaf084-B49","author":"Zhang","year":"2023"},{"key":"2025062707350022600_ocaf084-B50","doi-asserted-by":"publisher","first-page":"1821","DOI":"10.1093\/jamia\/ocae122","article-title":"BioInstruct: instruction tuning of large language models for biomedical natural language processing","volume":"31","author":"Tran","year":"2024","journal-title":"J Am Med Inf Assoc"},{"key":"2025062707350022600_ocaf084-B51","author":"Bleys","year":"2023"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/7\/1164\/63428244\/ocaf084.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/7\/1164\/63428244\/ocaf084.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,27]],"date-time":"2025-06-27T11:35:09Z","timestamp":1751024109000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/7\/1164\/8155977"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,3]]},"references-count":51,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2025,6,3]]},"published-print":{"date-parts":[[2025,7,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf084","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,7]]},"published":{"date-parts":[[2025,6,3]]}}}