{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T19:05:29Z","timestamp":1789585529048,"version":"build-2803163510"},"reference-count":36,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T00:00:00Z","timestamp":1781827200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","award":["R01LM011934"],"award-info":[{"award-number":["R01LM011934"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","award":["R01LM014508"],"award-info":[{"award-number":["R01LM014508"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000060","name":"National Institute of Allergy & Infectious Diseases","doi-asserted-by":"publisher","award":["R21AI164100"],"award-info":[{"award-number":["R21AI164100"]}],"id":[{"id":"10.13039\/100000060","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>Generative information extraction using large language models (LLMs), particularly through prompting combined with few-shot learning, has become a popular method. In many ways such prompts with examples resemble the annotation guidelines long used for manual labeling of data for information extraction, and indeed studies have demonstrated the direct use of these guidelines as effective prompts. However, constructing annotation guidelines is both labor- and knowledge-intensive. Instead, this paper proposes to leverage LLMs\u2019 impressive ability to automatically create such annotation guidelines.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>Specifically, we propose a zero-shot hierarchical prompt engineering method that harvests the knowledge summarization and text generation capacity of LLMs to synthesize annotation guidelines to improve downstream LLMs while requiring minimal human input.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Zero-shot clinical named entity recognition benchmarks, 2012 i2b2 EVENT, 2012 i2b2 TIMEX, 2014 i2b2, and 2018 n2c2 showed improvements of 0.2% to 25.86% for Llama 3.1 and 5.82% to 16.13% for GPT-OSS in strict F1 scores from the no-guideline baseline. The LLM-synthesized guidelines showed equivalent or better performance compared to human-written guidelines by 0.23% to 10.00% in most tasks.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>LLMs generate high-quality annotation guidelines following a consistent pattern (eg, title, entity types, examples) without human guidance, indicating that a representation of such a concept has been encoded during the pre-training. Nuances in definitions, however, still require adjustment by researchers to align with the project.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>This study proposes a novel hierarchical prompt engineering method that requires minimal knowledge transfer from a human expert and is applicable to multiple biomedical domains.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocag084","type":"journal-article","created":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T12:01:20Z","timestamp":1778760080000},"page":"1735-1747","source":"Crossref","is-referenced-by-count":0,"title":["Synthesized annotation guidelines are knowledge-lite boosters for clinical information extraction"],"prefix":"10.1093","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8137-0171","authenticated-orcid":false,"given":"Enshuo","family":"Hsu","sequence":"first","affiliation":[{"name":"McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]},{"name":"Enterprise Development and Integration, University of Texas MD Anderson Cancer Center , Houston, TX 77030,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martin","family":"Ugbala","sequence":"additional","affiliation":[{"name":"School of Dentistry, University of Texas Health Science Center at Houston , Houston, TX 77054,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Krishna Kumar","family":"Kookal","sequence":"additional","affiliation":[{"name":"School of Dentistry, University of Texas Health Science Center at Houston , Houston, TX 77054,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kawtar","family":"Zouaidi","sequence":"additional","affiliation":[{"name":"School of Dentistry, University of Texas Health Science Center at Houston , Houston, TX 77054,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicholas L","family":"Rider","sequence":"additional","affiliation":[{"name":"Department of Health Systems & Implementation Science, Virginia Tech Carilion School of Medicine , Roanoke, VA 24016,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Muhammad F","family":"Walji","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]},{"name":"School of Dentistry, University of Texas Health Science Center at Houston , Houston, TX 77054,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6525-5213","authenticated-orcid":false,"given":"Kirk","family":"Roberts","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,6,19]]},"reference":[{"issue":"6357.","key":"2026091614320212400_ocag084-B1","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1007\/s11704-024-40555-y","article-title":"Large language models for generative information extraction: a survey","volume":"18","author":"Xu","year":"2024","journal-title":"Front Comput Sci"},{"key":"2026091614320212400_ocag084-B2","first-page":"1998"},{"key":"2026091614320212400_ocag084-B3","first-page":"2049"},{"key":"2026091614320212400_ocag084-B4","doi-asserted-by":"publisher","first-page":"1812","DOI":"10.1093\/jamia\/ocad259","article-title":"Improving large language models for clinical named entity recognition via prompt engineering","volume":"31","author":"Hu","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2026091614320212400_ocag084-B5"},{"key":"2026091614320212400_ocag084-B6","doi-asserted-by":"crossref","first-page":"178","DOI":"10.18653\/v1\/2023.bionlp-1.15","volume-title":"The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks","author":"Meoni","year":"2023"},{"key":"2026091614320212400_ocag084-B7","doi-asserted-by":"publisher","first-page":"104478","DOI":"10.1016\/j.jbi.2023.104478","article-title":"Annotated dataset creation through large language models for non-English medical NLP","volume":"145","author":"Frei","year":"2023","journal-title":"J Biomed Inform"},{"key":"2026091614320212400_ocag084-B8","doi-asserted-by":"publisher","first-page":"1929","DOI":"10.1093\/jamia\/ocae095","article-title":"RT: a Retrieving and Chain-of-Thought framework for few-shot medical named entity recognition","volume":"31","author":"Li","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2026091614320212400_ocag084-B9","doi-asserted-by":"publisher","first-page":"8241","DOI":"10.1038\/s41598-024-68168-2","article-title":"Leveraging large language models for knowledge-free weak supervision in clinical natural language processing","volume":"15","author":"Hsu","year":"2025","journal-title":"Sci Rep"},{"key":"2026091614320212400_ocag084-B10","doi-asserted-by":"publisher","first-page":"806","DOI":"10.1136\/amiajnl-2013-001628","article-title":"Evaluating temporal relations in clinical text: 2012 i2b2 Challenge","volume":"20","author":"Sun","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2026091614320212400_ocag084-B11","doi-asserted-by":"publisher","first-page":"S11","DOI":"10.1016\/j.jbi.2015.06.007","article-title":"Automated systems for the de-identification of longitudinal clinical narratives: overview of 2014 i2b2\/UTHealth shared task Track 1","volume":"58 Suppl","author":"Stubbs","year":"2015","journal-title":"J Biomed Inform"},{"key":"2026091614320212400_ocag084-B12","volume":"27","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2026091614320212400_ocag084-B13","first-page":"47083"},{"key":"2026091614320212400_ocag084-B14"},{"key":"2026091614320212400_ocag084-B15"},{"key":"2026091614320212400_ocag084-B16","doi-asserted-by":"crossref","first-page":"15496","DOI":"10.18653\/v1\/2024.findings-acl.916","volume-title":"Findings of the Association for Computational Linguistics ACL 2024","author":"Xu","year":"2024"},{"key":"2026091614320212400_ocag084-B17","doi-asserted-by":"crossref","first-page":"15566","DOI":"10.18653\/v1\/2023.acl-long.868","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Wadhwa","year":"2023"},{"key":"2026091614320212400_ocag084-B18"},{"key":"2026091614320212400_ocag084-B19","first-page":"8241","year":"2025"},{"key":"2026091614320212400_ocag084-B20","doi-asserted-by":"publisher","first-page":"290","DOI":"10.1186\/s12859-023-05411-z","article-title":"Leveraging pre-trained language models for mining microbiome-disease relationships","volume":"24","author":"Karkera","year":"2023","journal-title":"BMC Bioinformatics"},{"key":"2026091614320212400_ocag084-B21"},{"key":"2026091614320212400_ocag084-B22","doi-asserted-by":"crossref","first-page":"145","DOI":"10.63317\/5mszwmbhujbq","volume-title":"Proceedings of the First Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC-COLING 2024","author":"Andrew","year":"2024"},{"key":"2026091614320212400_ocag084-B23","first-page":"15898"},{"key":"2026091614320212400_ocag084-B24"},{"key":"2026091614320212400_ocag084-B25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3458754","article-title":"Domain-specific language model pretraining for biomedical natural language processing","volume":"3","author":"Gu","year":"2022","journal-title":"ACM Trans Comput Healthcare"},{"key":"2026091614320212400_ocag084-B26","first-page":"182","author":"Yao","year":"2024"},{"key":"2026091614320212400_ocag084-B27","article-title":"pjmathematician at MultiClinSUM 2025: a novel automated prompt optimization framework for multilingual clinical summarization","author":"Vachharajani","year":"2025","journal-title":"CLEF"},{"key":"2026091614320212400_ocag084-B28"},{"key":"2026091614320212400_ocag084-B29","first-page":"2025","author":"Li","year":"2025"},{"key":"2026091614320212400_ocag084-B30"},{"key":"2026091614320212400_ocag084-B31","doi-asserted-by":"crossref","first-page":"701","DOI":"10.1093\/jamia\/ocac001","article-title":"BigMouth: development and maintenance of a successful dental data repository","volume":"29","author":"Walji","year":"2022","journal-title":"J Am Med Inform Assoc"},{"key":"2026091614320212400_ocag084-B32","doi-asserted-by":"crossref","first-page":"e540","DOI":"10.1097\/PTS.0000000000000407","article-title":"Classifying adverse events in the dental office","volume":"17","author":"Kalenderian","year":"2021","journal-title":"J Patient Saf"},{"key":"2026091614320212400_ocag084-B33","doi-asserted-by":"crossref","first-page":"470","DOI":"10.1097\/PTS.0000000000000959","article-title":"Building consensus for a shared definition of adverse events: a case study in the profession of dentistry","volume":"18","author":"Franklin","year":"2022","journal-title":"J Patient Saf"},{"key":"2026091614320212400_ocag084-B34"},{"key":"2026091614320212400_ocag084-B35","first-page":"611"},{"key":"2026091614320212400_ocag084-B36","author":"Ollama"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocag084\/68556722\/ocag084.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/9\/1735\/68556722\/ocag084.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/9\/1735\/68556722\/ocag084.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T18:32:11Z","timestamp":1789583531000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/33\/9\/1735\/8711444"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,19]]},"references-count":36,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2026,6,19]]},"published-print":{"date-parts":[[2026,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocag084","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,9]]},"published":{"date-parts":[[2026,6,19]]}}}