{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T06:47:39Z","timestamp":1775198859061,"version":"3.50.1"},"reference-count":42,"publisher":"Oxford University Press (OUP)","issue":"10","license":[{"start":{"date-parts":[[2024,6,20]],"date-time":"2024-06-20T00:00:00Z","timestamp":1718841600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["P30CA082103"],"award-info":[{"award-number":["P30CA082103"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000038","name":"FDA","doi-asserted-by":"publisher","award":["U01FD005978"],"award-info":[{"award-number":["U01FD005978"]}],"id":[{"id":"10.13039\/100000038","id-type":"DOI","asserted-by":"publisher"}]},{"name":"UCSF\u2013Stanford Center of Excellence in Regulatory Sciences and Innovation"},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["UL1 TR001872"],"award-info":[{"award-number":["UL1 TR001872"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,10,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Although supervised machine learning is popular for information extraction from clinical notes, creating large annotated datasets requires extensive domain expertise and is time-consuming. Meanwhile, large language models (LLMs) have demonstrated promising transfer learning capability. In this study, we explored whether recent LLMs could reduce the need for large-scale data annotations.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We curated a dataset of 769 breast cancer pathology reports, manually labeled with 12 categories, to compare zero-shot classification capability of the following LLMs: GPT-4, GPT-3.5, Starling, and ClinicalCamel, with task-specific supervised classification performance of 3 models: random forests, long short-term memory networks with attention (LSTM-Att), and the UCSF-BERT model.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Across all 12 tasks, the GPT-4 model performed either significantly better than or as well as the best supervised model, LSTM-Att (average macro F1-score of 0.86 vs 0.75), with advantage on tasks with high label imbalance. Other LLMs demonstrated poor performance. Frequent GPT-4 error categories included incorrect inferences from multiple samples and from history, and complex task design, and several LSTM-Att errors were related to poor generalization to the test set.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>On tasks where large annotated datasets cannot be easily collected, LLMs can reduce the burden of data labeling. However, if the use of LLMs is prohibitive, the use of simpler models with large annotated datasets can provide comparable results.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>GPT-4 demonstrated the potential to speed up the execution of clinical NLP studies by reducing the need for large annotated datasets. This may increase the utilization of NLP-based variables and outcomes in clinical studies.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae146","type":"journal-article","created":{"date-parts":[[2024,6,20]],"date-time":"2024-06-20T15:18:48Z","timestamp":1718896728000},"page":"2315-2327","source":"Crossref","is-referenced-by-count":35,"title":["A comparative study of large language model-based zero-shot inference and task-specific supervised classification of breast cancer pathology reports"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7884-0526","authenticated-orcid":false,"given":"Madhumita","family":"Sushil","sequence":"first","affiliation":[{"name":"Bakar Computational Health Sciences Institute, University of California, San Francisco , San Francisco, CA 94158, United States"}]},{"given":"Travis","family":"Zack","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute, University of California, San Francisco , San Francisco, CA 94158, United States"},{"name":"Helen Diller Family Comprehensive Cancer Center, University of California, San Francisco , San Francisco, CA 94158, United States"}]},{"given":"Divneet","family":"Mandair","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute, University of California, San Francisco , San Francisco, CA 94158, United States"},{"name":"Helen Diller Family Comprehensive Cancer Center, University of California, San Francisco , San Francisco, CA 94158, United States"}]},{"given":"Zhiwei","family":"Zheng","sequence":"additional","affiliation":[{"name":"University of California, Berkeley , Berkeley, CA 94720, United States"}]},{"given":"Ahmed","family":"Wali","sequence":"additional","affiliation":[{"name":"University of California, Berkeley , Berkeley, CA 94720, United States"}]},{"given":"Yan-Ning","family":"Yu","sequence":"additional","affiliation":[{"name":"University of California, Berkeley , Berkeley, CA 94720, United States"}]},{"given":"Yuwei","family":"Quan","sequence":"additional","affiliation":[{"name":"University of California, Berkeley , Berkeley, CA 94720, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0543-0758","authenticated-orcid":false,"given":"Dmytro","family":"Lituiev","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute, University of California, San Francisco , San Francisco, CA 94158, United States"}]},{"given":"Atul J","family":"Butte","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute, University of California, San Francisco , San Francisco, CA 94158, United States"},{"name":"Helen Diller Family Comprehensive Cancer Center, University of California, San Francisco , San Francisco, CA 94158, United States"},{"name":"Center for Data-driven Insights and Innovation, University of California, Office of the President , Oakland, CA 94607, United States"},{"name":"Department of Pediatrics, University of California, San Francisco , San Francisco, CA 94158, United States"}]}],"member":"286","published-online":{"date-parts":[[2024,6,20]]},"reference":[{"issue":"1","key":"2024092007534398000_ocae146-B1","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1038\/s41746-022-00730-6","article-title":"A survey on clinical natural language processing in the United Kingdom from 2007 to 2022","volume":"5","author":"Wu","year":"2022","journal-title":"Digit Med"},{"issue":"3","key":"2024092007534398000_ocae146-B2","doi-asserted-by":"crossref","first-page":"398","DOI":"10.1111\/cts.13463","article-title":"Recommended practices and ethical considerations for natural language processing-assisted observational research: a scoping review","volume":"16","author":"Fu","year":"2023","journal-title":"Clin Transl Sci"},{"key":"2024092007534398000_ocae146-B3","first-page":"1877","volume-title":"Advances in Neural Information Processing Systems","author":"Brown","year":"2020"},{"key":"2024092007534398000_ocae146-B4","first-page":"22199","article-title":". Large language models are zero-shot reasoners","volume":"35","author":"Kojima","year":"2022","journal-title":"Adv Neural Inform Process Syst"},{"key":"2024092007534398000_ocae146-B5","author":"Agrawal","year":"2022"},{"issue":"1","key":"2024092007534398000_ocae146-B6","doi-asserted-by":"crossref","first-page":"AIp2300031","DOI":"10.1056\/AIp2300031","article-title":"Use of GPT-4 to diagnose complex clinical cases","volume":"1","author":"Eriksen","year":"2023","journal-title":"NEJM AI"},{"key":"2024092007534398000_ocae146-B7","author":"Wang","year":"2023"},{"issue":"3","key":"2024092007534398000_ocae146-B8","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1001\/jamapediatrics.2023.5750","article-title":"Diagnostic accuracy of a large language model in pediatric case studies","volume":"178","author":"Barile","year":"2024","journal-title":"JAMA Pediatr"},{"key":"2024092007534398000_ocae146-B9","author":"Nori"},{"issue":"1","key":"2024092007534398000_ocae146-B10","doi-asserted-by":"crossref","first-page":"16492","DOI":"10.1038\/s41598-023-43436-9","article-title":"Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments","volume":"13","author":"Brin","year":"2023","journal-title":"Sci Rep"},{"key":"2024092007534398000_ocae146-B11","doi-asserted-by":"publisher","first-page":"14414","DOI":"10.18653\/v1\/2023.emnlp-main.891","author":"Liu","year":"2023"},{"issue":"3","key":"2024092007534398000_ocae146-B12","doi-asserted-by":"crossref","first-page":"e231362","DOI":"10.1148\/radiol.231362","article-title":"Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancer","volume":"308","author":"Fink","year":"2023","journal-title":"Radiology"},{"issue":"1","key":"2024092007534398000_ocae146-B13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-023-00957-x","article-title":"Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models","volume":"6","author":"Alsentzer","year":"2023","journal-title":"NPJ Digit Med"},{"issue":"1","key":"2024092007534398000_ocae146-B14","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-023-00970-0","article-title":"Large language models to identify social determinants of health in electronic health records","volume":"7","author":"Guevara","year":"2024","journal-title":"NPJ Digit Med"},{"issue":"4","key":"2024092007534398000_ocae146-B15","doi-asserted-by":"publisher","first-page":"AIdbp2300110","DOI":"10.1056\/AIdbp2300110","article-title":"CORAL: expert-curated oncology reports to advance language model inference","volume":"1","author":"Sushil","year":"2024","journal-title":"NEJM AI"},{"issue":"3","key":"2024092007534398000_ocae146-B16","doi-asserted-by":"crossref","first-page":"310","DOI":"10.1002\/path.6232","article-title":"Extracting structured information from unstructured histopathology reports using generative pre-trained transformer 4 (GPT-4)","volume":"262","author":"Truhn","year":"2024","journal-title":"J Pathol"},{"key":"2024092007534398000_ocae146-B17","first-page":"846","author":"Wong","year":"2023"},{"issue":"2","key":"2024092007534398000_ocae146-B18","doi-asserted-by":"crossref","first-page":"375","DOI":"10.1093\/jamia\/ocad218","article-title":"AutoCriteria: a generalizable clinical trial eligibility criteria extraction system powered by large language models","volume":"31","author":"Datta","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2024092007534398000_ocae146-B19","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocad259","article-title":"Improving large language models for clinical named entity recognition via prompt engineering","author":"Hu","year":"2024","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2024092007534398000_ocae146-B20","doi-asserted-by":"crossref","first-page":"e243201","DOI":"10.1001\/jamanetworkopen.2024.3201","article-title":"Artificial intelligence\u2014generated draft replies to patient inbox messages","volume":"7","author":"Garcia","year":"2024","journal-title":"JAMA Netw Open"},{"issue":"6","key":"2024092007534398000_ocae146-B21","doi-asserted-by":"publisher","first-page":"1341","DOI":"10.1093\/jamia\/ocae067","article-title":"Can large language models provide secondary reliable opinion on treatment options for dermatological diseases?","volume":"31","author":"Iqbal","year":"2024","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"2024092007534398000_ocae146-B22","doi-asserted-by":"crossref","first-page":"AIcs2300145","DOI":"10.1056\/AIcs2300145","article-title":"Using ChatGPT to facilitate truly informed medical consent","volume":"1","author":"Mirza","year":"2024","journal-title":"NEJM AI"},{"issue":"3","key":"2024092007534398000_ocae146-B23","doi-asserted-by":"crossref","first-page":"e240357","DOI":"10.1001\/jamanetworkopen.2024.0357","article-title":"Generative artificial intelligence to transform inpatient discharge summaries to patient-friendly language and format","volume":"7","author":"Zaretsky","year":"2024","journal-title":"JAMA Netw Open"},{"issue":"3","key":"2024092007534398000_ocae146-B24","doi-asserted-by":"crossref","first-page":"ooad045","DOI":"10.1093\/jamiaopen\/ooad045","article-title":"A certified de-identification system for all clinical text documents for information extraction at scale","volume":"6","author":"Radhakrishnan","year":"2023","journal-title":"JAMIA Open"},{"issue":"3","key":"2024092007534398000_ocae146-B25","doi-asserted-by":"crossref","first-page":"431","DOI":"10.1093\/jamiaopen\/ooaa029","article-title":"Natural language processing systems for pathology parsing in limited data environments with uncertainty estimation","volume":"3","author":"Odisho","year":"2020","journal-title":"JAMIA Open"},{"issue":"1","key":"2024092007534398000_ocae146-B26","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1007\/s10278-018-0105-8","article-title":"Large scale semi-automated labeling of routine free-text clinical records for deep learning","volume":"32","author":"Trivedi","year":"2019","journal-title":"J Digit Imaging"},{"key":"2024092007534398000_ocae146-B27","year":"2024"},{"key":"2024092007534398000_ocae146-B28","year":"2023"},{"key":"2024092007534398000_ocae146-B29","year":"2024"},{"key":"2024092007534398000_ocae146-B30","author":"Toma","year":"2023"},{"key":"2024092007534398000_ocae146-B31","author":"Wolf","year":"2020"},{"issue":"2","key":"2024092007534398000_ocae146-B32","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1080\/00223980.1969.10543491","article-title":"Approximate randomization tests","volume":"72","author":"Edgington","year":"1969","journal-title":"J Psychol"},{"key":"2024092007534398000_ocae146-B33","volume-title":"Content Analysis: An Introduction to Its Methodology","author":"Krippendorff","year":"2018"},{"key":"2024092007534398000_ocae146-B34","author":"Sushil","year":"2022"},{"key":"2024092007534398000_ocae146-B35","doi-asserted-by":"crossref","first-page":"108189","DOI":"10.1016\/j.compbiomed.2024.108189","article-title":"A comprehensive evaluation of large language models on benchmark biomedical text processing tasks","volume":"171(1527-974X)","author":"Jahan","year":"2024","journal-title":"Comput Biol Med"},{"key":"2024092007534398000_ocae146-B36","author":"Chen","year":"2024"},{"issue":"10","key":"2024092007534398000_ocae146-B37","doi-asserted-by":"crossref","first-page":"1797","DOI":"10.1093\/jamia\/ocac127","article-title":"A scoping review of publicly available language tasks in clinical natural language processing","volume":"29","author":"Gao","year":"2022","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2024092007534398000_ocae146-B38","doi-asserted-by":"crossref","first-page":"18562","DOI":"10.1038\/s41598-023-45837-2","article-title":"Comparative performance of humans versus GPT-4.0 and GPT-3.5 in the self-assessment program of American Academy of Ophthalmology","volume":"13","author":"Taloni","year":"2023","journal-title":"Sci Rep"},{"key":"2024092007534398000_ocae146-B39","author":"Nori","year":"2023"},{"key":"2024092007534398000_ocae146-B40","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1162\/tacl_a_00638","article-title":"Lost in the middle: how language models use long contexts","volume":"12(2307-387X)","author":"Liu","year":"2024","journal-title":"Trans Assoc Computat Linguist"},{"key":"2024092007534398000_ocae146-B41","doi-asserted-by":"publisher","first-page":"e2200006","DOI":"10.1200\/CCI.22.00006","article-title":"Assessment of electronic health record for cancer research and patient care through a scoping review of cancer natural language processing","volume":"6","author":"Wang","year":"2022","journal-title":"JCO Clin Cancer Inform"},{"issue":"1","key":"2024092007534398000_ocae146-B42","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1186\/s12859-023-05480-0","article-title":"Extracting cancer concepts from clinical notes using natural language processing: a systematic review","volume":"24","author":"Gholipour","year":"2023","journal-title":"BMC Bioinformatics"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/10\/2315\/59206405\/ocae146.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/10\/2315\/59206405\/ocae146.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,20]],"date-time":"2024-09-20T07:54:03Z","timestamp":1726818843000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/10\/2315\/7696538"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,20]]},"references-count":42,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2024,6,20]]},"published-print":{"date-parts":[[2024,10,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae146","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,10]]},"published":{"date-parts":[[2024,6,20]]}}}