{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T11:33:20Z","timestamp":1791545600387,"version":"4.3.4"},"reference-count":21,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Popularized by ChatGPT, large language models (LLMs) are poised to transform the scalability of clinical natural language processing (NLP) downstream tasks such as medical question answering (MQA) and automated data extraction from clinical narrative reports. However, the use of LLMs in the health care setting is limited by cost, computing power, and patient privacy concerns. Specifically, as interest in LLM-based clinical applications grows, regulatory safeguards must be established to avoid exposure of patient data through the public domain. The use of open-source LLMs deployed behind institutional firewalls may ensure the protection of private patient data. In this study, we evaluated the extraction performance of a locally deployed LLM for automated MQA from surgical pathology reports.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n We compared the performance of human reviewers and a locally deployed LLM tasked with extracting key histologic and staging information from surgical pathology reports.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n A total of 84 thyroid cancer surgical pathology reports were assessed by two independent reviewers and the open-source FastChat-T5 3B-parameter LLM using institutional computing resources. Longer text reports were split into 1200-character-long segments, followed by conversion to embeddings. Three segments with the highest similarity scores were integrated to create the final context for the LLM. The context was then made part of the question it was directed to answer. Twelve medical questions for staging and thyroid cancer recurrence risk data extraction were formulated and answered for each report. The time to respond and concordance of answers were evaluated. The concordance rate for each pairwise comparison (human-LLM and human-human) was calculated as the total number of concordant answers divided by the total number of answers for each of the 12 questions. The average concordance rate and associated error of all questions were tabulated for each pairwise comparison and evaluated with two-sided\n t<\/jats:italic>\n tests.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Out of a total of 1008 questions answered, reviewers 1 and 2 had an average (SD) concordance rate of responses of 99% (1%; 999\/1008 responses). The LLM was concordant with reviewers 1 and 2 at an overall average (SD) rate of 89% (7%; 896\/1008 responses) and 89% (7.2%; 903\/1008 responses). The overall time to review and answer questions for all reports was 170.7, 115, and 19.56 minutes for Reviewers 1, 2, and the LLM, respectively.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n The locally deployed LLM can be used for MQA with considerable time-saving and acceptable accuracy in responses. Prompt engineering and fine-tuning may further augment automated data extraction from clinical narratives for the provision of real-time, essential clinical insights.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/64544","type":"journal-article","created":{"date-parts":[[2025,4,7]],"date-time":"2025-04-07T17:49:19Z","timestamp":1744048159000},"page":"e64544-e64544","source":"Crossref","is-referenced-by-count":35,"title":["Using Large Language Models to Automate Data Extraction From Surgical Pathology Reports: Retrospective Cohort Study"],"prefix":"10.2196","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-9203-925X","authenticated-orcid":false,"given":"Denise","family":"Lee","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3343-744X","authenticated-orcid":false,"given":"Akhil","family":"Vaid","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4589-4661","authenticated-orcid":false,"given":"Kartikeya M","family":"Menon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4946-6533","authenticated-orcid":false,"given":"Robert","family":"Freeman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2674-0387","authenticated-orcid":false,"given":"David S","family":"Matteson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-0030-8801","authenticated-orcid":false,"given":"Michael L","family":"Marin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6319-4314","authenticated-orcid":false,"given":"Girish N","family":"Nadkarni","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,4,7]]},"reference":[{"issue":"12","key":"R1","doi-asserted-by":"publisher","first-page":"1722","DOI":"10.1001\/jamaoncol.2016.6905","article-title":"Using the national cancer database for outcomes research: a review","volume":"3","author":"Boffa","journal-title":"JAMA Oncol"},{"issue":"12","key":"R2","first-page":"1117","volume":"8","author":"Hankey","journal-title":"Cancer Epidemiol Biomarkers Prev"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1136\/jclinpath-2016-203872","article-title":"Natural language processing in pathology: a scoping review","author":"Burger","journal-title":"J Clin Pathol"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.1200\/CCI.23.00156","article-title":"DeepPhe-CR: natural language processing software services for cancer registrar case abstraction","volume":"7","author":"Hochheiser","journal-title":"JCO Clin Cancer Inform"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2021.103864","article-title":"Development of a generalizable natural language processing pipeline to extract physician-reported pain from clinical reports: Generated using publicly-available datasets and tested on institutional clinical reports for cancer patients with bone metastases","volume":"120","author":"Naseri","journal-title":"J Biomed Inform"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1200\/CCI.22.00014","article-title":"Natural language processing of computed tomography reports to label metastatic phenotypes with prognostic significance in patients with colorectal cancer","volume":"6","author":"Causa Andrieu","journal-title":"JCO Clin Cancer Inform"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1200\/CCI.20.00165","article-title":"Development and use of natural language processing for identification of distant cancer recurrence and sites of distant recurrence using unstructured electronic health record data","volume":"5","author":"Karimi","journal-title":"JCO Clin Cancer Inform"},{"issue":"6","key":"R8","doi-asserted-by":"publisher","first-page":"797","DOI":"10.1001\/jamaoncol.2016.0213","article-title":"Natural language processing in oncology: a review","volume":"2","author":"Yim","journal-title":"JAMA Oncol"},{"issue":"8","key":"R9","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","journal-title":"Nat Med"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-022-00742-2","article-title":"A large language model for electronic health records","volume":"5","author":"Yang","journal-title":"NPJ Digit Med"},{"issue":"4","key":"R11","doi-asserted-by":"publisher","first-page":"445","DOI":"10.1001\/jamasurg.2023.6496","article-title":"Academic surgery in the era of large language models: a review","volume":"159","author":"Rengers","journal-title":"JAMA Surg"},{"key":"R12","unstructured":"LLMSurgicalPathologyNoteParserPublic. Github. URL: https:\/\/github.com\/akhilvaid\/LLMSurgicalPathologyNoteParser [Accessed 07-04-2025]"},{"issue":"6","key":"R13","doi-asserted-by":"publisher","first-page":"751","DOI":"10.1089\/thy.2017.0102","article-title":"Updated American joint committee on cancer\/tumor-node-metastasis staging system for differentiated and anaplastic thyroid cancer (eighth edition): what changed and why?","volume":"27","author":"Tuttle","journal-title":"Thyroid"},{"issue":"1","key":"R14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1089\/thy.2015.0020","article-title":"2015 American thyroid association management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: the American thyroid association guidelines task force on thyroid nodules and differentiated thyroid cancer","volume":"26","author":"Haugen","journal-title":"Thyroid"},{"issue":"21","key":"R15","doi-asserted-by":"publisher","first-page":"5463","DOI":"10.1158\/0008-5472.CAN-19-0579","article-title":"Use of natural language processing to extract clinical cancer phenotypes from electronic medical records","volume":"79","author":"Savova","journal-title":"Cancer Res"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2019.103301","article-title":"A frame semantic overview of NLP-based information extraction for cancer-related EHR notes","volume":"100","author":"Datta","journal-title":"J Biomed Inform"},{"key":"R17","volume":"2003","author":"Schadow","journal-title":"AMIA Annu Symp Proc"},{"issue":"3","key":"R18","doi-asserted-by":"publisher","DOI":"10.2196\/27210","article-title":"A question-and-answer system to extract data from free-text oncological pathology reports (CancerBERT network): development study","volume":"24","author":"Mitchell","journal-title":"J Med Internet Res"},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2024.105443","article-title":"Evaluation of large language models performance against humans for summarizing MRI knee radiology reports: a feasibility study","volume":"187","author":"L\u00f3pez-\u00dabeda","journal-title":"Int J Med Inform"},{"issue":"5","key":"R20","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.12767","article-title":"Use of artificial intelligence chatbots in interpretation of pathology reports","volume":"7","author":"Steimetz","journal-title":"JAMA Netw Open"},{"issue":"3","key":"R21","doi-asserted-by":"publisher","first-page":"209","DOI":"10.3857\/roj.2023.00633","article-title":"Developing prompts from large language model for extracting clinical information from pathology and ultrasound reports in breast cancer","volume":"41","author":"Choi","journal-title":"Radiat Oncol J"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/formative.jmir.org\/2025\/1\/e64544","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:30:22Z","timestamp":1785537022000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2025\/1\/e64544"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,7]]},"references-count":21,"URL":"https:\/\/doi.org\/10.2196\/64544","relation":{},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,7]]},"article-number":"v9i12e64544"}}