{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,12]],"date-time":"2026-03-12T12:39:37Z","timestamp":1773319177158,"version":"3.50.1"},"reference-count":18,"publisher":"Springer Science and Business Media LLC","license":[{"start":{"date-parts":[[2026,3,11]],"date-time":"2026-03-11T00:00:00Z","timestamp":1773187200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,3,11]],"date-time":"2026-03-11T00:00:00Z","timestamp":1773187200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"April Krueger v. Wyeth, Inc.","award":["Case No. 03-cv-2496"],"award-info":[{"award-number":["Case No. 03-cv-2496"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Digit Imaging. Inform. med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Prompt engineering techniques which aid in the use of generative artificial intelligence to address classification tasks have expanded considerably in the last 2\u00a0years. The success of such methods varies depending on context, and their efficacy in extracting structured data from unstructured medical text is not well understood. In this paper, five large language prompting strategies were evaluated on a structured categorical question about unstructured radiology reports. Three categories were typically explicit in the text, while one required extrapolation from medical knowledge. The five prompting strategies each contained one or more of the following prompting techniques: external knowledge source, recursive criticism and improvement, chain-of-thought. The efficacy of each strategy was assessed by measuring accuracy and rate of non-determinism. Accuracy was measured by overall correctness and sensitivity to the non-explicit category. Non-determinism was assessed by running each strategy multiple times per exam, tracking both the frequency of exams with non-deterministic outputs and the number of runs needed before stabilization. The presence of an external knowledge source increased sensitivity to the non-explicit category from 10 to 66%, with minimal impact on overall accuracy. Sensitivity further increased to 78% with the introduction of recursive criticism and improvement and chain-of-thought but at the cost of increased non-determinism, with the proportion of exams with non-deterministic results increasing from 7% with only external knowledge source to 14%.<\/jats:p>","DOI":"10.1007\/s10278-026-01882-7","type":"journal-article","created":{"date-parts":[[2026,3,11]],"date-time":"2026-03-11T18:58:09Z","timestamp":1773255489000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Advanced Prompting Techniques Informed by Clinical Expertise Improve the Accuracy of LLM Data Extraction but Increase Non-Determinism"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1632-2050","authenticated-orcid":false,"given":"Yifei","family":"Wang","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alejandro Alejandrez","family":"Cisneros","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Carly","family":"Stewart","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matthew","family":"Malekhedayat","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jason","family":"Luong","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rebecca","family":"Smith-Bindman","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John","family":"Mongan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2026,3,11]]},"reference":[{"key":"1882_CR1","doi-asserted-by":"crossref","unstructured":"Kooragayala K, Crudeli C, Kalola A, Bhat V, Lou J, Sensenig R, et al. Utilization of natural language processing software to identify worrisome pancreatic lesions. Ann Surg Oncol. 2022;29(13):8513\u20138519.","DOI":"10.1245\/s10434-022-12391-6"},{"key":"1882_CR2","doi-asserted-by":"crossref","unstructured":"Dublin S, Baldwin E, Walker RL, Christensen LM, Haug PJ, Jackson ML, et al. Natural language processing to identify pneumonia from radiology reports. Pharmacoepidemiol Drug Saf. 2013;22(8):834\u2013841.","DOI":"10.1002\/pds.3418"},{"key":"1882_CR3","doi-asserted-by":"crossref","unstructured":"Bozkurt S, Alkim E, Banerjee I, Rubin DL. Automated detection of measurements and their descriptors in radiology reports using a hybrid natural language processing algorithm. J Digit Imaging. 2019;32(4):544\u2013553.","DOI":"10.1007\/s10278-019-00237-9"},{"key":"1882_CR4","doi-asserted-by":"crossref","unstructured":"Yalamanchili A, Sengupta B, Song J, Lim S, Thomas TO, Mittal BB, et al. Quality of large language model responses to radiation oncology patient care questions. JAMA Netw Open. 2024;7(4):e244630.","DOI":"10.1001\/jamanetworkopen.2024.4630"},{"key":"1882_CR5","doi-asserted-by":"crossref","unstructured":"Doshi R, Amin KS, Khosla P, Bajaj SS, Chheang S, Forman HP. Quantitative evaluation of large language models to streamline radiology report impressions: a multimodal retrospective analysis. Radiology. 2024;310(3):e231593.","DOI":"10.1148\/radiol.231593"},{"key":"1882_CR6","doi-asserted-by":"crossref","unstructured":"Yamashita R, Bird K, Cheung PYC, Decker JH, Flory MN, Goff D, et al. Automated identification and measurement extraction of pancreatic cystic lesions from free-text radiology reports using natural language processing. Radiol Artif Intell. 2022;4(2):e210092.","DOI":"10.1148\/ryai.210092"},{"key":"1882_CR7","doi-asserted-by":"crossref","unstructured":"Le Guellec B, Lef\u00e8vre A, Geay C, Shorten L, Bruge C, Hacein-Bey L, et al. Performance of an open-source large language model in extracting information from free-text radiology reports. Radiol Artif Intell. 2024;6(4):e230364.","DOI":"10.1148\/ryai.230364"},{"key":"1882_CR8","unstructured":"Kim G, Baldi P, McAleer S. Language models can solve computer tasks [Internet]. arXiv; 2023 [cited 2025 Mar 22]. Available from: http:\/\/arxiv.org\/abs\/2303.17491"},{"key":"1882_CR9","unstructured":"Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, et al. Chain-of-thought prompting elicits reasoning in large language models [Internet]. arXiv; 2023 [cited 2025 Mar 22]. Available from: http:\/\/arxiv.org\/abs\/2201.11903"},{"key":"1882_CR10","doi-asserted-by":"crossref","unstructured":"Kim TT, Makutonin M, Sirous R, Javan R. Optimizing large language models in radiology and mitigating pitfalls: prompt engineering and fine-tuning. RadioGraphics. 2025;45(4):e240073.","DOI":"10.1148\/rg.240073"},{"key":"1882_CR11","doi-asserted-by":"crossref","unstructured":"Ouyang S, Zhang JM, Harman M, Wang M. An empirical study of the non-determinism of ChatGPT in code generation. ACM Trans Softw Eng Methodol. 2025;34(2):1\u201328.","DOI":"10.1145\/3697010"},{"key":"1882_CR12","unstructured":"Klishevich E, Denisov-Blanch Y, Obstbaum S, Ciobanu I, Kosinski M. Measuring determinism in large language models for software code review. arXiv; 2025 [v1 2025 Feb 28]. Available from: http:\/\/arxiv.org\/abs\/2502.20747"},{"key":"1882_CR13","unstructured":"Atil B, Aykent S, Chittams A, Fu L, Passonneau RJ, Radcliffe E, Rajagopal GR, Sloan A, Tudrej T, Ture F, Wu Z, Xu L, Baldwin B. Non-determinism of \"deterministic\" LLM settings. arXiv; 2025 [v5 cited 2025 Apr 2]. Available from: https:\/\/arxiv.org\/abs\/2408.04667"},{"key":"1882_CR14","unstructured":"OpenAI. Pricing. OpenAI website. https:\/\/platform.openai.com\/docs\/pricing. Published 2024. Accessed May 20, 2025."},{"key":"1882_CR15","unstructured":"R Core Team. R: A language and environment for statistical computing. Version 4.3.2 [software]. Vienna, Austria: R Foundation for Statistical Computing; 2023. Available from: https:\/\/www.R-project.org\/"},{"key":"1882_CR16","unstructured":"Yuan J, Li H, Ding X, Xie W, Li Y-J, Zhao W, Wan K, Shi J, Hu X, Liu Z. Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference. arXiv; 2025 [v1 2025 Jun 11]. Available from: http:\/\/arxiv.org\/abs\/2506.09501"},{"key":"1882_CR17","doi-asserted-by":"crossref","unstructured":"Shanmugavelu S, Taillefumier M, Culver C, Hernandez O, Coletti M, Sedova A. Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications. arXiv; 2024 [v1 2024 Aug 09]. Available from: https:\/\/arxiv.org\/abs\/2408.05148","DOI":"10.1109\/SCW63240.2024.00028"},{"key":"1882_CR18","unstructured":"Liu R, Geng J, Wu AJ, Sucholutsky I, Lombrozo T, Griffiths TL. Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse. arXiv; 2024 [v1 2024 Oct 27]. Available from: https:\/\/arxiv.org\/abs\/2410.21333"}],"container-title":["Journal of Imaging Informatics in Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-026-01882-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10278-026-01882-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-026-01882-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,11]],"date-time":"2026-03-11T18:58:16Z","timestamp":1773255496000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10278-026-01882-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,11]]},"references-count":18,"alternative-id":["1882"],"URL":"https:\/\/doi.org\/10.1007\/s10278-026-01882-7","relation":{},"ISSN":["2948-2933"],"issn-type":[{"value":"2948-2933","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,11]]},"assertion":[{"value":"6 October 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 February 2026","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 February 2026","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 March 2026","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This study is HIPAA-compliant and approved by the University of California, San Francisco Institutional Review Board with a waiver of individual informed consent.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics Approval and Consent to Participate"}},{"value":"Dr. Alejandrez is a current active-duty military diagnostic radiology resident affiliated with SAUSHEC. However, please note that SAUSHEC and the military\/government had no role in this study. The views expressed in this manuscript are those of the author(s) and do not necessarily reflect the official policy or position of the Defense Health Agency, the Department of Defense, or any agencies under the US Government. Dr. Smith-Bindman is a co-founder of Alara Imaging, Inc., a new company focused on improving the clinical and operational aspects of health systems, including collecting and reporting CT radiation dose and image quality as part of payer-led quality programs. Alara Imaging had no role in the study, and Dr. Smith-Bindman has not received funding from Alara Imaging. Dr. Mongan has the following disclosures to report\u2014unpaid advisor: Microsoft (Nuance), royalties: GE, spouse employment: Annexon Biosciences; Bristol Myers Squibb, spouse shareholder: Annexon Biosciences; Bristol Myers Squibb, expert witness: Gibson Dunn; Covington.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}