{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T17:26:45Z","timestamp":1783013205843,"version":"3.54.6"},"reference-count":29,"publisher":"Wiley","issue":"4","license":[{"start":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T00:00:00Z","timestamp":1774915200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T00:00:00Z","timestamp":1774915200000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"content-domain":{"domain":["aapm.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["J Applied Clin Med Phys"],"published-print":{"date-parts":[[2026,4]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Purpose<\/jats:title>\n                    <jats:p>Ambiguous or incomplete documentation is a recurrent bottleneck in radiation oncology workflows, leading to inefficiencies in communication and potential treatment delays. Large language models (LLMs) pose a solution to addressing these ambiguities without added burden to clinical staff. We aim to assess the effectiveness of Meta's open\u2010source Llama 3.3 model in using physician consultation notes to isolate and classify anatomical treatment sites and create helpful extractive summaries for each patient.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Semi\u2010structured interviews with five radiation therapists revealed that CT simulation orders lack the necessary details to acquire the appropriate image. A retrospective cohort of 100 patient notes was used for iterative prompt engineering. The final model was evaluated on an independent test cohort of 52 patient notes. The LLM's accuracy in identifying the treatment site was benchmarked against two human observers (a medical physicist and a physician) as well as the final delivered treatment plan (ground truth). The helpfulness and accuracy of the AI\u2010generated summaries were also rated by both observers on a 5\u2010point Likert scale.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Llama 3.3 achieved a weighted accuracy of 94.2% [95%CI: 89.4%\u201398.1%] when compared to sites isolated by either observer. When compared to the sites isolated from the retrospectively delivered plans, the model reached a weighted accuracy of 92.3% [95% CI: 87.5%\u201397.1%]. The model classified the anatomical sites with a weighted accuracy of 96.2% [95%CI: 87.0% \u201398.9%]. The AI\u2010generated summaries were highly rated by both observers (Observer 1: 4.96 [95%CI: 4.87\u20135.00] and Observer 2: 4.58 [95% CI: 4.38\u20134.73]).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>This pilot study provides foundational evidence that LLMs can classify data with high accuracy, achieve benchmarks comparable to human experts when isolating anatomical treatment sites, and produce clinically helpful summaries. Our results suggest that LLMs can be effectively integrated to streamline complex radiotherapy workflows in the clinic.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1002\/acm2.70558","type":"journal-article","created":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T10:03:39Z","timestamp":1774951419000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Automated identification of radiotherapy treatment sites from unstructured physician notes"],"prefix":"10.1002","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1309-7246","authenticated-orcid":false,"given":"Klea","family":"Hoxha","sequence":"first","affiliation":[{"name":"Department of Radiation Oncology University of California, Los Angeles  Los Angeles California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ricky R.","family":"Savjani","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology University of California, Los Angeles  Los Angeles California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3421-7481","authenticated-orcid":false,"given":"Jack","family":"Neylon","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology University of California, Los Angeles  Los Angeles California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathan","family":"Pham","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology University of California, Los Angeles  Los Angeles California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rojine","family":"Ariani","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology University of California, Los Angeles  Los Angeles California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0752-5825","authenticated-orcid":false,"given":"Dylan P.","family":"O'Connell","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology University of California, Los Angeles  Los Angeles California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2026,3,31]]},"reference":[{"key":"e_1_2_10_2_1","doi-asserted-by":"publisher","DOI":"10.3322\/caac.21871"},{"key":"e_1_2_10_3_1","doi-asserted-by":"publisher","DOI":"10.1111\/jep.70189"},{"key":"e_1_2_10_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11606\u2010024\u201008956\u20108"},{"key":"e_1_2_10_5_1","doi-asserted-by":"publisher","DOI":"10.1002\/acm2.70310"},{"key":"e_1_2_10_6_1","doi-asserted-by":"publisher","DOI":"10.1002\/acm2.13640"},{"key":"e_1_2_10_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.prro.2017.10.007"},{"key":"e_1_2_10_8_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2020.34065"},{"key":"e_1_2_10_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tipsro.2024.100258"},{"key":"e_1_2_10_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jmir.2019.08.008"},{"key":"e_1_2_10_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.prro.2023.08.010"},{"key":"e_1_2_10_12_1","doi-asserted-by":"publisher","DOI":"10.2196\/49724"},{"key":"e_1_2_10_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijrobp.2024.07.1386"},{"key":"e_1_2_10_14_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41698\u2010024\u201000733\u20104"},{"key":"e_1_2_10_15_1","doi-asserted-by":"publisher","DOI":"10.2196\/68998"},{"key":"e_1_2_10_16_1","doi-asserted-by":"publisher","DOI":"10.4258\/hir.2019.25.1.1"},{"key":"e_1_2_10_17_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746\u2010025\u201001527\u2010z"},{"key":"e_1_2_10_18_1","doi-asserted-by":"publisher","DOI":"10.2196\/17984"},{"key":"e_1_2_10_19_1","unstructured":"NelsonJ AugustineM MatthewS Natural Language Processing (NLP) in Clinical Documentation [Internet].2025Feb [cited 2025 Oct 10]. Available from: ResearchGate.https:\/\/www.researchgate.net\/publication\/391950535_Natural_Language_Processing_NLP_in_Clinical_Documentation"},{"key":"e_1_2_10_20_1","doi-asserted-by":"publisher","DOI":"10.2196\/12239"},{"key":"e_1_2_10_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/s41666\u2010019\u201000049\u20100"},{"key":"e_1_2_10_22_1","doi-asserted-by":"publisher","DOI":"10.1177\/21501319231166921"},{"key":"e_1_2_10_23_1","doi-asserted-by":"publisher","DOI":"10.2196\/85206"},{"key":"e_1_2_10_24_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00502"},{"key":"e_1_2_10_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jhsa.2024.01.006"},{"key":"e_1_2_10_26_1","unstructured":"WeiJ WangX SchuurmansD et\u00a0al.Chain\u2010of\u2010thought prompting elicits reasoning in large language models. January 10 2023. doi:10.48550\/arXiv.2201.11903"},{"key":"e_1_2_10_27_1","unstructured":"KojimaT GuSS ReidM MatsuoY IwasawaY Large language models are zero\u2010shot reasoners. January 29 2023. doi:10.48550\/arXiv.2205.11916"},{"key":"e_1_2_10_28_1","unstructured":"WangJ ShiE YuS et\u00a0al.Prompt engineering for healthcare: methodologies and applications. March 23 2024. doi:10.48550\/arXiv.2304.14670"},{"key":"e_1_2_10_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2022.3209479"},{"key":"e_1_2_10_30_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591\u2010024\u201002855\u20105"}],"container-title":["Journal of Applied Clinical Medical Physics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/acm2.70558","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/acm2.70558","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/acm2.70558","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T17:09:13Z","timestamp":1783012153000},"score":1,"resource":{"primary":{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/10.1002\/acm2.70558"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,31]]},"references-count":29,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2026,4]]}},"alternative-id":["10.1002\/acm2.70558"],"URL":"https:\/\/doi.org\/10.1002\/acm2.70558","archive":["Portico"],"relation":{},"ISSN":["1526-9914","1526-9914"],"issn-type":[{"value":"1526-9914","type":"print"},{"value":"1526-9914","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,31]]},"assertion":[{"value":"2025-10-14","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-03-05","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-03-31","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e70558"}}