{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,16]],"date-time":"2026-10-16T13:00:45Z","timestamp":1792155645324,"version":"4.4.0"},"reference-count":41,"publisher":"Wiley","issue":"11","license":[{"start":{"date-parts":[[2025,10,27]],"date-time":"2025-10-27T00:00:00Z","timestamp":1761523200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"publisher","award":["R01CA280134"],"award-info":[{"award-number":["R01CA280134"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["aapm.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["J Applied Clin Med Phys"],"published-print":{"date-parts":[[2025,11]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>In the current clinical workflow of radiation oncology, therapists manually summarize the physician\u2010issued Computed Tomography (CT) simulation orders to prepare patients' CT simulations. This process increases the workload, introduces variability in documentation quality, and is prone to human errors.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Purpose<\/jats:title>\n                    <jats:p>This study aims to address these challenges using a large language model (LLM) to automate the generation of summaries from the CT simulation orders and evaluate its performance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>A total of 607 CT simulation orders were collected from the Aria database at our institution. A locally hosted LLaMa 3.1 405B model, accessed via the Application Programming Interface (API) service, was used to extract keywords from the CT simulation orders and generate summaries. The downloaded CT simulation orders were categorized into seven groups based on treatment modalities and disease sites. For each group, a customized instruction prompt was developed collaboratively with therapists to guide the LLaMa 3.1 405B model in generating summaries. The ground truth for the corresponding summaries was manually derived by carefully reviewing each CT simulation order and subsequently verified by therapists. The accuracy of the LLM\u2010generated summaries was evaluated by therapists using the ground truth.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Over 98% of the LLM\u2010generated summaries aligned with the ground truth in terms of accuracy. Our evaluations showed an improved consistency in format and enhanced readability from the LLM\u2010generated summaries compared to the corresponding therapist\u2010generated summaries. This automated approach demonstrated consistent performance across all groups, regardless of treatment modality or disease site.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>This study demonstrated the high precision and consistency of the LLaMa 3.1 405B model in extracting keywords and summarizing CT simulation orders, suggesting that LLMs have great potential to assist in this task, reduce the workload of CT simulation therapists and improve radiation oncology workflow efficiency.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1002\/acm2.70310","type":"journal-article","created":{"date-parts":[[2025,10,28]],"date-time":"2025-10-28T13:27:05Z","timestamp":1761658025000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Using large language models to automate summarization of CT simulation orders in radiation oncology"],"prefix":"10.1002","volume":"26","author":[{"given":"Meiyun","family":"Cao","sequence":"first","affiliation":[{"name":"Department of Radiation Oncology Mayo Clinic Hospital  Phoenix Arizona USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shaw","family":"Hu","sequence":"additional","affiliation":[{"name":"Corcoran College of Arts and Sciences George Washington University\u2010Washington DC  Washington DC USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jason","family":"Sharp","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Mayo Clinic Hospital  Phoenix Arizona USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-5900-9433","authenticated-orcid":false,"given":"Edward Lee","family":"Clouser","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Mayo Clinic Hospital  Phoenix Arizona USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jason M.","family":"Holmes","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Mayo Clinic Hospital  Phoenix Arizona USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Linda L","family":"Lam","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Mayo Clinic Hospital  Phoenix Arizona USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaoning","family":"Ding","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Mayo Clinic Hospital  Phoenix Arizona USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diego Santos","family":"Toesca","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Mayo Clinic Hospital  Phoenix Arizona USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brianne C","family":"Raulston","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Mayo Clinic Hospital  Phoenix Arizona USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wendy S","family":"Lindholm","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Mayo Clinic Hospital  Phoenix Arizona USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samir H.","family":"Patel","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Mayo Clinic Hospital  Phoenix Arizona USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sujay A","family":"Vora","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Mayo Clinic Hospital  Phoenix Arizona USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6730-1524","authenticated-orcid":false,"given":"Peilong","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Mayo Clinic Hospital  Phoenix Arizona USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0935-3999","authenticated-orcid":false,"given":"Wei","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Mayo Clinic Hospital  Phoenix Arizona USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2025,10,27]]},"reference":[{"key":"e_1_2_9_2_1","doi-asserted-by":"crossref","unstructured":"ChangY WangX WangJ et al.A survey on evaluation of large language models.ACM transactions on intelligent systems and technology.2024;15(3):1\u201045.","DOI":"10.1145\/3641289"},{"key":"e_1_2_9_3_1","article-title":"Attention is all you need","volume":"30","author":"Vaswani A","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_9_4_1","doi-asserted-by":"publisher","DOI":"10.4103\/jimr.jimr_50_23"},{"key":"e_1_2_9_5_1","volume-title":"Surviving Chatgpt in Healthcare","author":"Liu Z","year":"2023"},{"key":"e_1_2_9_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.prro.2024.04.017"},{"key":"e_1_2_9_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.mcpdig.2025.100198"},{"key":"e_1_2_9_8_1","first-page":"100428","volume-title":"Fine\u2010Tuning Large Language Models for Radiation Oncology, A Highly Specialized Healthcare Domain","author":"Liu Z","year":"2024"},{"key":"e_1_2_9_9_1","first-page":"57","volume-title":"Large Language Models in Healthcare and Medical Domain: A Review","author":"Nazi ZA","year":"2024"},{"key":"e_1_2_9_10_1","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare11202776"},{"key":"e_1_2_9_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.metrad.2023.100045"},{"key":"e_1_2_9_12_1","unstructured":"LiuZ WangP LiY et\u00a0al.Radonc\u2010gpt: A large language model for radiation oncology. arXiv [Preprint].2023. doi:10.48550\/arXiv.2309.10160"},{"key":"e_1_2_9_13_1","unstructured":"ShiY ShuP LiuZ et\u00a0al.Mgh radiology llama: A llama 3 70b model for radiology. arXiv [Preprint]. 2024. doi:10.48550\/arXiv.2408.11848"},{"key":"e_1_2_9_14_1","doi-asserted-by":"crossref","unstructured":"WangP HolmesJ LiuZ et\u00a0al.A recent evaluation on the performance of LLMs on radiation oncology physics using questions of randomly shuffled options. arXiv [Preprint].2024. doi:10.48550\/arXiv.2412.10622","DOI":"10.3389\/fonc.2025.1557064"},{"key":"e_1_2_9_15_1","doi-asserted-by":"publisher","DOI":"10.1056\/AIoa2300151"},{"key":"e_1_2_9_16_1","first-page":"464","volume-title":"Tailoring Large Language Models to Radiology: A Preliminary Approach to LLM Adaptation for a Highly Specialized Domain","author":"Liu Z","year":"2023"},{"key":"e_1_2_9_17_1","first-page":"1884","volume-title":"The Potential And Pitfalls of Using a Large Language Model Such as ChatGPT, GPT\u20104, or LLaMA as a Clinical Assistant","author":"Zhang J","year":"2024"},{"key":"e_1_2_9_18_1","doi-asserted-by":"crossref","unstructured":"WuZ ZhangL CaoC et\u00a0al. Exploring the Trade\u2010Offs: Unified Large Language Models vs Local Fine\u2010Tuned Models for Highly\u2010Specific Radiology NLI Task. IEEE Transactions on Big Data. 2025 Feb 13.","DOI":"10.1109\/TBDATA.2025.3536928"},{"key":"e_1_2_9_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.meddos.2005.12.004"},{"key":"e_1_2_9_20_1","doi-asserted-by":"publisher","DOI":"10.1088\/0031\u20109155\/56\/24\/003"},{"key":"e_1_2_9_21_1","doi-asserted-by":"publisher","DOI":"10.1118\/1.4813900"},{"key":"e_1_2_9_22_1","first-page":"5248","volume-title":"Uncertainty Incorporated Beam Angle Optimization for IMPT Treatment Planning","author":"Cao W","year":"2012"},{"key":"e_1_2_9_23_1","first-page":"6138","volume-title":"Robust Intensity\u2010Modulated Proton Therapy to Reduce High Linear Energy Transfer in Organs at Risk","author":"An Y","year":"2017"},{"key":"e_1_2_9_24_1","first-page":"5428","volume-title":"Intensity\u2010Modulated Proton Therapy (IMPT) Interplay Effect Evaluation of Asymmetric Breathing With Simultaneous Uncertainty Considerations in Patients With Non\u2010Small Cell Lung Cancer","author":"Shan J","year":"2020"},{"key":"e_1_2_9_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.bjps.2016.08.027"},{"key":"e_1_2_9_26_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12913\u2010025\u201012780\u2010z"},{"key":"e_1_2_9_27_1","unstructured":"GrattafioriA DubeyA JauhriA et\u00a0al.The llama 3 herd of models.arXiv preprint arXiv:240721783.2024."},{"key":"e_1_2_9_28_1","unstructured":"MetaAI. Meta LLaMA 3.1\u2013405B model card [Internet]. Hugging Face; 2024 [cited 2025 Jul 31].https:\/\/huggingface.co\/meta\u2010llama\/Meta\u2010Llama\u20103.1\u2010405B"},{"key":"e_1_2_9_29_1","unstructured":"Hugging Face.LLaMA 3.1 models overview and benchmarks [Internet]. Hugging Face; 2024 [cited 2025 Jul 31].https:\/\/huggingface.co\/models"},{"key":"e_1_2_9_30_1","unstructured":"QinM.The Uniqueness of LLaMA3\u201070B Series with Per\u2010Channel Quantization. arXiv [Preprint]. 2024. doi:10.48550\/arXiv.240815301"},{"key":"e_1_2_9_31_1","unstructured":"LiY YinR LeeD XiaoS PandaP.GPTQv2: Efficient Finetuning\u2010Free Quantization for Asymmetric Calibration. arXiv [Preprint]. 2025. doi:10.48550\/arXiv.2504.02692"},{"key":"e_1_2_9_32_1","unstructured":"DaoT FuT ErmonS R\u00e9C RudraA.FlashAttention: Fast and memory\u2010efficient exact attention with IO\u2010awareness. arXiv [Preprint]. 2022. doi:10.48550.arXiv.2205.14135"},{"key":"e_1_2_9_33_1","unstructured":"VaswaniA ShazeerN ParmarN et\u00a0al.Attention is all you need. arXiv [Preprint]. 2017. doi.10.48550\/arXiv.1706.03762"},{"key":"e_1_2_9_34_1","unstructured":"HintonG VinyalsO DeanJ.Distilling the knowledge in a neural network.arXiv [Preprint]. 2015. doi:10.48550\/arXiv.1503.02531"},{"key":"e_1_2_9_35_1","unstructured":"ZhangY.A simple yet deep explanation of FlashAttention v1 and v2. Medium [Internet]. Published August 9 2023. Accessed July 31 2025. Available from:https:\/\/medium.com\/@yuhezhang\/a\u2010simple\u2010yet\u2010deep\u2010explanation\u2010of\u2010flashattention\u2010v1\u2010and\u2010v2\u20108aa067d9451c"},{"key":"e_1_2_9_36_1","unstructured":"JurafskyD MartinJH.Speech and language processing (3rd ed. draft) [Internet]. Published August 24 2025. Accessed September 6 2025. Available from:https:\/\/web.stanford.edu\/~jurafsky\/slp3\/"},{"key":"e_1_2_9_37_1","doi-asserted-by":"crossref","unstructured":"RenzeM.The effect of sampling temperature on problem solving in large language models.Findings of the Association for Computational Linguistics: EMNLP 2024.ACL Anthology;2024:7346\u20107356. doi:10.18653\/v1\/2024.findings\u2010emnlp.432","DOI":"10.18653\/v1\/2024.findings-emnlp.432"},{"key":"e_1_2_9_38_1","doi-asserted-by":"crossref","unstructured":"PatelD TimsinaP RautG et\u00a0al.Exploring temperature effects on large language models across various clinical tasks. medRxiv.2024.","DOI":"10.1101\/2024.07.22.24310824"},{"key":"e_1_2_9_39_1","unstructured":"Prompt Engineering Guide.Examples of prompts [Internet]. Published June 6 2025. Accessed July 31 2025. Available from:https:\/\/www.promptingguide.ai\/introduction\/examples"},{"key":"e_1_2_9_40_1","unstructured":"SiriwardhanaS McquadeM et\u00a0al.Domain Adaptation of Llama3\u201070B\u2010Instruct through Continual Pre\u2010Training and Model Merging: A Comprehensive Evaluation. arXiv [Preprint]. 2024. doi:10.48550\/arXiv.2406.14971"},{"key":"e_1_2_9_41_1","unstructured":"ZhangR HanJ et\u00a0al.LLAMA\u2010Adapter: Efficient Fine\u2010Tuning Of Large Language MOD\u2010ELS With Zero\u2010Initialized Attention. InThe Twelfth International Conference on Learning Representations.2024."},{"key":"e_1_2_9_42_1","doi-asserted-by":"crossref","unstructured":"ZhangJ WangX et\u00a0al.Ratt: a thought structure for coherent and correct llm reasoning. In:Proceedings of the AAAI Conference on Artificial Intelligence.AAAI;2025.","DOI":"10.1609\/aaai.v39i25.34876"}],"container-title":["Journal of Applied Clinical Medical Physics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/acm2.70310","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,15]],"date-time":"2025-11-15T11:45:18Z","timestamp":1763207118000},"score":1,"resource":{"primary":{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/10.1002\/acm2.70310"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,27]]},"references-count":41,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2025,11]]}},"alternative-id":["10.1002\/acm2.70310"],"URL":"https:\/\/doi.org\/10.1002\/acm2.70310","archive":["Portico"],"relation":{},"ISSN":["1526-9914","1526-9914"],"issn-type":[{"value":"1526-9914","type":"print"},{"value":"1526-9914","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,27]]},"assertion":[{"value":"2025-03-20","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-09-24","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e70310"}}