{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,14]],"date-time":"2025-02-14T06:10:01Z","timestamp":1739513401476,"version":"3.37.0"},"reference-count":33,"publisher":"IOP Publishing","issue":"1","license":[{"start":{"date-parts":[[2025,2,14]],"date-time":"2025-02-14T00:00:00Z","timestamp":1739491200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2025,2,14]],"date-time":"2025-02-14T00:00:00Z","timestamp":1739491200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"name":"NIH","award":["R01CA237269"],"award-info":[{"award-number":["R01CA237269"]}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2025,3,31]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Introduction: Auto-segmentation of tumor volumes and organs at risk (OARs) is a critical step in cancer radiotherapy treatment planning, where rapid, precise adjustments to treatment plans are required to match the patient anatomy. Although auto-segmentation has been clinically accepted for most OARs, auto-segmentation of tumor volumes, particularly clinical target volumes (CTVs), remains a challenge. This difficulty arises because images alone are often insufficient to capture the necessary information for accurate delineation of microscopic tumor invasion invisible on the image itself. Methods: We propose a deep learning-based medical image segmentation framework designed to mimic the clinical process of delineating CTVs and OARs. At its core, the model performs precise segmentation of medical images while enhancing accuracy by integrating clinical information in text format. A transformer-based text encoder converts textual clinical data into vectors, which are incorporated into the segmentation process with image features. This integration bridges the gap between traditional automated segmentation methods and clinician-guided, context-rich delineations. The framework\u2019s effectiveness is demonstrated through a prostate segmentation example in the context of radiation therapy for localized prostate cancer, where incorporating clinical context significantly impacts the delineation process. Results: In our experiments, we included additional clinical information potentially influencing clinicians\u2019 prostate segmentation. The results show that our proposed method not only outperforms the baseline model, but also surpasses current state-of-the-art methods, with or without clinical contexts. Furthermore, our method demonstrates high performance even with limited data. Conclusion: This proposed segmentation framework has shown to significantly improve auto-segmentation, particularly for CTVs, in cancer radiotherapy.<\/jats:p>","DOI":"10.1088\/2632-2153\/adb371","type":"journal-article","created":{"date-parts":[[2025,2,6]],"date-time":"2025-02-06T23:00:06Z","timestamp":1738882806000},"page":"015040","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Medical image segmentation assisted with clinical inputs via language encoder in a deep learning framework"],"prefix":"10.1088","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6712-5823","authenticated-orcid":true,"given":"Hengrui","family":"Zhao","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6894-8645","authenticated-orcid":true,"given":"Biling","family":"Wang","sequence":"additional","affiliation":[]},{"given":"Deepkumar","family":"Mistry","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8491-4146","authenticated-orcid":true,"given":"Jing","family":"Wang","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9043-1490","authenticated-orcid":false,"given":"Michael","family":"Dohopolski","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6268-4653","authenticated-orcid":false,"given":"Daniel","family":"Yang","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3036-7287","authenticated-orcid":true,"given":"Weiguo","family":"Lu","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3083-6752","authenticated-orcid":true,"given":"Steve","family":"Jiang","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9590-0655","authenticated-orcid":true,"given":"Dan","family":"Nguyen","sequence":"additional","affiliation":[]}],"member":"266","published-online":{"date-parts":[[2025,2,14]]},"reference":[{"key":"mlstadb371bib1","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1016\/s0360-3016(03)00405-x","article-title":"Intensity-modulated radiation therapy for head-and-neck cancer: the UCSF experience focusing on target volume delineation","volume":"57","author":"Lee","year":"2003","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"mlstadb371bib2","doi-asserted-by":"publisher","first-page":"777","DOI":"10.1007\/978-3-030-00934-2_86","article-title":"Tumor-aware, adversarial domain adaptation from CT to MRI for lung cancer segmentation","author":"Jiang","year":"2018"},{"key":"mlstadb371bib3","doi-asserted-by":"publisher","first-page":"1535","DOI":"10.1016\/j.ijrobp.2004.11.037","article-title":"Four-dimensional image-based treatment planning: target volume segmentation and dose calculation in the presence of respiratory motion","volume":"61","author":"Rietzel","year":"2005","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"mlstadb371bib4","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1118\/1.597064","article-title":"Automatic segmentation of liver structure in CT images","volume":"20","author":"Bae","year":"1993","journal-title":"Med. Phys."},{"key":"mlstadb371bib5","doi-asserted-by":"publisher","first-page":"2625","DOI":"10.1109\/TBME.2011.2160262","article-title":"Automatic segmentation of intracochlear anatomy in conventional CT","volume":"58","author":"Noble","year":"2011","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"mlstadb371bib6","doi-asserted-by":"publisher","first-page":"98","DOI":"10.1109\/42.668699","article-title":"Fully automatic segmentation of the brain in MRI","volume":"17","author":"Atkins","year":"1998","journal-title":"IEEE Trans. Med. Imaging"},{"key":"mlstadb371bib7","doi-asserted-by":"publisher","first-page":"564","DOI":"10.1016\/j.neuroimage.2009.04.068","article-title":"Automatic segmentation of newborn brain MRI","volume":"47","author":"Weisenfeld","year":"2009","journal-title":"Neuroimage"},{"key":"mlstadb371bib8","doi-asserted-by":"publisher","first-page":"892","DOI":"10.1016\/j.ijrobp.2006.07.013","article-title":"Online adaptive radiotherapy of the bladder: small bowel irradiated-volume reduction","volume":"66","author":"Burridge","year":"2006","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"mlstadb371bib9","doi-asserted-by":"publisher","first-page":"994","DOI":"10.1016\/j.ijrobp.2017.04.023","article-title":"Online adaptive radiation therapy","volume":"99","author":"Lim-Reinders","year":"2017","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"mlstadb371bib10","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1080\/03091900412331289889","article-title":"Automatic segmentation of medical images using image registration: diagnostic and simulation applications","volume":"29","author":"Barber","year":"2005","journal-title":"J. Med. Eng. Technol."},{"key":"mlstadb371bib11","doi-asserted-by":"publisher","first-page":"582","DOI":"10.1007\/s10278-019-00227-x","article-title":"Deep learning techniques for medical image segmentation: achievements and challenges","volume":"32","author":"Hesamian","year":"2019","journal-title":"J. Digit. Imaging"},{"key":"mlstadb371bib12","doi-asserted-by":"publisher","first-page":"1224","DOI":"10.3390\/su13031224","article-title":"A review of deep-learning-based medical image segmentation methods","volume":"13","author":"Liu","year":"2021","journal-title":"Sustainability"},{"key":"mlstadb371bib13","doi-asserted-by":"publisher","first-page":"22","DOI":"10.47760\/ijcsmc","article-title":"Medical image segmentation: a review","volume":"2","author":"Patil","year":"2013","journal-title":"Int. J. Comput. Sci. Mobile Comput."},{"article-title":"Attention U-Net: learning where to look for the pancreas","year":"2018","author":"Oktay","key":"mlstadb371bib14"},{"key":"mlstadb371bib15","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/978-3-030-00889-5_1","article-title":"UNet++: a nested U-Net architecture for medical image segmentation","author":"Zhou","year":"2018"},{"article-title":"3D U-Net: learning dense volumetric segmentation from sparse annotation","year":"2016","author":"Zhou","key":"mlstadb371bib16"},{"key":"mlstadb371bib17","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-24574-4_28","article-title":"U-Net: convolutional networks for biomedical image segmentation","author":"Ronneberger","year":"2015"},{"article-title":"nnU-Net: self-adapting framework for u-net-based medical image segmentation","year":"2018","author":"Isensee","key":"mlstadb371bib18"},{"article-title":"MIS-FM: 3D medical image segmentation using foundation models pretrained on a large-scale unannotated dataset","year":"2023","author":"Wang","key":"mlstadb371bib19"},{"key":"mlstadb371bib20","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1016\/j.radonc.2018.01.014","article-title":"ESTRO ACROP consensus guideline on CT-and MRI-based target volume delineation for primary radiation therapy of localized prostate cancer","volume":"127","author":"Salembier","year":"2018","journal-title":"Radiother. Oncol."},{"article-title":"Attention is all you need","year":"2017","author":"Vaswani","key":"mlstadb371bib21"},{"article-title":"An image is worth 16\u00d716 words: transformers for image recognition at scale","year":"2020","author":"Dosovitskiy","key":"mlstadb371bib22"},{"key":"mlstadb371bib23","doi-asserted-by":"publisher","first-page":"23716","DOI":"10.48550\/arXiv.2204.14198","article-title":"Flamingo: a visual language model for few-shot learning","volume":"35","author":"Alayrac","year":"2022","journal-title":"Advances in Neural Information Processing Systems"},{"key":"mlstadb371bib24","first-page":"2","article-title":"Contrastive learning of medical visual representations from paired images and text","author":"Zhang","year":"2022"},{"key":"mlstadb371bib25","first-page":"11162","article-title":"Learning visual representations from textual annotations","author":"Desai","year":"2021"},{"key":"mlstadb371bib26","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1007\/978-3-030-58598-3_10","article-title":"Learning visual representations with caption annotations","author":"Sariyildiz","year":"2020"},{"article-title":"Learning transferable visual models from natural language supervision","year":"2021","author":"Radford","key":"mlstadb371bib27"},{"key":"mlstadb371bib28","doi-asserted-by":"publisher","first-page":"21152","DOI":"10.48550\/arXiv.2301.00785","article-title":"Clip-driven universal model for organ segmentation and tumor detection","author":"Liu","year":"2023"},{"article-title":"Improving medical multi-modal contrastive learning with expert annotations","year":"2024","author":"Kumar","key":"mlstadb371bib29"},{"key":"mlstadb371bib30","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/ad580f(2022)","article-title":"Performance deterioration of deep learning models after clinical deployment: a case study with auto-segmentation for definitive prostate cancer radiotherapy","volume":"5","author":"Wang","year":"2022","journal-title":"Mach. Learn.: Sci. Technol."},{"key":"mlstadb371bib31","doi-asserted-by":"publisher","first-page":"e74","DOI":"10.1016\/j.urology.2021.05.013","article-title":"SpaceOAR hydrogel spacer for reducing radiation toxicity during radiotherapy for prostate cancer. A systematic review","volume":"156","author":"Armstrong","year":"2021","journal-title":"Urology"},{"key":"mlstadb371bib32","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2021.102101","article-title":"A deep learning-based framework for segmenting invisible clinical target volumes with estimated uncertainties for post-operative prostate cancer radiotherapy","volume":"72","author":"Balagopal","year":"2021","journal-title":"Med. Image Anal."},{"key":"mlstadb371bib33","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1016\/j.radonc.2017.10.016","article-title":"Delineation of the primary tumour clinical target volumes (CTV-P) in laryngeal, hypopharyngeal, oropharyngeal and oral cavity squamous cell carcinoma: AIRO, CACA, DAHANCA, EORTC, GEORCC, GORTEC, HKNPCSG, HNCIG, IAG-KHT, LPRHHT, NCIC CTG, NCRI, NRG Oncology, PHNS, SBRT, SOMERA, SRO, SSHNO, TROG consensus guidelines","volume":"126","author":"Gr\u00e9goire","year":"2018","journal-title":"Radiother. Oncol."}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adb371","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adb371\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adb371","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adb371\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adb371\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adb371\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adb371\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adb371\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,14]],"date-time":"2025-02-14T05:33:27Z","timestamp":1739511207000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adb371"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,14]]},"references-count":33,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,2,14]]},"published-print":{"date-parts":[[2025,3,31]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/adb371","relation":{},"ISSN":["2632-2153"],"issn-type":[{"type":"electronic","value":"2632-2153"}],"subject":[],"published":{"date-parts":[[2025,2,14]]},"assertion":[{"value":"Medical image segmentation assisted with clinical inputs via language encoder in a deep learning framework","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2025 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2024-09-09","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2025-02-06","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2025-02-14","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}