{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,27]],"date-time":"2026-10-27T19:24:39Z","timestamp":1793129079872,"version":"4.5.0"},"reference-count":32,"publisher":"IOP Publishing","issue":"20","license":[{"start":{"date-parts":[[2024,10,10]],"date-time":"2024-10-10T00:00:00Z","timestamp":1728518400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2024,10,10]],"date-time":"2024-10-10T00:00:00Z","timestamp":1728518400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"name":"Wilson-Olegario Foundation"},{"DOI":"10.13039\/100007065","name":"NVIDIA Corporation","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100007065","id-type":"DOI","asserted-by":"crossref"}]},{"name":"City of London Radiation Research Unit at the Cancer Research UK","award":["C7893\/A28990"],"award-info":[{"award-number":["C7893\/A28990"]}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Phys. Med. Biol."],"published-print":{"date-parts":[[2024,10,21]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>\n                  <jats:italic>Objective.<\/jats:italic> Normal tissue complication probability (NTCP) modelling is rapidly embracing deep learning (DL) methods, acknowledging the importance of spatial dose information. Finding effective ways to combine information from radiation dose distribution maps (dosiomics) and clinical data involves technical challenges and requires domain knowledge. We propose different multi-modality data fusion strategies to facilitate future DL-based NTCP studies. <jats:italic>Approach.<\/jats:italic> Early, joint and late DL multi-modality fusion strategies were compared using clinical and mandibular radiation dose distribution volumes. These were contrasted with single-modality models: a random forest trained on non-image data (clinical, demographic and dose-volume metrics) and a 3D DenseNet-40 trained on image data (mandibular dose distribution maps). The study involved a matched cohort of 92 osteoradionecrosis cases and 92 controls from a single institution. <jats:italic>Main results.<\/jats:italic> The late fusion model exhibited superior discrimination and calibration performance, while the join fusion achieved a more balanced distribution of the predicted probabilities. Discrimination performance did not significantly differ between strategies. Late fusion, though less technically complex, lacks crucial inter-modality interactions for NTCP modelling. In contrast, joint fusion, despite its complexity, resulted in a single network training process which included intra- and inter-modality interactions in its model parameter optimisation. <jats:italic>Significance.<\/jats:italic> This study is a pioneering effort in comparing different strategies for including image data into DL-based NTCP models in combination with lower dimensional data such as clinical variables. The discrimination performance of such multi-modality NTCP models and the choice of fusion strategy will depend on the distribution and quality of both types of data. Multiple data fusion strategies should be compared and reported in multi-modality NTCP modelling using DL.<\/jats:p>","DOI":"10.1088\/1361-6560\/ad8290","type":"journal-article","created":{"date-parts":[[2024,10,2]],"date-time":"2024-10-02T22:58:27Z","timestamp":1727909907000},"page":"20NT01","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Comparison of deep-learning multimodality data fusion strategies in mandibular osteoradionecrosis NTCP modelling using clinical variables and radiation dose distribution volumes"],"prefix":"10.1088","volume":"69","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8005-6770","authenticated-orcid":true,"given":"Laia","family":"Humbert-Vidan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vinod","family":"Patel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew P","family":"King","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Teresa Guerrero","family":"Urbano","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2024,10,10]]},"reference":[{"key":"pmbad8290bib1","doi-asserted-by":"publisher","first-page":"1373","DOI":"10.1080\/0284186X.2019.1643037","article-title":"Osteoradionecrosis of the mandible after radiotherapy for head and neck cancer: risk factors and dose-volume correlations","volume":"58","author":"Aarup-Kristensen","year":"2019","journal-title":"Acta Oncol."},{"key":"pmbad8290bib2","doi-asserted-by":"publisher","first-page":"e87","DOI":"10.1016\/j.clon.2021.12.002","article-title":"Deep learning for radiotherapy outcome prediction using dose data\u2014a review","volume":"34","author":"Appelt","year":"2022","journal-title":"Clin. Oncol."},{"key":"pmbad8290bib3","doi-asserted-by":"publisher","first-page":"3279","DOI":"10.1109\/TKDE.2021.3126456","article-title":"A general survey on attention mechanisms in deep learning","volume":"35","author":"Brauwers","year":"2023","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"pmbad8290bib4","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"pmbad8290bib5","doi-asserted-by":"publisher","DOI":"10.1136\/bmj-2023-078378","article-title":"TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods","volume":"385","author":"Collins","year":"2024","journal-title":"BMJ"},{"key":"pmbad8290bib6","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1007\/s11912-020-00954-3","article-title":"Radiation therapy and mandibular osteoradionecrosis: state of the art","volume":"22","author":"De Felice","year":"2020","journal-title":"Curr. Oncol. Rep."},{"key":"pmbad8290bib7","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1016\/j.radonc.2016.05.015","article-title":"Normal tissue complication probability (ntcp) modelling using spatial dose metrics and machine learning methods for severe acute oral mucositis resulting from head and neck radiotherapy","volume":"120","author":"Dean","year":"2016","journal-title":"Radiother. Oncol."},{"key":"pmbad8290bib8","doi-asserted-by":"publisher","first-page":"837","DOI":"10.2307\/2531595","article-title":"Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach","volume":"44","author":"DeLong","year":"1988","journal-title":"Biometrics"},{"key":"pmbad8290bib9","doi-asserted-by":"publisher","first-page":"12TR01","DOI":"10.1088\/1361-6560\/ac0681","article-title":"Spatial descriptions of radiotherapy dose: normal tissue complication models and statistical associations","volume":"66","author":"Ebert","year":"2021","journal-title":"Phys. Med. Biol."},{"key":"pmbad8290bib10","author":"El Naqa","year":"2022"},{"key":"pmbad8290bib11","doi-asserted-by":"publisher","first-page":"1206","DOI":"10.1016\/j.ijrobp.2020.12.043","article-title":"Osteoradionecrosis: exposing the evidence not the bone","volume":"109","author":"Frankart","year":"2021","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"pmbad8290bib12","doi-asserted-by":"publisher","first-page":"35","DOI":"10.3389\/fonc.2018.00035","article-title":"Design and selection of machine learning methods using radiomics and dosiomics for normal tissue complication probability modeling of xerostomia","volume":"8","author":"Gabry\u015b","year":"2018","journal-title":"Front. Oncol."},{"key":"pmbad8290bib13","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR52688.2022.02005","article-title":"Multimodal dynamics: dynamical fusion for trustworthy multimodal classification","author":"Han","year":"2022"},{"key":"pmbad8290bib14","doi-asserted-by":"publisher","first-page":"283","DOI":"10.1016\/S0167-8140(00)00236-X","article-title":"Volume effects in radiobiology as applied to radiotherapy","volume":"56","author":"Hopewell","year":"2000","journal-title":"Radiother. Oncol."},{"key":"pmbad8290bib15","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2017.243","article-title":"Densely connected convolutional networks","author":"Huang","year":"2017"},{"key":"pmbad8290bib16","doi-asserted-by":"publisher","first-page":"136","DOI":"10.1038\/s41746-020-00341-z","article-title":"Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines","volume":"3","author":"Huang","year":"2020","journal-title":"npj Digit. Med."},{"key":"pmbad8290bib17","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1016\/j.radonc.2022.09.014","article-title":"Protocol letter: a multi-institutional retrospective case-control cohort investigating PREDiction models for Mandibular OsteoRadioNecrosis in head and neck cancer (PREDMORN)","volume":"176","author":"Humbert-Vidan","year":"2022a","journal-title":"Radiother. Oncol."},{"key":"pmbad8290bib18","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1007\/978-3-031-17721-7_6","article-title":"Prediction of mandibular ORN incidence from 3D radiation dose distribution maps using deep learning","volume":"13540","author":"Humbert-Vidan","year":"2022b","journal-title":"Appl. Med. Artif. Intell."},{"key":"pmbad8290bib19","doi-asserted-by":"publisher","first-page":"4763","DOI":"10.1002\/mp.13122","article-title":"Development of deep neural network for individualized hepatobiliary toxicity prediction after liver SBRT","volume":"45","author":"Ibragimov","year":"2018","journal-title":"Med. Phys."},{"key":"pmbad8290bib20","doi-asserted-by":"publisher","first-page":"S10","DOI":"10.1016\/j.ijrobp.2009.07.1754","article-title":"Use of normal tissue complication probability models in the clinic","volume":"76","author":"Marks","year":"2010","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"pmbad8290bib21","doi-asserted-by":"publisher","first-page":"440","DOI":"10.1016\/j.ijrobp.2019.06.009","article-title":"A deep learning model for predicting xerostomia due to radiation therapy for head and neck squamous cell carcinoma in the RTOG 0522 clinical trial","volume":"105","author":"Men","year":"2019","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"pmbad8290bib22","doi-asserted-by":"publisher","first-page":"232","DOI":"10.1016\/j.radonc.2017.06.026","article-title":"Dose-volume correlates of mandibular osteoradionecrosis in oropharynx cancer patients receiving intensity modulated radiotherapy: results from a case-matched comparison","volume":"124","author":"Mohamed","year":"2017","journal-title":"Radiother. Oncol."},{"key":"pmbad8290bib23","doi-asserted-by":"publisher","first-page":"7220","DOI":"10.1038\/s41598-017-07586-x","article-title":"Voxel-based analysis unveils regional dose differences associated with radiation-induced morbidity in head and neck cancer patients","volume":"7","author":"Monti","year":"2017","journal-title":"Sci. Rep."},{"key":"pmbad8290bib24","doi-asserted-by":"publisher","first-page":"98","DOI":"10.1016\/j.oraloncology.2017.07.014","article-title":"Incidence of, and risk factors for, mandibular osteoradionecrosis in patients with oral cavity and oropharynx cancers","volume":"72","author":"Moon","year":"2017","journal-title":"Oral Oncol."},{"key":"pmbad8290bib25","doi-asserted-by":"publisher","DOI":"10.1016\/j.oraloncology.2022.106056","article-title":"Osteoradionecrosis after postoperative radiotherapy for oral cavity cancer: a retrospective cohort study","volume":"133","author":"M\u00f6ring","year":"2022","journal-title":"Oral Oncol."},{"key":"pmbad8290bib26","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1016\/S0915-6992(02)80014-0","article-title":"Osteoradionecrosis of the mandible\u2014factors influencing severity","volume":"14","author":"Notani","year":"2002","journal-title":"Asian J. Oral Maxillofac. Surg."},{"key":"pmbad8290bib27","doi-asserted-by":"publisher","first-page":"64","DOI":"10.12968\/denu.2022.49.1.64","article-title":"Osteoradionecrosis in the current era of radiation treatment","volume":"49","author":"Patel","year":"2022","journal-title":"Dental Update"},{"key":"pmbad8290bib28","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1109\/MSP.2017.2738401","article-title":"Deep multimodal learning: a survey on recent advances and trends","volume":"34","author":"Ramachandram","year":"2017","journal-title":"IEEE Signal Process. Mag."},{"key":"pmbad8290bib29","doi-asserted-by":"publisher","DOI":"10.1016\/j.adro.2022.101163","article-title":"Comparison of machine-learning and deep-learning methods for the prediction of osteoradionecrosis resulting from head and neck cancer radiation therapy","volume":"8","author":"Reber","year":"2022","journal-title":"Adv. Radiat. Oncol."},{"key":"pmbad8290bib30","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1186\/1471-2105-12-77","article-title":"proc: an open-source package for R and S+ to analyze and compare ROC curves","volume":"12","author":"Robin","year":"2011","journal-title":"BMC Bioinf."},{"key":"pmbad8290bib31","doi-asserted-by":"publisher","first-page":"549","DOI":"10.1016\/j.ijrobp.2021.04.042","article-title":"Normal tissue complication probability (NTCP) prediction model for osteoradionecrosis of the mandible in patients with head and neck cancer after radiation therapy: large-scale observational cohort","volume":"111","author":"van Dijk","year":"2021","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"pmbad8290bib32","article-title":"Dynamic multimodal fusion","author":"Xue","year":"2022"}],"container-title":["Physics in Medicine &amp; Biology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad8290","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad8290\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad8290","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad8290\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad8290\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad8290\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad8290\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad8290\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,10]],"date-time":"2024-10-10T13:40:45Z","timestamp":1728567645000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad8290"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,10]]},"references-count":32,"journal-issue":{"issue":"20","published-online":{"date-parts":[[2024,10,10]]},"published-print":{"date-parts":[[2024,10,21]]}},"URL":"https:\/\/doi.org\/10.1088\/1361-6560\/ad8290","relation":{},"ISSN":["0031-9155","1361-6560"],"issn-type":[{"value":"0031-9155","type":"print"},{"value":"1361-6560","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,10]]},"assertion":[{"value":"Comparison of deep-learning multimodality data fusion strategies in mandibular osteoradionecrosis NTCP modelling using clinical variables and radiation dose distribution volumes","name":"article_title","label":"Article Title"},{"value":"Physics in Medicine & Biology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2024 The Author(s). Published on behalf of Institute of Physics and Engineering in Medicine by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2024-05-14","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-10-01","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-10-10","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}