{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T20:13:29Z","timestamp":1788984809772,"version":"build-2803163510"},"reference-count":43,"publisher":"Wiley","issue":"4","license":[{"start":{"date-parts":[[2022,12,17]],"date-time":"2022-12-17T00:00:00Z","timestamp":1671235200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01CA251792"],"award-info":[{"award-number":["R01CA251792"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["aapm.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Medical Physics"],"published-print":{"date-parts":[[2023,4]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Purpose<\/jats:title>\n                    <jats:p>A reliable locoregional recurrence (LRR) prediction model is important for the personalized management of head and neck cancers (HNC) patients who received radiotherapy. This work aims to develop a delta\u2010radiomics feature\u2010based multi\u2010classifier, multi\u2010objective, and multi\u2010modality (Delta\u2010mCOM) model for post\u2010treatment HNC LRR prediction. Furthermore, we aim to adopt a learning with rejection option (LRO) strategy to boost the reliability of Delta\u2010mCOM model by rejecting prediction for samples with high prediction uncertainties.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>In this retrospective study, we collected PET\/CT image and clinical data from 224 HNC patients who received radiotherapy (RT) at our institution. We calculated the differences between radiomics features extracted from PET\/CT images acquired before and after radiotherapy and used them in conjunction with pre\u2010treatment radiomics features as the input features. Using clinical parameters, PET radiomics features, and CT radiomics features, we built and optimized three separate single\u2010modality models. We used multiple classifiers for model construction and employed sensitivity and specificity simultaneously as the training objectives for each of them. Then, for testing samples, we fused the output probabilities from all these single\u2010modality models to obtain the final output probabilities of the Delta\u2010mCOM model. In the LRO strategy, we estimated the epistemic and aleatoric uncertainties when predicting with a trained Delta\u2010mCOM model and identified patients associated with prediction of higher reliability (low uncertainty estimates). The epistemic and aleatoric uncertainties were estimated using an AutoEncoder\u2010style anomaly detection model and test\u2010time augmentation (TTA) with predictions made from the Delta\u2010mCOM model, respectively. Predictions with higher epistemic uncertainty or higher aleatoric uncertainty than given thresholds were deemed unreliable, and they were rejected before providing a final prediction. In this study, different thresholds corresponding to different low\u2010reliability prediction rejection ratios were applied. Their values are based on the estimated epistemic and aleatoric uncertainties distribution of the validation data.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The Delta\u2010mCOM model performed significantly better than the single\u2010modality models, whether trained with pre\u2010, post\u2010treatment radiomics features or concatenated BaseLine and Delta\u2010Radiomics Features (BL\u2010DRFs). It was numerically superior to the PET and CT fused BL\u2010DRF model (nonstatistically significant). Using the LRO strategy for the Delta\u2010mCOM model, most of the evaluation metrics improved as the rejection ratio increased from 0% to around 25%. Utilizing both epistemic and aleatoric uncertainty for rejection yielded nonstatistically significant improved metrics compared to each alone at approximately a 25% rejection ratio. Metrics were significantly better than the no\u2010rejection method when the reject ratio was higher than 50%.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>The inclusion of the delta\u2010radiomics feature improved the accuracy of HNC LRR prediction, and the proposed Delta\u2010mCOM model can give more reliable predictions by rejecting predictions for samples of high uncertainty using the LRO strategy.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1002\/mp.16132","type":"journal-article","created":{"date-parts":[[2022,12,9]],"date-time":"2022-12-09T06:10:23Z","timestamp":1670566223000},"page":"2212-2223","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":25,"title":["Towards reliable head and neck cancers locoregional recurrence prediction using delta\u2010radiomics and learning with rejection option"],"prefix":"10.1002","volume":"50","author":[{"given":"Kai","family":"Wang","sequence":"first","affiliation":[{"name":"Department of Radiation Oncology UT Southwestern Medical Center  Dallas Texas USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Dohopolski","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology UT Southwestern Medical Center  Dallas Texas USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qiongwen","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology UT Southwestern Medical Center  Dallas Texas USA"},{"name":"Department of Head and Neck Oncology State Key Laboratory of Biotherapy and Cancer Center West China Hospital Sichuan University  Chengdu China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Sher","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology UT Southwestern Medical Center  Dallas Texas USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jing","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology UT Southwestern Medical Center  Dallas Texas USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2022,12,17]]},"reference":[{"key":"e_1_2_9_2_1","doi-asserted-by":"publisher","DOI":"10.1002\/lary.23341"},{"key":"e_1_2_9_3_1","doi-asserted-by":"publisher","DOI":"10.1186\/1748-717X-6-54"},{"key":"e_1_2_9_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00259-011-1893-y"},{"key":"e_1_2_9_5_1","doi-asserted-by":"publisher","DOI":"10.2214\/AJR.17.18301"},{"key":"e_1_2_9_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0959-8049(99)00229-4"},{"key":"e_1_2_9_7_1","doi-asserted-by":"publisher","DOI":"10.2214\/AJR.14.14166"},{"key":"e_1_2_9_8_1","doi-asserted-by":"publisher","DOI":"10.1038\/bjc.2013.668"},{"key":"e_1_2_9_9_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.14388"},{"key":"e_1_2_9_10_1","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2015.00272"},{"key":"e_1_2_9_11_1","article-title":"Radiomics strategies for risk assessment of tumour failure in head\u2010and\u2010neck cancer","volume":"7","author":"Vallieres M","year":"2017","journal-title":"Sci Rep\u2010Uk"},{"key":"e_1_2_9_12_1","doi-asserted-by":"publisher","DOI":"10.21037\/tcr.2016.07.18"},{"key":"e_1_2_9_13_1","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/ab8956"},{"key":"e_1_2_9_14_1","doi-asserted-by":"publisher","DOI":"10.1038\/srep13087"},{"key":"e_1_2_9_15_1","doi-asserted-by":"publisher","DOI":"10.1186\/s40644-019-0283-8"},{"key":"e_1_2_9_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2884126"},{"key":"e_1_2_9_17_1","doi-asserted-by":"crossref","unstructured":"MorganHE WangK DohopolskiM et\u00a0al.Exploratory ensemble interpretable model for predicting local failure in head and neck cancer: the additive benefit of CT and intra\u2010treatment cone\u2010beam computed tomography features.2021;11(12):4781\u20104796.","DOI":"10.21037\/qims-21-274"},{"key":"e_1_2_9_18_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13014-019-1246-8"},{"key":"e_1_2_9_19_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.14896"},{"key":"e_1_2_9_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2021.104916"},{"key":"e_1_2_9_21_1","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/ab6f51"},{"key":"e_1_2_9_22_1","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.2015151169"},{"key":"e_1_2_9_23_1","doi-asserted-by":"crossref","unstructured":"GaoJ YaoJ ShaoY Towards Reliable Learning for High Stakes Applications. Paper presented at: Proceedings of the AAAI Conference on Artificial Intelligence 2019.","DOI":"10.1609\/aaai.v33i01.33013614"},{"key":"e_1_2_9_24_1","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/ab083a"},{"key":"e_1_2_9_25_1","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/aa6ae5"},{"key":"e_1_2_9_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijrobp.2018.07.128"},{"key":"e_1_2_9_27_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-017-00665-z"},{"key":"e_1_2_9_28_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13014-019-1246-8"},{"key":"e_1_2_9_29_1","doi-asserted-by":"crossref","unstructured":"CortesC DeSalvoG MohriM Learning with rejection. Paper presented at: International Conference on Algorithmic Learning Theory 2016.","DOI":"10.1007\/978-3-319-46379-7_5"},{"key":"e_1_2_9_30_1","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.2020191145"},{"key":"e_1_2_9_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2005.159"},{"key":"e_1_2_9_32_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.953"},{"issue":"8","key":"e_1_2_9_33_1","first-page":"1823","article-title":"Classification with a reject option using a hinge loss","volume":"9","author":"Bartlett PL","year":"2008","journal-title":"J Mach Learn Res"},{"key":"e_1_2_9_34_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2013.07.030"},{"key":"e_1_2_9_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcp.2010.03.003"},{"key":"e_1_2_9_36_1","article-title":"What uncertainties do we need in bayesian deep learning for computer vision?","volume":"30","author":"Kendall A","year":"2017","journal-title":"Adv Neural Inf Process Syst"},{"key":"e_1_2_9_37_1","first-page":"1","article-title":"Pyod: a python toolbox for scalable outlier detection","volume":"20","author":"Zhao Y","year":"2019","journal-title":"J Mach Learn Res"},{"key":"e_1_2_9_38_1","doi-asserted-by":"crossref","unstructured":"BaurC WiestlerB AlbarqouniS NavabN Deep autoencoding models for unsupervised anomaly segmentation in brain MR images. Paper presented at: International MICCAI Brainlesion Workshop 2018.","DOI":"10.1007\/978-3-030-11723-8_16"},{"key":"e_1_2_9_39_1","unstructured":"AyhanMS BerensP.Test\u2010time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks. Paper presented at: MIDL2018."},{"key":"e_1_2_9_40_1","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/abb71c"},{"key":"e_1_2_9_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/LSP.2014.2337313"},{"key":"e_1_2_9_42_1","unstructured":"TeamRC.R: a language and environment for statistical computing.2013."},{"key":"e_1_2_9_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00259-019-04391-8"},{"key":"e_1_2_9_44_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijrobp.2021.02.004"}],"container-title":["Medical Physics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/mp.16132","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/mp.16132","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/mp.16132","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,1]],"date-time":"2025-08-01T10:23:53Z","timestamp":1754043833000},"score":1,"resource":{"primary":{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/10.1002\/mp.16132"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,17]]},"references-count":43,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,4]]}},"alternative-id":["10.1002\/mp.16132"],"URL":"https:\/\/doi.org\/10.1002\/mp.16132","archive":["Portico"],"relation":{},"ISSN":["0094-2405","2473-4209"],"issn-type":[{"value":"0094-2405","type":"print"},{"value":"2473-4209","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,17]]},"assertion":[{"value":"2021-10-21","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-11-20","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-12-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}