{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T20:13:23Z","timestamp":1788984803453,"version":"build-2803163510"},"reference-count":49,"publisher":"IOP Publishing","issue":"9","license":[{"start":{"date-parts":[[2023,4,25]],"date-time":"2023-04-25T00:00:00Z","timestamp":1682380800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/publishingsupport.iopscience.iop.org\/iop-standard\/v1"},{"start":{"date-parts":[[2023,4,25]],"date-time":"2023-04-25T00:00:00Z","timestamp":1682380800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"name":"National Institute of Health","award":["R01 EB027898"],"award-info":[{"award-number":["R01 EB027898"]}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Phys. Med. Biol."],"published-print":{"date-parts":[[2023,5,7]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    <jats:italic>Objective<\/jats:italic>\n                    . Accurate diagnosis of lymph node metastasis (LNM) is critical in treatment management for patients with head and neck cancer. Positron emission tomography and computed tomography are routinely used for identifying LNM status. However, for small or less fluorodeoxyglucose (FDG) avid nodes, there are always uncertainties in LNM diagnosis. We are aiming to develop a reliable prediction model is for identifying LNM.\n                    <jats:italic>Approach<\/jats:italic>\n                    . In this study, a new automated and reliable multi-objective learning model (ARMO) is proposed. In ARMO, a multi-objective model is introduced to obtain balanced sensitivity and specificity. Meanwhile, confidence is calibrated by introducing individual reliability, whilst the model uncertainty is estimated by a newly defined overall reliability in ARMO. In the training stage, a Pareto-optimal model set is generated. Then all the Pareto-optimal models are used, and a reliable fusion strategy that introduces individual reliability is developed for calibrating the confidence of each output. The overall reliability is calculated to estimate the model uncertainty for each test sample.\n                    <jats:italic>Main results<\/jats:italic>\n                    . The experimental results demonstrated that ARMO obtained more promising results, which the area under the curve, accuracy, sensitivity and specificity can achieve 0.97, 0.93, 0.88 and 0.94, respectively. Meanwhile, based on calibrated confidence and overall reliability, clinicians could pay particular attention to highly uncertain predictions.\n                    <jats:italic>Significance<\/jats:italic>\n                    . In this study, we developed a unified model that can achieve balanced prediction, confidence calibration and uncertainty estimation simultaneously. The experimental results demonstrated that ARMO can obtain accurate and reliable prediction performance.\n                  <\/jats:p>","DOI":"10.1088\/1361-6560\/acca5b","type":"journal-article","created":{"date-parts":[[2023,4,5]],"date-time":"2023-04-05T05:01:06Z","timestamp":1680670866000},"page":"095012","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["ARMO: automated and reliable multi-objective model for lymph node metastasis prediction in head and neck cancer"],"prefix":"10.1088","volume":"68","author":[{"given":"Zhiguo","family":"Zhou","sequence":"first","affiliation":[{"name":"University of Kansas Medical Center"},{"name":"University of Kansas Cancer Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Liyuan","family":"Chen","sequence":"additional","affiliation":[{"name":"University of Texas Southwestern Medical Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Dohopolski","sequence":"additional","affiliation":[{"name":"University of Texas Southwestern Medical Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Sher","sequence":"additional","affiliation":[{"name":"University of Texas Southwestern Medical Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8491-4146","authenticated-orcid":true,"given":"Jing","family":"Wang","sequence":"additional","affiliation":[{"name":"University of Texas Southwestern Medical Center"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2023,4,25]]},"reference":[{"key":"pmbacca5bbib1","doi-asserted-by":"publisher","first-page":"6334","DOI":"10.1002\/mp.14538","type":"journal-article","article-title":"Machine learning prediction of axillary lymph node metastasis in breast cancer: 2D versus 3D radiomic features","volume":"47","author":"Arefan","year":"2020","journal-title":"Med. Phys."},{"key":"pmbacca5bbib2","type":"journal-article","article-title":"Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks","author":"Ayhan","year":"2018","journal-title":"Medical Imaging with Deep Learning"},{"key":"pmbacca5bbib3","first-page":"215","type":"journal-article","article-title":"Ensemble learning in Bayesian neural networks","volume":"168","author":"Barber","year":"1998","journal-title":"Nato ASI Ser. F Comput. Syst. Sci."},{"key":"pmbacca5bbib4","first-page":"p 4","author":"Bishop","year":"2006","type":"book"},{"key":"pmbacca5bbib5","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2020.101653","type":"journal-article","article-title":"Deep learning uncertainty and confidence calibration for the five-class polyp classification from colonoscopy","volume":"62","author":"Carneiro","year":"2020","journal-title":"Med. Image Anal."},{"key":"pmbacca5bbib6","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/ab083a","type":"journal-article","article-title":"Combining many-objective radiomics and 3-dimensional convolutional neural network through evidential reasoning to predict lymph node metastasis in head and neck cancer","author":"Chen","year":"2019","journal-title":"Phys. Med. Biol."},{"key":"pmbacca5bbib7","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1007\/s10479-007-0186-0","type":"journal-article","article-title":"A survey of recent developments in multiobjective optimization","volume":"154","author":"Chinchuluun","year":"2007","journal-title":"Ann. Oper. Res."},{"key":"pmbacca5bbib8","doi-asserted-by":"publisher","first-page":"1911","DOI":"10.1002\/cncr.23654","type":"journal-article","article-title":"Head and neck cancer","volume":"113","author":"Cognetti","year":"2008","journal-title":"Cancer"},{"key":"pmbacca5bbib9","doi-asserted-by":"publisher","first-page":"379","DOI":"10.1097\/CM9.0000000000000095","type":"journal-article","article-title":"Artificial intelligence system of faster region-based convolutional neural network surpassing senior radiologists in evaluation of metastatic lymph nodes of rectal cancer","volume":"132","author":"Ding","year":"2019","journal-title":"Chin. Med. J."},{"key":"pmbacca5bbib10","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/abb71c","type":"journal-article","article-title":"Predicting lymph node metastasis in patients with oropharyngeal cancer by using a convolutional neural network with associated epistemic and aleatoric uncertainty","volume":"65","author":"Dohopolski","year":"2020","journal-title":"Phys. Med. Biol."},{"key":"pmbacca5bbib11","first-page":"1050","author":"Gal","year":"2016","type":"conference-proceedings"},{"key":"pmbacca5bbib12","article-title":"A survey of uncertainty in deep neural networks","author":"Gawlikowski","year":"2021","type":"preprint"},{"key":"pmbacca5bbib13","first-page":"1321","author":"Guo","year":"2017","type":"conference-proceedings"},{"key":"pmbacca5bbib14","doi-asserted-by":"publisher","first-page":"2157","DOI":"10.1200\/JCO.2015.65.9128","type":"journal-article","article-title":"Development and validation of a radiomics nomogram for preoperative prediction of lymph node metastasis in colorectal cancer","volume":"34","author":"Huang","year":"2016","journal-title":"J. Clin. Oncol."},{"key":"pmbacca5bbib15","type":"conference-proceedings","article-title":"Simple and scalable predictive uncertainty estimation using deep ensembles","volume":"30","author":"Lakshminarayanan","year":"2017"},{"key":"pmbacca5bbib16","doi-asserted-by":"publisher","first-page":"3868","DOI":"10.1109\/TMI.2020.3006437","type":"journal-article","article-title":"Confidence calibration and predictive uncertainty estimation for deep medical image segmentation","volume":"39","author":"Mehrtash","year":"2020","journal-title":"IEEE Trans. Med. Imaging"},{"key":"pmbacca5bbib17","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1016\/j.inffus.2019.12.001","type":"journal-article","article-title":"A survey on machine learning for data fusion","volume":"57","author":"Meng","year":"2020","journal-title":"Inf. Fusion"},{"key":"pmbacca5bbib18","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1038\/s42256-020-00278-8","type":"journal-article","article-title":"A biological perspective on evolutionary computation","volume":"3","author":"Miikkulainen","year":"2021","journal-title":"Nat. Mach. Intell."},{"key":"pmbacca5bbib19","doi-asserted-by":"publisher","first-page":"1561","DOI":"10.1097\/01.mlg.0000173202.56739.9f","type":"journal-article","article-title":"Lymph node metastases from cutaneous squamous cell carcinoma of the head and neck","volume":"115","author":"Moore","year":"2005","journal-title":"Laryngoscope"},{"key":"pmbacca5bbib20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-020-61808-3","type":"journal-article","article-title":"Test-time augmentation for deep learning-based cell segmentation on microscopy images","volume":"10","author":"Moshkov","year":"2020","journal-title":"Sci. Rep."},{"key":"pmbacca5bbib21","type":"conference-proceedings","article-title":"When does label smoothing help?","volume":"32","author":"M\u00fcller","year":"2019"},{"key":"pmbacca5bbib22","first-page":"38","author":"Nixon","year":"2019","type":"conference-proceedings"},{"key":"pmbacca5bbib23","doi-asserted-by":"publisher","first-page":"448","DOI":"10.1038\/nrc1370","type":"journal-article","article-title":"Dissecting the metastatic cascade","volume":"4","author":"Pantel","year":"2004","journal-title":"Nat. Rev. Cancer"},{"key":"pmbacca5bbib24","first-page":"5281","author":"Raghu","year":"2019","type":"conference-proceedings"},{"key":"pmbacca5bbib25","author":"Ritter","year":"2018","type":"conference-proceedings"},{"key":"pmbacca5bbib26","volume":"vol 1","author":"Shafer","year":"1976","type":"book"},{"key":"pmbacca5bbib27","type":"journal-article","article-title":"On mixup training: improved calibration and predictive uncertainty for deep neural networks","volume":"32","author":"Thulasidasan","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"pmbacca5bbib28","article-title":"Deep sub-ensembles for fast uncertainty estimation in image classification","author":"Valdenegro-Toro","year":"2019","type":"preprint"},{"key":"pmbacca5bbib29","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1016\/j.neucom.2019.01.103","type":"journal-article","article-title":"Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks","volume":"338","author":"Wang","year":"2019","journal-title":"Neurocomputing"},{"key":"pmbacca5bbib30","first-page":"61","author":"Wang","year":"2018","type":"conference-proceedings"},{"key":"pmbacca5bbib31","doi-asserted-by":"publisher","first-page":"1885","DOI":"10.1016\/j.ejor.2004.09.059","type":"journal-article","article-title":"Environmental impact assessment using the evidential reasoning approach","volume":"174","author":"Wang","year":"2006","journal-title":"Eur. J. Oper. Res."},{"key":"pmbacca5bbib32","doi-asserted-by":"publisher","first-page":"455","DOI":"10.1097\/RCT.0000000000000555","type":"journal-article","article-title":"Support vector machines model of computed tomography for assessing lymph node metastasis in esophageal cancer with neoadjuvant chemotherapy","volume":"41","author":"Wang","year":"2017","journal-title":"J. Comput. Assist. Tomogr."},{"key":"pmbacca5bbib33","article-title":"Batchensemble: an alternative approach to efficient ensemble and lifelong learning","author":"Wen","year":"2020","type":"preprint"},{"key":"pmbacca5bbib34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-019-40831-z","type":"journal-article","article-title":"Preoperative prediction of axillary lymph node metastasis in breast cancer using mammography-based radiomics method","volume":"9","author":"Yang","year":"2019","journal-title":"Sci. Rep."},{"key":"pmbacca5bbib35","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1109\/TSMCA.2002.802746","type":"journal-article","article-title":"On the evidential reasoning algorithm for multiple attribute decision analysis under uncertainty","volume":"32","author":"Yang","year":"2002","journal-title":"IEEE Trans. Syst. Man Cybern. A"},{"key":"pmbacca5bbib36","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.artint.2013.09.003","type":"journal-article","article-title":"Evidential reasoning rule for evidence combination","volume":"205","author":"Yang","year":"2013","journal-title":"Artif. Intell."},{"key":"pmbacca5bbib37","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41467-020-18497-3","type":"journal-article","article-title":"Lymph node metastasis prediction of papillary thyroid carcinoma based on transfer learning radiomics","volume":"11","author":"Yu","year":"2020","journal-title":"Nat. Commun."},{"key":"pmbacca5bbib38","article-title":"Confidence calibration for convolutional neural networks using structured dropout","author":"Zhang","year":"2019","type":"preprint"},{"key":"pmbacca5bbib39","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1016\/j.lungcan.2020.04.014","type":"journal-article","article-title":"A cross-modal 3D deep learning for accurate lymph node metastasis prediction in clinical stage T1 lung adenocarcinoma","volume":"145","author":"Zhao","year":"2020a","journal-title":"Lung Cancer"},{"key":"pmbacca5bbib40","first-page":"4837","author":"Zhao","year":"2020b","type":"conference-proceedings"},{"key":"pmbacca5bbib41","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1148\/radiol.2019190372","type":"journal-article","article-title":"Lymph node metastasis prediction from primary breast cancer US images using deep learning","volume":"294","author":"Zhou","year":"2020a","journal-title":"Radiology"},{"key":"pmbacca5bbib42","first-page":"1","author":"Zhou","year":"2018a","type":"conference-proceedings"},{"key":"pmbacca5bbib43","doi-asserted-by":"crossref","DOI":"10.1109\/EMBC.2018.8513070","type":"preprint","article-title":"Predicting lymph node metastasis in head and neck cancer by combining many-objective radiomics and 3-dimensioal convolutional neural network through evidential reasoning","author":"Zhou","year":"2018b"},{"key":"pmbacca5bbib44","first-page":"1","author":"Zhou","year":"2019","type":"conference-proceedings"},{"key":"pmbacca5bbib45","doi-asserted-by":"publisher","first-page":"501","DOI":"10.1016\/j.radonc.2016.04.029","type":"journal-article","article-title":"Predicting distant failure in early stage NSCLC treated with SBRT using clinical parameters","volume":"119","author":"Zhou","year":"2016","journal-title":"Radiother. Oncol."},{"key":"pmbacca5bbib46","doi-asserted-by":"publisher","first-page":"4460","DOI":"10.1088\/1361-6560\/aa6ae5","type":"journal-article","article-title":"Multi-objective radiomics model for predicting distant failure in lung SBRT","volume":"62","author":"Zhou","year":"2017a","journal-title":"Phys. Med. Biol."},{"key":"pmbacca5bbib47","doi-asserted-by":"publisher","first-page":"4460","DOI":"10.1088\/1361-6560\/aa6ae5","type":"journal-article","article-title":"Multi-objective radiomics model for predicting distant failure in lung SBRT","volume":"62","author":"Zhou","year":"2017b","journal-title":"Phys. Med. Biol."},{"key":"pmbacca5bbib48","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/ab8956","type":"journal-article","article-title":"Multifaceted radiomics for distant metastasis prediction in head & neck cancer","volume":"65","author":"Zhou","year":"2020b","journal-title":"Phys. Med. Biol."},{"key":"pmbacca5bbib49","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1016\/j.knosys.2015.04.019","type":"journal-article","article-title":"A cooperative belief rule based decision support system for lymph node metastasis diagnosis in gastric cancer","volume":"85","author":"Zhou","year":"2015","journal-title":"Knowl.-Based Syst."}],"container-title":["Physics in Medicine &amp; Biology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/acca5b","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/acca5b\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/acca5b","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/acca5b\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/acca5b\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/acca5b\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/acca5b\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/acca5b\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T16:26:00Z","timestamp":1778775960000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/acca5b"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,25]]},"references-count":49,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2023,4,25]]},"published-print":{"date-parts":[[2023,5,7]]}},"URL":"https:\/\/doi.org\/10.1088\/1361-6560\/acca5b","relation":{},"ISSN":["0031-9155","1361-6560"],"issn-type":[{"value":"0031-9155","type":"print"},{"value":"1361-6560","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,25]]},"assertion":[{"value":"ARMO: automated and reliable multi-objective model for lymph node metastasis prediction in head and neck cancer","name":"article_title","label":"Article Title"},{"value":"Physics in Medicine & Biology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2023 Institute of Physics and Engineering in Medicine. All rights, including for text and data mining, AI training, and similar technologies, are reserved.","name":"copyright_information","label":"Copyright Information"},{"value":"2022-11-07","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2023-04-04","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2023-04-25","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}