{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T04:35:59Z","timestamp":1787027759141,"version":"build-2736575974"},"reference-count":53,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2023,5,16]],"date-time":"2023-05-16T00:00:00Z","timestamp":1684195200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institute of General Medical Sciences of the National Institutes of Health","doi-asserted-by":"publisher","award":["T32GM008620"],"award-info":[{"award-number":["T32GM008620"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Cancers"],"abstract":"<jats:p>Despite diagnostic advancements, the development of reliable prognostic systems for assessing the risk of cancer recurrence still remains a challenge. In this study, we developed a novel framework to generate highly representative machine-learning prediction models for oral tongue squamous cell carcinoma (OTSCC) cancer recurrence. We identified cases of 5- and 10-year OTSCC recurrence from the SEER database. Four classification models were trained using the H2O ai platform, whose performances were assessed according to their accuracy, recall, precision, and the area under the curve (AUC) of their receiver operating characteristic (ROC) curves. By evaluating Shapley additive explanation contribution plots, feature importance was studied. Of the 130,979 patients studied, 36,042 (27.5%) were female, and the mean (SD) age was 58.2 (13.7) years. The Gradient Boosting Machine model performed the best, achieving 81.8% accuracy and 97.7% precision for 5-year prediction. Moreover, 10-year predictions demonstrated 80.0% accuracy and 94.0% precision. The number of prior tumors, patient age, the site of cancer recurrence, and tumor histology were the most significant predictors. The implementation of our novel SEER framework enabled the successful identification of patients with OTSCC recurrence, with which highly accurate and sensitive prediction models were generated. Thus, we demonstrate our framework\u2019s potential for application in various cancers to build generalizable screening tools to predict tumor recurrence.<\/jats:p>","DOI":"10.3390\/cancers15102769","type":"journal-article","created":{"date-parts":[[2023,5,16]],"date-time":"2023-05-16T02:27:04Z","timestamp":1684204024000},"page":"2769","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Development of a Machine Learning Model to Predict Recurrence of Oral Tongue Squamous Cell Carcinoma"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8660-1621","authenticated-orcid":false,"given":"Yasaman","family":"Fatapour","sequence":"first","affiliation":[{"name":"Department of Biomedical Engineering, University of California, Irvine, CA 92617, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2656-1060","authenticated-orcid":false,"given":"Arash","family":"Abiri","sequence":"additional","affiliation":[{"name":"Department of Biomedical Engineering, University of California, Irvine, CA 92617, USA"},{"name":"Department of Otolaryngology-Head and Neck Surgery, University of California, Irvine, CA 92604, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Edward C.","family":"Kuan","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology-Head and Neck Surgery, University of California, Irvine, CA 92604, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7995-5197","authenticated-orcid":false,"given":"James P.","family":"Brody","sequence":"additional","affiliation":[{"name":"Department of Biomedical Engineering, University of California, Irvine, CA 92617, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,5,16]]},"reference":[{"key":"ref_1","unstructured":"Noone, A.M., Howlader, N., Krapcho, M., Miller, D., Brest, A., Yu, M., and Cronin, K.A. (2018). SEER Cancer Statistics Review, National Cancer Institute."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"7877","DOI":"10.1038\/s41598-020-64748-0","article-title":"Increasing Incidence and Improving Survival of Oral Tongue Squamous Cell Carcinoma","volume":"10","author":"Kim","year":"2020","journal-title":"Sci. Rep."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1488","DOI":"10.1200\/JCO.2010.31.7883","article-title":"Increasing Incidence of Oral Tongue Squamous Cell Carcinoma in Young White Women, Age 18 to 44 Years","volume":"29","author":"Patel","year":"2011","journal-title":"J. Clin. Oncol. Off. J. Am. Soc. Clin. Oncol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1016\/j.oraloncology.2008.06.002","article-title":"Global Epidemiology of Oral and Oropharyngeal Cancer","volume":"45","author":"Warnakulasuriya","year":"2009","journal-title":"Oral Oncol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2076","DOI":"10.1002\/lary.27720","article-title":"Oral Tongue Squamous Cell Carcinoma Survival as Stratified by Age and Sex: A Surveillance, Epidemiology, and End Results Analysis","volume":"129","author":"Mukdad","year":"2019","journal-title":"Laryngoscope"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"170","DOI":"10.1159\/000328340","article-title":"de Q.C. Oral Squamous Cell Carcinoma: Clinicopathological Features in Patients with and without Recurrence","volume":"73","author":"Camisasca","year":"2011","journal-title":"ORL J. Oto-Rhino-Laryngol. Its Relat. Spec."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1309","DOI":"10.1016\/j.jcms.2015.05.011","article-title":"Recurrence Rate and Shift in Histopathological Differentiation of Oral Squamous Cell Carcinoma\u2014A Long-Term Retrospective Study over a Period of 13.5 Years","volume":"43","author":"Ermer","year":"2015","journal-title":"J. Cranio-Maxillo-Facial Surg. Off. Publ. Eur. Assoc. Cranio-Maxillo-Facial Surg."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"614","DOI":"10.5732\/cjc.012.10219","article-title":"The Recurrence and Survival of Oral Squamous Cell Carcinoma: A Report of 275 Cases","volume":"32","author":"Wang","year":"2013","journal-title":"Chin. J. Cancer"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"e2114723","DOI":"10.1001\/jamanetworkopen.2021.14723","article-title":"Performance of a Machine Learning Algorithm Using Electronic Health Record Data to Identify and Estimate Survival in a Longitudinal Cohort of Patients with Lung Cancer","volume":"4","author":"Yuan","year":"2021","journal-title":"JAMA Netw. Open"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"100352","DOI":"10.1016\/j.ctarc.2021.100352","article-title":"A Genetic Risk Score for Glioblastoma Multiforme Based on Copy Number Variations","volume":"27","author":"Ko","year":"2021","journal-title":"Cancer Treat. Res. Commun."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Karatza, P., Dalakleidi, K., Athanasiou, M., and Nikita, K.S. (2021, January 1\u20135). Interpretability Methods of Machine Learning Algorithms with Applications in Breast Cancer Diagnosis. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine & Biology Society, Virtual.","DOI":"10.1109\/EMBC46164.2021.9630556"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"e2025881","DOI":"10.1001\/jamanetworkopen.2020.25881","article-title":"Machine Learning-Guided Adjuvant Treatment of Head and Neck Cancer","volume":"3","author":"Howard","year":"2020","journal-title":"JAMA Netw. Open"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"288","DOI":"10.3389\/fendo.2019.00288","article-title":"Creating Prognostic Systems for Well-Differentiated Thyroid Cancer Using Machine Learning","volume":"10","author":"Yang","year":"2019","journal-title":"Front. Endocrinol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1186\/s13040-021-00253-y","article-title":"Genetic Risk Score for Ovarian Cancer Based on Chromosomal-Scale Length Variation","volume":"14","author":"Toh","year":"2021","journal-title":"BioData Min."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1571","DOI":"10.1038\/s41598-021-81188-6","article-title":"A Machine Learning-Based Predictor for the Identification of the Recurrence of Patients with Gastric Cancer after Operation","volume":"11","author":"Zhou","year":"2021","journal-title":"Sci. Rep."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1176935118810215","DOI":"10.1177\/1176935118810215","article-title":"Machine Learning with K-Means Dimensional Reduction for Predicting Survival Outcomes in Patients with Breast Cancer","volume":"17","author":"Zhao","year":"2018","journal-title":"Cancer Inform."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ijmedinf.2017.09.013","article-title":"Prediction of Lung Cancer Patient Survival via Supervised Machine Learning Classification Techniques","volume":"108","author":"Lynch","year":"2017","journal-title":"Int. J. Med. Inform."},{"key":"ref_18","first-page":"977","article-title":"Machine Learning and Treatment Outcome Prediction for Oral Cancer","volume":"49","author":"Chu","year":"2020","journal-title":"J. Oral Pathol. Med. Off. Publ. Int. Assoc. Oral Pathol. Am. Acad. Oral Pathol."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Sarkar, S., Min, K., Ikram, W., Tatton, R.W., Riaz, I.B., Silva, A.C., Bryce, A.H., Moore, C., Ho, T.H., and Sonpavde, G. (2023). Performing Automatic Identification and Staging of Urothelial Carcinoma in Bladder Cancer Patients Using a Hybrid Deep-Machine Learning Approach. Cancers, 15.","DOI":"10.3390\/cancers15061673"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"O\u2019Donnell, A., Wolsztynski, E., Cronin, M., and Moghaddam, S. (2023). Improving the Post-Operative Prediction of BCR-Free Survival Time with MRNA Variables and Machine Learning. Cancers, 15.","DOI":"10.3390\/cancers15041276"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Umer, M., Naveed, M., Alrowais, F., Ishaq, A., Hejaili, A.A., Alsubai, S., Eshmawi, A., Mohamed, A., and Ashraf, I. (2022). Breast Cancer Detection Using Convoluted Features and Ensemble Machine Learning Algorithm. Cancers, 14.","DOI":"10.3390\/cancers14236015"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"104068","DOI":"10.1016\/j.ijmedinf.2019.104068","article-title":"Comparison of Supervised Machine Learning Classification Techniques in Prediction of Locoregional Recurrences in Early Oral Tongue Cancer","volume":"136","author":"Alabi","year":"2020","journal-title":"Int. J. Med. Inform."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1115","DOI":"10.1001\/jamaoto.2019.0981","article-title":"Development and Assessment of a Machine Learning Model to Help Predict Survival Among Patients with Oral Squamous Cell Carcinoma","volume":"145","author":"Karadaghy","year":"2019","journal-title":"JAMA Otolaryngol.\u2013Head Neck Surg."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"S41","DOI":"10.1007\/s12105-012-0373-4","article-title":"Morphologic Features of Conventional Squamous Cell Carcinoma of the Oropharynx: \u201ckeratinizing\u201d and \u201cNonkeratinizing\u201d Histologic Types as the Basis for a Consistent Classification System","volume":"6","author":"Chernock","year":"2012","journal-title":"Head Neck Pathol."},{"key":"ref_25","unstructured":"Gijsbers, P., LeDell, E., Thomas, J., Poirier, S., Bischl, B., and Vanschoren, J. (2019). An Open Source AutoML Benchmark. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Caruana, R., and Niculescu-Mizil, A. (2006, January 25\u201329). An Empirical Comparison of Supervised Learning Algorithms. Proceedings of the 23rd International Conference on Machine Learning, New York, NY, USA.","DOI":"10.1145\/1143844.1143865"},{"key":"ref_27","unstructured":"(2021, February 12). H2O.ai Generalized Linear Model (GLM)\u2013H2O Documentation. Available online: https:\/\/docs.h2o.ai\/h2o\/latest-stable\/h2o-docs\/data-science\/glm.html."},{"key":"ref_28","unstructured":"(2021, February 12). H2O.ai Distributed Random Forest (DRF)\u2013H2O Documentation. Available online: https:\/\/docs.h2o.ai\/h2o\/latest-stable\/h2o-docs\/data-science\/drf.html."},{"key":"ref_29","unstructured":"(2021, February 12). H2O.ai Deep Learning Neural Networks\u2013H2O Documentation. Available online: https:\/\/docs.h2o.ai\/h2o\/latest-stable\/h2o-docs\/data-science\/deep-learning.html%0A."},{"key":"ref_30","unstructured":"(2021, February 12). H2O.ai Gradient Boosting Machine (GBM)\u2013H2O Documentation. Available online: https:\/\/docs.h2o.ai\/h2o\/latest-stable\/h2o-docs\/data-science\/gbm.html."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1189","DOI":"10.1214\/aos\/1013203451","article-title":"Greedy Function Approximation: A Gradient Boosting Machine","volume":"29","author":"Friedman","year":"2001","journal-title":"Ann. Stat."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: Synthetic Minority over-Sampling Technique","volume":"16","author":"Chawla","year":"2002","journal-title":"J. Artif. Intell. Res."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Garcia-Pedrajas, N., Herrera, F., Fyfe, C., S\u00e1nchez, J.M.B., and Ali, M. (2010, January 1\u20134). Trends in Applied Intelligent Systems. Proceedings of the 23rd International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, IEA\/AIE 2010, Cordoba, Spain.","DOI":"10.1007\/978-3-642-13025-0"},{"key":"ref_34","unstructured":"Lundberg, S.M., and Lee, S.-I. (2017, January 4\u20139). A Unified Approach to Interpreting Model Predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"166","DOI":"10.3342\/ceo.2008.1.3.166","article-title":"Factors Related to Regional Recurrence in Early Stage Squamous Cell Carcinoma of the Oral Tongue","volume":"1","author":"An","year":"2008","journal-title":"Clin. Exp. Otorhinolaryngol."},{"key":"ref_36","first-page":"2559","article-title":"A Prospective Study of Prognostic Factors for Recurrence in Early Oral Tongue Cancer","volume":"7","author":"Sharma","year":"2013","journal-title":"J. Clin. Diagn. Res. JCDR"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1007\/s12105-018-0925-3","article-title":"Measuring Depth of Invasion in Early Squamous Cell Carcinoma of the Oral Tongue: Positive Deep Margin, Extratumoral Perineural Invasion, and Other Challenges","volume":"13","author":"Berdugo","year":"2019","journal-title":"Head Neck Pathol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1749","DOI":"10.1016\/j.jcms.2017.07.012","article-title":"Analysis of Clinicopathological Risk Factors for Locoregional Recurrence of Oral Squamous Cell Carcinoma\u2013Retrospective Analysis of 517 Patients","volume":"45","author":"Safi","year":"2017","journal-title":"J. Cranio-Maxillo-Facial Surg. Off. Publ. Eur. Assoc. Cranio-Maxillo-Facial Surg."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1724","DOI":"10.1016\/j.joms.2011.06.228","article-title":"Predictors for Tumor Recurrence after Primary Definitive Surgery for Oral Cancer","volume":"70","author":"Seoane","year":"2012","journal-title":"J. Oral Maxillofac. Surg. Off. J. Am. Assoc. Oral Maxillofac. Surg."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1016\/j.ijrobp.2013.09.034","article-title":"Genetic Susceptibility to Head and Neck Squamous Cell Carcinoma","volume":"89","author":"Lacko","year":"2014","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1001\/archotol.1995.01890020019005","article-title":"Role of Genetic Factors in the Etiology of Squamous Cell Carcinoma of the Head and Neck","volume":"121","author":"Copper","year":"1995","journal-title":"Arch. Otolaryngol.\u2013Head Neck Surg."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1007\/s00405-005-0947-9","article-title":"Influential Factors on Tumor Recurrence in Head and Neck Cancer Patients","volume":"263","author":"Matthias","year":"2006","journal-title":"Eur. Arch. Oto-Rhino-Laryngol. Head Neck"},{"key":"ref_43","first-page":"80","article-title":"Multiple Cancers of the Head and Neck","volume":"8","author":"Danciu","year":"2013","journal-title":"Maedica"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1186\/1758-3284-2-9","article-title":"Clinicopathological Parameters, Recurrence, Locoregional and Distant Metastasis in 115 T1-T2 Oral Squamous Cell Carcinoma Patients","volume":"2","author":"Jerjes","year":"2010","journal-title":"Head Neck Oncol."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"444","DOI":"10.1016\/j.joms.2017.06.034","article-title":"Degree of Keratinization Is an Independent Prognostic Factor in Oral Squamous Cell Carcinoma","volume":"76","author":"Wolfer","year":"2018","journal-title":"J. Oral Maxillofac. Surg."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"772","DOI":"10.1038\/modpathol.2017.183","article-title":"The Histologic Risk Model Is a Useful and Inexpensive Tool to Assess Risk of Recurrence and Death in Stage I or II Squamous Cell Carcinoma of Tongue and Floor of Mouth","volume":"31","author":"Sinha","year":"2018","journal-title":"Mod. Pathol."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1097\/01.pas.0000149687.90710.21","article-title":"Oral Squamous Cell Carcinoma: Histologic Risk Assessment, but Not Margin Status, Is Strongly Predictive of Local Disease-Free and Overall Survival","volume":"29","author":"Teixeira","year":"2005","journal-title":"Am. J. Surg. Pathol."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"616","DOI":"10.1007\/s12105-019-01082-6","article-title":"Validation of the Brandwein Gensler Risk Model in Patients of Oral Cavity Squamous Cell Carcinoma in North India","volume":"14","author":"Chaturvedi","year":"2020","journal-title":"Head Neck Pathol."},{"key":"ref_49","first-page":"e377","article-title":"Histopathologic Risk Factors in Oral and Oropharyngeal Squamous Cell Carcinoma Variants: An Update with Special Reference to HPV-Related Carcinomas","volume":"19","year":"2014","journal-title":"Med. Oral Patol. Oral Y Cir. Bucal"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"386","DOI":"10.1016\/S1368-8375(02)00142-2","article-title":"Tumour Thickness Predicts Cervical Nodal Metastases and Survival in Early Oral Tongue Cancer","volume":"39","author":"Pillai","year":"2003","journal-title":"Oral Oncol."},{"key":"ref_51","unstructured":"(2021, February 12). H2O.ai XGBoost\u2013H2O Documentation. Available online: https:\/\/docs.h2o.ai\/h2o\/latest-stable\/h2o-docs\/data-science\/xgboost.html."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Chen, T., and Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. arXiv.","DOI":"10.1145\/2939672.2939785"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1016\/j.patrec.2020.05.035","article-title":"Imbalance-XGBoost: Leveraging Weighted and Focal Losses for Binary Label-Imbalanced Classification with XGBoost","volume":"136","author":"Wang","year":"2020","journal-title":"Pattern Recognit. Lett."}],"container-title":["Cancers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-6694\/15\/10\/2769\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:35:43Z","timestamp":1760124943000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-6694\/15\/10\/2769"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,16]]},"references-count":53,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2023,5]]}},"alternative-id":["cancers15102769"],"URL":"https:\/\/doi.org\/10.3390\/cancers15102769","relation":{},"ISSN":["2072-6694"],"issn-type":[{"value":"2072-6694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,16]]}}}