{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,21]],"date-time":"2026-09-21T17:29:21Z","timestamp":1790011761460,"version":"4.0.1"},"reference-count":51,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2025,3,25]],"date-time":"2025-03-25T00:00:00Z","timestamp":1742860800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"U.S. Department of Defense","award":["W81XWH1910495"],"award-info":[{"award-number":["W81XWH1910495"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Cancers"],"abstract":"<jats:p>Background: A grid search, at the cost of training and testing a large number of models, is an effective way to optimize the prediction performance of deep learning models. A challenging task concerning grid search is time management. Without a good time management scheme, a grid search can easily be set off as a \u201cmission\u201d that will not finish in our lifetime. In this study, we introduce a heuristic three-stage mechanism for managing the running time of low-budget grid searches with deep learning, sweet-spot grid search (SSGS) and randomized grid search (RGS) strategies for improving model prediction performance, in an application of predicting the 5-year, 10-year, and 15-year risk of breast cancer metastasis. Methods: We develop deep feedforward neural network (DFNN) models and optimize the prediction performance of these models through grid searches. We conduct eight cycles of grid searches in three stages, focusing on learning a reasonable range of values for each of the adjustable hyperparameters in Stage 1, learning the sweet-spot values of the set of hyperparameters and estimating the unit grid search time in Stage 2, and conducting multiple cycles of timed grid searches to refine model prediction performance with SSGS and RGS in Stage 3. We conduct various SHAP analyses to explain the prediction, including a unique type of SHAP analyses to interpret the contributions of the DFNN-model hyperparameters. Results: The grid searches we conducted improved the risk prediction of 5-year, 10-year, and 15-year breast cancer metastasis by 18.6%, 16.3%, and 17.3%, respectively, over the average performance of all corresponding models we trained using the RGS strategy. Conclusions: Grid search can greatly improve model prediction. Our result analyses not only demonstrate best model performance but also characterize grid searches from various aspects such as their capabilities of discovering decent models and the unit grid search time. The three-stage mechanism worked effectively. It not only made our low-budget grid searches feasible and manageable but also helped improve the model prediction performance of the DFNN models. Our SHAP analyses not only identified clinical risk factors important for the prediction of future risk of breast cancer metastasis, but also DFNN-model hyperparameters important to the prediction of performance scores.<\/jats:p>","DOI":"10.3390\/cancers17071092","type":"journal-article","created":{"date-parts":[[2025,3,25]],"date-time":"2025-03-25T06:25:15Z","timestamp":1742883915000},"page":"1092","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Deep Learning: A Heuristic Three-Stage Mechanism for Grid Searches to Optimize the Future Risk Prediction of Breast Cancer Metastasis Using EHR-Based Clinical Data"],"prefix":"10.3390","volume":"17","author":[{"given":"Xia","family":"Jiang","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA 15206, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yijun","family":"Zhou","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA 15206, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chuhan","family":"Xu","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA 15206, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8080-7960","authenticated-orcid":false,"given":"Adam","family":"Brufsky","sequence":"additional","affiliation":[{"name":"Division of Hematology\/Oncology, University of Pittsburgh School of Medicine, Pittsburgh, PA 15213, USA"},{"name":"UPMC Hillman Cancer Center, Pittsburgh, PA 15232, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1637-8150","authenticated-orcid":false,"given":"Alan","family":"Wells","sequence":"additional","affiliation":[{"name":"UPMC Hillman Cancer Center, Pittsburgh, PA 15232, USA"},{"name":"Department of Pathology, University of Pittsburgh and Pittsburgh VA Health System, Pittsburgh, PA 15261, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,3,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"679","DOI":"10.1016\/j.cell.2006.11.001","article-title":"Cancer Metastasis: Building a Framework","volume":"127","author":"Gupta","year":"2006","journal-title":"Cell"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Jiang, X., Wells, A., Brufsky, A., and Neapolitan, R. (2019). A Clinical Decision Support System Learned from Data to Personalize Treatment Recommendations towards Preventing Breast Cancer Metastasis. PLoS ONE, 14.","DOI":"10.1371\/journal.pone.0213292"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1016\/j.neunet.2014.09.003","article-title":"Deep Learning in Neural Networks: An Overview","volume":"61","author":"Schmidhuber","year":"2015","journal-title":"Neural Netw."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Jiang, X., and Xu, C. (2022). Deep Learning and Machine Learning with Grid Search to Predict Later Occurrence of Breast Cancer Metastasis Using Clinical Data. J. Clin. Med., 11.","DOI":"10.3390\/jcm11195772"},{"key":"ref_5","unstructured":"Chen, L.C., Schwing, A.G., Yuille, A.L., and Urtasun, R. (2015, January 6\u201311). Learning Deep Structured Models. Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France."},{"key":"ref_6","unstructured":"Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., and Isard, M. (2016, January 2\u20134). TensorFlow: A System for Large-Scale Machine Learning. Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA."},{"key":"ref_7","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"351","DOI":"10.4258\/hir.2016.22.4.351","article-title":"Deep Learning Bible","volume":"22","author":"Goodfellow","year":"2016","journal-title":"Healthc. Inform. Res."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Cai, B., and Jiang, X. (2016). Computational Methods for Ubiquitination Site Prediction Using Physicochemical Properties of Protein Sequences. BMC Bioinform., 17.","DOI":"10.1186\/s12859-016-0959-z"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"e312","DOI":"10.1136\/amiajnl-2013-002358","article-title":"A Comparative Analysis of Methods for Predicting Clinical Outcomes Using High-Dimensional Genomic Datasets","volume":"21","author":"Jiang","year":"2014","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_11","unstructured":"Goodfellow, I., Bengio, Y., Courville, A., RegGoodfellow, I., Bengio, Y., and Courville, A. (2016). Regularization for Deep Learning. Deep Learning, MIT Press."},{"key":"ref_12","unstructured":"Brownlee, J. (2019). How to Choose Loss Functions When Training Deep Learning Neural Networks, Machine Learning Mastery."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1080\/00401706.1970.10488634","article-title":"Ridge Regression: Biased Estimation for Nonorthogonal Problems","volume":"12","author":"Hoerl","year":"1970","journal-title":"Technometrics"},{"key":"ref_14","unstructured":"Ghojogh, B., and Crowley, M. (2019). The Theory Behind Overfitting, Cross Validation, Regularization, Bagging, and Boosting: Tutorial. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"912","DOI":"10.1093\/bib\/bbv010","article-title":"Evaluation of a Two-Stage Framework for Prediction Using Big Genomic Data","volume":"16","author":"Jiang","year":"2015","journal-title":"Brief. Bioinform."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1016\/j.jbi.2013.12.009","article-title":"A Novel Artificial Neural Network Method for Biomedical Prediction Based on Matrix Pseudo-Inversion","volume":"48","author":"Cai","year":"2014","journal-title":"J. Biomed. Inform."},{"key":"ref_17","unstructured":"Glorot, X., and Bengio, Y. (2010, January 13\u201315). Understanding the Difficulty of Training Deep Feedforward Neural Networks. Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), Sardinia, Italy."},{"key":"ref_18","unstructured":"Kim, L.S. (1993, January 25\u201329). Understanding the Difficulty of Training Deep Feedforward Neural Networks Xavier. Proceedings of the International Joint Conference on Neural Networks, Nagoya, Japan."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Xu, C., Coen-Pirani, P., and Jiang, X. (2023). Empirical Study of Overfitting in Deep Learning for Predicting Breast Cancer Metastasis. Cancers, 15.","DOI":"10.3390\/cancers15071969"},{"key":"ref_20","first-page":"1742","article-title":"Grid Search of Multilayer Perceptron Based on the Walk-Forward Validation Methodology","volume":"11","author":"Ngoc","year":"2021","journal-title":"Int. J. Electr. Comput. Eng."},{"key":"ref_21","first-page":"173","article-title":"Grid Search Tuning of Hyperparameters in Random Forest Classifier for Customer Feedback Sentiment Prediction","volume":"11","author":"Sumathi","year":"2020","journal-title":"Int. J. Adv. Comput. Sci. Appl."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1007\/s10549-023-06966-4","article-title":"Breast cancer risk prediction combining a convolutional neural network-based mammographic evaluation with clinical factors","volume":"200","author":"Michel","year":"2023","journal-title":"Breast Cancer Res. Treat."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Hamedi, S.Z., Emami, H., Khayamzadeh, M., Rabiei, R., Aria, M., Akrami, M., and Zangouri, V. (2024). Application of machine learning in breast cancer survival prediction using a multimethod approach. Sci. Rep., 14.","DOI":"10.1038\/s41598-024-81734-y"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1148\/radiol.2019182622","article-title":"Predicting Breast Cancer by Applying Deep Learning to Linked Health Records and Mammograms","volume":"292","author":"Chorev","year":"2019","journal-title":"Radiology"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1007\/BF02478259","article-title":"A Logical Calculus of the Ideas Immanent in Nervous Activity","volume":"5","author":"McCulloch","year":"1943","journal-title":"Bull. Math. Biophys."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1109\/TIT.1954.1057468","article-title":"Simulation of Self-Organizing Systems by Digital Computer","volume":"4","author":"Farley","year":"1954","journal-title":"IRE Prof. Group Inf. Theory"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Neapolitan, R.E., and Jiang, X. (2018). Neural Networks and Deep Learning. Artificial Intelligence, Chapman and Hall\/CRC.","DOI":"10.1201\/b22400-15"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Ran, L., Zhang, Y., Zhang, Q., and Yang, T. (2017). Convolutional Neural Network-Based Robot Navigation Using Uncalibrated Spherical Images. Sensors, 17.","DOI":"10.3390\/s17061341"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Deng, L., Tur, G., He, X., and Hakkani-Tur, D. (2012, January 2\u20135). Use of Kernel Deep Convex Networks and End-to-End Learning for Spoken Language Understanding. Proceedings of the 2012 IEEE Workshop on Spoken Language Technology, SLT 2012\u2014Proceedings, Miami, FL, USA.","DOI":"10.1109\/SLT.2012.6424224"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Fern\u00e1ndez, S., Graves, A., and Schmidhuber, J. (2007). An Application of Recurrent Neural Networks to Discriminative Keyword Spotting. Artificial Neural Networks\u2014ICANN 2007, Springer.","DOI":"10.1007\/978-3-540-74695-9_23"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"5727","DOI":"10.1038\/s41467-020-19334-3","article-title":"Deep Learning-Enabled Breast Cancer Hormonal Receptor Status Determination from Base-Level H&E Stains","volume":"11","author":"Naik","year":"2020","journal-title":"Nat. Commun."},{"key":"ref_32","first-page":"851","article-title":"Deep Learning in Bioinformatics","volume":"18","author":"Min","year":"2017","journal-title":"Brief. Bioinform."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1016\/j.zemedi.2018.11.002","article-title":"An Overview of Deep Learning in Medical Imaging Focusing on MRI","volume":"29","author":"Lundervold","year":"2019","journal-title":"Z. Med. Phys."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Lee, Y.W., Huang, C.S., Shih, C.C., and Chang, R.F. (2021). Axillary Lymph Node Metastasis Status Prediction of Early-Stage Breast Cancer Using Convolutional Neural Networks. Comput. Biol. Med., 130.","DOI":"10.1016\/j.compbiomed.2020.104206"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Papandrianos, N., Papageorgiou, E., Anagnostis, A., and Feleki, A. (2020). A Deep-Learning Approach for Diagnosis of Metastatic Breast Cancer in Bones from Whole-Body Scans. Appl. Sci., 10.","DOI":"10.3390\/app10030997"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1148\/radiol.2019190372","article-title":"Lymph Node Metastasis Prediction from Primary Breast Cancer US Images Using Deep Learning","volume":"294","author":"Zhou","year":"2020","journal-title":"Radiology"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.media.2017.07.005","article-title":"A Survey on Deep Learning in Medical Image Analysis","volume":"42","author":"Litjens","year":"2017","journal-title":"Med. Image Anal."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Mohanty, S.P., Hughes, D.P., and Salath\u00e9, M. (2016). Using Deep Learning for Image-Based Plant Disease Detection. Front. Plant Sci., 7.","DOI":"10.3389\/fpls.2016.01419"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Shen, H. (2018, January 18\u201323). Towards a Mathematical Understanding of the Difficulty in Learning with Feedforward Neural Networks. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00091"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Jiang, X., Wells, A., Brufsky, A., Shetty, D., Shajihan, K., and Neapolitan, R.E. (2020). Leveraging Bayesian Networks and Information Theory to Learn Risk Factors for Breast Cancer Metastasis. BMC Bioinform., 21.","DOI":"10.1186\/s12859-020-03638-8"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"861","DOI":"10.1016\/j.patrec.2005.10.010","article-title":"An Introduction to ROC Analysis","volume":"27","author":"Fawcett","year":"2006","journal-title":"Pattern Recognit. Lett."},{"key":"ref_42","unstructured":"Shapley, L.S. (1951). Notes on the N-Person Game II: The Value of an n-Person Game, Rand Corporation. Rand Corporation Research Memoranda; RM-670."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Shapley, L.S. (1953). The Value of an N-Person Game. Contributions to the Theory of Games (AM-28), Volume II. Classics in Game Theory, Princeton University Press.","DOI":"10.1515\/9781400881970-018"},{"key":"ref_44","unstructured":"Lundberg, S.M., and Lee, S.I. (2017, January 4\u20139). A Unified Approach to Interpreting Model Predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 12\u201317). \u201cWhy Should I Trust You?\u201d Explaining the Predictions of Any Classifier. Proceedings of the NAACL-HLT 2016\u20142016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Demonstrations Session, San Diego, CA, USA.","DOI":"10.18653\/v1\/N16-3020"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"800","DOI":"10.1038\/bjc.2012.338","article-title":"PREDICT Plus: Development and validation of a prognostic model for early breast cancer that includes HER2","volume":"107","author":"Wishart","year":"2012","journal-title":"Br. J. Cancer"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1007\/s10549-022-06682-5","article-title":"A prognostic survival model for women diagnosed with invasive breast cancer in Queensland, Australia","volume":"195","author":"Baade","year":"2022","journal-title":"Breast Cancer Res. Treat."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1007\/s10549-007-9556-1","article-title":"An overview of prognostic factors for long-term survivors of breast cancer","volume":"107","author":"Soerjomataram","year":"2008","journal-title":"Breast Cancer Res. Treat."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"1836","DOI":"10.1056\/NEJMoa1701830","article-title":"20-year risks of breast-cancer recurrence after stopping endocrine therapy at 5 years","volume":"377","author":"Pan","year":"2017","journal-title":"N. Engl. J. Med."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1016\/j.breast.2022.09.001","article-title":"Treatment and survival outcomes in older women with primary breast cancer: A retrospective propensity score-matched analysis","volume":"66","author":"Sang","year":"2022","journal-title":"Breast"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","article-title":"Regression Shrinkage and Selection Via the Lasso","volume":"58","author":"Tibshirani","year":"1996","journal-title":"J. R. Stat. Soc. Ser. B (Methodol.)"}],"container-title":["Cancers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-6694\/17\/7\/1092\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T16:59:44Z","timestamp":1760029184000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-6694\/17\/7\/1092"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,25]]},"references-count":51,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2025,4]]}},"alternative-id":["cancers17071092"],"URL":"https:\/\/doi.org\/10.3390\/cancers17071092","relation":{},"ISSN":["2072-6694"],"issn-type":[{"value":"2072-6694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,25]]}}}