{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,3]],"date-time":"2026-08-03T15:22:41Z","timestamp":1785770561628,"version":"3.56.0"},"reference-count":30,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2023,1,12]],"date-time":"2023-01-12T00:00:00Z","timestamp":1673481600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,1,12]],"date-time":"2023-01-12T00:00:00Z","timestamp":1673481600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["UL1TR002537"],"award-info":[{"award-number":["UL1TR002537"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R00HL116769"],"award-info":[{"award-number":["R00HL116769"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R21EB026518"],"award-info":[{"award-number":["R21EB026518"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01HD08994"],"award-info":[{"award-number":["R01HD08994"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["P50HD105556"],"award-info":[{"award-number":["P50HD105556"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["SN COMPUT. SCI."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Early stopping is an extremely common tool to minimize overfitting, which would otherwise be a cause of poor generalization of the model to novel data. However, early stopping is a heuristic that, while effective, primarily relies on ad hoc parameters and metrics. Optimizing when to stop remains a challenge. In this paper, we suggest that for some biomedical applications, a natural dichotomy of invasive\/non-invasive measurements, or more generally proximal vs distal measurements of a biological system can be exploited to provide objective advice on early stopping. We discuss the conditions where invasive measurements of a biological process should provide better predictions than non-invasive measurements, or at best offer parity. Hence, if data from an invasive measurement are available locally, or from the literature, that information can be leveraged to know with high certainty whether a model of non-invasive data is overfitted. We present paired invasive\/non-invasive cardiac and coronary artery measurements from two mouse strains, one of which spontaneously develops type 2 diabetes, posed as a classification problem. Examination of the various stopping rules shows that generalization is reduced with more training epochs and commonly applied stopping rules give widely different generalization error estimates. The use of an empirically derived training ceiling is demonstrated to be helpful as added information to leverage early stopping in order to reduce overfitting.<\/jats:p>","DOI":"10.1007\/s42979-022-01553-8","type":"journal-article","created":{"date-parts":[[2023,1,12]],"date-time":"2023-01-12T14:02:58Z","timestamp":1673532178000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Invasive or More Direct Measurements Can Provide an Objective Early-Stopping Ceiling for Training Deep Neural Networks on Non-invasive or Less-Direct Biomedical Data"],"prefix":"10.1007","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7837-6348","authenticated-orcid":false,"given":"Christopher W.","family":"Bartlett","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jamie","family":"Bossenbroek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yukie","family":"Ueyama","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patricia","family":"McCallinhart","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Olivia A.","family":"Peters","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Donna A.","family":"Santillan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark K.","family":"Santillan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aaron J.","family":"Trask","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"William C.","family":"Ray","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,1,12]]},"reference":[{"key":"1553_CR1","doi-asserted-by":"crossref","unstructured":"Bartlett C, Bossenbroek J, Ueyama Y, Mccallinhart P, Trask A, Ray WC. Submit invasive measurements can provide an objective ceiling for non-invasive machine learning predictions. In: Proceedings of the 18th International Conference on Signal Processing and Multimedia Applications - SIGMAP. 2021. p. 73\u201380.","DOI":"10.5220\/0010582000730080"},{"issue":"4","key":"1553_CR2","doi-asserted-by":"publisher","first-page":"761","DOI":"10.1016\/S0893-6080(98)00010-0","volume":"11","author":"L Prechelt","year":"1998","unstructured":"Prechelt L. Automatic early stopping using cross validation: quantifying the criteria. Neural Netw. 1998;11(4):761\u20137.","journal-title":"Neural Netw."},{"key":"1553_CR3","unstructured":"Deng J-Q, Kwok Y-K. Large vocabulary automatic chord estimation with an even chance training scheme. In: ISMIR, 2017;p. 531\u201336."},{"key":"1553_CR4","unstructured":"Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean, J, Devin M, Ghemawat S, Irving G, Isard M, et al. Tensorflow: a system for large-scale machine learning. In: 12th USENIX Symposium on Operating and Systems Design and Implementation (OSDI 16),2016; p. 265\u201383"},{"issue":"2","key":"1553_CR5","volume":"1168","author":"X Ying","year":"2019","unstructured":"Ying X. An overview of overfitting and its solutions. J Phys. 2019;1168(2): 022022.","journal-title":"J Phys"},{"key":"1553_CR6","volume-title":"Deep learning","author":"I Goodfellow","year":"2016","unstructured":"Goodfellow I, Bengio Y, Courville A. Deep learning. MIT Press; 2016."},{"key":"1553_CR7","doi-asserted-by":"crossref","unstructured":"Prakash S.S, Visakha K. Breast cancer malignancy prediction using deep learning neural networks. In: 2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA), 2020;p. 88\u201392.","DOI":"10.1109\/ICIRCA48905.2020.9183378"},{"key":"1553_CR8","doi-asserted-by":"crossref","unstructured":"Liu K, Song J, Zhang W, Yang X. Alleviating over-fitting in attribute reduction: an early stopping strategy. In: 2018 International Conference on Wavelet Analysis and Pattern Recognition (ICWAPR), 2018; pp. 190\u20135.","DOI":"10.1109\/ICWAPR.2018.8521316"},{"key":"1553_CR9","doi-asserted-by":"crossref","unstructured":"Wu X, Liu J. A new early stopping algorithm for improving neural network generalization. In: 2009 Second international conference on intelligent computation technology and automation, vol. 1,2009; p. 15\u20138.","DOI":"10.1109\/ICICTA.2009.11"},{"key":"1553_CR10","doi-asserted-by":"publisher","unstructured":"Blanchard N, Kinnison J, RichardWebster B, Bashivan P, Scheirer WJ. A neurobiological evaluation metric for neural network model search. 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2019. https:\/\/doi.org\/10.1109\/cvpr.2019.00555.","DOI":"10.1109\/cvpr.2019.00555"},{"issue":"1","key":"1553_CR11","doi-asserted-by":"publisher","first-page":"17268","DOI":"10.1038\/s41598-018-35572-4","volume":"8","author":"IL Sunyecz","year":"2018","unstructured":"Sunyecz IL, McCallinhart PE, Patel KU, McDermott MR, Trask AJ. Defining coronary flow patterns: comprehensive automation of transthoracic doppler coronary blood flow. Sci Rep. 2018;8(1):17268.","journal-title":"Sci Rep"},{"issue":"11","key":"1553_CR12","doi-asserted-by":"publisher","first-page":"2760","DOI":"10.1007\/s10439-015-1333-4","volume":"43","author":"M Anghelescu","year":"2015","unstructured":"Anghelescu M, Tonniges JR, Calomeni E, Shamhart PE, Agarwal G, Gooch KJ, Trask AJ. Vascular mechanics in decellularized aortas and coronary resistance microvessels in type 2 diabetic db\/db mice. Ann Biomed Eng. 2015;43(11):2760\u201370.","journal-title":"Ann Biomed Eng"},{"key":"1553_CR13","doi-asserted-by":"crossref","unstructured":"Gooch KJ, Trask AJ. Tissue-specific vascular remodeling and stiffness associated with metabolic diseases. 2015.","DOI":"10.1152\/ajpheart.00545.2015"},{"issue":"6","key":"1553_CR14","doi-asserted-by":"publisher","first-page":"1123","DOI":"10.1007\/s00395-011-0201-0","volume":"106","author":"PS Katz","year":"2011","unstructured":"Katz PS, Trask AJ, Souza-Smith FM, Hutchinson KR, Galantowicz ML, Lord KC, Stewart J, James A, Cismowski MJ, Varner KJ, Lucchesi PA. Coronary arterioles in type 2 diabetic (db\/db) mice undergo a distinct pattern of remodeling associated with decreased vessel stiffness. Basic Res Cardiol. 2011;106(6):1123\u201334.","journal-title":"Basic Res Cardiol."},{"key":"1553_CR15","doi-asserted-by":"publisher","first-page":"114","DOI":"10.1016\/j.phrs.2017.07.004","volume":"123","author":"H Labazi","year":"2017","unstructured":"Labazi H, Trask AJ. Coronary microvascular disease as an early culprit in the pathophysiology of diabetes and metabolic syndrome. Pharmacol Res. 2017;123:114\u201321.","journal-title":"Pharmacol Res."},{"issue":"2","key":"1553_CR16","first-page":"306","volume":"301","author":"S Lee","year":"2011","unstructured":"Lee S, Park Y, Dellsperger KC, Zhang C. Exercise training improves endothelial function via adiponectin-dependent and independent pathways in type 2 diabetic mice. Am J Physiol. 2011;301(2):306\u201314.","journal-title":"Am J Physiol"},{"issue":"4","key":"1553_CR17","doi-asserted-by":"publisher","first-page":"241","DOI":"10.5099\/aj110400241","volume":"3","author":"S Lee","year":"2011","unstructured":"Lee S, Park Y, Zhang C. Exercise training prevents coronary endothelial dysfunction in type 2 diabetic mice. Am J Biomed Sci. 2011;3(4):241\u201352.","journal-title":"Am J Biomed Sci"},{"issue":"5","key":"1553_CR18","first-page":"1982","volume":"295","author":"Y Park","year":"2008","unstructured":"Park Y, Capobianco S, Gao X, Falck JR, Dellsperger KC, Zhang C. Role of edhf in type 2 diabetes-induced endothelial dysfunction. Am J Physiol. 2008;295(5):1982\u20138.","journal-title":"Am J Physiol"},{"issue":"1","key":"1553_CR19","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1007\/s00395-010-0129-9","volume":"106","author":"Y Park","year":"2011","unstructured":"Park Y, Yang J, Zhang H, Chen X, Zhang C. Effect of par2 in regulating tnf-\u03b1 and nad(p)h oxidase in coronary arterioles in type 2 diabetic mice. Basic Res Cardiol. 2011;106(1):111\u201323.","journal-title":"Basic Res Cardiol"},{"issue":"5\u20136","key":"1553_CR20","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1016\/j.vph.2012.07.007","volume":"57","author":"AJ Trask","year":"2012","unstructured":"Trask AJ, Delbin MA, Katz PS, Zanesco A, Lucchesi PA. Differential coronary resistance microvessel remodeling between type 1 and type 2 diabetic mice: impact of exercise training. Vascul Pharmacol. 2012;57(5\u20136):187\u201393.","journal-title":"Vascul Pharmacol"},{"issue":"7","key":"1553_CR21","doi-asserted-by":"publisher","first-page":"1128","DOI":"10.1152\/japplphysiol.00604.2012","volume":"113","author":"AJ Trask","year":"2012","unstructured":"Trask AJ, Katz PS, Kelly AP, Galantowicz ML, Cismowski MJ, West TA, Neeb ZP, Berwick ZC, Goodwill AG, Alloosh M, et al. Dynamic micro- and macro-vascular remodeling in coronary circulation of obese ossabaw pigs with metabolic syndrome. J Appl Physiol. 2012;113(7):1128\u201340.","journal-title":"J Appl Physiol"},{"issue":"1","key":"1553_CR22","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1080\/14767050500127674","volume":"18","author":"ME Kleijer","year":"2005","unstructured":"Kleijer ME, Dekker GA, Heard AR. Risk factors for intrauterine growth restriction in a socio-economically disadvantaged region. J Maternal-Fetal Neonatal Med. 2005;18(1):23\u201330.","journal-title":"J Maternal-Fetal Neonatal Med"},{"issue":"4","key":"1553_CR23","doi-asserted-by":"publisher","first-page":"409-e1","DOI":"10.1016\/j.ajog.2008.11.025","volume":"200","author":"SP Chauhan","year":"2009","unstructured":"Chauhan SP, Gupta LM, Hendrix NW, Berghella V. Intrauterine growth restriction: comparison of American college of obstetricians and gynecologists practice bulletin with other national guidelines. Am J Obstet Gynecol. 2009;200(4):409-e1.","journal-title":"Am J Obstet Gynecol"},{"issue":"4","key":"1553_CR24","doi-asserted-by":"publisher","first-page":"300","DOI":"10.1016\/j.ajog.2012.01.022","volume":"206","author":"E Berkley","year":"2012","unstructured":"Berkley E, Chauhan SP, Abuhamad A, for Maternal-Fetal Medicine Publications\u00a0Committee, S, et al. Doppler assessment of the fetus with intrauterine growth restriction. Am J Obstest Gynecol. 2012;206(4):300\u20138.","journal-title":"Am J Obstest Gynecol"},{"issue":"17","key":"1553_CR25","doi-asserted-by":"publisher","first-page":"2996","DOI":"10.1080\/14767058.2019.1566900","volume":"33","author":"J Morales-Rosell\u00f3","year":"2020","unstructured":"Morales-Rosell\u00f3 J, Galindo A, Herraiz I, Gil MM, Brik M, De Paco-Matallana C, Ciammela R, Sanchez Ajenjo C, Ca\u00f1ada Martinez AJ, Delgado JL, et al. Is it possible to predict late antepartum stillbirth by means of cerebroplacental ratio and maternal characteristics? J Maternal-Fetal Neonatal Med. 2020;33(17):2996\u20133002.","journal-title":"J Maternal-Fetal Neonatal Med"},{"key":"1553_CR26","doi-asserted-by":"crossref","unstructured":"Blue NR, Allshouse AA, Grobman WA, Day RC, Haas DM, Simhan HN, Parry S, Saade GR, Silver RM. Developing a predictive model for perinatal morbidity among small for gestational age infants. J Maternal-Fetal Neonatal Med. 2021;1\u201310.","DOI":"10.1080\/14767058.2021.1980533"},{"key":"1553_CR27","doi-asserted-by":"publisher","first-page":"28","DOI":"10.1016\/j.vph.2015.06.013","volume":"76","author":"KE Husarek","year":"2016","unstructured":"Husarek KE, Katz PS, Trask AJ, Galantowicz ML, Cismowski MJ, Lucchesi PA. The angiotensin receptor blocker losartan reduces coronary arteriole remodeling in type 2 diabetic mice. Vascul Pharmacol. 2016;76:28\u201336.","journal-title":"Vascul Pharmacol"},{"issue":"6","key":"1553_CR28","doi-asserted-by":"publisher","first-page":"687","DOI":"10.1038\/ajh.2010.51","volume":"23","author":"AJ Trask","year":"2010","unstructured":"Trask AJ, Groban L, Westwood BM, Varagic J, Ganten D, Gallagher PE, Chappell MC, Ferrario CM. Inhibition of angiotensin-converting enzyme 2 exacerbates cardiac hypertrophy and fibrosis in ren-2 hypertensive rats. Am J Hypertens. 2010;23(6):687\u201393.","journal-title":"Am J Hypertens"},{"key":"1553_CR29","unstructured":"Pedregosa F, Varoquaux G, Gramfort A. Scikit-learn: machine learning in python. of machine Learning... 2011."},{"key":"1553_CR30","unstructured":"Kingma DP, Ba J. Adam: a method for stochastic optimization. arXiv 2014."}],"container-title":["SN Computer Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-022-01553-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s42979-022-01553-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-022-01553-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,12]],"date-time":"2023-01-12T14:36:32Z","timestamp":1673534192000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s42979-022-01553-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,12]]},"references-count":30,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,3]]}},"alternative-id":["1553"],"URL":"https:\/\/doi.org\/10.1007\/s42979-022-01553-8","relation":{},"ISSN":["2661-8907"],"issn-type":[{"value":"2661-8907","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,12]]},"assertion":[{"value":"19 February 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 December 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 January 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that there is no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}},{"value":"Animal work was approved by the Nationwide Children\u2019s Hospital Institutional Animal Care and Use Committee (protocol number: AR18-0001). For analysis of the de-identified placenta data, the Nationwide Children\u2019s Hospital Institutional Review Board reviewed the project and determined that it did not require further review (STUDY00000950). For the University of Iowa, a human subjects research determination was completed for this study and the IRB determined that our use of coded samples and data are not human subjects research (IRB202201716).","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical Approval"}}],"article-number":"161"}}