{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T20:11:40Z","timestamp":1775074300010,"version":"3.50.1"},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,12,1]],"date-time":"2021-12-01T00:00:00Z","timestamp":1638316800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,12,18]],"date-time":"2021-12-18T00:00:00Z","timestamp":1639785600000},"content-version":"vor","delay-in-days":17,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Res Methodol"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Background<\/jats:title><jats:p>While machine learning (ML) algorithms may predict cardiovascular outcomes more accurately than statistical models, their result is usually not representable by a transparent formula. Hence, it is often unclear how specific values of predictors lead to the predictions. We aimed to demonstrate with graphical tools how predictor-risk relations in cardiovascular risk prediction models fitted by ML algorithms and by statistical approaches may differ, and how sample size affects the stability of the estimated relations.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>We reanalyzed data from a large registry of 1.5 million participants in a national health screening program. Three data analysts developed analytical strategies to predict cardiovascular events within 1 year from health screening. This was done for the full data set and with gradually reduced sample sizes, and each data analyst followed their favorite modeling approach. Predictor-risk relations were visualized by partial dependence and individual conditional expectation plots.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>When comparing the modeling algorithms, we found some similarities between these visualizations but also occasional divergence. The smaller the sample size, the more the predictor-risk relation depended on the modeling algorithm used, and also sampling variability played an increased role. Predictive performance was similar if the models were derived on the full data set, whereas smaller sample sizes favored simpler models.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>Predictor-risk relations from ML models may differ from those obtained by statistical models, even with large sample sizes. Hence, predictors may assume different roles in risk prediction models. As long as sample size is sufficient, predictive accuracy is not largely affected by the choice of algorithm.<\/jats:p><\/jats:sec>","DOI":"10.1186\/s12874-021-01487-4","type":"journal-article","created":{"date-parts":[[2021,12,18]],"date-time":"2021-12-18T08:02:16Z","timestamp":1639814536000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["The roles of predictors in cardiovascular risk models - a question of modeling culture?"],"prefix":"10.1186","volume":"21","author":[{"given":"Christine","family":"Wallisch","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Asan","family":"Agibetov","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daniela","family":"Dunkler","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Maria","family":"Haller","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matthias","family":"Samwald","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Georg","family":"Dorffner","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Georg","family":"Heinze","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,12,18]]},"reference":[{"issue":"6","key":"1487_CR1","doi-asserted-by":"publisher","first-page":"743","DOI":"10.1161\/CIRCULATIONAHA.107.699579","volume":"117","author":"RB D'Agostino Sr","year":"2008","unstructured":"D'Agostino RB Sr, Vasan RS, Pencina MJ, Wolf PA, Cobain M, Massaro JM, et al. General cardiovascular risk profile for use in primary care: the Framingham heart study. Circulation. 2008;117(6):743\u201353.","journal-title":"Circulation."},{"issue":"1","key":"1487_CR2","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1186\/s12874-019-0666-3","volume":"19","author":"A Perperoglou","year":"2019","unstructured":"Perperoglou A, Sauerbrei W, Abrahamowicz M, Schmid M. A review of spline function procedures in R. BMC Med Res Methodol. 2019;19(1):46.","journal-title":"BMC Med Res Methodol"},{"issue":"1","key":"1487_CR3","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1186\/s41512-020-00074-3","volume":"4","author":"W Sauerbrei","year":"2020","unstructured":"Sauerbrei W, Perperoglou A, Schmid M, Abrahamowicz M, Becher H, Binder H, et al. State of the art in selection of variables and functional forms in multivariable analysis\u2014outstanding issues. Diagn Progn Res. 2020;4(1):3.","journal-title":"Diagn Progn Res"},{"issue":"30","key":"1487_CR4","doi-asserted-by":"publisher","first-page":"5512","DOI":"10.1002\/sim.3148","volume":"26","author":"W Sauerbrei","year":"2007","unstructured":"Sauerbrei W, Royston P, Binder H. Selection of important variables and determination of functional form for continuous predictors in multivariable model building. Stat Med. 2007;26(30):5512\u201328.","journal-title":"Stat Med"},{"key":"1487_CR5","volume-title":"Pattern recognition and machine learning","author":"CM Bishop","year":"2016","unstructured":"Bishop CM. Pattern recognition and machine learning. New York: Springer; 2016."},{"key":"1487_CR6","doi-asserted-by":"publisher","first-page":"785","DOI":"10.1145\/2939672.2939785","volume-title":"Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining","author":"T Chen","year":"2016","unstructured":"Chen T, Guestrin C. XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. San Francisco: Association for Computing Machinery; 2016. p. 785\u201394."},{"key":"1487_CR7","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-28954-6","volume-title":"Explainable AI: interpreting, explaining and visualizing deep learning","author":"W Samek","year":"2019","unstructured":"Samek W, Montavon G, Vedaldi A, Hansen LK, M\u00fcller KR. Explainable AI: interpreting, explaining and visualizing deep learning: Springer International Publishing; 2019."},{"issue":"4","key":"1487_CR8","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0231166","volume":"15","author":"E Zihni","year":"2020","unstructured":"Zihni E, Madai VI, Livne M, Galinovic I, Khalil AA, Fiebach JB, et al. Opening the black box of artificial intelligence for clinical decision support: a study predicting stroke outcome. PLoS One. 2020;15(4):e0231166.","journal-title":"PLoS One"},{"issue":"1","key":"1487_CR9","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1080\/10618600.2014.907095","volume":"24","author":"A Goldstein","year":"2015","unstructured":"Goldstein A, Kapelner A, Bleich J, Pitkin E. Peeking inside the black box: visualizing statistical learning with plots of individual conditional expectation. J Comput Graph Stat. 2015;24(1):44\u201365.","journal-title":"J Comput Graph Stat"},{"key":"1487_CR10","doi-asserted-by":"crossref","unstructured":"Zhao QY, Hastie T. Causal interpretations of black-box models. J Bus Econ Stat. 2021;39(1):272\u201381.","DOI":"10.1080\/07350015.2019.1624293"},{"issue":"3","key":"1487_CR11","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1214\/ss\/1009213726","volume":"16","author":"L Breiman","year":"2001","unstructured":"Breiman L. Statistical modeling: the two cultures. Stat Sci. 2001;16(3):199\u2013215.","journal-title":"Stat Sci"},{"key":"1487_CR12","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1016\/j.jclinepi.2020.03.002","volume":"122","author":"S Nusinovici","year":"2020","unstructured":"Nusinovici S, Tham YC, Chak Yan MY, Wei Ting DS, Li J, Sabanayagam C, et al. Logistic regression was as good as machine learning for predicting major chronic diseases. J Clin Epidemiol. 2020;122:56\u201369.","journal-title":"J Clin Epidemiol"},{"key":"1487_CR13","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1016\/j.jclinepi.2019.02.004","volume":"110","author":"E Christodoulou","year":"2019","unstructured":"Christodoulou E, Ma J, Collins GS, Steyerberg EW, Verbakel JY, Van Calster B. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol. 2019;110:12\u201322.","journal-title":"J Clin Epidemiol"},{"key":"1487_CR14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12916-014-0241-z","volume":"13","author":"GS Collins","year":"2015","unstructured":"Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMC Med. 2015;13:1.","journal-title":"BMC Med"},{"key":"1487_CR15","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1016\/j.ijcard.2018.11.001","volume":"283","author":"C Wallisch","year":"2019","unstructured":"Wallisch C, Heinze G, Rinner C, Mundigler G, Winkelmayer WC, Dunkler D. External validation of two Framingham cardiovascular risk equations and the pooled cohort equations: a nationwide registry analysis. Int J Cardiol. 2019;283:165\u201370.","journal-title":"Int J Cardiol"},{"issue":"1","key":"1487_CR16","doi-asserted-by":"publisher","first-page":"8140","DOI":"10.1038\/s41598-020-64629-6","volume":"10","author":"C Wallisch","year":"2020","unstructured":"Wallisch C, Heinze G, Rinner C, Mundigler G, Winkelmayer WC, Dunkler D. Re-estimation improved the performance of two Framingham cardiovascular risk equations and the pooled cohort equations: a nationwide registry analysis. Sci Rep. 2020;10(1):8140.","journal-title":"Sci Rep"},{"key":"1487_CR17","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-19425-7","volume-title":"Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis","author":"F Harrell","year":"2015","unstructured":"Harrell F. Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. New York, Berlin, Heidelberg: Springer; 2015."},{"key":"1487_CR18","volume-title":"Generalized additive models","author":"TJ Hastie","year":"1990","unstructured":"Hastie TJ, Tibshirani RJ. Generalized additive models. Boca Raton: Chapman & Hall\/CRC Press; 1990."},{"key":"1487_CR19","doi-asserted-by":"crossref","DOI":"10.1002\/9780470770771","volume-title":"Multivariable model-building. A pragmatic approach to regression analysis based on fractional polynomials for modelling continuous variables","author":"P Royston","year":"2008","unstructured":"Royston P, Sauerbrei W. In: Shewhart WA, Wilks SS, editors. Multivariable model-building. A pragmatic approach to regression analysis based on fractional polynomials for modelling continuous variables. Chichester: Wiley; 2008."},{"issue":"23","key":"1487_CR20","first-page":"1805","volume":"38","author":"BA Goldstein","year":"2017","unstructured":"Goldstein BA, Navar AM, Carter RE. Moving beyond regression techniques in cardiovascular risk prediction: applying machine learning to address analytic challenges. Eur Heart J. 2017;38(23):1805\u201314.","journal-title":"Eur Heart J"},{"issue":"3","key":"1487_CR21","doi-asserted-by":"publisher","first-page":"431","DOI":"10.1002\/bimj.201700067","volume":"60","author":"G Heinze","year":"2018","unstructured":"Heinze G, Wallisch C, Dunkler D. Variable selection - a review and recommendations for the practicing statistician. Biom J. 2018;60(3):431\u201349.","journal-title":"Biom J"},{"key":"1487_CR22","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-84858-7","volume-title":"The elements of statistical learning: data mining, inference, and prediction","author":"T Hastie","year":"2009","unstructured":"Hastie T, Tibshirani R, Friedman J. The elements of statistical learning: data mining, inference, and prediction. New York: Springer; 2009."},{"issue":"5","key":"1487_CR23","doi-asserted-by":"publisher","first-page":"1189","DOI":"10.1214\/aos\/1013203451","volume":"29","author":"JH Friedman","year":"2001","unstructured":"Friedman JH. Greedy function approximation: a gradient boosting machine. Ann Stat. 2001;29(5):1189\u2013232.","journal-title":"Ann Stat"},{"issue":"1","key":"1487_CR24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1175\/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2","volume":"78","author":"GW Brier","year":"1950","unstructured":"Brier GW. Verification of forecasts expressed in terms of probability. Mon Weather Rev. 1950;78(1):1\u20133.","journal-title":"Mon Weather Rev"},{"issue":"4","key":"1487_CR25","doi-asserted-by":"publisher","first-page":"366","DOI":"10.1198\/tast.2009.08210","volume":"63","author":"T Tjur","year":"2009","unstructured":"Tjur T. Coefficients of determination in logistic regression models -a new proposal: the coefficient of discrimination. Am Stat. 2009;63(4):366\u201372.","journal-title":"Am Stat"},{"key":"1487_CR26","doi-asserted-by":"crossref","unstructured":"Riley RD, Ensor J, Snell KIE, Harrell FE, Martin GP, Reitsma JB, et al. Calculating the sample size required for developing a clinical prediction model. Bmj. 2020;368:m441.","DOI":"10.1136\/bmj.m441"},{"issue":"14","key":"1487_CR27","doi-asserted-by":"publisher","first-page":"1156","DOI":"10.1136\/heartjnl-2017-311198","volume":"104","author":"K Shameer","year":"2018","unstructured":"Shameer K, Johnson KW, Glicksberg BS, Dudley JT, Sengupta PP. Machine learning in cardiovascular medicine: are we there yet? Heart. 2018;104(14):1156\u201364.","journal-title":"Heart."},{"issue":"5","key":"1487_CR28","doi-asserted-by":"publisher","first-page":"1015","DOI":"10.1016\/j.mayocp.2020.01.038","volume":"95","author":"F Lopez-Jimenez","year":"2020","unstructured":"Lopez-Jimenez F, Attia Z, Arruda-Olson AM, Carter R, Chareonthaitawee P, Jouni H, et al. Artificial intelligence in cardiology: present and future. Mayo Clin Proc. 2020;95(5):1015\u201339.","journal-title":"Mayo Clin Proc"},{"issue":"4","key":"1487_CR29","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0174944","volume":"12","author":"SF Weng","year":"2017","unstructured":"Weng SF, Reps J, Kai J, Garibaldi JM, Qureshi N. Can machine-learning improve cardiovascular risk prediction using routine clinical data? PLoS One. 2017;12(4):e0174944.","journal-title":"PLoS One"},{"issue":"9","key":"1487_CR30","doi-asserted-by":"publisher","first-page":"1092","DOI":"10.1161\/CIRCRESAHA.117.311312","volume":"121","author":"B Ambale-Venkatesh","year":"2017","unstructured":"Ambale-Venkatesh B, Yang X, Wu CO, Liu K, Hundley WG, McClelland R, et al. Cardiovascular event prediction by machine learning. Circ Res. 2017;121(9):1092\u2013101.","journal-title":"Circ Res"},{"issue":"5","key":"1487_CR31","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0213653","volume":"14","author":"AM Alaa","year":"2019","unstructured":"Alaa AM, Bolton T, Di Angelantonio E, Rudd JHF, van der Schaar M. Cardiovascular disease risk prediction using automated machine learning: a prospective study of 423,604 UK biobank participants. PLoS One. 2019;14(5):e0213653.","journal-title":"PLoS One"},{"issue":"1","key":"1487_CR32","doi-asserted-by":"publisher","first-page":"230","DOI":"10.1186\/s12916-019-1466-7","volume":"17","author":"B Van Calster","year":"2019","unstructured":"Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW, Topic Group \u2018Evaluating diagnostic tests prediction models\u2019 of the Stratos initiative. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17(1):230.","journal-title":"BMC Med"},{"key":"1487_CR33","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1186\/1471-2288-14-137","volume":"14","author":"T van der Ploeg","year":"2014","unstructured":"van der Ploeg T, Austin PC, Steyerberg EW. Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints. BMC Med Res Methodol. 2014;14:137.","journal-title":"BMC Med Res Methodol"},{"issue":"6","key":"1487_CR34","first-page":"618","volume":"9","author":"RC Deo","year":"2016","unstructured":"Deo RC, Nallamothu BK. Learning about machine learning: the promise and pitfalls of big data and the electronic health record. Circ: Cardiovasc Qual Outcomes. 2016;9(6):618\u201320.","journal-title":"Circ: Cardiovasc Qual Outcomes"},{"issue":"8","key":"1487_CR35","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1007\/s11936-020-00814-0","volume":"22","author":"DE Schlesinger","year":"2020","unstructured":"Schlesinger DE, Stultz CM. Deep learning for cardiovascular risk stratification. Curr Treat Options Cardiovasc Med. 2020;22(8):15.","journal-title":"Curr Treat Options Cardiovasc Med"},{"key":"1487_CR36","unstructured":"Molnar C. Interpretable machine learning. A guide for making black box models explainable.\u00a02020. https:\/\/christophm.github.io\/interpretable-ml-book\/. Accessed 10 Nov 2021."},{"key":"1487_CR37","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.m3919","volume":"371","author":"Y Li","year":"2020","unstructured":"Li Y, Sperrin M, Ashcroft DM, van Staa TP. Consistency of variety of machine learning and statistical models in predicting clinical risks of individual patients: longitudinal cohort study using cardiovascular disease as exemplar. BMJ. 2020;371:m3919.","journal-title":"BMJ."},{"issue":"7","key":"1487_CR38","doi-asserted-by":"publisher","first-page":"1262","DOI":"10.1002\/sim.7993","volume":"38","author":"RD Riley","year":"2019","unstructured":"Riley RD, Snell KIE, Ensor J, Burke DL, Harrell FE Jr, Moons KGM, et al. Minimum sample size for developing a multivariable prediction model: Part I - continuous outcomes. Stat Med. 2019;38(7):1262\u201375.","journal-title":"Stat Med"},{"issue":"7","key":"1487_CR39","doi-asserted-by":"publisher","first-page":"1276","DOI":"10.1002\/sim.7992","volume":"38","author":"RD Riley","year":"2019","unstructured":"Riley RD, Snell KI, Ensor J, Burke DL, Harrell FE Jr, Moons KG, et al. Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes. Stat Med. 2019;38(7):1276\u201396.","journal-title":"Stat Med"}],"container-title":["BMC Medical Research Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12874-021-01487-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12874-021-01487-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12874-021-01487-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,19]],"date-time":"2023-01-19T03:14:15Z","timestamp":1674098055000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedresmethodol.biomedcentral.com\/articles\/10.1186\/s12874-021-01487-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12]]},"references-count":39,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["1487"],"URL":"https:\/\/doi.org\/10.1186\/s12874-021-01487-4","relation":{},"ISSN":["1471-2288"],"issn-type":[{"value":"1471-2288","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12]]},"assertion":[{"value":"14 June 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 November 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 December 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The study protocol and the exempt from the need to obtain informed consent was approved by the Ethics Committee of the Medical University of Vienna (ECS 1232\/2014).","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"284"}}