{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T01:22:40Z","timestamp":1784251360628,"version":"3.55.0"},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,10,14]],"date-time":"2025-10-14T00:00:00Z","timestamp":1760400000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,10,14]],"date-time":"2025-10-14T00:00:00Z","timestamp":1760400000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000272","name":"National Institute for Health and Care Research","doi-asserted-by":"publisher","award":["NIHR300100"],"award-info":[{"award-number":["NIHR300100"]}],"id":[{"id":"10.13039\/501100000272","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000289","name":"Cancer Research UK","doi-asserted-by":"publisher","award":["C14183\/A29739"],"award-info":[{"award-number":["C14183\/A29739"]}],"id":[{"id":"10.13039\/501100000289","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000289","name":"Cancer Research UK","doi-asserted-by":"publisher","award":["C41379\/A27583"],"award-info":[{"award-number":["C41379\/A27583"]}],"id":[{"id":"10.13039\/501100000289","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002794","name":"Cancerfonden","doi-asserted-by":"publisher","award":["2018\/744"],"award-info":[{"award-number":["2018\/744"]}],"id":[{"id":"10.13039\/501100002794","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004359","name":"Vetenskapsr\u00e5det","doi-asserted-by":"publisher","award":["2017 -01591"],"award-info":[{"award-number":["2017 -01591"]}],"id":[{"id":"10.13039\/501100004359","id-type":"DOI","asserted-by":"publisher"}]},{"name":"NIHR-MRC Better Methods Better Research","award":["MR\/Z503873\/1"],"award-info":[{"award-number":["MR\/Z503873\/1"]}]},{"DOI":"10.13039\/501100018952","name":"Birmingham Biomedical Research Centre","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100018952","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004047","name":"Karolinska Institute","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004047","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Diagn Progn Res"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>When developing\/validating prognostic models, it is typical to assess calibration between predicted and observed risks \u2014 either in the development dataset or in an external sample. For competing risks data, correct specification of more than one model may be required to ensure well-calibrated predicted risks for the event of interest. Furthermore, interest may be in the predicted risks of the event of interest, competing events and all-causes. Therefore, calibration must be assessed simultaneously using various measures.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We focus on the calibration of prediction models for external validation using a cause-specific hazards approach. We propose that miscalibration for cause-specific hazard models be assessed using components specific to each model through the complement of the cause-specific survival alongside the assessment of the calibration of the cause-specific absolute risks. We simulated a range of scenarios to illustrate how to identify which model(s) are mis-specified in an external validation setting. Calibration plots and calibration statistics (calibration slope, calibration-in-the-large) are presented alongside performance measures such as the Brier score and Index of Prediction Accuracy. We use pseudo-observations to calculate observed risks and generate a smooth calibration curve with restricted cubic splines. We fitted flexible parametric survival models to the simulated data to flexibly estimate baseline cause-specific hazards for the prediction of individual cause-specific absolute risks.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Our simulations illustrate that miscalibration due to changes in the baseline cause-specific hazards in external validation data is better identified using components from each cause-specific model. A mis-calibrated model on one cause could lead to poor calibration of the predicted absolute risks for each cause of interest, including the all-cause absolute risk. This is because prediction of a single cause-specific absolute risk is impacted by effects of variables on the cause of interest and competing events.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>If accurate predictions for both all-cause and each cause-specific absolute risks are of interest, this is best achieved by developing and validating models via the cause-specific hazards approach. For each cause-specific model, researchers should evaluate calibration plots separately using the complement of the cause-specific survival function to reveal the cause of any miscalibration. However, this also requires careful consideration of dependent censoring which must be sufficiently accounted for.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s41512-025-00197-5","type":"journal-article","created":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T22:03:15Z","timestamp":1760392995000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Calibration of cause-specific absolute risk for external validation using each cause-specific hazards model in the presence of competing events"],"prefix":"10.1186","volume":"9","author":[{"given":"Sarwar I.","family":"Mozumder","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sarah","family":"Booth","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richard D.","family":"Riley","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark J.","family":"Rutherford","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paul C.","family":"Lambert","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,10,14]]},"reference":[{"issue":"18","key":"197_CR1","doi-asserted-by":"publisher","first-page":"3191","DOI":"10.1002\/sim.6152","volume":"33","author":"TA Gerds","year":"2014","unstructured":"Gerds TA, Andersen PK, Kattan MW. Calibration plots for risk prediction models in the presence of competing risks. Stat Med. 2014;33(18):3191\u2013203.","journal-title":"Stat Med"},{"key":"197_CR2","volume-title":"others","author":"EW Steyerberg","year":"2019","unstructured":"Steyerberg EW. others. Clinical prediction models: Springer; 2019."},{"key":"197_CR3","doi-asserted-by":"crossref","unstructured":"Riley RD, van der Windt D, Croft P, Moons KG. Prognosis research in healthcare: concepts, methods, and impact.\u00a0Oxford:\u00a0Oxford University Press; 2019.","DOI":"10.1093\/med\/9780198796619.001.0001"},{"key":"197_CR4","doi-asserted-by":"crossref","unstructured":"Wolbers M, Koller MT, Witteman JC, Steyerberg EW. Prognostic models with competing risks: methods and application to coronary risk prediction. Epidemiology. 2009;15(3):555\u201361.","DOI":"10.1097\/EDE.0b013e3181a39056"},{"issue":"3","key":"197_CR5","doi-asserted-by":"publisher","first-page":"526","DOI":"10.1093\/biostatistics\/kxt059","volume":"15","author":"M Wolbers","year":"2014","unstructured":"Wolbers M, Blanche P, Koller MT, Witteman JC, Gerds TA. Concordance for prognostic models with competing risks. Biostatistics. 2014;15(3):526\u201339.","journal-title":"Biostatistics"},{"issue":"19","key":"197_CR6","doi-asserted-by":"publisher","first-page":"4200","DOI":"10.1002\/sim.9023","volume":"40","author":"PC Austin","year":"2021","unstructured":"Austin PC, Steyerberg EW, Putter H. Fine-Gray subdistribution hazard models to simultaneously estimate the absolute risk of different event types: cumulative total failure probability may exceed 1. Stat Med. 2021;40(19):4200\u201312.","journal-title":"Stat Med"},{"key":"197_CR7","doi-asserted-by":"crossref","unstructured":"van Geloven N, Giardiello D, Bonneville EF, Teece L, Ramspek CL, van Smeden M, et al. Validation of prediction models in the presence of competing risks: a guide through modern methods. BMJ. 2022;24(377).","DOI":"10.1136\/bmj-2021-069249"},{"issue":"1","key":"197_CR8","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1186\/s41512-021-00114-6","volume":"6","author":"PC Austin","year":"2022","unstructured":"Austin PC, Putter H, Giardiello D, van Klaveren D. Graphical calibration curves and the Integrated Calibration Index (ICI) for competing risk models. Diagn Progn Res. 2022;6(1):2.","journal-title":"Diagn Progn Res"},{"key":"197_CR9","doi-asserted-by":"crossref","unstructured":"Abdel-Qadir H, Fang J, Lee DS, Tu JV, Amir E, Austin PC, et al. Importance of considering competing risks in time-to-event analyses.\u00a0Circ Cardiovasc Qual Outcomes.\u00a02018;11(7):e004580.","DOI":"10.1161\/CIRCOUTCOMES.118.004580"},{"issue":"16","key":"197_CR10","doi-asserted-by":"publisher","first-page":"325","DOI":"10.21037\/atm.2018.07.38","volume":"6","author":"Z Zhang","year":"2018","unstructured":"Zhang Z, Cortese G, Combescure C, Marshall R, Lee M, Lim HJ, et al. Overview of model validation for survival regression model with competing risks using melanoma study data. Ann Transl Med. 2018;6(16):325.","journal-title":"Ann Transl Med"},{"key":"197_CR11","doi-asserted-by":"crossref","unstructured":"Hippisley-Cox J, Coupland C. Development and validation of risk prediction equations to estimate survival in patients with colorectal cancer: cohort study. BMJ. 2017;15(357).","DOI":"10.1136\/bmj.j2497"},{"key":"197_CR12","doi-asserted-by":"publisher","first-page":"496","DOI":"10.1080\/01621459.1999.10474144","volume":"446","author":"JP Fine","year":"1999","unstructured":"Fine JP, Gray RJ. A proportional hazards model for the subdistribution of a competing risk. J Am Stat Assoc. 1999;446:496\u2013509.","journal-title":"J Am Stat Assoc"},{"issue":"1","key":"197_CR13","doi-asserted-by":"publisher","first-page":"29","DOI":"10.15256\/joc.2014.4.41","volume":"4","author":"EA Bayliss","year":"2014","unstructured":"Bayliss EA, Reifler LM, Zeng C, McQuillan DB, Ellis JL, Steiner JF. Competing risks of cancer mortality and cardiovascular events in individuals with multimorbidity. Journal of comorbidity. 2014;4(1):29\u201336.","journal-title":"Journal of comorbidity"},{"key":"197_CR14","doi-asserted-by":"publisher","first-page":"2936","DOI":"10.1093\/eurheartj\/ehu131","volume":"35","author":"M Wolbers","year":"2014","unstructured":"Wolbers M, Koller MT, Stel VS, Schaer B, Jager KJ, Leffondre K, et al. Competing risks analyses: objectives and approaches. Eur Heart J. 2014;35:2936\u201341.","journal-title":"Eur Heart J."},{"issue":"6","key":"197_CR15","doi-asserted-by":"publisher","first-page":"956","DOI":"10.1002\/sim.3516","volume":"28","author":"J Beyersmann","year":"2009","unstructured":"Beyersmann J, Latouche A, Buchholz A, Schumacher M. Simulating competing risks data in survival analysis. Stat Med. 2009;28(6):956\u201371.","journal-title":"Stat Med"},{"issue":"11","key":"197_CR16","doi-asserted-by":"publisher","first-page":"2389","DOI":"10.1002\/sim.2712","volume":"26","author":"H Putter","year":"2007","unstructured":"Putter H, Fiocco M, Geskus RB. Tutorial in biostatistics: competing risks and multi-state models. Stat Med. 2007May;26(11):2389\u2013430.","journal-title":"Stat Med"},{"key":"197_CR17","doi-asserted-by":"publisher","unstructured":"Lambert PC, Wilkes SR, Crowther MJ. Flexible parametric modelling of the cause-specific cumulative incidence function. Statist Med. 2017;36: 1429\u201346. https:\/\/doi.org\/10.1002\/sim.7208.","DOI":"10.1002\/sim.7208"},{"issue":"1","key":"197_CR18","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1002\/sim.7498","volume":"37","author":"SI Mozumder","year":"2018","unstructured":"Mozumder SI, Rutherford M, Lambert P. Direct likelihood inference on the cause-specific cumulative incidence function: a flexible parametric regression modelling approach. Stat Med. 2018;37(1):82\u201397.","journal-title":"Stat Med"},{"key":"197_CR19","unstructured":"Aalen OO, Johansen S. An empirical transition matrix for non-homogeneous Markov chains based on censored observations. Scand J Stat. 1978;141\u201350."},{"key":"197_CR20","doi-asserted-by":"publisher","first-page":"861","DOI":"10.1093\/ije\/dyr213","volume":"41","author":"PK Andersen","year":"2012","unstructured":"Andersen PK, Geskus RB, de Witte T, Putter H. Competing risks in epidemiology: possibilities and pitfalls. Int J Epidemiol. 2012;41:861\u201370.","journal-title":"Int J Epidemiol"},{"issue":"4","key":"197_CR21","doi-asserted-by":"publisher","first-page":"541","DOI":"10.2307\/2530374","volume":"34","author":"RL Prentice","year":"1978","unstructured":"Prentice RL, Kalbfleisch JD, Peterson AV Jr, Flournoy N, Farewell VT, Breslow NE. The analysis of failure times in the presence of competing risks. Biometrics. 1978;34(4):541\u201354.","journal-title":"Biometrics"},{"issue":"135","key":"197_CR22","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1016\/j.jclinepi.2021.02.011","volume":"1","author":"KIE Snell","year":"2021","unstructured":"Snell KIE, Archer L, Ensor J, Bonnett LJ, Debray TPA, Phillips B, et al. External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb. J Clin Epidemiol. 2021;1(135):79\u201389.","journal-title":"J Clin Epidemiol"},{"key":"197_CR23","first-page":"265","volume":"9","author":"PC Lambert","year":"2009","unstructured":"Lambert PC, Royston P. Further development of flexible parametric models for survival analysis. Stand Genomic Sci. 2009;9:265\u201390.","journal-title":"Stand Genomic Sci"},{"issue":"1","key":"197_CR24","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1002\/sim.8757","volume":"40","author":"PK Andersen","year":"2021","unstructured":"Andersen PK, Pohar Perme M, van Houwelingen HC, Cook RJ, Joly P, Martinussen T, et al. Analysis of time-to-event for observational studies: guidance to the use of intensity models. Stat Med. 2021;40(1):185\u2013211.","journal-title":"Stat Med"},{"issue":"4","key":"197_CR25","first-page":"738","volume":"14","author":"P Royston","year":"2014","unstructured":"Royston P. Tools for checking calibration of a Cox model in external validation: approach based on individual event probabilities. Stand Genomic Sci. 2014;14(4):738\u201355.","journal-title":"Stand Genomic Sci"},{"issue":"1","key":"197_CR26","doi-asserted-by":"publisher","first-page":"230","DOI":"10.1186\/s12916-019-1466-7","volume":"17","author":"B Van Calster","year":"2019","unstructured":"Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW, Bossuyt P, et al. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17(1):230.","journal-title":"BMC Med"},{"issue":"1","key":"197_CR27","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1177\/0962280209105020","volume":"19","author":"PK Andersen","year":"2010","unstructured":"Andersen PK, Perme MP. Pseudo-observations in survival analysis. Stat Methods Med Res. 2010;19(1):71\u201399.","journal-title":"Stat Methods Med Res"},{"issue":"5","key":"197_CR28","doi-asserted-by":"publisher","first-page":"551","DOI":"10.1002\/sim.4780080504","volume":"8","author":"S Durrleman","year":"1989","unstructured":"Durrleman S, Simon R. Flexible regression models with cubic splines. Stat Med. 1989;8(5):551\u201361.","journal-title":"Stat Med"},{"key":"197_CR29","doi-asserted-by":"crossref","unstructured":"Harrell FE. Regression modeling strategies with application to linear models, logistic regression and survival analysis.\u00a0Oxford: Springer; 2001.","DOI":"10.1007\/978-1-4757-3462-1"},{"issue":"1","key":"197_CR30","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1097\/EDE.0b013e3181c30fb2","volume":"21","author":"EW Steyerberg","year":"2010","unstructured":"Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128\u201338.","journal-title":"Epidemiology"},{"issue":"18","key":"197_CR31","doi-asserted-by":"publisher","first-page":"3089","DOI":"10.1002\/sim.5773","volume":"32","author":"G Cortese","year":"2013","unstructured":"Cortese G, Gerds TA, Andersen PK. Comparing predictions among competing risks models with time-dependent covariates. Stat Med. 2013;32(18):3089\u2013101.","journal-title":"Stat Med"},{"issue":"7","key":"197_CR32","doi-asserted-by":"publisher","first-page":"890","DOI":"10.1093\/bioinformatics\/btp088","volume":"25","author":"H Binder","year":"2009","unstructured":"Binder H, Allignol A, Schumacher M, Beyersmann J. Boosting for high-dimensional time-to-event data with competing risks. Bioinformatics. 2009;25(7):890\u20136.","journal-title":"Bioinformatics"},{"issue":"1","key":"197_CR33","doi-asserted-by":"publisher","first-page":"88","DOI":"10.1002\/bimj.201000073","volume":"53","author":"R Schoop","year":"2011","unstructured":"Schoop R, Beyersmann J, Schumacher M, Binder H. Quantifying the predictive accuracy of time-to-event models in the presence of competing risks. Biom J. 2011;53(1):88\u2013112.","journal-title":"Biom J"},{"issue":"1","key":"197_CR34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s41512-018-0029-2","volume":"2","author":"MW Kattan","year":"2018","unstructured":"Kattan MW, Gerds TA. The index of prediction accuracy: an intuitive measure useful for evaluating risk prediction models. Diagnostic and prognostic research. 2018;2(1):1\u20137.","journal-title":"Diagnostic and prognostic research"},{"key":"197_CR35","unstructured":"Linden A, Gerds TA, Huber C. STBRIER: Stata module for estimating the Brier score for survival (censored) data. [Internet]. Statistical Software Components S458486, Boston College Department of Economics,; 2017. Available from: https:\/\/ideas.repec.org\/c\/boc\/bocode\/s458368.html"},{"key":"197_CR36","unstructured":"Ensor J, Snell KI, Martin EC. PMCALPLOT: Stata module to produce calibration plot of prediction model performance [Internet]. Statistical Software Components S458486, Boston College Department of Economics; 2018. Available from: https:\/\/ideas.repec.org\/c\/boc\/bocode\/s458486.html"},{"issue":"4","key":"197_CR37","doi-asserted-by":"publisher","first-page":"708","DOI":"10.1093\/biostatistics\/kxw017","volume":"17","author":"F Ambrogi","year":"2016","unstructured":"Ambrogi F, Scheike TH. Penalized estimation for competing risks regression with applications to high-dimensional covariates. Biostatistics. 2016;17(4):708\u201321.","journal-title":"Biostatistics"},{"issue":"2","key":"197_CR38","doi-asserted-by":"publisher","first-page":"288","DOI":"10.1002\/bimj.201600242","volume":"60","author":"M Saadati","year":"2018","unstructured":"Saadati M, Beyersmann J, Kopp-Schneider A, Benner A. Prediction accuracy and variable selection for penalized cause-specific hazards models. Biom J. 2018;60(2):288\u2013306.","journal-title":"Biom J"},{"issue":"1","key":"197_CR39","doi-asserted-by":"publisher","first-page":"W1","DOI":"10.7326\/M14-0698","volume":"162","author":"KGM Moons","year":"2015","unstructured":"Moons KGM, Altman DG, Reitsma JB, Ioannidis JPA, Macaskill P, Steyerberg EW, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1-73.","journal-title":"Ann Intern Med"}],"container-title":["Diagnostic and Prognostic Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s41512-025-00197-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s41512-025-00197-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s41512-025-00197-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T22:03:19Z","timestamp":1760392999000},"score":1,"resource":{"primary":{"URL":"https:\/\/diagnprognres.biomedcentral.com\/articles\/10.1186\/s41512-025-00197-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,14]]},"references-count":39,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["197"],"URL":"https:\/\/doi.org\/10.1186\/s41512-025-00197-5","relation":{"has-preprint":[{"id-type":"doi","id":"10.21203\/rs.3.rs-3416448\/v1","asserted-by":"object"}]},"ISSN":["2397-7523"],"issn-type":[{"value":"2397-7523","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,14]]},"assertion":[{"value":"10 October 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 June 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 October 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"S. I. M. was employed by AstraZeneca throughout the review process and is currently employed by GSK at the time of submission of this manuscript. The other authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"23"}}