{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,28]],"date-time":"2026-10-28T07:41:04Z","timestamp":1793173264623,"version":"4.5.0"},"reference-count":26,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,7,1]],"date-time":"2025-07-01T00:00:00Z","timestamp":1751328000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,7,1]],"date-time":"2025-07-01T00:00:00Z","timestamp":1751328000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100018952","name":"Birmingham Biomedical Research Centre","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100018952","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/Y018516\/1"],"award-info":[{"award-number":["EP\/Y018516\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/Y018516\/1"],"award-info":[{"award-number":["EP\/Y018516\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/Y018516\/1"],"award-info":[{"award-number":["EP\/Y018516\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/Y018516\/1"],"award-info":[{"award-number":["EP\/Y018516\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/Y018516\/1"],"award-info":[{"award-number":["EP\/Y018516\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/Y018516\/1"],"award-info":[{"award-number":["EP\/Y018516\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/Y018516\/1"],"award-info":[{"award-number":["EP\/Y018516\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/Y018516\/1"],"award-info":[{"award-number":["EP\/Y018516\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/Y018516\/1"],"award-info":[{"award-number":["EP\/Y018516\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/Y018516\/1"],"award-info":[{"award-number":["EP\/Y018516\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000289","name":"Cancer Research UK","doi-asserted-by":"publisher","award":["C49297\/A27294"],"award-info":[{"award-number":["C49297\/A27294"]}],"id":[{"id":"10.13039\/501100000289","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000289","name":"Cancer Research UK","doi-asserted-by":"publisher","award":["PRCPJT-Nov21\\100021"],"award-info":[{"award-number":["PRCPJT-Nov21\\100021"]}],"id":[{"id":"10.13039\/501100000289","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Res Methodol"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    When evaluating the performance of a model for individualised risk prediction, the sample size needs to be large enough to precisely estimate the performance measures of interest. Current sample size guidance is based on precisely estimating calibration, discrimination, and net benefit, which should be the first stage of calculating the minimum required sample size. However, when a clinically important threshold is used for classification, other performance measures are also often reported. We extend the previously published guidance to precisely estimate threshold-based performance measures. We have reported closed-form solutions to estimate the sample size required to target sufficiently precise estimates of accuracy, specificity, sensitivity, positive predictive value (PPV), negative predictive value (NPV), and an iterative method to estimate the sample size required to target a sufficiently precise estimate of the F1-score, in an external evaluation study of a prediction model with a binary outcome. This approach requires the user to pre-specify the target standard error and the expected value for each performance measure alongside the outcome prevalence. We describe how the sample size formulae were derived and demonstrate their use in an example. Extension to time-to-event outcomes is also considered. In our examples, the minimum sample size required was lower than that required to precisely estimate the calibration slope, and we expect this would most often be the case. Our formulae, along with corresponding Python code and updated R, Stata and Python commands (\n                    <jats:italic>pmvalsampsize<\/jats:italic>\n                    ), enable researchers to calculate the minimum sample size needed to precisely estimate threshold-based performance measures in an external evaluation study. These criteria should be used alongside previously published criteria to precisely estimate the calibration, discrimination, and net-benefit.\n                  <\/jats:p>","DOI":"10.1186\/s12874-025-02592-4","type":"journal-article","created":{"date-parts":[[2025,7,1]],"date-time":"2025-07-01T04:42:29Z","timestamp":1751344949000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Extended sample size calculations for evaluation of prediction models using a threshold for classification"],"prefix":"10.1186","volume":"25","author":[{"given":"Rebecca","family":"Whittle","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joie","family":"Ensor","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lucinda","family":"Archer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gary S.","family":"Collins","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paula","family":"Dhiman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alastair","family":"Denniston","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joseph","family":"Alderman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amardeep","family":"Legha","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maarten","family":"van Smeden","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Karel G.","family":"Moons","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jean-Baptiste","family":"Cazier","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richard D.","family":"Riley","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kym I. E","family":"Snell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,7,1]]},"reference":[{"issue":"1","key":"2592_CR1","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1186\/s41512-022-00136-8","volume":"6","author":"M Sperrin","year":"2022","unstructured":"Sperrin M, Riley RD, Collins GS, Martin GP. Targeted validation: validating clinical prediction models in their intended population and setting. Diagn Progn Res. 2022Dec 22;6(1):24.","journal-title":"Diagn Progn Res"},{"key":"2592_CR2","doi-asserted-by":"crossref","unstructured":"Richard D Riley, Kym I E Snell, Lucinda Archer, Joie Ensor, Thomas P A Debray, Ben van Calster, et al. Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study. BMJ. 2024 Jan 22;384:e074821.","DOI":"10.1136\/bmj-2023-074821"},{"key":"2592_CR3","doi-asserted-by":"publisher","DOI":"10.1136\/bmj-2023-074820","volume":"15","author":"RD Riley","year":"2024","unstructured":"Riley RD, Archer L, Snell KIE, Ensor J, Dhiman P, Martin GP, et al. Evaluation of clinical prediction models (part 2): how to undertake an external validation study. BMJ. 2024Jan;15: e074820.","journal-title":"BMJ"},{"issue":"1","key":"2592_CR4","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1002\/sim.8766","volume":"40","author":"L Archer","year":"2021","unstructured":"Archer L, Snell KIE, Ensor J, Hudda MT, Collins GS, Riley RD. Minimum sample size for external validation of a clinical prediction model with a continuous outcome. Stat Med. 2021Jan 15;40(1):133\u201346.","journal-title":"Stat Med"},{"issue":"19","key":"2592_CR5","doi-asserted-by":"publisher","first-page":"4230","DOI":"10.1002\/sim.9025","volume":"40","author":"RD Riley","year":"2021","unstructured":"Riley RD, Debray TPA, Collins GS, Archer L, Ensor J, Van Smeden M, et al. Minimum sample size for external validation of a clinical prediction model with a binary outcome. Stat Med. 2021Aug 30;40(19):4230\u201351.","journal-title":"Stat Med"},{"key":"2592_CR6","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1016\/j.jclinepi.2021.02.011","volume":"135","author":"KIE Snell","year":"2021","unstructured":"Snell KIE, Archer L, Ensor J, Bonnett LJ, Debray TPA, Phillips B, et al. External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb. J Clin Epidemiol. 2021Jul;135:79\u201389.","journal-title":"J Clin Epidemiol"},{"issue":"7","key":"2592_CR7","doi-asserted-by":"publisher","first-page":"1280","DOI":"10.1002\/sim.9275","volume":"41","author":"RD Riley","year":"2022","unstructured":"Riley RD, Collins GS, Ensor J, Archer L, Booth S, Mozumder SI, et al. Minimum sample size calculations for external validation of a clinical prediction model with a time-to-event outcome. Stat Med. 2022Mar 30;41(7):1280\u201395.","journal-title":"Stat Med"},{"issue":"154","key":"2592_CR8","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1016\/j.jclinepi.2022.11.015","volume":"1","author":"CL Andaur Navarro","year":"2023","unstructured":"Andaur Navarro CL, Damen JAA, van Smeden M, Takada T, Nijman SWJ, Dhiman P, et al. Systematic review identifies the design and methodological conduct of studies on machine learning-based prediction models. J Clin Epidemiol. 2023Feb;1(154):8\u201322.","journal-title":"J Clin Epidemiol"},{"key":"2592_CR9","doi-asserted-by":"publisher","unstructured":"Wainstein M, Flanagan E, Johnson DW, Shrapnel S. Systematic review of externally validated machine learning models for predicting acute kidney injury in general hospital patients. Frontiers in Nephrology. 2023;3. Available from: https:\/\/www.frontiersin.org\/articles\/https:\/\/doi.org\/10.3389\/fneph.2023.1220214","DOI":"10.3389\/fneph.2023.1220214"},{"key":"2592_CR10","doi-asserted-by":"publisher","first-page":"76","DOI":"10.1016\/j.ejim.2023.11.009","volume":"121","author":"T Frondelius","year":"2024","unstructured":"Frondelius T, Atkova I, Miettunen J, Rello J, Vesty G, Chew HSJ, Jansson M. Early prediction of ventilator-associated pneumonia with machine learning models: A systematic review and meta-analysis of prediction model performance\u2730. Eur J Intern Med. 2024;121:76\u201387. https:\/\/doi.org\/10.1016\/j.ejim.2023.11.009. Epub 2023 Nov 18 PMID: 37981529.","journal-title":"Eur J Intern Med"},{"key":"2592_CR11","doi-asserted-by":"crossref","unstructured":"Ensor J. PMVALSAMPSIZE: Stata module to calculate the minimum sample size required for external validation of a multivariable prediction model. Published online; 2023. Available from: https:\/\/EconPapers.repec.org\/RePEc:boc:bocode:s459226","DOI":"10.32614\/CRAN.package.pmvalsampsize"},{"key":"2592_CR12","doi-asserted-by":"crossref","unstructured":"Ensor J. pmvalsampsize: Sample Size for External Validation of a Prediction Model. Published online; 2023. Available from: https:\/\/github.com\/JoieEnsor\/pmvalsampsize","DOI":"10.32614\/CRAN.package.pmvalsampsize"},{"key":"2592_CR13","doi-asserted-by":"crossref","unstructured":"Newcombe RG. Confidence intervals for an effect size measure based on the Mann\u2013Whitney statistic. Part 2: asymptotic methods and evaluation. Statist Med. 2006;25(4):559\u201373.","DOI":"10.1002\/sim.2324"},{"issue":"3","key":"2592_CR14","doi-asserted-by":"publisher","first-page":"843","DOI":"10.1111\/biom.13190","volume":"76","author":"TL Marsh","year":"2020","unstructured":"Marsh TL, Janes H, Pepe MS. Statistical inference for net benefit measures in biomarker validation studies. Biometrics. 2020Sep;76(3):843\u201352.","journal-title":"Biometrics"},{"issue":"5","key":"2592_CR15","doi-asserted-by":"publisher","first-page":"4961","DOI":"10.1007\/s10489-021-02635-5","volume":"52","author":"K Takahashi","year":"2022","unstructured":"Takahashi K, Yamamoto K, Kuchiba A, Koyama T. Confidence interval for micro-averaged F 1 and macro-averaged F 1 scores. Appl Intell (Dordr). 2022Mar;52(5):4961\u201372.","journal-title":"Appl Intell (Dordr)"},{"issue":"4","key":"2592_CR16","doi-asserted-by":"publisher","first-page":"349","DOI":"10.1016\/S2213-2600(20)30559-2","volume":"9","author":"RK Gupta","year":"2021","unstructured":"Gupta RK, Harrison EM, Ho A, Docherty AB, Knight SR, Van Smeden M, et al. Development and validation of the ISARIC 4C Deterioration model for adults hospitalised with COVID-19: a prospective cohort study. Lancet Respir Med. 2021Apr;9(4):349\u201359.","journal-title":"Lancet Respir Med"},{"issue":"2","key":"2592_CR17","first-page":"119","volume":"52","author":"A Agresti","year":"1998","unstructured":"Agresti A, Coull BA. Approximate Is Better than \u2018Exact\u2019 for Interval Estimation of Binomial Proportions. Am Stat. 1998May;52(2):119.","journal-title":"Am Stat"},{"issue":"3","key":"2592_CR18","first-page":"408","volume":"10","author":"ET Parner","year":"2010","unstructured":"Parner ET, Andersen PK. Regression Analysis of Censored Data Using Pseudo-observations. Stand Genomic Sci. 2010Sep;10(3):408\u201322.","journal-title":"Stand Genomic Sci"},{"issue":"1","key":"2592_CR19","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1177\/0962280209105020","volume":"19","author":"PK Andersen","year":"2010","unstructured":"Andersen PK, Pohar PM. Pseudo-observations in survival analysis. Stat Methods Med Res. 2010Feb;19(1):71\u201399.","journal-title":"Stat Methods Med Res"},{"key":"2592_CR20","doi-asserted-by":"crossref","unstructured":"Ensor J, Riley RD, Jowett S, Monahan M, Snell KI, Bayliss S, et al. Prediction of risk of recurrence of venous thromboembolism following treatment for a first unprovoked venous thromboembolism: systematic review, prognostic model and clinical decision rule, and economic evaluation. Health Technol Assess. 2016 Feb;20(12):i\u2013xxxiii, 1\u2013190.","DOI":"10.3310\/hta20120"},{"issue":"3","key":"2592_CR21","volume":"3","author":"A Negida","year":"2019","unstructured":"Negida A, Fahim NK, Negida Y. Sample Size Calculation Guide - Part 4: How to Calculate the Sample Size for a Diagnostic Test Accuracy Study based on Sensitivity, Specificity, and the Area Under the ROC Curve. Adv J Emerg Med. 2019;3(3): e33.","journal-title":"Adv J Emerg Med"},{"key":"2592_CR22","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1016\/j.jbi.2014.02.013","volume":"48","author":"K Hajian-Tilaki","year":"2014","unstructured":"Hajian-Tilaki K. Sample size estimation in diagnostic test studies of biomedical informatics. J Biomed Inform. 2014Apr;48:193\u2013204.","journal-title":"J Biomed Inform"},{"issue":"21","key":"2592_CR23","doi-asserted-by":"publisher","first-page":"2714","DOI":"10.1002\/sim.8570","volume":"39","author":"PC Austin","year":"2020","unstructured":"Austin PC, Harrell FE, Van Klaveren D. Graphical calibration curves and the integrated calibration index (ICI) for survival models. Stat Med. 2020Sep 20;39(21):2714\u201342.","journal-title":"Stat Med"},{"issue":"1","key":"2592_CR24","doi-asserted-by":"publisher","first-page":"192","DOI":"10.1186\/s12916-019-1425-3","volume":"17","author":"L Wynants","year":"2019","unstructured":"Wynants L, Van Smeden M, McLernon DJ, Timmerman D, Steyerberg EW, Van Calster B, et al. Three myths about risk thresholds for prediction models. BMC Med. 2019Dec;17(1):192.","journal-title":"BMC Med"},{"issue":"1","key":"2592_CR25","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1097\/00001648-199701000-00002","volume":"8","author":"KGM Moons","year":"1997","unstructured":"Moons KGM, Van Es GA, Deckers JW, Habbema JDF, Grobbee DE. Limitations of Sensitivity, Specificity, Likelihood Ratio, and Bayes\u02bc Theorem in Assessing Diagnostic Probabilities: A Clinical Example. Epidemiology. 1997Jan;8(1):12\u20137.","journal-title":"Epidemiology"},{"issue":"6","key":"2592_CR26","doi-asserted-by":"publisher","first-page":"670","DOI":"10.1016\/S1076-6332(03)80087-9","volume":"10","author":"KGM Moons","year":"2003","unstructured":"Moons KGM, Harrell FE. Sensitivity and Specificity should be De-emphasized in Diagnostic Accuracy Studies. Acad Radiol. 2003Jun;10(6):670\u20132.","journal-title":"Acad Radiol"}],"container-title":["BMC Medical Research Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12874-025-02592-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12874-025-02592-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12874-025-02592-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,1]],"date-time":"2025-07-01T04:42:31Z","timestamp":1751344951000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedresmethodol.biomedcentral.com\/articles\/10.1186\/s12874-025-02592-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,1]]},"references-count":26,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["2592"],"URL":"https:\/\/doi.org\/10.1186\/s12874-025-02592-4","relation":{},"ISSN":["1471-2288"],"issn-type":[{"value":"1471-2288","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,1]]},"assertion":[{"value":"20 November 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 May 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 July 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"RDR receives royalties for the textbooks \u2018Prognosis Research in Healthcare\u2019 and \u2018Individual Participant Data Meta-Analysis\u2019.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"170"}}