{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,22]],"date-time":"2026-07-22T12:47:52Z","timestamp":1784724472654,"version":"3.55.0"},"reference-count":21,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,10,15]],"date-time":"2019-10-15T00:00:00Z","timestamp":1571097600000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2019,10,15]],"date-time":"2019-10-15T00:00:00Z","timestamp":1571097600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"crossref","award":["P50-CA92629"],"award-info":[{"award-number":["P50-CA92629"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"crossref","award":["P30-CA008748"],"award-info":[{"award-number":["P30-CA008748"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"crossref","award":["R01-CA179115"],"award-info":[{"award-number":["R01-CA179115"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"crossref","award":["P30-CA054174"],"award-info":[{"award-number":["P30-CA054174"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"crossref","award":["K24-CA160653"],"award-info":[{"award-number":["K24-CA160653"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000090","name":"Congressionally Directed Medical Research Programs","doi-asserted-by":"crossref","award":["W81XWH-15-1-0441"],"award-info":[{"award-number":["W81XWH-15-1-0441"]}],"id":[{"id":"10.13039\/100000090","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Res Methodol"],"published-print":{"date-parts":[[2019,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n              <jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>Online clinical risk prediction tools built on data from multiple cohorts are increasingly being utilized for contemporary doctor-patient decision-making and validation. This report outlines a comprehensive data science strategy for building such tools with application to the Prostate Biopsy Collaborative Group prostate cancer risk prediction tool.<\/jats:p>\n              <\/jats:sec>\n              <jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>We created models for high-grade prostate cancer risk using six established risk factors. The data comprised 8492 prostate biopsies collected from ten institutions, 2 in Europe and 8 across North America. We calculated area under the receiver operating characteristic curve (AUC) for discrimination, the Hosmer-Lemeshow test statistic (HLS) for calibration and the clinical net benefit at risk threshold 15%. We implemented several internal cross-validation schemes to assess the influence of modeling method and individual cohort on validation performance.<\/jats:p>\n              <\/jats:sec>\n              <jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>High-grade disease prevalence ranged from 18% in Zurich (1863 biopsies) to 39% in UT Health San Antonio (899 biopsies). Visualization revealed outliers in terms of risk factors, including San Juan VA (51% abnormal digital rectal exam), Durham VA (63% African American), and Zurich (2.8% family history). Exclusion of any cohort did not significantly affect the AUC or HLS, nor did the choice of prediction model (pooled, random-effects, meta-analysis). Excluding the lowest-prevalence Zurich cohort from training sets did not statistically significantly change the validation metrics for any of the\u00a0individual cohorts, except for Sunnybrook, where the effect on the AUC was minimal. Therefore the final multivariable logistic model was built by pooling the data from all cohorts using logistic regression. Higher prostate-specific antigen and age, abnormal digital rectal exam, African ancestry and a family history of prostate cancer increased risk of high-grade prostate cancer, while a history of a prior negative prostate biopsy decreased risk (all <jats:italic>p<\/jats:italic>-values &lt;\u20090.004).<\/jats:p>\n              <\/jats:sec>\n              <jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>We have outlined a multi-cohort model-building internal validation strategy for developing globally accessible and scalable risk prediction tools.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12874-019-0839-0","type":"journal-article","created":{"date-parts":[[2019,10,15]],"date-time":"2019-10-15T11:22:10Z","timestamp":1571138530000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Multi-cohort modeling strategies for scalable globally accessible prostate cancer risk tools"],"prefix":"10.1186","volume":"19","author":[{"given":"Johanna","family":"Tolksdorf","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael W.","family":"Kattan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephen A.","family":"Boorjian","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephen J.","family":"Freedland","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Karim","family":"Saba","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cedric","family":"Poyet","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lourdes","family":"Guerrios","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amanda","family":"De Hoedt","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael A.","family":"Liss","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robin J.","family":"Leach","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Javier","family":"Hernandez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emily","family":"Vertosick","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew J.","family":"Vickers","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4229-2462","authenticated-orcid":false,"given":"Donna P.","family":"Ankerst","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2019,10,15]]},"reference":[{"issue":"13","key":"839_CR1","doi-asserted-by":"publisher","first-page":"1320","DOI":"10.1056\/NEJMoa0810084","volume":"360","author":"FH Schr\u00f6der","year":"2009","unstructured":"Schr\u00f6der FH, Hugosson J, Roobol MJ, Tammela TL, Ciatto S, Nelen V, Kwiatkowski M, Lujan M, Lilja H, Zappa M, Denis LJ, Recker F, Berenguer A, M\u00e4\u00e4tt\u00e4nen L, Bangma CH, Aus G, Villers A, Rebillard X, van der Kwast T, Blijenberg BG, Moss SM, de Koning HJ, Auvinen A, ERSPC Investigators. Screening and prostate-cancer mortality in a randomized European study. N Engl J Med. 2009;360(13):1320\u20138. \n                    https:\/\/doi.org\/10.1056\/NEJMoa0810084\n                    \n                  .","journal-title":"N Engl J Med"},{"issue":"8","key":"839_CR2","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1093\/jnci\/djj131","volume":"98","author":"IM Thompson","year":"2006","unstructured":"Thompson IM, Ankerst DP, Chi C, Goodman PJ, Tangen CM, Lucia MS, Feng Z, Parnes HL, Coltman CA Jr. Assessing prostate cancer risk: results from the Prostate Cancer Prevention Trial. J Natl Cancer Inst. 2006;98(8):529\u201334. \n                    https:\/\/doi.org\/10.1093\/jnci\/djj131\n                    \n                  .","journal-title":"J Natl Cancer Inst"},{"issue":"1","key":"839_CR3","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1016\/j.juro.2016.08.103","volume":"197","author":"R Chen","year":"2017","unstructured":"Chen R, Sjoberg DD, Huang Y, Xie L, Zhou L, He D, Vickers AJ, Sun Y, Chinese Prostate Cancer Consortium, Prostate Biopsy Collaborative Group. Prostate specific antigen and prostate cancer in Chinese men undergoing initial prostate biopsies compared with western cohorts. J Urol. 2017;197(1):90\u20136. \n                    https:\/\/doi.org\/10.1016\/j.juro.2016.08.103\n                    \n                  .","journal-title":"J Urol"},{"issue":"1","key":"839_CR4","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1007\/s00345-012-0869-2","volume":"32","author":"DP Ankerst","year":"2014","unstructured":"Ankerst DP, Boeck A, Freedland SJ, Jones JS, Cronin AM, Roobol MJ, Hugosson J, Kattan MW, Klein EA, Hamdy F, Neal D, Donovan J, Parekh DJ, Klocker H, Horninger W, Benchikh A, Salama G, Villers A, Moreira DM, Schr\u00f6der FH, Lilja H, Vickers AJ, Thompson IM. Evaluating the prostate cancer prevention trial high grade prostate cancer risk calculator in 10 international biopsy cohorts: results from the Prostate Biopsy Collaborative Group. World J Urol. 2014;32(1):185\u201391. \n                    https:\/\/doi.org\/10.1007\/s00345-012-0869-2\n                    \n                  .","journal-title":"World J Urol"},{"issue":"1","key":"839_CR5","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1016\/j.juro.2015.01.092","volume":"194","author":"AN Strobl","year":"2015","unstructured":"Strobl AN, Thompson IM, Vickers AJ, Ankerst DP. The next generation of clinical decision making tools: development of a real-time prediction tool for outcome of prostate biopsy in response to a continuously evolving prostate cancer landscape. J Urol. 2015;194(1):58\u201364. \n                    https:\/\/doi.org\/10.1016\/j.juro.2015.01.092\n                    \n                  .","journal-title":"J Urol"},{"issue":"17","key":"839_CR6","doi-asserted-by":"publisher","first-page":"4374","DOI":"10.1158\/1078-0432.CCR-10-1328","volume":"16","author":"AJ Vickers","year":"2010","unstructured":"Vickers AJ, Cronin AM, Roobol MJ, Hugosson J, Jones JS, Kattan MW, Klein E, Hamdy F, Neal D, Donovan J, Parekh DJ, Ankerst D, Bartsch G, Klocker H, Horninger W, Benchikh A, Salama G, Villers A, Freedland SJ, Moreira DM, Schr\u00f6der FH, Lilja H. The relationship between prostate-specific antigen and prostate cancer risk: the Prostate Biopsy Collaborative Group. Clin Cancer Res. 2010;16(17):4374\u201381. \n                    https:\/\/doi.org\/10.1158\/1078-0432.CCR-10-1328\n                    \n                  .","journal-title":"Clin Cancer Res"},{"issue":"2","key":"839_CR7","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1016\/j.eururo.2018.05.003","volume":"74","author":"DP Ankerst","year":"2018","unstructured":"Ankerst DP, Straubinger J, Selig K, Guerrios L, De Hoedt A, Hernandez J, Liss MA, Leach RJ, Freedland SJ, Kattan MW, Nam R, Haese A, Montorsi F, Boorjian SA, Cooperberg MR, Poyet C, Vertosick E, Vickers AJ. A contemporary prostate biopsy risk calculator based on multiple heterogeneous cohorts. Eur Urol. 2018;74(2):197\u2013203. \n                    https:\/\/doi.org\/10.1016\/j.eururo.2018.05.003\n                    \n                  .","journal-title":"Eur Urol"},{"issue":"4","key":"839_CR8","doi-asserted-by":"publisher","first-page":"e60650","DOI":"10.1371\/journal.pone.0060650","volume":"8","author":"TP Debray","year":"2013","unstructured":"Debray TP, Moons KG, Abo-Zaid GM, Koffijberg H, Riley RD. Individual participant data meta-analysis for a binary outcome: one-stage or two-stage? PLoS One. 2013;8(4):e60650. \n                    https:\/\/doi.org\/10.1371\/journal.pone.0060650\n                    \n                  .","journal-title":"PLoS One"},{"key":"839_CR9","volume-title":"R: a language and environment for statistical computing","author":"R Core Team","year":"2019","unstructured":"R Core Team. R: a language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2019. \n                    https:\/\/www.R-project.org\/"},{"key":"839_CR10","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24277-4","volume-title":"ggplot2: elegant graphics for data analysis","author":"H Wickham","year":"2016","unstructured":"Wickham H. ggplot2: elegant graphics for data analysis. New York: Springer-Verlag; 2016."},{"key":"839_CR11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12916-014-0241-z","volume":"13","author":"GS Collins","year":"2015","unstructured":"Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMC Med. 2015;13:1. \n                    https:\/\/doi.org\/10.1186\/s12916-014-0241-z\n                    \n                  .","journal-title":"BMC Med"},{"issue":"3","key":"839_CR12","doi-asserted-by":"publisher","first-page":"659","DOI":"10.1111\/j.1467-985X.2009.00587.x","volume":"172","author":"A Skrondal","year":"2009","unstructured":"Skrondal A, Rabe-Hesketh S. Prediction in multilevel generalized linear models. J R Stat Soc Ser A. 2009;172(3):659\u201387. \n                    https:\/\/doi.org\/10.1111\/j.1467-985X.2009.00587.x\n                    \n                  .","journal-title":"J R Stat Soc Ser A"},{"issue":"7","key":"839_CR13","doi-asserted-by":"publisher","first-page":"721","DOI":"10.1016\/j.jclinepi.2009.12.008","volume":"63","author":"KJ Janssen","year":"2010","unstructured":"Janssen KJ, Donders AR, Harrell FE Jr, Vergouwe Y, Chen Q, Grobbee DE, Moons KG. Missing covariate data in medical research: to impute is better than to ignore. J Clin Epidemiol. 2010;63(7):721\u20137. \n                    https:\/\/doi.org\/10.1016\/j.jclinepi.2009.12.008\n                    \n                  .","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"839_CR14","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1148\/radiology.143.1.7063747","volume":"143","author":"JA Hanley","year":"1982","unstructured":"Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29\u201336. \n                    https:\/\/doi.org\/10.1148\/radiology.143.1.7063747\n                    \n                  .","journal-title":"Radiology"},{"key":"839_CR15","doi-asserted-by":"publisher","DOI":"10.1002\/9781118548387","volume-title":"Applied logistic regression","author":"DW Hosmer","year":"2013","unstructured":"Hosmer DW, Lemeshow S. Applied logistic regression. 3rd ed. New Jersey: Wiley; 2013. \n                    https:\/\/doi.org\/10.1002\/9781118548387\n                    \n                  .","edition":"3"},{"issue":"6","key":"839_CR16","doi-asserted-by":"publisher","first-page":"565","DOI":"10.1177\/0272989X06295361","volume":"26","author":"AJ Vickers","year":"2006","unstructured":"Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26(6):565\u201374. \n                    https:\/\/doi.org\/10.1177\/0272989X06295361\n                    \n                  .","journal-title":"Med Decis Making"},{"issue":"18","key":"839_CR17","doi-asserted-by":"publisher","first-page":"3158","DOI":"10.1002\/sim.5732","volume":"32","author":"TPA Debray","year":"2013","unstructured":"Debray TPA, Moons KGM, Ahmed I, Koffijberg H, Riley RD. A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-analysis. Stat Med. 2013;32(18):3158\u201380. \n                    https:\/\/doi.org\/10.1002\/sim.5732\n                    \n                  .","journal-title":"Stat Med"},{"issue":"4","key":"839_CR18","doi-asserted-by":"publisher","first-page":"676","DOI":"10.1214\/088342304000000396","volume":"19","author":"MD Ernst","year":"2004","unstructured":"Ernst MD. Permutation methods: a basis for exact inference. Statistical Sci. 2004;19(4):676\u201385. \n                    https:\/\/doi.org\/10.1214\/088342304000000396\n                    \n                  .","journal-title":"Statistical Sci"},{"key":"839_CR19","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1186\/1471-2288-14-3","volume":"14","author":"I Ahmed","year":"2014","unstructured":"Ahmed I, Debray TP, Moons KG, Riley RD. Developing and validating risk prediction models in an individual participant data meta-analysis. BMC Med Res Methodol. 2014;14:3. \n                    https:\/\/doi.org\/10.1186\/1471-2288-14-3\n                    \n                  .","journal-title":"BMC Med Res Methodol"},{"issue":"10","key":"839_CR20","doi-asserted-by":"publisher","first-page":"e1001886","DOI":"10.1371\/journal.pmed.1001886","volume":"12","author":"TPA Debray","year":"2015","unstructured":"Debray TPA, Riley RD, Rovers MM, Reitsma JB, Moons KGM, Cochrane IPD Meta-analysis Methods group. Individual participant data (IPD) meta-analyses of diagnostic and prognostic modeling studies: guidance on their use. PLoS Med. 2015;12(10):e1001886. \n                    https:\/\/doi.org\/10.1371\/journal.pmed.1001886\n                    \n                  .","journal-title":"PLoS Med"},{"key":"839_CR21","doi-asserted-by":"publisher","first-page":"2676","DOI":"10.12688\/f1000research.8231.3","volume":"5","author":"S Poelsterl","year":"2016","unstructured":"Poelsterl S, Gupta P, Wang L, Conjeti S, Katouzian A, Navab N. Heterogeneous ensembles for predicting survival of metastatic, castrate-resistant prostate cancer patients. F1000Res. 2016;5:2676. \n                    https:\/\/doi.org\/10.12688\/f1000research.8231.3\n                    \n                  .","journal-title":"F1000Res"}],"container-title":["BMC Medical Research Methodology"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12874-019-0839-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s12874-019-0839-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12874-019-0839-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,10,13]],"date-time":"2020-10-13T23:04:57Z","timestamp":1602630297000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedresmethodol.biomedcentral.com\/articles\/10.1186\/s12874-019-0839-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,15]]},"references-count":21,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,12]]}},"alternative-id":["839"],"URL":"https:\/\/doi.org\/10.1186\/s12874-019-0839-0","relation":{},"ISSN":["1471-2288"],"issn-type":[{"value":"1471-2288","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,10,15]]},"assertion":[{"value":"14 November 2018","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 September 2019","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 October 2019","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Collection of all data and materials used in this report were approved by University of Texas Health Science Center of San Antonio Institutional Review Board (IRB), Memorial Sloan Kettering Cancer Center IRB, Mayo Clinic IRB, University of California San Francisco Human Research Protection Program Committee on Human Research, Cleveland Clinic IRB and Human Research Protections, Sunnybrook Health Sciences Centre Research Ethics Board, VA Caribbean Healthcare System IRB, VA Durham Subcommittee on Research Safety, San Raffaele Hospital IRB, and University Hospital Zurich IRB. Patient consent was waived as only retrospective de-identified data were used.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"AJV has a patent on a panel of markers to detect prostate cancer and receives royalties from the sale of the test; MWK is a consultant for Stratify Genomics. All other authors declare that they have no financial conflict of interest.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"191"}}