{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T12:07:30Z","timestamp":1787314050948,"version":"build-2736575974"},"reference-count":37,"publisher":"Public Library of Science (PLoS)","issue":"1","license":[{"start":{"date-parts":[[2023,1,12]],"date-time":"2023-01-12T00:00:00Z","timestamp":1673481600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000292","name":"Cystic Fibrosis Trust","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000292","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000897","name":"Cystic Fibrosis Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000897","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000292","name":"Cystic Fibrosis Trust","doi-asserted-by":"publisher","award":["DHRP016"],"award-info":[{"award-number":["DHRP016"]}],"id":[{"id":"10.13039\/501100000292","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosdigitalhealth.org"],"crossmark-restriction":false},"short-container-title":["PLOS Digit Health"],"abstract":"<jats:p>Precise and timely referral for lung transplantation is critical for the survival of cystic fibrosis patients with terminal illness. While machine learning (ML) models have been shown to achieve significant improvement in prognostic accuracy over current referral guidelines, the external validity of these models and their resulting referral policies has not been fully investigated. Here, we studied the external validity of machine learning-based prognostic models using annual follow-up data from the UK and Canadian Cystic Fibrosis Registries. Using a state-of-the-art automated ML framework, we derived a model for predicting poor clinical outcomes in patients enrolled in the UK registry, and conducted external validation of the derived model using the Canadian Cystic Fibrosis Registry. In particular, we studied the effect of (1) natural variations in patient characteristics across populations and (2) differences in clinical practice on the external validity of ML-based prognostic scores. Overall, decrease in prognostic accuracy on the external validation set (AUCROC: 0.88, 95% CI 0.88-0.88) was observed compared to the internal validation accuracy (AUCROC: 0.91, 95% CI 0.90-0.92). Based on our ML model, analysis on feature contributions and risk strata revealed that, while external validation of ML models exhibited high precision on average, both factors (1) and (2) can undermine the external validity of ML models in patient subgroups with moderate risk for poor outcomes. A significant boost in prognostic power (F1 score) from 0.33 (95% CI 0.31-0.35) to 0.45 (95% CI 0.45-0.45) was observed in external validation when variations in these subgroups were accounted in our model. Our study highlighted the significance of external validation of ML models for cystic fibrosis prognostication. The uncovered insights on key risk factors and patient subgroups can be used to guide the cross-population adaptation of ML-based models and inspire new research on applying transfer learning methods for fine-tuning ML models to cope with regional variations in clinical care.<\/jats:p>","DOI":"10.1371\/journal.pdig.0000179","type":"journal-article","created":{"date-parts":[[2023,1,12]],"date-time":"2023-01-12T13:30:44Z","timestamp":1673530244000},"page":"e0000179","update-policy":"https:\/\/doi.org\/10.1371\/journal.pdig.corrections_policy","source":"Crossref","is-referenced-by-count":5,"title":["External validity of machine learning-based prognostic scores for cystic fibrosis: A retrospective study using the UK and Canadian registries"],"prefix":"10.1371","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0396-5142","authenticated-orcid":true,"given":"Yuchao","family":"Qin","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ahmed","family":"Alaa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andres","family":"Floto","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mihaela van der","family":"Schaar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2023,1,12]]},"reference":[{"issue":"3","key":"pdig.0000179.ref001","doi-asserted-by":"crossref","first-page":"344","DOI":"10.1016\/j.jcf.2020.02.015","article-title":"Cystic Fibrosis Foundation consensus guidelines for the care of individuals with advanced cystic fibrosis lung disease","volume":"19","author":"SG Kapnadak","year":"2020","journal-title":"Journal of Cystic Fibrosis"},{"issue":"3","key":"pdig.0000179.ref002","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1038\/s44161-022-00030-y","article-title":"Identifying and preventing cardiovascular disease in patients with cystic fibrosis","volume":"1","author":"T Saunders","year":"2022","journal-title":"Nature Cardiovascular Research"},{"issue":"6","key":"pdig.0000179.ref003","doi-asserted-by":"crossref","first-page":"553","DOI":"10.1016\/j.healun.2020.02.010","article-title":"Lung transplantation for cystic fibrosis","volume":"39","author":"JC Yeung","year":"2020","journal-title":"The Journal of Heart and Lung Transplantation"},{"issue":"18","key":"pdig.0000179.ref004","doi-asserted-by":"crossref","first-page":"1187","DOI":"10.1056\/NEJM199204303261804","article-title":"Prediction of mortality in patients with cystic fibrosis","volume":"326","author":"E Kerem","year":"1992","journal-title":"New England Journal of Medicine"},{"issue":"4","key":"pdig.0000179.ref005","doi-asserted-by":"crossref","first-page":"1133","DOI":"10.1109\/JBHI.2016.2574857","article-title":"Discovery and clinical decision support for personalized healthcare","volume":"21","author":"J Yoon","year":"2016","journal-title":"IEEE Journal of Biomedical and Health Informatics"},{"issue":"1","key":"pdig.0000179.ref006","first-page":"1","article-title":"Prognostication and risk factors for cystic fibrosis via automated machine learning","volume":"8","author":"AM Alaa","year":"2018","journal-title":"Scientific Reports"},{"issue":"6","key":"pdig.0000179.ref007","doi-asserted-by":"crossref","first-page":"589","DOI":"10.3390\/genes11060589","article-title":"The changing epidemiology of cystic fibrosis: incidence, survival and impact of the CFTR gene discovery","volume":"11","author":"V Scotet","year":"2020","journal-title":"Genes"},{"key":"pdig.0000179.ref008","doi-asserted-by":"crossref","first-page":"106504","DOI":"10.1016\/j.cmpb.2021.106504","article-title":"Effects of dataset size and interactions on the prediction performance of logistic regression and deep learning models","volume":"213","author":"A Bailly","year":"2022","journal-title":"Computer Methods and Programs in Biomedicine"},{"issue":"8","key":"pdig.0000179.ref009","doi-asserted-by":"crossref","first-page":"716","DOI":"10.1038\/s42256-021-00353-8","article-title":"Machine learning to guide the use of adjuvant therapies for breast cancer","volume":"3","author":"AM Alaa","year":"2021","journal-title":"Nature Machine Intelligence"},{"key":"pdig.0000179.ref010","unstructured":"Alaa A, Schaar M. Autoprognosis: Automated clinical prognostic modeling via bayesian optimization with structured kernel learning. In: International Conference on Machine Learning. Proceedings of Machine Learning Research. 2018:139\u2013148."},{"key":"pdig.0000179.ref011","doi-asserted-by":"crossref","unstructured":"Imrie F, Cebere B, McKinney EF, and Schaar M. Autoprognosis 2.0: Democratizing diagnostic and prognostic modeling in healthcare with automated machine learning. arXiv preprint arXiv:2210.12090. 2022.","DOI":"10.1371\/journal.pdig.0000276"},{"key":"pdig.0000179.ref012","unstructured":"AutoPrognosis\u2014A System for Automating the Design of Predictive Modeling Pipelines Tailored for Clinical Prognosis; 2022. Available from: https:\/\/github.com\/vanderschaarlab\/autoprognosis."},{"issue":"4","key":"pdig.0000179.ref013","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1016\/j.jcf.2013.12.003","article-title":"The relative frequency of CFTR mutation classes in European patients with cystic fibrosis","volume":"13","author":"K De Boeck","year":"2014","journal-title":"Journal of Cystic Fibrosis"},{"key":"pdig.0000179.ref014","unstructured":"UK Cystic Fibrosis Registry 2014 Annual Data Report; 2015. Available from: https:\/\/www.cysticfibrosis.org.uk\/sites\/default\/files\/2020-12\/2014 Registry Annual Data Report final.pdf."},{"key":"pdig.0000179.ref015","unstructured":"The Canadian Cystic Fibrosis Registry 2014 Annual Report; 2016. Available from: https:\/\/www.cysticfibrosis.ca\/registry\/2014AnnualDataReport.pdf."},{"key":"pdig.0000179.ref016","unstructured":"Cardiothoracic Advisory Group (CTAG) on behalf of NHSBT. Donor lung distribution and allocation.; 2017. Available from: http:\/\/odt.nhs.uk\/pdf\/lung_allocation_policy.pdf."},{"issue":"1","key":"pdig.0000179.ref017","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1136\/thoraxjnl-2018-211731","article-title":"Patient outcomes from time of listing for lung transplantation in the UK: are there disease-specific differences?","volume":"74","author":"A Kourliouros","year":"2019","journal-title":"Thorax"},{"issue":"7","key":"pdig.0000179.ref018","doi-asserted-by":"crossref","first-page":"e13870","DOI":"10.1111\/ctr.13870","article-title":"Clinical judgment versus lung allocation score in predicting lung transplant waitlist mortality","volume":"34","author":"A Hirji","year":"2020","journal-title":"Clinical Transplantation"},{"key":"pdig.0000179.ref019","article-title":"Cystic fibrosis patients in Canada live nearly a decade longer than those in US, study finds","volume":"356","author":"M McCarthy","year":"2017","journal-title":"British Medical Journal"},{"issue":"8","key":"pdig.0000179.ref020","doi-asserted-by":"crossref","first-page":"537","DOI":"10.7326\/M16-0858","article-title":"Survival comparison of patients with cystic fibrosis in Canada and the United States: a population-based cohort study","volume":"166","author":"AL Stephenson","year":"2017","journal-title":"Annals of Internal Medicine"},{"key":"pdig.0000179.ref021","unstructured":"International Registry in Organ Donation and Transplantation; 2022. Available from: https:\/\/www.irodat.org\/?p=database."},{"issue":"3","key":"pdig.0000179.ref022","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1016\/j.jcf.2019.03.002","article-title":"Lung transplant referral for individuals with cystic fibrosis: Cystic Fibrosis Foundation consensus guidelines","volume":"18","author":"KJ Ramos","year":"2019","journal-title":"Journal of Cystic Fibrosis"},{"issue":"6","key":"pdig.0000179.ref023","doi-asserted-by":"crossref","first-page":"702","DOI":"10.1016\/j.jcf.2017.03.004","article-title":"A 3-year prognostic score for adults with cystic fibrosis","volume":"16","author":"L Nkam","year":"2017","journal-title":"Journal of Cystic Fibrosis"},{"issue":"4","key":"pdig.0000179.ref024","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1093\/aje\/153.4.345","article-title":"Predictive 5-year survivorship model of cystic fibrosis","volume":"153","author":"TG Liou","year":"2001","journal-title":"American Journal of Epidemiology"},{"issue":"1-2","key":"pdig.0000179.ref025","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1016\/S0167-5877(00)00115-X","article-title":"Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests","volume":"45","author":"M Greiner","year":"2000","journal-title":"Preventive Veterinary Medicine"},{"issue":"9","key":"pdig.0000179.ref026","doi-asserted-by":"crossref","first-page":"1012","DOI":"10.1164\/rccm.201405-0973OC","article-title":"Body composition and mortality after adult lung transplantation in the United States","volume":"190","author":"JP Singer","year":"2014","journal-title":"American Journal of Respiratory and Critical Care Medicine"},{"key":"pdig.0000179.ref027","article-title":"Low body mass index as a barrier to lung transplant in cystic fibrosis","author":"AL Jennerich","year":"2021","journal-title":"Journal of Cystic Fibrosis"},{"key":"pdig.0000179.ref028","doi-asserted-by":"crossref","first-page":"1506","DOI":"10.1016\/j.transproceed.2014.04.004","article-title":"Body mass index in lung transplant candidates: a contra-indication to transplant or not?","volume":"46","author":"D Ruttens","year":"2014","journal-title":"Transplantation Proceedings"},{"issue":"4","key":"pdig.0000179.ref029","doi-asserted-by":"crossref","first-page":"898","DOI":"10.1016\/j.chest.2019.11.043","article-title":"Underweight patients with cystic fibrosis have acceptable survival following lung transplantation: a united network for organ sharing registry study","volume":"157","author":"KJ Ramos","year":"2020","journal-title":"Chest"},{"issue":"11","key":"pdig.0000179.ref030","doi-asserted-by":"crossref","first-page":"1336","DOI":"10.1016\/j.jclinepi.2014.12.010","article-title":"A statistical model to predict one-year risk of death in patients with cystic fibrosis","volume":"68","author":"SD Aaron","year":"2015","journal-title":"Journal of Clinical Epidemiology"},{"issue":"6","key":"pdig.0000179.ref031","doi-asserted-by":"crossref","first-page":"525","DOI":"10.1002\/ppul.20619","article-title":"Predictors of mortality in adults with cystic fibrosis","volume":"42","author":"J Courtney","year":"2007","journal-title":"Pediatric Pulmonology"},{"issue":"3","key":"pdig.0000179.ref032","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1177\/1753465816682677","article-title":"Macrolides: a promising pharmacologic therapy for chronic obstructive pulmonary disease","volume":"11","author":"S Qiu","year":"2017","journal-title":"Therapeutic Advances in Respiratory Disease"},{"issue":"1","key":"pdig.0000179.ref033","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/j.1440-1843.2001.00289.x","article-title":"Allergic bronchopulmonary aspergillosis: new concepts of pathogenesis and treatment","volume":"6","author":"PA Wark","year":"2001","journal-title":"Respirology"},{"issue":"9","key":"pdig.0000179.ref034","doi-asserted-by":"crossref","first-page":"482","DOI":"10.5694\/j.1326-5377.2005.tb06792.x","article-title":"Life-threatening allergic bronchopulmonary aspergillosis in a well child with cystic fibrosis","volume":"182","author":"E Skowronski","year":"2005","journal-title":"The Medical Journal of Australia"},{"issue":"1","key":"pdig.0000179.ref035","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1016\/0002-9343(94)90112-0","article-title":"Osteopenia in adults with cystic fibrosis","volume":"96","author":"LK Bachrach","year":"1994","journal-title":"The American Journal of Medicine"},{"key":"pdig.0000179.ref036","article-title":"A roadmap to the brittle bones of cystic fibrosis","volume":"2011","author":"AP Gore","year":"2010","journal-title":"Journal of Osteoporosis"},{"issue":"1","key":"pdig.0000179.ref037","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1097\/TP.0000000000003664","article-title":"An intention-to-treat view of lung transplantation for interstitial lung disease: successful strategies to minimize waiting list and posttransplant mortality","volume":"106","author":"MZ Zhu","year":"2022","journal-title":"Transplantation"}],"container-title":["PLOS Digital Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0000179","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,4]],"date-time":"2023-12-04T11:35:25Z","timestamp":1701689725000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0000179"}},"subtitle":[],"editor":[{"given":"Mecit Can Emre","family":"Simsekler","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2023,1,12]]},"references-count":37,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,1,12]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pdig.0000179","relation":{},"ISSN":["2767-3170"],"issn-type":[{"value":"2767-3170","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,12]]}}}