{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T22:18:11Z","timestamp":1779401891654,"version":"3.53.1"},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,10,23]],"date-time":"2021-10-23T00:00:00Z","timestamp":1634947200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,10,23]],"date-time":"2021-10-23T00:00:00Z","timestamp":1634947200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100002954","name":"Universit\u00e0 degli Studi di Milano - Bicocca","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100002954","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Health Inf Sci Syst"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Purpose<\/jats:title>\n                <jats:p>The rRT-PCR for COVID-19 diagnosis is affected by long turnaround time, potential shortage of reagents, high false-negative rates and high costs. Routine hematochemical tests are a faster and less expensive alternative for diagnosis. Thus, Machine Learning (ML) has been applied to hematological parameters to develop diagnostic tools and help clinicians in promptly managing positive patients. However, few ML models have been externally validated, making their real-world applicability unclear.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>We externally validate 6 state-of-the-art diagnostic ML models, based on Complete Blood Count (CBC) and trained on a dataset encompassing 816 COVID-19 positive cases. The external validation was performed based on two datasets, collected at two different hospitals in northern Italy and encompassing 163 and 104 COVID-19 positive cases, in terms of both error rate and calibration.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results and Conclusion<\/jats:title>\n                <jats:p>We report an average AUC of 95% and average Brier score of 0.11, out-performing existing ML methods, and showing good cross-site transportability. The best performing model (SVM) reported an average AUC of 97.5% (Sensitivity: 87.5%, Specificity: 94%), comparable with the performance of RT-PCR, and was also the best calibrated. The validated models can be useful in the early identification of potential COVID-19 patients, due to the rapid availability of CBC exams, and in multiple test settings.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1007\/s13755-021-00167-3","type":"journal-article","created":{"date-parts":[[2021,10,23]],"date-time":"2021-10-23T15:02:40Z","timestamp":1635001360000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":27,"title":["External validation of Machine Learning models for COVID-19 detection based on Complete Blood Count"],"prefix":"10.1007","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0027-5157","authenticated-orcid":false,"given":"Andrea","family":"Campagner","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anna","family":"Carobene","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Federico","family":"Cabitza","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,10,23]]},"reference":[{"issue":"14","key":"167_CR1","doi-asserted-by":"publisher","first-page":"1377","DOI":"10.1001\/jama.2017.12126","volume":"318","author":"AC Alba","year":"2017","unstructured":"Alba AC, Agoritsas T, Walsh M, Hanna S, Iorio A, Devereaux P, McGinn T, Guyatt G. Discrimination and calibration of clinical prediction models: users guides to the medical literature. Jama. 2017;318(14):1377\u201384.","journal-title":"Jama."},{"issue":"21","key":"167_CR2","doi-asserted-by":"publisher","first-page":"3589","DOI":"10.1002\/sim.2672","volume":"25","author":"D Ashby","year":"2006","unstructured":"Ashby D. Bayesian statistics in medicine: a 25 year review. Stat Med. 2006;25(21):3589\u2013631.","journal-title":"Stat. Med."},{"issue":"4","key":"167_CR3","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1001\/jama.2019.20866","volume":"323","author":"AL Beam","year":"2020","unstructured":"Beam AL, Manrai AK, Ghassemi M. Challenges to the reproducibility of machine learning models in health care. Jama. 2020;323(4):305\u20136.","journal-title":"Jama."},{"issue":"1","key":"167_CR4","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1016\/j.ajic.2020.07.011","volume":"49","author":"B B\u00f6ger","year":"2021","unstructured":"B\u00f6ger B, Fachi MM, Vilhena RO, Cobre AF, Tonin FS, Pontarolo R. Systematic review with meta-analysis of the accuracy of diagnostic tests for Covid-19. Am J Infect Control. 2021;49(1):21\u20139.","journal-title":"Am J Infect Control."},{"key":"167_CR5","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1007\/s10916-020-01597-4","volume":"44","author":"D Brinati","year":"2020","unstructured":"Brinati D, Campagner A, Ferrari D, Locatelli M, Banfi G, Cabitza F. Detection of COVID-19 infection from routine blood exams with machine learning: a feasibility study. J Med Syst. 2020;44:135. https:\/\/doi.org\/10.1007\/s10916-020-01597-4.","journal-title":"J Med Syst."},{"key":"167_CR6","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1016\/j.cca.2017.07.009","volume":"473","author":"S Buoro","year":"2017","unstructured":"Buoro S, Carobene A, Seghezzi M, Manenti B, Pacioni A, Ceriotti F, Ottomano C, Lippi G. Short-and medium-term biological variation estimates of leukocytes extended to differential count and morphology-structural parameters (cell population data) in blood samples obtained from healthy people. Clin Chim Acta. 2017;473:147\u201356.","journal-title":"Clin Chim Acta."},{"key":"167_CR7","doi-asserted-by":"publisher","unstructured":"Cabitza F, Campagner A (2021) The need to separate the wheat from the chaff in medical informatics: Introducing a comprehensive checklist for the (self)-assessment of medical AI studies. Int J Med Inform. p 104510, https:\/\/doi.org\/10.1016\/j.ijmedinf.2021.104510","DOI":"10.1016\/j.ijmedinf.2021.104510"},{"key":"167_CR8","doi-asserted-by":"publisher","first-page":"8","DOI":"10.21037\/atm.2019.04.07","volume":"7","author":"F Cabitza","year":"2019","unstructured":"Cabitza F, Zeitoun JD. The proof of the pudding: in praise of a culture of real-world validation for medical artificial intelligence. Ann Transl Med. 2019;7:8.","journal-title":"Ann Transl Med."},{"issue":"2","key":"167_CR9","doi-asserted-by":"publisher","first-page":"421","DOI":"10.1515\/cclm-2020-1294","volume":"59","author":"F Cabitza","year":"2021","unstructured":"Cabitza F, Campagner A, Ferrari D, Di Resta C, Ceriotti D, Sabetta E, Colombini A, De Vecchi E, Banfi G, Locatelli M, et al. Development, evaluation, and validation of machine learning models for Covid-19 detection based on routine blood tests. Clin Chem Lab Med (CCLM). 2021;59(2):421\u201331.","journal-title":"Clin Chem Lab Med. (CCLM)"},{"issue":"2","key":"167_CR10","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1177\/0004867419885448","volume":"54","author":"M Cearns","year":"2020","unstructured":"Cearns M, Hahn T, Clark S, Baune B. Machine learning probability calibration for high-risk clinical decision-making. Aust N Z J Psychiatry. 2020;54(2):123.","journal-title":"Aust N Z J Psychiatry"},{"key":"167_CR11","doi-asserted-by":"publisher","first-page":"109272","DOI":"10.1016\/j.ejrad.2020.109272","volume":"132","author":"A Cozzi","year":"2020","unstructured":"Cozzi A, Schiaffino S, Arpaia F, Della Pepa G, Tritella S, Bertolotti P, Menicagli L, Monaco CG, Carbonaro LA, Spairani R, et al. Chest x-ray in the Covid-19 pandemic: radiologists real-world reader performance. Eur J Radiol. 2020;132:109272.","journal-title":"Eur J Radiol"},{"key":"167_CR12","doi-asserted-by":"publisher","unstructured":"Dinnes J, Deeks JJ, Berhane S, Taylor M, Adriano A, Davenport C, Dittrich S, Emperador D, Takwoingi Y, Cunningham J, Beese S, Domen J, Dretzke J, Ferrante di Ruffano L, Harris IM, Price MJ, Taylor-Phillips S, Hooft L, Leeflang MMG, McInnes MDF, Spijker R, Van den Bruel A. Rapid, point\u2010of\u2010care antigen and molecular\u2010based tests for diagnosis of SARS\u2010CoV\u20102 infection. Cochrane Database Syst Rev. 2021. https:\/\/doi.org\/10.1002\/14651858.CD013705.pub2.","DOI":"10.1002\/14651858.CD013705.pub2"},{"issue":"7","key":"167_CR13","doi-asserted-by":"publisher","first-page":"1095","DOI":"10.1515\/cclm-2020-0398","volume":"58","author":"D Ferrari","year":"2020","unstructured":"Ferrari D, Motta A, Strollo M, Banfi G, Locatelli M. Routine blood tests as a potential diagnostic tool for COVID-19. Clin Chem Lab Med (CCLM). 2020;58(7):1095\u20139.","journal-title":"Clin Chem Lab Med. (CCLM)"},{"issue":"16","key":"167_CR14","doi-asserted-by":"publisher","first-page":"1574","DOI":"10.1001\/jama.2020.5394","volume":"323","author":"G Grasselli","year":"2020","unstructured":"Grasselli G, Zangrillo A, Zanella A, Antonelli M, Cabrini L, Castelli A, Cereda D, Coluccello A, Foti G, Fumagalli R, et al. Baseline characteristics and outcomes of 1591 patients infected with Sars-Cov-2 admitted to ICUS of the Lombardy region, italy. Jama. 2020;323(16):1574\u201381.","journal-title":"Jama."},{"issue":"1\u20133","key":"167_CR15","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1023\/A:1012487302797","volume":"46","author":"I Guyon","year":"2002","unstructured":"Guyon I, Weston J, Barnhill S, Vapnik V. Gene selection for cancer classification using support vector machines. Mach Learn. 2002;46(1\u20133):389\u2013422.","journal-title":"Mach Learn."},{"issue":"4","key":"167_CR16","doi-asserted-by":"publisher","first-page":"349","DOI":"10.1515\/dx-2020-0058","volume":"7","author":"TA Harahwa","year":"2020","unstructured":"Harahwa TA, Yau THL, Lim-Cooke MS, Al-Haddi S, Zeinah M, Harky A. The optimal diagnostic methods for Covid-19. Diagnosis. 2020;7(4):349\u201356.","journal-title":"Diagnosis."},{"issue":"12","key":"167_CR17","doi-asserted-by":"publisher","first-page":"2011","DOI":"10.1093\/jamia\/ocaa088","volume":"27","author":"T Hernandez-Boussard","year":"2020","unstructured":"Hernandez-Boussard T, Bozkurt S, Ioannidis JP, Shah NH. Minimar (minimum information for medical AI reporting): developing reporting standards for artificial intelligence in health care. J Am Med Inform Assoc. 2020;27(12):2011\u20135.","journal-title":"J Am Med Inform Assoc."},{"issue":"5","key":"167_CR18","doi-asserted-by":"publisher","first-page":"2819","DOI":"10.1007\/s00330-020-07347-x","volume":"31","author":"A Kov\u00e1cs","year":"2020","unstructured":"Kov\u00e1cs A, Pal\u00e1sti P, Ver\u00e9b D, Bozsik B, Palk\u00f3 A, Kincses ZT. The sensitivity and specificity of chest CT in the diagnosis of Covid-19. Eur Radiol. 2020;31(5):2819\u201324.","journal-title":"Eur Radiol."},{"key":"167_CR19","doi-asserted-by":"crossref","unstructured":"Kukar M, Gun\u010dar G, Vovko T, Podnar S, \u010cernel\u010d P, Brvar M, Zalaznik M, Notar M, Mo\u0161kon S, Notar M (2020) Covid-19 diagnosis by routine blood tests using machine learning. arXiv preprint arXiv:200603476","DOI":"10.1038\/s41598-021-90265-9"},{"key":"167_CR20","doi-asserted-by":"publisher","DOI":"10.1090\/mbk\/107","volume-title":"Markov chains and mixing times","author":"DA Levin","year":"2017","unstructured":"Levin DA, Peres Y (2017) Markov chains and mixing times, vol 107. American Mathematical Soc., Providence.","edition":"107"},{"key":"167_CR21","first-page":"4765","volume-title":"Advances in neural information processing systems","author":"SM Lundberg","year":"2017","unstructured":"Lundberg SM, Lee SI. A unified approach to interpreting model predictions. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R, editors. Advances in neural information processing systems, vol. 30. Red Hook: Curran Associates, Inc.; 2017. p. 4765\u201374."},{"issue":"11","key":"167_CR22","doi-asserted-by":"publisher","first-page":"1365","DOI":"10.1093\/clinchem\/hvaa212","volume":"66","author":"CR McCudden","year":"2020","unstructured":"McCudden CR. Deus Ex machina? Predicting SARS-CoV-2 infection from lab tests using machine learning. Clin Chem. 2020;66(11):1365\u20136. https:\/\/doi.org\/10.1093\/clinchem\/hvaa212.","journal-title":"Clin Chem."},{"key":"167_CR23","doi-asserted-by":"crossref","unstructured":"Nevin L. Advancing the beneficial use of machine learning in health care and medicine: Toward a community understanding. PLoS Med. 2018;15(11):e1002708.","DOI":"10.1371\/journal.pmed.1002708"},{"issue":"10","key":"167_CR24","doi-asserted-by":"publisher","first-page":"1645","DOI":"10.1515\/cclm-2020-0748","volume":"58","author":"AA Onyeaghala","year":"2020","unstructured":"Onyeaghala AA, Olajide I. Managing COVID-19 outbreak in Nigeria: matters arising. Clin Chem Lab Med (CCLM). 2020;58(10):1645\u201350.","journal-title":"Clin Chem Lab Med. (CCLM)"},{"key":"167_CR25","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E. Scikit-learn: machine learning in python. J Mach Learn Res. 2011;12:2825\u201330.","journal-title":"J Mach Learn Res"},{"issue":"12","key":"167_CR26","doi-asserted-by":"publisher","first-page":"e24048","DOI":"10.2196\/24048","volume":"22","author":"TB Plante","year":"2020","unstructured":"Plante TB, Blau AM, Berg AN, Weinberg AS, Jun IC, Tapson VF, Kanigan TS, Adib AB. Development and external validation of a machine learning tool to rule out Covid-19 among adults in the emergency department using routine blood tests: A large, multicenter, real-world study. J Med Internet Res. 2020;22(12):e24048.","journal-title":"J Med Internet Res."},{"key":"167_CR27","doi-asserted-by":"publisher","DOI":"10.1016\/j.chaos.2020.110337","author":"J Rasheed","year":"2020","unstructured":"Rasheed J, Jamil A, Hameed AA, Aftab U, Aftab J, Shah SA, Draheim D (2020) A survey on artificial intelligence approaches in supporting frontline workers and decision makers for covid-19 pandemic. Chaos, Solitons & Fractals https:\/\/doi.org\/10.1016\/j.chaos.2020.110337","journal-title":"Chaos, Solitons & Fractals"},{"key":"167_CR28","doi-asserted-by":"crossref","unstructured":"Riley RD, Debray TP, Collins GS, Archer L, Ensor J, van Smeden M, Snell KI. Minimum sample size for external validation of a clinical prediction model with a binary outcome. Statistics. 2021 (in Medicine).","DOI":"10.1002\/sim.9025"},{"issue":"3","key":"167_CR29","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1038\/s42256-021-00307-0","volume":"3","author":"M Roberts","year":"2021","unstructured":"Roberts M, Driggs D, Thorpe M, Gilbey J, Yeung M, Ursprung S, Aviles-Rivero AI, Etmann C, McCague C, Beer L, et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for Covid-19 using chest radiographs and CT scans. Nat Mach Intell. 2021;3(3):199\u2013217.","journal-title":"Nat Mach Intell."},{"issue":"4","key":"167_CR30","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1016\/j.ipm.2009.03.002","volume":"45","author":"M Sokolova","year":"2009","unstructured":"Sokolova M, Lapalme G. A systematic analysis of performance measures for classification tasks. Inform Process Manage. 2009;45(4):427\u201337.","journal-title":"Inform Process Manage."},{"issue":"2","key":"167_CR31","doi-asserted-by":"publisher","first-page":"e78","DOI":"10.1016\/S2589-7500(20)30274-0","volume":"3","author":"AA Soltan","year":"2020","unstructured":"Soltan AA, Kouchaki S, Zhu T, Kiyasseh D, Taylor T, Hussain ZB, Peto T, Brent AJ, Eyre DW, Clifton DA. Rapid triage for covid-19 using routine clinical data for patients attending hospital: development and prospective validation of an artificial intelligence screening test. The Lancet Digital Health. 2020;3(2),e78-e87","journal-title":"Lancet Digital Health."},{"issue":"1","key":"167_CR32","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1186\/1471-2105-7-91","volume":"7","author":"S Varma","year":"2006","unstructured":"Varma S, Simon R. Bias in error estimation when using cross-validation for model selection. BMC Bioinformat. 2006;7(1):91.","journal-title":"BMC Bioinformat."},{"key":"167_CR33","doi-asserted-by":"crossref","unstructured":"Wang Y, Hou H, Wang W, Wang W. Combination of CT and RT-PCR in the screening or diagnosis of Covid-19. J Global Health. 2020;10(1).","DOI":"10.7189\/jogh.10.010347"},{"key":"167_CR34","doi-asserted-by":"publisher","unstructured":"Watson J, Whiting PF, Brush JE. Interpreting a covid-19 test result. BMJ. 2020;369:m1808. https:\/\/doi.org\/10.1136\/bmj.m1808.","DOI":"10.1136\/bmj.m1808"},{"key":"167_CR35","doi-asserted-by":"publisher","unstructured":"Wu J, Zhang P, Zhang L, Meng W, Li J, Tong C, Li Y, Cai J, Yang Z, Zhu J, Zhao M, Huang H, Xie X, Li S. Rapid and accurate identification of covid-19 infection through machine learning based on clinical available blood test results. medRxiv. 2020. https:\/\/doi.org\/10.1101\/2020.04.02.20051136","DOI":"10.1101\/2020.04.02.20051136"},{"key":"167_CR36","doi-asserted-by":"publisher","unstructured":"Wynants L, Van Calster B, Collins G S, Riley R D, Heinze G, Schuit E, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal BMJ. 2020;369:m1328. https:\/\/doi.org\/10.1136\/bmj.m1328.","DOI":"10.1136\/bmj.m1328"},{"issue":"11","key":"167_CR37","doi-asserted-by":"publisher","first-page":"1396","DOI":"10.1093\/clinchem\/hvaa200","volume":"66","author":"HS Yang","year":"2020","unstructured":"Yang HS, Hou Y, Vasovic LV, Steel PA, Chadburn A, Racine-Brzostek SE, Velu P, Cushing MM, Loda M, Kaushal R, et al. Routine laboratory blood tests predict Sars-Cov-2 infection using machine learning. Clin Chem. 2020;66(11):1396\u2013404.","journal-title":"Clin Chem."},{"key":"167_CR38","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1016\/j.cca.2020.04.018","volume":"507","author":"H Yun","year":"2020","unstructured":"Yun H, Sun Z, Wu J, Tang A, Hu M, Xiang Z. Laboratory data analysis of novel coronavirus (Covid-19) screening in 2510 patients. Clin Chim Acta. 2020;507:94\u20137.","journal-title":"Clin Chim Acta"}],"container-title":["Health Information Science and Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13755-021-00167-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s13755-021-00167-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13755-021-00167-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,12,3]],"date-time":"2021-12-03T14:27:33Z","timestamp":1638541653000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s13755-021-00167-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,23]]},"references-count":38,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["167"],"URL":"https:\/\/doi.org\/10.1007\/s13755-021-00167-3","relation":{},"ISSN":["2047-2501"],"issn-type":[{"value":"2047-2501","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,23]]},"assertion":[{"value":"8 April 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 September 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 October 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors report no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Research involving human subjects complied with all relevant national regulations, institutional policies and is in accordance with the tenets of the Helsinki Declaration (as revised in 2013), and was approved by the authors\u2019 Institutional Review Board (70\/INT\/2020).","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"Individuals signed an informed consent authorizing the use of their anonymously collected data for retrospective observational studies (article 9.2.j; EU general data protection regulation 2016\/679 [GDPR]), according to the IRCCS San Raffaele Hospital policy (IOG075\/2016), and the appropriate institutional forms have been archived.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}}],"article-number":"37"}}