{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T13:11:06Z","timestamp":1780492266461,"version":"3.54.1"},"reference-count":18,"publisher":"Wiley","issue":"25","license":[{"start":{"date-parts":[[2024,9,5]],"date-time":"2024-09-05T00:00:00Z","timestamp":1725494400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000276","name":"Department of Health and Social Care","doi-asserted-by":"publisher","award":["2020\/045"],"award-info":[{"award-number":["2020\/045"]}],"id":[{"id":"10.13039\/501100000276","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100012100","name":"Royal Statistical Society","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100012100","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100012338","name":"Alan Turing Institute","doi-asserted-by":"publisher","award":["EP\/W037211\/1"],"award-info":[{"award-number":["EP\/W037211\/1"]}],"id":[{"id":"10.13039\/100012338","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100013373","name":"NIHR Oxford Biomedical Research Centre","doi-asserted-by":"publisher","award":["NIHR203311"],"award-info":[{"award-number":["NIHR203311"]}],"id":[{"id":"10.13039\/501100013373","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Statistics in Medicine"],"published-print":{"date-parts":[[2024,11,10]]},"abstract":"<jats:title>ABSTRACT<\/jats:title><jats:p>From early in the coronavirus disease 2019 (COVID\u201019) pandemic, there was interest in using machine learning methods to predict COVID\u201019 infection status based on vocal audio signals, for example, cough recordings. However, early studies had limitations in terms of data collection and of how the performances of the proposed predictive models were assessed. This article describes how these limitations have been overcome in a study carried out by the Turing\u2010RSS Health Data Laboratory and the UK Health Security Agency. As part of the study, the UK Health Security Agency collected a dataset of acoustic recordings, SARS\u2010CoV\u20102 infection status and extensive study participant meta\u2010data. This allowed us to rigorously assess state\u2010of\u2010the\u2010art machine learning techniques to predict SARS\u2010CoV\u20102 infection status based on vocal audio signals. The lessons learned from this project should inform future studies on statistical evaluation methods to assess the performance of machine learning techniques for public health tasks.<\/jats:p>","DOI":"10.1002\/sim.10211","type":"journal-article","created":{"date-parts":[[2024,9,5]],"date-time":"2024-09-05T23:50:35Z","timestamp":1725580235000},"page":"4861-4871","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Assessing the Performance of Machine Learning Methods Trained on Public Health Observational Data: A Case Study From COVID\u201019"],"prefix":"10.1002","volume":"43","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4591-4167","authenticated-orcid":false,"given":"Davide","family":"Pigoli","sequence":"first","affiliation":[{"name":"Department of Mathematics King's College London  UK"},{"name":"The Alan Turing Institute  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kieran","family":"Baker","sequence":"additional","affiliation":[{"name":"Department of Mathematics King's College London  UK"},{"name":"The Alan Turing Institute  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jobie","family":"Budd","sequence":"additional","affiliation":[{"name":"Division of Medicine University College London  UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lorraine","family":"Butler","sequence":"additional","affiliation":[{"name":"UK Health Security Agency  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Harry","family":"Coppock","sequence":"additional","affiliation":[{"name":"The Alan Turing Institute  London UK"},{"name":"Group on Language Audio &amp; Music Imperial College London  UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sabrina","family":"Egglestone","sequence":"additional","affiliation":[{"name":"UK Health Security Agency  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Steven G.","family":"Gilmour","sequence":"additional","affiliation":[{"name":"Department of Mathematics King's College London  UK"},{"name":"The Alan Turing Institute  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6667-4943","authenticated-orcid":false,"given":"Chris","family":"Holmes","sequence":"additional","affiliation":[{"name":"The Alan Turing Institute  London UK"},{"name":"Department of Statistics University of Oxford  UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Hurley","sequence":"additional","affiliation":[{"name":"UK Health Security Agency  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Radka","family":"Jersakova","sequence":"additional","affiliation":[{"name":"The Alan Turing Institute  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ivan","family":"Kiskin","sequence":"additional","affiliation":[{"name":"Centre for Vision, Speech and Signal Processing University of Surrey  UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vasiliki","family":"Koutra","sequence":"additional","affiliation":[{"name":"Department of Mathematics King's College London  UK"},{"name":"The Alan Turing Institute  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathon","family":"Mellor","sequence":"additional","affiliation":[{"name":"UK Health Security Agency  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"George","family":"Nicholson","sequence":"additional","affiliation":[{"name":"The Alan Turing Institute  London UK"},{"name":"Department of Statistics University of Oxford  UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joe","family":"Packham","sequence":"additional","affiliation":[{"name":"UK Health Security Agency  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Selina","family":"Patel","sequence":"additional","affiliation":[{"name":"Division of Medicine University College London  UK"},{"name":"UK Health Security Agency  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richard","family":"Payne","sequence":"additional","affiliation":[{"name":"UK Health Security Agency  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephen J.","family":"Roberts","sequence":"additional","affiliation":[{"name":"Department of Engineering Science University of Oxford  UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6478-8699","authenticated-orcid":false,"given":"Bj\u00f6rn W.","family":"Schuller","sequence":"additional","affiliation":[{"name":"The Alan Turing Institute  London UK"},{"name":"Group on Language Audio &amp; Music Imperial College London  UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ana","family":"Tendero\u2010Ca\u00f1adas","sequence":"additional","affiliation":[{"name":"UK Health Security Agency  London UK"},{"name":"Centre for Lifelong Health University of Brighton  UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9276-052X","authenticated-orcid":false,"given":"Tracey","family":"Thornley","sequence":"additional","affiliation":[{"name":"Pharmacy Practice and Policy Division University of Nottingham  UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexander","family":"Titcomb","sequence":"additional","affiliation":[{"name":"UK Health Security Agency  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2024,9,5]]},"reference":[{"key":"e_1_2_12_2_1","unstructured":"World Health Organization \u201cWHO Director\u2010General's Opening Remarks at the Media Briefing on COVID\u201019\u201316 March 2020 \u201daccessed September 15 2022 https:\/\/www.who.int\/director\u2010general\/speeches\/detail\/who\u2010director\u2010general\u2010s\u2010opening\u2010remarks\u2010at\u2010the\u2010media\u2010briefing\u2010on\u2010covid\u201019\u201016\u2010march\u20102020."},{"key":"e_1_2_12_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/OJEMB.2020.3026928"},{"key":"e_1_2_12_4_1","doi-asserted-by":"crossref","unstructured":"J.Han C.Brown J.Chauhan et al. \u201cExploring Automatic COVID\u201019 Diagnosis via Voice and Symptoms From Crowdsourced Data \u201d inICASSP 2021\u20102021 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP)(IEEE 2021) 8328\u20138332.","DOI":"10.1109\/ICASSP39728.2021.9414576"},{"key":"e_1_2_12_5_1","doi-asserted-by":"crossref","unstructured":"C.Brown J.Chauhan A.Grammenos et al. \u201cExploring Automatic Diagnosis of COVID\u201019 From Crowdsourced Respiratory Sound Data\u201d (2020). arXiv preprint arXiv:2006.05919.","DOI":"10.1145\/3394486.3412865"},{"key":"e_1_2_12_6_1","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-023-00773-8"},{"key":"e_1_2_12_7_1","article-title":"Machine Learning and Artificial Intelligence Research for Patient Benefit: 20 Critical Questions on Transparency, Replicability, Ethics, and Effectiveness","volume":"368","author":"Vollmer S.","year":"2020","journal-title":"British Medical Journal"},{"key":"e_1_2_12_8_1","doi-asserted-by":"publisher","DOI":"10.1126\/scitranslmed.abb1655"},{"key":"e_1_2_12_9_1","doi-asserted-by":"publisher","DOI":"10.7326\/M14-0697"},{"key":"e_1_2_12_10_1","doi-asserted-by":"crossref","unstructured":"A.Apicella F.Isgr\u00f2 andR.Prevete \u201cDon't Push the Button! Exploring Data Leakage Risks in Machine Learning and Transfer Learning\u201d (2024). arXiv preprint arXiv:2401.13796.","DOI":"10.2139\/ssrn.4733889"},{"key":"e_1_2_12_11_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0118432"},{"key":"e_1_2_12_12_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-021-00553-x"},{"key":"e_1_2_12_13_1","unstructured":"\u201cUK Health Security Agency. Speak up and Help Beat Coronavirus \u201d2021 https:\/\/www.gov.uk\/government\/news\/speak\u2010up\u2010and\u2010help\u2010beat\u2010coronavirus\u2010covid\u201019."},{"key":"e_1_2_12_14_1","unstructured":"J.Budd K.Baker E.Karoune et al. \u201cA Large\u2010Scale and PCR\u2010Referenced Vocal Audio Dataset for COVID\u201019\u201d (2022) arXiv preprint."},{"key":"e_1_2_12_15_1","first-page":"20","article-title":"Real\u2010Time Assessment of Community Transmission (REACT) of SARS\u2010CoV\u20102 Virus: Study Protocol","author":"Riley S.","year":"2020","journal-title":"Wellcome Open Research"},{"key":"e_1_2_12_16_1","doi-asserted-by":"crossref","unstructured":"F.Eyben F.Weninger F.Gross andB.Schuller \u201cRecent Developments in Opensmile the Munich Open\u2010Source Multimedia Feature Extractor \u201d inProceedings of the 21st ACM International Conference on Multimedia(Newyork NY: Association for Computing Machinery 2013) 835\u2013838.","DOI":"10.1145\/2502081.2502224"},{"key":"e_1_2_12_17_1","volume-title":"2010 IEEE 18th Signal Processing and Communications Applications Conference","author":"Schuller B.","year":"2009"},{"key":"e_1_2_12_18_1","doi-asserted-by":"crossref","unstructured":"B. E.Boser I. M.Guyon andV. N.Vapnik \u201cA Training Algorithm for Optimal Margin Classifiers \u201d inProceedings of the Fifth Annual Workshop on Computational Learning Theory(Association for Computing Machinery 1992) 144\u2013152.","DOI":"10.1145\/130385.130401"},{"key":"e_1_2_12_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2019.10.118"}],"container-title":["Statistics in Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/sim.10211","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,17]],"date-time":"2024-10-17T10:37:12Z","timestamp":1729161432000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/sim.10211"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,5]]},"references-count":18,"journal-issue":{"issue":"25","published-print":{"date-parts":[[2024,11,10]]}},"alternative-id":["10.1002\/sim.10211"],"URL":"https:\/\/doi.org\/10.1002\/sim.10211","archive":["Portico"],"relation":{},"ISSN":["0277-6715","1097-0258"],"issn-type":[{"value":"0277-6715","type":"print"},{"value":"1097-0258","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,5]]},"assertion":[{"value":"2023-10-04","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-08-15","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-09-05","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}