{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,11]],"date-time":"2026-03-11T05:17:35Z","timestamp":1773206255520,"version":"3.50.1"},"reference-count":29,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,6,19]],"date-time":"2020-06-19T00:00:00Z","timestamp":1592524800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,6,19]],"date-time":"2020-06-19T00:00:00Z","timestamp":1592524800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Benchmark datasets have a powerful normative influence: by determining how the real world is represented in data, they define which problems will first be solved by algorithms built using the datasets and, by extension, who these algorithms will work <jats:italic>for<\/jats:italic>. It is desirable for these datasets to serve four functions: (1) enabling the creation of clinically relevant algorithms; (2) facilitating like-for-like comparison of algorithmic performance; (3) ensuring reproducibility of algorithms; (4) asserting a normative influence on the clinical domains and diversity of patients that will potentially benefit from technological advances. Without benchmark datasets that satisfy these functions, it is impossible to address two perennial concerns of clinicians experienced in computational research: \u201cthe data scientists just go where the data is rather than where the needs are,\u201d and, \u201cyes, but will this work for my patients?\u201d If algorithms are to be developed and applied for the care of patients, then it is prudent for the research community to create benchmark datasets proactively, across specialties. As yet, best practice in this area has not been defined. Broadly speaking, efforts will include design of the dataset; compliance and contracting issues relating to the sharing of sensitive data; enabling access and reuse; and planning for translation of algorithms to the clinical environment. If a deliberate and systematic approach is not followed, not only will the considerable benefits of clinical algorithms fail to be realized, but the potential harms may be regressively incurred across existing gradients of social inequity.<\/jats:p>","DOI":"10.1038\/s41746-020-0295-6","type":"journal-article","created":{"date-parts":[[2020,6,19]],"date-time":"2020-06-19T10:03:27Z","timestamp":1592561007000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["\u201cYes, but will it work for my patients?\u201d Driving clinically relevant research with benchmark datasets"],"prefix":"10.1038","volume":"3","author":[{"given":"Trishan","family":"Panch","sequence":"first","affiliation":[]},{"given":"Tom J.","family":"Pollard","sequence":"additional","affiliation":[]},{"given":"Heather","family":"Mattie","sequence":"additional","affiliation":[]},{"given":"Emily","family":"Lindemer","sequence":"additional","affiliation":[]},{"given":"Pearse A.","family":"Keane","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6712-6626","authenticated-orcid":false,"given":"Leo Anthony","family":"Celi","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,6,19]]},"reference":[{"key":"295_CR1","unstructured":"Krizhevsky, A., Sutskever, I., & Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Proceedings of Advances in neural information processing systems, 1097\u20131105 (Association for Computing Machinery, 2012)."},{"key":"295_CR2","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248\u2013255 (IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, 2009).","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"295_CR3","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436\u2013444 (2015).","journal-title":"Nature"},{"key":"295_CR4","doi-asserted-by":"publisher","first-page":"1684","DOI":"10.1056\/NEJMsb1616595","volume":"376","author":"BE Bierer","year":"2017","unstructured":"Bierer, B. E., Crosas, M. & Pierce, H. H. Data authorship as an incentive to data sharing. N. Engl. J. Med. 376, 1684\u20131687 (2017).","journal-title":"N. Engl. J. Med."},{"key":"295_CR5","unstructured":"Crawford, K. & Paglen, T. Excavating AI: the politics of training sets for machine learning. https:\/\/excavating.ai (The AI Now Institute, NYU, 2019)."},{"key":"295_CR6","unstructured":"Solly, M. Art project shows racial biases in artificial intelligence system. Smithsonian Mag. https:\/\/www.smithsonianmag.com\/smart-news\/art-project-exposed-racial-biases-artificial-intelligence-system-180973207\/#AgkvdCpeVrC8hqGV.99 (2019)."},{"key":"295_CR7","doi-asserted-by":"crossref","unstructured":"Yang, K., Qinami, K., Fei-Fei, L., Deng, J., & Russakovsky, O. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. https:\/\/arxiv.org\/abs\/1912.07726 (2019).","DOI":"10.1145\/3351095.3375709"},{"key":"295_CR8","doi-asserted-by":"publisher","first-page":"e198","DOI":"10.1016\/S2589-7500(19)30112-8","volume":"1","author":"TJ Pollard","year":"2019","unstructured":"Pollard, T. J. et al. Turning the crank for machine learning: ease, at what expense? Lancet Digit. Health 1, e198\u2013e199 (2019).","journal-title":"Lancet Digit. Health"},{"key":"295_CR9","doi-asserted-by":"publisher","DOI":"10.1016\/S2589-7500(20)30082-0","volume":"2","author":"CV Cosgriv","year":"2020","unstructured":"Cosgriv, C. V., Ebner, D. E. & Celi, L. A. Data sharing in the era of COVID-19. Lancet Digit. Health 2, e224 (2020).","journal-title":"Lancet Digit. Health"},{"key":"295_CR10","doi-asserted-by":"publisher","first-page":"810","DOI":"10.1126\/science.aaw0029","volume":"363","author":"RB Parikh","year":"2019","unstructured":"Parikh, R. B., Obermeyer, Z. & Navathe, A. S. Regulation of predictive analytics in medicine. Science 363, 810\u2013812 (2019).","journal-title":"Science"},{"key":"295_CR11","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2018.178","volume":"5","author":"T Pollard","year":"2018","unstructured":"Pollard, T. et al. The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Sci. Data 5, 180178 (2018).","journal-title":"Sci. Data"},{"key":"295_CR12","doi-asserted-by":"crossref","unstructured":"Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., & Dawn, Song. Natural adversarial examples. https:\/\/arxiv.org\/abs\/1907.07174 (2020).","DOI":"10.1109\/CVPR46437.2021.01501"},{"key":"295_CR13","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-019-52737-x","volume":"9","author":"V Sandfort","year":"2019","unstructured":"Sandfort, V. et al. Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks. Sci. Rep. 9, 16884 (2019).","journal-title":"Sci. Rep."},{"key":"295_CR14","unstructured":"Alder, S. De-identification of protected health information: how to anonymize PHI. HIPAA J. https:\/\/www.hipaajournal.com\/de-identification-protected-health-information\/ (2017)"},{"key":"295_CR15","unstructured":"U.S. Department of Health and Human Services. Guidance Regarding Methods for De-identification of Protected Health Information in Accordance with the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule (U.S. Department of Health and Human Services, 2020) https:\/\/www.hhs.gov\/hipaa\/for-professionals\/privacy\/special-topics\/de-identification\/index.html."},{"key":"295_CR16","doi-asserted-by":"publisher","first-page":"010318","DOI":"10.7189\/jogh.09.020318","volume":"9","author":"T Panch","year":"2019","unstructured":"Panch, T., Mattie, H. & Atun, R. Artificial intelligence and algorithmic bias: implications for health systems. J. Glob. Health 9, 010318 (2019).","journal-title":"J. Glob. Health"},{"key":"295_CR17","unstructured":"National Health Service. Review of data security, consent and opt-outs. https:\/\/www.gov.uk\/government\/publications\/review-of-data-security-consent-and-opt-outs. (National Health Service, 2017)."},{"key":"295_CR18","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2016.18","volume":"3","author":"M Wilkinson","year":"2016","unstructured":"Wilkinson, M. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci. Data 3, 160018 (2016).","journal-title":"Sci. Data"},{"key":"295_CR19","doi-asserted-by":"crossref","unstructured":"Johnson, A. E. W. et al. MIMIC-III, a freely accessible critical care database. Sci. Data. http:\/\/www.nature.com\/articles\/sdata201635 (2016).","DOI":"10.1038\/sdata.2016.35"},{"key":"295_CR20","doi-asserted-by":"crossref","unstructured":"Reiz, A., N\u00fa\u00f1ez, & Organizing Committee of the Madrid. Big data and machine learning in critical care: opportunities for collaborative research. Med. Intensiv. 43(1), 52\u201357 (2019).","DOI":"10.1016\/j.medine.2018.06.006"},{"key":"295_CR21","doi-asserted-by":"publisher","first-page":"51","DOI":"10.5626\/JCSE.2012.6.1.51","volume":"6","author":"LeoA Celi","year":"2012","unstructured":"Celi, LeoA. et al. Collective experience: a database-fuelled, inter-disciplinary team-led learning system. J. Comput. Sci. Eng. JCSE 6, 51\u201359 (2012).","journal-title":"J. Comput. Sci. Eng. JCSE"},{"key":"295_CR22","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1093\/jamia\/ocx084","volume":"25","author":"AEW Johnson","year":"2018","unstructured":"Johnson, A. E. W., Stone, D. J., Celi, L. A. & Pollard, T. J. The MIMIC Code Repository: enabling reproducibility in critical care research. J. Am. Med. Inform. Assoc. 25, 32\u201339 (2018).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"295_CR23","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-019-1426-2","volume":"17","author":"CJ Kelly","year":"2019","unstructured":"Kelly, C. J. et al. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 17, 195 (2019).","journal-title":"BMC Med."},{"key":"295_CR24","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1038\/s41586-019-1799-6","volume":"577","author":"SM McKinney","year":"2020","unstructured":"McKinney, S. M. et al. International evaluation of an AI system for breast cancer screening. Nature 577, 89\u201394 (2020).","journal-title":"Nature"},{"key":"295_CR25","doi-asserted-by":"publisher","first-page":"487","DOI":"10.1148\/radiol.2019192515","volume":"294","author":"DA Bluemke","year":"2020","unstructured":"Bluemke, D. A. et al. Assessing radiology research on artificial intelligence: a brief guide for authors, reviewers, and readers-from the radiology editorial board. Radiology 294, 487\u2013489 (2020).","journal-title":"Radiology"},{"key":"295_CR26","doi-asserted-by":"publisher","first-page":"1035","DOI":"10.13063\/2327-9214.1035","volume":"1","author":"BJ Wells","year":"2013","unstructured":"Wells, B. J. et al. Strategies for handling missing data in electronic health record derived data. EGEMS 1, 1035 (2013).","journal-title":"EGEMS"},{"issue":"6","key":"295_CR27","doi-asserted-by":"publisher","first-page":"1052","DOI":"10.1093\/jamia\/ocx030","volume":"24","author":"SE Davis","year":"2017","unstructured":"Davis, S. E. et al. Calibration drift in regression and machine learning models for acute kidney injury. J. Am. Med. Inform. Assoc. 24(6), 1052\u20131061 (2017).","journal-title":"J. Am. Med. Inform. Assoc."},{"issue":"7436","key":"295_CR28","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1038\/494155a","volume":"494","author":"D Butler","year":"2013","unstructured":"Butler, D. When Google got flu wrong: US outbreak foxes a leading web-based method for tracking seasonal flu. Nature 494(7436), 155\u2013157 (2013).","journal-title":"Nature"},{"key":"295_CR29","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1038\/s41746-019-0155-4","volume":"2","author":"T Panch","year":"2019","unstructured":"Panch, T., Mattie, H. & Celi, L. A. The \u201cinconvenient truth\u201d about AI in healthcare. npj Digit. Med. 2, 77 (2019).","journal-title":"npj Digit. Med."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0295-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0295-6","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0295-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T02:21:21Z","timestamp":1670379681000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0295-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,19]]},"references-count":29,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["295"],"URL":"https:\/\/doi.org\/10.1038\/s41746-020-0295-6","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,19]]},"assertion":[{"value":"21 January 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 May 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 June 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"87"}}