{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,9]],"date-time":"2026-03-09T14:38:06Z","timestamp":1773067086398,"version":"3.50.1"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,10,11]],"date-time":"2019-10-11T00:00:00Z","timestamp":1570752000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,10,11]],"date-time":"2019-10-11T00:00:00Z","timestamp":1570752000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000867","name":"Robert Wood Johnson Foundation","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100000867","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Collection of high-dimensional, longitudinal digital health data has the potential to support a wide-variety of research and clinical applications including diagnostics and longitudinal health tracking. Algorithms that process these data and inform digital diagnostics are typically developed using training and test sets generated from multiple repeated measures collected across a set of individuals. However, the inclusion of repeated measurements is not always appropriately taken into account in the analytical evaluations of predictive performance. The assignment of repeated measurements from each individual to both the training and the test sets (\u201crecord-wise\u201d data split) is a common practice and can lead to massive underestimation of the prediction error due to the presence of \u201cidentity confounding.\u201d In essence, these models learn to identify subjects, in addition to diagnostic signal. Here, we present a method that can be used to effectively calculate the amount of identity confounding learned by classifiers developed using a record-wise data split. By applying this method to several real datasets, we demonstrate that identity confounding is a serious issue in digital health studies and that record-wise data splits for machine learning- based applications need to be avoided.<\/jats:p>","DOI":"10.1038\/s41746-019-0178-x","type":"journal-article","created":{"date-parts":[[2019,10,11]],"date-time":"2019-10-11T10:04:04Z","timestamp":1570788244000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":63,"title":["Detecting the impact of subject characteristics on machine learning-based diagnostic applications"],"prefix":"10.1038","volume":"2","author":[{"given":"Elias","family":"Chaibub Neto","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5289-6932","authenticated-orcid":false,"given":"Abhishek","family":"Pratap","sequence":"additional","affiliation":[]},{"given":"Thanneer M.","family":"Perumal","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0741-8683","authenticated-orcid":false,"given":"Meghasyam","family":"Tummalacherla","sequence":"additional","affiliation":[]},{"given":"Phil","family":"Snyder","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2412-6826","authenticated-orcid":false,"given":"Brian M.","family":"Bot","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7070-0139","authenticated-orcid":false,"given":"Andrew D.","family":"Trister","sequence":"additional","affiliation":[]},{"given":"Stephen H.","family":"Friend","sequence":"additional","affiliation":[]},{"given":"Lara","family":"Mangravite","sequence":"additional","affiliation":[]},{"given":"Larsson","family":"Omberg","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2019,10,11]]},"reference":[{"key":"178_CR1","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1159\/000486347","volume":"2","author":"B Perry","year":"2018","unstructured":"Perry, B. et al. Use of mobile devices to measure outcomes in clinical research, 2010-2016: a systematic literature review. Digital Biomark. 2, 11\u201330 (2018).","journal-title":"Digital Biomark."},{"key":"178_CR2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-019-0084-2","volume":"2","author":"LC Kourtis","year":"2019","unstructured":"Kourtis, L. C., Regele, O. B., Wright, J. M. & Jones, G. B. Digital biomarkers for Alzheimer\u2019s disease: the mobile\/wearable devices opportunity. npj Digital Med. 2, 9 (2019).","journal-title":"npj Digital Med."},{"key":"178_CR3","doi-asserted-by":"publisher","first-page":"1287","DOI":"10.1002\/mds.27376","volume":"33","author":"F Lipsmeier","year":"2018","unstructured":"Lipsmeier, F. et al. Evaluation of smartphone-based testing to generate exploratory outcome measures in a phase 1 Parkinson\u2019s disease clinical trial. Mov. Disord. 33, 1287\u20131297 (2018).","journal-title":"Mov. Disord."},{"key":"178_CR4","doi-asserted-by":"publisher","unstructured":"Dagum, P. Digital biomarkers of cognitive function. npj Digital Med. 1, https:\/\/doi.org\/10.1038\/s41746-018-0018-4 (2018).","DOI":"10.1038\/s41746-018-0018-4"},{"key":"178_CR5","doi-asserted-by":"publisher","unstructured":"Coravos, A., Khozin, S. & Mandl, K. D. Developing and adopting safe and effective digital biomarkers to improve patient outcomes. npj Digital Med. 2, https:\/\/doi.org\/10.1038\/s41746-019-0090-4 (2019).","DOI":"10.1038\/s41746-019-0090-4"},{"key":"178_CR6","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1159\/000477383","volume":"1","author":"RE Dorsey","year":"2017","unstructured":"Dorsey, R. E., Papapetropoulos, S., Xiong, M. & Kieburtz, K. The first frontier: digital biomarkers for neurodegenerative disorders. Digital Biomark. 1, 6\u201313 (2017).","journal-title":"Digital Biomark."},{"key":"178_CR7","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2018.6040","volume":"1","author":"L Na","year":"2018","unstructured":"Na, L. et al. Feasibility of reidentifying individuals in large national physical activity data sets from which protected health information has been removed with use of machine learning. JAMA Netw. Open 1, e166040 (2018).","journal-title":"JAMA Netw. Open"},{"key":"178_CR8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1093\/gigascience\/gix019","volume":"6","author":"S Saeb","year":"2017","unstructured":"Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C. & Kording, K. P. The need to approximate the use-case in clinical machine learning. GigaScience 6, 1\u20139 (2017).","journal-title":"GigaScience"},{"key":"178_CR9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1093\/gigascience\/gix020","volume":"6","author":"M Little","year":"2017","unstructured":"Little, M. et al. Using and understanding cross-validation strategies. Perspectives on Saeb et al. GigaScience 6, 1\u20139 (2017).","journal-title":"GigaScience"},{"key":"178_CR10","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1038\/532020a","volume":"532","author":"E Anthes","year":"2016","unstructured":"Anthes, E. Mental health: there\u2019s an app for that. Nature 532, 20\u201323 (2016).","journal-title":"Nature"},{"key":"178_CR11","doi-asserted-by":"publisher","DOI":"10.1186\/1475-925X-6-23","volume":"6","author":"MA Little","year":"2017","unstructured":"Little, M. A., McSharry, P. E., Roberts, S. J., Costello, D. A. E. & Moroz, I. M. Exploiting nonlinear recurrence and fractal scaling properties for voice disorder detection. Biomed. Eng. OnLine 6, 23 (2017).","journal-title":"Biomed. Eng. OnLine"},{"key":"178_CR12","doi-asserted-by":"publisher","first-page":"828","DOI":"10.1109\/JBHI.2013.2245674","volume":"17","author":"BE Sarkar","year":"2013","unstructured":"Sarkar, B. E. et al. Collection and analysis of a Parkinson speeach dataset with multiple types of sound recordings. IEEE J. Biomed. Health Inform. 17, 828\u2013834 (2013).","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"178_CR13","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2016.11","volume":"3","author":"BM Bot","year":"2016","unstructured":"Bot, B. M. et al. The mPower study, Parkinson disease mobile data collected using ResearchKit. Sci. Data 3, 160011 (2016).","journal-title":"Sci. Data"},{"key":"178_CR14","doi-asserted-by":"crossref","unstructured":"Trister, A. D., Dorsey, E. R., & Friend, S. H. Smartphones as new tools in the management and understanding of Parkinson\u2019s disease. npj Parkinson\u2019s Dis. 16006 (2016).","DOI":"10.1038\/npjparkd.2016.6"},{"key":"178_CR15","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-3235-1","volume-title":"Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypothesis.","author":"P Good","year":"2000","unstructured":"Good, P. Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypothesis. 2nd edn (Springer, New York, 2000).","edition":"2nd edn"},{"key":"178_CR16","doi-asserted-by":"publisher","first-page":"591","DOI":"10.1007\/s10916-009-9272-y","volume":"34","author":"CO Sarkar","year":"2010","unstructured":"Sarkar, C. O. & Kursun, O. Telediagnosis of Parkinson\u2019s disease using measurements of dysphonia. J. Med. Syst. 34, 591\u2013599 (2010).","journal-title":"J. Med. Syst."},{"key":"178_CR17","unstructured":"Chaibub Neto, E. et al. On the analysis of personalized medication response and classification of case vs control patients in mobile health studies: the mPower case study. arXiv:1706.09574 (2017)."},{"key":"178_CR18","doi-asserted-by":"crossref","unstructured":"Chaibub Neto, E. et al. Using permutations to assess confounding in machine learning applications for digital health. Machine Learning for Health (ML4H) Workshop at the Third Second Neural Information Processing Systems Conference, NeurIPS 2018. arXiv:1811.11920 (2018).","DOI":"10.1145\/3292500.3330903"},{"key":"178_CR19","doi-asserted-by":"crossref","unstructured":"Chaibub Neto, E. et al. A permutation approach to assess confounding in machine learning applications for digital health. In The 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD'19), August 04-08, 2019, Anchorage, AK, USA, 54\u201364 (2019).","DOI":"10.1145\/3292500.3330903"},{"key":"178_CR20","first-page":"1","volume":"56","author":"MA Little","year":"2008","unstructured":"Little, M. A., McSharry, P. E., Hunter, E. J., Spielman, J. & Ramig, L. O. Suitability of dysphonia measurements for telemonitoring of Parkinson\u2019s disease. IEEE Trans. Biomed. Eng. 56, 1\u20139 (2008).","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"178_CR21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v042.i08","volume":"42","author":"D Ho","year":"2011","unstructured":"Ho, D., Imai, K., King, G. & Stuart, E. A. MatchIt: Non-parametric preprocessing for parametric causal inference. J. Stat. Softw. 42, 1\u201328 (2011).","journal-title":"J. Stat. Softw."},{"key":"178_CR22","doi-asserted-by":"publisher","first-page":"842","DOI":"10.1098\/rsif.2010.0456","volume":"8","author":"A Tsanas","year":"2011","unstructured":"Tsanas, A., Little, M. A., McSharry, P. E. & Ramig, L. O. Nonlinear speech analysis algorithms mapped to a standard metric achieve clinically useful quantification of average Parkinson\u2019s disease symptom severity. J. R. Soc. Interface 8, 842\u2013855 (2011).","journal-title":"J. R. Soc. Interface"},{"key":"178_CR23","doi-asserted-by":"publisher","first-page":"650","DOI":"10.1016\/j.parkreldis.2015.02.026","volume":"21","author":"S Arora","year":"2015","unstructured":"Arora, S. et al. Detecting and monitoring the symptoms of Parkinson\u2019s disease using smartphones: A pilot study. Parkinsonism Relat. Disord. 21, 650\u2013653 (2015).","journal-title":"Parkinsonism Relat. Disord."},{"key":"178_CR24","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","volume":"45","author":"L Breiman","year":"2001","unstructured":"Breiman, L. Random forests. Mach. Learn. 45, 5\u201332 (2001).","journal-title":"Mach. Learn."},{"key":"178_CR25","first-page":"18","volume":"2","author":"A Liaw","year":"2002","unstructured":"Liaw, A. & Wiener, M. Classification and regression by randomForest. R. News 2, 18\u201322 (2002).","journal-title":"R. News"},{"key":"178_CR26","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2105-12-77","volume":"12","author":"X Xavier Robin","year":"2011","unstructured":"Xavier Robin, X. et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinforma. 12, 77 (2011).","journal-title":"BMC Bioinforma."},{"key":"178_CR27","unstructured":"Dua, D. & Graff, C. UCI Machine Learning Repository. http:\/\/archive.ics.uci.edu\/ml, (University of California, School of Information and Computer Science, Irvine, CA, 2019)."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0178-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0178-x","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0178-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,17]],"date-time":"2022-12-17T18:33:59Z","timestamp":1671302039000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0178-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,11]]},"references-count":27,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2019,12]]}},"alternative-id":["178"],"URL":"https:\/\/doi.org\/10.1038\/s41746-019-0178-x","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,10,11]]},"assertion":[{"value":"4 June 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 September 2019","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 October 2019","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"99"}}