{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,5]],"date-time":"2026-11-05T20:27:10Z","timestamp":1793910430353,"version":"4.8.0"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,4,22]],"date-time":"2024-04-22T00:00:00Z","timestamp":1713744000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,4,22]],"date-time":"2024-04-22T00:00:00Z","timestamp":1713744000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Commun Med"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Machine learning (ML) models are evaluated in a test set to estimate model performance after deployment. The design of the test set is therefore of importance because if the data distribution after deployment differs too much, the model performance decreases. At the same time, the data often contains undetected groups. For example, multiple assessments from one user may constitute a group, which is usually the case in mHealth scenarios.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>In this work, we evaluate a model\u2019s performance using several cross-validation train-test-split approaches, in some cases deliberately ignoring the groups. By sorting the groups (in our case: Users) by time, we additionally simulate a concept drift scenario for better external validity. For this evaluation, we use 7 longitudinal mHealth datasets, all containing Ecological Momentary Assessments (EMA). Further, we compared the model performance with baseline heuristics, questioning the essential utility of a complex ML model.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Hidden groups in the dataset leads to overestimation of ML performance after deployment. For prediction, a user\u2019s last completed questionnaire is a reasonable heuristic for the next response, and potentially outperforms a complex ML model. Because we included 7 studies, low variance appears to be a more fundamental phenomenon of mHealth datasets.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>The way mHealth-based data are generated by EMA leads to questions of user and assessment level and appropriate validation of ML models. Our analysis shows that further research needs to follow to obtain robust ML models. In addition, simple heuristics can be considered as an alternative for ML. Domain experts should be consulted to find potentially hidden groups in the data.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1038\/s43856-024-00468-0","type":"journal-article","created":{"date-parts":[[2024,4,22]],"date-time":"2024-04-22T09:01:45Z","timestamp":1713776505000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":27,"title":["Practical approaches in evaluating validation and biases of machine learning applied to mobile health studies"],"prefix":"10.1038","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9051-2004","authenticated-orcid":false,"given":"Johannes","family":"Allgaier","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1522-785X","authenticated-orcid":false,"given":"R\u00fcdiger","family":"Pryss","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,4,22]]},"reference":[{"key":"468_CR1","doi-asserted-by":"crossref","unstructured":"Kunjan, S. et al. The necessity of leave one subject out (loso) cross validation for eeg disease diagnosis. In Brain Informatics: 14th International Conference, BI 2021, Virtual Event, September 17\u201319, 2021, Proceedings vol. 14, 558\u2013567 (Springer, 2021).","DOI":"10.1007\/978-3-030-86993-9_50"},{"key":"468_CR2","unstructured":"Kohavi, R. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI), vol. 14, 1137\u20131145 (Montreal, Canada, 1995)."},{"key":"468_CR3","doi-asserted-by":"publisher","first-page":"1895","DOI":"10.1162\/089976698300017197","volume":"10","author":"TG Dietterich","year":"1998","unstructured":"Dietterich, T. G. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Comput 10, 1895\u20131923 (1998).","journal-title":"Neural Comput"},{"key":"468_CR4","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","volume":"1","author":"C Rudin","year":"2019","unstructured":"Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206\u2013215 (2019).","journal-title":"Nat. Mach. Intell."},{"key":"468_CR5","first-page":"1","volume":"9","author":"P Chapman","year":"2000","unstructured":"Chapman, P. et al. Crisp-dm 1.0: Step-by-step data mining guide. SPSS Inc 9, 1\u201373 (2000).","journal-title":"SPSS Inc"},{"key":"468_CR6","doi-asserted-by":"publisher","first-page":"7395","DOI":"10.3390\/ijerph18147395","volume":"18","author":"F Beierle","year":"2021","unstructured":"Beierle, F. et al. Corona health\u2013a study-and sensor-based mobile app platform exploring aspects of the covid-19 pandemic. Int. J. Environ. Res. Public Health 18, 7395 (2021).","journal-title":"Int. J. Environ. Res. Public Health"},{"key":"468_CR7","doi-asserted-by":"crossref","unstructured":"Vogel, C., Schobel, J., Schlee, W., Engelke, M. & Pryss, R. Uniti mobile\u2013emi-apps for a large-scale european study on tinnitus. In 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), vol. 43, 2358\u20132362 (IEEE, 2021).","DOI":"10.1109\/EMBC46164.2021.9630482"},{"key":"468_CR8","doi-asserted-by":"publisher","first-page":"164","DOI":"10.3389\/fnins.2020.00164","volume":"14","author":"R Kraft","year":"2020","unstructured":"Kraft, R. et al. Combining mobile crowdsensing and ecological momentary assessments in the healthcare domain. Front. Neurosci. 14, 164 (2020).","journal-title":"Front. Neurosci."},{"key":"468_CR9","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-020-79527-0","volume":"10","author":"M Schleicher","year":"2020","unstructured":"Schleicher, M. et al. Understanding adherence to the recording of ecological momentary assessments in the example of tinnitus monitoring. Sci. Rep. 10, 22459 (2020).","journal-title":"Sci. Rep."},{"key":"468_CR10","first-page":"2079","volume":"11","author":"GC Cawley","year":"2010","unstructured":"Cawley, G. C. & Talbot, N. L. On over-fitting in model selection and subsequent selection bias in performance evaluation. J. Mach. Learn. Res. 11, 2079\u20132107 (2010).","journal-title":"J. Mach. Learn. Res."},{"key":"468_CR11","doi-asserted-by":"publisher","first-page":"532","DOI":"10.1007\/978-0-387-39940-9_565","volume":"5","author":"P Refaeilzadeh","year":"2009","unstructured":"Refaeilzadeh, P., Tang, L. & Liu, H. Cross-validation. Encyclopedia Database Syst. 5, 532\u2013538 (2009).","journal-title":"Encyclopedia Database Syst."},{"key":"468_CR12","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1016\/j.ecolmodel.2019.06.002","volume":"406","author":"P Schratz","year":"2019","unstructured":"Schratz, P., Muenchow, J., Iturritxa, E., Richter, J. & Brenning, A. Hyperparameter tuning and performance assessment of statistical and machine-learning algorithms using spatial data. Ecolog. Model. 406, 109\u2013120 (2019).","journal-title":"Ecolog. Model."},{"key":"468_CR13","doi-asserted-by":"publisher","first-page":"486","DOI":"10.1080\/01621459.1993.10476299","volume":"88","author":"J Shao","year":"1993","unstructured":"Shao, J. Linear model selection by cross-validation. J. Am. Stat. Associat. 88, 486\u2013494 (1993).","journal-title":"J. Am. Stat. Associat."},{"key":"468_CR14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.envsoft.2017.12.001","volume":"101","author":"H Meyer","year":"2018","unstructured":"Meyer, H., Reudenbach, C., Hengl, T., Katurji, M. & Nauss, T. Improving performance of spatio-temporal machine learning models using forward feature selection and target-oriented validation. Environ. Model. Software 101, 1\u20139 (2018).","journal-title":"Environ. Model. Software"},{"key":"468_CR15","doi-asserted-by":"publisher","first-page":"4270","DOI":"10.3390\/jcm11154270","volume":"11","author":"J Allgaier","year":"2022","unstructured":"Allgaier, J., Schlee, W., Probst, T. & Pryss, R. Prediction of tinnitus perception based on daily life mhealth data using country origin and season. J. Clin. Med. 11, 4270 (2022).","journal-title":"J. Clin. Med."},{"key":"468_CR16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1146\/annurev.clinpsy.3.022806.091415","volume":"4","author":"S Shiffman","year":"2008","unstructured":"Shiffman, S., Stone, A. A. & Hufford, M. R. Ecological momentary assessment. Annu. Rev. Clin. Psychol. 4, 1\u201332 (2008).","journal-title":"Annu. Rev. Clin. Psychol."},{"key":"468_CR17","first-page":"206","volume":"4","author":"M Holfelder","year":"2021","unstructured":"Holfelder, M. et al. Medical device regulation efforts for mhealth apps during the covid-19 pandemic\u2013an experience report of corona check and corona health. J 4, 206\u2013222 (2021).","journal-title":"J"},{"key":"468_CR18","doi-asserted-by":"crossref","unstructured":"Pryss, R., Reichert, M., Herrmann, J., Langguth, B. & Schlee, W. Mobile crowd sensing in clinical and psychological trials\u2013a case study. In 2015 IEEE 28th international symposium on computer-based medical systems, 23\u201324 (IEEE, 2015).","DOI":"10.1109\/CBMS.2015.26"},{"key":"468_CR19","doi-asserted-by":"publisher","first-page":"441","DOI":"10.1016\/bs.pbr.2020.12.005","volume":"260","author":"W Schlee","year":"2021","unstructured":"Schlee, W. et al. Towards a unification of treatments and interventions for tinnitus patients: The eu research and innovation action uniti. Progress Brain Res. 260, 441\u2013451 (2021).","journal-title":"Progress Brain Res."},{"key":"468_CR20","doi-asserted-by":"publisher","DOI":"10.1186\/s13063-023-07303-2","volume":"24","author":"JP Simoes","year":"2023","unstructured":"Simoes, J. P. et al. The statistical analysis plan for the unification of treatments and interventions for tinnitus patients randomized clinical trial (uniti-rct). Trials 24, 472 (2023).","journal-title":"Trials"},{"key":"468_CR21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-021-96731-8","volume":"11","author":"J Allgaier","year":"2021","unstructured":"Allgaier, J., Schlee, W., Langguth, B., Probst, T. & Pryss, R. Predicting the gender of individuals with tinnitus based on daily life data of the trackyourtinnitus mhealth platform. Sci. Rep. 11, 1\u201314 (2021).","journal-title":"Sci. Rep."},{"key":"468_CR22","doi-asserted-by":"crossref","unstructured":"Beierle, F. et al. Self-assessment of having covid-19 with the corona check mhealth app. IEEE J Biomed Health Inform. 27, 2794\u20132805 (2023).","DOI":"10.1109\/JBHI.2023.3264999"},{"key":"468_CR23","doi-asserted-by":"publisher","first-page":"e40958","DOI":"10.2196\/40958","volume":"9","author":"E Humer","year":"2023","unstructured":"Humer, E. et al. Associations of country-specific and sociodemographic factors with self-reported covid-19\u2013related symptoms: Multivariable analysis of data from the coronacheck mobile health platform. JMIR Public Health Surveil. 9, e40958 (2023).","journal-title":"JMIR Public Health Surveil."},{"key":"468_CR24","doi-asserted-by":"publisher","first-page":"6212","DOI":"10.3390\/ijerph18126212","volume":"18","author":"B Wetzel","year":"2021","unstructured":"Wetzel, B. et al. \"How come you don\u2019t call me?\u201d Smartphone communication app usage as an indicator of loneliness and social well-being across the adult lifespan during the COVID-19 pandemic. Int. Environ. Res. Public Health 18, 6212 (2021).","journal-title":"Int. Environ. Res. Public Health"},{"key":"468_CR25","doi-asserted-by":"publisher","first-page":"606","DOI":"10.1046\/j.1525-1497.2001.016009606.x","volume":"16","author":"K Kroenke","year":"2001","unstructured":"Kroenke, K., Spitzer, R. L. & Williams, J. B. The phq-9: validity of a brief depression severity measure. J. General Internal Med. 16, 606\u2013613 (2001).","journal-title":"J. General Internal Med."},{"key":"468_CR26","first-page":"1","volume":"10","author":"S Cohen","year":"1994","unstructured":"Cohen, S., Kamarck, T. & Mermelstein, R. et al. Perceived stress scale. Measur. Stress: Guider Health Social Scient. 10, 1\u20132 (1994).","journal-title":"Measur. Stress: Guider Health Social Scient."},{"key":"468_CR27","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1111\/j.2517-6161.1974.tb00994.x","volume":"36","author":"M Stone","year":"1974","unstructured":"Stone, M. Cross-validatory choice and assessment of statistical predictions. J. Royal Stat. Society: Series B (Methodological) 36, 111\u2013133 (1974).","journal-title":"J. Royal Stat. Society: Series B (Methodological)"},{"key":"468_CR28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/00401706.1968.10490530","volume":"10","author":"PA Lachenbruch","year":"1968","unstructured":"Lachenbruch, P. A. & Mickey, M. R. Estimation of error rates in discriminant analysis. Technometrics 10, 1\u201311 (1968).","journal-title":"Technometrics"},{"key":"468_CR29","doi-asserted-by":"publisher","first-page":"320","DOI":"10.1080\/01621459.1975.10479865","volume":"70","author":"S Geisser","year":"1975","unstructured":"Geisser, S. The predictive sample reuse method with applications. J. Am. Stat. Associa. 70, 320\u2013328 (1975).","journal-title":"J. Am. Stat. Associa."},{"key":"468_CR30","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa, F. et al. Scikit-learn: Machine learning in python. J. Mach. Learn. Res. 12, 2825\u20132830 (2011).","journal-title":"J. Mach. Learn. Res."},{"key":"468_CR31","doi-asserted-by":"publisher","first-page":"447","DOI":"10.3389\/fnagi.2017.00447","volume":"9","author":"W Schlee","year":"2018","unstructured":"Schlee, W. et al. Innovations in doctoral training and research on tinnitus: The european school on interdisciplinary tinnitus research (esit) perspective. Front. Aging Neurosci 9, 447 (2018).","journal-title":"Front. Aging Neurosci"},{"key":"468_CR32","doi-asserted-by":"publisher","unstructured":"Allgaier, J. Github repository \u2223 from hidden groups to robust models: How to better estimate performance of mobile health models. Zenodo https:\/\/doi.org\/10.5281\/zenodo.10401660 (2023).","DOI":"10.5281\/zenodo.10401660"}],"container-title":["Communications Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s43856-024-00468-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s43856-024-00468-0","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s43856-024-00468-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,22]],"date-time":"2024-04-22T09:08:35Z","timestamp":1713776915000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s43856-024-00468-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,22]]},"references-count":32,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["468"],"URL":"https:\/\/doi.org\/10.1038\/s43856-024-00468-0","relation":{},"ISSN":["2730-664X"],"issn-type":[{"value":"2730-664X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,22]]},"assertion":[{"value":"21 March 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 February 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 April 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"76"}}