{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,18]],"date-time":"2026-04-18T16:58:40Z","timestamp":1776531520799,"version":"3.51.2"},"reference-count":42,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2023,2,6]],"date-time":"2023-02-06T00:00:00Z","timestamp":1675641600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"University of Technology of Compi\u00e8gne"},{"name":"Waterloo University"},{"name":"Idex Sorbonne University"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Stress is an increasingly prevalent mental health condition across the world. In Europe, for example, stress is considered one of the most common health problems, and over USD 300 billion are spent on stress treatments annually. Therefore, monitoring, identification and prevention of stress are of the utmost importance. While most stress monitoring is carried out through self-reporting, there are now several studies on stress detection from physiological signals using Artificial Intelligence algorithms. However, the generalizability of these models is only rarely discussed. The main goal of this work is to provide a monitoring proof-of-concept tool exploring the generalization capabilities of Heart Rate Variability-based machine learning models. To this end, two Machine Learning models are used, Logistic Regression and Random Forest to analyze and classify stress in two datasets differing in terms of protocol, stressors and recording devices. First, the models are evaluated using leave-one-subject-out cross-validation with train and test samples from the same dataset. Next, a cross-dataset validation of the models is performed, that is, leave-one-subject-out models trained on a Multi-modal Dataset for Real-time, Continuous Stress Detection from Physiological Signals dataset and validated using the University of Waterloo stress dataset. While both logistic regression and random forest models achieve good classification results in the independent dataset analysis, the random forest model demonstrates better generalization capabilities with a stable F1 score of 61%. This indicates that the random forest can be used to generalize HRV-based stress detection models, which can lead to better analyses in the mental health and medical research field through training and integrating different models.<\/jats:p>","DOI":"10.3390\/s23041807","type":"journal-article","created":{"date-parts":[[2023,2,6]],"date-time":"2023-02-06T04:08:07Z","timestamp":1675656487000},"page":"1807","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":32,"title":["Cross Dataset Analysis for Generalizability of HRV-Based Stress Detection Models"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5833-5776","authenticated-orcid":false,"given":"Mouna","family":"Benchekroun","sequence":"first","affiliation":[{"name":"Biomechanics and Bioengineering Lab, University of Technology of Compi\u00e8gne (UMR CNRS 7338), 60200 Compi\u00e8gne, France"},{"name":"Heudiasyc Lab (Heuristics and Diagnosis of Complex Systems), University of Technology of Compi\u00e8gne (UMR CNRS 7338), 60200 Compi\u00e8gne, France"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7539-3193","authenticated-orcid":false,"given":"Pedro Elkind","family":"Velmovitsky","sequence":"additional","affiliation":[{"name":"School of Public Health and Health Systems, University of Waterloo, Waterloo, ON N2L 3G1, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5906-4947","authenticated-orcid":false,"given":"Dan","family":"Istrate","sequence":"additional","affiliation":[{"name":"Biomechanics and Bioengineering Lab, University of Technology of Compi\u00e8gne (UMR CNRS 7338), 60200 Compi\u00e8gne, France"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5325-6649","authenticated-orcid":false,"given":"Vincent","family":"Zalc","sequence":"additional","affiliation":[{"name":"Biomechanics and Bioengineering Lab, University of Technology of Compi\u00e8gne (UMR CNRS 7338), 60200 Compi\u00e8gne, France"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9515-6478","authenticated-orcid":false,"given":"Plinio Pelegrini","family":"Morita","sequence":"additional","affiliation":[{"name":"School of Public Health and Health Systems, University of Waterloo, Waterloo, ON N2L 3G1, Canada"},{"name":"Research Institute for Aging, University of Waterloo, Waterloo, ON N2J 0E2, Canada"},{"name":"Department of Systems Design Engineering, University of Waterloo, Waterloo, ON N2L 3G1, Canada"},{"name":"Institute of Health Policy, Management, and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, ON M5T 3M6, Canada"},{"name":"Centre for Digital Therapeutics, Techna Institute, University Health Network, Toronto, ON M5G 2C4, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6225-8854","authenticated-orcid":false,"given":"Dominique","family":"Lenne","sequence":"additional","affiliation":[{"name":"Heudiasyc Lab (Heuristics and Diagnosis of Complex Systems), University of Technology of Compi\u00e8gne (UMR CNRS 7338), 60200 Compi\u00e8gne, France"}]}],"member":"1968","published-online":{"date-parts":[[2023,2,6]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"103139","DOI":"10.1016\/j.jbi.2019.103139","article-title":"Stress detection in daily life scenarios using smart phones and wearable sensors: A survey","volume":"92","author":"Can","year":"2019","journal-title":"J. Biomed. Inform."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1109\/TMSCS.2017.2703613","article-title":"Keep the stress away with SoDA: Stress detection and alleviation system","volume":"3","author":"Akmandor","year":"2017","journal-title":"IEEE Trans. Multi-Scale Comput. Syst."},{"key":"ref_3","first-page":"1","article-title":"Perceived stress scale","volume":"Volume 10","author":"Cohen","year":"1994","journal-title":"Measuring Stress: A Guide for Health and Social Scientists"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Can, Y.S., Gokay, D., K\u0131l\u0131\u00e7, D.R., Ekiz, D., Chalabianloo, N., and Ersoy, C. (2020). How laboratory experiments can be exploited for monitoring stress in the wild: A bridge between laboratory and daily life. Sensors, 20.","DOI":"10.3390\/s20030838"},{"key":"ref_5","first-page":"13","article-title":"Ecological momentary assessment: What it is and why it is a method of the future in clinical psychopharmacology","volume":"31","author":"Moskowitz","year":"2006","journal-title":"J. Psychiatry Neurosci."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"84045","DOI":"10.1109\/ACCESS.2021.3085502","article-title":"A review on mental stress detection using wearable sensors and machine learning techniques","volume":"9","author":"Gedam","year":"2021","journal-title":"IEEE Access"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Kristoffersson, A., and Lind\u00e9n, M. (2020). A systematic review on the use of wearable body sensors for health monitoring: A qualitative synthesis. Sensors, 20.","DOI":"10.3390\/s20051502"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"444","DOI":"10.1016\/j.bbe.2019.01.004","article-title":"A survey of machine learning techniques in physiology based mental stress detection systems","volume":"39","author":"Panicker","year":"2019","journal-title":"Biocybern. Biomed. Eng."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Giannakakis, G., Marias, K., and Tsiknakis, M. (2019, January 3\u20136). A stress recognition system using HRV parameters and machine learning techniques. Proceedings of the 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), Cambridge, UK.","DOI":"10.1109\/ACIIW.2019.8925142"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Castaldo, R., Xu, W., Melillo, P., Pecchia, L., Santamaria, L., and James, C. (2016, January 16\u201320). Detection of mental stress due to oral academic examination via ultra-short-term HRV analysis. Proceedings of the 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA.","DOI":"10.1109\/EMBC.2016.7591557"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Hovsepian, K., Al\u2019Absi, M., Ertin, E., Kamarck, T., Nakajima, M., and Kumar, S. (2015, January 7\u201311). cStress: Towards a gold standard for continuous stress assessment in the mobile environment. Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Osaka, Japan.","DOI":"10.1145\/2750858.2807526"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Gjoreski, M., Gjoreski, H., Lu\u0161trek, M., and Gams, M. (2016, January 12\u201316). Continuous stress detection using a wrist device: In laboratory and real life. Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct, Heidelberg, Germany.","DOI":"10.1145\/2968219.2968306"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1109\/TITS.2005.848368","article-title":"Detecting stress during real-world driving tasks using physiological sensors","volume":"6","author":"Healey","year":"2005","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Koldijk, S., Sappelli, M., Verberne, S., Neerincx, M.A., and Kraaij, W. (2014, January 12\u201316). The swell knowledge work dataset for stress and user modeling research. Proceedings of the 16th International Conference on Multimodal Interaction, Istanbul, Turkey.","DOI":"10.1145\/2663204.2663257"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/T-AFFC.2011.15","article-title":"Deap: A database for emotion analysis; using physiological signals","volume":"3","author":"Koelstra","year":"2011","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Schmidt, P., Reiss, A., Duerichen, R., Marberger, C., and Van Laerhoven, K. (2018, January 16\u201320). Introducing wesad, a multimodal dataset for wearable stress and affect detection. Proceedings of the 20th ACM International Conference on Multimodal Interaction, Boulder, CO, USA.","DOI":"10.1145\/3242969.3242985"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1109\/T-AFFC.2011.25","article-title":"A Multimodal Database for Affect Recognition and Implicit Tagging","volume":"3","author":"Soleymani","year":"2012","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Benchekroun, M., Istrate, D., Zalc, V., and Lenne, D. (2022, January 9\u201311). Mmsd: A Multi-modal Dataset for Real-time, Continuous Stress Detection from Physiological Signals. Proceedings of the HEALTHINF, Online.","DOI":"10.5220\/0010985400003123"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1207\/s15327752jpa4801_16","article-title":"Psychometric properties of the STAI: A reply to Ramanaiah, Franzen, and Schill","volume":"48","author":"Speilberger","year":"1984","journal-title":"J. Personal. Assess."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Velmovitsky, P.E., Alencar, P., Leatherdale, S.T., Cowan, D., and Morita, P.P. (2021, January 15\u201318). Towards Real-Time Public Health: A Novel Mobile Health Monitoring System. Proceedings of the 2021 IEEE International Conference on Big Data (Big Data), Orlando, FL, USA.","DOI":"10.1109\/BigData52589.2021.9672059"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1322","DOI":"10.1002\/jclp.21908","article-title":"The depression anxiety stress Scales\u201421 (DASS-21): Further examination of dimensions, scale reliability, and correlates","volume":"68","author":"Osman","year":"2012","journal-title":"J. Clin. Psychol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1097\/FBP.0000000000000383","article-title":"Heart-rate variability: A biomarker to study the influence of nutrition on physiological and psychological health?","volume":"29","author":"Young","year":"2018","journal-title":"Behav. Pharmacol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1031","DOI":"10.1007\/s11517-006-0119-0","article-title":"Heart rate variability: A review","volume":"44","author":"Kannathal","year":"2006","journal-title":"Med. Biol. Eng. Comput."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1093\/cvr\/4.2.160","article-title":"The normal range and determinants of the intrinsic heart rate in man","volume":"4","author":"Jose","year":"1970","journal-title":"Cardiovasc. Res."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"747","DOI":"10.1016\/j.neubiorev.2011.11.009","article-title":"A meta-analysis of heart rate variability and neuroimaging studies: Implications for heart rate variability as a marker of stress and health","volume":"36","author":"Thayer","year":"2012","journal-title":"Neurosci. Biobehav. Rev."},{"key":"ref_26","unstructured":"Bartels, R., and Pe\u00e7anha, T. (2021, September 23). HRV: A Pythonic Package for Heart Rate Variability Analysis. Github July 2020. Available online: https:\/\/github.com\/rhenanbartels\/hrv\/tree\/0.2.8."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"100456","DOI":"10.1016\/j.softx.2020.100456","article-title":"TSFEL: Time series feature extraction library","volume":"11","author":"Barandas","year":"2020","journal-title":"SoftwareX"},{"key":"ref_28","unstructured":"Gomes, P., Margaritoff, P., and Silva, H. (2019, January 3\u20136). pyHRV: Development and evaluation of an open-source python toolbox for heart rate variability (HRV). Proceedings of the International Conference on Electrical, Electronic, and Computing Engineering (IcETRAN), Silver Lake, Serbia."},{"key":"ref_29","unstructured":"Benchekroun, M., Chevallier, B., Zalc, V., Istrate, D., Lenne, D., and Vera, N. (2021, January 20\u201321). Analysis of the Impact of Inter-Beat-Interval Interpolation on real-time HRV Feature Estimation for e-Health Applications. Proceedings of the JETSAN 2021\u20148th E-Health and Biomedical Devices Study Conference 2021, Toulouse, France."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1485","DOI":"10.1088\/0967-3334\/28\/12\/003","article-title":"Effect of missing RR-interval data on heart rate variability analysis in the time domain","volume":"28","author":"Kim","year":"2007","journal-title":"Physiol. Meas."},{"key":"ref_31","unstructured":"Task Force of The European Society of Cardiology, and The North American Society of Pacing and Electrophysiology (1996). Heart rate variability: Standards of measurement, physiological interpretation and clinical use. Circulation, 93, 1043\u20131065."},{"key":"ref_32","unstructured":"Caridade Gomes, P.M. (2019). Development of an Open-Source Python Toolbox for Heart Rate Variability (HRV). [Ph.D. Thesis, Hochschule f\u00fcr Angewandte Wissenschaften Hamburg]."},{"key":"ref_33","unstructured":"Vinutha, H., Poornima, B., and Sagar, B. (2018). Information and Decision Sciences, Springer."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Han, H., Wang, W.Y., and Mao, B.H. (2005, January 23\u201326). Borderline-SMOTE: A new over-sampling method in imbalanced data sets learning. Proceedings of the International Conference on Intelligent Computing, Hefei, China.","DOI":"10.1007\/11538059_91"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Muaremi, A., Bexheti, A., Gravenhorst, F., Arnrich, B., and Tr\u00f6ster, G. (2014, January 1\u20134). Monitoring the impact of stress on the sleep patterns of pilgrims using wearable sensors. Proceedings of the IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), Valencia, Spain.","DOI":"10.1109\/BHI.2014.6864335"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Dalmeida, K.M., and Masala, G.L. (2021). HRV features as viable physiological markers for stress detection using wearable devices. Sensors, 21.","DOI":"10.3390\/s21082873"},{"key":"ref_37","first-page":"444","article-title":"HRVBased Stress Recognizing by Random Forest","volume":"Volume 2016","author":"Sun","year":"2016","journal-title":"Fuzzy Systems and Data MiningII: Proceedings of FSDM 2016, Macau, China, 1\u201314 December 2016"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"2793","DOI":"10.1007\/s00500-020-05338-0","article-title":"Toward soft real-time stress detection using wrist-worn devices for human workspaces","volume":"25","author":"Khowaja","year":"2021","journal-title":"Soft Comput."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Benchekroun, M., Chevallier, B., Beaouiss, H., Istrate, D., Zalc, V., Khalil, M., and Lenne, D. (2022, January 11\u201315). Comparison of Stress Detection through ECG and PPG signals using a Random Forest-based Algorithm. Proceedings of the 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Scotland, UK.","DOI":"10.1109\/EMBC48229.2022.9870984"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Nkurikiyeyezu, K., Yokokubo, A., and Lopez, G. (2019). The effect of person-specific biometrics in improving generic stress predictive models. arXiv.","DOI":"10.18494\/SAM.2020.2650"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Prajod, P., and Andr\u00e9, E. (2022). On the Generalizability of ECG-based Stress Detection Models. arXiv.","DOI":"10.1109\/ICMLA55696.2022.00090"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1145\/3432220","article-title":"Evaluating the reproducibility of physiological stress detection models","volume":"4","author":"Mishra","year":"2020","journal-title":"Proc. Acm Interact. Mob. Wearable Ubiquitous Technol."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/4\/1807\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:25:26Z","timestamp":1760120726000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/4\/1807"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,6]]},"references-count":42,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["s23041807"],"URL":"https:\/\/doi.org\/10.3390\/s23041807","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,6]]}}}