{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T12:39:50Z","timestamp":1791549590336,"version":"4.3.4"},"reference-count":68,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2020,12,17]],"date-time":"2020-12-17T00:00:00Z","timestamp":1608163200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000026","name":"National Institute on Drug Abuse","doi-asserted-by":"publisher","award":["P30DA029926"],"award-info":[{"award-number":["P30DA029926"]}],"id":[{"id":"10.13039\/100000026","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["CNS-1314281,CNS-1314342,CNS-1619970,CNS-1619950"],"award-info":[{"award-number":["CNS-1314281,CNS-1314342,CNS-1619970,CNS-1619950"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2020,12,17]]},"abstract":"<jats:p>Recent advances in wearable sensor technologies have led to a variety of approaches for detecting physiological stress. Even with over a decade of research in the domain, there still exist many significant challenges, including a near-total lack of reproducibility across studies. Researchers often use some physiological sensors (custom-made or off-the-shelf), conduct a study to collect data, and build machine-learning models to detect stress. There is little effort to test the applicability of the model with similar physiological data collected from different devices, or the efficacy of the model on data collected from different studies, populations, or demographics.<\/jats:p>\n          <jats:p>This paper takes the first step towards testing reproducibility and validity of methods and machine-learning models for stress detection. To this end, we analyzed data from 90 participants, from four independent controlled studies, using two different types of sensors, with different study protocols and research goals. We started by evaluating the performance of models built using data from one study and tested on data from other studies. Next, we evaluated new methods to improve the performance of stress-detection models and found that our methods led to a consistent increase in performance across all studies, irrespective of the device type, sensor type, or the type of stressor. Finally, we developed and evaluated a clustering approach to determine the stressed\/not-stressed classification when applying models on data from different studies, and found that our approach performed better than selecting a threshold based on training data. This paper's thorough exploration of reproducibility in a controlled environment provides a critical foundation for deeper study of such methods, and is a prerequisite for tackling reproducibility in free-living conditions.<\/jats:p>","DOI":"10.1145\/3432220","type":"journal-article","created":{"date-parts":[[2020,12,18]],"date-time":"2020-12-18T15:39:14Z","timestamp":1608305954000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":82,"title":["Evaluating the Reproducibility of Physiological Stress Detection Models"],"prefix":"10.1145","volume":"4","author":[{"given":"Varun","family":"Mishra","sequence":"first","affiliation":[{"name":"Dartmouth College"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sougata","family":"Sen","sequence":"additional","affiliation":[{"name":"Northwestern University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Grace","family":"Chen","sequence":"additional","affiliation":[{"name":"Washington University in St. Louis"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tian","family":"Hao","sequence":"additional","affiliation":[{"name":"IBM T.J. Watson Research Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeffrey","family":"Rogers","sequence":"additional","affiliation":[{"name":"IBM T.J. Watson Research Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ching-Hua","family":"Chen","sequence":"additional","affiliation":[{"name":"IBM T.J. Watson Research Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Kotz","sequence":"additional","affiliation":[{"name":"Dartmouth College"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,12,18]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Adrenocortical responses to psychological stress and risk for hypertension. Biomedicine & pharmacotherapy 54, 5","author":"Al'Absi M","year":"2000"},{"key":"e_1_2_1_2_1","volume-title":"Stress in America","author":"APA.","year":"2019"},{"key":"e_1_2_1_3_1","volume-title":"Dunn","author":"Bent Brinnae","year":"2020"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 30th International Conference on International Conference on Machine Learning -","volume":"28","author":"Bergstra J","year":"2013"},{"key":"e_1_2_1_5_1","doi-asserted-by":"crossref","unstructured":"George E Billman. 2013. The LF\/HF ratio does not accurately measure cardiac sympatho-vagal balance. 26 pages. https:\/\/doi.org\/10.3389\/fphys.2013.00026  George E Billman. 2013. The LF\/HF ratio does not accurately measure cardiac sympatho-vagal balance. 26 pages. https:\/\/doi.org\/10.3389\/fphys.2013.00026","DOI":"10.3389\/fphys.2013.00026"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3300061.3345432"},{"key":"e_1_2_1_7_1","first-page":"1","article-title":"A Guide for Analysing Electrodermal Activity (EDA) & Skin Conductance Responses (SCRs) for Psychological Experiments","volume":"49","author":"Braithwaite JJ","year":"2013","journal-title":"Psychophysiology"},{"key":"e_1_2_1_8_1","volume-title":"Definitions of stress. Occasional paper","author":"Butler Gillian","year":"1993"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1961189.1961199"},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","unstructured":"Grace Chen Varun Mishra and Ching-Hua Chen. 2019. Temporal Factors of Listening to Music on Stress Reduction. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers (UbiComp\/ISWC '19 Adjunct). ACM New York NY USA 907--914. https:\/\/doi.org\/10.1145\/3341162.3346272  Grace Chen Varun Mishra and Ching-Hua Chen. 2019. Temporal Factors of Listening to Music on Stress Reduction. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers (UbiComp\/ISWC '19 Adjunct). ACM New York NY USA 907--914. https:\/\/doi.org\/10.1145\/3341162.3346272","DOI":"10.1145\/3341162.3346272"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/TITB.2011.2169804"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.1992.03480090092034"},{"key":"e_1_2_1_13_1","volume-title":"cvxEDA. https:\/\/github.com\/lciti\/cvxEDA. [Online","author":"Citi Luca","year":"2020"},{"key":"e_1_2_1_14_1","volume-title":"A global measure of perceived stress. Journal of Health and Social Behavior","author":"Cohen Sheldon","year":"1983"},{"key":"e_1_2_1_15_1","volume-title":"Learning from the past: approaches for reproducibility in computational neuroscience. In 20 Years of Computational Neuroscience","author":"Crook Sharon M"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3264913"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/PERCOMW.2017.7917644"},{"key":"e_1_2_1_18_1","volume-title":"https:\/\/www.empatica.com\/en-eu\/research\/e4\/. [Online","author":"Empatica","year":"2020"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2070942.2070970"},{"key":"e_1_2_1_20_1","volume-title":"Artifact Review and Badging. https:\/\/www.acm.org\/publications\/policies\/artifact-review-badging. [Online","author":"Association for Computing Machinery. 2016.","year":"2020"},{"key":"e_1_2_1_21_1","volume-title":"2014 IEEE International Conference on Image Processing (ICIP). 5961--5965","author":"Gao H."},{"key":"e_1_2_1_22_1","doi-asserted-by":"crossref","unstructured":"Maurizio Garbarino Matteo Lai Simone Tognetti Rosalind Picard and Daniel Bender. 2014. Empatica E3 - A wearable wireless multi-sensor device for real-time computerized biofeedback and data acquisition. In Proceedings of the 4th International Conference on Wireless Mobile Communication and Healthcare - \"Transforming healthcare through innovations in mobile and wireless technologies\". ICST. https:\/\/doi.org\/10.4108\/icst.mobihealth.2014.257418  Maurizio Garbarino Matteo Lai Simone Tognetti Rosalind Picard and Daniel Bender. 2014. Empatica E3 - A wearable wireless multi-sensor device for real-time computerized biofeedback and data acquisition. In Proceedings of the 4th International Conference on Wireless Mobile Communication and Healthcare - \"Transforming healthcare through innovations in mobile and wireless technologies\". ICST. https:\/\/doi.org\/10.4108\/icst.mobihealth.2014.257418","DOI":"10.4108\/icst.mobihealth.2014.257418"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/JBHI.2015.2446195"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1002\/for.3980040103"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3314400"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1183\/23120541.00006-2019"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2968219.2968306"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2017.08.006"},{"key":"e_1_2_1_29_1","first-page":"797","article-title":"cvxEDA: A convex optimization approach to electrodermal activity processing","volume":"63","author":"Greco Alberto","year":"2015","journal-title":"IEEE Transactions on Biomedical Engineering"},{"key":"e_1_2_1_30_1","unstructured":"Kristina Grifantini. 2010. Sensor Detects Emotions through the Skin. https:\/\/www.technologyreview.com\/s\/421316\/sensor-detects-emotions-through-the-skin\/  Kristina Grifantini. 2010. Sensor Detects Emotions through the Skin. https:\/\/www.technologyreview.com\/s\/421316\/sensor-detects-emotions-through-the-skin\/"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1148\/radiology.143.1.7063747"},{"key":"e_1_2_1_32_1","volume-title":"MEDINFO 2017: Precision Healthcare Through Informatics: Proceedings of the 16th World Congress on Medical and Health Informatics","volume":"245","author":"Hao Tian","year":"2018"},{"key":"e_1_2_1_33_1","volume-title":"Towards Precision Stress Management: Design and Evaluation of a Practical Wearable Sensing System for Monitoring Everyday Stress. iproc 3, 1 (22","author":"Hao Tian","year":"2017"},{"key":"e_1_2_1_34_1","volume-title":"StressHacker: Towards Practical Stress Monitoring in the Wild with Smartwatches. In AMIA Annual Symposium. Washington D.C.","author":"Hao Tian","year":"2017"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2005.848368"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.psyneuen.2008.10.026"},{"key":"e_1_2_1_37_1","volume-title":"Picard","author":"Hernandez Javier","year":"2011"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2750858.2807526"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351249"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/MPRV.2012.56"},{"key":"e_1_2_1_41_1","doi-asserted-by":"crossref","unstructured":"Sylvain Laborde Emma Mosley and Julian F Thayer. 2017. Heart rate variability and cardiac vagal tone in psychophysiological research - Recommendations for experiment planning data analysis and data reporting. 213 pages. https:\/\/doi.org\/10.3389\/fpsyg.2017.00213  Sylvain Laborde Emma Mosley and Julian F Thayer. 2017. Heart rate variability and cardiac vagal tone in psychophysiological research - Recommendations for experiment planning data analysis and data reporting. 213 pages. https:\/\/doi.org\/10.3389\/fpsyg.2017.00213","DOI":"10.3389\/fpsyg.2017.00213"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2370216.2370270"},{"key":"e_1_2_1_43_1","volume-title":"https:\/\/www.mc10inc.com\/our-products\/biostamprc. [Online","author":"RC.","year":"2020"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1001\/archinte.1993.00410180039004"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267305.3267537"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267305.3267538"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3361562"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/BHI.2014.6864335"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2010.5596746"},{"key":"e_1_2_1_50_1","volume-title":"Deep Adaptive Input Normalization for Price Forecasting using Limit Order Book Data. arXiv preprint arXiv:1902.07892","author":"Passalis Nikolaos","year":"2019"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the 10th ACM\/IEEE International Conference on Information Processing in Sensor Networks. 97--108","author":"Plarre Kurt","year":"2011"},{"key":"e_1_2_1_53_1","volume-title":"Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods","author":"Platt John C."},{"key":"e_1_2_1_54_1","volume-title":"Reproducibility vs. replicability: a brief history of a confused terminology. Frontiers in neuroinformatics 11","author":"Plesser Hans E","year":"2018"},{"key":"e_1_2_1_55_1","volume-title":"https:\/\/support.polar.com\/us-en\/support\/H7_heart_rate_sensor. [Online","author":"Polar","year":"2020"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/BSN.2018.8329691"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0026-0495(98)90321-3"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-020-1530-z"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/2750858.2806897"},{"key":"e_1_2_1_60_1","volume-title":"Stress Recognition Using Wearable Sensors and Mobile Phones. In Humaine Association Conference on Affective Computing and Intelligent Interaction. IEEE, 671--676","author":"Sano Akane","year":"2013"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858218"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3242969.3242985"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.3389\/fpubh.2017.00258"},{"key":"e_1_2_1_64_1","volume-title":"A healthy heart is not a metronome: an integrative review of the heart's anatomy and heart rate variability. Frontiers in psychology 5","author":"Shaffer Fred","year":"2014"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/2493432.2493508"},{"key":"e_1_2_1_66_1","first-page":"1","article-title":"Activity-aware Mental Stress Detection Using Physiological Sensors","volume":"76","author":"Sun Feng-Tso","year":"2012","journal-title":"Mobile Computing, Applications, and Services"},{"key":"e_1_2_1_67_1","volume-title":"2019 IEEE EMBS International Conference on Biomedical and Health Informatics, BHI 2019 - Proceedings. Institute of Electrical and Electronics Engineers Inc. https:\/\/doi.org\/10","author":"Umematsu Terumi","year":"2019"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/2632048.2632054"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3432220","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3432220","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3432220","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:47:09Z","timestamp":1750193229000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3432220"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12,17]]},"references-count":68,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,12,17]]}},"alternative-id":["10.1145\/3432220"],"URL":"https:\/\/doi.org\/10.1145\/3432220","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12,17]]},"assertion":[{"value":"2020-12-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}