{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,13]],"date-time":"2026-03-13T00:52:15Z","timestamp":1773363135253,"version":"3.50.1"},"reference-count":48,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,6,7]],"date-time":"2019-06-07T00:00:00Z","timestamp":1559865600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,6,7]],"date-time":"2019-06-07T00:00:00Z","timestamp":1559865600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Accurately measuring sleep and its quality with polysomnography (PSG) is an expensive task. Actigraphy, an alternative, has been proven cheap and relatively accurate. However, the largest experiments conducted to date, have had only hundreds of participants. In this work, we processed the data of the recently published Multi-Ethnic Study of Atherosclerosis (MESA) Sleep study to have both PSG and actigraphy data synchronized. We propose the adoption of this publicly available large dataset, which is at least one order of magnitude larger than any other dataset, to systematically compare existing methods for the detection of sleep-wake stages, thus fostering the creation of new algorithms. We also implemented and compared state-of-the-art methods to score sleep-wake stages, which range from the widely used traditional algorithms to recent machine learning approaches. We identified among the traditional algorithms, two approaches that perform better than the algorithm implemented by the actigraphy device used in the MESA Sleep experiments. The performance, in regards to accuracy and <jats:italic>F<\/jats:italic><jats:sub>1<\/jats:sub> score of the machine learning algorithms, was also superior to the device\u2019s native algorithm and comparable to human annotation. Future research in developing new sleep-wake scoring algorithms, in particular, machine learning approaches, will be highly facilitated by the cohort used here. We exemplify this potential by showing that two particular deep-learning architectures, CNN and LSTM, among the many recently created, can achieve accuracy scores significantly higher than other methods for the same tasks.<\/jats:p>","DOI":"10.1038\/s41746-019-0126-9","type":"journal-article","created":{"date-parts":[[2019,6,7]],"date-time":"2019-06-07T10:02:44Z","timestamp":1559901764000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":85,"title":["Benchmark on a large cohort for sleep-wake classification with machine learning techniques"],"prefix":"10.1038","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7099-9716","authenticated-orcid":false,"given":"Joao","family":"Palotti","sequence":"first","affiliation":[]},{"given":"Raghvendra","family":"Mall","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6321-5242","authenticated-orcid":false,"given":"Michael","family":"Aupetit","sequence":"additional","affiliation":[]},{"given":"Michael","family":"Rueschman","sequence":"additional","affiliation":[]},{"given":"Meghna","family":"Singh","sequence":"additional","affiliation":[]},{"given":"Aarti","family":"Sathyanarayana","sequence":"additional","affiliation":[]},{"given":"Shahrad","family":"Taheri","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8165-9904","authenticated-orcid":false,"given":"Luis","family":"Fernandez-Luque","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2019,6,7]]},"reference":[{"key":"126_CR1","doi-asserted-by":"publisher","first-page":"881","DOI":"10.1136\/adc.2005.093013","volume":"91","author":"S Taheri","year":"2006","unstructured":"Taheri, S. The link between short sleep duration and obesity: we should recommend more sleep to prevent obesity. Arch. Dis. Child. 91, 881\u2013884 (2006).","journal-title":"Arch. Dis. Child."},{"key":"126_CR2","doi-asserted-by":"publisher","first-page":"833","DOI":"10.1161\/01.HYP.0000217362.34748.e0","volume":"47","author":"JE Gangwisch","year":"2006","unstructured":"Gangwisch, J. E. et al. Short sleep duration as a risk factor for hypertension: analyses of the first national health and nutrition examination survey. Hypertension 47, 833\u2013839 (2006).","journal-title":"Hypertension"},{"key":"126_CR3","doi-asserted-by":"publisher","first-page":"608","DOI":"10.2337\/diacare.24.3.608","volume":"24","author":"H Shigeta","year":"2001","unstructured":"Shigeta, H., Shigeta, M., Nakazawa, A., Nakamura, N. & Yoshikawa, T. Lifestyle, obesity, and insulin resistance. Diabetes Care 24, 608\u2013608 (2001).","journal-title":"Diabetes Care"},{"key":"126_CR4","doi-asserted-by":"publisher","first-page":"294","DOI":"10.1016\/j.pcad.2008.10.003","volume":"51","author":"JM Mullington","year":"2009","unstructured":"Mullington, J. M., Haack, M., Toth, M., Serrador, J. M. & Meier-Ewert, H. K. Cardiovascular, inflammatory, and metabolic consequences of sleep deprivation. Prog. Cardiovasc. Dis. 51, 294\u2013302 (2009).","journal-title":"Prog. Cardiovasc. Dis."},{"key":"126_CR5","doi-asserted-by":"publisher","first-page":"201","DOI":"10.1093\/sleep\/17.3.201","volume":"17","author":"A Sadeh","year":"1994","unstructured":"Sadeh, A., Sharkey, M. & Carskadon, M. A. Activity-based sleep-wake identification: an empirical test of methodological issues. Sleep 17, 201\u2013207 (1994).","journal-title":"Sleep"},{"key":"126_CR6","doi-asserted-by":"publisher","first-page":"461","DOI":"10.1093\/sleep\/15.5.461","volume":"15","author":"RJ Cole","year":"1992","unstructured":"Cole, R. J., Kripke, D. F., Gruen, W., Mullaney, D. J. & Gillin, J. C. Automatic sleep\/wake identification from wrist activity. Sleep 15, 461\u2013469 (1992).","journal-title":"Sleep"},{"key":"126_CR7","doi-asserted-by":"publisher","first-page":"659","DOI":"10.1016\/S0031-9384(98)00213-3","volume":"65","author":"G Jean-Louis","year":"1998","unstructured":"Jean-Louis, G., Zizi, F., Von Gizycki, H. & Hauri, P. Actigraphic assessment of sleep in insomnia: application of the actigraph data analysis software (adas). Physiol. Behav. 65, 659\u2013663 (1998).","journal-title":"Physiol. Behav."},{"key":"126_CR8","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1093\/sleep\/26.1.81","volume":"26","author":"L de Souza","year":"2003","unstructured":"de Souza, L. et al. Further validation of actigraphy for sleep studies. Sleep 26, 81\u201385 (2003).","journal-title":"Sleep"},{"key":"126_CR9","doi-asserted-by":"publisher","first-page":"1291","DOI":"10.1088\/0967-3334\/25\/5\/018","volume":"25","author":"E Sazonov","year":"2004","unstructured":"Sazonov, E. et al. Activity-based sleep-wake identification in infants. Physiol. Meas. 25, 1291 (2004).","journal-title":"Physiol. Meas."},{"key":"126_CR10","doi-asserted-by":"publisher","first-page":"337","DOI":"10.1093\/sleep\/26.3.337","volume":"26","author":"M Littner","year":"2003","unstructured":"Littner, M. et al. Practice parameters for the role of actigraphy in the study of sleep and circadian rhythms: an update for 2002. Sleep 26, 337\u2013341 (2003).","journal-title":"Sleep"},{"key":"126_CR11","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1111\/j.1365-2869.2008.00706.x","volume":"18","author":"J Tilmanne","year":"2009","unstructured":"Tilmanne, J., Urbain, J., Kothare, M. V., Wouwer, A. V. & Kothare, S. V. Algorithms for sleep\u2013wake identification using actigraphy: a comparative study and new results. J. Sleep. Res. 18, 85\u201398 (2009).","journal-title":"J. Sleep. Res."},{"key":"126_CR12","doi-asserted-by":"publisher","first-page":"328","DOI":"10.1111\/j.1479-8425.2012.00578.x","volume":"10","author":"MF Hjorth","year":"2012","unstructured":"Hjorth, M. F. et al. Measure of sleep and physical activity by a single accelerometer: can a waist-worn actigraph adequately measure sleep in children? Sleep. Biol. Rhythms 10, 328\u2013335 (2012).","journal-title":"Sleep. Biol. Rhythms"},{"key":"126_CR13","unstructured":"Granovsky, L., Shalev, G., Yacovzada, N., Frank, Y. & Fine, S. Actigraphy-based sleep\/wake pattern detection using convolutional neural networks. arXiv preprint arXiv:1802.07945 (2018)."},{"key":"126_CR14","doi-asserted-by":"publisher","first-page":"1231","DOI":"10.5664\/jcsm.7230","volume":"14","author":"MT Smith","year":"2018","unstructured":"Smith, M. T. et al. Use of actigraphy for the evaluation of sleep disorders and circadian rhythm sleep-wake disorders: an american academy of sleep medicine clinical practice guideline. J. Clin. Sleep. Med. 14, 1231\u20131237 (2018).","journal-title":"J. Clin. Sleep. Med."},{"key":"126_CR15","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1016\/j.smrv.2010.10.001","volume":"15","author":"A Sadeh","year":"2011","unstructured":"Sadeh, A. The role and validity of actigraphy in sleep medicine: an update. Sleep. Med. Rev. 15, 259\u2013267 (2011).","journal-title":"Sleep. Med. Rev."},{"key":"126_CR16","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1080\/07420520801897228","volume":"25","author":"L Tonetti","year":"2008","unstructured":"Tonetti, L., Pasquini, F., Fabbri, M., Belluzzi, M. & Natale, V. Comparison of two different actigraphs with polysomnography in healthy young subjects. Chronobiol. Int. 25, 145\u2013153 (2008).","journal-title":"Chronobiol. Int."},{"key":"126_CR17","doi-asserted-by":"publisher","first-page":"1151","DOI":"10.5665\/sleep.5774","volume":"39","author":"DA Dean","year":"2016","unstructured":"Dean, D. A. et al. Scaling up scientific discovery in sleep medicine: the national sleep research resource. Sleep 39, 1151\u20131164 (2016).","journal-title":"Sleep"},{"issue":"10","key":"126_CR18","doi-asserted-by":"publisher","first-page":"1351","DOI":"10.1093\/jamia\/ocy064","volume":"25","author":"Guo-Qiang Zhang","year":"2018","unstructured":"Zhang, G. -Q. et al. The national sleep research resource: towards a sleep data commons. J. Am. Med. Infor. Assoc. 25, 1351\u20131358 (2018).","journal-title":"Journal of the American Medical Informatics Association"},{"key":"126_CR19","unstructured":"MESA: Multi-Ethnic Study of Atherosclerosis. MESA Actigraphy Scoring and Processing Guidelines. Tech. Rep. (2016). Report available at https:\/\/sleepdata.org\/datasets\/mesa\/files\/documentation\/MESA_Sleep_Actigraphy_Scoring_Manual.pdf. Accessed on 24 March, 2019."},{"key":"126_CR20","unstructured":"MESA: Multi-Ethnic Study of Atherosclerosis. MESA Exam 5-Sleep Data Documentation Guide. Tech. Rep. (2014). Report available at https:\/\/sleepdata.org\/datasets\/mesa\/files\/m\/browser\/documentation\/MESA_Sleep_Data_Documentation_Guide.pdf. Accessed on 24 March, 2019."},{"key":"126_CR21","doi-asserted-by":"publisher","first-page":"612","DOI":"10.1111\/j.1365-2869.2010.00835.x","volume":"19","author":"DF Kripke","year":"2010","unstructured":"Kripke, D. F. et al. Wrist actigraphic scoring for sleep laboratory patients: algorithm development. J. Sleep. Res. 19, 612\u2013619 (2010).","journal-title":"J. Sleep. Res."},{"key":"126_CR22","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1016\/S1389-9457(00)00098-8","volume":"2","author":"CA Kushida","year":"2001","unstructured":"Kushida, C. A. et al. Comparison of actigraphic, polysomnographic, and subjective assessment of sleep parameters in sleep-disordered patients. Sleep. Med. 2, 389\u2013396 (2001).","journal-title":"Sleep. Med."},{"key":"126_CR23","doi-asserted-by":"publisher","first-page":"1747","DOI":"10.5665\/sleep.3142","volume":"36","author":"M Marino","year":"2013","unstructured":"Marino, M. et al. Measuring sleep: accuracy, sensitivity, and specificity of wrist actigraphy compared to polysomnography. Sleep 36, 1747\u20131755 (2013).","journal-title":"Sleep"},{"key":"126_CR24","doi-asserted-by":"crossref","unstructured":"Lonini, L. et al. Wearable sensors for parkinsons disease: which data are worth collecting for training symptom detection models. Npj Digit. Med. 1 (2018). https:\/\/www.nature.com\/articles\/s41746-018-0071-z.","DOI":"10.1038\/s41746-018-0071-z"},{"key":"126_CR25","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1038\/s41746-017-0010-4","volume":"1","author":"AI Luik","year":"2018","unstructured":"Luik, A. I., Machado, P. F. & Espie, C. A. Delivering digital cognitive behavioral therapy for insomnia at scale: does using a wearable device to estimate sleep influence therapy?. Npj Digit. Med. 1, 3 (2018).","journal-title":"Npj Digit. Med."},{"key":"126_CR26","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1001953","volume":"13","author":"L Piwek","year":"2016","unstructured":"Piwek, L., Ellis, D. A., Andrews, S. & Joinson, A. The rise of consumer health wearables: promises and barriers. PLoS Med. 13, e1001953 (2016).","journal-title":"PLoS Med."},{"key":"126_CR27","doi-asserted-by":"publisher","first-page":"871","DOI":"10.1093\/aje\/kwf113","volume":"156","author":"DE Bild","year":"2002","unstructured":"Bild, D. E. et al. Multi-ethnic study of atherosclerosis: objectives and design. Am. J. Epidemiol. 156, 871\u2013881 (2002).","journal-title":"Am. J. Epidemiol."},{"key":"126_CR28","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1093\/sleep\/5.4.389","volume":"5","author":"JB Webster","year":"1982","unstructured":"Webster, J. B., Kripke, D. F., Messin, S., Mullaney, D. J. & Wyborney, G. An activity-based sleep monitor system for ambulatory use. Sleep 5, 389\u2013399 (1982).","journal-title":"Sleep"},{"key":"126_CR29","doi-asserted-by":"publisher","first-page":"342","DOI":"10.1093\/sleep\/26.3.342","volume":"26","author":"S Ancoli-Israel","year":"2003","unstructured":"Ancoli-Israel, S. et al. The role of actigraphy in the study of sleep and circadian rhythms. Sleep 26, 342\u2013392 (2003).","journal-title":"Sleep"},{"key":"126_CR30","doi-asserted-by":"publisher","first-page":"1497","DOI":"10.5665\/sleep.4998","volume":"38","author":"SR Patel","year":"2015","unstructured":"Patel, S. R. et al. Reproducibility of a standardized actigraphy scoring algorithm for sleep in a us hispanic\/latino population. Sleep 38, 1497\u20131503 (2015).","journal-title":"Sleep"},{"key":"126_CR31","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1139\/apnm-2013-0173","volume":"39","author":"C Tudor-Locke","year":"2013","unstructured":"Tudor-Locke, C., Barreira, T. V., Schuna, J. M. Jr, Mire, E. F. & Katzmarzyk, P. T. Fully automated waist-worn accelerometer algorithm for detecting children\u2019s sleep-period time separate from 24-h physical activity or sedentary behaviors. Appl. Physiol., Nutr. Metab. 39, 53\u201357 (2013).","journal-title":"Appl. Physiol., Nutr. Metab."},{"key":"126_CR32","unstructured":"Oakley, N. Validation with Polysomnography of the Sleepwatch Sleep\/wake Scoring Algorithm Used by the Actiwatch Activity Monitoring System. (Technical report to Mini Mitter, Cambridge Neurotechnology, 1997)."},{"key":"126_CR33","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1016\/S0031-9384(00)00355-3","volume":"72","author":"G Jean-Louis","year":"2001","unstructured":"Jean-Louis, G., Kripke, D. F., Cole, R. J., Assmus, J. D. & Langer, R. D. Sleep detection with an accelerometer actigraph: comparisons with polysomnography. Physiol. Behav. 72, 21\u201328 (2001).","journal-title":"Physiol. Behav."},{"key":"126_CR34","first-page":"1995","volume":"3361","author":"Y LeCun","year":"1995","unstructured":"LeCun, Y. et al. Convolutional networks for images, speech, and time series. Handb. Brain Theory Neural Netw. 3361, 1995 (1995).","journal-title":"Handb. Brain Theory Neural Netw."},{"key":"126_CR35","doi-asserted-by":"crossref","unstructured":"McCullagh, P. & Nelder, J. Generalized Linear Models 2nd edn. (Chapman & Hall, Boca Raton, Florida, 1989).","DOI":"10.1007\/978-1-4899-3242-6"},{"key":"126_CR36","first-page":"273","volume":"20","author":"C Cortes","year":"1995","unstructured":"Cortes, C. & Vapnik, V. Support-vector networks. Mach. Learn. 20, 273\u2013297 (1995).","journal-title":"Mach. Learn."},{"key":"126_CR37","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/s10994-006-6226-1","volume":"63","author":"P Geurts","year":"2006","unstructured":"Geurts, P., Ernst, D. & Wehenkel, L. Extremely randomized trees. Mach. Learn. 63, 3\u201342 (2006).","journal-title":"Mach. Learn."},{"key":"126_CR38","doi-asserted-by":"publisher","first-page":"277","DOI":"10.1023\/A:1007662407062","volume":"37","author":"Y Freund","year":"1999","unstructured":"Freund, Y. & Schapire, R. E. Large margin classification using the perceptron algorithm. Mach. Learn. 37, 277\u2013296 (1999).","journal-title":"Mach. Learn."},{"issue":"15","key":"126_CR39","doi-asserted-by":"publisher","first-page":"2605","DOI":"10.1093\/bioinformatics\/bty166","volume":"34","author":"Sameer Khurana","year":"2018","unstructured":"Khurana, S. et al. Deepsol: a deep learning framework for sequence-based protein solubility prediction. Bioinformatics 34, 2605\u20132613 (2018).","journal-title":"Bioinformatics"},{"key":"126_CR40","doi-asserted-by":"crossref","unstructured":"Elbasir, A. et al. Deepcrystal: A deep learning framework for sequence-based protein crystallization prediction. Bioinformatics bty953, (2018).","DOI":"10.1109\/BIBM.2018.8621202"},{"key":"126_CR41","doi-asserted-by":"publisher","first-page":"e39","DOI":"10.1093\/nar\/gky015","volume":"46","author":"R Mall","year":"2018","unstructured":"Mall, R. et al. Rgbm: regularized gradient boosting machines for identification of the transcriptional regulators of discrete glioma subtypes. Nucl. Acids Res. 46, e39\u2013e39 (2018).","journal-title":"Nucl. Acids Res."},{"key":"126_CR42","unstructured":"Mall, R. et al. Differential community detection in paired biological networks. In Proc. 8th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, (eds Haspel, N., Cowen, L. J., Shehu, A., Kahveci, T. & Pozzi, G.) 330\u2013339 (ACM, Boston, Massachusetts, USA, 2017)."},{"key":"126_CR43","doi-asserted-by":"publisher","DOI":"10.1186\/s12918-017-0412-6","volume":"11","author":"R Mall","year":"2017","unstructured":"Mall, R., Cerulo, L., Bensmail, H., Iavarone, A. & Ceccarelli, M. Detection of statistically significant network changes in complex biological networks. BMC Syst. Biol. 11, 32 (2017).","journal-title":"BMC Syst. Biol."},{"key":"126_CR44","doi-asserted-by":"crossref","unstructured":"Sathyanarayana, A. et al. Robust automated human activity recognition and its application to sleep research. In Data Mining Workshops (ICDMW), 2016 IEEE 16th International Conference on, 495\u2013502 (IEEE, Barcelona, Spain, 2016).","DOI":"10.1109\/ICDMW.2016.0077"},{"key":"126_CR45","doi-asserted-by":"crossref","unstructured":"Gers, F. A., Schmidhuber, J. & Cummins, F. Learning to forget: Continual prediction with lstm. In International Conference on Artificial Neural Networks ICANN, 850\u2013855 (IEEE, Edinburgh, UK, 1999).","DOI":"10.1049\/cp:19991218"},{"key":"126_CR46","volume-title":"Modern Elementary Statistics","author":"JE Freund","year":"1988","unstructured":"Freund, J. E. Modern Elementary Statistics. (Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 1988)."},{"key":"126_CR47","doi-asserted-by":"publisher","first-page":"1560","DOI":"10.1093\/sleep\/27.8.1560","volume":"27","author":"J Hedner","year":"2004","unstructured":"Hedner, J. et al. A novel adaptive wrist actigraphy algorithm for sleep-wake assessment in sleep apnea patients. Sleep 27, 1560\u20131566 (2004).","journal-title":"Sleep"},{"key":"126_CR48","doi-asserted-by":"publisher","first-page":"621","DOI":"10.1016\/j.sleep.2008.07.009","volume":"10","author":"KY Chae","year":"2009","unstructured":"Chae, K. Y. et al. Evaluation of immobility time for sleep latency in actigraphy. Sleep. Med. 10, 621\u2013625 (2009).","journal-title":"Sleep. Med."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0126-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0126-9","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0126-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,17]],"date-time":"2022-12-17T18:28:24Z","timestamp":1671301704000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0126-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,6,7]]},"references-count":48,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2019,12]]}},"alternative-id":["126"],"URL":"https:\/\/doi.org\/10.1038\/s41746-019-0126-9","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,6,7]]},"assertion":[{"value":"17 January 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 May 2019","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 June 2019","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"50"}}