{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T13:31:59Z","timestamp":1784899919493,"version":"3.55.0"},"reference-count":38,"publisher":"Public Library of Science (PLoS)","issue":"3","license":[{"start":{"date-parts":[[2023,3,28]],"date-time":"2023-03-28T00:00:00Z","timestamp":1679961600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000864","name":"Michael J. Fox Foundation for Parkinson's Research","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000864","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosdigitalhealth.org"],"crossmark-restriction":false},"short-container-title":["PLOS Digit Health"],"abstract":"<jats:p>One of the promising opportunities of digital health is its potential to lead to more holistic understandings of diseases by interacting with the daily life of patients and through the collection of large amounts of real-world data. Validating and benchmarking indicators of disease severity in the home setting is difficult, however, given the large number of confounders present in the real world and the challenges in collecting ground truth data in the home. Here we leverage two datasets collected from patients with Parkinson\u2019s disease, which couples continuous wrist-worn accelerometer data with frequent symptom reports in the home setting, to develop digital biomarkers of symptom severity. Using these data, we performed a public benchmarking challenge in which participants were asked to build measures of severity across 3 symptoms (on\/off medication, dyskinesia, and tremor). 42 teams participated and performance was improved over baseline models for each subchallenge. Additional ensemble modeling across submissions further improved performance, and the top models validated in a subset of patients whose symptoms were observed and rated by trained clinicians.<\/jats:p>","DOI":"10.1371\/journal.pdig.0000208","type":"journal-article","created":{"date-parts":[[2023,3,28]],"date-time":"2023-03-28T17:49:27Z","timestamp":1680025767000},"page":"e0000208","update-policy":"https:\/\/doi.org\/10.1371\/journal.pdig.corrections_policy","source":"Crossref","is-referenced-by-count":9,"title":["Developing better digital health measures of Parkinson\u2019s disease using free living data and a crowdsourced data analysis challenge"],"prefix":"10.1371","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1033-0954","authenticated-orcid":true,"given":"Solveig K.","family":"Sieberts","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Henryk","family":"Borzymowski","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuanfang","family":"Guan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yidi","family":"Huang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ayala","family":"Matzner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5562-3115","authenticated-orcid":true,"given":"Alex","family":"Page","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2999-4083","authenticated-orcid":true,"given":"Izhar","family":"Bar-Gad","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brett","family":"Beaulieu-Jones","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuval","family":"El-Hanani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jann","family":"Goschenhofer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Monica","family":"Javidnia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark S.","family":"Keller","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yan-chak","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8171-2581","authenticated-orcid":true,"given":"Mohammed","family":"Saqib","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Greta","family":"Smith","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ana","family":"Stanescu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Charles S.","family":"Venuto","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert","family":"Zielinski","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"name":"the BEAT-PD DREAM Challenge Consortium","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arun","family":"Jayaraman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luc J. W.","family":"Evers","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luca","family":"Foschini","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alex","family":"Mariakakis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gaurav","family":"Pandey","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicholas","family":"Shawen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Phil","family":"Synder","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Larsson","family":"Omberg","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2023,3,28]]},"reference":[{"key":"pdig.0000208.ref001","first-page":"1","article-title":"Remote smartphone monitoring of Parkinson\u2019s disease and individual response to therapy","author":"L Omberg","year":"2021","journal-title":"Nat Biotechnol"},{"key":"pdig.0000208.ref002","unstructured":"Now you see me: The voice of people with Parkinson\u2019s in clinical trials. AD\/PD; 2019. Available: https:\/\/www.youtube.com\/watch?v=Gj1iXHUWhSk"},{"key":"pdig.0000208.ref003","unstructured":"FDA rejects Verily\u2019s filing for wearable Parkinson\u2019s disease clinical trial tool. In: MobiHealthNews [Internet]. 16 Jun 2021 [cited 7 Oct 2021]. Available: https:\/\/www.mobihealthnews.com\/news\/fda-rejects-verilys-filing-wearable-parkinsons-disease-clinical-trial-tool"},{"key":"pdig.0000208.ref004","doi-asserted-by":"crossref","first-page":"2293","DOI":"10.1056\/NEJMsb1609216","article-title":"Real-World Evidence\u2014What Is It and What Can It Tell Us?","volume":"375","author":"RE Sherman","year":"2016","journal-title":"N Engl J Med"},{"key":"pdig.0000208.ref005","first-page":"429","article-title":"Systematic Review Looking at the Use of Technology to Measure Free-Living Symptom and Activity Outcomes in Parkinson\u2019s Disease in the Home or a Home-like Environment.","volume":"10","author":"C Morgan","year":"2020","journal-title":"J Park Dis."},{"key":"pdig.0000208.ref006","doi-asserted-by":"crossref","first-page":"e26006","DOI":"10.2196\/26006","article-title":"Smartphone-Based VO2max Measurement With Heart Snapshot in Clinical and Real-world Settings With a Diverse Population: Validation Study.","volume":"9","author":"DE Webster","year":"2021","journal-title":"JMIR MHealth UHealth"},{"key":"pdig.0000208.ref007","doi-asserted-by":"crossref","DOI":"10.1038\/s41746-021-00414-7","article-title":"Crowdsourcing digital health measures to predict Parkinson\u2019s disease severity: the Parkinson\u2019s Disease Digital Biomarker DREAM Challenge.","volume":"4","author":"SK Sieberts","year":"2021","journal-title":"NPJ Digit Med."},{"key":"pdig.0000208.ref008","doi-asserted-by":"crossref","first-page":"636","DOI":"10.1016\/j.cels.2021.05.015","article-title":"Advances in systems biology modeling: 10 years of crowdsourcing DREAM challenges.","volume":"12","author":"P Meyer","year":"2021","journal-title":"Cell Syst."},{"key":"pdig.0000208.ref009","volume-title":"Clinician-Input Study (CIS-PD): how the Fox Wearable Companion Application can influence treatment and care in Parkinson\u2019s disease (P3. 048).","author":"M Daeschler","year":"2018"},{"key":"pdig.0000208.ref010","first-page":"1","article-title":"Feasibility and utility of a clinician dashboard from wearable and mobile application Parkinson\u2019s disease data.","volume":"2","author":"JJ Elm","year":"2019","journal-title":"NPJ Digit Med"},{"key":"pdig.0000208.ref011","doi-asserted-by":"crossref","first-page":"e19068","DOI":"10.2196\/19068","article-title":"Real-Life Gait Performance as a Digital Biomarker for Motor Fluctuations: The Parkinson@Home Validation Study","volume":"22","author":"LJ Evers","year":"2020","journal-title":"J Med Internet Res"},{"key":"pdig.0000208.ref012","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1097\/00002826-200003000-00003","article-title":"A home diary to assess functional status in patients with Parkinson\u2019s disease with motor fluctuations and dyskinesia.","volume":"23","author":"RA Hauser","year":"2000","journal-title":"Clin Neuropharmacol."},{"key":"pdig.0000208.ref013","unstructured":"Ending Parkinson\u2019s Disease. 2020. Available: https:\/\/www.hachette.co.uk\/titles\/bastiaan-r-bloem\/ending-parkinsons-disease\/9781541724501\/"},{"key":"pdig.0000208.ref014","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-019-0178-x","article-title":"Detecting the impact of subject characteristics on machine learning-based diagnostic applications.","volume":"2","author":"E Chaibub Neto","year":"2019","journal-title":"Npj Digit Med"},{"key":"pdig.0000208.ref015","first-page":"1","volume-title":"Supporting Coping with Parkinson\u2019s Disease Through Self Tracking. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems","author":"SR Mishra","year":"2019"},{"key":"pdig.0000208.ref016","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random Forests.","volume":"45","author":"L. Breiman","year":"2001","journal-title":"Mach Learn"},{"key":"pdig.0000208.ref017","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1038\/s42256-019-0138-9","article-title":"From local explanations to global understanding with explainable AI for trees","volume":"2","author":"SM Lundberg","year":"2020","journal-title":"Nat Mach Intell"},{"key":"pdig.0000208.ref018","doi-asserted-by":"crossref","DOI":"10.1126\/scitranslmed.abd7865","article-title":"Smartwatch inertial sensors continuously monitor real-world motor fluctuations in Parkinson\u2019s disease.","volume":"13","author":"R Powers","year":"2021","journal-title":"Sci Transl Med."},{"key":"pdig.0000208.ref019","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1016\/j.ymeth.2015.08.016","article-title":"Predicting protein function and other biomedical characteristics with heterogeneous ensembles.","volume":"93","author":"S Whalen","year":"2016","journal-title":"Methods"},{"key":"pdig.0000208.ref020","first-page":"288","article-title":"Learning parsimonious ensembles for unbalanced computational genomics problems","volume":"22","author":"A Stanescu","year":"2017","journal-title":"Pac Symp Biocomput"},{"key":"pdig.0000208.ref021","doi-asserted-by":"crossref","DOI":"10.12688\/f1000research.16415.1","article-title":"Large-scale protein function prediction using heterogeneous ensembles.","volume":"7","author":"L Wang","year":"2018","journal-title":"F1000Research."},{"key":"pdig.0000208.ref022","author":"A Stanescu","year":"2021","journal-title":"Developing parsimonious ensembles using predictor diversity within a reinforcement learning framework"},{"key":"pdig.0000208.ref023","doi-asserted-by":"crossref","first-page":"2129","DOI":"10.1002\/mds.22340","article-title":"Movement Disorder Society-sponsored revision of the Unified Parkinson\u2019s Disease Rating Scale (MDS-UPDRS): Scale presentation and clinimetric testing results.","volume":"23","author":"CG Goetz","year":"2008","journal-title":"Mov Disord"},{"key":"pdig.0000208.ref024","doi-asserted-by":"crossref","DOI":"10.3390\/s20205817","article-title":"Parkinson\u2019s Disease Tremor Detection in the Wild Using Wearable Accelerometers.","volume":"20","author":"R San-Segundo","year":"2020","journal-title":"Sensors"},{"key":"pdig.0000208.ref025","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1016\/j.neucom.2018.03.067","article-title":"Time Series FeatuRe Extraction on basis of Scalable Hypothesis tests (tsfresh\u2013A Python package).","volume":"307","author":"M Christ","year":"2018","journal-title":"Neurocomputing"},{"key":"pdig.0000208.ref026","doi-asserted-by":"crossref","first-page":"955","DOI":"10.1001\/archneur.1989.00520450025014","article-title":"Accuracy of Self-Reported Disability in Patients With Parkinsonism","volume":"46","author":"RG Brown","year":"1989","journal-title":"Arch Neurol"},{"key":"pdig.0000208.ref027","doi-asserted-by":"crossref","first-page":"160011","DOI":"10.1038\/sdata.2016.11","article-title":"The mPower study, Parkinson disease mobile data collected using ResearchKit.","volume":"3","author":"BM Bot","year":"2016","journal-title":"Sci Data"},{"key":"pdig.0000208.ref028","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-018-0071-z","article-title":"Wearable sensors for Parkinson\u2019s disease: which data are worth collecting for training symptom detection models.","volume":"1","author":"L Lonini","year":"2018","journal-title":"NPJ Digit Med"},{"key":"pdig.0000208.ref029","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12984-020-00684-4","article-title":"Role of data measurement characteristics in the accurate detection of Parkinson\u2019s disease symptoms using wearable sensors.","volume":"17","author":"N Shawen","year":"2020","journal-title":"J Neuroengineering Rehabil."},{"key":"pdig.0000208.ref030","doi-asserted-by":"crossref","first-page":"12460","DOI":"10.1038\/ncomms12460","article-title":"Crowdsourced assessment of common genetic contribution to predicting anti-TNF treatment response in rheumatoid arthritis.","volume":"7","author":"SK Sieberts","year":"2016","journal-title":"Nat Commun"},{"key":"pdig.0000208.ref031","doi-asserted-by":"crossref","first-page":"389","DOI":"10.1023\/A:1012487302797","article-title":"Gene Selection for Cancer Classification using Support Vector Machines.","volume":"46","author":"I Guyon","year":"2002","journal-title":"Mach Learn."},{"key":"pdig.0000208.ref032","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python.","volume":"12","author":"F Pedregosa","year":"2011","journal-title":"J Mach Learn Res."},{"key":"pdig.0000208.ref033","first-page":"18","volume-title":"Ensemble selection from libraries of models. Proceedings of the twenty-first international conference on Machine learning.","author":"R Caruana","year":"2004"},{"key":"pdig.0000208.ref034","doi-asserted-by":"crossref","first-page":"828","DOI":"10.1109\/ICDM.2006.76","article-title":"Getting the Most Out of Ensemble Selection.","author":"R Caruana","year":"2006","journal-title":"Sixth International Conference on Data Mining (ICDM\u201906)."},{"key":"pdig.0000208.ref035","volume-title":"Reinforcement Learning: An Introduction","author":"RS Sutton","year":"1998","edition":"2"},{"key":"pdig.0000208.ref036","doi-asserted-by":"crossref","first-page":"525","DOI":"10.22237\/jmasm\/1067646360","article-title":"JMASM9: Converting Kendall\u2019s Tau For Correlational Or Meta-Analytic Analyses.","volume":"2","author":"DA Walker","year":"2003","journal-title":"J Mod Appl Stat Methods."},{"key":"pdig.0000208.ref037","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1136\/ebmental-2019-300117","article-title":"How to perform a meta-analysis with R: a practical tutorial.","volume":"22","author":"S Balduzzi","year":"2019","journal-title":"Evid Based Ment Health"},{"key":"pdig.0000208.ref038","volume-title":"Statistical Methods for Research Workers","author":"RA Fisher","year":"1932","edition":"4"}],"container-title":["PLOS Digital Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0000208","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,3,28]],"date-time":"2023-03-28T17:50:15Z","timestamp":1680025815000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0000208"}},"subtitle":[],"editor":[{"given":"Crina","family":"Grosan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2023,3,28]]},"references-count":38,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2023,3,28]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pdig.0000208","relation":{},"ISSN":["2767-3170"],"issn-type":[{"value":"2767-3170","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,28]]}}}