{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,27]],"date-time":"2026-07-27T08:01:12Z","timestamp":1785139272807,"version":"3.55.0"},"reference-count":70,"publisher":"SAGE Publications","license":[{"start":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T00:00:00Z","timestamp":1784764800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T00:00:00Z","timestamp":1784764800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Journal of Parkinson\u2019s Disease"],"abstract":"<jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To review the application of crowdsourcing and machine learning contests in Parkinson's disease (PD) research, identify best practices for successful implementation, and highlight future opportunities.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>This paper analyzes the landscape of crowdsourcing in PD research through a literature survey and a comparative case study of two major machine learning contests: the MJFF Freezing of Gait (FOG) Challenge and the AMP PD Proteomics Challenge. We also describe a taxonomy of crowdsourcing projects and a framework of success characteristics for machine learning contest design.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The analysis of previous crowdsourcing and machine learning contests revealed that contest success is highly dependent on specific design factors. The FOG challenge, which addressed a \u201csolvable but not yet solved\u201d problem with a suitable scoring metric, successfully produced a high-performing algorithm with real-world clinical value. In contrast, the Proteomics challenge did not yield biologically meaningful results, as winning models bypassed the core proteomic data, highlighting issues of data signal and metric selection. The review also identified underutilized crowdsourcing approaches in PD research, including gamification and community-based open-source development.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Machine learning contests offer a powerful, open-science-aligned method to address complex problems in PD. Success requires careful design, particularly a solvable problem and an appropriate scoring metric. There is significant potential to expand the use of diverse crowdsourcing techniques to accelerate progress in PD research and clinical care.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/1877718x261452116","type":"journal-article","created":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T03:39:55Z","timestamp":1784864395000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":0,"title":["Crowdsourcing and machine learning contests in Parkinson's disease research - when do they work?"],"prefix":"10.1177","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-5874-1593","authenticated-orcid":false,"given":"Leslie C","family":"Kirsch","sequence":"first","affiliation":[{"name":"The Michael J. Fox Foundation for Parkinson's Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeffrey M","family":"Hausdorff","sequence":"additional","affiliation":[{"name":"Center for the Study of Movement, Cognition and Mobility, Neurological Institute, Tel Aviv Medical Center, Tel Aviv, Israel"},{"name":"Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel"},{"name":"Department of Physical Therapy, Gray Faculty of Medical &amp; Health Sciences, Tel Aviv University, Tel Aviv, Israel"},{"name":"Department of Orthopedic Surgery and Rush Alzheimer's Disease Center, Rush University Medical Center, Chicago, IL USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8771-2649","authenticated-orcid":false,"given":"Victoria J","family":"Dardov","sequence":"additional","affiliation":[{"name":"Technome.io, Herndon, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2026,7,23]]},"reference":[{"key":"e_1_3_3_2_2","volume-title":"Crowdsourcing: Why the power of the crowd is driving the future of business","author":"Howe J","year":"2009","unstructured":"Howe J. Crowdsourcing: Why the power of the crowd is driving the future of business. New York, NY: Three Rivers Press, 2009."},{"issue":"6","key":"e_1_3_3_3_2","first-page":"1","article-title":"A taxonomy of crowdsourcing based on task complexity","volume":"40","author":"Nakatsu RT","year":"2014","unstructured":"Nakatsu RT, Grossman EB, Iacovou CL. A taxonomy of crowdsourcing based on task complexity. J Info Sci 2014; 40(6): 1\u201313.","journal-title":"J Info Sci"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.technovation.2014.01.011"},{"key":"e_1_3_3_5_2","unstructured":"Howe J. The rise of crowdsourcing. Wired Magazine 1 June 2006."},{"key":"e_1_3_3_6_2","doi-asserted-by":"crossref","unstructured":"Bennett J Lanning S. The Netflix Prize. Proceedings of KDD Cup and Workshop 2007 2007 August 12 San Jose California.","DOI":"10.1145\/1345448.1345459"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-021-03819-2"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1038\/nbt.3051"},{"issue":"6","key":"e_1_3_3_9_2","first-page":"1","article-title":"A taxonomy of crowdsourcing based on task complexity","volume":"40","author":"Nakatsu RT","year":"2014","unstructured":"Nakatsu RT, Grossman EB, Iacovou CL. A taxonomy of crowdsourcing based on task complexity. J Info Sci 2014; 40(6): 1\u201313.","journal-title":"J Info Sci"},{"key":"e_1_3_3_10_2","unstructured":"Amazon. Amazon mechanical turk https:\/\/www.mturk.com\/ (nd accessed 11 April 2025)."},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2018.06.032"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature09304"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMsr1809937"},{"key":"e_1_3_3_14_2","first-page":"1","article-title":"Consumer behavior and exposure to parabens, bisphenols, triclosan, dichlorophenols, and benzophenone-3: Results from a crowdsourced biomonitoring study","volume":"230","author":"Dodson RE","year":"2020","unstructured":"Dodson RE, Boronow KE, Susmann H. Consumer behavior and exposure to parabens, bisphenols, triclosan, dichlorophenols, and benzophenone-3: Results from a crowdsourced biomonitoring study. Intl J Hyg Env Hea 2020; 230: 1\u201313.","journal-title":"Intl J Hyg Env Hea"},{"key":"e_1_3_3_15_2","unstructured":"Rheapley. Case Studies: Allina Health. https:\/\/rheaply.com\/resources\/case-studies\/allina-health\/ (2024 accessed 27 May 2025)."},{"key":"e_1_3_3_16_2","unstructured":"Coleridge Initiative. https:\/\/www.kaggle.com\/c\/coleridgeinitiative-show-us-the-data (2021 accessed 22 April 2025)."},{"key":"e_1_3_3_17_2","unstructured":"St. Jude Research. St. Judge Research Biohackathon. https:\/\/www.stjude.org\/research\/why-st-jude\/biohackathon.html (nd accessed 27 May 2025)."},{"key":"e_1_3_3_18_2","unstructured":"Wazoku. The story of InnoCentive https:\/\/www.wazokucrowd.com\/story-of-innocentive\/ (nd accessed 11 April 2025)."},{"issue":"139","key":"e_1_3_3_19_2","first-page":"1","article-title":"Meeting the challenge of genomic analysis: A collaboratively developed workshop for pangenomics and topological data analysis","volume":"00","author":"Contreras-Peruyero H","year":"2024","unstructured":"Contreras-Peruyero H, Guerrero-Flores S, Zirion-Martinez C. Meeting the challenge of genomic analysis: A collaboratively developed workshop for pangenomics and topological data analysis. Bioinf Adv 2024; 00(vbae139): 1\u20139.","journal-title":"Bioinf Adv"},{"key":"e_1_3_3_20_2","first-page":"25","article-title":"A standards organization for open and FAIR neuroscience: the international neuroinformatics coordinating facility","volume":"20","author":"Abrams MB","year":"2022","unstructured":"Abrams MB, Bjaalie JG, Das S, et al. A standards organization for open and FAIR neuroscience: the international neuroinformatics coordinating facility. Neuron 2022; 20: 25\u201336.","journal-title":"Neuron"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1038\/nbt.2495"},{"key":"e_1_3_3_22_2","unstructured":"Kaggle. setup guide: https:\/\/www.kaggle.com\/community-competitions-setup-guide."},{"key":"e_1_3_3_23_2","unstructured":"Carlens H. State of competitive machine learning in 2022. ML Contests Research https:\/\/mlcontests.com\/state-of-competitive-machine-learning-2022\/ (2023 accessed 11 April 2025)."},{"key":"e_1_3_3_24_2","unstructured":"Carlens H. State of competitive machine learning in 2024. ML Contests Research https:\/\/mlcontests.com\/state-of-machine-learning-competitions-2024. (2025 accessed 11 April 2025)."},{"key":"e_1_3_3_25_2","unstructured":"Kaggle. Kaggle Rankings. https:\/\/www.kaggle.com\/rankings. (nd accessed 27 May 2025)."},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-021-00444-8"},{"key":"e_1_3_3_27_2","unstructured":"Pavao A. Methodology for design and analysis of machine learning competitions. PhD Thesis. University Paris-Orsay: Saclay2023."},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1038\/nbt.2495"},{"key":"e_1_3_3_29_2","unstructured":"Alzheimer\u2019s Disease Data Initiative. AD Data Initiative announces results of inaugural data challenge. (nd accessed 22 April 2025) https:\/\/www.alzheimersdata.org\/news\/ad-data-initiative-announces-results-of-inaugural-data-challenge."},{"key":"e_1_3_3_30_2","unstructured":"DrivenData. PREPARE: Pioneering Research for Early Prediction of Alzheimer\u2019s and Related Dementias EUREKA Challenge. (nd. accessed 22 April 2025) https:\/\/www.drivendata.org\/competitions\/group\/nih-nia-alzheimers-adrd-competition\/."},{"key":"e_1_3_3_31_2","volume-title":"Artificial Intelligence for High Energy Physics","author":"Rousseau D","year":"2023","unstructured":"Rousseau D, Ustyuzhanin A. Machine learning scientific competitions and datasets. Artificial Intelligence for High Energy Physics 2023. https:\/\/doi.org\/10.48550\/arXiv.2012.08520."},{"key":"e_1_3_3_32_2","unstructured":"Kaggle. Kaggle\u00ae Competitions Best-Practice Guidelines. Nd."},{"key":"e_1_3_3_33_2","unstructured":"Lipstein G Bull P Dorne E et al. 10 Takeaways from 10 years of data science for social good. (2024 accessed 11 April 2025) https:\/\/drivendata.co\/blog\/10-years-of-data-science-for-social-good."},{"key":"e_1_3_3_34_2","unstructured":"Kaggle. How to Use Kaggle: Competitions Setup. https:\/\/www.kaggle.com\/docs\/competitions-setup (nd accessed 27 May 2025)."},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.techfore.2016.11.015"},{"key":"e_1_3_3_36_2","volume-title":"The ladder: A reliable leaderboard for machine learning competitions","author":"Hardt M","unstructured":"Hardt M, Blum A. The ladder: A reliable leaderboard for machine learning competitions. In: Proceedings of the 32nd International Conference on Machine Learning 2015: JMLR: W&CP volume 37, Lille, France."},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.24251\/HICSS.2021.599"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1044\/2023_JSLHR-22-00694"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1044\/2015_AJSLP-15-0059"},{"key":"e_1_3_3_40_2","first-page":"160011","article-title":"The mPower study, Parkinson disease mobile data collected using ResearchKit","volume":"3","author":"Bot BM","year":"2016","unstructured":"Bot BM, Suver C, Neto EC, et al. The mPower study, Parkinson disease mobile data collected using ResearchKit. Nature Sci Data 2016; 3: 160011.","journal-title":"Nature Sci Data"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1088\/2057-1976\/ab39a8"},{"key":"e_1_3_3_42_2","first-page":"1","article-title":"Fox Insight at 5 years - a cohort of 54,000 participants contributing longitudinal patient-reported outcome, genetic, and microbiome data relating to Parkinson\u2019s disease","volume":"11","author":"Gottesman J","year":"2024","unstructured":"Gottesman J, Karim Y, Forbes J, et al. Fox Insight at 5 years - a cohort of 54,000 participants contributing longitudinal patient-reported outcome, genetic, and microbiome data relating to Parkinson\u2019s disease. Nat Sci Data 2024; 11: 1\u201312.","journal-title":"Nat Sci Data"},{"key":"e_1_3_3_43_2","first-page":"1","article-title":"Parkinson\u2019s Progression Markers Initiative (PPMI) online expands biomarker research in Parkinson\u2019s disease (PD)","volume":"98","author":"Brown E","year":"2022","unstructured":"Brown E, Alonso D, Chahine L, et al. Parkinson\u2019s Progression Markers Initiative (PPMI) online expands biomarker research in Parkinson\u2019s disease (PD). Neur 2022; 98: 1\u20131.","journal-title":"Neur"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2023.103062"},{"key":"e_1_3_3_45_2","first-page":"1","article-title":"The IPDGC\/GP2 Hackathon - an open science event for training in data science, genomics, and collaboration using Parkinson\u2019s disease data","volume":"9","author":"Leonard HL","year":"2023","unstructured":"Leonard HL, Murtadha R, Martinez-Carrasco A, et al. The IPDGC\/GP2 Hackathon - an open science event for training in data science, genomics, and collaboration using Parkinson\u2019s disease data. Npj PD 2023; 9: 1\u201333.","journal-title":"Npj PD"},{"key":"e_1_3_3_46_2","unstructured":"The Michael J. Fox Foundation for Parkinson\u2019s Research. MJFF Data Community of Practice Makes it Easier to Work with PD Data. https:\/\/www.michaeljfox.org\/news\/mjff-data-community-practice-makes-it-easier-work-pd-data (2025 accessed 27 May 2025)."},{"key":"e_1_3_3_47_2","unstructured":"Roussos G. Introducing PDkit a Python toolkit for analysing data from mobile apps and wearables. https:\/\/rcop.michaeljfox.org\/t\/introducing-pdkit-a-python-toolkit-for-analysing-data-from-mobile-apps-and-wearables\/912 (2025 access 27 May 2025)."},{"key":"e_1_3_3_48_2","unstructured":"U.S. Food and Drug Administration. Digitally-derived endpoints for freezing-of-gait detection (DEFoGD) challenge. https:\/\/precision.fda.gov\/challenges\/33\/results (2024 accessed 9 April 2025)."},{"key":"e_1_3_3_49_2","unstructured":"Howard A Salomon A Gazit E et al. Parkinson\u2019s freezing of gait prediction. https:\/\/www.kaggle.com\/competitions\/tlvmc-parkinsons-freezing-gait-prediction\/ (2023 accessed 9 April 2025)."},{"key":"e_1_3_3_50_2","unstructured":"Kirsch LC Dane S Adam S et al. AMP-Parkinson\u2019s Disease progression prediction (2023 accessed 9 April 2025) https:\/\/www.kaggle.com\/competitions\/amp-parkinsons-disease-progression-prediction\/overview."},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000208"},{"key":"e_1_3_3_52_2","first-page":"1","article-title":"Crowdsourcing digital health measures to predict Parkinson\u2019s disease severity: the Parkinson\u2019s Disease Digital Biomarker DREAM challenge","volume":"53","author":"Sieberts SK","year":"2021","unstructured":"Sieberts SK, Schaff J, Duda M, et al. Crowdsourcing digital health measures to predict Parkinson\u2019s disease severity: the Parkinson\u2019s Disease Digital Biomarker DREAM challenge. Npj Digi Med 2021; 53: 1\u201353.","journal-title":"Npj Digi Med"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1002\/mds.22340"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ddtec.2021.06.006"},{"key":"e_1_3_3_55_2","unstructured":"AMP-PD Progression Prediction: Discussion. https:\/\/www.kaggle.com\/c\/amp-parkinsons-disease-progression-prediction\/discussion (nd accessed 6 June 2025)."},{"key":"e_1_3_3_56_2","unstructured":"AMP-PD Progression Prediction: 1st Place Solution. https:\/\/www.kaggle.com\/competitions\/amp-parkinsons-disease-progression-prediction\/discussion\/411505 (nd accessed 6 June 2025)."},{"key":"e_1_3_3_57_2","unstructured":"AMP-PD Progression Prediction: Summarizing the current situation. https:\/\/www.kaggle.com\/competitions\/amp-parkinsons-disease-progression-prediction\/discussion\/392894 (nd accessed 6 June 2025)."},{"key":"e_1_3_3_58_2","unstructured":"AMP PD. Untargeted Proteomics Data. https:\/\/amp-pd.org\/data\/untargeted-proteomics-data (nd access 6 June 2025)."},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1002\/mds.20115"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1474-4422(11)70143-0"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1002\/mdc3.13636"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1186\/s40035-022-00298-x"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1177\/1877718X241301065"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-49027-0"},{"key":"e_1_3_3_65_2","unstructured":"Kaggle. SMAPE: A good evaluation metric? https:\/\/www.kaggle.com\/competitions\/amp-parkinsons-disease-progression-prediction\/discussion\/390682 (2023 accessed 27 May 2025)."},{"key":"e_1_3_3_66_2","unstructured":"Thingiverse. Thingiverse homepage. https:\/\/www.thingiverse.com\/ (accessed 27 May 2025)."},{"key":"e_1_3_3_67_2","unstructured":"Johnson JS. How a TikTok video inspired a new pill bottle design for Parkinson\u2019s disease. https:\/\/parkinsonseurope.org\/2022\/02\/03\/tiktok-pill-bottle-parkinsons\/ (03 February 2022 accessed 27 May 2025)."},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.3390\/app12146967"},{"key":"e_1_3_3_69_2","first-page":"7","article-title":"Ensembling crowdsourced seizure prediction algorithms using long-term human intracranial EEG","volume":"61","author":"Reuben C","year":"2019","unstructured":"Reuben C, Karoly P, Freestone DR, et al. Ensembling crowdsourced seizure prediction algorithms using long-term human intracranial EEG. Epilepsia 2019; 61: 7\u201312.","journal-title":"Epilepsia"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cjca.2021.09.004"},{"key":"e_1_3_3_71_2","first-page":"2","article-title":"Why are we using black box models in AI when we don\u2019t need to?","volume":"1","author":"Rudin C","year":"2019","unstructured":"Rudin C, Radin J. Why are we using black box models in AI when we don\u2019t need to? A Lesson from an Explainable AI Competition. Harvard Data Sci Rev 2019; 1: 2.","journal-title":"A Lesson from an Explainable AI Competition. Harvard Data Sci Rev"}],"container-title":["Journal of Parkinson\u2019s Disease"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1877718X261452116","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/1877718X261452116","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1877718X261452116","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,27]],"date-time":"2026-07-27T07:19:33Z","timestamp":1785136773000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/1877718X261452116"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,23]]},"references-count":70,"alternative-id":["10.1177\/1877718X261452116"],"URL":"https:\/\/doi.org\/10.1177\/1877718x261452116","relation":{},"ISSN":["1877-7171","1877-718X"],"issn-type":[{"value":"1877-7171","type":"print"},{"value":"1877-718X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,23]]},"article-number":"1877718X261452116"}}