{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,28]],"date-time":"2026-08-28T09:33:12Z","timestamp":1787909592396,"version":"build-2784847793"},"reference-count":90,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"<jats:p>Modern machine learning approaches have led to performant diagnostic models for a variety of health conditions. Several machine learning approaches, such as decision trees and deep neural networks, can, in principle, approximate any function. However, this power can be considered to be both a gift and a curse, as the propensity toward overfitting is magnified when the input data are heterogeneous and high dimensional and the output class is highly nonlinear. This issue can especially plague diagnostic systems that predict behavioral and psychiatric conditions that are diagnosed with subjective criteria. An emerging solution to this issue is crowdsourcing, where crowd workers are paid to annotate complex behavioral features in return for monetary compensation or a gamified experience. These labels can then be used to derive a diagnosis, either directly or by using the labels as inputs to a diagnostic machine learning model. This viewpoint describes existing work in this emerging field and discusses ongoing challenges and opportunities with crowd-powered diagnostic systems, a nascent field of study. With the correct considerations, the addition of crowdsourcing to human-in-the-loop machine learning workflows for the prediction of complex and nuanced health conditions can accelerate screening, diagnostics, and ultimately access to care.<\/jats:p>","DOI":"10.2196\/51138","type":"journal-article","created":{"date-parts":[[2024,1,30]],"date-time":"2024-01-30T01:52:06Z","timestamp":1706579526000},"page":"e51138","source":"Crossref","is-referenced-by-count":11,"title":["A Perspective on Crowdsourcing and Human-in-the-Loop Workflows in Precision Health"],"prefix":"10.2196","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3276-4411","authenticated-orcid":false,"given":"Peter","family":"Washington","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2024,4,11]]},"reference":[{"key":"ref1","unstructured":"HoweJThe rise of crowdsourcingWired200606012024-03-29https:\/\/sistemas-humano-computacionais.wdfiles.com\/local--files\/capitulo%3Aredes-sociais\/Howe_The_Rise_of_Crowdsourcing.pdf"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.1016\/j.amepre.2013.10.016"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.1016\/S2589-7500(20)30055-8"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.2196\/31306"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.1016\/j.ypmed.2017.07.006"},{"key":"ref6","doi-asserted-by":"publisher","DOI":"10.1016\/j.watres.2019.115209"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1707.03778"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.2810"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.7189\/jogh.07.020602"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1038\/s41563-019-0345-0"},{"key":"ref11","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-40017-9","author":"Dua, S","year":"2014","journal-title":"Machine Learning in Healthcare Informatics, Volume 56"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-018-0316-z"},{"key":"ref13","first-page":"191","volume":"2020","author":"Ghassemi, M","year":"2020","journal-title":"AMIA Jt Summits Transl Sci Proc"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.1109\/iceca.2018.8474918"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.1038\/s41551-018-0305-z"},{"key":"ref16","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-biodatasci-072018-021139"},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.1016\/j.ebiom.2015.05.009"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0037245"},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.1126\/scitranslmed.3009172"},{"key":"ref20","doi-asserted-by":"publisher","DOI":"10.1117\/12.2077884"},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.2338"},{"key":"ref22","doi-asserted-by":"publisher","DOI":"10.1145\/2159365.2159367"},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.1177\/1075547015609322"},{"key":"ref24","doi-asserted-by":"publisher","DOI":"10.1038\/nbt.2109"},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btx283"},{"key":"ref26","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2112979119"},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-023-35827-3"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1313039111"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1803.03146"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.1016\/j.tibs.2014.08.005"},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-022-01605-0"},{"key":"ref32","doi-asserted-by":"publisher","DOI":"10.1016\/S2213-2600(16)00048-5"},{"key":"ref33","doi-asserted-by":"publisher","DOI":"10.1109\/tg.2018.2877325"},{"key":"ref34","doi-asserted-by":"publisher","DOI":"10.1109\/ICHI.2018.00052"},{"key":"ref35","doi-asserted-by":"publisher","DOI":"10.1055\/s-0041-1736626"},{"key":"ref36","doi-asserted-by":"publisher","DOI":"10.1007\/s41666-018-0034-9"},{"key":"ref37","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2019.06.004"},{"key":"ref38","doi-asserted-by":"publisher","DOI":"10.1101\/2021.07.28.21260646"},{"key":"ref39","doi-asserted-by":"publisher","DOI":"10.2196\/13174"},{"key":"ref40","doi-asserted-by":"publisher","DOI":"10.2196\/26760"},{"key":"ref41","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2012.08678"},{"key":"ref42","doi-asserted-by":"publisher","DOI":"10.2196\/31830"},{"key":"ref43","doi-asserted-by":"publisher","DOI":"10.2196\/35406"},{"key":"ref44","doi-asserted-by":"publisher","DOI":"10.2196\/33771"},{"key":"ref45","doi-asserted-by":"publisher","DOI":"10.1145\/3411763.3451701"},{"key":"ref46","doi-asserted-by":"publisher","DOI":"10.1017\/s1930297500002205"},{"key":"ref47","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"ref48","doi-asserted-by":"publisher","DOI":"10.1002\/cam4.1165"},{"key":"ref49","unstructured":"RossJZaldivarAIraniLTomlinsonBWho are the turkers? Worker demographics in Amazon Mechanical TurkResearchGate20092024-03-22https:\/\/www.researchgate.net\/publication\/268427703_Who_are_the_Turkers_Worker_Demographics_in_Amazon_Mechanical_Turk"},{"key":"ref50","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"key":"ref51","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33019837"},{"key":"ref52","doi-asserted-by":"publisher","DOI":"10.1109\/cvprw.2008.4562953"},{"key":"ref53","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2014.06.044"},{"key":"ref54","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470742"},{"key":"ref55","doi-asserted-by":"publisher","DOI":"10.1145\/2627392"},{"key":"ref56","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2105.12762"},{"key":"ref57","doi-asserted-by":"publisher","DOI":"10.1007\/s10606-018-9343-z"},{"key":"ref58","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2014.08.006"},{"key":"ref59","doi-asserted-by":"publisher","DOI":"10.1108\/intr-01-2020-0008"},{"key":"ref60","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.3807"},{"key":"ref61","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0117401"},{"key":"ref62","unstructured":"ChengJManoharanMZhangYLeaseMIs there a doctor in the crowd? Diagnosis needed! (For less than $5)iConference 2015 Proceedings20152024-03-22https:\/\/www.ideals.illinois.edu\/items\/73844"},{"key":"ref63","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-020-76874-w"},{"key":"ref64","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1002705"},{"key":"ref65","doi-asserted-by":"publisher","DOI":"10.2196\/13822"},{"key":"ref66","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-021-87059-4"},{"key":"ref67","doi-asserted-by":"publisher","DOI":"10.3390\/jpm10030086"},{"key":"ref68","first-page":"14","volume":"26","author":"Washington, P","year":"2021","journal-title":"Pac Symp Biocomput"},{"key":"ref69","doi-asserted-by":"publisher","DOI":"10.2196\/13668"},{"key":"ref70","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-021-87059-4"},{"key":"ref71","doi-asserted-by":"publisher","DOI":"10.1016\/j.ibmed.2022.100056"},{"key":"ref72","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2017.11295"},{"key":"ref73","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-019-0166-1"},{"key":"ref74","doi-asserted-by":"publisher","DOI":"10.1038\/s41386-020-0771-3"},{"key":"ref75","doi-asserted-by":"publisher","DOI":"10.1038\/npp.2016.7"},{"key":"ref76","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2201.11197"},{"key":"ref77","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-31620-4","author":"Baumeister, H","year":"2019","journal-title":"Digital Phenotyping and Mobile Sensing: New Developments in Psychoinformatics"},{"key":"ref78","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-019-01346-1"},{"key":"ref79","doi-asserted-by":"publisher","DOI":"10.3390\/s131217292"},{"key":"ref80","doi-asserted-by":"publisher","DOI":"10.1155\/2019\/5931315"},{"key":"ref81","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2015.06.003"},{"key":"ref82","doi-asserted-by":"publisher","DOI":"10.1056\/nejmra1806949"},{"key":"ref83","doi-asserted-by":"publisher","DOI":"10.1109\/comst.2014.2381246"},{"key":"ref84","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/1140.001.0001","author":"Picard, RW","year":"2000","journal-title":"Affective Computing"},{"key":"ref85","doi-asserted-by":"publisher","DOI":"10.1016\/s1071-5819(03)00052-1"},{"key":"ref86","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2017.02.003"},{"key":"ref87","author":"Scherer, KR","year":"2010","journal-title":"A Blueprint for Affective Computing: a Sourcebook and Manual"},{"key":"ref88","doi-asserted-by":"publisher","DOI":"10.1007\/11573548_125"},{"key":"ref89","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2022.03.009"},{"key":"ref90","doi-asserted-by":"publisher","DOI":"10.1145\/3363560"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2024\/1\/e51138","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:34:41Z","timestamp":1782840881000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2024\/1\/e51138"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,11]]},"references-count":90,"URL":"https:\/\/doi.org\/10.2196\/51138","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.51138","asserted-by":"object"}]},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,11]]}}}