{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T01:09:35Z","timestamp":1784077775140,"version":"3.55.0"},"reference-count":10,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2017,5,25]],"date-time":"2017-05-25T00:00:00Z","timestamp":1495670400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2017,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objectives<\/jats:title><jats:p>Identifying all published reports of randomized controlled trials (RCTs) is an important aim, but it requires extensive manual effort to separate RCTs from non-RCTs, even using current machine learning (ML) approaches. We aimed to make this process more efficient via a hybrid approach using both crowdsourcing and ML.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>We trained a classifier to discriminate between citations that describe RCTs and those that do not. We then adopted a simple strategy of automatically excluding citations deemed very unlikely to be RCTs by the classifier and deferring to crowdworkers otherwise.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Combining ML and crowdsourcing provides a highly sensitive RCT identification strategy (our estimates suggest 95%\u201399% recall) with substantially less effort (we observed a reduction of around 60%\u201380%) than relying on manual screening alone.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>Hybrid crowd-ML strategies warrant further exploration for biomedical curation\/annotation tasks.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocx053","type":"journal-article","created":{"date-parts":[[2017,5,18]],"date-time":"2017-05-18T20:17:35Z","timestamp":1495138655000},"page":"1165-1168","source":"Crossref","is-referenced-by-count":132,"title":["Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach"],"prefix":"10.1093","volume":"24","author":[{"given":"Byron C","family":"Wallace","sequence":"first","affiliation":[{"name":"College of Computer and Information Science, Northeastern University, Boston MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anna","family":"Noel-Storr","sequence":"additional","affiliation":[{"name":"Radcliffe Department of Medicine, University of Oxford, Oxford, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Iain J","family":"Marshall","sequence":"additional","affiliation":[{"name":"Department of Primary Care and Public Health Sciences, King\u2019s College London, London, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aaron M","family":"Cohen","sequence":"additional","affiliation":[{"name":"Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland, OR, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Neil R","family":"Smalheiser","sequence":"additional","affiliation":[{"name":"Department of Psychiatry and Psychiatric Institute, University of Illinois College of Medicine, Chicago, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James","family":"Thomas","sequence":"additional","affiliation":[{"name":"EPPI-Centre, Department of Social Science, University College London, London, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2017,5,25]]},"reference":[{"issue":"1","key":"2020110612452151400_ocx053-B1","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1111\/j.1749-6632.1993.tb26345.x","article-title":"The Cochrane collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care","volume":"703","author":"Chalmers","year":"1993","journal-title":"Ann NY Acad Sci."},{"issue":"3","key":"2020110612452151400_ocx053-B2","doi-asserted-by":"crossref","first-page":"707","DOI":"10.1093\/jamia\/ocu025","article-title":"Automated confidence ranked classification of randomized controlled trial articles: an aid to evidence-based medicine","volume":"22","author":"Cohen","year":"2015","journal-title":"J Am Med Inform Assoc."},{"issue":"3","key":"2020110612452151400_ocx053-B3","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1111\/j.1471-1842.2008.00827.x","article-title":"Retrieving randomized controlled trials from Medline: a comparison of 38 published search filters","volume":"26","author":"McKibbon","year":"2009","journal-title":"Health Info Libr J."},{"key":"2020110612452151400_ocx053-B4","doi-asserted-by":"crossref","first-page":"2008","DOI":"10.1136\/bmj.d7501","article-title":"Understanding why evidence from randomised clinical trials may not be retrieved from Medline: comparison of indexed and non-indexed records","volume":"344","author":"Wieland","year":"2012","journal-title":"Brit Med J."},{"issue":"9","key":"2020110612452151400_ocx053-B5","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1371\/journal.pmed.1000326","article-title":"Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?","volume":"7","author":"Bastian","year":"2010","journal-title":"PLoS Med."},{"issue":"1","key":"2020110612452151400_ocx053-B6","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1186\/1471-2105-11-55","article-title":"Semi-automated screening of biomedical citations for systematic reviews","volume":"11","author":"Wallace","year":"2010","journal-title":"BMC Bioninformatics."},{"key":"2020110612452151400_ocx053-B7","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1197\/jamia.M1929","article-title":"Reducing workload in systematic review preparation using automated citation classification","volume":"13(2)","author":"Cohen","year":"2006","journal-title":"J Am Med Inform Assoc."},{"key":"2020110612452151400_ocx053-B8","doi-asserted-by":"crossref","DOI":"10.1007\/BFb0026683","article-title":"Text categorization with support vector machines: learning with many relevant features","volume-title":"European Conference on Machine Learning (ECML)","author":"Joachims","year":"1998"},{"key":"2020110612452151400_ocx053-B9","doi-asserted-by":"crossref","DOI":"10.1137\/1.9781611972818.16","article-title":"Who should label what? Instance allocation in multiple expert active learning","volume-title":"SIAM International Conference on Data Mining","author":"Wallace","year":"2011"},{"key":"2020110612452151400_ocx053-B10","doi-asserted-by":"crossref","DOI":"10.1609\/hcomp.v3i1.13225","article-title":"Combining crowd and expert labels using decision theoretic active learning","volume-title":"Human Computation and Crowdsourcing (HCOMP)","author":"Nguyen","year":"2015"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/24\/6\/1165\/34149207\/ocx053.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/24\/6\/1165\/34149207\/ocx053.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,23]],"date-time":"2023-08-23T16:51:31Z","timestamp":1692809491000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/24\/6\/1165\/3852629"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,5,25]]},"references-count":10,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2017,5,25]]},"published-print":{"date-parts":[[2017,11,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocx053","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2017,11]]},"published":{"date-parts":[[2017,5,25]]}}}