{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T05:41:52Z","timestamp":1775194912315,"version":"3.50.1"},"reference-count":26,"publisher":"Oxford University Press (OUP)","issue":"12","license":[{"start":{"date-parts":[[2020,9,17]],"date-time":"2020-09-17T00:00:00Z","timestamp":1600300800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000265","name":"UK Medical Research Council","doi-asserted-by":"crossref","award":["MR\/N015185\/1"],"award-info":[{"award-number":["MR\/N015185\/1"]}],"id":[{"id":"10.13039\/501100000265","id-type":"DOI","asserted-by":"crossref"}]},{"name":"National Institutes of Health under the National Library of Medicine","award":["R01-LM012086"],"award-info":[{"award-number":["R01-LM012086"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,12,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>Randomized controlled trials (RCTs) are the gold standard method for evaluating whether a treatment works in health care but can be difficult to find and make use of. We describe the development and evaluation of a system to automatically find and categorize all new RCT reports.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>Trialstreamer continuously monitors PubMed and the World Health Organization International Clinical Trials Registry Platform, looking for new RCTs in humans using a validated classifier. We combine machine learning and rule-based methods to extract information from the RCT abstracts, including free-text descriptions of trial PICO (populations, interventions\/comparators, and outcomes) elements and map these snippets to normalized MeSH (Medical Subject Headings) vocabulary terms. We additionally identify sample sizes, predict the risk of bias, and extract text conveying key findings. We store all extracted data in a database, which we make freely available for download, and via a search portal, which allows users to enter structured clinical queries. Results are ranked automatically to prioritize larger and higher-quality studies.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>As of early June 2020, we have indexed 673\u00a0191 publications of RCTs, of which 22\u00a0363 were published in the first 5 months of 2020 (142 per day). We additionally include 304\u00a0111 trial registrations from the International Clinical Trials Registry Platform. The median trial sample size was 66.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>We present an automated system for finding and categorizing RCTs. This yields a novel resource: a database of structured information automatically extracted for all published RCTs in humans. We make daily updates of this database available on our website (https:\/\/trialstreamer.robotreviewer.net).<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocaa163","type":"journal-article","created":{"date-parts":[[2020,7,7]],"date-time":"2020-07-07T07:12:55Z","timestamp":1594105975000},"page":"1903-1912","source":"Crossref","is-referenced-by-count":65,"title":["Trialstreamer: A living, automatically updated database of clinical trial reports"],"prefix":"10.1093","volume":"27","author":[{"given":"Iain J","family":"Marshall","sequence":"first","affiliation":[{"name":"School of Population Health and Environmental Sciences, King\u2019s College London, London, United Kingdom"}]},{"given":"Benjamin","family":"Nye","sequence":"additional","affiliation":[{"name":"Khoury College of Computer Sciences, Northeastern University, Boston, Massachusetts, USA"}]},{"given":"Jo\u00ebl","family":"Kuiper","sequence":"additional","affiliation":[{"name":"Vortext Systems, Groningen, the Netherlands"}]},{"given":"Anna","family":"Noel-Storr","sequence":"additional","affiliation":[{"name":"Cochrane Dementia Group, University of Oxford, Oxford, United Kingdom"}]},{"given":"Rachel","family":"Marshall","sequence":"additional","affiliation":[{"name":"Cochrane Editorial and Methods Department, London, United Kingdom"}]},{"given":"Rory","family":"Maclean","sequence":"additional","affiliation":[{"name":"School of Population Health and Environmental Sciences, King\u2019s College London, London, United Kingdom"}]},{"given":"Frank","family":"Soboczenski","sequence":"additional","affiliation":[{"name":"School of Population Health and Environmental Sciences, King\u2019s College London, London, United Kingdom"}]},{"given":"Ani","family":"Nenkova","sequence":"additional","affiliation":[{"name":"Computer and Information Science, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}]},{"given":"James","family":"Thomas","sequence":"additional","affiliation":[{"name":"EPPI-Centre, UCL Social Research Institute, University College London, London, United Kingdom"}]},{"given":"Byron C","family":"Wallace","sequence":"additional","affiliation":[{"name":"Khoury College of Computer Sciences, Northeastern University, Boston, Massachusetts, USA"}]}],"member":"286","published-online":{"date-parts":[[2020,9,17]]},"reference":[{"key":"2020121009242198000_ocaa163-B1","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1111\/j.1749-6632.1993.tb26345.x","article-title":"The Cochrane Collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care","volume":"703","author":"Chalmers","year":"1993","journal-title":"Ann N Y Acad Sci"},{"key":"2020121009242198000_ocaa163-B2","first-page":"489","article-title":"Becoming an information master: a guidebook to the medical information jungle","volume":"39","author":"Shaughnessy","year":"1994","journal-title":"J Fam Pract"},{"issue":"9","key":"2020121009242198000_ocaa163-B3","doi-asserted-by":"crossref","first-page":"e1000326","DOI":"10.1371\/journal.pmed.1000326","article-title":"Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?","volume":"7","author":"Bastian","year":"2010","journal-title":"PLoS Med"},{"key":"2020121009242198000_ocaa163-B4","doi-asserted-by":"crossref","DOI":"10.1002\/jrsm.1287","article-title":"Machine learning for identifying randomized controlled trials: an evaluation and practitioner\u2019s guide","author":"Marshall","year":"2018","journal-title":"Res Synth Methods"},{"key":"2020121009242198000_ocaa163-B5","volume-title":"Cochrane Handbook for Systematic Reviews of Interventions","author":"Thomas","year":"2019"},{"issue":"1","key":"2020121009242198000_ocaa163-B6","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1186\/s13643-019-1074-9","article-title":"Toward systematic review automation: a practical guide to using machine learning tools in research synthesis","volume":"8","author":"Marshall","year":"2019","journal-title":"Syst Rev"},{"issue":"3","key":"2020121009242198000_ocaa163-B7","doi-asserted-by":"crossref","first-page":"707","DOI":"10.1093\/jamia\/ocu025","article-title":"Automated confidence ranked classification of randomized controlled trial articles: an aid to evidence-based medicine","volume":"22","author":"Cohen","year":"2015","journal-title":"J Am Med Inform Assoc"},{"issue":"6","key":"2020121009242198000_ocaa163-B8","doi-asserted-by":"crossref","first-page":"1165","DOI":"10.1093\/jamia\/ocx053","article-title":"Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach","volume":"24","author":"Wallace","year":"2017","journal-title":"J Am Med Inform Assoc"},{"key":"2020121009242198000_ocaa163-B9","doi-asserted-by":"crossref","DOI":"10.1093\/database\/bay079","article-title":"A probabilistic automated tagger to identify human-related publications","volume":"2018.","author":"Cohen","year":"2018","journal-title":"Database"},{"key":"2020121009242198000_ocaa163-B10","first-page":"372","article-title":"Automatic summarization of results from clinical trials","volume":"2011","author":"Summerscales","year":"2011","journal-title":". Proceedings (IEEE Int Conf Bioinformatics Biomed)"},{"key":"2020121009242198000_ocaa163-B11","first-page":"1","article-title":"Extracting PICO sentences from clinical trial reports using supervised","volume":"17","author":"Wallace","year":"2016","journal-title":"J Mach Learn Res"},{"key":"2020121009242198000_ocaa163-B12","first-page":"197","article-title":"A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature","volume":"2018","author":"Nye","year":"2018","journal-title":"Proc Conf Assoc Comput Linguist Meet"},{"key":"2020121009242198000_ocaa163-B13","first-page":"88","author":"Marshall","year":"2014"},{"issue":"1","key":"2020121009242198000_ocaa163-B14","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1093\/jamia\/ocv044","article-title":"RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials","volume":"23","author":"Marshall","year":"2016","journal-title":"J Am Med Inform Assoc"},{"key":"2020121009242198000_ocaa163-B15","first-page":"795","author":"Zhang","year":"2016"},{"issue":"1","key":"2020121009242198000_ocaa163-B16","doi-asserted-by":"crossref","first-page":"266","DOI":"10.1093\/ije\/dyv306","article-title":"Machine learning to assist risk-of-bias assessments in systematic reviews","volume":"45","author":"Millard","year":"2016","journal-title":"Int J Epidemiol"},{"issue":"4","key":"2020121009242198000_ocaa163-B17","doi-asserted-by":"crossref","first-page":"841","DOI":"10.1093\/jamia\/ocw177","article-title":"MetaMap Lite: an evaluation of a new Java implementation of MetaMap","volume":"24","author":"Demner-Fushman","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"7482","key":"2020121009242198000_ocaa163-B18","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1136\/bmj.38336.804167.47","article-title":"Optimal search strategies for retrieving systematic reviews from Medline: analytical survey","volume":"330","author":"Montori","year":"2005","journal-title":"BMJ"},{"issue":"8","key":"2020121009242198000_ocaa163-B19","doi-asserted-by":"crossref","first-page":"774","DOI":"10.1016\/S0895-4356(01)00341-9","article-title":"Internal validation of predictive models: efficiency of some procedures","volume":"54","author":"Steyerberg","year":"2001","journal-title":"J Clin Epidemiol"},{"key":"2020121009242198000_ocaa163-B20","first-page":"3111","volume-title":"Advances in Neural Information Processing Systems 26","author":"Mikolov","year":"2013"},{"key":"2020121009242198000_ocaa163-B21","author":"Pyysalo","year":"2013"},{"key":"2020121009242198000_ocaa163-B22","article-title":"Bidirectional LSTM-CRF models for sequence tagging","author":"Huang","year":"2015","journal-title":"arXiv:1508.01991 Cs"},{"key":"2020121009242198000_ocaa163-B23","first-page":"451","article-title":"A simple algorithm for identifying abbreviation definitions in biomedical text","volume":"2003","author":"Schwartz","year":"2003","journal-title":"Pac Symp Biocomput"},{"key":"2020121009242198000_ocaa163-B24","doi-asserted-by":"crossref","first-page":"d5928","DOI":"10.1136\/bmj.d5928","article-title":"The Cochrane Collaboration\u2019s tool for assessing risk of bias in randomised trials","volume":"343","author":"Higgins","year":"2011","journal-title":"BMJ"},{"key":"2020121009242198000_ocaa163-B25","doi-asserted-by":"crossref","first-page":"7.1","DOI":"10.1002\/9780470712184","volume-title":"Cochrane Handbook for Systematic Reviews of Interventions","author":"Higgins","year":"2008"},{"key":"2020121009242198000_ocaa163-B26","doi-asserted-by":"crossref","DOI":"10.1145\/1015330.1015435","article-title":"Feature selection, L 1 vs. L 2 regularization, and rotational invariance","author":"Ng","year":"2004","journal-title":"proceedings of the Twenty-First International Conference on Machine Learning (ICML \u201904)"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/12\/1903\/34838541\/ocaa163.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/12\/1903\/34838541\/ocaa163.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,12,10]],"date-time":"2020-12-10T09:34:11Z","timestamp":1607592851000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/27\/12\/1903\/5907063"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,17]]},"references-count":26,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2020,9,17]]},"published-print":{"date-parts":[[2020,12,9]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaa163","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2020.05.15.20103044","asserted-by":"object"}]},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,12]]},"published":{"date-parts":[[2020,9,17]]}}}