{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,15]],"date-time":"2026-05-15T08:09:11Z","timestamp":1778832551329,"version":"3.51.4"},"reference-count":38,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2019,2,8]],"date-time":"2019-02-08T00:00:00Z","timestamp":1549584000000},"content-version":"vor","delay-in-days":1,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","award":["R01LM009886"],"award-info":[{"award-number":["R01LM009886"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","award":["CU15-2317"],"award-info":[{"award-number":["CU15-2317"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006108","name":"National Center for Advancing Translational Science","doi-asserted-by":"publisher","award":["3OT3TR002027-01S1"],"award-info":[{"award-number":["3OT3TR002027-01S1"]}],"id":[{"id":"10.13039\/100006108","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006108","name":"National Center for Advancing Translational Science","doi-asserted-by":"publisher","award":["UL1TR001873"],"award-info":[{"award-number":["UL1TR001873"]}],"id":[{"id":"10.13039\/100006108","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,4,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>Cohort definition is a bottleneck for conducting clinical research and depends on subjective decisions by domain experts. Data-driven cohort definition is appealing but requires substantial knowledge of terminologies and clinical data models. Criteria2Query is a natural language interface that facilitates human-computer collaboration for cohort definition and execution using clinical databases.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and Methods<\/jats:title><jats:p>Criteria2Query uses a hybrid information extraction pipeline combining machine learning and rule-based methods to systematically parse eligibility criteria text, transforms it first into a structured criteria representation and next into sharable and executable clinical data queries represented as SQL queries conforming to the OMOP Common Data Model. Users can interactively review, refine, and execute queries in the ATLAS web application. To test effectiveness, we evaluated 125 criteria across different disease domains from ClinicalTrials.gov and 52 user-entered criteria. We evaluated F1 score and accuracy against 2 domain experts and calculated the average computation time for fully automated query formulation. We conducted an anonymous survey evaluating usability.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Criteria2Query achieved 0.795 and 0.805 F1 score for entity recognition and relation extraction, respectively. Accuracies for negation detection, logic detection, entity normalization, and attribute normalization were 0.984, 0.864, 0.514 and 0.793, respectively. Fully automatic query formulation took 1.22 seconds\/criterion. More than 80% (11+ of 13) of users would use Criteria2Query in their future cohort definition tasks.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>We contribute a novel natural language interface to clinical databases. It is open source and supports fully automated and interactive modes for autonomous data-driven cohort definition by researchers with minimal human effort. We demonstrate its promising user friendliness and usability.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocy178","type":"journal-article","created":{"date-parts":[[2018,11,29]],"date-time":"2018-11-29T20:17:21Z","timestamp":1543522641000},"page":"294-305","source":"Crossref","is-referenced-by-count":138,"title":["Criteria2Query: a natural language interface to clinical databases for cohort definition"],"prefix":"10.1093","volume":"26","author":[{"given":"Chi","family":"Yuan","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, New York, USA"},{"name":"Department of Computer Science and Technology, Nanjing University of Science and Technology, Nanjing, Jiangsu Province, P.R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Patrick B","family":"Ryan","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, New York, USA"},{"name":"Epidemiology Analytics, Janssen Research and Development, Titusville, New Jersey, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Casey","family":"Ta","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, New York, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yixuan","family":"Guo","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, New York, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ziran","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, New York, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jill","family":"Hardin","sequence":"additional","affiliation":[{"name":"Epidemiology Analytics, Janssen Research and Development, Titusville, New Jersey, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rupa","family":"Makadia","sequence":"additional","affiliation":[{"name":"Epidemiology Analytics, Janssen Research and Development, Titusville, New Jersey, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Peng","family":"Jin","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, New York, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ning","family":"Shang","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, New York, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tian","family":"Kang","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, New York, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chunhua","family":"Weng","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, New York, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2019,2,7]]},"reference":[{"key":"2020110613060435600_ocy178-B1","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1016\/j.ijmedinf.2007.09.001","article-title":"Definition, structure, content, use and impacts of electronic health records: a review of the research literature","volume":"77","author":"H\u00e4yrinen","year":"2008","journal-title":"Int J Med Inf"},{"issue":"3","key":"2020110613060435600_ocy178-B2","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1016\/j.cct.2010.03.005","article-title":"Automated matching software for clinical trials eligibility: measuring efficiency and flexibility","volume":"31","author":"Penberthy","year":"2010","journal-title":"Contemp Clin Trials"},{"issue":"6","key":"2020110613060435600_ocy178-B3","doi-asserted-by":"crossref","first-page":"869","DOI":"10.1197\/jamia.M3119","article-title":"Electronic screening improves efficiency in clinical trial recruitment","volume":"16","author":"Thadani","year":"2009","journal-title":"J Am Med Inform Assoc"},{"issue":"6","key":"2020110613060435600_ocy178-B4","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1200\/JOP.2012.000646","article-title":"Effort required in eligibility screening for clinical trials","volume":"8","author":"Penberthy","year":"2012","journal-title":"J Oncol Pract"},{"key":"2020110613060435600_ocy178-B5","first-page":"291","article-title":"Knowledge engineering for a clinical trial advice system: uncovering errors in protocol specification","volume":"74","author":"Musen","year":"1987","journal-title":"Bull Cancer"},{"issue":"11","key":"2020110613060435600_ocy178-B6","doi-asserted-by":"crossref","first-page":"706","DOI":"10.1016\/j.tips.2015.08.007","article-title":"Optimizing clinical research participant selection with informatics","volume":"36","author":"Weng","year":"2015","journal-title":"Trends Pharmacol Sci"},{"issue":"6","key":"2020110613060435600_ocy178-B7","doi-asserted-by":"crossref","first-page":"1062","DOI":"10.1093\/jamia\/ocx019","article-title":"EliIE: an open-source information extraction system for clinical trial eligibility criteria","volume":"24","author":"Kang","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"2020110613060435600_ocy178-B8","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1197\/jamia.M3092","article-title":"A \u201cfundamental theorem\u201d of biomedical informatics","volume":"16","author":"Friedman","year":"2009","journal-title":"J Am Med Inform Assoc JAMIA"},{"issue":"3","key":"2020110613060435600_ocy178-B9","doi-asserted-by":"crossref","first-page":"451","DOI":"10.1016\/j.jbi.2009.12.004","article-title":"Formal representation of eligibility criteria: a literature review","volume":"43","author":"Weng","year":"2010","journal-title":"J Biomed Inform"},{"key":"2020110613060435600_ocy178-B10"},{"issue":"2","key":"2020110613060435600_ocy178-B11","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1016\/j.jbi.2010.09.007","article-title":"A practical method for transforming free-text eligibility criteria into computable criteria","volume":"44","author":"Tu","year":"2011","journal-title":"J Biomed Inform"},{"issue":"5","key":"2020110613060435600_ocy178-B12","doi-asserted-by":"crossref","first-page":"805","DOI":"10.1016\/j.jbi.2013.06.001","article-title":"Analysis of eligibility criteria representation in industry-standard clinical trial protocols","volume":"46","author":"Bhattacharya","year":"2013","journal-title":"J Biomed Inform"},{"key":"2020110613060435600_ocy178-B13","doi-asserted-by":"crossref","first-page":"i116","DOI":"10.1136\/amiajnl-2011-000321","article-title":"EliXR: an approach to eligibility criteria extraction and representation","volume":"18 (Suppl 1)","author":"Weng","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613060435600_ocy178-B14","first-page":"71","article-title":"EliXR-TIME: a temporal knowledge representation for clinical research eligibility criteria","volume":"2012","author":"Boland","year":"2012","journal-title":"AMIA Summits Transl Sci Proc"},{"issue":"3","key":"2020110613060435600_ocy178-B15","doi-asserted-by":"crossref","first-page":"266","DOI":"10.3414\/ME15-01-0112","article-title":"Valx: a system for extracting and structuring numeric lab test comparison statements from text","volume":"55","author":"Hao","year":"2016","journal-title":"Methods Inf Med"},{"key":"2020110613060435600_ocy178-B16","author":"Parker","year":"2005"},{"key":"2020110613060435600_ocy178-B17"},{"key":"2020110613060435600_ocy178-B18","first-page":"574","article-title":"Observational Health Data Sciences and Informatics (OHDSI): opportunities for observational researchers","volume":"216","author":"Hripcsak","year":"2015","journal-title":"Stud Health Technol Inform"},{"key":"2020110613060435600_ocy178-B19"},{"issue":"1","key":"2020110613060435600_ocy178-B20","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1017\/S135132490000005X","article-title":"Natural language interfaces to databases\u2014an introduction","volume":"1","author":"Androutsopoulos","year":"1995","journal-title":"Nat Lang Eng"},{"issue":"4","key":"2020110613060435600_ocy178-B21","doi-asserted-by":"crossref","first-page":"225\u201349.","DOI":"10.1017\/S0269888900005476","article-title":"Natural language interfaces to databases","volume":"5","author":"Copestake","year":"1990","journal-title":"Knowl Eng Rev"},{"key":"2020110613060435600_ocy178-B22","author":"Woods","year":"1973"},{"key":"2020110613060435600_ocy178-B23","first-page":"320","article-title":"Natural language access to a melanoma data base","volume":"1978","author":"Epstein","year":"1978","journal-title":"Proc Annu Symp Comput Appl Med Care"},{"key":"2020110613060435600_ocy178-B24","author":"Chandra"},{"key":"2020110613060435600_ocy178-B25","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1007\/978-3-642-33021-6_36","volume-title":"Recent Advances in Hybrid Intelligence Systems","author":"Pazos","year":"2013"},{"issue":"1","key":"2020110613060435600_ocy178-B26","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1016\/0010-4809(81)90040-9","article-title":"A natural language interface to a clinical data base management system","volume":"14","author":"Woodyard","year":"1981","journal-title":"Comput Biomed Res"},{"key":"2020110613060435600_ocy178-B27","first-page":"157","article-title":"Toward a natural language interface for EHR questions","volume":"2015","author":"Roberts","year":"2015","journal-title":"AMIA Summits Transl Sci Proc"},{"key":"2020110613060435600_ocy178-B28","year":"2018"},{"key":"2020110613060435600_ocy178-B29","author":"Manning","year":"2014"},{"issue":"5","key":"2020110613060435600_ocy178-B30","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1006\/jbin.2001.1029","article-title":"A simple algorithm for identifying negated findings and diseases in discharge summaries","volume":"34","author":"Chapman","year":"2001","journal-title":"J Biomed Inform"},{"key":"2020110613060435600_ocy178-B31","author":"Schuster"},{"key":"2020110613060435600_ocy178-B32","year":"2018"},{"issue":"90001","key":"2020110613060435600_ocy178-B33","doi-asserted-by":"crossref","first-page":"267D","DOI":"10.1093\/nar\/gkh061","article-title":"The Unified Medical Language System (UMLS): integrating biomedical terminology","volume":"32","author":"Bodenreider","year":"2004","journal-title":"Nucleic Acids Res"},{"key":"2020110613060435600_ocy178-B34"},{"key":"2020110613060435600_ocy178-B35"},{"key":"2020110613060435600_ocy178-B36","first-page":"3735","volume-title":"Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012).","author":"Chang","year":"2012"},{"key":"2020110613060435600_ocy178-B37","author":"Dogan","year":"2012"},{"issue":"4","key":"2020110613060435600_ocy178-B38","doi-asserted-by":"crossref","first-page":"626","DOI":"10.1016\/j.jbi.2011.11.006","article-title":"Mapping partners master drug dictionary to RxNorm using an NLP-based approach","volume":"45","author":"Zhou","year":"2012","journal-title":"J Biomed Inform"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/4\/294\/34151511\/ocy178.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/4\/294\/34151511\/ocy178.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,16]],"date-time":"2020-11-16T10:15:10Z","timestamp":1605521710000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/26\/4\/294\/5308980"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,2,7]]},"references-count":38,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2019,2,7]]},"published-print":{"date-parts":[[2019,4,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocy178","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,4]]},"published":{"date-parts":[[2019,2,7]]}}}