{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T23:12:58Z","timestamp":1780355578148,"version":"3.54.1"},"reference-count":43,"publisher":"Oxford University Press (OUP)","issue":"8","license":[{"start":{"date-parts":[[2021,5,6]],"date-time":"2021-05-06T00:00:00Z","timestamp":1620259200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"Bridging the semantic gap between research eligibility criteria and clinical data","award":["5R01LM009886-11"],"award-info":[{"award-number":["5R01LM009886-11"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,7,30]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>We introduce Medical evidence Dependency (MD)\u2013informed attention, a novel neuro-symbolic model for understanding free-text clinical trial publications with generalizability and interpretability.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and Methods<\/jats:title><jats:p>We trained one head in the multi-head self-attention model to attend to the Medical evidence Ddependency (MD) and to pass linguistic and domain knowledge on to later layers (MD informed). This MD-informed attention model was integrated into BioBERT and tested on 2 public machine reading comprehension benchmarks for clinical trial publications: Evidence Inference 2.0 and PubMedQA. We also curated a small set of recently published articles reporting randomized controlled trials on COVID-19 (coronavirus disease 2019) following the Evidence Inference 2.0 guidelines to evaluate the model\u2019s robustness to unseen data.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The integration of MD-informed attention head improves BioBERT substantially in both benchmark tasks\u2014as large as an increase of +30% in the F1 score\u2014and achieves the new state-of-the-art performance on the Evidence Inference 2.0. It achieves 84% and 82% in overall accuracy and F1 score, respectively, on the unseen COVID-19 data.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>MD-informed attention empowers neural reading comprehension models with interpretability and generalizability via reusable domain knowledge. Its compositionality can benefit any transformer-based architecture for machine reading comprehension of free-text medical evidence.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocab077","type":"journal-article","created":{"date-parts":[[2021,4,9]],"date-time":"2021-04-09T11:54:04Z","timestamp":1617969244000},"page":"1703-1711","source":"Crossref","is-referenced-by-count":10,"title":["A neuro-symbolic method for understanding free-text medical evidence"],"prefix":"10.1093","volume":"28","author":[{"given":"Tian","family":"Kang","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ali","family":"Turfah","sequence":"additional","affiliation":[{"name":"Department of Statistics, Columbia University, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jaehyun","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6695-0282","authenticated-orcid":false,"given":"Adler","family":"Perotte","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9624-0214","authenticated-orcid":false,"given":"Chunhua","family":"Weng","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2021,5,6]]},"reference":[{"issue":"1","key":"2021073020281337700_ocab077-B1","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/S0146-0005(97)80013-4","article-title":"Evidence-based medicine","volume":"21","author":"Sackett","year":"1997","journal-title":"Semin Perinatol"},{"key":"2021073020281337700_ocab077-B2","author":"DeYoung","year":"2020"},{"issue":"6","key":"2021073020281337700_ocab077-B3","doi-asserted-by":"crossref","first-page":"1192","DOI":"10.1093\/jamia\/ocx050","article-title":"Evidence appraisal: a scoping review, conceptual framework, and research agenda","volume":"24","author":"Goldstein","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"7206","key":"2021073020281337700_ocab077-B4","doi-asserted-by":"crossref","first-page":"358","DOI":"10.1136\/bmj.319.7206.358","article-title":"Analysis of questions asked by family doctors regarding patient care","volume":"319","author":"Ely","year":"1999","journal-title":"BMJ"},{"issue":"4","key":"2021073020281337700_ocab077-B5","doi-asserted-by":"crossref","first-page":"440","DOI":"10.1177\/0272989X0002000408","article-title":"Electronic trial banks: a complementary method for reporting randomized trials","volume":"20","author":"Sim","year":"2000","journal-title":"Med Decis Making"},{"issue":"11","key":"2021073020281337700_ocab077-B6","doi-asserted-by":"crossref","first-page":"e365","DOI":"10.1371\/journal.pmed.0020365","article-title":"Beyond trial registration: a global trial bank for clinical trial reporting","volume":"2","author":"Sim","year":"2005","journal-title":"PLoS Med"},{"key":"2021073020281337700_ocab077-B7","author":"Verspoor","year":"2020"},{"issue":"12","key":"2021073020281337700_ocab077-B8","doi-asserted-by":"crossref","first-page":"1903","DOI":"10.1093\/jamia\/ocaa163","article-title":"Trialstreamer: A living, automatically updated database of clinical trial reports","volume":"27","author":"Marshall","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2021073020281337700_ocab077-B9","first-page":"197","article-title":"A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature","volume":"2018","author":"Nye","year":"2018","journal-title":"Proc Conf Assoc Comput Linguist Meet"},{"key":"2021073020281337700_ocab077-B10","first-page":"188","article-title":"Pretraining to recognize piCO elements from randomized controlled trial literature","volume":"264","author":"Kang","year":"2019","journal-title":"Stud Health Technol Inform"},{"key":"2021073020281337700_ocab077-B11","first-page":"124","author":"Amini","year":"2012"},{"key":"2021073020281337700_ocab077-B12","first-page":"67","author":"Jin","year":"2018"},{"key":"2021073020281337700_ocab077-B13","doi-asserted-by":"crossref","first-page":"S5","DOI":"10.1186\/1471-2105-12-S2-S5","article-title":"Automatic classification of sentences to support evidence based medicine","volume":"12 (suppl 2","author":"Kim","year":"2011","journal-title":"BMC Bioinform"},{"issue":"1","key":"2021073020281337700_ocab077-B14","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1186\/s13643-019-1074-9","article-title":"Toward systematic review automation: a practical guide to using machine learning tools in research synthesis","volume":"8","author":"Marshall","year":"2019","journal-title":"Syst Rev"},{"issue":"1","key":"2021073020281337700_ocab077-B15","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1093\/jamia\/ocv044","article-title":"RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials","volume":"23","author":"Marshall","year":"2016","journal-title":"J Am Med Inform Assoc"},{"key":"2021073020281337700_ocab077-B16","first-page":"7","article-title":"Automating biomedical evidence synthesis: robot reviewer","volume":"2017","author":"Marshall","year":"2017","journal-title":"Proc Conf Assoc Comput Linguist Meet"},{"key":"2021073020281337700_ocab077-B17","first-page":"469","article-title":"Beyond information retrieval\u2014medical question answering","volume":"2006","author":"Lee","year":"2006","journal-title":"AMIA Ann Symp Proc"},{"key":"2021073020281337700_ocab077-B18","first-page":"841","author":"Demner-Fushman","year":"2006"},{"key":"2021073020281337700_ocab077-B19","first-page":"38","author":"Schulze","year":"2016"},{"key":"2021073020281337700_ocab077-B20","first-page":"297","article-title":"Medical question answering for clinical decision support","volume":"2016","author":"Goodwin","year":"2016","journal-title":"Proc ACM Int Conf Inf Knowl Manag"},{"key":"2021073020281337700_ocab077-B21","author":"Yoon","year":"2019"},{"key":"2021073020281337700_ocab077-B22","doi-asserted-by":"crossref","first-page":"101166","DOI":"10.1016\/j.ctcp.2020.101166","article-title":"Respiratory rehabilitation in elderly patients with COVID-19: a randomized controlled study","volume":"39","author":"Liu","year":"2020","journal-title":"Complement Ther Clin Pract"},{"issue":"2","key":"2021073020281337700_ocab077-B23","first-page":"217","article-title":"The UMLS Metathesaurus: representing different views of biomedical concepts","volume":"81","author":"Schuyler","year":"1993","journal-title":"Bull Med Libr Assoc"},{"key":"2021073020281337700_ocab077-B24","author":"Peters","year":"2018"},{"key":"2021073020281337700_ocab077-B25","author":"Devlin","year":"2019"},{"key":"2021073020281337700_ocab077-B26","author":"Du","year":"2018"},{"key":"2021073020281337700_ocab077-B27","author":"Xiong","year":"2018"},{"key":"2021073020281337700_ocab077-B28","author":"Wiese","year":"2017"},{"key":"2021073020281337700_ocab077-B29","author":"Weissenborn","year":"2017"},{"key":"2021073020281337700_ocab077-B30","author":"Kursuncu","year":"2020"},{"key":"2021073020281337700_ocab077-B31","author":"Chen","year":"2019"},{"key":"2021073020281337700_ocab077-B32","author":"Lehman","year":"2019"},{"key":"2021073020281337700_ocab077-B33","first-page":"670","author":"Oita","year":"2019"},{"key":"2021073020281337700_ocab077-B34","author":"Bahdanau","year":"2016"},{"key":"2021073020281337700_ocab077-B35","first-page":"6000","author":"Vaswani","year":"2017"},{"key":"2021073020281337700_ocab077-B36","author":"Yang","year":"2019"},{"issue":"8","key":"2021073020281337700_ocab077-B37","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","year":"2019","journal-title":"OpenAI Blog"},{"key":"2021073020281337700_ocab077-B38","author":"Strubell","year":"2018"},{"key":"2021073020281337700_ocab077-B39","author":"Peng","year":"2019"},{"key":"2021073020281337700_ocab077-B40","author":"Jin","year":"2019"},{"key":"2021073020281337700_ocab077-B41","author":"Liu","year":"2019"},{"issue":"4","key":"2021073020281337700_ocab077-B42","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2021073020281337700_ocab077-B43","author":"Kingma","year":"2014"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/8\/1703\/39502326\/ocab077.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/8\/1703\/39502326\/ocab077.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,1]],"date-time":"2023-11-01T21:55:15Z","timestamp":1698875715000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/28\/8\/1703\/6270974"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,6]]},"references-count":43,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2021,5,6]]},"published-print":{"date-parts":[[2021,7,30]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocab077","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,8,1]]},"published":{"date-parts":[[2021,5,6]]}}}