{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T20:01:59Z","timestamp":1779912119508,"version":"3.53.1"},"reference-count":38,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2023,3,15]],"date-time":"2023-03-15T00:00:00Z","timestamp":1678838400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"name":"Bridging the semantic gap between research eligibility criteria and clinical data","award":["5R01LM009886-11"],"award-info":[{"award-number":["5R01LM009886-11"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,5,19]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To develop a computable representation for medical evidence and to contribute a gold standard dataset of annotated randomized controlled trial (RCT) abstracts, along with a natural language processing (NLP) pipeline for transforming free-text RCT evidence in PubMed into the structured representation.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and methods<\/jats:title>\n                  <jats:p>Our representation, EvidenceMap, consists of 3 levels of abstraction: Medical Evidence Entity, Proposition and Map, to represent the hierarchical structure of medical evidence composition. Randomly selected RCT abstracts were annotated following EvidenceMap based on the consensus of 2 independent annotators to train an NLP pipeline. Via a user study, we measured how the EvidenceMap improved evidence comprehension and analyzed its representative capacity by comparing the evidence annotation with EvidenceMap representation and without following any specific guidelines.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Two corpora including 229 disease-agnostic and 80 COVID-19 RCT abstracts were annotated, yielding 12\u00a0725 entities and 1602 propositions. EvidenceMap saves users 51.9% of the time compared to reading raw-text abstracts. Most evidence elements identified during the freeform annotation were successfully represented by EvidenceMap, and users gave the enrollment, study design, and study Results sections mean 5-scale Likert ratings of 4.85, 4.70, and 4.20, respectively. The end-to-end evaluations of the pipeline show that the evidence proposition formulation achieves F1 scores of 0.84 and 0.86 in the adjusted random index score.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>EvidenceMap extends the participant, intervention, comparator, and outcome framework into 3 levels of abstraction for transforming free-text evidence from the clinical literature into a computable structure. It can be used as an interoperable format for better evidence retrieval and synthesis and an interpretable representation to efficiently comprehend RCT findings.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocad036","type":"journal-article","created":{"date-parts":[[2023,3,15]],"date-time":"2023-03-15T20:56:58Z","timestamp":1678913818000},"page":"1022-1031","source":"Crossref","is-referenced-by-count":15,"title":["<i>EvidenceMap<\/i>: a three-level knowledge representation for medical evidence computation and comprehension"],"prefix":"10.1093","volume":"30","author":[{"given":"Tian","family":"Kang","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University , New York, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8693-5768","authenticated-orcid":false,"given":"Yingcheng","family":"Sun","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University , New York, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jae Hyun","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University , New York, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4679-805X","authenticated-orcid":false,"given":"Casey","family":"Ta","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University , New York, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6695-0282","authenticated-orcid":false,"given":"Adler","family":"Perotte","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University , New York, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kayla","family":"Schiffer","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University , New York, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mutong","family":"Wu","sequence":"additional","affiliation":[{"name":"Department of Statistics, Columbia University , New York, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yang","family":"Zhao","sequence":"additional","affiliation":[{"name":"Department of Statistics, Columbia University , New York, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nour","family":"Moustafa-Fahmy","sequence":"additional","affiliation":[{"name":"Department of Statistics, Columbia University , New York, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9309-8331","authenticated-orcid":false,"given":"Yifan","family":"Peng","sequence":"additional","affiliation":[{"name":"Department of Population Health Sciences, Weill Cornell Medicine , New York, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chunhua","family":"Weng","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University , New York, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2023,3,15]]},"reference":[{"issue":"1","key":"2023051917300470500_ocad036-B1","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1097\/PRS.0b013e318219c171","article-title":"The levels of evidence and their role in evidence-based medicine","volume":"128","author":"Burns","year":"2011","journal-title":"Plast Reconstr Surg"},{"key":"2023051917300470500_ocad036-B2","volume-title":"Trial Banks: An Informatics Foundation for Evidence-Based Medicine","author":"Sim","year":"1998"},{"issue":"1","key":"2023051917300470500_ocad036-B3","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1472-6947-7-16","article-title":"Utilization of the PICO framework to improve searching PubMed for clinical questions","volume":"7","author":"Schardt","year":"2007","journal-title":"BMC Med Inform Decis Mak"},{"issue":"2","key":"2023051917300470500_ocad036-B4","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1016\/j.jbi.2009.11.001","article-title":"Beyond genes, proteins, and abstracts: identifying scientific claims from full-text biomedical articles","volume":"43","author":"Blake","year":"2010","journal-title":"J Biomed Inform"},{"key":"2023051917300470500_ocad036-B5","first-page":"279","article-title":"Classification of PICO elements by text features systematically extracted from PubMed abstracts","author":"Huang","year":"2011"},{"issue":"1","key":"2023051917300470500_ocad036-B6","first-page":"4572","article-title":"Extracting PICO sentences from clinical trial reports using supervised distant supervision","volume":"17","author":"Wallace","year":"2016","journal-title":"J Mach Learn Res"},{"key":"2023051917300470500_ocad036-B7","first-page":"67","author":"Jin","year":"2018"},{"key":"2023051917300470500_ocad036-B8","first-page":"188","article-title":"Pretraining to recognize PICO elements from randomized controlled trial literature","volume":"264","author":"Kang","year":"2019","journal-title":"Stud Health Technol Inform"},{"key":"2023051917300470500_ocad036-B9","first-page":"197","article-title":"A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature","author":"Nye","year":"2018","journal-title":"NIH Public Access"},{"issue":"1","key":"2023051917300470500_ocad036-B10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12911-018-0699-2","article-title":"Combination of conditional random field with a rule based method in the extraction of PICO elements","volume":"18","author":"Chabou","year":"2018","journal-title":"BMC Med Inform Decis Mak"},{"issue":"1","key":"2023051917300470500_ocad036-B11","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1093\/jamia\/ocv044","article-title":"RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials","volume":"23","author":"Marshall","year":"2016","journal-title":"J Am Med Inform Assoc"},{"key":"2023051917300470500_ocad036-B12","first-page":"7","article-title":"Automating biomedical evidence synthesis: RobotReviewer","author":"Marshall","year":"2017","journal-title":"NIH Public Access"},{"key":"2023051917300470500_ocad036-B13","first-page":"359","article-title":"Evaluation of PICO as a knowledge representation for clinical questions","author":"Huang","year":"2006","journal-title":"Am Med Inform Assoc"},{"key":"2023051917300470500_ocad036-B14","author":"Lehman","year":"2019"},{"key":"2023051917300470500_ocad036-B15","first-page":"123","author":"DeYoung","year":"2020"},{"key":"2023051917300470500_ocad036-B16","first-page":"63","article-title":"Trialstreamer: mapping and browsing medical evidence in real-time","author":"Nye","year":"2020"},{"key":"2023051917300470500_ocad036-B17","author":"Health NIo","year":"2022"},{"key":"2023051917300470500_ocad036-B18","doi-asserted-by":"crossref","DOI":"10.1093\/acprof:oso\/9780199693764.001.0001","volume-title":"New Thinking about Propositions","author":"King","year":"2014"},{"key":"2023051917300470500_ocad036-B19","volume-title":"A Dictionary of Linguistics and Phonetics","author":"Crystal","year":"2011"},{"issue":"4","key":"2023051917300470500_ocad036-B20","doi-asserted-by":"crossref","first-page":"636","DOI":"10.1108\/JDOC-09-2015-0111","article-title":"On the composition of scientific abstracts","volume":"72","author":"Atanassova","year":"2016","journal-title":"J Doc"},{"key":"2023051917300470500_ocad036-B21","first-page":"102","author":"Stenetorp","year":"2012"},{"issue":"8","key":"2023051917300470500_ocad036-B22","doi-asserted-by":"crossref","first-page":"1703","DOI":"10.1093\/jamia\/ocab077","article-title":"A neuro-symbolic method for understanding free-text medical evidence","volume":"28","author":"Kang","year":"2021","journal-title":"J Am Med Inform Assoc"},{"key":"2023051917300470500_ocad036-B23","author":"Peng","year":"2019"},{"issue":"5","key":"2023051917300470500_ocad036-B24","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1006\/jbin.2001.1029","article-title":"A simple algorithm for identifying negated findings and diseases in discharge summaries","volume":"34","author":"Chapman","year":"2001","journal-title":"J Biomed Inform"},{"issue":"1","key":"2023051917300470500_ocad036-B25","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2458-14-177","article-title":"Protocol for the DREAM Project (Diabetes Research, Education, and Action for Minorities): a randomized trial of a community health worker intervention to improve diabetic management and control among Bangladeshi adults in NYC","volume":"14","author":"Islam","year":"2014","journal-title":"BMC Public Health"},{"issue":"9","key":"2023051917300470500_ocad036-B26","doi-asserted-by":"crossref","first-page":"e0257238","DOI":"10.1371\/journal.pone.0257238","article-title":"Hydroxychloroquine for the treatment of severe respiratory infection by covid-19: a randomized controlled trial","volume":"16","author":"Hernandez-Cardenas","year":"2021","journal-title":"PLoS One"},{"issue":"9","key":"2023051917300470500_ocad036-B27","doi-asserted-by":"crossref","first-page":"1959","DOI":"10.1007\/s00345-018-2597-8","article-title":"The effect of platelet-rich plasma injection on post-internal urethrotomy stricture recurrence","volume":"37","author":"Rezaei","year":"2019","journal-title":"World J Urol"},{"issue":"1","key":"2023051917300470500_ocad036-B28","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.jmpt.2018.04.007","article-title":"Joint mobilization of the hands of patients with rheumatoid arthritis: results from an assessor-blinded, randomized crossover trial","volume":"42","author":"Levitsky","year":"2019","journal-title":"J Manip Physiol Ther"},{"issue":"1","key":"2023051917300470500_ocad036-B29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12931-021-01833-6","article-title":"The efficacy of corticosteroids therapy in patients with moderate to severe SARS-CoV-2 infection: a multicenter, randomized, open-label trial","volume":"22","author":"Ghanei","year":"2021","journal-title":"Respir Res"},{"issue":"1","key":"2023051917300470500_ocad036-B30","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1136\/emermed-2020-210256","article-title":"Prehospital continuous positive airway pressure (CPAP) for acute respiratory distress: a randomised controlled trial","volume":"39","author":"Finn","year":"2022","journal-title":"Emerg Med J"},{"issue":"10175","key":"2023051917300470500_ocad036-B31","doi-asserted-by":"crossref","first-page":"1021","DOI":"10.1016\/S0140-6736(19)30195-3","article-title":"Efficacy and safety of minimally invasive surgery with thrombolysis in intracerebral haemorrhage evacuation (MISTIE III): a randomised, controlled, open-label, blinded endpoint phase 3 trial","volume":"393","author":"Hanley","year":"2019","journal-title":"Lancet"},{"issue":"1","key":"2023051917300470500_ocad036-B32","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1001\/jamaneurol.2021.4161","article-title":"Amyloid-related imaging abnormalities in 2 phase 3 studies evaluating aducanumab in patients with early Alzheimer disease","volume":"79","author":"Salloway","year":"2022","journal-title":"JAMA Neurol"},{"issue":"5","key":"2023051917300470500_ocad036-B33","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1016\/j.jacc.2020.11.061","article-title":"Rivaroxaban plus aspirin in obese and overweight patients with vascular disease in the COMPASS trial","volume":"77","author":"Guzik","year":"2021","journal-title":"J Am Coll Cardiol"},{"issue":"6","key":"2023051917300470500_ocad036-B34","doi-asserted-by":"crossref","first-page":"824","DOI":"10.1016\/S1470-2045(21)00149-2","article-title":"Neoadjuvant durvalumab with or without stereotactic body radiotherapy in patients with early-stage non-small-cell lung cancer: a single-centre, randomised phase 2 trial","volume":"22","author":"Altorki","year":"2021","journal-title":"Lancet Oncol"},{"issue":"336","key":"2023051917300470500_ocad036-B35","doi-asserted-by":"crossref","first-page":"846","DOI":"10.1080\/01621459.1971.10482356","article-title":"Objective criteria for the evaluation of clustering methods","volume":"66","author":"Rand","year":"1971","journal-title":"J Am Stat Assoc"},{"key":"2023051917300470500_ocad036-B36","first-page":"203","author":"Kolditz","year":"2019"},{"issue":"1","key":"2023051917300470500_ocad036-B37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13063-019-3442-0","article-title":"Abdominal drainage versus no drainage after distal pancreatectomy: study protocol for a randomized controlled trial","volume":"20","author":"Kaiser","year":"2019","journal-title":"Trials"},{"key":"2023051917300470500_ocad036-B38","doi-asserted-by":"crossref","DOI":"10.1136\/bmj.c723","article-title":"The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed","author":"Hopewell","year":"2010","journal-title":"BMJ"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/6\/1022\/50374770\/ocad036.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/6\/1022\/50374770\/ocad036.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,19]],"date-time":"2023-05-19T17:33:21Z","timestamp":1684517601000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/30\/6\/1022\/7078531"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,15]]},"references-count":38,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2023,3,15]]},"published-print":{"date-parts":[[2023,5,19]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocad036","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,6,1]]},"published":{"date-parts":[[2023,3,15]]}}}