{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,7]],"date-time":"2026-03-07T19:53:31Z","timestamp":1772913211490,"version":"3.50.1"},"reference-count":51,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2019,8,28]],"date-time":"2019-08-28T00:00:00Z","timestamp":1566950400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"University of Florida Clinical"},{"name":"Translational Science Institute"},{"DOI":"10.13039\/100006108","name":"NIH National Center for Advancing Translational Sciences","doi-asserted-by":"crossref","award":["UL1TR001427"],"award-info":[{"award-number":["UL1TR001427"]}],"id":[{"id":"10.13039\/100006108","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100006108","name":"NIH National Center for Advancing Translational Sciences","doi-asserted-by":"crossref","award":["NIA R21AG062884"],"award-info":[{"award-number":["NIA R21AG062884"]}],"id":[{"id":"10.13039\/100006108","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,1,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>To develop a natural language processing system that identifies relations of medications with adverse drug events from clinical narratives. This project is part of the 2018 n2c2 challenge.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and Methods<\/jats:title><jats:p>We developed a novel clinical named entity recognition method based on an recurrent convolutional neural network and compared it to a recurrent neural network implemented using the long-short term memory architecture, explored methods to integrate medical knowledge as embedding layers in neural networks, and investigated 3 machine learning models, including support vector machines, random forests and gradient boosting for relation classification. The performance of our system was evaluated using annotated data and scripts provided by the 2018 n2c2 organizers.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Our system was among the top ranked. Our best model submitted during this challenge (based on recurrent neural networks and support vector machines) achieved lenient F1 scores of 0.9287 for concept extraction (ranked third), 0.9459 for relation classification (ranked fourth), and 0.8778 for the end-to-end relation extraction (ranked second). We developed a novel named entity recognition model based on a recurrent convolutional neural network and further investigated gradient boosting for relation classification. The new methods improved the lenient F1 scores of the 3 subtasks to 0.9292, 0.9633, and 0.8880, respectively, which are comparable to the best performance reported in this challenge.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>This study demonstrated the feasibility of using machine learning methods to extract the relations of medications with adverse drug events from clinical narratives.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocz144","type":"journal-article","created":{"date-parts":[[2019,7,24]],"date-time":"2019-07-24T11:25:23Z","timestamp":1563967523000},"page":"65-72","source":"Crossref","is-referenced-by-count":69,"title":["Identifying relations of medications with adverse drug events using recurrent convolutional neural networks and gradient boosting"],"prefix":"10.1093","volume":"27","author":[{"given":"Xi","family":"Yang","sequence":"first","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA"}]},{"given":"Jiang","family":"Bian","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA"}]},{"given":"Ruogu","family":"Fang","sequence":"additional","affiliation":[{"name":"J. Crayton Pruitt Family Department of Biomedical Engineering, University of Florida, Gainesville, Florida, USA"}]},{"given":"Ragnhildur I","family":"Bjarnadottir","sequence":"additional","affiliation":[{"name":"Department of Family, Community and Health Systems Science, College of Nursing, University of Florida, Gainesville, Florida, USA"}]},{"given":"William R","family":"Hogan","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA"}]},{"given":"Yonghui","family":"Wu","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA"}]}],"member":"286","published-online":{"date-parts":[[2019,8,28]]},"reference":[{"key":"2020110613055562500_ocz144-B1","volume-title":"To Err Is Human: Building a Safer Health System","year":"2000"},{"issue":"6","key":"2020110613055562500_ocz144-B2","doi-asserted-by":"crossref","first-page":"635","DOI":"10.1002\/pds.4184","article-title":"Burden of hospitalizations related to adverse drug events in the USA: a retrospective analysis from large inpatient database","volume":"26","author":"Poudel","year":"2017","journal-title":"Pharmacoepidemiol Drug Saf"},{"key":"2020110613055562500_ocz144-B3","author":"Weiss","year":"2018"},{"key":"2020110613055562500_ocz144-B4","doi-asserted-by":"crossref","first-page":"125.","DOI":"10.1186\/1472-6963-14-125","article-title":"International prevalence of adverse drug events in hospitals: an analysis of routine data from England, Germany, and the USA","volume":"14","author":"Stausberg","year":"2014","journal-title":"BMC Health Serv Res"},{"issue":"5","key":"2020110613055562500_ocz144-B5","doi-asserted-by":"crossref","first-page":"544","DOI":"10.1136\/amiajnl-2011-000464","article-title":"Natural language processing: an introduction","volume":"18","author":"Nadkarni","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613055562500_ocz144-B6","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.jbi.2017.11.011","article-title":"Clinical information extraction applications: a literature review","volume":"77","author":"Wang","year":"2018","journal-title":"J Biomed Inform"},{"key":"2020110613055562500_ocz144-B7","doi-asserted-by":"crossref","first-page":"128","DOI":"10.1055\/s-0038-1638592","article-title":"Extracting information from textual documents in the electronic health record: a review of recent research","volume":"17","author":"Meystre","year":"2008","journal-title":"Yearb Med Inform"},{"issue":"5","key":"2020110613055562500_ocz144-B8","doi-asserted-by":"crossref","first-page":"765","DOI":"10.1016\/j.jbi.2013.06.004","article-title":"Natural language processing: state of the art and prospects for significant progress, a workshop sponsored by the National Library of Medicine","volume":"46","author":"Friedman","year":"2013","journal-title":"J Biomed Inform"},{"issue":"2","key":"2020110613055562500_ocz144-B9","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1136\/jamia.1994.95236146","article-title":"A general natural-language text processor for clinical radiology","volume":"1","author":"Friedman","year":"1994","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2020110613055562500_ocz144-B10","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1136\/jamia.2009.002733","article-title":"An overview of MetaMap: historical perspective and recent advances","volume":"17","author":"Aronson","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613055562500_ocz144-B11","first-page":"195","article-title":"The KnowledgeMap project: development of a concept-based medical school curriculum database","author":"Denny","year":"2003","journal-title":"AMIA Annu Symp Proc"},{"issue":"5","key":"2020110613055562500_ocz144-B12","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1136\/jamia.2009.001560","article-title":"Mayo clinical text analysis and knowledge extraction system (cTAKES): architecture, component evaluation and applications","volume":"17","author":"Savova","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613055562500_ocz144-B13","doi-asserted-by":"crossref","first-page":"D267","DOI":"10.1093\/nar\/gkh061","article-title":"The unified medical language system (UMLS): integrating biomedical terminology","volume":"32","author":"Bodenreider","year":"2004","journal-title":"Nucleic Acids Res"},{"key":"2020110613055562500_ocz144-B14","first-page":"282","volume-title":"Proceedings of the Eighteenth International Conference on Machine Learning","author":"Lafferty","year":"2001"},{"key":"2020110613055562500_ocz144-B15","first-page":"1453","article-title":"Large margin methods for structured and interdependent output variables","volume":"6","author":"Tsochantaridis","year":"2005","journal-title":"J Mach Learn Res"},{"issue":"5","key":"2020110613055562500_ocz144-B16","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","article-title":"2010 i2b2\/VA challenge on concepts, assertions, and relations in clinical text","volume":"18","author":"Uzuner","year":"2011","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2020110613055562500_ocz144-B17","doi-asserted-by":"crossref","first-page":"806","DOI":"10.1136\/amiajnl-2013-001628","article-title":"Evaluating temporal relations in clinical text: 2012 i2b2 Challenge","volume":"20","author":"Sun","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613055562500_ocz144-B18","author":"Pradhan"},{"key":"2020110613055562500_ocz144-B19","doi-asserted-by":"crossref","first-page":"212","DOI":"10.1007\/978-3-642-40802-1_24","volume-title":"Information Access Evaluation Multilinguality, Multimodality, and Visualization","author":"Suominen","year":"2013"},{"key":"2020110613055562500_ocz144-B20","doi-asserted-by":"crossref","first-page":"S1","DOI":"10.1186\/1472-6947-13-S1-S1","article-title":"Recognizing clinical entities in hospital discharge summaries using structural support vector machines with word representation features","volume":"13 Suppl 1","author":"Tang","year":"2013","journal-title":"BMC Med Inform Decis Mak"},{"issue":"5","key":"2020110613055562500_ocz144-B21","doi-asserted-by":"crossref","first-page":"601","DOI":"10.1136\/amiajnl-2011-000163","article-title":"A study of machine-learning-based approaches to extract clinical entities and their assertions from discharge summaries","volume":"18","author":"Jiang","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613055562500_ocz144-B22","first-page":"1326","article-title":"A study of neural word embeddings for named entity recognition in clinical text","author":"Wu","year":"2015","journal-title":"AMIA Annu Symp Proc"},{"issue":"5","key":"2020110613055562500_ocz144-B23","doi-asserted-by":"crossref","first-page":"557","DOI":"10.1136\/amiajnl-2011-000150","article-title":"Machine-learned solutions for three stages of clinical information extraction: the state of the art at i2b2 2010","volume":"18","author":"de Bruijn","year":"2011","journal-title":"J Am Med Inform Assoc"},{"issue":"7553","key":"2020110613055562500_ocz144-B24","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"2020110613055562500_ocz144-B25","first-page":"2493","article-title":"Natural language processing (almost) from scratch","volume":"12","author":"Collobert","year":"2011","journal-title":"J Mach Learn Res"},{"key":"2020110613055562500_ocz144-B26","article-title":"Efficient estimation of word representations in vector space","author":"Mikolov","year":"2013"},{"key":"2020110613055562500_ocz144-B27","author":"Pennington","year":"2014"},{"key":"2020110613055562500_ocz144-B28","first-page":"1812","article-title":"Clinical named entity recognition using deep learning models","author":"Wu","year":"2017","journal-title":"AMIA Annu Symp Proc"},{"issue":"S2","key":"2020110613055562500_ocz144-B29","doi-asserted-by":"crossref","first-page":"2018","DOI":"10.1186\/s12911-017-0468-7","article-title":"Entity recognition from clinical texts via recurrent neural network","volume":"17","author":"Liu","year":"2017","journal-title":"BMC Med Inform Decis Mak"},{"key":"2020110613055562500_ocz144-B30","first-page":"473","article-title":"Bidirectional RNN for medical event detection in electronic health records","volume":"2016","author":"Jagannatha","year":"2016","journal-title":"Proc Conf"},{"key":"2020110613055562500_ocz144-B31","first-page":"624","article-title":"Named entity recognition in chinese clinical text using deep neural network","volume":"216","author":"Wu","year":"2015","journal-title":"Stud Health Technol Inform"},{"issue":"8","key":"2020110613055562500_ocz144-B32","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput"},{"key":"2020110613055562500_ocz144-B33","doi-asserted-by":"crossref","article-title":"Adverse drug event detection from electronic health records using hierarchical recurrent neural networks with dual-level embedding","author":"Wunnava","DOI":"10.1007\/s40264-018-0765-9"},{"issue":"1","key":"2020110613055562500_ocz144-B34","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1007\/s40264-018-0761-0","article-title":"MADEx: a system for detecting medications, adverse drug events, and their relations from clinical notes","volume":"42","author":"Yang","year":"2019","journal-title":"Drug Saf"},{"key":"2020110613055562500_ocz144-B35","author":"Kumar","year":"2017"},{"key":"2020110613055562500_ocz144-B36","doi-asserted-by":"crossref","article-title":"Towards drug safety surveillance and pharmacovigilance: current progress in detecting medication and adverse drug events from electronic health records","author":"Liu","DOI":"10.1007\/s40264-018-0766-8"},{"issue":"5","key":"2020110613055562500_ocz144-B37","doi-asserted-by":"crossref","first-page":"828","DOI":"10.1136\/amiajnl-2013-001635","article-title":"A hybrid system for temporal information extraction from clinical text","volume":"20","author":"Tang","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613055562500_ocz144-B38","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1016\/j.neucom.2016.07.082","article-title":"Recurrent convolutional neural network for answer selection in community question answering","volume":"274","author":"Zhou","year":"2018","journal-title":"Neurocomputing"},{"key":"2020110613055562500_ocz144-B39","doi-asserted-by":"crossref","first-page":"785","DOI":"10.1145\/2939672.2939785","volume-title":"Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Chen","year":"2016"},{"issue":"1","key":"2020110613055562500_ocz144-B40","doi-asserted-by":"crossref","first-page":"160035","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Sci Data"},{"key":"2020110613055562500_ocz144-B41","author":"Lample","year":"2016"},{"key":"2020110613055562500_ocz144-B42","article-title":"Combine factual medical knowledge and distributed word representation to improve clinical named entity recognition","author":"Wu"},{"key":"2020110613055562500_ocz144-B43","doi-asserted-by":"crossref","first-page":"343.","DOI":"10.1038\/msb.2009.98","article-title":"A side effect resource to capture phenotypic effects of drugs","volume":"6","author":"Kuhn","year":"2010","journal-title":"Mol Syst Biol"},{"key":"2020110613055562500_ocz144-B44","author":"Reimers","year":"2017"},{"key":"2020110613055562500_ocz144-B45","author":"Joulin"},{"issue":"3","key":"2020110613055562500_ocz144-B46","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1961189.1961199","article-title":"LIBSVM: a library for support vector machines","volume":"2","author":"Chang","year":"2011","journal-title":"ACM Trans Intell Syst Technol"},{"key":"2020110613055562500_ocz144-B47","author":"Xu","year":"2016"},{"key":"2020110613055562500_ocz144-B48","author":"Abadi"},{"key":"2020110613055562500_ocz144-B49","author":"Mikolov"},{"key":"2020110613055562500_ocz144-B50","first-page":"1113","volume-title":"Proceedings of the 26th Annual International Conference on Machine Learning","author":"Weinberger","year":"2009"},{"issue":"8","key":"2020110613055562500_ocz144-B51","doi-asserted-by":"crossref","first-page":"1965","DOI":"10.1007\/s10489-017-0920-5","article-title":"Balanced undersampling: a novel sentence-based undersampling method to improve recognition of named entities in chemical and biomedical text","volume":"48","author":"Akkasi","year":"2018","journal-title":"Appl Intell"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/1\/65\/34152788\/ocz144.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/1\/65\/34152788\/ocz144.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,1,14]],"date-time":"2021-01-14T04:25:04Z","timestamp":1610598304000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/27\/1\/65\/5555856"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,8,28]]},"references-count":51,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2019,8,28]]},"published-print":{"date-parts":[[2020,1,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocz144","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,1]]},"published":{"date-parts":[[2019,8,28]]}}}