{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T18:14:48Z","timestamp":1780337688278,"version":"3.54.1"},"reference-count":53,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2023,6,14]],"date-time":"2023-06-14T00:00:00Z","timestamp":1686700800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100006093","name":"Patient-Centered Outcomes Research Institute","doi-asserted-by":"publisher","award":["ME-2018C3-14754"],"award-info":[{"award-number":["ME-2018C3-14754"]}],"id":[{"id":"10.13039\/100006093","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"publisher","award":["1R01CA246418 R01"],"award-info":[{"award-number":["1R01CA246418 R01"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000049","name":"National Institute on Aging","doi-asserted-by":"publisher","award":["NIA R21AG062884"],"award-info":[{"award-number":["NIA R21AG062884"]}],"id":[{"id":"10.13039\/100000049","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Cancer Informatics and eHealth"},{"DOI":"10.13039\/100018776","name":"UF Health Cancer Center","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100018776","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100018488","name":"UF Clinical and Translational Science Institute","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100018488","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,8,18]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To develop a natural language processing system that solves both clinical concept extraction and relation extraction in a unified prompt-based machine reading comprehension (MRC) architecture with good generalizability for cross-institution applications.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>We formulate both clinical concept extraction and relation extraction using a unified prompt-based MRC architecture and explore state-of-the-art transformer models. We compare our MRC models with existing deep learning models for concept extraction and end-to-end relation extraction using 2 benchmark datasets developed by the 2018 National NLP Clinical Challenges (n2c2) challenge (medications and adverse drug events) and the 2022 n2c2 challenge (relations of social determinants of health [SDoH]). We also evaluate the transfer learning ability of the proposed MRC models in a cross-institution setting. We perform error analyses and examine how different prompting strategies affect the performance of MRC models.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results and Conclusion<\/jats:title>\n                  <jats:p>The proposed MRC models achieve state-of-the-art performance for clinical concept and relation extraction on the 2 benchmark datasets, outperforming previous non-MRC transformer models. GatorTron-MRC achieves the best strict and lenient F1-scores for concept extraction, outperforming previous deep learning models on the 2 datasets by 1%\u20133% and 0.7%\u20131.3%, respectively. For end-to-end relation extraction, GatorTron-MRC and BERT-MIMIC-MRC achieve the best F1-scores, outperforming previous deep learning models by 0.9%\u20132.4% and 10%\u201311%, respectively. For cross-institution evaluation, GatorTron-MRC outperforms traditional GatorTron by 6.4% and 16% for the 2 datasets, respectively. The proposed method is better at handling nested\/overlapped concepts, extracting relations, and has good portability for cross-institute applications. Our clinical MRC package is publicly available at https:\/\/github.com\/uf-hobi-informatics-lab\/ClinicalTransformerMRC.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocad107","type":"journal-article","created":{"date-parts":[[2023,6,15]],"date-time":"2023-06-15T03:53:32Z","timestamp":1686801212000},"page":"1486-1493","source":"Crossref","is-referenced-by-count":27,"title":["Clinical concept and relation extraction using prompt-based machine reading comprehension"],"prefix":"10.1093","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1994-893X","authenticated-orcid":false,"given":"Cheng","family":"Peng","sequence":"first","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida , Gainesville, Florida, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xi","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida , Gainesville, Florida, USA"},{"name":"Cancer Informatics Shared Resource, University of Florida Health Cancer Center , Gainesville, Florida, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zehao","family":"Yu","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida , Gainesville, Florida, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2238-5429","authenticated-orcid":false,"given":"Jiang","family":"Bian","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida , Gainesville, Florida, USA"},{"name":"Cancer Informatics Shared Resource, University of Florida Health Cancer Center , Gainesville, Florida, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9881-1017","authenticated-orcid":false,"given":"William R","family":"Hogan","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida , Gainesville, Florida, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6780-6135","authenticated-orcid":false,"given":"Yonghui","family":"Wu","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida , Gainesville, Florida, USA"},{"name":"Cancer Informatics Shared Resource, University of Florida Health Cancer Center , Gainesville, Florida, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2023,6,14]]},"reference":[{"key":"2023081808483587400_ocad107-B1","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.jbi.2017.11.011","article-title":"Clinical information extraction applications: a literature review","volume":"77","author":"Wang","year":"2018","journal-title":"J Biomed Inform"},{"issue":"5","key":"2023081808483587400_ocad107-B2","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","article-title":"2010 i2b2\/VA challenge on concepts, assertions, and relations in clinical text","volume":"18","author":"Uzuner","year":"2011","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2023081808483587400_ocad107-B3","doi-asserted-by":"crossref","first-page":"806","DOI":"10.1136\/amiajnl-2013-001628","article-title":"Evaluating temporal relations in clinical text: 2012 i2b2 challenge","volume":"20","author":"Sun","year":"2013","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2023081808483587400_ocad107-B4","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1093\/jamia\/ocz166","article-title":"2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records","volume":"27","author":"Henry","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2023081808483587400_ocad107-B5","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocad012","article-title":"The 2022 n2c2\/UW shared task on extracting social determinants of health","author":"Lybarger","year":"2023","journal-title":"J Am Med Inform Assoc"},{"issue":"18","key":"2023081808483587400_ocad107-B6","doi-asserted-by":"crossref","first-page":"8319","DOI":"10.3390\/app11188319","article-title":"A survey on recent named entity recognition and relationship extraction techniques on clinical texts","volume":"11","author":"Bose","year":"2021","journal-title":"Appl Sci"},{"key":"2023081808483587400_ocad107-B7","doi-asserted-by":"crossref","first-page":"103526","DOI":"10.1016\/j.jbi.2020.103526","article-title":"Clinical concept extraction: a methodology review","volume":"109","author":"Fu","year":"2020","journal-title":"J Biomed Inform"},{"issue":"1","key":"2023081808483587400_ocad107-B8","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1093\/jamia\/ocz144","article-title":"Identifying relations of medications with adverse drug events using recurrent convolutional neural networks and gradient boosting","volume":"27","author":"Yang","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2023081808483587400_ocad107-B9","doi-asserted-by":"crossref","first-page":"103285","DOI":"10.1016\/j.jbi.2019.103285","article-title":"A two-stage deep learning approach for extracting entities and relationships from medical texts","volume":"99","author":"Su\u00e1rez-Paniagua","year":"2019","journal-title":"J Biomed Inform"},{"issue":"1","key":"2023081808483587400_ocad107-B10","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1007\/s40264-018-0761-0","article-title":"MADEx: a system for detecting medications, adverse drug events, and their relations from clinical notes","volume":"42","author":"Yang","year":"2019","journal-title":"Drug Saf"},{"issue":"Suppl 5","key":"2023081808483587400_ocad107-B11","doi-asserted-by":"crossref","first-page":"232","DOI":"10.1186\/s12911-019-0935-4","article-title":"A study of deep learning methods for de-identification of clinical notes in cross-institute settings","volume":"19","author":"Yang","year":"2019","journal-title":"BMC Med Inform Decis Mak"},{"issue":"2","key":"2023081808483587400_ocad107-B12","doi-asserted-by":"crossref","first-page":"560","DOI":"10.4338\/ACI-2016-12-RA-0211","article-title":"The effects of natural language processing on cross-institutional portability of influenza case detection for disease surveillance","volume":"8","author":"Ferraro","year":"2017","journal-title":"Appl Clin Inform"},{"issue":"3","key":"2023081808483587400_ocad107-B13","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1093\/jamia\/ocx138","article-title":"Clinical documentation variations and NLP system portability: a case study in asthma birth cohorts across institutions","volume":"25","author":"Sohn","year":"2018","journal-title":"J Am Med Inform Assoc"},{"key":"2023081808483587400_ocad107-B14","first-page":"5849","article-title":"A unified MRC framework for named entity recognition","author":"Li","year":"2020"},{"issue":"1","key":"2023081808483587400_ocad107-B15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-022-00742-2","article-title":"A large language model for electronic health records","volume":"5","author":"Yang","year":"2022","journal-title":"NPJ Digit Med"},{"key":"2023081808483587400_ocad107-B16","doi-asserted-by":"crossref","first-page":"272","DOI":"10.1016\/j.jbi.2014.06.006","article-title":"Complex epilepsy phenotype extraction from narrative clinical discharge summaries","volume":"51","author":"Cui","year":"2014","journal-title":"J Biomed Inform"},{"issue":"1","key":"2023081808483587400_ocad107-B17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-019-0122-0","article-title":"Deep learning and alternative learning strategies for retrospective real-world clinical data","volume":"2","author":"Chen","year":"2019","journal-title":"NPJ Digit Med"},{"key":"2023081808483587400_ocad107-B18","author":"Wang","year":"2021"},{"key":"2023081808483587400_ocad107-B19","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1016\/j.jbi.2015.02.009","article-title":"Regular expression-based learning to extract bodyweight values from clinical notes","volume":"54","author":"Murtaugh","year":"2015","journal-title":"J Biomed Inform"},{"issue":"3","key":"2023081808483587400_ocad107-B20","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1136\/jamia.2009.002733","article-title":"An overview of MetaMap: historical perspective and recent advances","volume":"17","author":"Aronson","year":"2010","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"2023081808483587400_ocad107-B21","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1136\/jamia.1994.95236146","article-title":"A general natural-language text processor for clinical radiology","volume":"1","author":"Friedman","year":"1994","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2023081808483587400_ocad107-B22","doi-asserted-by":"crossref","first-page":"425","DOI":"10.1016\/j.jbi.2013.01.006","article-title":"An enhanced CRFs-based system for information extraction from radiology reports","volume":"46","author":"Esuli","year":"2013","journal-title":"J Biomed Inform"},{"key":"2023081808483587400_ocad107-B23","doi-asserted-by":"crossref","first-page":"S1","DOI":"10.1186\/1472-6947-13-S1-S1","article-title":"Recognizing clinical entities in hospital discharge summaries using structural support vector machines with word representation features","volume":"13","author":"Tang","year":"2013","journal-title":"BMC Med Inform Decis Mak"},{"key":"2023081808483587400_ocad107-B24","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1016\/j.media.2017.04.002","article-title":"Convolutional neural network regression for short-axis left ventricle segmentation in cardiac cine MR sequences","volume":"39","author":"Tan","year":"2017","journal-title":"Med Image Anal"},{"key":"2023081808483587400_ocad107-B25","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1016\/j.jbi.2017.11.007","article-title":"Recurrent neural networks with specialized word embeddings for health-domain named-entity recognition","volume":"76","author":"Jauregi Unanue","year":"2017","journal-title":"J Biomed Inform"},{"key":"2023081808483587400_ocad107-B26","first-page":"65","author":"Baytas","year":"2017"},{"key":"2023081808483587400_ocad107-B27","first-page":"260","article-title":"Neural architectures for named entity recognition","author":"Lample","year":"2016"},{"key":"2023081808483587400_ocad107-B28","first-page":"6470","article-title":"Named entity recognition as dependency parsing","author":"Yu","year":"2020"},{"key":"2023081808483587400_ocad107-B29","doi-asserted-by":"crossref","first-page":"S34","DOI":"10.1016\/j.jbi.2017.05.023","article-title":"De-identification of clinical notes via recurrent neural network and conditional random field","volume":"75","author":"Liu","year":"2017","journal-title":"J Biomed Inform"},{"key":"2023081808483587400_ocad107-B30","doi-asserted-by":"publisher","first-page":"1532","DOI":"10.3115\/v1\/D14-1162","author":"Pennington","year":"2014"},{"key":"2023081808483587400_ocad107-B31","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1612.03651","article-title":"FastText.zip: compressing text classification models","author":"Joulin","year":"2016"},{"key":"2023081808483587400_ocad107-B32","first-page":"5998","author":"Vaswani","year":"2017"},{"key":"2023081808483587400_ocad107-B33","article-title":"BERT: pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2019"},{"key":"2023081808483587400_ocad107-B34","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1909.11942","article-title":"ALBERT: a lite BERT for self-supervised learning of language representations","author":"Lan","year":"2020"},{"key":"2023081808483587400_ocad107-B35","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1907.11692","article-title":"RoBERTa: a robustly optimized BERT pretraining approach","author":"Liu","year":"2019"},{"key":"2023081808483587400_ocad107-B36","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2003.10555","article-title":"ELECTRA: pre-training text encoders as discriminators rather than generators","author":"Clark","year":"2020"},{"issue":"12","key":"2023081808483587400_ocad107-B37","doi-asserted-by":"crossref","first-page":"1935","DOI":"10.1093\/jamia\/ocaa189","article-title":"Clinical concept extraction using transformers","volume":"27","author":"Yang","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2023081808483587400_ocad107-B38","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1145\/2390068.2390073","author":"Tang","year":"2012"},{"issue":"1","key":"2023081808483587400_ocad107-B39","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1007\/s40264-018-0763-y","article-title":"Detecting adverse drug events with rapidly trained classification models","volume":"42","author":"Chapman","year":"2019","journal-title":"Drug Saf"},{"issue":"Suppl 1","key":"2023081808483587400_ocad107-B40","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1186\/s12911-019-0736-9","article-title":"Integrating shortest dependency path and sentence sequence into a deep learning framework for relation extraction in clinical text","volume":"19","author":"Li","year":"2019","journal-title":"BMC Med Inform Decis Mak"},{"issue":"1","key":"2023081808483587400_ocad107-B41","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1093\/jamia\/ocz101","article-title":"Adverse drug events and medication relation extraction in electronic health records with ensemble deep learning methods","volume":"27","author":"Christopoulou","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2023081808483587400_ocad107-B42","article-title":"Clinical relation extraction using transformer-based models","author":"Yang","year":"2021"},{"issue":"1","key":"2023081808483587400_ocad107-B43","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1093\/jamia\/ocz075","article-title":"An ensemble of neural models for nested adverse drug events and medication extraction with subwords","volume":"27","author":"Ju","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2023081808483587400_ocad107-B44","first-page":"3816","article-title":"Making pre-trained language models better few-shot learners","author":"Gao","year":"2021"},{"key":"2023081808483587400_ocad107-B45","doi-asserted-by":"crossref","first-page":"103799","DOI":"10.1016\/j.jbi.2021.103799","article-title":"Biomedical named entity recognition using BERT in the machine reading comprehension framework","volume":"118","author":"Sun","year":"2021","journal-title":"J Biomed Inform"},{"key":"2023081808483587400_ocad107-B46","first-page":"333","article-title":"Zero-shot relation extraction via reading comprehension","author":"Levy","year":"2017"},{"key":"2023081808483587400_ocad107-B47","first-page":"1340","article-title":"Entity-relation extraction as multi-turn question answering","author":"Li","year":"2019"},{"key":"2023081808483587400_ocad107-B48","doi-asserted-by":"crossref","first-page":"160035","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Sci Data"},{"key":"2023081808483587400_ocad107-B49","first-page":"38","article-title":"HuggingFace\u2019s transformers: state-of-the-art natural language processing","author":"Wolf","year":"2020"},{"key":"2023081808483587400_ocad107-B50","first-page":"1","author":"Xu"},{"issue":"5","key":"2023081808483587400_ocad107-B51","doi-asserted-by":"crossref","first-page":"701","DOI":"10.2967\/jnumed.123.265687","article-title":"An opinion on ChatGPT in health care\u2014written by humans only","volume":"64","author":"Kleesiek","year":"2023","journal-title":"J Nucl Med"},{"key":"2023081808483587400_ocad107-B52","first-page":"4582","article-title":"Prefix-tuning: optimizing continuous prompts for generation","author":"Li","year":"2021"},{"key":"2023081808483587400_ocad107-B53","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2110.07602","article-title":"P-tuning v2: prompt tuning can be comparable to fine-tuning universally across scales and tasks","author":"Liu","year":"2022"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/9\/1486\/51141528\/ocad107.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/9\/1486\/51141528\/ocad107.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,18]],"date-time":"2023-08-18T10:33:08Z","timestamp":1692354788000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/30\/9\/1486\/7198189"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,14]]},"references-count":53,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2023,6,14]]},"published-print":{"date-parts":[[2023,8,18]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocad107","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,9,1]]},"published":{"date-parts":[[2023,6,14]]}}}