{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T02:06:14Z","timestamp":1781057174075,"version":"3.54.1"},"reference-count":37,"publisher":"Springer Science and Business Media LLC","issue":"S3","license":[{"start":{"date-parts":[[2022,9,6]],"date-time":"2022-09-06T00:00:00Z","timestamp":1662422400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,9,6]],"date-time":"2022-09-06T00:00:00Z","timestamp":1662422400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Covance","award":["Covance contract to the UTHealth"],"award-info":[{"award-number":["Covance contract to the UTHealth"]}]},{"name":"NLM","award":["R01LM009886"],"award-info":[{"award-number":["R01LM009886"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Background<\/jats:title><jats:p>Clinical trial protocols are the foundation for advancing medical sciences, however, the extraction of accurate and meaningful information from the original clinical trials is very challenging due to the complex and unstructured texts of such documents. Named entity recognition (NER) is a fundamental and necessary step to process and standardize the unstructured text in clinical trials using Natural Language Processing (NLP) techniques.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>In this study we fine-tuned pre-trained language models to support the NER task on clinical trial eligibility criteria. We systematically investigated four pre-trained contextual embedding models for the biomedical domain (i.e., BioBERT, BlueBERT, PubMedBERT, and SciBERT) and two models for the open domains (BERT and SpanBERT), for NER tasks using three existing clinical trial eligibility criteria corpora. In addition, we also investigated the feasibility of data augmentation approaches and evaluated their performance.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Our evaluation results using tenfold cross-validation show that domain-specific transformer models achieved better performance than the general transformer models, with the best performance obtained by the PubMedBERT model (F1-scores of 0.715, 0.836, and 0.622 for the three corpora respectively). The data augmentation results show that it is feasible to leverage additional corpora to improve NER performance.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>Findings from this study not only demonstrate the importance of contextual embeddings trained from domain-specific corpora, but also shed lights on the benefits of leveraging multiple data sources for the challenging NER task in clinical trial eligibility criteria text.<\/jats:p><\/jats:sec>","DOI":"10.1186\/s12911-022-01967-7","type":"journal-article","created":{"date-parts":[[2022,9,6]],"date-time":"2022-09-06T12:03:40Z","timestamp":1662465820000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":36,"title":["A comparative study of pre-trained language models for named entity recognition in clinical trial eligibility criteria from multiple corpora"],"prefix":"10.1186","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9949-7007","authenticated-orcid":false,"given":"Jianfu","family":"Li","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qiang","family":"Wei","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Omid","family":"Ghiasvand","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Miao","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Victor","family":"Lobanov","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chunhua","family":"Weng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hua","family":"Xu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,9,6]]},"reference":[{"issue":"3","key":"1967_CR1","doi-asserted-by":"crossref","first-page":"451","DOI":"10.1016\/j.jbi.2009.12.004","volume":"43","author":"C Weng","year":"2011","unstructured":"Weng C, Tu SW, Sim I, Richesson R. Formal representations of eligibility criteria: a literature review. J Biomed Inform. 2011;43(3):451\u201367.","journal-title":"J Biomed Inform"},{"issue":"27","key":"1967_CR2","doi-asserted-by":"crossref","first-page":"7329","DOI":"10.1073\/pnas.1510502113","volume":"113","author":"G Hripcsak","year":"2016","unstructured":"Hripcsak G, Ryan PB, Duke JD, Shah NH, Park RW, Huser V, et al. Characterizing treatment pathways at scale using the OHDSI network. Proc Natl Acad Sci. 2016;113(27):7329\u201336.","journal-title":"Proc Natl Acad Sci"},{"key":"1967_CR3","first-page":"569","volume":"216","author":"Z He","year":"2015","unstructured":"He Z, Wang S, Borhanian E, Weng C. Assessing the collective population representativeness of related type 2 diabetes trials by combining public data from Clinical Trials.gov and NHANES. Stud Health Technol Inform. 2015;216:569.","journal-title":"Stud Health Technol Inform."},{"issue":"April","key":"1967_CR4","doi-asserted-by":"crossref","first-page":"1062","DOI":"10.1093\/jamia\/ocx019","volume":"24","author":"T Kang","year":"2017","unstructured":"Kang T, Zhang S, Tang Y, Hruby GW, Rusanov A, Weng C. EliIE: an open-source information extraction system for clinical trial eligibility criteria. J Am Med Inf Assoc. 2017;24(April):1062\u201371.","journal-title":"J Am Med Inf Assoc"},{"key":"1967_CR5","unstructured":"Chen M, Du F, Lan G, Lobanov V. Using pre-trained transformer deep learning models to identify named entities and syntactic relations for clinical protocol analysis. In: AAAI spring symposium: combining machine learning with knowledge engineering. 2020."},{"key":"1967_CR6","unstructured":"Devlin J, Chang MW, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the conference on NAACL HLT 2019. Association for computational linguistics (ACL); 2019; p. 4171\u201386."},{"issue":"4","key":"1967_CR7","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","volume":"36","author":"J Lee","year":"2020","unstructured":"Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020;36(4):1234\u201340.","journal-title":"Bioinformatics"},{"key":"1967_CR8","doi-asserted-by":"crossref","unstructured":"Peng Y, Yan S, Lu Z. Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets. In: Proceedings of the BioNLP 2019 workshop. 2019; p. 58\u201365.","DOI":"10.18653\/v1\/W19-5006"},{"issue":"1","key":"1967_CR9","first-page":"1","volume":"3","author":"Y Gu","year":"2021","unstructured":"Gu Y, Tinn R, Cheng H, Lucas M, Usuyama N, Liu X, Naumann T, Gao J, Poon H. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans Comput Healthc (HEALTH). 2021;3(1):1\u201323.","journal-title":"ACM Trans Comput Healthc (HEALTH)"},{"key":"1967_CR10","doi-asserted-by":"crossref","unstructured":"Beltagy I, Lo K, Cohan A. SciBERT: a pretrained language model for scientific text. In: Proceedings of the 2019 conference on EMNLP-IJCNLP. 2019; p. 3615\u201320.","DOI":"10.18653\/v1\/D19-1371"},{"key":"1967_CR11","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1162\/tacl_a_00300","volume":"8","author":"M Joshi","year":"2019","unstructured":"Joshi M, Chen D, Liu Y, Weld DS, Zettlemoyer L, Levy O. SpanBERT: Improving pre-training by representing and predicting spans. Trans Assoc Comput Linguist. 2019;8:64\u201377.","journal-title":"Trans Assoc Comput Linguist"},{"key":"1967_CR12","doi-asserted-by":"crossref","unstructured":"Kury F, Butler A, Yuan C, Fu L heng, Sun Y, Liu H, et al. Chia, a large annotated corpus of clinical trial eligibility criteria. Sci Data. 2020;7(1).","DOI":"10.1038\/s41597-020-00620-0"},{"key":"1967_CR13","doi-asserted-by":"crossref","unstructured":"Armengol-Estap\u00e9 J, Soares F, Marimon M, Krallinger M. PharmacoNER tagger: a deep learning-based tool for automatically finding chemicals and drugs in spanish medical texts. Genom Inform. 2019;17(2).","DOI":"10.5808\/GI.2019.17.2.e15"},{"issue":"77","key":"1967_CR14","first-page":"34","volume":"2018","author":"Y Wang","year":"2017","unstructured":"Wang Y, Wang L, Rastegar-Mojarad M, Moon S, Shen F, Afzal N, et al. Clinical information extraction applications: a literature review. J Biomed Inform. 2017;2018(77):34\u201349.","journal-title":"J Biomed Inform"},{"issue":"1","key":"1967_CR15","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1093\/jamia\/ocz063","volume":"27","author":"Q Wei","year":"2020","unstructured":"Wei Q, Ji Z, Li Z, Du J, Wang J, Xu J, Xiang Y, Tiryaki F, Wu S, Zhang Y, Tao C. A study of deep learning approaches for medication and adverse drug event extraction from clinical text. J Am Med Inform Assoc. 2020;27(1):13\u201321.","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"1967_CR16","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","volume":"18","author":"\u00d6 Uzuner","year":"2011","unstructured":"Uzuner \u00d6, South BR, Shen S, DuVall SL. 2010 i2b2\/VA challenge on concepts, assertions, and relations in clinical text. J Am Med Inform Assoc. 2011;18(5):552\u20136.","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"1967_CR17","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1093\/jamia\/ocz166","volume":"27","author":"S Henry","year":"2020","unstructured":"Henry S, Buchan K, Filannino M, Stubbs A, Uzuner O. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. J Am Med Inform Assoc. 2020;27(1):3\u201312.","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"1967_CR18","doi-asserted-by":"crossref","first-page":"514","DOI":"10.1136\/jamia.2010.003947","volume":"17","author":"\u00d6 Uzuner","year":"2010","unstructured":"Uzuner \u00d6, Solti I, Cadag E. Extracting medication information from clinical text. J Am Med Inform Assoc. 2010;17(5):514\u20138.","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"1967_CR19","doi-asserted-by":"crossref","first-page":"601","DOI":"10.1136\/amiajnl-2011-000163","volume":"18","author":"M Jiang","year":"2011","unstructured":"Jiang M, Chen Y, Liu M, Rosenbloom ST, Mani S, Denny JC, et al. A study of machine-learning-based approaches to extract clinical entities and their assertions from discharge summaries. J Am Med Inform Assoc. 2011;18(5):601\u20136.","journal-title":"J Am Med Inform Assoc"},{"key":"1967_CR20","doi-asserted-by":"crossref","unstructured":"Tang B, Cao H, Wu Y, Jiang M, Xu H. Clinical entity recognition using structural support vector machines with rich features. In: Proceedings of the ACM sixth international workshop on data and text mining in biomedical informatics 2012; p. 13\u201320.","DOI":"10.1145\/2390068.2390073"},{"issue":"23","key":"1967_CR21","doi-asserted-by":"crossref","first-page":"4087","DOI":"10.1093\/bioinformatics\/bty449","volume":"34","author":"JM Giorgi","year":"2018","unstructured":"Giorgi JM, Bader GD. Transfer learning for biomedical named entity recognition with neural networks. Bioinformatics. 2018;34(23):4087\u201394.","journal-title":"Bioinformatics"},{"issue":"12","key":"1967_CR22","doi-asserted-by":"crossref","first-page":"1935","DOI":"10.1093\/jamia\/ocaa189","volume":"27","author":"X Yang","year":"2020","unstructured":"Yang X, Bian J, Hogan WR, Wu Y. Clinical concept extraction using transformers. J Am Med Inform Assoc. 2020;27(12):1935\u201342.","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"1967_CR23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-019-56847-4","volume":"10","author":"Y Kim","year":"2020","unstructured":"Kim Y, Lee JH, Choi S, Lee JM, Kim JH, Seok J, et al. Validation of deep learning natural language processing algorithm for keyword extraction from pathology reports in electronic health records. Sci Rep. 2020;10(1):1\u20139.","journal-title":"Sci Rep"},{"key":"1967_CR24","unstructured":"Tu SW, Peleg M, Carini S, Rubin D, Sim I. ERGO: a template-based expression language for encoding eligibility criteria. Technical report, 2009. (Accessed 03\/20\/2022 from https:\/\/storage.googleapis.com\/google-code-archive-downloads\/v2\/code.google.com\/ontology-of-clinical-research\/ERGO_Technical_Documentation.pdf)"},{"key":"1967_CR25","unstructured":"Tu SW, Musen MA. The EON model of intervention protocols and guidelines. In: Proceedings of the AMIA annual fall symposium. american medical informatics association; 1996; p. 587."},{"issue":"4","key":"1967_CR26","doi-asserted-by":"crossref","first-page":"781","DOI":"10.1093\/jamia\/ocw176","volume":"24","author":"K Zhang","year":"2017","unstructured":"Zhang K, Demner-Fushman D. Automated classification of eligibility criteria in clinical trials to facilitate patient-trial matching for specific patient populations. J Am Med Inform Assoc. 2017;24(4):781\u20137.","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"1967_CR27","doi-asserted-by":"crossref","first-page":"294","DOI":"10.1093\/jamia\/ocy178","volume":"26","author":"C Yuan","year":"2019","unstructured":"Yuan C, Ryan PB, Ta C, Guo Y, Li Z, Hardin J, et al. Criteria2Query: a natural language interface to clinical databases for cohort definition. J Am Med Inform Assoc. 2019;26(4):294\u2013305.","journal-title":"J Am Med Inform Assoc"},{"key":"1967_CR28","doi-asserted-by":"crossref","unstructured":"Zhu Y, Kiros R, Zemel R, Salakhutdinov R, Urtasun R, Torralba A, et al. Aligning books and movies: towards story-like visual explanations by watching movies and reading books. In: Proceedings of the IEEE international conference on computer vision. 2015; p. 19\u201327.","DOI":"10.1109\/ICCV.2015.11"},{"key":"1967_CR29","unstructured":"Wu Y, Schuster M, Chen Z, Le QV, Norouzi M, Macherey W, Krikun M, Cao Y, Gao Q, Macherey K, Klingner J. Google\u2019s neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. 2016."},{"issue":"3","key":"1967_CR30","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1093\/jamia\/ocx132","volume":"25","author":"E Soysal","year":"2018","unstructured":"Soysal E, Wang J, Jiang M, Wu Y, Pakhomov S, Liu H, et al. CLAMP\u2013a toolkit for efficiently building customized clinical natural language processing pipelines. J Am Med Inform Assoc. 2018;25(3):331\u20136.","journal-title":"J Am Med Inform Assoc"},{"key":"1967_CR31","doi-asserted-by":"crossref","unstructured":"Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, et al. HuggingFace\u2019s transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 EMNLP (Systems Demonstrations), 2020; p. 38\u201345.","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"1967_CR32","doi-asserted-by":"crossref","first-page":"S11","DOI":"10.1016\/j.jbi.2015.06.007","volume":"58","author":"A Stubbs","year":"2015","unstructured":"Stubbs A, Kotfila C, Uzuner \u00d6. Automated systems for the de-identification of longitudinal clinical narratives: overview of 2014 i2b2\/UTHealth shared task Track 1. J Biomed Inform. 2015;58:S11\u20139.","journal-title":"J Biomed Inform"},{"key":"1967_CR33","volume-title":"Probability and statistical inference","author":"RV Hogg","year":"2010","unstructured":"Hogg RV, Tanis EA, Zimmerman DL. Probability and statistical inference. Upper Saddle River: Pearson\/Prentice Hall; 2010."},{"key":"1967_CR34","unstructured":"Tang B, Chen Q, Wang X, Wu Y, Zhang Y, Jiang M, et al. Recognizing disjoint clinical concepts in clinical text using machine learning-based methods. In: AMIA annual symposium proceedings. American Medical Informatics Association; 2015; p. 1184."},{"issue":"8","key":"1967_CR35","doi-asserted-by":"crossref","first-page":"1098","DOI":"10.1093\/bioinformatics\/btq075","volume":"26","author":"D Dahlmeier","year":"2010","unstructured":"Dahlmeier D, Ng HT. Domain adaptation for semantic role labeling in the biomedical domain. Bioinformatics. 2010;26(8):1098\u2013104.","journal-title":"Bioinformatics"},{"key":"1967_CR36","doi-asserted-by":"crossref","unstructured":"Peng N, Dredze M. Multi-task domain adaptation for sequence tagging. In: Proceedings of the 2nd workshop on representation learning for NLP, 2016.","DOI":"10.18653\/v1\/W17-2612"},{"key":"1967_CR37","doi-asserted-by":"crossref","unstructured":"Lin BY, Lu W. Neural adaptation layers for cross-domain named entity recognition. In: Proceedings of the 2018 conference on empirical methods in natural language processing. 2018; p. 2012\u201322.","DOI":"10.18653\/v1\/D18-1226"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-022-01967-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-022-01967-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-022-01967-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,18]],"date-time":"2023-02-18T07:18:59Z","timestamp":1676704739000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-022-01967-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,6]]},"references-count":37,"journal-issue":{"issue":"S3","published-online":{"date-parts":[[2022,9]]}},"alternative-id":["1967"],"URL":"https:\/\/doi.org\/10.1186\/s12911-022-01967-7","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,6]]},"assertion":[{"value":"21 June 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 August 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 September 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"Dr. Xu and The University of Texas Health Science Center at Houston have research-related financial interests in Melax Technologies, Inc.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"235"}}