{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,23]],"date-time":"2026-04-23T10:09:27Z","timestamp":1776938967722,"version":"3.51.4"},"reference-count":82,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,2,22]],"date-time":"2021-02-22T00:00:00Z","timestamp":1613952000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,2,22]],"date-time":"2021-02-22T00:00:00Z","timestamp":1613952000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100010665","name":"H2020 Marie Sk\u0142odowska-Curie Actions","doi-asserted-by":"publisher","award":["713366"],"award-info":[{"award-number":["713366"]}],"id":[{"id":"10.13039\/100010665","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Background<\/jats:title><jats:p>The large volume of medical literature makes it difficult for healthcare professionals to keep abreast of the latest studies that support Evidence-Based Medicine. Natural language processing enhances the access to relevant information, and gold standard corpora are required to improve systems. To contribute with a new dataset for this domain, we collected the Clinical Trials for Evidence-Based Medicine in Spanish (CT-EBM-SP) corpus.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>We annotated 1200 texts about clinical trials with entities from the Unified Medical Language System semantic groups: anatomy (ANAT), pharmacological and chemical substances (CHEM), pathologies (DISO), and lab tests, diagnostic or therapeutic procedures (PROC). We doubly annotated 10% of the corpus and measured inter-annotator agreement (IAA) using F-measure. As use case, we run medical entity recognition experiments with neural network models.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>This resource contains 500 abstracts of journal articles about clinical trials and 700 announcements of trial protocols (292 173 tokens). We annotated 46 699 entities (13.98% are nested entities). Regarding IAA agreement, we obtained an average F-measure of 85.65% (\u00b14.79, strict match) and 93.94% (\u00b13.31, relaxed match). In the use case experiments, we achieved recognition results ranging from 80.28% (\u00b100.99) to 86.74% (\u00b100.19) of average F-measure.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>Our results show that this resource is adequate for experiments with state-of-the-art approaches to biomedical named entity recognition. It is freely distributed at:<jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"http:\/\/www.lllf.uam.es\/ESP\/nlpmedterm_en.html\">http:\/\/www.lllf.uam.es\/ESP\/nlpmedterm_en.html<\/jats:ext-link>. The methods are generalizable to other languages with similar available sources.<\/jats:p><\/jats:sec>","DOI":"10.1186\/s12911-021-01395-z","type":"journal-article","created":{"date-parts":[[2021,2,22]],"date-time":"2021-02-22T10:02:58Z","timestamp":1613988178000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":32,"title":["A clinical trials corpus annotated with UMLS entities to enhance the access to evidence-based medicine"],"prefix":"10.1186","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3040-1756","authenticated-orcid":false,"given":"Leonardo","family":"Campillos-Llanos","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ana","family":"Valverde-Mateos","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Adri\u00e1n","family":"Capllonch-Carri\u00f3n","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9029-2216","authenticated-orcid":false,"given":"Antonio","family":"Moreno-Sandoval","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,2,22]]},"reference":[{"key":"1395_CR1","unstructured":"Sackett D, Strauss D, Richardson W, Rosenberg W, Haynes R. Evidence-based medicine: how to practice and teach EBM. Churchill Livingstone, Edinburgh, 2nd Ed. (2000)"},{"key":"1395_CR2","unstructured":"National Library of Medicine. ClinicalTrials.gov;. https:\/\/clinicaltrials.gov\/. Accessed 5 Sep 2020."},{"key":"1395_CR3","unstructured":"European Medicines Agency. European Union Clinical Trials Register (EudraCT). http:\/\/www.clinicaltrialsregister.eu. Accessed 5 Sep 2020."},{"issue":"01","key":"1395_CR4","first-page":"216","volume":"84","author":"AT McCray","year":"2001","unstructured":"McCray AT, Burgun A, Bodenreider O. Aggregating UMLS semantic types for reducing conceptual complexity. Stud Health Technol Inform. 2001;84(01):216\u201320.","journal-title":"Stud Health Technol Inform"},{"issue":"suppl 1","key":"1395_CR5","doi-asserted-by":"publisher","first-page":"D267","DOI":"10.1093\/nar\/gkh061","volume":"32","author":"O Bodenreider","year":"2004","unstructured":"Bodenreider O. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2004;32(suppl 1):D267\u201370.","journal-title":"Nucleic Acids Res"},{"issue":"5","key":"1395_CR6","doi-asserted-by":"publisher","first-page":"514","DOI":"10.1136\/jamia.2010.003947","volume":"17","author":"O Uzuner","year":"2010","unstructured":"Uzuner O, Solti I, Cadag E. Extracting medication information from clinical text. J Am Med Inform Assoc. 2010;17(5):514\u20138.","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"1395_CR7","doi-asserted-by":"publisher","first-page":"806","DOI":"10.1136\/amiajnl-2013-001628","volume":"20","author":"W Sun","year":"2013","unstructured":"Sun W, Rumshisky A, Uzuner O. Evaluating temporal relations in clinical text: 2012 i2b2 challenge. J Am Med Inform Assoc. 2013;20(5):806\u201313.","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"1395_CR8","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1186\/1471-2105-9-10","volume":"9","author":"JD Kim","year":"2008","unstructured":"Kim JD, Ohta T, Tsujii J. Corpus annotation for mining biomedical events from literature. BMC Bioinform. 2008;9(1):10.","journal-title":"BMC Bioinform"},{"issue":"11","key":"1395_CR9","first-page":"1","volume":"9","author":"V Vincze","year":"2008","unstructured":"Vincze V, Szarvas G, Farkas R, M\u00f3ra G, Csirik J. The BioScope corpus: biomedical texts annotated for uncertainty, negation and their scopes. BMC Bioinform. 2008;9(11):1\u20139.","journal-title":"BMC Bioinform"},{"key":"1395_CR10","first-page":"950","volume":"42","author":"A Roberts","year":"2009","unstructured":"Roberts A, Gaizauskas R, Hepple M, Demetriou G, Guo Y, Roberts I, et al. Building a semantically annotated corpus of clinical texts. J Biomed Semant. 2009;42:950\u201366.","journal-title":"J Biomed Semant"},{"issue":"1","key":"1395_CR11","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1186\/1471-2105-13-161","volume":"13","author":"M Bada","year":"2012","unstructured":"Bada M, Eckert M, Evans D, Garcia K, Shipley K, Sitnikov D, et al. Concept annotation in the CRAFT corpus. BMC Bioinform. 2012;13(1):161.","journal-title":"BMC Bioinform"},{"issue":"5","key":"1395_CR12","doi-asserted-by":"publisher","first-page":"914","DOI":"10.1016\/j.jbi.2013.07.011","volume":"46","author":"M Herrero-Zazo","year":"2013","unstructured":"Herrero-Zazo M, Segura-Bedmar I, Mart\u00ednez P, Declerck T. The DDI corpus: An annotated corpus with pharmacological substances and drug-drug interactions. J Biomed Inform. 2013;46(5):914\u201320.","journal-title":"J Biomed Inform"},{"issue":"1","key":"1395_CR13","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1186\/s13326-018-0179-8","volume":"9","author":"A N\u00e9v\u00e9ol","year":"2018","unstructured":"N\u00e9v\u00e9ol A, Dalianis H, Velupillai S, Savova G, Zweigenbaum P. Clinical natural language processing in languages other than English: opportunities and challenges. J Biomed Semant. 2018;9(1):12.","journal-title":"J Biomed Semant"},{"issue":"S2","key":"1395_CR14","doi-asserted-by":"publisher","first-page":"S5","DOI":"10.1186\/1471-2105-12-S2-S5","volume":"12","author":"SN Kim","year":"2011","unstructured":"Kim SN, Martinez D, Cavedon L, Yencken L. Springer. Automatic classification of sentences to support evidence based medicine. BMC Bioinform. 2011;12(S2):S5.","journal-title":"BMC Bioinform"},{"issue":"1","key":"1395_CR15","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1186\/1472-6947-9-10","volume":"9","author":"GY Chung","year":"2009","unstructured":"Chung GY. Sentence retrieval for abstracts of randomized controlled trials. BMC Med Inform Decis. 2009;9(1):10.","journal-title":"BMC Med Inform Decis"},{"key":"1395_CR16","unstructured":"Del\u00e9ger L, Li Q, Lingren T, Kaiser M, Molnar K, et\u00a0al. Building gold standard corpora for medical natural language processing tasks. Proc AMIA Symp. 2012;p. 144\u201353."},{"issue":"4","key":"1395_CR17","doi-asserted-by":"publisher","first-page":"705","DOI":"10.1007\/s10579-015-9327-2","volume":"50","author":"D Moll\u00e1","year":"2016","unstructured":"Moll\u00e1 D, Santiago-Mart\u00ednez ME, Sarker A, Paris C. A corpus for research in text processing for evidence based medicine. Lang Resour Eval. 2016;50(4):705\u201327.","journal-title":"Lang Resour Eval"},{"key":"1395_CR18","doi-asserted-by":"crossref","unstructured":"Nye B, Li JJ, Patel R, Yang Y, Marshall IJ, Nenkova A, et\u00a0al. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics Melbourne, Australia, 15\u201320 July. 2018;p. 197\u2013207.","DOI":"10.18653\/v1\/P18-1019"},{"key":"1395_CR19","doi-asserted-by":"crossref","unstructured":"Lehman E, DeYoung J, Barzilay R, Wallace BC. Inferring which medical treatments work from reports of clinical trials. In: Proceeding of the 2019 Conference of North American Chapter of the Association for Computational Linguistics, vol 1 Minneapolis, MN, USA, 2\u20137 June. 2019;p. 3705\u201317.","DOI":"10.18653\/v1\/N19-1371"},{"key":"1395_CR20","doi-asserted-by":"crossref","first-page":"100058","DOI":"10.1016\/j.yjbinx.2019.100058","volume":"4","author":"A Koroleva","year":"2019","unstructured":"Koroleva A, Kamath S, Paroubek P. Measuring semantic similarity of clinical trial outcomes using deep pre-trained language representations. J Biomed Inform. 2019;4:100058.","journal-title":"J Biomed Inform"},{"key":"1395_CR21","unstructured":"Devlin J, Chang M, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, vol 1 Minneapolis, MN, USA, 2\u20137 June. 2019;p. 4171\u201386."},{"key":"1395_CR22","doi-asserted-by":"publisher","first-page":"103321","DOI":"10.1016\/j.jbi.2019.103321","volume":"100","author":"H Hassanzadeh","year":"2019","unstructured":"Hassanzadeh H, Nguyen A, Verspoor K. Quantifying semantic similarity of clinical evidence in the biomedical literature to facilitate related evidence synthesis. J Biomed Inform. 2019;100:103321.","journal-title":"J Biomed Inform"},{"key":"1395_CR23","doi-asserted-by":"crossref","unstructured":"Kim JD, Ohta T, Tsuruoka Y, Tateisi Y, Collier N. Introduction to the bio-entity recognition task at JNLPBA. In: Proceedings of the international joint workshop on natural language processing in biomedicine and its applications. 2004;p. 70\u20135.","DOI":"10.3115\/1567594.1567610"},{"issue":"1","key":"1395_CR24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41597-020-00620-0","volume":"7","author":"F Kury","year":"2020","unstructured":"Kury F, Butler A, Yuan C, Fu Lh, Sun Y, Liu H, et al. Chia, a large annotated corpus of clinical trial eligibility criteria. Sci Data. 2020;7(1):1\u201311.","journal-title":"Sci Data"},{"issue":"1","key":"1395_CR25","doi-asserted-by":"publisher","first-page":"i116","DOI":"10.1136\/amiajnl-2011-000321","volume":"18","author":"C Weng","year":"2011","unstructured":"Weng C, Wu X, Luo Z, Boland MR, Theodoratos D, Johnson SB. EliXR: an approach to eligibility criteria extraction and representation. J Am Med Inform Assoc. 2011;18(1):i116\u201324.","journal-title":"J Am Med Inform Assoc"},{"issue":"6","key":"1395_CR26","doi-asserted-by":"publisher","first-page":"1062","DOI":"10.1093\/jamia\/ocx019","volume":"24","author":"T Kang","year":"2017","unstructured":"Kang T, Zhang S, Tang Y, Hruby GW, Rusanov A, Elhadad N, et al. EliIE: an open-source information extraction system for clinical trial eligibility criteria. J Am Med Inform Assoc. 2017;24(6):1062\u201371.","journal-title":"J Am Med Inform Assoc"},{"key":"1395_CR27","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1016\/j.sbspro.2013.10.619","volume":"95","author":"A Moreno-Sandoval","year":"2013","unstructured":"Moreno-Sandoval A, Campillos-Llanos L. Design and annotation of multimedica-a multilingual text corpus of the biomedical domain. Procedia Soc Behav Sci. 2013;95:33\u20139.","journal-title":"Procedia Soc Behav Sci"},{"issue":"5","key":"1395_CR28","doi-asserted-by":"publisher","first-page":"948","DOI":"10.1093\/jamia\/ocv037","volume":"22","author":"JA Kors","year":"2015","unstructured":"Kors JA, Clematide S, Akhondi SA, van Mulligen EM, Rebholz-Schuhmann D. A multilingual gold-standard corpus for biomedical concept recognition: the Mantra GSC. J Am Med Inform Assoc. 2015;22(5):948\u201356.","journal-title":"J Am Med Inform Assoc"},{"key":"1395_CR29","doi-asserted-by":"publisher","first-page":"318","DOI":"10.1016\/j.jbi.2015.06.016","volume":"56","author":"M Oronoz","year":"2015","unstructured":"Oronoz M, Gojenola K, P\u00e9rez A, de Ilarraza AD, Casillas A. On the creation of a clinical gold standard corpus in Spanish: Mining adverse drug reactions. J Biomed Inform. 2015;56:318\u201332.","journal-title":"J Biomed Inform"},{"issue":"2","key":"1395_CR30","doi-asserted-by":"publisher","first-page":"S6","DOI":"10.1186\/1472-6947-15-S2-S6","volume":"15","author":"I Segura-Bedmar","year":"2015","unstructured":"Segura-Bedmar I, Mart\u00ednez P, Revert R, Moreno-Schneider J. Exploring Spanish health social media for detecting drug effects. BMC Med Inform Decis. 2015;15(2):S6.","journal-title":"BMC Med Inform Decis"},{"key":"1395_CR31","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1016\/j.jbi.2017.06.013","volume":"72","author":"I Moreno","year":"2017","unstructured":"Moreno I, Boldrini E, Moreda P, Rom\u00e1-Ferri MT. DrugSemantics: a corpus for named entity recognition in Spanish summaries of product characteristics. J Biomed Inform. 2017;72:8\u201322.","journal-title":"J Biomed Inform"},{"key":"1395_CR32","doi-asserted-by":"crossref","unstructured":"Marim\u00f3n M, Vivaldi J, Bel N. Annotation of negation in the IULA spanish clinical record corpus. In: Proceedings of SemBEaR 2017 comput semantics beyond events roles Valencia, Spain, 4 Apr. 2017;p. 43\u201352.","DOI":"10.18653\/v1\/W17-1807"},{"key":"1395_CR33","doi-asserted-by":"crossref","unstructured":"Cotik V, Filippo D, Roller R, Uszkoreit H, Xu F. Annotation of entities and relations in spanish radiology reports. In: Proceedings of RANLP Varna, Bulgaria, 4\u20136 Sept. 2017;p. 177\u201384.","DOI":"10.26615\/978-954-452-049-6_025"},{"key":"1395_CR34","unstructured":"Intxaurrondo A, de\u00a0la Torre JC, Rodr\u00edguez\u00a0Betanco H, Marim\u00f3n M, Lopez-Mart\u00edn JA, Gonzalez-Agirre A, et\u00a0al. Resources, guidelines and annotations for the recognition, definition resolution and concept normalization of Spanish clinical abbreviations: the BARR2 corpus. In: Proceedings of SEPLN. 2018; p. 1\u20139."},{"key":"1395_CR35","doi-asserted-by":"crossref","unstructured":"Gonzalez-Agirre A, Marimon M, Intxaurrondo A, Rabal O, Villegas M, Krallinger M. PharmaCoNER: Pharmacological substances, compounds and proteins named entity recognition track. In: Proceedings of the 5th workshop on BioNLP open shared tasks Hong Kong, China, 4 Nov. 2019;p. 1\u201310.","DOI":"10.18653\/v1\/D19-5701"},{"key":"1395_CR36","first-page":"279","volume":"121","author":"K Donnelly","year":"2006","unstructured":"Donnelly K. SNOMED-CT: the advanced terminology and coding system for eHealth. Stud Health Technol Inform. 2006;121:279\u201390.","journal-title":"Stud Health Technol Inform"},{"key":"1395_CR37","unstructured":"Biomedical Text Mining Unit. CODIESP challenge;. https:\/\/temu.bsc.es\/codiesp\/. Accessed 5 Sep 2020."},{"key":"1395_CR38","unstructured":"Biomedical Text Mining Unit. CANTEMIST challenge. https:\/\/temu.bsc.es\/cantemist\/. Accessed 5 Sep 2020."},{"key":"1395_CR39","doi-asserted-by":"publisher","first-page":"103172","DOI":"10.1016\/j.jbi.2019.103172","volume":"94","author":"A Piad-Morffis","year":"2019","unstructured":"Piad-Morffis A, Guti\u00e9rrez Y, Mu\u00f1oz R. A corpus to support eHealth knowledge discovery technologies. J Biomed Inform. 2019;94:103172.","journal-title":"J Biomed Inform"},{"key":"1395_CR40","unstructured":"Mart\u00ednez\u00a0C\u00e1mara E, Almeida\u00a0Cruz Y, D\u00edaz\u00a0Galiano MC, Est\u00e9vez-Velarde S, Garc\u00eda\u00a0Cumbreras M\u00c1, Garc\u00eda\u00a0Vega M, et\u00a0al. Overview of TASS 2018: opinions, health and emotions. In: Proceedings of TASS 2018 at SEPLN, vol 2172 Sevilla, Spain, 18 Sept. 2018; p. 13\u201327."},{"key":"1395_CR41","unstructured":"Lima S, P\u00e9rez N, Cuadros M, Rigau G. NUBes: A corpus of negation and uncertainty in Spanish clinical texts. In: Proceedings of the 12th LREC Marseille, France, 11\u201316 May. 2020. p. 5772\u20135781."},{"key":"1395_CR42","doi-asserted-by":"crossref","unstructured":"B\u00e1ez P, Villena F, Rojas M, Dur\u00e1n M, Dunstan J. The Chilean Waiting List Corpus: a new resource for clinical named entity recognition in Spanish. In: Proceedings of the 3rd clinical natural language processing workshop; 2020. p. 291\u2013300.","DOI":"10.18653\/v1\/2020.clinicalnlp-1.32"},{"key":"1395_CR43","unstructured":"FAPESP - BIREME. Scientific Library Online (SciELO). https:\/\/www.scielo.org\/es\/. Accessed 5 Sep 2020."},{"key":"1395_CR44","unstructured":"National Library of Medicine. PubMed. https:\/\/pubmed.ncbi.nlm.nih.gov\/. Accessed 5 Sep 2020."},{"key":"1395_CR45","unstructured":"AEMPS. Spanish Repository of Clinical Trials (Registro Espa\u00f1ol de Ensayos Cl\u00ednicos, REEC);. https:\/\/reec.aemps.es. Accessed 5 Sep 2020."},{"issue":"3","key":"1395_CR46","doi-asserted-by":"publisher","first-page":"406","DOI":"10.1136\/amiajnl-2013-001837","volume":"21","author":"T Lingren","year":"2014","unstructured":"Lingren T, Deleger L, Molnar K, Zhai H, Meinzen-Derr J, Kaiser M, et al. Evaluating the impact of pre-annotation on annotation speed and potential bias: natural language processing gold standard development for clinical named entity recognition in clinical trial announcements. J Am Med Inform Assoc. 2014;21(3):406\u201313.","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"1395_CR47","doi-asserted-by":"publisher","first-page":"571","DOI":"10.1007\/s10579-017-9382-y","volume":"52","author":"L Campillos-Llanos","year":"2018","unstructured":"Campillos-Llanos L, Del\u00e9ger L, Grouin C, Hamon T, Ligozat AL, N\u00e9v\u00e9ol A. A French clinical corpus with comprehensive semantic annotations: development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT). Lang Resour Eval. 2018;52(2):571\u2013601.","journal-title":"Lang Resour Eval"},{"key":"1395_CR48","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1214\/aoms\/1177729694","volume":"22","author":"S Kullback","year":"1951","unstructured":"Kullback S, Leibler RA. On information and sufficiency. Ann Math Stat. 1951;22:49\u201386.","journal-title":"Ann Math Stat"},{"key":"1395_CR49","doi-asserted-by":"crossref","unstructured":"Dai X, Karimi S, Hachey B, Paris C. Using similarity measures to select pretraining data for NER. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics, vol 1 Minneapolis, MN, USA, 2\u20137 June. 2019; p. 1460\u201370.","DOI":"10.18653\/v1\/N19-1149"},{"key":"1395_CR50","doi-asserted-by":"crossref","unstructured":"Chiu B, Crichton G, Korhonen A, Pyysalo S. How to train good word embeddings for biomedical NLP. In: Proceedings of BioNLP 2016, Berlin, Germany, 12th August; 2016. p. 166\u201374.","DOI":"10.18653\/v1\/W16-2922"},{"key":"1395_CR51","unstructured":"Honnibal M, Montani I. Spacy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. 2017."},{"key":"1395_CR52","doi-asserted-by":"crossref","unstructured":"Campillos-Llanos L. First steps towards building a medical Lexicon for Spanish with linguistic and semantic information. In: Proceedings of BioNLP 2019 Florence, Italy, 1st Aug. 2019. p. 152\u201364.","DOI":"10.18653\/v1\/W19-5017"},{"key":"1395_CR53","unstructured":"RANME. Diccionario de T\u00e9rminos M\u00e9dicos (DTM). Madrid: Editorial Panamericana; 2011. http:\/\/dtme.ranm.es\/accesoRestringido.aspx."},{"key":"1395_CR54","unstructured":"Aronson AR. Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. In: Proceedings of the AMIA symposium American medical informatics association; 2001. p. 17\u201321."},{"key":"1395_CR55","unstructured":"Stenetorp P, Pyysalo S, Topi\u0107 G, Ohta T, Ananiadou S, Tsujii J. BRAT: a web-based tool for nlp-assisted text annotation. In: Proceedings of the demonstrations session at EACL. 2012; p. 102\u20137."},{"key":"1395_CR56","doi-asserted-by":"crossref","unstructured":"Finkel JR, Manning CD. Nested named entity recognition. In: Proceedings of the 2009 conference on empirical methods in natural language processing. 2009; p. 141\u201350.","DOI":"10.3115\/1699510.1699529"},{"key":"1395_CR57","unstructured":"Ogren P, Savova G, Chute C. constructing evaluation corpora for automated clinical named entity recognition. In: Proceedings of the 6th LREC Marrakech, Morocco, 28\u201330 May. 2008;p. 3143\u201350."},{"issue":"3","key":"1395_CR58","doi-asserted-by":"publisher","first-page":"296","DOI":"10.1197\/jamia.M1733","volume":"12","author":"G Hripcsak","year":"2005","unstructured":"Hripcsak G, Rothschild AS. Agreement, the F-measure, and reliability in information retrieval. J Am Med Inform Assoc. 2005;12(3):296\u20138.","journal-title":"J Am Med Inform Assoc"},{"key":"1395_CR59","unstructured":"Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, et\u00a0al. language models are few-shot learners. Preprint at arXiv. 2020; arXiv:abs\/2005.14165"},{"key":"1395_CR60","unstructured":"Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed representations of words and phrases and their compositionality. In: Proceedings of advances in neural information processing systems. 2013; p. 3111\u20139."},{"key":"1395_CR61","doi-asserted-by":"crossref","unstructured":"Pennington J, Socher R, Manning CD. Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing. 2014;p. 1532\u20131543.","DOI":"10.3115\/v1\/D14-1162"},{"key":"1395_CR62","doi-asserted-by":"crossref","unstructured":"Rei M. Semi-supervised multitask learning for sequence labeling. In: Proceedings of the 55th annual meeting of the association for computational linguistics, vol 1 Vancouver, Canada, 30 July\u20134 Aug. 2017; p. 2121\u201330. https:\/\/github.com\/marekrei\/sequence-labeler.","DOI":"10.18653\/v1\/P17-1194"},{"key":"1395_CR63","doi-asserted-by":"crossref","unstructured":"Lample G, Ballesteros M, Subramanian S, Kawakami K, Dyer C. Neural architectures for named entity recognition. In: Proceedings of the North American chapter of the association for computational linguistics, vol 1 San Diego, CA, USA, 12\u201317 June. 2016; p. 260\u201370.","DOI":"10.18653\/v1\/N16-1030"},{"key":"1395_CR64","doi-asserted-by":"crossref","unstructured":"Tourille J, Doutreligne M, Ferret O, N\u00e9v\u00e9ol A, Paris N, Tannier X. Evaluation of a sequence tagging tool for biomedical texts. In: Proceedings of the 9th international workshop on health text mining and information analysis. 2018; p. 193\u2013203.","DOI":"10.18653\/v1\/W18-5622"},{"key":"1395_CR65","first-page":"135","volume":"5","author":"P Bojanowski","year":"2017","unstructured":"Bojanowski P, Grave E, Joulin A, Mikolov T. Enriching word vectors with subword information. T Assoc Comp Ling. 2017;5:135\u201346.","journal-title":"T Assoc Comp Ling"},{"key":"1395_CR66","unstructured":"Akbik A, Blythe D, Vollgraf R. Contextual string embeddings for sequence labeling. In: Proceedings of the 27th international conference on computational linguistics Santa Fe, NM, USA, 20\u201326 Aug. 2018;p. 1638\u201349."},{"key":"1395_CR67","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et\u00a0al. Attention is all you need. In: Proceedings of advances in neural information processing systems. 2017; p. 5998\u20136008."},{"key":"1395_CR68","unstructured":"Ca\u00f1ete J, Chaperon G, Fuentes R, P\u00e9rez J. Spanish pre-trained BERT model and evaluation data. PML4DC at ICLR 2020 Addis Ababa, Ethiopia, 26 Apr. 2020; p. 1\u201310."},{"key":"1395_CR69","doi-asserted-by":"crossref","unstructured":"Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, et\u00a0al. HuggingFace\u2019s transformers: state-of-the-art natural language processing. Preprint at arXiv. 2019; arXiv:abs\/1910.03771.","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"1395_CR70","doi-asserted-by":"crossref","unstructured":"Ratinov L, Roth D. Design challenges and misconceptions in named entity recognition. In: Proceedings of the 13th conference on computational natural language learning (CoNLL-2009). 2009;p. 147\u201355.","DOI":"10.3115\/1596374.1596399"},{"key":"1395_CR71","unstructured":"Tiedemann J. Parallel data, tools and interfaces in OPUS. In: Proceedings of the 8th LREC Istanbul, Turkey, 21\u201327 May. 2012; p. 2214\u201318."},{"key":"1395_CR72","doi-asserted-by":"crossref","unstructured":"Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;p. 159\u201374.","DOI":"10.2307\/2529310"},{"key":"1395_CR73","unstructured":"Holzinger A, Biemann C, Pattichis CS, Kell DB. What do we need to build explainable AI systems for the medical domain? Preprint at arXiv. 2017;Available from: arXiv:abs\/1712.09923."},{"key":"1395_CR74","unstructured":"Cohen KB, Roeder C, Baumgartner\u00a0Jr WA, Hunter LE, Verspoor K. Test suite design for ontology concept recognition systems. In: Proceedings of LREC. Valletta, Malta; 2010. p. 441\u20136."},{"issue":"4","key":"1395_CR75","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","volume":"36","author":"J Lee","year":"2020","unstructured":"Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020;36(4):1234\u201340.","journal-title":"Bioinformatics"},{"key":"1395_CR76","doi-asserted-by":"crossref","unstructured":"Weber L, S\u00e4nger M, M\u00fcnchmeyer J, Habibi M, Leser U. HunFlair: an easy-to-use tool for state-of-the-art biomedical named entity recognition. Preprint at arXiv. 2020; arXiv:abs\/2008.07347.","DOI":"10.1093\/bioinformatics\/btab042"},{"key":"1395_CR77","doi-asserted-by":"crossref","unstructured":"Peters M, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, et\u00a0al. Deep contextualized word representations. In: Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics, vol 1 New Orleans, LA, 1-6 June. 2018;p. 2227\u201337.","DOI":"10.18653\/v1\/N18-1202"},{"key":"1395_CR78","doi-asserted-by":"crossref","unstructured":"Akbik A, Bergmann T, Vollgraf R. Pooled contextualized embeddings for named entity recognition. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics, Vol 1 Minneapolis, MN, USA, 2\u20137 June. 2019;p. 724\u20138.","DOI":"10.18653\/v1\/N19-1078"},{"key":"1395_CR79","doi-asserted-by":"crossref","unstructured":"Akhtyamova L, Mart\u00ednez P, Verspoor K, Cardiff J. testing contextualized word embeddings to improve NER in Spanish clinical case narratives. IEEE Access. 2020;p. 1\u201311.","DOI":"10.21203\/rs.2.22697\/v1"},{"key":"1395_CR80","unstructured":"Abacha AB, Zweigenbaum P. Medical entity recognition: a comparaison of semantic and statistical methods. In: Proceedings of BioNLP 2011 workshop. 2011;p. 56\u201364."},{"key":"1395_CR81","first-page":"143","volume":"2","author":"WF Styler IV","year":"2014","unstructured":"Styler WF IV, Bethard S, Finan S, Palmer M, Pradhan S, De Groen PC, et al. Temporal annotation in the clinical domain. T Assoc Comp Ling. 2014;2:143\u201354.","journal-title":"T Assoc Comp Ling."},{"key":"1395_CR82","unstructured":"N\u00e9v\u00e9ol A, Yepes AJ, Neves L, Verspoor K. Parallel corpora for the biomedical domain. In: Proceedings of LREC. Miyazaki, Japan; 2018. ."}],"updated-by":[{"DOI":"10.1186\/s12911-021-01475-0","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2021,4,7]],"date-time":"2021-04-07T00:00:00Z","timestamp":1617753600000}}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-021-01395-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-021-01395-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-021-01395-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,21]],"date-time":"2023-10-21T18:17:49Z","timestamp":1697912269000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-021-01395-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,22]]},"references-count":82,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["1395"],"URL":"https:\/\/doi.org\/10.1186\/s12911-021-01395-z","relation":{"correction":[{"id-type":"doi","id":"10.1186\/s12911-021-01475-0","asserted-by":"object"}]},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,22]]},"assertion":[{"value":"29 September 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 January 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 February 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 April 2021","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Correction","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"A Correction to this paper has been published:","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"https:\/\/doi.org\/10.1186\/s12911-021-01475-0","URL":"https:\/\/doi.org\/10.1186\/s12911-021-01475-0","order":7,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Not applicable.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"69"}}