{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T07:42:24Z","timestamp":1790667744645,"version":"4.1.0"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2013,1,8]],"date-time":"2013-01-08T00:00:00Z","timestamp":1357603200000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/2.0"},{"start":{"date-parts":[[2013,1,8]],"date-time":"2013-01-08T00:00:00Z","timestamp":1357603200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/2.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Biomed Semant"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>The availability of annotated corpora has facilitated the application of machine learning algorithms to concept extraction from clinical notes. However, high expenditure and labor are required for creating the annotations. A potential alternative is to reuse existing corpora from other institutions by pooling with local corpora, for training machine taggers. In this paper we have investigated the latter approach by pooling corpora from 2010 i2b2\/VA NLP challenge and Mayo Clinic Rochester, to evaluate taggers for recognition of medical problems. The corpora were annotated for medical problems, but with different guidelines. The taggers were constructed using an existing tagging system MedTagger that consisted of dictionary lookup, part of speech (POS) tagging and machine learning for named entity prediction and concept extraction. We hope that our current work will be a useful case study for facilitating reuse of annotated corpora across institutions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We found that pooling was effective when the size of the local corpus was small and after some of the guideline differences were reconciled. The benefits of pooling, however, diminished as more locally annotated documents were included in the training data. We examined the annotation guidelines to identify factors that determine the effect of pooling.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>The effectiveness of pooling corpora, is dependent on several factors, which include compatibility of annotation guidelines, distribution of report types and size of local and foreign corpora. Simple methods to rectify some of the guideline differences can facilitate pooling. Our findings need to be confirmed with further studies on different corpora. To facilitate the pooling and reuse of annotated corpora, we suggest that \u2013 i) the NLP community should develop a standard annotation guideline that addresses the potential areas of guideline differences that are partly identified in this paper; ii) corpora should be annotated with a two-pass method that focuses first on concept recognition, followed by normalization to existing ontologies; and iii) metadata such as type of the report should be created during the annotation process.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/2041-1480-4-3","type":"journal-article","created":{"date-parts":[[2013,1,8]],"date-time":"2013-01-08T11:20:47Z","timestamp":1357644047000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Pooling annotated corpora for clinical concept extraction"],"prefix":"10.1186","volume":"4","author":[{"given":"Kavishwar B","family":"Wagholikar","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Manabu","family":"Torii","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Siddhartha R","family":"Jonnalagadda","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hongfang","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2013,1,8]]},"reference":[{"issue":"5","key":"110_CR1","doi-asserted-by":"publisher","first-page":"760","DOI":"10.1016\/j.jbi.2009.08.007","volume":"42","author":"D Demner-Fushman","year":"2009","unstructured":"Demner-Fushman D, Chapman WW, McDonald CJ: What can natural language processing do for clinical decision support?. J Biomed Inform. 2009, 42 (5): 760-772. 10.1016\/j.jbi.2009.08.007.","journal-title":"J Biomed Inform"},{"key":"110_CR2","doi-asserted-by":"crossref","unstructured":"Meystre SM, Savova GK, Kipper-Schuler KC, Hurdle JF: Extracting information from textual documents in the electronic health record: a review of recent research. Yearb Med Inform. 2008, 128-144. 1","DOI":"10.1055\/s-0038-1638592"},{"issue":"5","key":"110_CR3","doi-asserted-by":"publisher","first-page":"550","DOI":"10.1197\/jamia.M2444","volume":"14","author":"O Uzuner","year":"2007","unstructured":"Uzuner O, Luo Y, Szolovits P: Evaluating the state-of-the-art in automatic de-identification. J Am Med Inform Assoc. 2007, 14 (5): 550-563. 10.1197\/jamia.M2444.","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"110_CR4","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1197\/jamia.M2408","volume":"15","author":"O Uzuner","year":"2008","unstructured":"Uzuner O, Goldstein I, Luo Y, Kohane I: Identifying patient smoking status from medical discharge records. J Am Med Inform Assoc. 2008, 15 (1): 14-24.","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"110_CR5","doi-asserted-by":"publisher","first-page":"514","DOI":"10.1136\/jamia.2010.003947","volume":"17","author":"O Uzuner","year":"2010","unstructured":"Uzuner O, Solti I, Cadag E: Extracting medication information from clinical text. J Am Med Inform Assoc. 2010, 17 (5): 514-518. 10.1136\/jamia.2010.003947.","journal-title":"J Am Med Inform Assoc"},{"key":"110_CR6","first-page":"42","volume-title":"Proceedings of the workshop on biomedical information extraction","author":"Y Wang","year":"2009","unstructured":"Wang Y, Patrick J: Cascading classifiers for named entity recognition in clinical notes. Proceedings of the workshop on biomedical information extraction. 2009, Borovets, Bulgaria. 1859783: Association for Computational Linguistics, 42-49."},{"key":"110_CR7","doi-asserted-by":"publisher","first-page":"94","DOI":"10.3115\/1572306.1572326","volume-title":"Proceedings of the workshop on current trends in biomedical natural language processing","author":"D Li","year":"2008","unstructured":"Li D, Kipper-Schuler K, Savova G: Conditional random fields and support vector machines for disorder named entity recognition in clinical texts. Proceedings of the workshop on current trends in biomedical natural language processing. 2008, Columbus, Ohio. 1572326: Association for Computational Linguistics, 94-95."},{"key":"110_CR8","volume-title":"An effective approach to biomedical information extraction with limited training data (PhD Dissertation, Arizona State University)","author":"S Jonnalagadda","year":"2011","unstructured":"Jonnalagadda S: An effective approach to biomedical information extraction with limited training data (PhD Dissertation, Arizona State University). 2011, Phoenix, Arizona: PhD Phoenix"},{"issue":"2","key":"110_CR9","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1197\/jamia.M2844","volume":"16","author":"M Torii","year":"2009","unstructured":"Torii M, Hu Z, Wu CH, Liu H: BioTagger-GM: a gene\/protein name recognition system. J Am Med Inform Assoc. 2009, 16 (2): 247-255. 10.1197\/jamia.M2844.","journal-title":"J Am Med Inform Assoc"},{"key":"110_CR10","first-page":"142","volume-title":"Seventh conference on natural language learning","author":"EF Tjong Kim Sang","year":"2003","unstructured":"Tjong Kim Sang EF, De Meulder F: Introduction to the CoNLL-2003 shared task. Seventh conference on natural language learning. 2003, Edmonton, Canada, 142-147."},{"key":"110_CR11","first-page":"7","volume-title":"Proceedings of the second BioCreative challenge workshop","author":"J Wilbur","year":"2007","unstructured":"Wilbur J, Smith L, Tanabe T: BioCreative 2 gene mention task. Proceedings of the second BioCreative challenge workshop. 2007, Madrid, Spain: Proceedings of the second biocreative challenge evaluation workshop Vol: 23, 7-16."},{"issue":"Suppl 8","key":"110_CR12","doi-asserted-by":"publisher","first-page":"S4","DOI":"10.1186\/1471-2105-12-S8-S4","volume":"12","author":"CN Arighi","year":"2011","unstructured":"Arighi CN, Roberts PM, Agarwal S, Bhattacharya S, Cesareni G, Chatr-Aryamontri A, Clematide S, Gaudet P, Giglio MG, Harrow I, et al: BioCreative III interactive task: an overview. BMC Bioinformatics. 2011, 12 (Suppl 8): S4-10.1186\/1471-2105-12-S8-S4.","journal-title":"BMC Bioinformatics"},{"issue":"Suppl 11","key":"110_CR13","doi-asserted-by":"publisher","first-page":"S1","DOI":"10.1186\/1471-2105-13-S11-S1","volume":"13","author":"JD Kim","year":"2012","unstructured":"Kim JD, Nguyen N, Wang Y, Tsujii J, Takagi T, Yonezawa A: The genia event and protein coreference tasks of the BioNLP shared task 2011. BMC Bioinformatics. 2012, 13 (Suppl 11): S1-","journal-title":"BMC Bioinformatics"},{"issue":"3","key":"110_CR14","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1136\/jamia.2009.002733","volume":"17","author":"AR Aronson","year":"2010","unstructured":"Aronson AR, Lang FM: An overview of MetaMap: historical perspective and recent advances. J Am Med Inform Assoc. 2010, 17 (3): 229-236.","journal-title":"J Am Med Inform Assoc"},{"issue":"Pt 1","key":"110_CR15","first-page":"472","volume":"107","author":"S Bakken","year":"2004","unstructured":"Bakken S, Hyun S, Friedman C, Johnson S: A comparison of semantic categories of the ISO reference terminology models for nursing and the MedLEE natural language processing system. Stud Health Technol Inform. 2004, 107 (Pt 1): 472-476.","journal-title":"Stud Health Technol Inform"},{"key":"110_CR16","first-page":"247","volume-title":"Proc Annu Symp Comput Appl Med Care","author":"P Haug","year":"1994","unstructured":"Haug P, Koehler S, Lau LM, Wang P, Rocha R, Huff S: A natural language understanding system combining syntactic and semantic techniques. Proc Annu Symp Comput Appl Med Care. 1994, 247-251."},{"issue":"5","key":"110_CR17","doi-asserted-by":"publisher","first-page":"580","DOI":"10.1136\/amiajnl-2011-000155","volume":"18","author":"M Torii","year":"2011","unstructured":"Torii M, Wagholikar K, Liu H: Using machine learning for concept extraction on clinical documents from multiple data sources. J Am Med Inform Assoc. 2011, 18 (5): 580-587. 10.1136\/amiajnl-2011-000155.","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"110_CR18","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1186\/1747-5333-2-4","volume":"2","author":"HL Johnson","year":"2007","unstructured":"Johnson HL, Baumgartner WA, Krallinger M, Cohen KB, Hunter L: Corpus refactoring: a feasibility study. Journal of Biomedical Discovery and Collaboration. 2007, 2 (1): 4-10.1186\/1747-5333-2-4.","journal-title":"Journal of Biomedical Discovery and Collaboration"},{"key":"110_CR19","first-page":"106","volume-title":"Workshop on current trends in biomedical natural language processing","author":"T Ohta","year":"2009","unstructured":"Ohta T, Kim J-D, Pyysalo S, Wang Y, Tsujii J: Incorporating GENETAG-style annotation to GENIA corpus. Workshop on current trends in biomedical natural language processing. 2009, Stroudsburg, PA, USA: Association for Computational Linguistics, 106-107."},{"issue":"1","key":"110_CR20","doi-asserted-by":"publisher","first-page":"403","DOI":"10.1186\/1471-2105-10-403","volume":"10","author":"Y Wang","year":"2009","unstructured":"Wang Y, Kim J-D, S\u00e6tre R, Pyysalo S, Tsujii J: Investigating heterogeneous protein annotations toward cross-corpora utilization. BMC Bioinformatics. 2009, 10 (1): 403-10.1186\/1471-2105-10-403.","journal-title":"BMC Bioinformatics"},{"key":"110_CR21","unstructured":"The AIMed corpus.ftp:\/\/ftp.cs.utexas.edu\/pub\/mooney\/bio-data\/interactions.tar.gz,"},{"issue":"05","key":"110_CR22","doi-asserted-by":"publisher","first-page":"901","DOI":"10.1142\/S0219720010004999","volume":"08","author":"Y Wang","year":"2010","unstructured":"Wang Y, S\u00e6tre R, Kim J-D, Pyysalo S, Ohta T, Tsujii JI: Improving the inter-corpora compatibility for protein annotations. J Bioinform Comput Biol. 2010, 08 (05): 901-10.1142\/S0219720010004999.","journal-title":"J Bioinform Comput Biol"},{"key":"110_CR23","first-page":"382","volume":"2011","author":"JW Fan","year":"2011","unstructured":"Fan JW, Prasad R, Yabut RM, Loomis RM, Zisook DS, Mattison JE, Huang Y: Part-of-speech tagging for clinical text: wall or bridge between institutions?. AMIA Annu Symp Proc. 2011, 2011: 382-391.","journal-title":"AMIA Annu Symp Proc"},{"issue":"5","key":"110_CR24","doi-asserted-by":"publisher","first-page":"507","DOI":"10.1136\/jamia.2009.001560","volume":"17","author":"GK Savova","year":"2010","unstructured":"Savova GK, Masanz JJ, Ogren PV, Zheng J, Sohn S, Kipper-Schuler KC, Chute CG: Mayo clinical text analysis and knowledge extraction system (cTAKES): architecture, component evaluation and applications. Journal of American Medical Informatics Assocociation. 2010, 17 (5): 507-513. 10.1136\/jamia.2009.001560.","journal-title":"Journal of American Medical Informatics Assocociation"},{"issue":"5","key":"110_CR25","doi-asserted-by":"publisher","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","volume":"18","author":"O Uzuner","year":"2011","unstructured":"Uzuner O, South BR, Shen S, Duvall SL: 2010i2b2\/VA challenge on concepts, assertions, and relations in clinical text. J Am Med Inform Assoc. 2011, 18 (5): 552-556. 10.1136\/amiajnl-2011-000203.","journal-title":"J Am Med Inform Assoc"},{"key":"110_CR26","first-page":"63","volume-title":"AMIA summit on clinical research informatics","author":"K Wagholikar","year":"2012","unstructured":"Wagholikar K, Torii M, Jonnalagadda S, Liu H: Feasibility of pooling annotated corpora for clinical concept extraction. AMIA summit on clinical research informatics. 2012, San Francisco, CA: American Medical Informatics Summits on Translational Science Proceedings, 63-70."},{"key":"110_CR27","unstructured":"Annotation schema for marking spans of clinical conditions in clinical text.http:\/\/orbit.nlm.nih.gov\/resource\/annotation-schema-marking-spans-clinical-conditions-clinical-text,"},{"key":"110_CR28","first-page":"3143","volume-title":"LREC\u201908","author":"PVS Ogren","year":"2008","unstructured":"Ogren PVS, Guergana K, Chute Y, Christopher G: Constructing evaluation corpora for automated clinical named entity recognition. LREC\u201908. 2008, Marrakech, Morocco: Proceedings of the Sixth International Conference on Language Resources and Evaluation LREC'08, 3143-3150."},{"issue":"Database issue","key":"110_CR29","doi-asserted-by":"publisher","first-page":"D267","DOI":"10.1093\/nar\/gkh061","volume":"32","author":"O Bodenreider","year":"2004","unstructured":"Bodenreider O: The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2004, 32 (Database issue): D267-270.","journal-title":"Nucleic Acids Res"},{"issue":"3","key":"110_CR30","doi-asserted-by":"publisher","first-page":"189","DOI":"10.1016\/j.jbi.2003.08.005","volume":"36","author":"C Friedman","year":"2003","unstructured":"Friedman C, Liu H, Shagina L: A vocabulary development and visualization tool based on natural language processing and the mining of textual patient reports. J Biomed Inform. 2003, 36 (3): 189-201. 10.1016\/j.jbi.2003.08.005.","journal-title":"J Biomed Inform"},{"key":"110_CR31","first-page":"382","volume-title":"Advances in informatics - 10th panhellenic conference on informatics","author":"YT Yoshimasa Tsuruoka","year":"2005","unstructured":"Yoshimasa Tsuruoka YT, Jin-Dong K, Tomoko O, John MN, Sophia A, Junichi T: Developing a robust part-of-speech tagger for biomedical text. Advances in informatics - 10th panhellenic conference on informatics. 2005, Heidelberg, Berlin: Springer Berlin, 382-392."},{"key":"110_CR32","volume-title":"MALLET: a machine learning for language toolkit","author":"AK McCallum","year":"2002","unstructured":"McCallum AK: MALLET: a machine learning for language toolkit. 2002,http:\/\/mallet.cs.umass.edu,"}],"container-title":["Journal of Biomedical Semantics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/2041-1480-4-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/2041-1480-4-3","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/2041-1480-4-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T00:21:56Z","timestamp":1790036516000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1186\/2041-1480-4-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,1,8]]},"references-count":32,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2013,12]]}},"alternative-id":["110"],"URL":"https:\/\/doi.org\/10.1186\/2041-1480-4-3","relation":{},"ISSN":["2041-1480"],"issn-type":[{"value":"2041-1480","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,1,8]]},"assertion":[{"value":"26 June 2012","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 January 2013","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 January 2013","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"3"}}