{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T20:10:31Z","timestamp":1784232631262,"version":"3.55.0"},"reference-count":55,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,4,6]],"date-time":"2020-04-06T00:00:00Z","timestamp":1586131200000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2020,4,6]],"date-time":"2020-04-06T00:00:00Z","timestamp":1586131200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100010909","name":"Young Scientists Fund","doi-asserted-by":"publisher","award":["61801058"],"award-info":[{"award-number":["61801058"]}],"id":[{"id":"10.13039\/501100010909","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100010909","name":"Young Scientists Fund","doi-asserted-by":"publisher","award":["61501063"],"award-info":[{"award-number":["61501063"]}],"id":[{"id":"10.13039\/501100010909","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>In this study, we focus on building a fine-grained entity annotation corpus with the corresponding annotation guideline of traditional Chinese medicine (TCM) clinical records. Our aim is to provide a basis for the fine-grained corpus construction of TCM clinical records in future.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>We developed a four-step approach that is suitable for the construction of TCM medical records in our corpus. First, we determined the entity types included in this study through sample annotation. Then, we drafted a fine-grained annotation guideline by summarizing the characteristics of the dataset and referring to some existing guidelines. We iteratively updated the guidelines until the inter-annotator agreement (IAA) exceeded a Cohen\u2019s kappa value of 0.9. Comprehensive annotations were performed while keeping the IAA value above 0.9.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>We annotated the 10,197 clinical records in five rounds. Four entity categories involving 13 entity types were employed. The final fine-grained annotated entity corpus consists of 1104 entities and 67,799 tokens. The final IAAs are 0.936 on average (for three annotators), indicating that the fine-grained entity recognition corpus is of high quality.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>These results will provide a foundation for future research on corpus construction and named entity recognition tasks in the TCM clinical domain.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12911-020-1079-2","type":"journal-article","created":{"date-parts":[[2020,4,6]],"date-time":"2020-04-06T09:04:11Z","timestamp":1586163851000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Constructing fine-grained entity recognition corpora based on clinical records of traditional Chinese medicine"],"prefix":"10.1186","volume":"20","author":[{"given":"Tingting","family":"Zhang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yaqiang","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaofeng","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yafei","family":"Yang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9042-8008","authenticated-orcid":false,"given":"Ying","family":"Ye","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,4,6]]},"reference":[{"key":"1079_CR1","doi-asserted-by":"publisher","first-page":"126","DOI":"10.1038\/448126a","volume":"448","author":"J Qiu","year":"2007","unstructured":"Qiu J. Traditional medicine: a culture in the balance. Nature. 2007;448:126.","journal-title":"Nature."},{"key":"1079_CR2","unstructured":"Ministry of Health. Basic Specification for Eelectronic Medical Records (Trial). China's Health Qual Manage. 2010;17:22\u20133."},{"key":"1079_CR3","volume-title":"On construction of Chinese medicine ontology Concept's description architecture","author":"L Yao","year":"2008","unstructured":"Yao L, Chen X, Yang Z, Wang H, Wang Z. On construction of Chinese medicine ontology Concept's description architecture; 2008."},{"key":"1079_CR4","doi-asserted-by":"publisher","first-page":"544","DOI":"10.1136\/amiajnl-2011-000464","volume":"18","author":"P Nadkarni","year":"2011","unstructured":"Nadkarni P, Ohno-Machado L, Chapman W. Natural language processing: an introduction. J Am Med Inform Assn. 2011;18:544\u201351.","journal-title":"J Am Med Inform Assn."},{"key":"1079_CR5","doi-asserted-by":"publisher","first-page":"808","DOI":"10.1136\/amiajnl-2013-002381","volume":"21","author":"J Lei","year":"2014","unstructured":"Lei J, Tang B, Lu X, Gao K, Jiang M, Xu H. A comprehensive study of named entity recognition in Chinese clinical text. J Am Med Inform Assoc. 2014;21:808\u201314.","journal-title":"J Am Med Inform Assoc"},{"key":"1079_CR6","first-page":"267","volume":"32","author":"B Olivier","year":"2004","unstructured":"Olivier B. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2004;32:267\u201370.","journal-title":"Nucleic Acids Res"},{"key":"1079_CR7","first-page":"279","volume":"121","author":"K Donnelly","year":"2006","unstructured":"Donnelly K. SNOMED-CT: the advanced terminology and coding system for eHealth. Stud Health Technol Inform. 2006;121:279.","journal-title":"Stud Health Technol Inform"},{"key":"1079_CR8","doi-asserted-by":"publisher","first-page":"1091","DOI":"10.1093\/nar\/gkt1068","volume":"42","author":"L Vivian","year":"2014","unstructured":"Vivian L, Craig K, Yannick D, Tim J, Chi G, Yifeng L, Adam M, David A, Michael W, Vanessa N. DrugBank 4.0: shedding new light on drug metabolism. Nucleic Acids Res. 2014;42:1091\u20137.","journal-title":"Nucleic Acids Res"},{"key":"1079_CR9","doi-asserted-by":"publisher","first-page":"e84","DOI":"10.1136\/amiajnl-2013-001806","volume":"21","author":"Y Xu","year":"2014","unstructured":"Xu Y, Wang Y, Liu T, Liu J, Fan Y, Qian Y, Tsujii J, Chang E. Joint segmentation and named entity recognition using dual decomposition in Chinese discharge summaries. J Am Med Inform Assn. 2014;21:e84\u201392.","journal-title":"J Am Med Inform Assn."},{"key":"1079_CR10","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1016\/j.jbi.2017.04.006","volume":"69","author":"B He","year":"2017","unstructured":"He B, Dong B, Guan Y, Yang J, Jiang Z, Yu Q, Cheng J, Qu C. Building a comprehensive syntactic and semantic corpus of Chinese clinical texts. J Biomed Inform. 2017;69:203\u201317.","journal-title":"J Biomed Inform"},{"key":"1079_CR11","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1186\/s12911-019-0759-2","volume":"19","author":"Y Gao","year":"2019","unstructured":"Gao Y, Gu L, Wang Y, Wang Y, Yang F. Constructing a Chinese electronic medical record corpus for named entity recognition on resident admit notes. BMC Med Inform Decis Mak. 2019;19:56.","journal-title":"BMC Med Inform Decis Mak."},{"key":"1079_CR12","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1186\/s12911-019-0762-7","volume":"19","author":"X Cai","year":"2019","unstructured":"Cai X, Dong S, Hu J. A deep learning model incorporating part of speech and self-matching attention for named entity recognition of Chinese electronic medical records. BMC Med Inform Decis Mak. 2019;19:65.","journal-title":"BMC Med Inform Decis Mak"},{"key":"1079_CR13","first-page":"818","volume":"2016","author":"Z Liu","year":"2016","unstructured":"Liu Z, Tang B, Wang X, Chen Q, Li H, Bu J, Jiang J, Deng Q, Zhu S. CMedTEX: a rule-based temporal expression extraction and normalization system for Chinese clinical notes. AMIA Annu Symp Proc. 2016;2016:818\u201326.","journal-title":"AMIA Annu Symp Proc"},{"key":"1079_CR14","doi-asserted-by":"crossref","unstructured":"Chapman WW, Nadkarni PM, Hirschman L, D\u2019Avolio LW, Savova GK, Uzuner O. Overcoming barriers to NLP for clinical text: the role of shared tasks and the need for additional creative solutions. Am Med Inform Assoc. 2011;18:540\u20133.","DOI":"10.1136\/amiajnl-2011-000465"},{"key":"1079_CR15","first-page":"1537","volume":"40","author":"J Yang","year":"2014","unstructured":"Yang J, Yu Q, Guan Y, Jiang Z. An overview of research on electronic medical record oriented named entity recognition and entity relation extraction. Acta Automat Sin. 2014;40:1537\u201362.","journal-title":"Acta Automat Sin"},{"key":"1079_CR16","doi-asserted-by":"publisher","first-page":"210","DOI":"10.1016\/j.jbi.2011.10.010","volume":"45","author":"Y Wang","year":"2012","unstructured":"Wang Y, Yu Z, Jiang Y, Liu Y, Li C, Liu Y. A framework and its empirical study of automatic diagnosis of traditional Chinese medicine utilizing raw free-text clinical records. J Biomed Inform. 2012;45:210\u201323.","journal-title":"J Biomed Inform"},{"key":"1079_CR17","doi-asserted-by":"publisher","first-page":"540","DOI":"10.1136\/amiajnl-2011-000465","volume":"18","author":"W Chapman","year":"2011","unstructured":"Chapman W, Nadkarni P, Hirschman L, D'Avolio L, Savova G, Uzuner O. Overcoming barriers to NLP for clinical text: the role of shared tasks and the need for additional creative solutions. J Am Med Inform Assn. 2011;18:540\u20133.","journal-title":"J Am Med Inform Assn"},{"key":"1079_CR18","doi-asserted-by":"publisher","first-page":"i180","DOI":"10.1093\/bioinformatics\/btg1023","volume":"19 Suppl 1","author":"J Kim","year":"2003","unstructured":"Kim J, Ohta T, Tateisi Y, Tsujii J. GENIA corpus--semantically annotated corpus for bio-textmining. Bioinformatics. 2003;19 Suppl 1:i180.","journal-title":"Bioinformatics"},{"key":"1079_CR19","unstructured":"Dog\u01e7an RI, Lu Z. An Improved Corpus of Disease Mentions in Pubmed Citations[C]. Stroudsbury: Proceedings of the 2012 Workshop on Biomedical Natural Language Processing. 2012;91\u20139."},{"key":"1079_CR20","doi-asserted-by":"publisher","first-page":"914","DOI":"10.1016\/j.jbi.2013.07.011","volume":"46","author":"M Herrero-Zazo","year":"2013","unstructured":"Herrero-Zazo M, Segura-Bedmar I, Mart\u00ednez P, Declerck T. The DDI corpus: an annotated corpus with pharmacological substances and drug\u2013drug interactions. J Biomed Inform. 2013;46:914\u201320.","journal-title":"J Biomed Inform"},{"key":"1079_CR21","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1016\/j.jbi.2013.09.008","volume":"47","author":"Y Wang","year":"2014","unstructured":"Wang Y, Yu Z, Chen L, Chen Y, Liu Y, Hu X, Jiang Y. Supervised methods for symptom name recognition in free-text clinical records of traditional Chinese medicine: an empirical study. J Biomed Inform. 2014;47:91\u2013104.","journal-title":"J Biomed Inform"},{"key":"1079_CR22","doi-asserted-by":"publisher","first-page":"130","DOI":"10.1016\/j.jbi.2013.12.017","volume":"48","author":"H Wang","year":"2014","unstructured":"Wang H, Zhang W, Zeng Q, Li Z, Feng K, Liu L. Extracting important information from Chinese operation notes with natural language processing methods. J Biomed Inform. 2014;48:130\u20136.","journal-title":"J Biomed Inform"},{"key":"1079_CR23","doi-asserted-by":"publisher","first-page":"347","DOI":"10.1007\/s11684-014-0361-z","volume":"8","author":"C Cao","year":"2014","unstructured":"Cao C, Sun M, Wang S. Extracting terms from clinical records of traditional Chinese medicine. Front Med Prc. 2014;8:347\u201351.","journal-title":"Front Med Prc"},{"key":"1079_CR24","first-page":"1","volume":"2015","author":"Y Li","year":"2015","unstructured":"Li Y, Zhou X, Zhang R, Wang Y, Peng Y, Hu J, Xie Q, Xue Y, Xu L, Liu X, Liu B. Detection of herb-symptom associations from traditional chinese medicine clinical data. Evid Based Complement Alternat Med. 2015;2015:1\u201311.","journal-title":"Evid Based Complement Alternat Med"},{"key":"1079_CR25","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0136270","volume":"10","author":"D Xu","year":"2015","unstructured":"Xu D, Zhang M, Zhao T, Ge C, Gao W, Wei J, Zhu K. Data-driven information extraction from Chinese electronic medical records. PLoS One. 2015;10:e0136270.","journal-title":"PLoS One"},{"key":"1079_CR26","doi-asserted-by":"publisher","first-page":"334","DOI":"10.1016\/j.jbi.2016.02.011","volume":"60","author":"S Zhang","year":"2016","unstructured":"Zhang S, Kang T, Zhang X, Wen D, Elhadad N, Lei J. Speculation detection for Chinese clinical notes: impacts of word segmentation and embedding models. J Biomed Inform. 2016;60:334\u201341.","journal-title":"J Biomed Inform"},{"key":"1079_CR27","doi-asserted-by":"publisher","first-page":"356","DOI":"10.1093\/jamia\/ocv092","volume":"23","author":"H Wan","year":"2016","unstructured":"Wan H, Moens M, Luyten W, Zhou X, Mei Q, Liu L, Tang J. Extracting relations from traditional Chinese medicine literature via heterogeneous entity networks. J Am Med Inform Assn. 2016;23:356\u201365.","journal-title":"J Am Med Inform Assn."},{"key":"1079_CR28","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1186\/s13326-017-0145-x","volume":"8","author":"T Ruan","year":"2017","unstructured":"Ruan T, Wang M, Sun J, Wang T, Zeng L, Yin Y, Gao J. An automatic approach for constructing a knowledge base of symptoms in Chinese. J Biomed Semant. 2017;8:33.","journal-title":"J Biomed Semant"},{"key":"1079_CR29","doi-asserted-by":"publisher","DOI":"10.2196\/medinform.9965","volume":"6","author":"Y Zhang","year":"2018","unstructured":"Zhang Y, Wang X. Clinical named entity recognition from Chinese electronic health records via machine learning methods. JMIR Med Inf. 2018;6:e50.","journal-title":"JMIR Med Inf."},{"key":"1079_CR30","first-page":"17","volume":"119","author":"S Miao","year":"2018","unstructured":"Miao S, Xu T, Wu Y, Xie H, Wang J, Jing S, Zhang Y, Zhang X, Yang Y, Zhang X, Shan T, Wang L, Xu H, Wang S, Liu Y. Extraction of BI-RADS findings from breast ultrasound reports in Chinese using deep learning approaches. JMIR Med Inf. 2018;119:17\u201321.","journal-title":"JMIR Med Inf"},{"key":"1079_CR31","first-page":"256","volume":"50","author":"X Bao","year":"2018","unstructured":"Bao X, Huang W, Zhang K, Jin M, Li Y, Niu C. A customized method for information extraction from unstructured text data in the electronic medical records. Beijing Da Xue Xue Bao. 2018;50:256\u201363.","journal-title":"Beijing Da Xue Xue Bao"},{"key":"1079_CR32","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1016\/j.jbi.2019.103133","volume":"92","author":"Q Wang","year":"2019","unstructured":"Wang Q, Zhou Y, Ruan T, Gao D, Xia Y, He P. Incorporating dictionaries into deep neural networks for the Chinese clinical named entity recognition. J Biomed Inform. 2019;92:103\u201333.","journal-title":"J Biomed Inform"},{"key":"1079_CR33","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1186\/s12911-019-0770-7","volume":"19","author":"Y Xiong","year":"2019","unstructured":"Xiong Y, Wang Z, Jiang D, Wang X, Chen Q, Xu H, Yan J, Tang B. A fine-grained Chinese word segmentation and part-of-speech tagging corpus for clinical text. BMC Med Inform Decis. 2019;19:66.","journal-title":"BMC Med Inform Decis"},{"key":"1079_CR34","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1186\/1472-6882-8-58","volume":"8","author":"Y Fang","year":"2008","unstructured":"Fang Y, Huang H, Chen H, Juan H. TCMGeneDIT: a database for associated traditional Chinese medicine, gene and disease information using text mining. BMC Complem Altern M. 2008;8:58.","journal-title":"BMC Complem Altern M"},{"key":"1079_CR35","first-page":"8","volume":"15","author":"F Wang","year":"2019","unstructured":"Wang F, Li J. Studying the medical record of traditional Chinese medicine is the best way for school inheritors to acquire the academic experience of famous teachers. J Pediatrics Tradit Chinese Med. 2019;15:8\u201311.","journal-title":"J Pediatrics Tradit Chinese Med"},{"key":"1079_CR36","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1749-8546-7-1","volume":"7","author":"Z Li","year":"2012","unstructured":"Li Z, He X, Wang S, Xu C. Strategic thought on clinical experience and academic thoughts of famous old Chinese medicine doctors. World Chin Med. 2012;7:1\u20134.","journal-title":"World Chin Med"},{"key":"1079_CR37","unstructured":"Meng Q. On the value, characteristics and research methods of the traditional Chinese medicine. J Tradit Chin Med. 2006;21:568\u201370."},{"key":"1079_CR38","volume-title":"Traditional Chinese medical record science","author":"X Zhang","year":"1995","unstructured":"Zhang X. Traditional Chinese medical record science. Bei Jing: China Press of Traditional Chinese Medicine; 1995."},{"key":"1079_CR39","unstructured":"Gao Z, Dong J. From four TCM diagnostic methods used in combination to precision TCM syndrome-based treatment. China J Tradit Chinese Med Pharm. 2019;34:13\u20137."},{"key":"1079_CR40","doi-asserted-by":"publisher","first-page":"855","DOI":"10.1016\/j.cell.2011.08.024","volume":"146","author":"L Miller","year":"2011","unstructured":"Miller L, Su X. Artemisinin: discovery from the Chinese herbal garden. Cell. 2011;146:855\u20138.","journal-title":"Cell."},{"key":"1079_CR41","doi-asserted-by":"publisher","first-page":"194","DOI":"10.4103\/2225-4110.114901","volume":"3","author":"L Lo","year":"2013","unstructured":"Lo L, Cheng T, Chiang J, Damdinsuren N. Breast cancer index: a perspective on tongue diagnosis in traditional chinese medicine. J Tradit Complement Med. 2013;3:194\u2013203.","journal-title":"J Tradit Complement Med"},{"key":"1079_CR42","doi-asserted-by":"publisher","first-page":"705","DOI":"10.1016\/j.ctim.2015.07.001","volume":"23","author":"L Lo","year":"2015","unstructured":"Lo L, Cheng T, Chen Y, Natsagdorj S, Chiang J. TCM tongue diagnosis index of early-stage breast cancer. Complement Ther Med. 2015;23:705\u201313.","journal-title":"Complement Ther Med"},{"key":"1079_CR43","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1155\/2015\/564326","volume":"2015","author":"B Picciani","year":"2015","unstructured":"Picciani B, Souza T, Santos Vde C, Domingos T, Carneiro S, Avelleira J, Azulay D, Pinto J, Dias E. Geographic tongue and fissured tongue in 348 patients with psoriasis: correlation with disease severity. Sci World J. 2015;2015:1\u20137.","journal-title":"Sci World J"},{"key":"1079_CR44","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1155\/2015\/859192","volume":"2015","author":"W Nanyue","year":"2015","unstructured":"Nanyue W, Youhua Y, Dawei H, Bin X, Jia L, Tongda L, Liyuan X, Zengyu S, Yanping C, Jia W. Pulse diagnosis signals analysis of fatty liver disease and cirrhosis patients by using machine learning. Sci World J. 2015;2015:1\u20139.","journal-title":"Sci World J"},{"key":"1079_CR45","first-page":"1193","volume":"24","author":"Y Wang","year":"2015","unstructured":"Wang Y, Fan X. Study on the relationship between left string-like pulse and liver disease. J Emerg Tradit Chinese Med. 2015;24:1193\u20134.","journal-title":"J Emerg Tradit Chinese Med"},{"key":"1079_CR46","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1136\/hrt.13.4.423","volume":"13","author":"D Bedford","year":"1951","unstructured":"Bedford D. The ancient art of feeling the pulse. Br Heart J. 1951;13:423\u201337.","journal-title":"Br Heart J"},{"key":"1079_CR47","volume-title":"Diagnostics of traditional Chinese medicine China Press of Traditional Chinese Medicine","author":"W Zhu","year":"2007","unstructured":"Zhu W. Diagnostics of traditional Chinese medicine China Press of Traditional Chinese Medicine; 2007."},{"key":"1079_CR48","unstructured":"WHO International Standard terminologies on traditional medicine in the Western Pacific region; 2010. https:\/\/iris.wpro.who.int\/bitstream\/handle\/10665.1\/5395\/9789290612487_eng.pdf. Accessed 1 Sept 2019."},{"issue":"Suppl","key":"1079_CR49","doi-asserted-by":"publisher","first-page":"S111","DOI":"10.1016\/j.jbi.2015.06.010","volume":"58","author":"K Roberts","year":"2015","unstructured":"Roberts K, Shooshan S, Rodriguez L, Abhyankar S, Kilicoglu H, Demner-Fushman D. The role of fine-grained annotations in supervised recognition of risk factors for heart disease from EHRs. J Biomed Inform. 2015;58(Suppl):S111\u20139.","journal-title":"J Biomed Inform"},{"key":"1079_CR50","doi-asserted-by":"publisher","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","volume":"18","author":"O Uzuner","year":"2011","unstructured":"Uzuner O, South B, Shen S, DuVall S. 2010 i2b2\/VA challenge on concepts, assertions, and relations in clinical text. J Am Med Inform Assn. 2011;18:552\u20136.","journal-title":"J Am Med Inform Assn"},{"key":"1079_CR51","doi-asserted-by":"publisher","first-page":"1168","DOI":"10.1136\/amiajnl-2013-001810","volume":"20","author":"J Fan","year":"2013","unstructured":"Fan J, Yang E, Jiang M, Prasad R, Loomis R, Zisook D, Denny J, Xu H, Huang Y. Syntactic parsing of clinical text: guideline and corpus development with handling ill-formed sentences. J Am Med Inform Assn. 2013;20:1168\u201377.","journal-title":"J Am Med Inform Assn."},{"key":"1079_CR52","first-page":"1366","volume":"2012","author":"P Raghavan","year":"2012","unstructured":"Raghavan P, Fosler-Lussier E, Lai A. Inter-annotator reliability of medical events, coreferences and temporal relations in clinical narratives by annotators with varying levels of clinical expertise. AMIA Annu Symp Proc. 2012;2012:1366\u201374.","journal-title":"AMIA Annu Symp Proc."},{"key":"1079_CR53","first-page":"298","volume":"245","author":"M Boguslav","year":"2017","unstructured":"Boguslav M, Cohen K. Inter-annotator agreement and the upper limit on machine performance: evidence from biomedical natural language processing. Stud Health Technol Inform. 2017;245:298\u2013302.","journal-title":"Stud Health Technol Inform"},{"key":"1079_CR54","first-page":"62","volume":"27","author":"W Tang","year":"2015","unstructured":"Tang W, Hu J, Zhang H, Wu P, He H. Kappa coefficient: a popular measure of rater agreement. Shanghai Arch Psychiatry. 2015;27:62\u20137.","journal-title":"Shanghai Arch Psychiatry"},{"key":"1079_CR55","first-page":"249","volume":"22","author":"J Carletta","year":"1996","unstructured":"Carletta J. Assessing agreement on classification tasks: the kappa statistic. Computational Linguistics. 1996;22:249\u201354.","journal-title":"Computational Linguistics"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-1079-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s12911-020-1079-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-1079-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,4,5]],"date-time":"2021-04-05T23:22:26Z","timestamp":1617664946000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-020-1079-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,6]]},"references-count":55,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["1079"],"URL":"https:\/\/doi.org\/10.1186\/s12911-020-1079-2","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,4,6]]},"assertion":[{"value":"8 January 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 March 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 April 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Not applicable.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"64"}}