{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T19:28:55Z","timestamp":1772134135829,"version":"3.50.1"},"reference-count":38,"publisher":"Oxford University Press (OUP)","issue":"7","license":[{"start":{"date-parts":[[2024,5,14]],"date-time":"2024-05-14T00:00:00Z","timestamp":1715644800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000049","name":"National Institute on Aging","doi-asserted-by":"publisher","award":["R33 AG58738"],"award-info":[{"award-number":["R33 AG58738"]}],"id":[{"id":"10.13039\/100000049","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000049","name":"National Institute on Aging","doi-asserted-by":"publisher","award":["R01 AG34676"],"award-info":[{"award-number":["R01 AG34676"]}],"id":[{"id":"10.13039\/100000049","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000049","name":"National Institute on Aging","doi-asserted-by":"publisher","award":["R01 AG068007"],"award-info":[{"award-number":["R01 AG068007"]}],"id":[{"id":"10.13039\/100000049","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Mayo Clinic Eric"},{"name":"Wendy Schmidt fund for AI Research and Innovation"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,6,20]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Background<\/jats:title>\n                  <jats:p>Error analysis plays a crucial role in clinical concept extraction, a fundamental subtask within clinical natural language processing (NLP). The process typically involves a manual review of error types, such as contextual and linguistic factors contributing to their occurrence, and the identification of underlying causes to refine the NLP model and improve its performance. Conducting error analysis can be complex, requiring a combination of NLP expertise and domain-specific knowledge. Due to the high heterogeneity of electronic health record (EHR) settings across different institutions, challenges may arise when attempting to standardize and reproduce the error analysis process.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>This study aims to facilitate a collaborative effort to establish common definitions and taxonomies for capturing diverse error types, fostering community consensus on error analysis for clinical concept extraction tasks.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We iteratively developed and evaluated an error taxonomy based on existing literature, standards, real-world data, multisite case evaluations, and community feedback. The finalized taxonomy was released in both .dtd and .owl formats at the Open Health Natural Language Processing Consortium. The taxonomy is compatible with several different open-source annotation tools, including MAE, Brat, and MedTator.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The resulting error taxonomy comprises 43 distinct error classes, organized into 6 error dimensions and 4 properties, including model type (symbolic and statistical machine learning), evaluation subject (model and human), evaluation level (patient, document, sentence, and concept), and annotation examples. Internal and external evaluations revealed strong variations in error types across methodological approaches, tasks, and EHR settings. Key points emerged from community feedback, including the need to enhancing clarity, generalizability, and usability of the taxonomy, along with dissemination strategies.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>The proposed taxonomy can facilitate the acceleration and standardization of the error analysis process in multi-site settings, thus improving the provenance, interpretability, and portability of NLP models. Future researchers could explore the potential direction of developing automated or semi-automated methods to assist in the classification and standardization of error analysis.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae101","type":"journal-article","created":{"date-parts":[[2024,5,14]],"date-time":"2024-05-14T09:39:26Z","timestamp":1715679566000},"page":"1493-1502","source":"Crossref","is-referenced-by-count":7,"title":["A taxonomy for advancing systematic error analysis in multi-site electronic health record-based clinical concept extraction"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1691-5179","authenticated-orcid":false,"given":"Sunyang","family":"Fu","sequence":"first","affiliation":[{"name":"Department of AI and Informatics, Mayo Clinic , Rochester, MN 55902, United States"},{"name":"Center for Translational AI Excellence and Applications in Medicine, University of Texas Health Science Center at Houston , Houston, TX 77030, United States"}]},{"given":"Liwei","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of AI and Informatics, Mayo Clinic , Rochester, MN 55902, United States"},{"name":"Center for Translational AI Excellence and Applications in Medicine, University of Texas Health Science Center at Houston , Houston, TX 77030, United States"}]},{"given":"Huan","family":"He","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics & Data Science, Yale University , New Haven, CT 06520, United States"}]},{"given":"Andrew","family":"Wen","sequence":"additional","affiliation":[{"name":"Center for Translational AI Excellence and Applications in Medicine, University of Texas Health Science Center at Houston , Houston, TX 77030, United States"}]},{"given":"Nansu","family":"Zong","sequence":"additional","affiliation":[{"name":"Department of AI and Informatics, Mayo Clinic , Rochester, MN 55902, United States"}]},{"given":"Anamika","family":"Kumari","sequence":"additional","affiliation":[{"name":"Department of Population and Quantitative Health Sciences, University of Massachusetts Chan Medical School , Boston, MA 01655, United States"}]},{"given":"Feifan","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Population and Quantitative Health Sciences, University of Massachusetts Chan Medical School , Boston, MA 01655, United States"}]},{"given":"Sicheng","family":"Zhou","sequence":"additional","affiliation":[{"name":"Division of Computational Health Sciences, University of Minnesota , Minneapolis, MN 55455, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8258-3585","authenticated-orcid":false,"given":"Rui","family":"Zhang","sequence":"additional","affiliation":[{"name":"Division of Computational Health Sciences, University of Minnesota , Minneapolis, MN 55455, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7434-6571","authenticated-orcid":false,"given":"Chenyu","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Health Information Management, University of Pittsburgh , Pittsburgh, PA 15260, United States"}]},{"given":"Yanshan","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Health Information Management, University of Pittsburgh , Pittsburgh, PA 15260, United States"}]},{"given":"Jennifer","family":"St Sauver","sequence":"additional","affiliation":[{"name":"Department of Quantitative Health Sciences , Mayo Clinic, Rochester, MN 55902, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2570-3741","authenticated-orcid":false,"given":"Hongfang","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of AI and Informatics, Mayo Clinic , Rochester, MN 55902, United States"},{"name":"Center for Translational AI Excellence and Applications in Medicine, University of Texas Health Science Center at Houston , Houston, TX 77030, United States"}]},{"given":"Sunghwan","family":"Sohn","sequence":"additional","affiliation":[{"name":"Department of AI and Informatics, Mayo Clinic , Rochester, MN 55902, United States"}]}],"member":"286","published-online":{"date-parts":[[2024,5,14]]},"reference":[{"key":"2024062007513134700_ocae101-B1","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.jbi.2017.11.011","article-title":"Clinical information extraction applications: a literature review","volume":"77","author":"Wang","year":"2018","journal-title":"J Biomed Inform"},{"key":"2024062007513134700_ocae101-B2","doi-asserted-by":"crossref","first-page":"103526","DOI":"10.1016\/j.jbi.2020.103526","article-title":"Clinical concept extraction: a methodology review","volume":"109","author":"Fu","year":"2020","journal-title":"J Biomed Inform"},{"key":"2024062007513134700_ocae101-B3","doi-asserted-by":"crossref","first-page":"100430","DOI":"10.1016\/j.jadr.2022.100430","article-title":"Portability of natural language processing methods to detect suicidality from clinical text in US and UK electronic health records","volume":"10","author":"Cusick","year":"2022","journal-title":"J Affect Disord Rep"},{"key":"2024062007513134700_ocae101-B4","author":"Adekkanattu","year":"2019"},{"issue":"8","key":"2024062007513134700_ocae101-B5","doi-asserted-by":"crossref","first-page":"e38155","DOI":"10.2196\/38155","article-title":"Multicenter validation of natural language processing algorithms for the detection of common data elements in operative notes for total hip arthroplasty: algorithm development and validation","volume":"10","author":"Han","year":"2022","journal-title":"JMIR Med Inform"},{"key":"2024062007513134700_ocae101-B6","author":"Wu","year":"2019"},{"key":"2024062007513134700_ocae101-B7","author":"Yuan","year":"2022"},{"issue":"Suppl 1","key":"2024062007513134700_ocae101-B8","doi-asserted-by":"crossref","first-page":"S1","DOI":"10.1186\/1472-6947-13-S1-S1","article-title":"Recognizing clinical entities in hospital discharge summaries using structural support vector machines with word representation features","volume":"13","author":"Tang","year":"2013","journal-title":"BMC Med Inform Decis Mak"},{"issue":"3","key":"2024062007513134700_ocae101-B9","doi-asserted-by":"crossref","first-page":"425","DOI":"10.1016\/j.jbi.2013.01.006","article-title":"An enhanced CRFs-based system for information extraction from radiology reports","volume":"46","author":"Esuli","year":"2013","journal-title":"J Biomed Inform"},{"key":"2024062007513134700_ocae101-B10","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/j.artmed.2016.03.003","article-title":"Utilizing uncoded consultation notes from electronic medical records for predictive modeling of colorectal cancer","volume":"69","author":"Hoogendoorn","year":"2016","journal-title":"Artif Intell Med"},{"key":"2024062007513134700_ocae101-B11","first-page":"739","author":"Aramaki","year":"2010"},{"key":"2024062007513134700_ocae101-B12","author":"Doan","year":"2010"},{"issue":"4","key":"2024062007513134700_ocae101-B13","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1136\/jamia.2010.004804","article-title":"Extracting timing and status descriptors for colonoscopy testing from electronic medical records","volume":"17","author":"Denny","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"2024062007513134700_ocae101-B14","author":"Yoon","year":"2017"},{"key":"2024062007513134700_ocae101-B15","doi-asserted-by":"crossref","first-page":"196","DOI":"10.1016\/j.jbi.2014.11.002","article-title":"Portable automatic text classification for adverse drug reaction detection via multi-corpus training","volume":"53","author":"Sarker","year":"2015","journal-title":"J Biomed Inform"},{"key":"2024062007513134700_ocae101-B16","doi-asserted-by":"crossref","first-page":"S34","DOI":"10.1016\/j.jbi.2017.05.023","article-title":"De-identification of clinical notes via recurrent neural network and conditional random field","volume":"75S","author":"Liu","year":"2017","journal-title":"J Biomed Inform"},{"key":"2024062007513134700_ocae101-B17","author":"Wu","year":"2015"},{"issue":"2","key":"2024062007513134700_ocae101-B18","doi-asserted-by":"crossref","first-page":"e0192360","DOI":"10.1371\/journal.pone.0192360","article-title":"Comparing deep learning and concept extraction based methods for patient phenotyping from clinical narratives","volume":"13","author":"Gehrmann","year":"2018","journal-title":"PLoS One"},{"issue":"4","key":"2024062007513134700_ocae101-B19","doi-asserted-by":"crossref","first-page":"571","DOI":"10.1197\/jamia.M3083","article-title":"Description of a rule-based system for the i2b2 challenge in natural language processing for clinical data","volume":"16","author":"Childs","year":"2009","journal-title":"J Am Med Inform Assoc"},{"issue":"21","key":"2024062007513134700_ocae101-B20","doi-asserted-by":"crossref","first-page":"1931","DOI":"10.2106\/JBJS.19.00071","article-title":"Use of natural language processing algorithms to identify common data elements in operative notes for total hip arthroplasty","volume":"101","author":"Wyles","year":"2019","journal-title":"J Bone Joint Surg Am"},{"issue":"4","key":"2024062007513134700_ocae101-B21","doi-asserted-by":"crossref","first-page":"636","DOI":"10.1016\/j.thromres.2015.01.026","article-title":"Using multiple sources of data for surveillance of postoperative venous thromboembolism among surgical patients treated in Department of Veterans Affairs hospitals, 2005\u20132010","volume":"135","author":"Nelson","year":"2015","journal-title":"Thromb Res"},{"key":"2024062007513134700_ocae101-B22","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1186\/s13326-016-0065-1","article-title":"Extracting a stroke phenotype risk factor from Veteran Health Administration clinical reports: an information content analysis","volume":"7","author":"Mowery","year":"2016","journal-title":"J Biomed Semantics"},{"issue":"3","key":"2024062007513134700_ocae101-B23","doi-asserted-by":"crossref","first-page":"524","DOI":"10.1093\/gerona\/glaa275","article-title":"Ascertainment of delirium status using natural language processing from electronic health records","volume":"77","author":"Fu","year":"2022","journal-title":"J Gerontol A Biol Sci Med Sci"},{"key":"2024062007513134700_ocae101-B24","doi-asserted-by":"crossref","first-page":"104736","DOI":"10.1016\/j.ijmedinf.2022.104736","article-title":"A hybrid model to identify fall occurrence from electronic health records","volume":"162","author":"Fu","year":"2022","journal-title":"Int J Med Inform"},{"issue":"3","key":"2024062007513134700_ocae101-B25","doi-asserted-by":"crossref","first-page":"524","DOI":"10.1093\/gerona\/glaa275","article-title":"Ascertainment of delirium status using natural language processing from electronic health records","volume":"77","author":"Fu","year":"2020","journal-title":"J Gerontol Ser A"},{"key":"2024062007513134700_ocae101-B26","author":"Fu","year":"2021"},{"issue":"1","key":"2024062007513134700_ocae101-B27","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1186\/s12911-020-1072-9","article-title":"Assessment of the impact of EHR heterogeneity for clinical research through a case study of silent brain infarction","volume":"20","author":"Fu","year":"2020","journal-title":"BMC Med Inform Decis Mak"},{"issue":"7","key":"2024062007513134700_ocae101-B28","doi-asserted-by":"crossref","first-page":"1208","DOI":"10.1093\/jamia\/ocac040","article-title":"CancerBERT: a cancer domain-specific language model for extracting breast cancer phenotypes from electronic health records","volume":"29","author":"Zhou","year":"2022","journal-title":"J Am Med Inform Assoc"},{"key":"2024062007513134700_ocae101-B29","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1016\/j.csbj.2023.08.018","article-title":"A cross-institutional evaluation on breast cancer phenotyping NLP algorithms on electronic health records","volume":"22","author":"Zhou","year":"2023","journal-title":"Comput Struct Biotechnol J"},{"issue":"12","key":"2024062007513134700_ocae101-B30","doi-asserted-by":"crossref","first-page":"2036","DOI":"10.1093\/jamia\/ocad134","article-title":"An open natural language processing (NLP) framework for EHR-based clinical research: a case demonstration using the National COVID Cohort Collaborative (N3C)","volume":"30","author":"Liu","year":"2023","journal-title":"J Am Med Inform Assoc"},{"issue":"6","key":"2024062007513134700_ocae101-B31","doi-asserted-by":"crossref","first-page":"1776","DOI":"10.1093\/bioinformatics\/btab880","article-title":"MedTator: a serverless annotation tool for corpus development","volume":"38","author":"He","year":"2022","journal-title":"Bioinformatics"},{"key":"2024062007513134700_ocae101-B32","author":"Zhou","year":"2023"},{"key":"2024062007513134700_ocae101-B33","doi-asserted-by":"crossref","first-page":"958539","DOI":"10.3389\/fdgth.2022.958539","article-title":"Quality assessment of functional status documentation in EHRs across different healthcare institutions","volume":"4","author":"Fu","year":"2022","journal-title":"Front Digit Health"},{"issue":"3","key":"2024062007513134700_ocae101-B34","doi-asserted-by":"crossref","first-page":"398","DOI":"10.1111\/cts.13463","article-title":"Recommended practices and ethical considerations for natural language processing-assisted observational research: a scoping review","volume":"16","author":"Fu","year":"2023","journal-title":"Clin Transl Sci"},{"key":"2024062007513134700_ocae101-B35","volume-title":"Evaluation Methods in Biomedical Informatics","author":"Friedman","year":"2005"},{"key":"2024062007513134700_ocae101-B36","doi-asserted-by":"crossref","first-page":"364","DOI":"10.1016\/j.future.2022.05.014","article-title":"A survey of human-in-the-loop for machine learning","volume":"135","author":"Wu","year":"2022","journal-title":"Future Gener Comput Syst"},{"key":"2024062007513134700_ocae101-B37","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","volume":"33","author":"Lewis","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"2024062007513134700_ocae101-B38","author":"Wang","year":"2022"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/7\/1493\/58243742\/ocae101.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/7\/1493\/58243742\/ocae101.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,20]],"date-time":"2024-06-20T07:54:31Z","timestamp":1718870071000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/7\/1493\/7671274"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,14]]},"references-count":38,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2024,5,14]]},"published-print":{"date-parts":[[2024,6,20]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae101","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,7]]},"published":{"date-parts":[[2024,5,14]]}}}