{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T15:25:06Z","timestamp":1787066706000,"version":"build-2736575974"},"reference-count":47,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"\n Background<\/jats:title>\n Clinical natural language processing (NLP) researchers need access to directly comparable evaluation results for applications such as text deidentification across a range of corpus types and the means to easily test new systems or corpora within the same framework. Current systems, reported metrics, and the personally identifiable information (PII) categories evaluated are not easily comparable.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study presents an open-source and extensible end-to-end framework for comparing clinical NLP system performance across corpora even when the annotation categories do not align.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n As a use case for this framework, we use 6 off-the-shelf text deidentification systems (ie, CliniDeID, deid from PhysioNet, MITRE Identity Scrubber Toolkit [MIST], NeuroNER, National Library of Medicine [NLM] Scrubber, and Philter) across 3 standard clinical text corpora for the task (2 of which are publicly available) and 1 private corpus (all in English), with annotation categories that are not directly analogous. The framework is built on shell scripts that can be extended to include new systems, corpora, and performance metrics. We present this open tool, multiple means for aligning PII categories during evaluation, and our initial timing and performance metric findings. Code for running this framework with all settings needed to run all pairs are available via Codeberg and GitHub.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n From this case study, we found large differences in processing speed between systems. The fastest system (ie, MIST) processed an average of 24.57 (SD 26.23) notes per second, while the slowest (ie, CliniDeID) processed an average of 1.00 notes per second. No system uniformly outperformed the others at identifying PII across corpora and categories. Instead, a rich tapestry of performance trade-offs emerged for PII categories. CliniDeID and Philter prioritize recall over precision (with an average recall 6.9 and 11.2 points higher, respectively, for partially matching spans of text matching any PII category), while the other 4 systems consistently have higher precision (with MIST\u2019s precision scoring 20.2 points higher, NLM Scrubber scoring 4.4 points higher, NeuroNER scoring 7.2 points higher, and deid scoring 17.1 points higher). The macroaverage recall across corpora for identifying names, one of the more sensitive PII categories, included deid (48.8%) and MIST (66.9%) at the low end and NeuroNER (84.1%), NLM Scrubber (88.1%), and CliniDeID (95.9%) at the high end. A variety of metrics across categories and corpora are reported with a wider variety (eg, F2-score) available via the tool.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n NLP systems in general and deidentification systems and corpora in our use case tend to be evaluated in stand-alone research articles that only include a limited set of comparators. We hold that a single evaluation pipeline across multiple systems and corpora allows for more nuanced comparisons. Our open pipeline should reduce barriers to evaluation and system advancement.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/55676","type":"journal-article","created":{"date-parts":[[2024,4,13]],"date-time":"2024-04-13T00:04:50Z","timestamp":1712966690000},"page":"e55676","source":"Crossref","is-referenced-by-count":4,"title":["An Extensible Evaluation Framework Applied to Clinical Text Deidentification Natural Language Processing Tools: Multisystem and Multicorpus Study"],"prefix":"10.2196","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1589-4567","authenticated-orcid":false,"given":"Paul M","family":"Heider","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7632-9625","authenticated-orcid":false,"given":"St\u00e9phane M","family":"Meystre","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2024,5,28]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1109\/CIC.2005.1588104"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6947-8-32"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.2020190137"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1055\/a-2282-4340"},{"key":"ref5","first-page":"333","author":"Sweeney, L","year":"1996","journal-title":"Proc AMIA Annu Fall Symp"},{"key":"ref6","doi-asserted-by":"publisher","DOI":"10.1197\/jamia.M2444"},{"key":"ref7","unstructured":"MoritaMKanoYOhkumaTMiyabeMAramakiEOverview of the NTCIR-10 MedNLP taskProceedings of the 10th NTCIR Conference2013NTCIR 2013June 18-21, 2013Tokyo, Japan"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2015.06.007"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2017.06.011"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2021.100255"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-0258-y"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2022.104215"},{"key":"ref13","unstructured":"HeiderPMAccettaJKMeystreSMETUDE for easy and efficient NLP application evaluationProceedings of the AMIA NLP-WG Pre-Symposium2018AMIA NLP-WG Pre-Symposium 2018November 3, 2018San Francisco, CA"},{"key":"ref14","unstructured":"HeiderPMAccettaJKMeystreSMETUDE: demonstrating multiple matching styles and offset calculators within a natural language processing evaluation toolProceedings of the 17th World Congress on Medical and Health Informatics2019MEDINFO 2019August 25-30, 2019Lyon, France"},{"key":"ref15","unstructured":"45 CFR parts 160 and 164U.S. Department of Health and Human Services2024-05-01https:\/\/www.hhs.gov\/hipaa\/for-professionals\/privacy\/guidance\/introduction\/index.html"},{"key":"ref16","unstructured":"General data protection regulation compliance guidelinesEU General Data Protection Regulation2024-05-01https:\/\/gdpr.eu\/"},{"key":"ref17","unstructured":"MeystreSMHeiderPMKimYTriceAUnderwoodGClinical text automatic de-identification to support large scale data reuse and sharing: pilot resultsProceedings of the American Medical Informatics Association Annual Symposium2018AMIA 2018November 3-7, 2018San Francisco, CA"},{"key":"ref18","unstructured":"Clinacuity \/ CliniDeIDGitHub2024-04-08https:\/\/github.com\/Clinacuity\/CliniDeID"},{"key":"ref19","unstructured":"De-identification software packagePhysioNet200712182024-04-09https:\/\/physionet.org\/content\/deid\/1.1\/"},{"key":"ref20","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2010.09.007"},{"key":"ref21","unstructured":"MITRE identification scrubber toolkit filesSourceForge2024-04-08https:\/\/sourceforge.net\/projects\/mist-deid\/files\/"},{"key":"ref22","unstructured":"MIST: the MITRE identification scrubber toolkit (documentation)SourceForge2024-04-08https:\/\/mist-deid.sourceforge.net\/docs_2_0_4\/html\/index.html"},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/d17-2017"},{"key":"ref24","unstructured":"Franck-Dernoncourt \/ NeuroNERGitHub2024-04-08https:\/\/github.com\/Franck-Dernoncourt\/NeuroNER"},{"key":"ref25","first-page":"707","volume":"2015","author":"Kayaalp, M","year":"2015","journal-title":"AMIA Annu Symp Proc"},{"key":"ref26","unstructured":"KayaalpMSaganPBrowneACMcDonaldCJGuidelines for annotating personal identifiers in the clinical text repository of the National Institutes of Health (version 6\/28\/2016)Lister Hill National Center for Biomedical Communications, U.S. National Library of Medicine, National Institutes of Health20162022-12-23https:\/\/lhncbc.nlm.nih.gov\/scrubber\/annotation.html"},{"key":"ref27","unstructured":"NLM-scrubber downloadsNational Institutes of Health National Library of Medicine2024-04-08https:\/\/lhncbc.nlm.nih.gov\/scrubber\/download.html"},{"key":"ref28","unstructured":"BCHSI\/philter-ucsfGitHub2024-04-08https:\/\/github.com\/BCHSI\/philter-ucsf"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2012-001034"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/d14-1162"},{"key":"ref31","unstructured":"StenetorpPPyysaloSTopi\u0107GOhtaTAnaniadouSTsujiiJbrat: a web-based tool for NLP-assisted text annotationProceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics2012EACL 2012April 23-27, 2012Avignon, France"},{"key":"ref32","unstructured":"MUSC-TBIC\/corpus-utilsGitHub2020-07-27https:\/\/github.com\/MUSC-TBIC\/corpus-utils"},{"key":"ref33","first-page":"241","volume":"2020","author":"Heider, PM","year":"2020","journal-title":"AMIA Jt Summits Transl Sci Proc"},{"key":"ref34","unstructured":"HeiderPMHeiderLab\/article-addendaCodeberg2023-10-04https:\/\/codeberg.org\/HeiderLab\/article-addenda"},{"key":"ref35","unstructured":"MUSC-TBIC\/article-addendaGitHub2023-10-04https:\/\/github.com\/MUSC-TBIC\/article-addenda"},{"key":"ref36","doi-asserted-by":"publisher","DOI":"10.21105\/joss.02959"},{"key":"ref37","doi-asserted-by":"publisher","DOI":"10.1109\/ichi57859.2023.00129"},{"key":"ref38","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3593982"},{"key":"ref39","unstructured":"NLP sandboxGitHub2020-07-27https:\/\/github.com\/nlpsandbox"},{"key":"ref40","unstructured":"TiDE text deidSTAnford Medicine Research data Repository2023-10-02https:\/\/starr.stanford.edu\/methods\/tide-text-deid"},{"key":"ref41","unstructured":"KimYHeiderPMMeystreSMAutomated category alignment applied to different de-identification annotation schemataProceedings of the AMIA 2021 Virtual Informatics Summit2021AMIA 2021March 22-25, 2021Online"},{"key":"ref42","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocw156"},{"key":"ref43","doi-asserted-by":"publisher","DOI":"10.1161\/01.cir.101.23.e215"},{"key":"ref44","doi-asserted-by":"publisher","DOI":"10.1097\/CCM.0b013e31820a92c6"},{"key":"ref45","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-12-109"},{"key":"ref46","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2012-001020"},{"key":"ref47","unstructured":"Data setsNational NLP Clinical Challenges (n2c2)2022-03-08https:\/\/n2c2.dbmi.hms.harvard.edu\/data-sets"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2024\/1\/e55676","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:43:48Z","timestamp":1782841428000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2024\/1\/e55676"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,28]]},"references-count":47,"URL":"https:\/\/doi.org\/10.2196\/55676","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.55676","asserted-by":"object"}]},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,28]]}}}