{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T16:18:30Z","timestamp":1784391510291,"version":"3.55.0"},"reference-count":58,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2016,12,31]],"date-time":"2016-12-31T00:00:00Z","timestamp":1483142400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2017,5,1]]},"abstract":"<jats:p>Objective: Patient notes in electronic health records (EHRs) may contain critical information for medical investigations. However, the vast majority of medical investigators can only access de-identified notes, in order to protect the confidentiality of patients. In the United States, the Health Insurance Portability and Accountability Act (HIPAA) defines 18 types of protected health information that needs to be removed to de-identify patient notes. Manual de-identification is impractical given the size of electronic health record databases, the limited number of researchers with access to non-de-identified notes, and the frequent mistakes of human annotators. A reliable automated de-identification system would consequently be of high value.<\/jats:p><jats:p>Materials and Methods: We introduce the first de-identification system based on artificial neural networks (ANNs), which requires no handcrafted features or rules, unlike existing systems. We compare the performance of the system with state-of-the-art systems on two datasets: the i2b2 2014 de-identification challenge dataset, which is the largest publicly available de-identification dataset, and the MIMIC de-identification dataset, which we assembled and is twice as large as the i2b2 2014 dataset.<\/jats:p><jats:p>Results: Our ANN model outperforms the state-of-the-art systems. It yields an F1-score of 97.85 on the i2b2 2014 dataset, with a recall of 97.38 and a precision of 98.32, and an F1-score of 99.23 on the MIMIC de-identification dataset, with a recall of 99.25 and a precision of 99.21.<\/jats:p><jats:p>Conclusion: Our findings support the use of ANNs for de-identification of patient notes, as they show better performance than previously published systems while requiring no manual feature engineering.<\/jats:p>","DOI":"10.1093\/jamia\/ocw156","type":"journal-article","created":{"date-parts":[[2016,10,11]],"date-time":"2016-10-11T14:19:40Z","timestamp":1476195580000},"page":"596-606","source":"Crossref","is-referenced-by-count":251,"title":["De-identification of patient notes with recurrent neural networks"],"prefix":"10.1093","volume":"24","author":[{"given":"Franck","family":"Dernoncourt","sequence":"first","affiliation":[{"name":"Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ji Young","family":"Lee","sequence":"additional","affiliation":[{"name":"Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ozlem","family":"Uzuner","sequence":"additional","affiliation":[{"name":"Computer Science Department, University at Albany, SUNY, Albany, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter","family":"Szolovits","sequence":"additional","affiliation":[{"name":"Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2016,12,31]]},"reference":[{"key":"2021010612000783000_ocw156-B1","doi-asserted-by":"crossref","first-page":"1355","DOI":"10.1377\/hlthaff.2013.0469","article-title":"Some hospitals are falling behind in meeting \u201cmeaningful use\u201d criteria and could be vulnerable to penalties in 2015","volume":"32","author":"DesRoches","year":"2013","journal-title":"Health Affairs."},{"key":"2021010612000783000_ocw156-B2","doi-asserted-by":"crossref","first-page":"779","DOI":"10.1056\/NEJMc1213481","article-title":"Early results of the meaningful use program for electronic health records","volume":"368","author":"Wright","year":"2013","journal-title":"New Engl J Med."},{"key":"2021010612000783000_ocw156-B3","first-page":"53181","article-title":"Standards for privacy of individually identifiable health information","volume":"67","author":"Office for Civil Rights H","year":"2002","journal-title":"Final rule. Federal Register."},{"key":"2021010612000783000_ocw156-B4","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1472-6947-8-32","article-title":"Automated de-identification of free-text medical records","volume":"8","author":"Neamatullah","year":"2008","journal-title":"BMC Med Inform Decis Mak."},{"key":"2021010612000783000_ocw156-B5","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1109\/CIC.2005.1588104","article-title":"De-identification algorithm for free-text nursing notes","author":"Douglass","year":"2005","journal-title":"Comput Cardiol."},{"key":"2021010612000783000_ocw156-B6","first-page":"341","article-title":"Computer-assisted de-identification of free text in the MIMIC II database","author":"Douglas","year":"2004","journal-title":"Comput Cardiol."},{"key":"2021010612000783000_ocw156-B7","doi-asserted-by":"crossref","first-page":"e215","DOI":"10.1161\/01.CIR.101.23.e215","article-title":"Physiobank, physiotoolkit, and physionet components of a new research resource for complex physiologic signals","volume":"101","author":"Goldberger","year":"2000","journal-title":"Circulation."},{"key":"2021010612000783000_ocw156-B8","doi-asserted-by":"crossref","first-page":"952","DOI":"10.1097\/CCM.0b013e31820a92c6","article-title":"Multiparameter intelligent monitoring in intensive care II (MIMIC-II): A public-access intensive care unit database","volume":"39","author":"Saeed","year":"2011","journal-title":"Crit Care Med."},{"key":"2021010612000783000_ocw156-B9","doi-asserted-by":"crossref","DOI":"10.1109\/HISB.2012.33","article-title":"Pre-annotating clinical notes and clinical trial announcements for gold standard corpus development: Evaluating the impact on annotation speed and potential bias","volume-title":"Proc 2012 IEEE 2nd Conf Healthc Informatics, Imaging Syst Biol HISB 2012","author":"Lingren","year":"2012"},{"key":"2021010612000783000_ocw156-B10","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1016\/j.jbi.2014.05.002","article-title":"Evaluating the effects of machine pre-annotation and an interactive annotation interface on manual de-identification of clinical text","volume":"50","author":"South","year":"2014","journal-title":"J Biomed Inform."},{"issue":"9","key":"2021010612000783000_ocw156-B11","doi-asserted-by":"crossref","first-page":"821","DOI":"10.1016\/j.ijmedinf.2013.03.005","article-title":"Bootstrapping a de-identification system for narrative patient records: Cost-performance tradeoffs","volume":"82","author":"Hanauer","year":"2013","journal-title":"Int J Med Inform."},{"key":"2021010612000783000_ocw156-B12","doi-asserted-by":"crossref","first-page":"833","DOI":"10.1136\/amiajnl-2013-002255","article-title":"Assisted annotation of medical free text using RapTAT","volume":"21(5)","author":"Gobbel","year":"2014","journal-title":"J Am Med Inform Assoc."},{"key":"2021010612000783000_ocw156-B13","first-page":"827","article-title":"Rule-based information extraction is dead! Long live rule-based information extraction systems!","author":"Chiticariu","year":"2013","journal-title":"EMNLP"},{"key":"2021010612000783000_ocw156-B14","doi-asserted-by":"crossref","first-page":"680","DOI":"10.5858\/2003-127-680-CMDS","article-title":"Concept-match medical data scrubbing: how pathology text can be used in research","volume":"127","author":"Berman","year":"2003","journal-title":"Arch Pathol Lab Med."},{"key":"2021010612000783000_ocw156-B15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1472-6947-6-12","article-title":"Development and evaluation of an open source software tool for deidentification of pathology reports","volume":"6","author":"Beckwith","year":"2006","journal-title":"BMC Med Inform Decis Mak."},{"key":"2021010612000783000_ocw156-B16","first-page":"1590","article-title":"Algorithmic de-identification of VA medical exam text for HIPAA privacy compliance: preliminary findings","author":"Fielstein","year":"2004","journal-title":"Medinfo."},{"key":"2021010612000783000_ocw156-B17","doi-asserted-by":"crossref","first-page":"601","DOI":"10.1197\/jamia.M2702","article-title":"A software tool for removing patient identifying information from clinical documents","volume":"15","author":"Friedlin","year":"2008","journal-title":"J Am Med Inform Assoc."},{"key":"2021010612000783000_ocw156-B18","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1309\/E6K33GBPE5C27FYU","article-title":"Evaluation of a deidentification (De-Id) software engine to share pathology reports and clinical documents for research","volume":"121","author":"Gupta","year":"2004","journal-title":"Am J Clin Pathol."},{"key":"2021010612000783000_ocw156-B19","first-page":"37","article-title":"Repurposing the clinical record: Can an existing natural language processing system de-identify clinical notes? J Am Med Inform Assoc","volume":"16","author":"Morrison","journal-title":"2009"},{"key":"2021010612000783000_ocw156-B20","first-page":"729","article-title":"Medical document anonymization with a semantic lexicon","author":"Ruch","year":"2000","journal-title":"Proc AMIA Symp."},{"key":"2021010612000783000_ocw156-B21","first-page":"333","article-title":"Replacing personally identifying information in medical records, the Scrub system","author":"Sweeney","year":"1996","journal-title":"Proc AMIA Annual Fall Symp."},{"key":"2021010612000783000_ocw156-B22","first-page":"777","article-title":"A successful technique for removing names in pathology reports using an augmented search and replace method","author":"Thomas","year":"2002","journal-title":"Proc AMIA Symp."},{"key":"2021010612000783000_ocw156-B23","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1007\/11893318_27","article-title":"A multilingual named entity recognition system using boosting and c4.5 decision tree learning algorithms","volume":"4265","author":"Szarvas","year":"2006","journal-title":"Discovery Sci."},{"key":"2021010612000783000_ocw156-B24","first-page":"10","article-title":"Identifying personal health information using support vector machines","volume-title":"I2b2 Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Guo","year":"2006"},{"key":"2021010612000783000_ocw156-B25","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1016\/j.artmed.2007.10.001","article-title":"P. A de-identifier for medical discharge summaries","volume":"42","author":"Uzuner","year":"2008","journal-title":"Artif Intell Med."},{"key":"2021010612000783000_ocw156-B26","first-page":"10","article-title":"Applying a SVM based chunker and a text classifier to the deid challenge","volume-title":"I2b2 Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Hara","year":"2006"},{"key":"2021010612000783000_ocw156-B27","doi-asserted-by":"crossref","first-page":"849","DOI":"10.1016\/j.ijmedinf.2010.09.007","article-title":"The MITRE Identification Scrubber Toolkit: Design, training, and assessment","volume":"79","author":"Aberdeen","year":"2010","journal-title":"Int J Med Inform."},{"key":"2021010612000783000_ocw156-B28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2288-10-70","article-title":"Automatic de-identification of textual documents in the electronic health record: a review of recent research","volume":"10","author":"Meystre","year":"2010","journal-title":"BMC Med Res Methodol."},{"key":"2021010612000783000_ocw156-B29","doi-asserted-by":"crossref","first-page":"S11","DOI":"10.1016\/j.jbi.2015.06.007","article-title":"Automated systems for the de-identification of longitudinal clinical narratives: overview of 2014 i2b2\/UTHealth shared task track 1","volume":"58","author":"Stubbs","year":"2015","journal-title":"J Biomed Inform."},{"key":"2021010612000783000_ocw156-B30","first-page":"3111","article-title":"Distributed representations of words and phrases and their compositionality","author":"Mikolov","year":"2013","journal-title":"Adv Neural Inf Process Syst"},{"key":"2021010612000783000_ocw156-B31","first-page":"2493","article-title":"Natural language processing (almost) from scratch","volume":"12","author":"Collobert","year":"2011","journal-title":"J Mach Learning Res."},{"key":"2021010612000783000_ocw156-B32","doi-asserted-by":"crossref","first-page":"1532","DOI":"10.3115\/v1\/D14-1162","article-title":"GloVe: Global vectors for word representation","volume":"12","author":"Pennington","year":"2014","journal-title":"Proc Empiricial Methods Natural Language Processing (EMNLP 2014)."},{"key":"2021010612000783000_ocw156-B33","first-page":"3","article-title":"Recurrent neural network based language model","author":"Mikolov","year":"2010","journal-title":"Interspeech"},{"key":"2021010612000783000_ocw156-B34","first-page":"1642","article-title":"Recursive deep models for semantic compositionality over a sentiment treebank","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Socher","year":"2013"},{"key":"2021010612000783000_ocw156-B35","first-page":"1746","article-title":"Convolutional neural networks for sentence classification","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing","author":"Kim","year":"2014"},{"key":"2021010612000783000_ocw156-B36","first-page":"655","article-title":"A convolutional neural network for modelling sentences","volume-title":"Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics","author":"Blunsom","year":"2014"},{"key":"2021010612000783000_ocw156-B37","first-page":"515","article-title":"Sequential short-text classification with recurrent and convolutional neural networks","volume-title":"Human Language Technologies 2016: The Conference of the North American Chapter of the Association for Computational Linguistics","author":"Lee","year":"2016"},{"key":"2021010612000783000_ocw156-B38","article-title":"Towards AI-complete question answering: a set of prerequisite toy tasks","author":"Weston"},{"key":"2021010612000783000_ocw156-B39","first-page":"707","article-title":"A long short-term memory model for answer sentence selection in question answering","volume-title":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (vol. 2: short papers) [Internet]","author":"Wang","year":"2015"},{"key":"2021010612000783000_ocw156-B40","article-title":"Neural machine translation by jointly learning to align and translate","author":"Bahdanau"},{"key":"2021010612000783000_ocw156-B41","first-page":"1470","article-title":"Recurrent neural networks for word alignment model","volume":"52","author":"Tamura","year":"2014","journal-title":"ACL (1)"},{"key":"2021010612000783000_ocw156-B42","first-page":"14","article-title":"Translation modeling with bidirectional recurrent neural networks","author":"Sundermeyer","year":"2014","journal-title":"EMNLP"},{"key":"2021010612000783000_ocw156-B43","doi-asserted-by":"crossref","article-title":"Neural architectures for named entity recognition","author":"Lample","DOI":"10.18653\/v1\/N16-1030"},{"key":"2021010612000783000_ocw156-B44","doi-asserted-by":"crossref","first-page":"232","DOI":"10.18653\/v1\/D15-1025","article-title":"Non-lexical neural architecture for fine-grained POS tagging","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing [Internet]","author":"Labeau","year":"2015"},{"key":"2021010612000783000_ocw156-B45","doi-asserted-by":"crossref","article-title":"Character-aware neural language models","author":"Kim","DOI":"10.1609\/aaai.v30i1.10362"},{"key":"2021010612000783000_ocw156-B46","first-page":"624","article-title":"Named entity recognition in Chinese clinical text using deep neural network","volume":"216","author":"Wu","year":"2015","journal-title":"Stud Health Technol Inform."},{"key":"2021010612000783000_ocw156-B47","first-page":"1268","article-title":"UTA DLNLP at SemEval-2016 Task 12: deep learning based natural language processing system for clinical information identification from clinical notes and pathology reports","volume-title":"Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)","author":"Li","year":"2016"},{"key":"2021010612000783000_ocw156-B48","first-page":"1274","article-title":"Brundlefly at SemEval-2016 Task 12: recurrent neural networks vs. joint inference for clinical temporal information extraction","volume-title":"Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)","author":"Fries","year":"2016"},{"key":"2021010612000783000_ocw156-B49","first-page":"1689","article-title":"DeepDive: a data management system for automatic knowledge base construction","volume":"53","author":"Zhang","year":"2015","journal-title":"Thesis."},{"key":"2021010612000783000_ocw156-B50","doi-asserted-by":"crossref","first-page":"55","DOI":"10.3115\/v1\/P14-5010","article-title":"The Stanford CoreNLP Natural Language Processing Toolkit","volume-title":"Proc 52nd Annu Meet Assoc Comput Linguist Syst Demonstr","author":"Manning","year":"2014"},{"key":"2021010612000783000_ocw156-B51","doi-asserted-by":"crossref","first-page":"S30","DOI":"10.1016\/j.jbi.2015.06.015","article-title":"Automatic detection of protected health information from clinic narratives","volume":"58","author":"Yang","year":"2015","journal-title":"J Biomed Inform."},{"key":"2021010612000783000_ocw156-B52","article-title":"ManTIME: temporal expression identification and normalization in the TempEval-3 challenge","author":"Filannino","year":"2013","journal-title":"CoRR."},{"key":"2021010612000783000_ocw156-B53","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"2021010612000783000_ocw156-B54","article-title":"Efficient estimation of word representations in vector space","author":"Mikolov"},{"key":"2021010612000783000_ocw156-B55","first-page":"746","article-title":"Linguistic regularities in continuous space word representations","volume":"13","author":"Mikolov","year":"2013","journal-title":"HLT-NAACL"},{"key":"2021010612000783000_ocw156-B56","doi-asserted-by":"crossref","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Scientific Data."},{"key":"2021010612000783000_ocw156-B57","volume-title":"Computer-assisted De-identification of Free-Text Nursing Notes [Master\u2019s thesis]","author":"Douglass","year":"2005"},{"key":"2021010612000783000_ocw156-B58","volume-title":"English Gigaword Fifth Edition, Linguistic Data Consortium. Technical Report","author":"Parker","year":"2011"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/24\/3\/596\/34945844\/ocw156.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/24\/3\/596\/34945844\/ocw156.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,7,10]],"date-time":"2022-07-10T03:27:24Z","timestamp":1657423644000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/24\/3\/596\/2769353"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,12,31]]},"references-count":58,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2016,12,31]]},"published-print":{"date-parts":[[2017,5,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocw156","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2017,5]]},"published":{"date-parts":[[2016,12,31]]}}}