{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T20:23:14Z","timestamp":1782332594679,"version":"3.54.5"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"S5","license":[{"start":{"date-parts":[[2019,12,1]],"date-time":"2019-12-01T00:00:00Z","timestamp":1575158400000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2019,12,5]],"date-time":"2019-12-05T00:00:00Z","timestamp":1575504000000},"content-version":"vor","delay-in-days":4,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2019,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>De-identification is a critical technology to facilitate the use of unstructured clinical text while protecting patient privacy and confidentiality. The clinical natural language processing (NLP) community has invested great efforts in developing methods and corpora for de-identification of clinical notes. These annotated corpora are valuable resources for developing automated systems to de-identify clinical text at local hospitals. However, existing studies often utilized training and test data collected from the same institution. There are few studies to explore automated de-identification under cross-institute settings. The goal of this study is to examine deep learning-based de-identification methods at a cross-institute setting, identify the bottlenecks, and provide potential solutions.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>We created a de-identification corpus using a total 500 clinical notes from the University of Florida (UF) Health, developed deep learning-based de-identification models using 2014 i2b2\/UTHealth corpus, and evaluated the performance using UF corpus. We compared five different word embeddings trained from the general English text, clinical text, and biomedical literature, explored lexical and linguistic features, and compared two strategies to customize the deep learning models using UF notes and resources.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>Pre-trained word embeddings using a general English corpus achieved better performance than embeddings from de-identified clinical text and biomedical literature. The performance of deep learning models trained using only i2b2 corpus significantly dropped (strict and relax F1 scores dropped from 0.9547 and 0.9646 to 0.8568 and 0.8958) when applied to another corpus annotated at UF Health. Linguistic features could further improve the performance of de-identification in cross-institute settings. After customizing the models using UF notes and resource, the best model achieved the strict and relaxed F1 scores of 0.9288 and 0.9584, respectively.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>It is necessary to customize de-identification models using local clinical text and other resources when applied in cross-institute settings. Fine-tuning is a potential solution to re-use pre-trained parameters and reduce the training time to customize deep learning-based de-identification models trained using clinical corpus from a different institution.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12911-019-0935-4","type":"journal-article","created":{"date-parts":[[2019,12,5]],"date-time":"2019-12-05T01:02:23Z","timestamp":1575507743000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":62,"title":["A study of deep learning methods for de-identification of clinical notes in cross-institute settings"],"prefix":"10.1186","volume":"19","author":[{"given":"Xi","family":"Yang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tianchen","family":"Lyu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qian","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chih-Yin","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiang","family":"Bian","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"William R.","family":"Hogan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yonghui","family":"Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2019,12,5]]},"reference":[{"key":"935_CR1","doi-asserted-by":"publisher","first-page":"70","DOI":"10.1186\/1471-2288-10-70","volume":"10","author":"SM Meystre","year":"2010","unstructured":"Meystre SM, Friedlin FJ, South BR, Shen S, Samore MH. Automatic de-identification of textual documents in the electronic health record: a review of recent research. BMC Med Res Methodol. 2010;10:70.","journal-title":"BMC Med Res Methodol"},{"key":"935_CR2","doi-asserted-by":"publisher","first-page":"8","DOI":"10.4274\/balkanmedj.2017.0966","volume":"35","author":"M Kayaalp","year":"2018","unstructured":"Kayaalp M. Patient privacy in the era of big data. Balkan Med J. 2018;35:8\u201317.","journal-title":"Balkan Med J"},{"key":"935_CR3","first-page":"707","volume":"2015","author":"M Kayaalp","year":"2015","unstructured":"Kayaalp M, Browne AC, Sagan P, McGee T, McDonald CJ. Challenges and insights in using HIPAA privacy rule for clinical text annotation. AMIA Annu Symp Proc. 2015;2015:707\u201316.","journal-title":"AMIA Annu Symp Proc."},{"key":"935_CR4","doi-asserted-by":"publisher","first-page":"162","DOI":"10.1016\/j.jbi.2014.05.002","volume":"50","author":"BR South","year":"2014","unstructured":"South BR, Mowery D, Suo Y, Leng J, Ferr\u00e1ndez \u00d3, Meystre SM, et al. Evaluating the effects of machine pre-annotation and an interactive annotation interface on manual de-identification of clinical text. J Biomed Inform. 2014;50:162\u201372.","journal-title":"J Biomed Inform"},{"key":"935_CR5","first-page":"246","volume":"45","author":"DA Dorr","year":"2018","unstructured":"Dorr DA, Phillips WF, Phansalkar S, Sims SA, Hurdle JF. Assessing the difficulty and time cost of De-identification in clinical narratives. Methods Inf Med. 2018;45:246\u201352.","journal-title":"Methods Inf Med"},{"key":"935_CR6","unstructured":"Yogarajan V, Mayo M, Pfahringer B. A survey of automatic de-identification of longitudinal clinical narratives. CoRR. 2018;abs\/1810.06765."},{"key":"935_CR7","doi-asserted-by":"publisher","first-page":"550","DOI":"10.1197\/jamia.M2444","volume":"14","author":"O Uzuner","year":"2007","unstructured":"Uzuner O, Luo Y, Szolovits P. Evaluating the state-of-the-art in automatic de-identification. J Am Med Inform Assoc. 2007;14:550\u201363.","journal-title":"J Am Med Inform Assoc"},{"key":"935_CR8","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1186\/1472-6947-8-32","volume":"8","author":"I Neamatullah","year":"2008","unstructured":"Neamatullah I, Douglass MM, Lehman LH, Reisner A, Villarroel M, Long WJ, et al. Automated de-identification of free-text medical records. BMC Med Inform Decision Making. 2008;8:32.","journal-title":"BMC Med Inform Decision Making"},{"key":"935_CR9","doi-asserted-by":"publisher","first-page":"S20","DOI":"10.1016\/j.jbi.2015.07.020","volume":"58","author":"A Stubbs","year":"2015","unstructured":"Stubbs A, Uzuner \u00d6. Annotating longitudinal clinical narratives for de-identification: the 2014 i2b2\/UTHealth corpus. J Biomed Inform. 2015;58:S20\u20139.","journal-title":"J Biomed Inform"},{"key":"935_CR10","doi-asserted-by":"publisher","first-page":"S4","DOI":"10.1016\/j.jbi.2017.06.011","volume":"75S","author":"A Stubbs","year":"2017","unstructured":"Stubbs A, Filannino M, Uzuner \u00d6. De-identification of psychiatric intake records: overview of 2016 CEGS N-GRID shared tasks track 1. J Biomed Inform. 2017;75S:S4\u201318.","journal-title":"J Biomed Inform"},{"key":"935_CR11","first-page":"199","volume":"2012","author":"\u00d3 Ferr\u00e1ndez","year":"2012","unstructured":"Ferr\u00e1ndez \u00d3, South BR, Shen S, Friedlin FJ, Samore MH, Meystre SM. Generalizability and comparison of automatic clinical text de-identification methods and resources. AMIA Annu Symp Proc. 2012;2012:199\u2013208.","journal-title":"AMIA Annu Symp Proc"},{"key":"935_CR12","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1186\/1471-2288-12-109","volume":"12","author":"O Ferr\u00e1ndez","year":"2012","unstructured":"Ferr\u00e1ndez O, South BR, Shen S, Friedlin FJ, Samore MH, Meystre SM. Evaluating current automatic de-identification methods with Veteran\u2019s health administration clinical documents. BMC Med Res Methodol. 2012;12:109.","journal-title":"BMC Med Res Methodol"},{"key":"935_CR13","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1016\/j.artmed.2014.03.006","volume":"61","author":"G Zuccon","year":"2014","unstructured":"Zuccon G, Kotzur D, Nguyen A, Bergheim A. De-identification of health records using anonym: effectiveness and robustness across datasets. Artif Intell Med. 2014;61:145\u201351.","journal-title":"Artif Intell Med"},{"key":"935_CR14","doi-asserted-by":"publisher","first-page":"544","DOI":"10.1136\/amiajnl-2011-000464","volume":"18","author":"PM Nadkarni","year":"2011","unstructured":"Nadkarni PM, Ohno-Machado L, Chapman WW. Natural language processing: an introduction. J Am Med Inform Assoc. 2011;18:544\u201351.","journal-title":"J Am Med Inform Assoc"},{"key":"935_CR15","doi-asserted-by":"publisher","first-page":"S11","DOI":"10.1016\/j.jbi.2015.06.007","volume":"58","author":"A Stubbs","year":"2015","unstructured":"Stubbs A, Kotfila C, Uzuner \u00d6. Automated systems for the de-identification of longitudinal clinical narratives: overview of 2014 i2b2\/UTHealth shared task track 1. J Biomed Inform. 2015;58:S11\u20139.","journal-title":"J Biomed Inform"},{"key":"935_CR16","first-page":"282","volume-title":"Proceedings of the eighteenth international conference on machine learning","author":"JD Lafferty","year":"2001","unstructured":"Lafferty JD, McCallum A, Pereira FCN. Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of the eighteenth international conference on machine learning. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc.; 2001. p. 282\u20139. http:\/\/dl.acm.org\/citation.cfm?id=645530.655813."},{"key":"935_CR17","first-page":"1453","volume":"6","author":"I Tsochantaridis","year":"2005","unstructured":"Tsochantaridis I, Joachims T, Hofmann T, Altun Y. Large margin methods for structured and interdependent output variables. J Mach Learn Res. 2005;6:1453\u201384.","journal-title":"J Mach Learn Res"},{"key":"935_CR18","doi-asserted-by":"publisher","first-page":"S30","DOI":"10.1016\/j.jbi.2015.06.015","volume":"58","author":"H Yang","year":"2015","unstructured":"Yang H, Garibaldi JM. Automatic detection of protected health information from clinic narratives. J Biomed Inform. 2015;58:S30\u20138.","journal-title":"J Biomed Inform"},{"key":"935_CR19","doi-asserted-by":"publisher","first-page":"S47","DOI":"10.1016\/j.jbi.2015.06.009","volume":"58","author":"Z Liu","year":"2015","unstructured":"Liu Z, Chen Y, Tang B, Wang X, Chen Q, Li H, et al. Automatic de-identification of electronic medical records using token-level and character-level conditional random fields. J Biomed Inform. 2015;58:S47\u201352.","journal-title":"J Biomed Inform"},{"key":"935_CR20","doi-asserted-by":"publisher","first-page":"S19","DOI":"10.1016\/j.jbi.2017.06.006","volume":"75","author":"H-J Lee","year":"2017","unstructured":"Lee H-J, Wu Y, Zhang Y, Xu J, Xu H, Roberts K. A hybrid approach to automatic de-identification of psychiatric notes. J Biomed Inform. 2017;75:S19\u201327.","journal-title":"J Biomed Inform"},{"key":"935_CR21","doi-asserted-by":"publisher","first-page":"1589","DOI":"10.1109\/JBHI.2017.2767063","volume":"22","author":"B Shickel","year":"2018","unstructured":"Shickel B, Tighe PJ, Bihorac A, Rashidi P. Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE J Biomed Health Inform. 2018;22:1589\u2013604.","journal-title":"IEEE J Biomed Health Inform"},{"key":"935_CR22","doi-asserted-by":"publisher","first-page":"1419","DOI":"10.1093\/jamia\/ocy068","volume":"25","author":"C Xiao","year":"2018","unstructured":"Xiao C, Choi E, Sun J. Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review. J Am Med Inform Assoc. 2018;25:1419\u201328.","journal-title":"J Am Med Inform Assoc"},{"key":"935_CR23","first-page":"624","volume":"216","author":"Y Wu","year":"2015","unstructured":"Wu Y, Jiang M, Lei J, Xu H. Named entity recognition in Chinese clinical text using deep neural network. Stud Health Technol Inform. 2015;216:624\u20138.","journal-title":"Stud Health Technol Inform"},{"key":"935_CR24","first-page":"1812","volume":"2017","author":"Y Wu","year":"2017","unstructured":"Wu Y, Jiang M, Xu J, Zhi D, Xu H. Clinical named entity recognition using deep learning models. AMIA Annu Symp Proc. 2017;2017:1812\u20139.","journal-title":"AMIA Annu Symp Proc."},{"key":"935_CR25","first-page":"1110","volume":"2018","author":"Y Wu","year":"2018","unstructured":"Wu Y, Yang X, Bian J, Guo Y, Xu H, Hogan W. Combine factual medical knowledge and distributed word representation to improve clinical named entity recognition. AMIA Ann Symp Proc. 2018;2018:1110\u20137.","journal-title":"AMIA Ann Symp Proc"},{"key":"935_CR26","doi-asserted-by":"publisher","unstructured":"Yang X, Bian J, Gong Y, Hogan WR, Wu Y. MADEx: a system for detecting medications, adverse drug events, and their relations from clinical notes. Drug Saf. 2019. https:\/\/doi.org\/10.1007\/s40264-018-0761-0.","DOI":"10.1007\/s40264-018-0761-0"},{"key":"935_CR27","doi-asserted-by":"publisher","first-page":"S34","DOI":"10.1016\/j.jbi.2017.05.023","volume":"75","author":"Z Liu","year":"2017","unstructured":"Liu Z, Tang B, Wang X, Chen Q. De-identification of clinical notes via recurrent neural network and conditional random field. J Biomed Inform. 2017;75:S34\u201342.","journal-title":"J Biomed Inform"},{"key":"935_CR28","first-page":"596","volume":"24","author":"JY Lee","year":"2016","unstructured":"Lee JY, Szolovits P, Dernoncourt F, Uzuner O. De-identification of patient notes with recurrent neural networks. J Am Med Inform Assoc. 2016;24:596\u2013606.","journal-title":"J Am Med Inform Assoc"},{"key":"935_CR29","unstructured":"Ramshaw L, Marcus M. Text chunking using transformation-based learning. In: Third Workshop on Very Large Corpora. 1995. http:\/\/aclweb.org\/anthology\/W95-0107."},{"key":"935_CR30","doi-asserted-by":"publisher","first-page":"S43","DOI":"10.1016\/j.jbi.2017.10.003","volume":"75","author":"Z Jiang","year":"2017","unstructured":"Jiang Z, Zhao C, He B, Guan Y, Jiang J. De-identification of medical records using conditional random fields and long short-term memory networks. J Biomed Inform. 2017;75:S43\u201353.","journal-title":"J Biomed Inform"},{"key":"935_CR31","doi-asserted-by":"publisher","first-page":"601","DOI":"10.1136\/amiajnl-2011-000163","volume":"18","author":"M Jiang","year":"2011","unstructured":"Jiang M, Chen Y, Liu M, Rosenbloom ST, Mani S, Denny JC, et al. A study of machine-learning-based approaches to extract clinical entities and their assertions from discharge summaries. J Am Med Inform Assoc. 2011;18:601\u20136.","journal-title":"J Am Med Inform Assoc"},{"key":"935_CR32","unstructured":"Reimers N, Gurevych I. Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks. CoRR. 2017;abs\/1707.06799. http:\/\/arxiv.org\/abs\/1707.06799."},{"key":"935_CR33","first-page":"3111","volume-title":"Proceedings of the 26th international conference on neural information processing systems - volume 2","author":"T Mikolov","year":"2013","unstructured":"Mikolov T, Sutskever I, Chen K, Corrado G, Dean J. Distributed representations of words and phrases and their compositionality. In: Proceedings of the 26th international conference on neural information processing systems - volume 2. USA: Curran Associates Inc.; 2013. p. 3111\u20139. http:\/\/dl.acm.org\/citation.cfm?id=2999792.2999959."},{"key":"935_CR34","unstructured":"Joulin A, Grave E, Bojanowski P, Douze M, J\u00e9gou H, Mikolov T. FastText.zip: Compressing text classification models. arXiv preprint arXiv. 2016:161203651."},{"key":"935_CR35","unstructured":"Mikolov T, Grave E, Bojanowski P, Puhrsch C, Joulin A. Advances in Pre-Training Distributed Word Representations. CoRR. 2017;abs\/1712.09405."},{"key":"935_CR36","doi-asserted-by":"publisher","first-page":"160035","DOI":"10.1038\/sdata.2016.35","volume":"3","author":"AEW Johnson","year":"2016","unstructured":"Johnson AEW, Pollard TJ, Shen L, Lehman LH, Feng M, Ghassemi M, et al. MIMIC-III, a freely accessible critical care database. Scientific Data. 2016;3:160035.","journal-title":"Scientific Data"},{"key":"935_CR37","doi-asserted-by":"crossref","unstructured":"Jagannatha A, Yu H. Structured prediction models for RNN based sequence labeling in clinical text. CoRR. 2016;abs\/1608.00612. http:\/\/arxiv.org\/abs\/1608.00612.","DOI":"10.18653\/v1\/D16-1082"},{"key":"935_CR38","doi-asserted-by":"crossref","unstructured":"Howard J, Ruder S. Universal language model fine-tuning for text classification. In: ACL. 2018.","DOI":"10.18653\/v1\/P18-1031"},{"key":"935_CR39","unstructured":"Dai AM, Le QV. Semi-supervised sequence learning. In: NIPS. 2015."},{"key":"935_CR40","unstructured":"Radford A, Narasimhan K, Salimans T, Sutskever I. Improving language understanding by generative pre-training."},{"key":"935_CR41","unstructured":"Mart\u00edn Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. 2015. https:\/\/www.tensorflow.org\/."}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-019-0935-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s12911-019-0935-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-019-0935-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,12,4]],"date-time":"2020-12-04T00:17:24Z","timestamp":1607041044000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-019-0935-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12]]},"references-count":41,"journal-issue":{"issue":"S5","published-print":{"date-parts":[[2019,12]]}},"alternative-id":["935"],"URL":"https:\/\/doi.org\/10.1186\/s12911-019-0935-4","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,12]]},"assertion":[{"value":"5 December 2019","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Not applicable.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"232"}}