{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T10:56:17Z","timestamp":1777892177302,"version":"3.51.4"},"reference-count":49,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,5,15]],"date-time":"2024-05-15T00:00:00Z","timestamp":1715731200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,5,15]],"date-time":"2024-05-15T00:00:00Z","timestamp":1715731200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Language models (LMs) such as BERT and GPT have revolutionized natural language processing (NLP). However, the medical field faces challenges in training LMs due to limited data access and privacy constraints imposed by regulations like the Health Insurance Portability and Accountability Act (HIPPA) and the General Data Protection Regulation (GDPR). Federated learning (FL) offers a decentralized solution that enables collaborative learning while ensuring data privacy. In this study, we evaluated FL on 2 biomedical NLP tasks encompassing 8 corpora using 6 LMs. Our results show that: (1) FL models consistently outperformed models trained on individual clients\u2019 data and sometimes performed comparably with models trained with polled data; (2) with the fixed number of total data, FL models training with more clients produced inferior performance but pre-trained transformer-based models exhibited great resilience. (3) FL models significantly outperformed pre-trained LLMs with few-shot prompting.<\/jats:p>","DOI":"10.1038\/s41746-024-01126-4","type":"journal-article","created":{"date-parts":[[2024,5,15]],"date-time":"2024-05-15T10:02:27Z","timestamp":1715767347000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":39,"title":["An in-depth evaluation of federated learning on biomedical natural language processing for information extraction"],"prefix":"10.1038","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5728-5965","authenticated-orcid":false,"given":"Le","family":"Peng","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6172-9784","authenticated-orcid":false,"given":"Gaoxiang","family":"Luo","sequence":"additional","affiliation":[]},{"given":"Sicheng","family":"Zhou","sequence":"additional","affiliation":[]},{"given":"Jiandong","family":"Chen","sequence":"additional","affiliation":[]},{"given":"Ziyue","family":"Xu","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2017-5903","authenticated-orcid":false,"given":"Ju","family":"Sun","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8258-3585","authenticated-orcid":false,"given":"Rui","family":"Zhang","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,5,15]]},"reference":[{"key":"1126_CR1","unstructured":"Devlin, J. et al. \u201cBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.\u201d North American Chapter of the Association for Computational Linguistics 4171\u20134186 (2019)."},{"key":"1126_CR2","unstructured":"Radford, A., Narasimhan, K., Salimans, T. & Sutskever, I. Improving language understanding by generative pre-training. Preprint at arXiv https:\/\/arxiv.org\/pdf\/2012.11747 (2010).."},{"key":"1126_CR3","doi-asserted-by":"crossref","unstructured":"Sun, C. et al. \u201cHow to Fine-Tune BERT for Text Classification?\u201d China National Conference on Chinese Computational Linguistics (2019).","DOI":"10.1007\/978-3-030-32381-3_16"},{"key":"1126_CR4","unstructured":"Xu, H. et al. \u201cBERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis.\u201d North American Chapter of the Association for Computational Linguistics (2019)."},{"key":"1126_CR5","unstructured":"Dathathri, S. et al. Plug and play language models: a simple approach to controlled text generation. Findings of the Association for Computational Linguistics: EMNLP pp. 3973\u20133997 (2021)."},{"key":"1126_CR6","unstructured":"Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. BERTScore: evaluating text generation with BERT. International Conference on Learning Representations (2020)."},{"key":"1126_CR7","unstructured":"Shi, P. & Lin, J. Simple BERT models for relation extraction and semantic role labeling. Preprint at arXiv http:\/\/arxiv.org\/abs\/1904.05255 (2019)."},{"key":"1126_CR8","doi-asserted-by":"publisher","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","volume":"36","author":"J Lee","year":"2020","unstructured":"Lee, J. et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 1234\u20131240 (2020).","journal-title":"Bioinformatics"},{"key":"1126_CR9","doi-asserted-by":"publisher","unstructured":"Huang, K., Altosaar, J. & Ranganath, R. ClinicalBERT: modeling clinical notes and predicting hospital readmission. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.1904.05342 (2020).","DOI":"10.48550\/arXiv.1904.05342"},{"key":"1126_CR10","doi-asserted-by":"publisher","unstructured":"Yang, W. et al. End-to-end open-domain question answering with BERTserini. In Proceedings of the 2019 Conference of the North 72\u201377. https:\/\/doi.org\/10.18653\/v1\/N19-4013 (2019).","DOI":"10.18653\/v1\/N19-4013"},{"key":"1126_CR11","doi-asserted-by":"publisher","unstructured":"Qu, C. et al. BERT with history answer embedding for conversational question answering. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval 1133\u20131136 (ACM, 2019). https:\/\/doi.org\/10.1145\/3331184.3331341.","DOI":"10.1145\/3331184.3331341"},{"key":"1126_CR12","doi-asserted-by":"crossref","unstructured":"Peng, Y. et al. \u201cTransfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets.\u201d Proceedings of the 2019 Workshop on Biomedical Natural Language Processing (2019).","DOI":"10.18653\/v1\/W19-5006"},{"key":"1126_CR13","doi-asserted-by":"publisher","first-page":"100729","DOI":"10.1016\/j.patter.2023.100729","volume":"4","author":"R Tinn","year":"2023","unstructured":"Tinn, R. et al. Fine-tuning large neural language models for biomedical natural language processing. Patterns 4, 100729 (2023).","journal-title":"Patterns"},{"key":"1126_CR14","unstructured":"A Study of Abbreviations in Clinical Notes\u2014PMC. https:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2655910\/."},{"key":"1126_CR15","first-page":"572","volume":"42","author":"M Reisman","year":"2017","unstructured":"Reisman, M. EHRs: the challenge of making electronic data usable and interoperable. Pharm. Ther. 42, 572\u2013575 (2017).","journal-title":"Pharm. Ther."},{"key":"1126_CR16","doi-asserted-by":"publisher","first-page":"1208","DOI":"10.1093\/jamia\/ocac040","volume":"29","author":"S Zhou","year":"2022","unstructured":"Zhou, S., Wang, N., Wang, L., Liu, H. & Zhang, R. CancerBERT: a cancer domain-specific language model for extracting breast cancer phenotypes from electronic health records. J. Am. Med. Inform. Assoc. JAMIA 29, 1208\u20131216 (2022).","journal-title":"J. Am. Med. Inform. Assoc. JAMIA"},{"key":"1126_CR17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3458754","volume":"3","author":"Y Gu","year":"2022","unstructured":"Gu, Y. et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans. Comput. Healthc. 3, 1\u201323 (2022).","journal-title":"ACM Trans. Comput. Healthc."},{"key":"1126_CR18","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2016.35","volume":"3","author":"AEW Johnson","year":"2016","unstructured":"Johnson, A. E. W. et al. MIMIC-III, a freely accessible critical care database. Sci. Data 3, 160035 (2016).","journal-title":"Sci. Data"},{"key":"1126_CR19","unstructured":"Lafferty, J. D., McCallum, A. & Pereira, F. C. N. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proceedings of the Eighteenth International Conference on Machine Learning 282\u2013289 (Morgan Kaufmann Publishers Inc., 2001)."},{"key":"1126_CR20","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","volume":"9","author":"S Hochreiter","year":"1997","unstructured":"Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735\u20131780 (1997).","journal-title":"Neural Comput."},{"key":"1126_CR21","unstructured":"The genetic association database\u2014PubMed. https:\/\/pubmed.ncbi.nlm.nih.gov\/15118671\/."},{"key":"1126_CR22","unstructured":"Kone\u010dn\u00fd, J. et al. Federated learning: strategies for improving communication efficiency. NIPS Workshop on Private Multi-Party Machine Learning (2016)."},{"key":"1126_CR23","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1093\/jamia\/ocac188","volume":"30","author":"L Peng","year":"2023","unstructured":"Peng, L. et al. Evaluation of federated learning variations for COVID-19 diagnosis using chest radiographs from 42 US and European hospitals. J. Am. Med. Inform. Assoc. 30, 54\u201363 (2023).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"1126_CR24","doi-asserted-by":"crossref","unstructured":"Long, G, et al. \"Federated learning for open banking.\" Federated Learning: Privacy and Incentive 240\u2013254 (2020).","DOI":"10.1007\/978-3-030-63076-8_17"},{"key":"1126_CR25","doi-asserted-by":"crossref","unstructured":"Nguyen, A. et al. \u201cDeep Federated Learning for Autonomous Driving.\u201d 2022 IEEE Intelligent Vehicles Symposium (IV), 1824\u20131830 (2021).","DOI":"10.1109\/IV51971.2022.9827020"},{"key":"1126_CR26","unstructured":"Liu, M. et al. Federated learning meets natural language processing: a survey. Preprint at arXiv http:\/\/arxiv.org\/abs\/2107.12603 (2021)."},{"key":"1126_CR27","doi-asserted-by":"crossref","unstructured":"Lin, B. Y. et al. \u201cFedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks.\u201d NAACL-HLT (2021).","DOI":"10.18653\/v1\/2022.findings-naacl.13"},{"key":"1126_CR28","doi-asserted-by":"publisher","unstructured":"Sui, D. et al. FedED: federated learning via ensemble distillation for medical relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) 2118\u20132128 (Association for Computational Linguistics, 2020) https:\/\/doi.org\/10.18653\/v1\/2020.emnlp-main.165.","DOI":"10.18653\/v1\/2020.emnlp-main.165"},{"key":"1126_CR29","unstructured":"Liu, D. & Miller, T. Federated pretraining and fine tuning of BERT using clinical notes from multiple silos. Preprint at arXiv http:\/\/arxiv.org\/abs\/2002.08562 (2020)."},{"key":"1126_CR30","unstructured":"McMahan, B., Moore, E., Ramage, D., Hampson, S. & Arcas, B. A. Y. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics 1273\u20131282 (PMLR, 2017)."},{"key":"1126_CR31","unstructured":"Li, T. et al. \"Federated optimization in heterogeneous networks.\" Proceedings of Machine Learning and Systems 2, 429\u2013450 (2020)."},{"key":"1126_CR32","unstructured":"Chen, Q. et al. \"Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations.\" Preprint at arXiv https:\/\/arxiv.org\/pdf\/2305.16326 (2023)."},{"key":"1126_CR33","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-022-00742-2","volume":"5","author":"X Yang","year":"2022","unstructured":"Yang, X. et al. A large language model for electronic health records. Npj Digit. Med. 5, 1\u20139 (2022).","journal-title":"Npj Digit. Med."},{"key":"1126_CR34","unstructured":"Large language models encode clinical knowledge. Nature. https:\/\/www.nature.com\/articles\/s41586-023-06291-2."},{"key":"1126_CR35","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3339474","volume":"10","author":"Q Yang","year":"2019","unstructured":"Yang, Q., Liu, Y., Chen, T. & Tong, Y. Federated machine learning: concept and applications. ACM Trans. Intell. Syst. Technol. 10, 1\u201319 (2019).","journal-title":"ACM Trans. Intell. Syst. Technol."},{"key":"1126_CR36","unstructured":"Federated Learning With Differential Privacy: Algorithms and Performance Analysis. IEEE J. Mag. https:\/\/ieeexplore.ieee.org\/document\/9069945."},{"key":"1126_CR37","unstructured":"Zhang, C. et al. \u201cBatchCrypt: Efficient Homomorphic Encryption for Cross-Silo Federated Learning.\u201d USENIX Annual Technical Conference (2020)."},{"key":"1126_CR38","unstructured":"Radford, A. et al. \"Language models are unsupervised multitask learners.\" OpenAI blog 1.8 9 (2019)."},{"key":"1126_CR39","unstructured":"[1508.01991] Bidirectional LSTM-CRF Models for Sequence Tagging. https:\/\/arxiv.org\/abs\/1508.01991."},{"key":"1126_CR40","unstructured":"Alsentzer, E. et al. Publicly available clinical BERT embeddings. Preprint at arXiv http:\/\/arxiv.org\/abs\/1904.03323 (2019)."},{"key":"1126_CR41","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1093\/jamia\/ocz166","volume":"27","author":"S Henry","year":"2020","unstructured":"Henry, S., Buchan, K., Filannino, M., Stubbs, A. & Uzuner, O. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. J. Am. Med. Inform. Assoc. 27, 3\u201312 (2020).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"1126_CR42","doi-asserted-by":"publisher","DOI":"10.1186\/gb-2008-9-s2-s2","volume":"9","author":"L Smith","year":"2008","unstructured":"Smith, L. et al. Overview of BioCreative II gene mention recognition. Genome Biol. 9, S2 (2008).","journal-title":"Genome Biol."},{"key":"1126_CR43","doi-asserted-by":"publisher","DOI":"10.1186\/1758-2946-7-S1-S2","volume":"7","author":"M Krallinger","year":"2015","unstructured":"Krallinger, M. et al. The CHEMDNER corpus of chemicals and drugs and its annotation principles. J. Cheminformatics 7, S2 (2015).","journal-title":"J. Cheminformatics"},{"key":"1126_CR44","unstructured":"Collier, N., Ohta, T., Tsuruoka, Y., Tateisi, Y. & Kim, J.-D. Introduction to the bio-entity recognition task at JNLPBA. In Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA\/BioNLP) 73\u201378 (COLING, 2004)."},{"key":"1126_CR45","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.jbi.2013.12.006","volume":"47","author":"RI Do\u011fan","year":"2014","unstructured":"Do\u011fan, R. I., Leaman, R. & Lu, Z. NCBI disease corpus: a resource for disease name recognition and concept normalization. J. Biomed. Inform. 47, 1\u201310 (2014).","journal-title":"J. Biomed. Inform."},{"key":"1126_CR46","doi-asserted-by":"publisher","first-page":"879","DOI":"10.1016\/j.jbi.2012.04.004","volume":"45","author":"EM van Mulligen","year":"2012","unstructured":"van Mulligen, E. M. et al. The EU-ADR corpus: annotated drugs, diseases, targets, and their relationships. J. Biomed. Inform. 45, 879\u2013884 (2012).","journal-title":"J. Biomed. Inform."},{"key":"1126_CR47","unstructured":"OpenAI. GPT-4 Technical Report. Preprint at arXiv http:\/\/arxiv.org\/abs\/2303.08774 (2023)."},{"key":"1126_CR48","doi-asserted-by":"publisher","unstructured":"Anil, R. et al. PaLM 2 Technical Report. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2305.10403 (2023).","DOI":"10.48550\/arXiv.2305.10403"},{"key":"1126_CR49","unstructured":"Gemini Team et al. Gemini: a family of highly capable multimodal models. Preprint at arXiv http:\/\/arxiv.org\/abs\/2312.11805 (2023)."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-024-01126-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-024-01126-4","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-024-01126-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,15]],"date-time":"2024-05-15T10:12:44Z","timestamp":1715767964000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-024-01126-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,15]]},"references-count":49,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["1126"],"URL":"https:\/\/doi.org\/10.1038\/s41746-024-01126-4","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,15]]},"assertion":[{"value":"1 November 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 April 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 May 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"127"}}