{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T10:07:09Z","timestamp":1792058829112,"version":"4.4.0"},"reference-count":61,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2024,6,4]],"date-time":"2024-06-04T00:00:00Z","timestamp":1717459200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000056","name":"National Institute of Nursing Research","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000056","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000025","name":"National Institute of Mental Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000025","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["1R01NR020868"],"award-info":[{"award-number":["1R01NR020868"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01MH125027"],"award-info":[{"award-number":["R01MH125027"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"name":"National Center on Homelessness among Veterans"},{"name":"US Department of Veterans Affairs Homeless Programs Office"},{"DOI":"10.13039\/100000025","name":"NIMH","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000025","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000056","name":"NINR","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000056","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000738","name":"Department of Veterans Affairs","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000738","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>To enhance the performance of large language models (LLMs) in biomedical natural language processing (BioNLP) by introducing a domain-specific instruction dataset and examining its impact when combined with multi-task learning principles.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We created the BioInstruct, comprising 25\u00a0005 instructions to instruction-tune LLMs (LLaMA 1 and 2, 7B and 13B version). The instructions were created by prompting the GPT-4 language model with 3-seed samples randomly drawn from an 80 human curated instructions. We employed Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning. We then evaluated these instruction-tuned LLMs on several BioNLP tasks, which can be grouped into 3 major categories: question answering (QA), information extraction (IE), and text generation (GEN). We also examined whether categories (eg, QA, IE, and generation) of instructions impact model performance.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results and Discussion<\/jats:title>\n                  <jats:p>Comparing with LLMs without instruction-tuned, our instruction-tuned LLMs demonstrated marked performance gains: 17.3% in QA on average accuracy metric, 5.7% in IE on average F1 metric, and 96% in Generation tasks on average GPT-4 score metric. Our 7B-parameter instruction-tuned LLaMA 1 model was competitive or even surpassed other LLMs in the biomedical domain that were also fine-tuned from LLaMA 1 with vast domain-specific data or a variety of tasks. Our results also show that the performance gain is significantly higher when instruction fine-tuning is conducted with closely related tasks. Our findings align with the observations of multi-task learning, suggesting the synergies between 2 tasks.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>The BioInstruct dataset serves as a valuable resource and instruction tuned LLMs lead to the best performing BioNLP applications.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae122","type":"journal-article","created":{"date-parts":[[2024,6,5]],"date-time":"2024-06-05T03:50:42Z","timestamp":1717559442000},"page":"1821-1832","source":"Crossref","is-referenced-by-count":68,"title":["BioInstruct: instruction tuning of large language models for biomedical natural language processing"],"prefix":"10.1093","volume":"31","author":[{"given":"Hieu","family":"Tran","sequence":"first","affiliation":[{"name":"Manning College of Information and Computer Sciences, University of Massachusetts Amherst , Amherst, MA 01003,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhichao","family":"Yang","sequence":"additional","affiliation":[{"name":"Manning College of Information and Computer Sciences, University of Massachusetts Amherst , Amherst, MA 01003,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5707-8410","authenticated-orcid":false,"given":"Zonghai","family":"Yao","sequence":"additional","affiliation":[{"name":"Manning College of Information and Computer Sciences, University of Massachusetts Amherst , Amherst, MA 01003,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hong","family":"Yu","sequence":"additional","affiliation":[{"name":"Manning College of Information and Computer Sciences, University of Massachusetts Amherst , Amherst, MA 01003,","place":["United States"]},{"name":"Department of Medicine, University of Massachusetts Medical School , Worcester, MA 01655,","place":["United States"]},{"name":"Center for Biomedical and Health Research in Data Sciences, Miner School of Computer and Information Sciences, University of Massachusetts Lowell , Lowell, MA 01854,","place":["United States"]},{"name":"Center for Healthcare Organization and Implementation Research, VA Bedford Health Care , Bedford, MA 01730,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,6,4]]},"reference":[{"key":"2025060320012431800_ocae122-B1","first-page":"1877","volume-title":"Advances in Neural Information Processing Systems","author":"Brown","year":"2020"},{"key":"2025060320012431800_ocae122-B2","author":"Sanh","year":"2021"},{"key":"2025060320012431800_ocae122-B3","author":"Chowdhery","year":"2022"},{"key":"2025060320012431800_ocae122-B4","author":"Longpre","year":"2023"},{"key":"2025060320012431800_ocae122-B5","author":"OpenAI","year":"2023"},{"key":"2025060320012431800_ocae122-B6","author":"Yang","year":"2023"},{"key":"2025060320012431800_ocae122-B7","author":"Alsentzer"},{"key":"2025060320012431800_ocae122-B8","first-page":"58","author":"Peng"},{"key":"2025060320012431800_ocae122-B9","first-page":"881","author":"Van Aken"},{"key":"2025060320012431800_ocae122-B10","first-page":"578","author":"Hernandez"},{"key":"2025060320012431800_ocae122-B11","author":"Wei","year":"2021"},{"key":"2025060320012431800_ocae122-B12","author":"Chung","year":"2022"},{"key":"2025060320012431800_ocae122-B13","first-page":"3470","author":"Mishra","year":"2022"},{"key":"2025060320012431800_ocae122-B14","first-page":"5085","author":"Wang"},{"key":"2025060320012431800_ocae122-B15","first-page":"13","author":"Ouyang","year":"2022","journal-title":"Training Language Models to Follow Instructions with Human Feedback"},{"issue":"1","key":"2025060320012431800_ocae122-B16","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1038\/s41746-021-00455-y","article-title":"Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction","volume":"4","author":"Rasmy","year":"2021","journal-title":"NPJ Digit Med"},{"key":"2025060320012431800_ocae122-B17","first-page":"13484","author":"Wang"},{"key":"2025060320012431800_ocae122-B18","author":"Touvron","year":"2023"},{"key":"2025060320012431800_ocae122-B19","author":"Touvron","year":"2023"},{"issue":"2","key":"2025060320012431800_ocae122-B20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3490238","article-title":"Biomedical question answering: a survey of approaches and challenges","volume":"55","author":"Jin","year":"2022","journal-title":"ACM Comput Surv"},{"key":"2025060320012431800_ocae122-B21","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.jbi.2017.11.011","article-title":"Clinical information extraction applications: a literature review","volume":"77","author":"Wang","year":"2018","journal-title":"J Biomed Inform"},{"key":"2025060320012431800_ocae122-B22","first-page":"503","author":"Ben Abacha","year":"2023"},{"key":"2025060320012431800_ocae122-B23","author":"Wang","year":"2023"},{"key":"2025060320012431800_ocae122-B24","author":"Wang"},{"key":"2025060320012431800_ocae122-B25","first-page":"4958","author":"Krishna"},{"key":"2025060320012431800_ocae122-B26","first-page":"2604","author":"Yao"},{"key":"2025060320012431800_ocae122-B27","first-page":"9241","author":"Zeng","year":"2020"},{"key":"2025060320012431800_ocae122-B28","first-page":"3764","author":"Yang","year":"2020"},{"key":"2025060320012431800_ocae122-B29","author":"Wu"},{"key":"2025060320012431800_ocae122-B30","author":"Kweon","year":"2023"},{"key":"2025060320012431800_ocae122-B31","author":"Han","year":"2023"},{"key":"2025060320012431800_ocae122-B32","author":"Li","year":"2023"},{"issue":"4","key":"2025060320012431800_ocae122-B33","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2025060320012431800_ocae122-B34","first-page":"6943","author":"Schick","year":"2021"},{"key":"2025060320012431800_ocae122-B35","first-page":"24457","author":"Meng","year":"2023"},{"key":"2025060320012431800_ocae122-B36","first-page":"14409","author":"Honovich"},{"key":"2025060320012431800_ocae122-B37","author":"Zhang","year":"2023"},{"key":"2025060320012431800_ocae122-B38","author":"Taori","year":"2023"},{"issue":"14","key":"2025060320012431800_ocae122-B39","doi-asserted-by":"crossref","first-page":"6421","DOI":"10.3390\/app11146421","article-title":"What disease does this patient have? a large-scale open domain question answering dataset from medical exams","volume":"11","author":"Jin","year":"2021","journal-title":"Appl Sci"},{"key":"2025060320012431800_ocae122-B40","first-page":"248","author":"Pal","year":"2022"},{"key":"2025060320012431800_ocae122-B41","first-page":"2567","author":"Jin"},{"issue":"1","key":"2025060320012431800_ocae122-B42","doi-asserted-by":"crossref","first-page":"138","DOI":"10.1186\/s12859-015-0564-6","article-title":"An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition","volume":"16","author":"Tsatsaronis","year":"2015","journal-title":"BMC Bioinform"},{"key":"2025060320012431800_ocae122-B43","first-page":"1586","author":"Shivade","year":"2019"},{"key":"2025060320012431800_ocae122-B44","first-page":"1998","author":"Agrawal","year":"2022"},{"key":"2025060320012431800_ocae122-B45","author":"Fu","year":"2023"},{"key":"2025060320012431800_ocae122-B46","first-page":"2511","author":"Liu"},{"key":"2025060320012431800_ocae122-B47","first-page":"10859","author":"Zhang"},{"issue":"1","key":"2025060320012431800_ocae122-B48","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1093\/nsr\/nwx105","article-title":"An overview of multi-task learning","volume":"5","author":"Zhang","year":"2018","journal-title":"Nat Sci Rev"},{"issue":"3","key":"2025060320012431800_ocae122-B49","doi-asserted-by":"crossref","first-page":"e14830","DOI":"10.2196\/14830","article-title":"Fine-tuning bidirectional encoder representations from transformers (BERT)-based models on large-scale electronic health record notes: an empirical study","volume":"7","author":"Li","year":"2019","journal-title":"JMIR Med Inform"},{"key":"2025060320012431800_ocae122-B50","first-page":"5231","author":"Pruksachatkun","year":"2020"},{"key":"2025060320012431800_ocae122-B51","author":"McCann","year":"2018"},{"key":"2025060320012431800_ocae122-B52","first-page":"4465","author":"Wang","year":"2019"},{"key":"2025060320012431800_ocae122-B53","author":"Aribandi","year":"2022"},{"key":"2025060320012431800_ocae122-B54","first-page":"7882","author":"Vu","year":"2020"},{"key":"2025060320012431800_ocae122-B55","author":"Kim","year":"2023"},{"key":"2025060320012431800_ocae122-B56","first-page":"100","author":"Liu","year":"2022"},{"key":"2025060320012431800_ocae122-B57","author":"Su","year":"2022"},{"key":"2025060320012431800_ocae122-B58","first-page":"4895","author":"Ainslie"},{"key":"2025060320012431800_ocae122-B59","first-page":"4582","author":"Li"},{"key":"2025060320012431800_ocae122-B60","author":"Hu","year":"2021"},{"key":"2025060320012431800_ocae122-B61","first-page":"7319","author":"Aghajanyan"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/1821\/58868340\/ocae122.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/1821\/58868340\/ocae122.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,4]],"date-time":"2025-06-04T00:01:31Z","timestamp":1748995291000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/9\/1821\/7687618"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,4]]},"references-count":61,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,6,4]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae122","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,6,4]]}}}