{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"medRxiv"}],"indexed":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T17:10:22Z","timestamp":1790183422420,"version":"4.1.0"},"posted":{"date-parts":[[2026,9,20]]},"group-title":"Health Informatics","reference-count":29,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2026,9,20]],"date-time":"2026-09-20T00:00:00Z","timestamp":1789862400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/publicdomain\/zero\/1.0\/"}],"funder":[{"name":"National Institutes of Health","award":["ZIA TR000548"],"award-info":[{"award-number":["ZIA TR000548"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2026,9,20]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:sec>\n                  <jats:title>Background<\/jats:title>\n                  <jats:p>Rare diseases affect an estimated 300 million people worldwide, yet the research needed to guide diagnosis and treatment is often fragmented across multiple unstructured literature sources. Natural history studies (NHS) are a key source of this evidence, but manually extracting structured information from NHS publications can be tedious and does not scale.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>We have developed a proof-of-concept for an information extraction pipeline testing three open-source large-language models (LLMs) \u2013 Athena-v3-AWQ, Google\u2019s Gemma3-27B, and Meta\u2019s Llama-3.1-70B-Instruct, to extract key NHS characteristics from PubMed abstracts curated from a Chan Zuckerberg Initiative disease research state model corpus (302 gold-standard and 8,338 full-corpus abstracts), and compared the models on efficiency, extraction completeness, and expert-rated accuracy.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>All three models processed abstracts with success rates exceeding 99%. However, Gemma achieved the best overall performance, with the highest expert-rated accuracy (68.0% of outputs rated \u201cgood\u201d vs. 36.0% for Llama and 10.0% for Athena) and the fastest runtime on the full corpus (\u223c16 minutes for 3,547 abstracts), despite Llama scoring higher on the automated Token F1 metric (0.874 vs. 0.723), highlighting a divergence between automated and human evaluation. Athena\u2019s lower performance was largely attributable to verbatim copying rather than synthesis of extracted content.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>These findings illustrate how locally deployed open-source LLMs can extract structured NHS characteristics at scale, thus supporting their use to accelerate evidence synthesis in rare disease research.<\/jats:p>\n                <\/jats:sec>","DOI":"10.64898\/2026.09.17.26363303","type":"posted-content","created":{"date-parts":[[2026,9,20]],"date-time":"2026-09-20T22:10:17Z","timestamp":1789942217000},"source":"Crossref","is-referenced-by-count":0,"title":["LLM-enabled Natural History Study Analysis to Support Rare Disease Research"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2038-7217","authenticated-orcid":false,"given":"Kevin","family":"Li","sequence":"first","affiliation":[{"name":"Division of Preclinical Innovations, National Center for Advancing Translational Sciences, Rockville, MD, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7697-3026","authenticated-orcid":false,"given":"Eric","family":"Sid","sequence":"additional","affiliation":[{"name":"Division of Rare Disease Research Innovations, National Center for Advancing Translational Sciences, Bethesda, MD, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4858-6333","authenticated-orcid":false,"given":"Qian","family":"Zhu","sequence":"additional","affiliation":[{"name":"Division of Preclinical Innovations, National Center for Advancing Translational Sciences, Rockville, MD, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"54368","reference":[{"key":"2026092309451259000_2026.09.17.26363303v1.1","unstructured":"Orphan Drug Act of 1983, Pub. L. No. 97-414, 21 U.S.C. \u00a7 360bb."},{"key":"2026092309451259000_2026.09.17.26363303v1.2","doi-asserted-by":"publisher","DOI":"10.1038\/d41573-019-00180-y"},{"key":"2026092309451259000_2026.09.17.26363303v1.3","doi-asserted-by":"publisher","DOI":"10.1038\/s41431-019-0508-0"},{"key":"2026092309451259000_2026.09.17.26363303v1.4","unstructured":"National Human Genome Research Institute. Rare Diseases FAQ. https:\/\/www.genome.gov\/FAQ\/Rare-Diseases"},{"key":"2026092309451259000_2026.09.17.26363303v1.5","unstructured":"GARD - Genetic and Rare Diseases Information Center. About GARD. National Center for Advancing Translational Sciences, NIH. https:\/\/rarediseases.info.nih.gov\/about"},{"issue":"3","key":"2026092309451259000_2026.09.17.26363303v1.6","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1007\/s40290-020-00332-1","article-title":"Establishing Patient Registries for Rare Diseases: Rationale and Challenges","volume":"34","year":"2020","journal-title":"Pharm Med"},{"key":"2026092309451259000_2026.09.17.26363303v1.7","unstructured":"U.S. Food and Drug Administration. Rare Diseases: Natural History Studies for Drug Development. Draft Guidance for Industry. Docket No. FDA-2019-D-0481; March 2019."},{"key":"2026092309451259000_2026.09.17.26363303v1.8","doi-asserted-by":"publisher","DOI":"10.1177\/0883073813495959"},{"key":"2026092309451259000_2026.09.17.26363303v1.9","unstructured":"European Medicines Agency. Guideline on Registry-Based Studies. EMA\/502388\/2020; October 2021."},{"key":"2026092309451259000_2026.09.17.26363303v1.10","doi-asserted-by":"publisher","DOI":"10.1002\/jcph.2134"},{"key":"2026092309451259000_2026.09.17.26363303v1.11","unstructured":"Aronson AR . Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. Proc AMIA Symp. 2001:17\u201321."},{"key":"2026092309451259000_2026.09.17.26363303v1.12","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6947-13-S2-S1"},{"key":"2026092309451259000_2026.09.17.26363303v1.13","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2105-9-207"},{"key":"2026092309451259000_2026.09.17.26363303v1.14","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btz682"},{"key":"2026092309451259000_2026.09.17.26363303v1.15","doi-asserted-by":"publisher","DOI":"10.1145\/3458754"},{"key":"2026092309451259000_2026.09.17.26363303v1.16","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"2026092309451259000_2026.09.17.26363303v1.17","doi-asserted-by":"crossref","unstructured":"Agrawal M , Hegselmann S , Lang H , Kim Y , Sontag D . Large language models are few-shot clinical information extractors . Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2022:1998\u20132022.","DOI":"10.18653\/v1\/2022.emnlp-main.130"},{"key":"2026092309451259000_2026.09.17.26363303v1.18","doi-asserted-by":"publisher","DOI":"10.1007\/s44162-025-00115-9"},{"key":"2026092309451259000_2026.09.17.26363303v1.19","unstructured":"TheBloke. Athena V3 - AWQ [Internet]. Hugging Face; [cited 2026 Aug 5]. Available from: https:\/\/huggingface.co\/TheBloke\/Athena-v3-AWQ. Based on Athena V3 by IkariDev and Undi95, available from: https:\/\/huggingface.co\/IkariDev\/Athena-v3."},{"key":"2026092309451259000_2026.09.17.26363303v1.20","unstructured":"Lin J , Tang J , Tang H , Yang S , Chen WM , Wang WC , Xiao G , Dang X , Gan C , Han S . AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. Proceedings of Machine Learning and Systems (MLSys). 2024;6. arXiv:2306.00978."},{"key":"2026092309451259000_2026.09.17.26363303v1.21","unstructured":"Kamath A , et al. (Gemma Team, Google DeepMind). Gemma 3 Technical Report. arXiv:2503.19786. Published March 12, 2025."},{"key":"2026092309451259000_2026.09.17.26363303v1.22","unstructured":"Grattafiori A , et al. (Llama Team, Meta AI). The Llama 3 Herd of Models. arXiv:2407.21783. Published July 31, 2024."},{"key":"2026092309451259000_2026.09.17.26363303v1.23","doi-asserted-by":"publisher","DOI":"10.1097\/MPG.0000000000001368"},{"key":"2026092309451259000_2026.09.17.26363303v1.24","doi-asserted-by":"crossref","unstructured":"Kwon W , Li Z , Zhuang S , Sheng Y , Zheng L , Yu CH , Gonzalez JE , Zhang H , Stoica I . Efficient Memory Management for Large Language Model Serving with PagedAttention. Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP \u201823). 2023:611\u2013626. arXiv:2309.06180.","DOI":"10.1145\/3600006.3613165"},{"key":"2026092309451259000_2026.09.17.26363303v1.25","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gkaa1043"},{"key":"2026092309451259000_2026.09.17.26363303v1.26","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2011-000116"},{"key":"2026092309451259000_2026.09.17.26363303v1.27","doi-asserted-by":"publisher","DOI":"10.1016\/j.esmoop.2020.100042"},{"key":"2026092309451259000_2026.09.17.26363303v1.28","doi-asserted-by":"publisher","DOI":"10.1002\/lary.26742"},{"key":"2026092309451259000_2026.09.17.26363303v1.29","unstructured":"National Center for Advancing Translational Sciences. NHS Information Extraction Pipeline [Internet]. GitHub. Available from: https:\/\/github.com\/ncats\/RDAS\/tree\/nhs_extraction_dockerization\/NHS_info_extraction."}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.64898\/2026.09.17.26363303","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T16:45:26Z","timestamp":1790181926000},"score":1,"resource":{"primary":{"URL":"http:\/\/medrxiv.org\/lookup\/doi\/10.64898\/2026.09.17.26363303"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,20]]},"references-count":29,"URL":"https:\/\/doi.org\/10.64898\/2026.09.17.26363303","relation":{},"subject":[],"published":{"date-parts":[[2026,9,20]]},"subtype":"preprint"}}