{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T16:42:41Z","timestamp":1791996161957,"version":"4.4.0"},"reference-count":26,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2025,6,14]],"date-time":"2025-06-14T00:00:00Z","timestamp":1749859200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,6,14]],"date-time":"2025-06-14T00:00:00Z","timestamp":1749859200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Ther Innov Regul Sci"],"published-print":{"date-parts":[[2025,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:sec>\n            <jats:title>Introduction<\/jats:title>\n            <jats:p>Generative artificial intelligence (AI) has the potential to transform and accelerate how information is accessed during the regulation of human drug and biologic products.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Objectives<\/jats:title>\n            <jats:p>Determine whether a generative AI-supported application with retrieval-augmented generation (RAG) architecture can be used to correctly answer questions about the information contained in FDA guidance documents.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Methods<\/jats:title>\n            <jats:p>Five large language models (LLMs): Flan-UL2, GPT-3.5 Turbo, GPT-4 Turbo, Granite, and Llama 2, were evaluated in conjunction with the RAG application Golden Retriever to assess their ability to answer questions about the information contained in clinically oriented FDA guidance documents. Models were configured to precise mode with a low temperature parameter setting to generate precise, non-creative answers, ensuring reliable clinical regulatory review guidance for users.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Results<\/jats:title>\n            <jats:p>During preliminary testing, GPT-4 Turbo was the highest performing LLM. It was therefore selected for additional evaluation where it generated a correct response with additional helpful information 33.9% of the time, a correct response 35.7% of the time, a response with some of the required correct information 17.0% of the time, and a response with any incorrect information 13.4% of the time. The RAG application was able to cite the correct source document 89.2% of the time.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Conclusion<\/jats:title>\n            <jats:p>The ability of the generative AI application to identify the correct guidance document and answer questions could significantly reduce the time in finding the correct answer for questions about FDA guidance documents. However, as the information in FDA guidance documents may be relied on by sponsors and FDA staff to guide important drug development decisions, the use of incorrect information could have a significantly negative impact on the drug development process. Based on our results, the correct citation documents can be used to reduce the time in finding the correct document that contains the information, but further research into the refinement of generative AI will likely be required before this tool can be relied on to answer questions about information contained in FDA guidance documents. Rephrasing questions by including additional context information, reconfiguring the embedding and chunking parameters, and other prompt engineering techniques may improve the rate of fully correct and complete responses.<\/jats:p>\n          <\/jats:sec>","DOI":"10.1007\/s43441-025-00798-8","type":"journal-article","created":{"date-parts":[[2025,6,14]],"date-time":"2025-06-14T12:34:38Z","timestamp":1749904478000},"page":"1148-1159","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Semantic Search of FDA Guidance Documents Using Generative AI"],"prefix":"10.1007","volume":"59","author":[{"given":"Scott","family":"Proestel","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Linda J. B.","family":"Jeng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher","family":"Smith","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew","family":"Deady","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Omar","family":"Amer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohamed","family":"Ahmed","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sarah","family":"Rodgers","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,6,14]]},"reference":[{"key":"798_CR1","unstructured":"Bommasani R, Hudson DA, Adeli E, et al. On the Opportunities and Risks of Foundation Models. 2022. https:\/\/arxiv.org\/abs\/2108.07258"},{"issue":"10","key":"798_CR2","doi-asserted-by":"publisher","first-page":"855","DOI":"10.1038\/s41573-023-00769-4","volume":"22","author":"JU Stegmann","year":"2023","unstructured":"Stegmann JU, Littlebury R, Trengove M, Goetz L, Bate A, Branson KM. Trustworthy AI for safe medicines. Nat Rev Drug Discov. 2023;22(10):855\u20136.","journal-title":"Nat Rev Drug Discov"},{"key":"798_CR3","doi-asserted-by":"publisher","first-page":"204209862412933","DOI":"10.1177\/20420986241293303","volume":"15","author":"M Glaser","year":"2024","unstructured":"Glaser M, Littlebury R. Governance of artificial intelligence and machine learning in pharmacovigilance: what works today and what more is needed? Ther Adv Drug Saf. 2024;15:20420986241293304.","journal-title":"Ther Adv Drug Saf"},{"key":"798_CR4","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.3403301","author":"D Leslie","year":"2019","unstructured":"Leslie D. Understanding artificial intelligence ethics and safety: A guide for the responsible design and implementation of AI systems in the public sector. 2019. Zenodo. https:\/\/doi.org\/10.2139\/ssrn.3403301."},{"key":"798_CR5","unstructured":"Wang A, Pruksachatkun Y, Nangia N, et al. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. 2019. https:\/\/arxiv.org\/abs\/1905.00537"},{"key":"798_CR6","doi-asserted-by":"crossref","unstructured":"Wang A, Singh A, Michael J, Hill F, Levy O, Bowman SR. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. 2018. https:\/\/arxiv.org\/abs\/1804.07461","DOI":"10.18653\/v1\/W18-5446"},{"key":"798_CR7","unstructured":"Srivastava A, Rastogi A, Rao A, et al. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. 2022. https:\/\/arxiv.org\/abs\/2206.04615"},{"key":"798_CR8","doi-asserted-by":"publisher","DOI":"10.3389\/fdsfr.2024.1379260","author":"D Li","year":"2024","unstructured":"Li D, Wu L, Zhang M, et al. Assessing the performance of large language models in literature screening for pharmacovigilance: a comparative study. Front Drug Safety Regul. 2024. https:\/\/doi.org\/10.3389\/fdsfr.2024.1379260.","journal-title":"Front Drug Safety Regul"},{"key":"798_CR9","unstructured":"U.S. Food and Drug Administration. Search for FDA Guidance Documents. https:\/\/www.fda.gov\/regulatory-information\/search-fda-guidance-documents"},{"key":"798_CR10","doi-asserted-by":"publisher","first-page":"104662","DOI":"10.1016\/j.jbi.2024.104662","volume":"156","author":"M Alkhalaf","year":"2024","unstructured":"Alkhalaf M, Yu P, Yin M, Deng C. Applying generative AI with retrieval augmented generation to summarize and extract key clinical information from electronic health records. J Biomed Inform. 2024;156:104662.","journal-title":"J Biomed Inform"},{"issue":"2","key":"798_CR11","doi-asserted-by":"publisher","first-page":"210","DOI":"10.7326\/M23-2772","volume":"177","author":"JA Omiye","year":"2024","unstructured":"Omiye JA, Gui H, Rezaei SJ, Zou J, Daneshjou R. Large language models in medicine: the potentials and pitfalls: a narrative review. Ann Intern Med. 2024;177(2):210\u201320.","journal-title":"Ann Intern Med"},{"key":"798_CR12","unstructured":"Tay Y, Dehghani M, Tran VQ, et al. UL2: Unifying Language Learning Paradigms. presented at: International Conference on Learning Representations; 2022; Virtual."},{"key":"798_CR13","unstructured":"OpenAI. Models overview: GPT-3.5 Turbo. Accessed 18 Oct 2024. https:\/\/platform.openai.com\/docs\/models#gpt-3-5-turbo"},{"key":"798_CR14","unstructured":"OpenAI. GPT-4 Technical Report. 2023. March 27. https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"798_CR15","unstructured":"IBM. Granite Foundation Models. presented at: IBM Research; May 31 2024; Virtual."},{"key":"798_CR16","unstructured":"Touvron H, Martin L, Stone K, et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. 2023. July 19. https:\/\/arxiv.org\/abs\/2307.09288"},{"key":"798_CR17","unstructured":"Bayarri-Planas J, Gururajan AK, Garcia-Gasulla D. Boosting Healthcare LLMs Through Retrieved Context. 2024. September 23. https:\/\/arxiv.org\/html\/2409.15127v1"},{"key":"798_CR18","unstructured":"An Z, Ding X, Fu Y-C, Chu C-C, Li Y, Du W. Golden-Retriever: High-Fidelity Agentic Retrieval Augmented Generation for Industrial Knowledge Base. 2024. https:\/\/arxiv.org\/html\/2408.00798v1"},{"issue":"2","key":"798_CR19","doi-asserted-by":"publisher","first-page":"207","DOI":"10.2307\/2529972","volume":"36","author":"HC Kraemer","year":"1980","unstructured":"Kraemer HC. Extension of the kappa coefficient. Biometrics. 1980;36(2):207\u201316.","journal-title":"Biometrics"},{"issue":"2","key":"798_CR20","first-page":"119","volume":"52","author":"A Agresti","year":"1998","unstructured":"Agresti A, Coull BA. Approximate is better than \u201cExact\u201d for interval estimation of binomial proportions. Am Stat. 1998;52(2):119\u201326.","journal-title":"Am Stat"},{"key":"798_CR21","doi-asserted-by":"crossref","unstructured":"Reimers N, Gurevych I. Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks. presented at: Conference on Empirical Methods in Natural Language Processing; 2019; Hong Kong, China.","DOI":"10.18653\/v1\/D19-1410"},{"issue":"1","key":"798_CR22","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","volume":"33","author":"JR Landis","year":"1977","unstructured":"Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159\u201374.","journal-title":"Biometrics"},{"key":"798_CR23","unstructured":"U.S. Food and Drug Administration. Guidance for Industry: Statistical Approaches to Establishing Bioequivalence. 2001; https:\/\/www.fda.gov\/regulatory-information\/search-fda-guidance-documents\/statistical-approaches-establishing-bioequivalence"},{"key":"798_CR24","unstructured":"U.S. Food and Drug Administration. Draft Guidance for Review Staff and Industry: Good Review Management Principles and Practices for PDUFA Products. 2005; https:\/\/www.fda.gov\/regulatory-information\/search-fda-guidance-documents\/guidance-review-staff-and-industry-good-review-management-principles-and-practices-pdufa-products"},{"key":"798_CR25","unstructured":"AI@Meta. The Llama 3 Herd of Models. 2024. July 23. https:\/\/arxiv.org\/abs\/2407.21783"},{"key":"798_CR26","unstructured":"OpenAI. Hello GPT-4o. Accessed 30 Aug 2024. https:\/\/openai.com\/index\/hello-gpt-4o\/"}],"container-title":["Therapeutic Innovation &amp; Regulatory Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s43441-025-00798-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s43441-025-00798-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s43441-025-00798-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,18]],"date-time":"2025-09-18T20:54:10Z","timestamp":1758228850000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s43441-025-00798-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,14]]},"references-count":26,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,9]]}},"alternative-id":["798"],"URL":"https:\/\/doi.org\/10.1007\/s43441-025-00798-8","relation":{"references":[{"id-type":"doi","id":"10.2139\/ssrn.3403301","asserted-by":"subject"}]},"ISSN":["2168-4790","2168-4804"],"issn-type":[{"value":"2168-4790","type":"print"},{"value":"2168-4804","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,14]]},"assertion":[{"value":"3 December 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 May 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 June 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Sarah Rodgers, Omar Amer, Matthew Deady, and Mohamed Ahmed were employees of IBM at the time of this work, and this company also created the RAG application Golden Retriever that was used during this research.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of interest"}},{"value":"The views expressed in this report are those of the authors and do not necessarily reflect the opinions of their institutions or the official policy of the Food and Drug Administration. No official support or endorsement by the FDA is intended or should be inferred.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Disclosures"}}]}}