{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"Research Square"}],"indexed":{"date-parts":[[2026,10,12]],"date-time":"2026-10-12T16:49:21Z","timestamp":1791823761977,"version":"4.3.4"},"posted":{"date-parts":[[2026,2,9]]},"group-title":"In Review","reference-count":24,"publisher":"Springer Science and Business Media LLC","license":[{"start":{"date-parts":[[2026,2,9]],"date-time":"2026-02-09T00:00:00Z","timestamp":1770595200000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2026,1,31]]},"abstract":"<title>Abstract<\/title>\n                <p>Healthcare data exchange increasingly relies on HL7 FHIR, but FHIR's implementation complexity creates barriers for clinical workflows. Large language model (LLM) agents could bridge this gap by translating natural language requests into structured FHIR operations, yet their reliability remains unproven. We present FHIR-AgentEval, an extensible evaluation sandbox comprising 43 modular tasks for benchmarking LLM agents on realistic appointment management and genetic testing workflows. Each task executes against a resettable FHIR server with custom deterministic validation of both agent responses and resulting server state. We run an ablation study of five agent configurations, varying access to an on-demand FHIR R4 specifications server and long-term memory trained with or without specification grounding. Across four experimental settings, memory consistently improves task success and reduces strategic failures such as incorrect tool selection and resource-type confusion. On held-out tasks, the best memory configuration improves success by 9.1% over baseline, offering a potential pathway toward more robust clinical deployment.<\/p>","DOI":"10.21203\/rs.3.rs-8746188\/v1","type":"posted-content","created":{"date-parts":[[2026,2,9]],"date-time":"2026-02-09T11:07:11Z","timestamp":1770635231000},"source":"Crossref","is-referenced-by-count":1,"title":["FHIR-AgentEval: A Modular Sandbox for Benchmarking Clinical LLM Agents with an Evaluation of Memory-Augmented Configurations"],"prefix":"10.21203","author":[{"given":"Youssef","family":"Mokssit","sequence":"first","affiliation":[{"name":"Boston Children's Hospital"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kamalakkannan","family":"Ravi","sequence":"additional","affiliation":[{"name":"Boston Children's Hospital"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mengshu","family":"Nie","sequence":"additional","affiliation":[{"name":"Boston Children's Hospital"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Junyoung","family":"Kim","sequence":"additional","affiliation":[{"name":"Boston Children's Hospital"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cong","family":"Liu","sequence":"additional","affiliation":[{"name":"Boston Children's Hospital"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","reference":[{"key":"ref1","doi-asserted-by":"crossref","first-page":"1642","DOI":"10.1093\/jamia\/ocac105","article-title":"HL7 FHIR-based tools and initiatives to support clinical research: a scoping review","volume":"29","author":"Duda SN","year":"2022","unstructured":"Duda, S. N. et al. HL7 FHIR-based tools and initiatives to support clinical research: a scoping review. J. Am. Med. Inform. Assoc. JAMIA 29, 1642\u20131653 (2022).","journal-title":"J. Am. Med. Inform. Assoc. JAMIA"},{"key":"ref2","doi-asserted-by":"crossref","first-page":"e35724","DOI":"10.2196\/35724","article-title":"Fast Healthcare Interoperability Resources (FHIR) for Interoperability in Health Research: Systematic Review","volume":"10","author":"Vorisek CN","year":"2022","unstructured":"Vorisek, C. N. et al. Fast Healthcare Interoperability Resources (FHIR) for Interoperability in Health Research: Systematic Review. JMIR Med. Inform. 10, e35724 (2022).","journal-title":"JMIR Med. Inform."},{"key":"ref3","first-page":"52","article-title":"The Use of FHIR in Digital Health - A Review of the Scientific Literature","volume":"267","author":"Lehne M","year":"2019","unstructured":"Lehne, M., Luijten, S., Vom Felde Genannt Imbusch, P. & Thun, S. The Use of FHIR in Digital Health - A Review of the Scientific Literature. Stud. Health Technol. Inform. 267, 52\u201358 (2019).","journal-title":"Stud. Health Technol. Inform."},{"key":"ref4","first-page":"113","article-title":"Implementation of HL7 FHIR-Based Interoperability Profiles to Manage Care Plans for Multimorbid Patients with Mild Dementia","volume":"302","author":"Gencturk M","year":"2023","unstructured":"Gencturk, M., Akpinar, A. E. & Laleci Erturkmen, G. B. Implementation of HL7 FHIR-Based Interoperability Profiles to Manage Care Plans for Multimorbid Patients with Mild Dementia. Stud. Health Technol. Inform. 302, 113\u2013117 (2023).","journal-title":"Stud. Health Technol. Inform."},{"key":"ref5","doi-asserted-by":"crossref","first-page":"e58445","DOI":"10.2196\/58445","article-title":"State-of-the-Art Fast Healthcare Interoperability Resources (FHIR)-Based Data Model and Structure Implementations: Systematic Scoping Review","volume":"12","author":"Tabari P","year":"2024","unstructured":"Tabari, P., Costagliola, G., De Rosa, M. & Boeker, M. State-of-the-Art Fast Healthcare Interoperability Resources (FHIR)-Based Data Model and Structure Implementations: Systematic Scoping Review. JMIR Med. Inform. 12, e58445 (2024).","journal-title":"JMIR Med. Inform."},{"key":"ref6","author":"Ehtesham A","year":"2025","unstructured":"Ehtesham, A., Singh, A. & Kumar, S. Enhancing Clinical Decision Support and EHR Insights through LLMs and the Model Context Protocol: An Open-Source MCP-FHIR Framework. arXiv.org https:\/\/arxiv.org\/abs\/2506.13800v1 (2025)."},{"key":"ref7","doi-asserted-by":"crossref","first-page":"101780","DOI":"10.1016\/j.jacadv.2025.101780","article-title":"LLMonFHIR: A Physician-Validated, Large Language Model-Based Mobile Application for Querying Patient Electronic Health Data","volume":"4","author":"Schmiedmayer P","year":"2025","unstructured":"Schmiedmayer, P. et al. LLMonFHIR: A Physician-Validated, Large Language Model-Based Mobile Application for Querying Patient Electronic Health Data. JACC Adv. 4, 101780 (2025).","journal-title":"JACC Adv."},{"key":"ref8","article-title":"FHIR-GPT Enhances Health Interoperability with Large Language Models","volume":"1","author":"Li Y","year":"2024","unstructured":"Li, Y., Wang, H., Yerebakan, H. Z., Shinagawa, Y. & Luo, Y. FHIR-GPT Enhances Health Interoperability with Large Language Models. Nejm Ai 1, (2024).","journal-title":"Nejm Ai"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2508.20148","author":"Heydari AA","year":"2025","unstructured":"Heydari, A. A. et al. The Anatomy of a Personal Health Agent. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2508.20148 (2025)."},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2501.14654","article-title":"MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents","author":"Jiang Y","year":"2025","unstructured":"Jiang, Y. et al. MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2501.14654 (2025)."},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2210.03629","author":"Yao S","year":"2023","unstructured":"Yao, S. et al. ReAct: Synergizing Reasoning and Acting in Language Models. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2210.03629 (2023)."},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.11366","author":"Shinn N","year":"2023","unstructured":"Shinn, N. et al. Reflexion: Language Agents with Verbal Reinforcement Learning. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2303.11366 (2023)."},{"key":"ref13","unstructured":"Anthropic. Introducing the Model Context Protocol. https:\/\/www.anthropic.com\/news\/model-context-protocol."},{"key":"ref14","unstructured":"hapifhir\/hapi-fhir. HAPI FHIR (2026)."},{"key":"ref15","unstructured":"LangChain. create_openai_tools_agent \u2014 LangChain documentation. https:\/\/reference.langchain.com\/v0.3\/python\/langchain\/agents\/langchain.agents.openai_tools.base.create_openai_tools_agent.html."},{"key":"ref16","unstructured":"GPT-4.1 Model | OpenAI API. https:\/\/platform.openai.com."},{"key":"ref17","unstructured":"HL7 International. hl7.fhir.r4.core. FHIR Package Registry https:\/\/registry.fhir.org\/package\/hl7.fhir.r4.core%7C4.0.1."},{"key":"ref18","unstructured":"GPT-4.1 mini Model | OpenAI API. https:\/\/platform.openai.com."},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2601.03236","article-title":"A Multi-Graph based Agentic Memory Architecture for AI Agents","author":"Jiang D","year":"2026","unstructured":"Jiang, D., Li, Y., Li, G. & Li, B. MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agents. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2601.03236 (2026).","journal-title":"Preprint at"},{"key":"ref20","unstructured":"o4-mini Model | OpenAI API. https:\/\/platform.openai.com."},{"key":"ref21","author":"Faiss","unstructured":"Faiss. https:\/\/ai.meta.com\/tools\/faiss."},{"key":"ref22","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2509.19319","author":"Lee G","year":"2025","unstructured":"Lee, G. et al. FHIR-AgentBench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answering. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2509.19319 (2025)."},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.acl-long.663","author":"Liao Y","year":"2025","unstructured":"Liao, Y., Jiang, S., Wang, Y. & Wang, Y. ReflecTool: Towards Reflection-Aware Tool-Augmented Clinical Agents. in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) 13507\u201313531 (Association for Computational Linguistics, Vienna, Austria, 2025). doi:10.18653\/v1\/2025.acl-long.663."},{"key":"ref24","author":"Yang A","year":"2024","unstructured":"Yang, A. et al. Qwen2 Technical Report. arXiv.org https:\/\/arxiv.org\/abs\/2407.10671v4 (2024)."}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/www.researchsquare.com\/article\/rs-8746188\/v1","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.researchsquare.com\/article\/rs-8746188\/v1.html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,12]],"date-time":"2026-10-12T16:04:16Z","timestamp":1791821056000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.researchsquare.com\/article\/rs-8746188\/v1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,9]]},"references-count":24,"URL":"https:\/\/doi.org\/10.21203\/rs.3.rs-8746188\/v1","relation":{},"subject":[],"published":{"date-parts":[[2026,2,9]]},"subtype":"preprint"}}