{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,13]],"date-time":"2026-10-13T18:17:04Z","timestamp":1791915424823,"version":"4.3.4"},"reference-count":51,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec sec-type=\"background\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Real-world data\u2013based feasibility assessments enhance clinical trial design, but automating eligibility criteria conversion to database queries is hindered by challenges related to ensuring high accuracy and generating clear, usable outputs.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"objective\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>The aim of this study is to develop an automated system converting free-text eligibility criteria from ClinicalTrials.gov into Observational Medical Outcomes Partnership Common Data Model (OMOP CDM)\u2013compatible Structured Query Language (SQL) queries and systematically evaluate hallucination patterns across multiple large language models (LLMs) to identify the optimal deployment strategies.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"methods\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Our system employs a three-stage preprocessing pipeline (segmentation, filtering, and simplification) achieving 58.2% token reduction while preserving clinical semantics. We compared GPT-4 concept mapping performance against USAGI using 357 clinical terms from 30 trials. For comprehensive evaluation, we analyzed 760 SQL generation attempts (19 trials\u00d78 LLMs\u00d75 prompting strategies) using the SynPUF (Synthetic Public Use Files) dataset and validated selected queries against National COVID Cohort Collaborative reference concept sets using Asan Medical Center\u2019s OMOP CDM database.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"results\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      GPT-4 achieved a 48.5% concept mapping accuracy versus USAGI\u2019s 32.0% (\n                      <jats:italic>P<\/jats:italic>\n                      &lt;.001), with domain-specific performance ranging from 72.7% (drug) to 38.3% (measurement). Surprisingly, the open-source llama3: 8b model achieved the highest effective SQL rate (75.8%) compared to GPT-4 (45.3%), attributed to lower hallucination rates (21.1% vs 33.7%). The overall hallucination rate was 32.7%, with wrong domain assignments (34.2%) and placeholder insertions (28.7%) being the most common. Clinical validation revealed mixed performance: high concordance for type 1 diabetes (Jaccard=0.81), complete failure for pregnancy (Jaccard=0.00), and minimal overlap for type 2 diabetes (Jaccard=0.03), despite perfect overlap coefficients in both diabetes cases. Moderate performance was observed for uncontrolled hypertension (Jaccard=0.18).\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"conclusions\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>While LLMs can accelerate eligibility criteria transformation, hallucination rates of 21\u201050% necessitate careful model selection and validation strategies. Our findings challenge assumptions about model superiority, demonstrating that smaller, cost-effective models can outperform larger commercial alternatives. Future work should focus on hybrid approaches combining LLM capabilities with rule-based methods for handling complex clinical concepts.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/71252","type":"journal-article","created":{"date-parts":[[2025,8,5]],"date-time":"2025-08-05T20:40:06Z","timestamp":1754426406000},"page":"e71252-e71252","source":"Crossref","is-referenced-by-count":13,"title":["Large Language Models for Automating Clinical Trial Criteria Conversion to Observational Medical Outcomes Partnership Common Data Model Queries: Validation and Evaluation Study"],"prefix":"10.2196","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7593-7020","authenticated-orcid":false,"given":"Kye Hwa","family":"Lee","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-2596-5444","authenticated-orcid":false,"given":"Sujung","family":"Jang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4610-4994","authenticated-orcid":false,"given":"Grace Juyun","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-3938-4618","authenticated-orcid":false,"given":"Sukyoung","family":"Park","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-4916-5209","authenticated-orcid":false,"given":"Doeun","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-4150-6630","authenticated-orcid":false,"given":"Oh Jin","family":"Kwon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2619-1231","authenticated-orcid":false,"given":"Jae-Ho","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3610-486X","authenticated-orcid":false,"given":"Young-Hak","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,10,16]]},"reference":[{"issue":"5","key":"R1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0127242","article-title":"Terminated trials in the ClinicalTrials.gov results database: evaluation of availability of primary outcome data and reasons for termination","volume":"10","author":"Williams","journal-title":"PLoS ONE"},{"issue":"3","key":"R2","doi-asserted-by":"publisher","DOI":"10.3390\/medsci12030039","article-title":"Successes and challenges in clinical trial recruitment: the experience of a new study team","volume":"12","author":"Hung","journal-title":"Med Sci (Basel)"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2023.104553","article-title":"Electronic health records (EHRs) in clinical research and platform trials: application of the innovative EHR-based methods developed by EU-PEARL","volume":"148","author":"Lombardo","journal-title":"J Biomed Inform"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1177\/1740774514558307","article-title":"Unsuccessful trial accrual and human subjects protections: an empirical analysis of recently closed trials","volume":"12","author":"Carlisle","journal-title":"Clin Trials"},{"issue":"4","key":"R5","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1001\/jama.2013.282542","article-title":"Scientific and regulatory reasons for delay and denial of FDA approval of initial applications for new drugs, 2000-2012","volume":"311","author":"Sacks","journal-title":"JAMA"},{"issue":"7","key":"R6","doi-asserted-by":"publisher","first-page":"3049","DOI":"10.1016\/j.apsb.2022.02.002","article-title":"Why 90% of clinical drug development fails and how to improve it?","volume":"12","author":"Sun","journal-title":"Acta Pharm Sin B"},{"issue":"3","key":"R7","doi-asserted-by":"crossref","first-page":"106","DOI":"10.4103\/2229-3485.71867","volume":"1","author":"Rajadhyaksha","journal-title":"Perspect Clin Res"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.1016\/j.conctc.2018.08.001","article-title":"Factors associated with clinical trials that fail and opportunities for improving the likelihood of success: a review","volume":"11","author":"Fogel","journal-title":"Contemp Clin Trials Commun"},{"issue":"10","key":"R9","doi-asserted-by":"publisher","first-page":"1201","DOI":"10.1002\/pds.4932","article-title":"Trial designs using real-world data: The changing landscape of the regulatory approval process","volume":"29","author":"Baumfeld Andre","journal-title":"Pharmacoepidemiol Drug Saf"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1002\/cpt.2480","article-title":"Use of real-world evidence to drive drug development strategy and inform clinical trial design","volume":"111","author":"Dagenais","journal-title":"Clin Pharmacol Ther"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.2147\/COPD.S244942","article-title":"Comparing randomized controlled trials and real-world studies in chronic obstructive pulmonary disease pharmacotherapy","volume":"15","author":"Tashkin","journal-title":"Int J Chron Obstruct Pulmon Dis"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1093\/jamia\/ocaa224","article-title":"Contemporary use of real-world data for clinical trial conduct in the United States: a scoping review","volume":"28","author":"Rogers","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"R13","doi-asserted-by":"publisher","first-page":"545","DOI":"10.1007\/s43441-020-00248-7","article-title":"Real-world data for planning eligibility criteria and enhancing recruitment: recommendations from the clinical trials transformation initiative","volume":"55","author":"Evans","journal-title":"Ther Innov Regul Sci"},{"issue":"2","key":"R14","doi-asserted-by":"publisher","first-page":"565","DOI":"10.1172\/JCI129197","article-title":"Opportunities and challenges in using real-world data for health care","volume":"130","author":"Rudrapatna","journal-title":"J Clin Invest"},{"issue":"7855","key":"R15","doi-asserted-by":"publisher","first-page":"629","DOI":"10.1038\/s41586-021-03430-5","article-title":"Evaluating eligibility criteria of oncology trials using real-world data and AI","volume":"592","author":"Liu","journal-title":"Nature New Biol"},{"key":"R16","unstructured":"Umapathi LK Pal A Sankarasubbu M . Med-HALT: medical domain hallucination test for large language models. arXiv. Preprint posted online on Jul 27, 2023. [doi: 10.48550\/arXiv.2307.15343]"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01670-7","article-title":"A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation","volume":"8","author":"Asgari","journal-title":"NPJ Digit Med"},{"key":"R18","unstructured":"OpenAI. GPT-4 technical report. arXiv. Preprint posted online on Mar 27, 2023. [doi: 10.48550\/arXiv.2303.08774]"},{"key":"R19","unstructured":"SynPUF dataset documentation. Centers for Medicare & Medicaid Services (CMS). URL: https:\/\/www.cms.gov [Accessed 10-07-2025]"},{"key":"R20","volume":"216","author":"Hripcsak","journal-title":"Stud Health Technol Inform"},{"key":"R21","unstructured":"Aggregate analysis of clinicaltrials.gov (AACT) database. Clinical Trials Transformation Initiative (CTTI). URL: https:\/\/aact.ctti-clinicaltrials.org\/ [Accessed 18-07-2023]"},{"key":"R22","unstructured":"Usagi: a tool for semi-automated mapping to standardized vocabularies. Observational Health Data Sciences and Informatics (OHDSI). GitHub repository; URL: https:\/\/github.com\/OHDSI\/Usagi [Accessed 27-07-2023]"},{"key":"R23","unstructured":"GPT-35 technical overview. OpenAI. 2023. URL: https:\/\/platform.openai.com\/docs [Accessed 10-07-2025]"},{"key":"R24","unstructured":"Anthropic. Claude 3 Technical Overview. 2024. URL: https:\/\/www.anthropic.com\/index\/claude-3 [Accessed 10-07-2025]"},{"key":"R25","unstructured":"Grattafiori A Dubey A Jauhri A et al. The llama 3 herd of models. arXiv. Preprint posted online on 2024. [doi: 10.48550\/arXiv:2407.21783]"},{"key":"R26","doi-asserted-by":"crossref","unstructured":"Guo X Yang D Zhang H et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. 2025;645:633-638. [doi: 10.1038\/s41586-025-09422-z]","DOI":"10.1038\/s41586-025-09422-z"},{"key":"R27","unstructured":"Yang A Yang B Zhang B . Qwen2.5 Technical Report. arXiv. Preprint posted online on 2024. [doi: 10.48550\/arXiv.2412.15115]"},{"key":"R28","unstructured":"Microsoft. Phi-3 technical report: a family of open language models. arXiv. Preprint posted online on Apr 22, 2024. [doi: 10.48550\/arXiv.2404.14219]"},{"key":"R29","unstructured":"Gemma: open models based on gemini research and technology. Google DeepMind. 2024. URL: https:\/\/ai.google.dev\/gemma [Accessed 10-07-2025]"},{"issue":"3","key":"R30","doi-asserted-by":"publisher","first-page":"352","DOI":"10.1038\/s41592-020-0772-5","article-title":"Author correction: SciPy 1.0: fundamental algorithms for scientific computing in Python","volume":"17","author":"Virtanen","journal-title":"Nat Methods"},{"key":"R31","doi-asserted-by":"crossref","unstructured":"Seabold S Perktold J . Statsmodels: econometric and statistical modeling with python. Presented at: Python in Science Conference; Jun 28, 2010; Austin, Texas. 2010.[doi: 10.25080\/Majora-92bf1922-011]","DOI":"10.25080\/Majora-92bf1922-011"},{"issue":"3","key":"R32","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1093\/jamia\/ocaa196","article-title":"The National COVID Cohort Collaborative (N3C): Rationale, design, infrastructure, and deployment","volume":"28","author":"Haendel","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"R33","doi-asserted-by":"publisher","first-page":"294","DOI":"10.1093\/jamia\/ocy178","article-title":"Criteria2Query: a natural language interface to clinical databases for cohort definition","volume":"26","author":"Yuan","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"R34","doi-asserted-by":"publisher","first-page":"230","DOI":"10.1093\/jamia\/ocx079","article-title":"Synthea: an approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record","volume":"25","author":"Walonoski","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"R35","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1136\/amiajnl-2011-000681","article-title":"Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research","volume":"20","author":"Weiskopf","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"R36","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000082","article-title":"Synthetic data in health care: a narrative review","volume":"2","author":"Gonzales","journal-title":"PLOS Digit Health"},{"issue":"1","key":"R37","doi-asserted-by":"publisher","DOI":"10.13063\/2327-9214.1244","article-title":"A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data","volume":"4","author":"Kahn","journal-title":"EGEMS (Wash DC)"},{"issue":"2","key":"R38","doi-asserted-by":"publisher","first-page":"214","DOI":"10.1002\/sim.6787","article-title":"Sample size considerations for the external validation of a multivariable prognostic model: a resampling study","volume":"35","author":"Collins","journal-title":"Stat Med"},{"issue":"9836","key":"R39","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1016\/S0140-6736(12)60240-2","article-title":"Epidemiology of multimorbidity and implications for health care, research, and medical education: a cross-sectional study","volume":"380","author":"Barnett","journal-title":"Lancet"},{"issue":"8","key":"R40","doi-asserted-by":"publisher","DOI":"10.2196\/33703","article-title":"Evaluation of the clinical, technical, and financial aspects of cost-effectiveness analysis of artificial intelligence in medicine: scoping review and framework of analysis","volume":"10","author":"Gomez Rossi","journal-title":"JMIR Med Inform"},{"key":"R41","unstructured":"OpenAI. URL: https:\/\/openai.com\/pricing [Accessed 10-07-2025]"},{"key":"R42","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2024.104649","article-title":"Criteria2Query 3.0: Leveraging generative large language models for clinical trial eligibility query generation","volume":"154","author":"Park","journal-title":"J Biomed Inform"},{"issue":"12","key":"R43","doi-asserted-by":"publisher","first-page":"1954","DOI":"10.1093\/jamia\/ocad149","article-title":"LeafAI: query generator for clinical cohort discovery rivaling a human programmer","volume":"30","author":"Dobbins","journal-title":"J Am Med Inform Assoc"},{"key":"R44","doi-asserted-by":"publisher","DOI":"10.1016\/j.imu.2022.100996","article-title":"Use of automatic SQL generation interface to enhance transparency and validity of health-data analysis","volume":"31","author":"Wagholikar","journal-title":"Inform Med Unlocked"},{"issue":"6","key":"R45","doi-asserted-by":"publisher","first-page":"788","DOI":"10.1016\/j.jpurol.2022.05.004","article-title":"Feasibility of establishing a multi-center research database using the electronic health record: the PURSUIT network","volume":"18","author":"Vemulakonda","journal-title":"J Pediatr Urol"},{"issue":"3","key":"R46","doi-asserted-by":"publisher","DOI":"10.2196\/14185","article-title":"A real-time automated patient screening system for clinical trials eligibility in an emergency department: design and evaluation","volume":"7","author":"Ni","journal-title":"JMIR Med Inform"},{"issue":"3","key":"R47","doi-asserted-by":"publisher","first-page":"244","DOI":"10.1093\/jnci\/djaa155","article-title":"\u201cWhen offered to participate\u201d: a systematic review and meta-analysis of patient agreement to participate in cancer clinical trials","volume":"113","author":"Unger","journal-title":"J Natl Cancer Inst"},{"issue":"1","key":"R48","doi-asserted-by":"publisher","first-page":"166","DOI":"10.1136\/amiajnl-2014-002887","article-title":"Automated clinical trial eligibility prescreening: increasing the efficiency of patient identification for clinical trials in the emergency department","volume":"22","author":"Ni","journal-title":"J Am Med Inform Assoc"},{"key":"R49","doi-asserted-by":"publisher","DOI":"10.1016\/j.gore.2025.101691","article-title":"Transforming clinical trial enrollment: leveraging technology and innovation to reach patients where they are","volume":"57","author":"Somasegar","journal-title":"Gynecol Oncol Rep"},{"issue":"2","key":"R50","doi-asserted-by":"publisher","first-page":"e16","DOI":"10.2105\/AJPH.2013.301706","article-title":"A systematic review of barriers and facilitators to minority research participation among African Americans, Latinos, Asian Americans, and Pacific Islanders","volume":"104","author":"George","journal-title":"Am J Public Health"},{"key":"R51","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-00323-1","article-title":"The future of digital health with federated learning","volume":"3","author":"Rieke","journal-title":"NPJ Digit Med"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2025\/1\/e71252","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:00:29Z","timestamp":1782842429000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2025\/1\/e71252"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,16]]},"references-count":51,"URL":"https:\/\/doi.org\/10.2196\/71252","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,16]]},"article-number":"v13i5e71252"}}