{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T07:15:43Z","timestamp":1781766943800,"version":"3.54.5"},"reference-count":35,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T00:00:00Z","timestamp":1740096000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Generating accurate SQL queries from natural language is critical for enabling non-experts to interact with complex databases, particularly in high-stakes domains like healthcare. This paper presents an extensive evaluation of state-of-the-art large language models (LLM), including LLaMA 3.3, Mixtral, Gemini, Claude 3.5, GPT-4o, and Qwen for transforming medical questions into executable SQL queries using the MIMIC-3 and TREQS datasets. Our approach employs LLMs with various prompts across 1000 natural language questions. The experiments are repeated multiple times to assess performance consistency, token efficiency, and cost-effectiveness. We explore the impact of prompt design on model accuracy through an ablation study, focusing on the role of table data samples and one-shot learning examples. The results highlight substantial trade-offs between accuracy, consistency, and computational cost between the models. This study also underscores the limitations of current models in handling medical terminology and provides insights to improve SQL query generation in the healthcare domain. Future directions include implementing RAG pipelines based on embeddings and reranking models, integrating ICD taxonomies, and refining evaluation metrics for medical query performance. By bridging these gaps, language models can become reliable tools for medical database interaction, enhancing accessibility and decision-making in clinical settings.<\/jats:p>","DOI":"10.3390\/a18030124","type":"journal-article","created":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T10:12:11Z","timestamp":1740132731000},"page":"124","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Transforming Medical Data Access: The Role and Challenges of Recent Language Models in SQL Query Automation"],"prefix":"10.3390","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3936-1244","authenticated-orcid":false,"given":"Nikola","family":"Tankovi\u0107","sequence":"first","affiliation":[{"name":"Faculty of Informatics, Juraj Dobrila University of Pula, 52100 Pula, Croatia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert","family":"\u0160ajina","sequence":"additional","affiliation":[{"name":"Faculty of Informatics, Juraj Dobrila University of Pula, 52100 Pula, Croatia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ivan","family":"Lorencin","sequence":"additional","affiliation":[{"name":"Faculty of Informatics, Juraj Dobrila University of Pula, 52100 Pula, Croatia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,2,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1477","DOI":"10.1056\/NEJMp0901592","article-title":"Stimulating the adoption of health information technology","volume":"360","author":"Blumenthal","year":"2009","journal-title":"N. Engl. J. Med."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"113137","DOI":"10.1016\/j.dss.2019.113137","article-title":"Electronic health records in IS research: Quality issues, essential thresholds and remedial actions","volume":"126","author":"Jetley","year":"2019","journal-title":"Decis. Support Syst."},{"key":"ref_3","unstructured":"Huang, K., Altosaar, J., and Ranganath, R. (2020). ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission. arXiv."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Alsentzer, E., Murphy, J.R., Boag, W., Weng, W.H., Jin, D., Naumann, T., and McDermott, M.B.A. (2019). Publicly Available Clinical BERT Embeddings. arXiv.","DOI":"10.18653\/v1\/W19-1909"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"ref_6","unstructured":"Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). Language models are few-shot learners. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Ayaz, M., Pasha, M.F., Alzahrani, M.Y., Budiarto, R., and Stiawan, D. (2021). The Fast Health Interoperability Resources (FHIR) Standard: Systematic Literature Review of Implementations, Applications, Challenges and Opportunities. JMIR Med. Inform., 9, Erratum in JMIR Med. Inform. 2021, 9, e32869.","DOI":"10.2196\/32869"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"22","DOI":"10.22215\/timreview\/1111","article-title":"Hitching Healthcare to the Chain: An Introduction to Blockchain Technology in the Healthcare Sector","volume":"7","author":"Engelhardt","year":"2017","journal-title":"Technol. Innov. Manag. Rev."},{"key":"ref_9","unstructured":"European Union (2018). General Data Protection Regulation (GDPR)."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"836","DOI":"10.1016\/j.mayocp.2016.05.007","article-title":"Relationship Between Clerical Burden and Characteristics of the Electronic Environment With Physician Burnout and Professional Satisfaction","volume":"91","author":"Shanafelt","year":"2016","journal-title":"Mayo Clin. Proc."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Frost, M.J., Tran, J.B., Khatun, F., Friberg, I.K., and Rodr\u00edguez, D.C. (2018). What Does It Take to Be an Effective National Steward of Digital Health Integration for Health Systems Strengthening in Low- and Middle-Income Countries?. Glob. Health Sci. Pract., 6.","DOI":"10.9745\/GHSP-D-18-00270"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Wang, J., Deng, H., Liu, B., Hu, A., Liang, J., Fan, L., Zheng, X., Wang, T., and Lei, J. (2020). Systematic Evaluation of Research Progress on Natural Language Processing in Medicine Over the Past 20 Years: Bibliometric Study on PubMed. J. Med. Internet Res., 22.","DOI":"10.2196\/16816"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Au Yeung, J., Shek, A., Searle, T., Kraljevic, Z., Dinu, V., Ratas, M., Al-Agil, M., Foy, A., Rafferty, B., and Oliynyk, V. (2024). Natural language processing data services for healthcare providers. BMC Med. Inform. Decis. Mak., 24.","DOI":"10.1186\/s12911-024-02713-x"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Marshan, A., AlMutairi, A.N., Ioannou, A., Bell, D., Monaghan, A., and Arzoky, M. (2024). MedT5SQL: A transformers-based large language model for text-to-SQL conversion in the healthcare domain. Front. Big Data, 7.","DOI":"10.3389\/fdata.2024.1371680"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"6699","DOI":"10.1109\/TKDE.2024.3400824","article-title":"Natural Language Interfaces for Tabular Data Querying and Visualization: A Survey","volume":"36","author":"Zhang","year":"2023","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"73","DOI":"10.14778\/2735461.2735468","article-title":"Constructing an Interactive Natural Language Interface for Relational Databases","volume":"8","author":"Li","year":"2014","journal-title":"Proc. VLDB Endow."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"905","DOI":"10.1007\/s00778-022-00776-8","article-title":"A survey on deep learning approaches for text-to-SQL","volume":"32","author":"Koutrika","year":"2023","journal-title":"VLDB J."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Iacob, R., Brad, F., Apostol, E., Truic\u0103, C.O., Hosu, I.A., and Rebedea, T. (2020, January 8\u201313). Neural Approaches for Natural Language Interfaces to Databases: A Survey. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain.","DOI":"10.18653\/v1\/2020.coling-main.34"},{"key":"ref_19","unstructured":"Qin, B., Hui, B., Wang, L., Yang, M., Li, J., Li, B., Geng, R., Cao, R., Sun, J., and Si, L. (2022). A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Jin, Y., Chandra, M., Verma, G., Hu, Y., De Choudhury, M., and Kumar, S. (2024, January 13\u201317). Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries. Proceedings of the ACM Web Conference 2024, Singapore.","DOI":"10.1145\/3589334.3645643"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: A pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"ref_22","unstructured":"Zhu, X., Li, Q., Cui, L., and Liu, Y. (2024). Large Language Model Enhanced Text-to-SQL Generation: A Survey. arXiv."},{"key":"ref_23","unstructured":"Liu, X., Shen, S., Li, B., Ma, P., Jiang, R., Zhang, Y., Fan, J., Li, G., Tang, N., and Luo, Y. (2024). A Survey of NL2SQL with Large Language Models: Where are we, and where are we going?. arXiv."},{"key":"ref_24","unstructured":"Hong, Z., Yuan, Z., Zhang, Q., Chen, H., Dong, J., Huang, F., and Huang, X. (2024). Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1038\/s44401-024-00004-1","article-title":"Retrieval-augmented generation for generative artificial intelligence in health care","volume":"2","author":"Yang","year":"2025","journal-title":"npj Health Syst."},{"key":"ref_26","unstructured":"Yang, E., Amar, J., Lee, J.H., Kumar, B., and Jia, Y. (2024). The Geometry of Queries: Query-Based Innovations in Retrieval-Augmented Generation. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"e1312","DOI":"10.1002\/widm.1312","article-title":"Causability and Explainability of AI in Medicine","volume":"9","author":"Holzinger","year":"2019","journal-title":"Wiley Interdiscip. Rev. Data Min. Knowl. Discov."},{"key":"ref_28","first-page":"5998","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"ref_29","unstructured":"Meta-AI (2024, December 11). LLaMA 3.3-70B. Available online: https:\/\/www.llama.com\/."},{"key":"ref_30","unstructured":"Mistral AI (2024, December 11). Mixtral 8x22B: Mixture-of-Experts Model. Available online: https:\/\/mistral.ai\/news\/mixtral-8x22b."},{"key":"ref_31","unstructured":"Google DeepMind (2024, December 11). Gemini Pro. Available online: https:\/\/deepmind.google\/technologies\/gemini\/pro\/."},{"key":"ref_32","unstructured":"Anthropic (2024, December 11). Claude 3.5. Available online: https:\/\/www.anthropic.com\/news\/claude-3-5-sonnet."},{"key":"ref_33","unstructured":"OpenAI (2024, December 11). GPT-4o. Available online: https:\/\/openai.com\/index\/hello-gpt-4o\/."},{"key":"ref_34","unstructured":"OpenAI (2024, December 11). GPT-4o-Mini: Advancing Cost-Efficient Intelligence. Available online: https:\/\/openai.com\/index\/gpt-4o-mini-advancing-cost-efficient-intelligence\/."},{"key":"ref_35","unstructured":"QwenLM (2024, December 11). Qwen2.5. Available online: https:\/\/github.com\/QwenLM\/Qwen2.5."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/3\/124\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T16:40:03Z","timestamp":1760028003000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/3\/124"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,21]]},"references-count":35,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,3]]}},"alternative-id":["a18030124"],"URL":"https:\/\/doi.org\/10.3390\/a18030124","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,21]]}}}