{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,4]],"date-time":"2026-09-04T18:12:35Z","timestamp":1788545555609,"version":"build-2803163510"},"reference-count":60,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Differentiating among liver disease entities such as autoimmune liver disease (AILD), drug-induced liver injury (DILI), and chronic hepatitis B (CHB) remains clinically challenging due to overlapping clinical manifestations and nonspecific laboratory findings. Conventional machine learning (ML) approaches rely mainly on structured laboratory data, whereas free-text clinical reports and other heterogeneous electronic medical record data are often underused. Large language models (LLMs) may provide a strategy for encoding heterogeneous clinical information, yet their usefulness for liver disease classification remains insufficiently evaluated.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to evaluate the usefulness of LLM-derived embeddings for clinical data mining in liver disease and to determine whether integrating these embeddings with laboratory variables improves classification across broad disease categories and closely related subtypes.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n We retrospectively analyzed electronic medical record data from 7543 patients with nonoverlapping liver disease etiologies treated at Beijing Youan Hospital, Capital Medical University, between 2010 and 2025. Three LLMs (Qwen3, Huatuo-o1, and II-Medical) generated semantic embeddings from standardized clinical text, combining free-text examination reports, and structured clinical observations. Performance was assessed in a 3-class etiological task (AILD, DILI, and CHB) and a 4-class task further subclassifying AILD into autoimmune hepatitis and primary biliary cholangitis. We compared embedding-only models, LLM-integrated ML models, and an ML-only baseline using the same structured variable set and preprocessing pipeline, with lightweight natural language processing encoders and zero-shot LLM reasoning as additional comparators. Models were developed using 5-fold cross-validation and evaluated on an internal holdout set using accuracy, macroaveraged precision, recall, and\n F<\/jats:italic>\n 1<\/jats:sub>\n -score.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n In the 3-class task, the LLM-integrated ML models achieved macro\n F<\/jats:italic>\n 1<\/jats:sub>\n -scores of 0.835\u20100.837, compared with 0.791 for the ML-only baseline, with corresponding accuracies of 0.925\u20100.929 versus 0.893. In the 4-class task, the LLM-integrated ML models achieved macro\n F<\/jats:italic>\n 1<\/jats:sub>\n -scores of 0.717\u20100.734, compared with 0.665 for the ML-only baseline, with corresponding accuracies of 0.920\u20100.922 versus 0.874. A temporal split sensitivity analysis using cases from 2010 to 2019 for training and cases from 2020 to 2025 for testing showed that the relative advantage of LLM-integrated ML models over the ML-only baseline was preserved. Direct zero-shot LLM reasoning and lightweight natural language processing encoders performed below the embedding-based integrated models.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n In this single-center retrospective cohort of patients with clear-cut, nonoverlapping liver disease etiologies, LLM-derived embeddings provided complementary information to structured laboratory variables for multiclass liver disease classification. The integrated framework showed improved internal validation performance compared with the ML-only model, particularly for non-CHB categories and fine-grained subtype discrimination. Because patients with overlapping liver disease etiologies were excluded, the reported performance may overestimate diagnostic accuracy in broader real-world clinical settings where overlapping syndromes are common. Multicenter external validation and prospective evaluation in more heterogeneous patient populations are needed before clinical implementation.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/92921","type":"journal-article","created":{"date-parts":[[2026,9,4]],"date-time":"2026-09-04T17:24:45Z","timestamp":1788542685000},"page":"e92921-e92921","source":"Crossref","is-referenced-by-count":0,"title":["Large Language Models for Heterogeneous Data Mining in Liver Disease: Framework Development and Retrospective Validation Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-8205-075X","authenticated-orcid":false,"given":"Haiping","family":"Zhang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-1132-6792","authenticated-orcid":false,"given":"Xinming","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2439-0221","authenticated-orcid":false,"given":"Kechi","family":"Fang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-4178-5542","authenticated-orcid":false,"given":"Yinxue","family":"Ma","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-8090-1177","authenticated-orcid":false,"given":"Lijuan","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5980-7675","authenticated-orcid":false,"given":"Huiping","family":"Yan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8950-1340","authenticated-orcid":false,"given":"Yanmin","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-9916-5354","authenticated-orcid":false,"given":"Yanhua","family":"Yu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2512-0223","authenticated-orcid":false,"given":"Jing","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,4]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.1093\/gastro\/goae069","article-title":"Current epidemiology of chronic liver disease","volume":"12","author":"Manikat","journal-title":"Gastroenterol Rep (Oxf)"},{"issue":"12","key":"R2","doi-asserted-by":"publisher","first-page":"2650","DOI":"10.1016\/j.cgh.2019.07.060","article-title":"Contemporary epidemiology of chronic liver disease and cirrhosis","volume":"18","author":"Moon","journal-title":"Clin Gastroenterol Hepatol"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1155\/2019\/9437043","article-title":"Autoimmune hepatitis-immunologically triggered liver pathogenesis-diagnostic and therapeutic strategies","volume":"2019","author":"Sucher","journal-title":"J Immunol Res"},{"issue":"30","key":"R4","doi-asserted-by":"publisher","first-page":"109028","DOI":"10.12998\/wjcc.v13.i30.109028","article-title":"Advancing the management of primary biliary cholangitis: from pathogenesis to emerging therapies","volume":"13","author":"Curto","journal-title":"World J Clin Cases"},{"issue":"6","key":"R5","doi-asserted-by":"publisher","first-page":"1924","DOI":"10.1053\/j.gastro.2008.09.011","article-title":"Causes, clinical features, and outcomes from a prospective study of drug-induced liver injury in the United States","volume":"135","author":"Chalasani","journal-title":"Gastroenterology"},{"issue":"6","key":"R6","doi-asserted-by":"publisher","DOI":"10.3390\/v13061169","article-title":"Novel antivirals in clinical development for chronic hepatitis B infection","volume":"13","author":"Mak","journal-title":"Viruses"},{"issue":"17","key":"R7","doi-asserted-by":"publisher","first-page":"1394","DOI":"10.21037\/atm-21-2264","article-title":"Wilson disease and the differential diagnosis of its hepatic manifestations: a narrative review of clinical, laboratory, and liver histological features","volume":"9","author":"Schroeder","journal-title":"Ann Transl Med"},{"issue":"4","key":"R8","doi-asserted-by":"publisher","first-page":"240","DOI":"10.1136\/flgastro-2016-100698","article-title":"Autoimmune liver disease: evaluating overlapping and cross-over presentations-a case-based discussion","volume":"7","author":"Corrigan","journal-title":"Frontline Gastroenterol"},{"issue":"10","key":"R9","doi-asserted-by":"publisher","DOI":"10.1016\/j.jhepr.2022.100560","article-title":"Differentiating between liver diseases by applying multiclass machine learning approaches to transcriptomics of liver tissue or blood-based samples","volume":"4","author":"Listopad","journal-title":"JHEP Rep"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","first-page":"12081","DOI":"10.1038\/s41598-024-63095-8","article-title":"A machine learning-based model analysis for serum markers of liver fibrosis in chronic hepatitis B patients","volume":"14","author":"Zhang","journal-title":"Sci Rep"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-024-02550-y","article-title":"Improved liver disease prediction from clinical data through an evaluation of ensemble learning approaches","volume":"24","author":"Ganie","journal-title":"BMC Med Inform Decis Mak"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","first-page":"10","DOI":"10.14366\/usg.22110","article-title":"Artificial intelligence models for the diagnosis and management of liver diseases","volume":"42","author":"Nishida","journal-title":"Ultrasonography"},{"issue":"9","key":"R13","doi-asserted-by":"publisher","first-page":"101383","DOI":"10.3748\/wjg.v31.i9.101383","article-title":"Machine learning-based models for advanced fibrosis in non-alcoholic steatohepatitis patients: a cohort study","volume":"31","author":"Xiong","journal-title":"World J Gastroenterol"},{"issue":"1","key":"R14","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1136\/amiajnl-2011-000681","article-title":"Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research","volume":"20","author":"Weiskopf","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"R15","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0231300","article-title":"Feature engineering with clinical expert knowledge: a case study assessment of machine learning model complexity and performance","volume":"15","author":"Roe","journal-title":"PLoS One"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.3389\/fdgth.2022.940330","article-title":"Challenges and recommendations for high quality research using electronic health records","volume":"4","author":"Honeyford","journal-title":"Front Digit Health"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-019-0747-6","article-title":"Importance of medical data preprocessing in predictive modeling and risk factor discovery for the frailty syndrome","volume":"19","author":"Hassler","journal-title":"BMC Med Inform Decis Mak"},{"issue":"6","key":"R18","doi-asserted-by":"publisher","DOI":"10.3390\/bioengineering12060631","article-title":"Large language models in healthcare and medical applications: a review","volume":"12","author":"Maity","journal-title":"Bioengineering (Basel)"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","DOI":"10.1186\/s12929-025-01131-z","article-title":"Impact of large language model (ChatGPT) in healthcare: an umbrella review and evidence synthesis","volume":"32","author":"Iqbal","journal-title":"J Biomed Sci"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.2196\/69929","article-title":"Performance of ChatGPT-4o and four open-source large language models in generating diagnoses based on China\u2019s rare disease catalog: comparative study","volume":"27","author":"Zhong","journal-title":"J Med Internet Res"},{"issue":"1","key":"R21","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01869-8","article-title":"Enhancing EHR-based pancreatic cancer prediction with LLM-derived embeddings","volume":"8","author":"Park","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R22","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-025-02954-4","article-title":"A systematic review of large language model (LLM) evaluations in clinical medicine","volume":"25","author":"Shool","journal-title":"BMC Med Inform Decis Mak"},{"issue":"1","key":"R23","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1002\/hep.22322","article-title":"Simplified criteria for the diagnosis of autoimmune hepatitis","volume":"48","author":"Hennes","journal-title":"Hepatology"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1016\/j.jhep.2017.03.022","article-title":"EASL clinical practice guidelines: the diagnosis and management of patients with primary biliary cholangitis","volume":"67","author":"Hirschfield","journal-title":"J Hepatol"},{"issue":"3","key":"R25","doi-asserted-by":"publisher","first-page":"736","DOI":"10.14218\/JCTH.2022.00347","article-title":"Guidelines on the diagnosis and management of primary biliary cholangitis (2021)","volume":"11","author":"You","journal-title":"J Clin Transl Hepatol"},{"issue":"1","key":"R26","doi-asserted-by":"publisher","DOI":"10.3390\/ijms17010014","article-title":"RUCAM in drug and herb induced liver injury: the update","volume":"17","author":"Danan","journal-title":"Int J Mol Sci"},{"issue":"2","key":"R27","doi-asserted-by":"publisher","first-page":"384","DOI":"10.1007\/s12072-023-10633-7","article-title":"Chinese guideline for the diagnosis and treatment of drug-induced liver injury: an update","volume":"18","author":"Mao","journal-title":"Hepatol Int"},{"issue":"4","key":"R28","doi-asserted-by":"publisher","first-page":"1560","DOI":"10.1002\/hep.29800","article-title":"Update on prevention, diagnosis, and treatment of chronic hepatitis B: AASLD 2018 hepatitis B guidance","volume":"67","author":"Terrault","journal-title":"Hepatology"},{"issue":"6","key":"R29","doi-asserted-by":"publisher","first-page":"1425","DOI":"10.14218\/JCTH.2023.00320","article-title":"Guidelines for the prevention and treatment of chronic hepatitis B (version 2022)","volume":"11","author":"You","journal-title":"J Clin Transl Hepatol"},{"issue":"1","key":"R30","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-015-0088-9","article-title":"A measure of the impact of CV incompleteness on prediction error estimation with application to PCA and normalization","volume":"15","author":"Hornung","journal-title":"BMC Med Res Methodol"},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2015.04.005","article-title":"Prediction models need appropriate internal, internal-external, and external validation","volume":"69","author":"Steyerberg","journal-title":"J Clin Epidemiol"},{"key":"R32","unstructured":"Chen J Cai Z Ji K et al. HuatuoGPT-o1, towards medical complex reasoning with LLMs. arXiv. Preprint posted online on Dec 25, 2024. [doi: 10.48550\/arXiv.2412.18925]"},{"key":"R33","unstructured":"Intelligent internet: II-medical-8B: medical reasoning model. Hugging Face. 2025. URL: https:\/\/huggingface.co\/Intelligent-Internet\/II-Medical-8B [Accessed 10-08-2026]"},{"key":"R34","unstructured":"Yang A Li A Yang B et al. Qwen3 technical report. arXiv. Preprint posted online on May 14, 2025. [doi: 10.48550\/arXiv.2505.09388]"},{"issue":"5","key":"R35","doi-asserted-by":"publisher","DOI":"10.1007\/s42979-021-00807-1","article-title":"Specialists, scientists, and sentiments: word2Vec and doc2Vec in analysis of scientific and medical texts","volume":"2","author":"Chen","journal-title":"SN Comput Sci"},{"key":"R36","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2025.1508455","article-title":"Early prediction of colorectal adenoma risk: leveraging large-language model for clinical electronic medical record data","volume":"15","author":"Yang","journal-title":"Front Oncol"},{"issue":"12","key":"R37","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2025.50454","article-title":"Diagnostic codes in AI prediction models and label leakage of same-admission clinical outcomes","volume":"8","author":"Ramadan","journal-title":"JAMA Netw Open"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2025.1658316","article-title":"Prompt engineering for accurate statistical reasoning with large language models in medical research","volume":"8","author":"Vilakati","journal-title":"Front Artif Intell"},{"issue":"6","key":"R39","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2025.101260","article-title":"Unleashing the potential of prompt engineering for large language models","volume":"6","author":"Chen","journal-title":"Patterns (N Y)"},{"key":"R40","doi-asserted-by":"publisher","DOI":"10.2196\/55318","article-title":"An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing: algorithm development and validation study","volume":"12","author":"Sivarajkumar","journal-title":"JMIR Med Inform"},{"issue":"18","key":"R41","doi-asserted-by":"publisher","first-page":"2283","DOI":"10.3390\/healthcare13182283","article-title":"Concordance of an artificial intelligence model (ChatGPT 4.0) with physician decisions in smoking cessation clinics: a comparative evaluation","volume":"13","author":"Gokseven Arda","journal-title":"Health Care (Don Mills)"},{"issue":"1","key":"R42","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-026-02539-z","article-title":"ClinicRealm: re-evaluating large language models with conventional machine learning for non-generative clinical prediction tasks","volume":"9","author":"Zhu","journal-title":"NPJ Digit Med"},{"issue":"9","key":"R43","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2023.100804","article-title":"Leakage and the reproducibility crisis in machine-learning-based science","volume":"4","author":"Kapoor","journal-title":"Patterns (N Y)"},{"issue":"1","key":"R44","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1177\/1536867X20909688","article-title":"The random forest algorithm for statistical learning","volume":"20","author":"Schonlau","journal-title":"Stata J"},{"issue":"6088","key":"R45","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1038\/323533a0","article-title":"Learning representations by back-propagating errors","volume":"323","author":"Rumelhart","journal-title":"Nature"},{"issue":"1","key":"R46","doi-asserted-by":"publisher","first-page":"112","DOI":"10.1186\/cc3045","article-title":"Statistics review 14: logistic regression","volume":"9","author":"Bewick","journal-title":"Crit Care"},{"key":"R47","doi-asserted-by":"crossref","unstructured":"Chen T Guestrin C . XGBoost: a scalable tree boosting system. Presented at: The 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; Aug 13-17, 2016:785-794; San Francisco, CA. 2016.[doi: 10.1145\/2939672.2939785]","DOI":"10.1145\/2939672.2939785"},{"issue":"3","key":"R48","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1007\/BF00994018","article-title":"Support-vector networks","volume":"20","author":"Cortes","journal-title":"Mach Learn"},{"key":"R49","doi-asserted-by":"publisher","DOI":"10.12688\/f1000research.169456.2","article-title":"A differential evolution-based optimized ensemble for balanced and imbalanced medical datasets","volume":"14","author":"Das","journal-title":"F1000Res"},{"issue":"5","key":"R50","doi-asserted-by":"publisher","first-page":"632","DOI":"10.1007\/s00535-025-02221-3","article-title":"A clinicopathological study of IgG4-related autoimmune hepatitis and IgG4-hepatopathy","volume":"60","author":"Tanaka","journal-title":"J Gastroenterol"},{"issue":"1","key":"R51","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1016\/j.cld.2023.07.002","article-title":"Autoimmune markers in primary biliary cholangitis","volume":"28","author":"Shah","journal-title":"Clin Liver Dis"},{"issue":"2","key":"R52","doi-asserted-by":"publisher","first-page":"474","DOI":"10.3350\/cmh.2024.0416","article-title":"Prediction of primary biliary cholangitis among health check-up population with anti-mitochondrial M2 antibody positive","volume":"31","author":"Li","journal-title":"Clin Mol Hepatol"},{"issue":"8","key":"R53","doi-asserted-by":"publisher","DOI":"10.3390\/v14081691","article-title":"HBeAg-negative\/anti-HBe-positive chronic hepatitis B: a 40-year-old history","volume":"14","author":"Bonino","journal-title":"Viruses"},{"issue":"10","key":"R54","doi-asserted-by":"publisher","DOI":"10.1016\/j.jhepr.2025.101470","article-title":"Autoantibodies are associated with worse outcomes in MASLD","volume":"7","author":"Soria","journal-title":"JHEP Rep"},{"issue":"2","key":"R55","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1007\/s12016-021-08888-9","article-title":"Autoimmune hepatitis: serum autoantibodies in clinical practice","volume":"63","author":"Terziroli Beretta-Piccoli","journal-title":"Clin Rev Allergy Immunol"},{"issue":"1","key":"R56","doi-asserted-by":"publisher","first-page":"1550","DOI":"10.1515\/med-2022-0526","article-title":"Noninvasive diagnosis of AIH\/PBC overlap syndrome based on prediction models","volume":"17","author":"Wang","journal-title":"Open Med"},{"key":"R57","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2022.3210347","article-title":"Machine learning model for hepatitis C diagnosis customized to each patient","volume":"10","author":"Chen","journal-title":"IEEE Access"},{"key":"R58","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpi.2025.100521","article-title":"Leveraging large language models for structured information extraction from pathology reports","volume":"19","author":"Balasubramanian","journal-title":"J Pathol Inform"},{"issue":"1","key":"R59","doi-asserted-by":"publisher","DOI":"10.1136\/bmjgast-2025-001896","article-title":"Large language models for extracting histopathologic diagnoses of colorectal cancer and dysplasia from electronic health records","volume":"12","author":"Johnson","journal-title":"BMJ Open Gastroenterol"},{"key":"R60","unstructured":"Zhang H Li X Fang K et al. LLM-based-liverdisease-model: code repository for large language models for heterogeneous data mining in liver disease. GitHub. 2025. URL: https:\/\/github.com\/BioInfor-coder\/LLM-based-LiverDisease-Model [Accessed 12-08-2026]"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,4]],"date-time":"2026-09-04T17:24:47Z","timestamp":1788542687000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e92921"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,4]]},"references-count":60,"URL":"https:\/\/doi.org\/10.2196\/92921","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,4]]},"article-number":"v28i12e92921"}}