{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,20]],"date-time":"2026-10-20T09:48:54Z","timestamp":1792489734684,"version":"4.5.0"},"reference-count":14,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Cancer"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Electronic health records contain inconsistently structured or free-text data, requiring efficient preprocessing to enable predictive health care models. Although artificial intelligence\u2013driven natural language processing tools show promise for automating diagnosis classification, their comparative performance and clinical reliability require systematic evaluation.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n The aim of this study is to evaluate the performance of 4 large language models (GPT-3.5, GPT-4o, Llama 3.2, and Gemini 1.5) and BioBERT in classifying cancer diagnoses from structured and unstructured electronic health records data.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We analyzed 762 unique diagnoses (326 International Classification of Diseases [ICD] code descriptions, 436 free-text entries) from 3456 records of patients with cancer. Models were tested on their ability to categorize diagnoses into 14 predefined categories. Two oncology experts validated classifications.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n BioBERT achieved the highest weighted macro\n F<\/jats:italic>\n 1<\/jats:sub>\n -score for ICD codes (84.2) and matched GPT-4o in ICD code accuracy (90.8). For free-text diagnoses, GPT-4o outperformed BioBERT in weighted macro\n F<\/jats:italic>\n 1<\/jats:sub>\n -score (71.8 vs 61.5) and achieved slightly higher accuracy (81.9 vs 81.6). GPT-3.5, Gemini, and Llama showed lower overall performance on both formats. Common misclassification patterns included confusion between metastasis and central nervous system tumors, as well as errors involving ambiguous or overlapping clinical terminology.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Although current performance levels appear sufficient for administrative and research use, reliable clinical applications will require standardized documentation practices alongside robust human oversight for high-stakes decision-making.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/72005","type":"journal-article","created":{"date-parts":[[2025,10,2]],"date-time":"2025-10-02T19:11:41Z","timestamp":1759432301000},"page":"e72005-e72005","source":"Crossref","is-referenced-by-count":5,"title":["Cancer Diagnosis Categorization in Electronic Health Records Using Large Language Models and BioBERT: Model Performance Evaluation Study"],"prefix":"10.2196","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7750-6294","authenticated-orcid":false,"given":"Soheil","family":"Hashtarkhani","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1343-5364","authenticated-orcid":false,"given":"Rezaur","family":"Rashid","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2958-599X","authenticated-orcid":false,"given":"Christopher L","family":"Brett","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1258-8353","authenticated-orcid":false,"given":"Lokesh","family":"Chinthala","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6700-4810","authenticated-orcid":false,"given":"Fekede Asefa","family":"Kumsa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0261-4917","authenticated-orcid":false,"given":"Janet A","family":"Zink","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8807-0019","authenticated-orcid":false,"given":"Robert L","family":"Davis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7235-5586","authenticated-orcid":false,"given":"David L","family":"Schwartz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2047-4759","authenticated-orcid":false,"given":"Arash","family":"Shaban-Nejad","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,10,2]]},"reference":[{"issue":"6","key":"R1","doi-asserted-by":"publisher","first-page":"395","DOI":"10.1038\/nrg3208","article-title":"Mining electronic health records: towards better research applications and clinical care","volume":"13","author":"Jensen","journal-title":"Nat Rev Genet"},{"issue":"7","key":"R2","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2021.14723","article-title":"Performance of a machine learning algorithm using electronic health record data to identify and estimate survival in a longitudinal cohort of patients with lung cancer","volume":"4","author":"Yuan","journal-title":"JAMA Netw Open"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.3233\/978-1-61499-564-7-40","article-title":"Using EHRs and machine learning for heart failure survival analysis","volume":"216","author":"Panahiazar","journal-title":"Stud Health Technol Inform"},{"issue":"5","key":"R4","doi-asserted-by":"publisher","first-page":"1007","DOI":"10.1093\/jamia\/ocv180","article-title":"Extracting information from the text of electronic medical records to improve case detection: a systematic review","volume":"23","author":"Ford","journal-title":"J Am Med Inform Assoc"},{"issue":"24","key":"R5","doi-asserted-by":"publisher","first-page":"2497","DOI":"10.1177\/15353702241226801","article-title":"Creative and generative artificial intelligence for personalized medicine and healthcare: hype, reality, or hyperreality?","volume":"248","author":"Shaban-Nejad","journal-title":"Exp Biol Med (Maywood)"},{"issue":"6","key":"R6","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1109\/MNET.2024.3367788","article-title":"A comprehensive overview of backdoor attacks in large language models within communication networks","volume":"38","author":"Yang","journal-title":"IEEE Netw"},{"issue":"1","key":"R7","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-024-02459-6","article-title":"Assessing the research landscape and clinical utility of large language models: a scoping review","volume":"24","author":"Park","journal-title":"BMC Med Inform Decis Mak"},{"issue":"5","key":"R8","doi-asserted-by":"publisher","first-page":"783","DOI":"10.1016\/j.jpedsurg.2024.01.033","article-title":"Using artificial intelligence to label free-text operative and ultrasound reports for grading pediatric appendicitis","volume":"59","author":"Abu-Ashour","journal-title":"J Pediatr Surg"},{"issue":"4","key":"R9","doi-asserted-by":"publisher","DOI":"10.2196\/medinform.5544","article-title":"Natural language processing-enabled and conventional data capture methods for input to electronic health records: a comparative usability study","volume":"4","author":"Kaufman","journal-title":"JMIR Med Inform"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3119621","article-title":"Machine learning techniques for biomedical natural language processing: a comprehensive review","volume":"9","author":"Houssein","journal-title":"IEEE Access"},{"issue":"2","key":"R11","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooac045","article-title":"Deep learning-based NLP data pipeline for EHR-scanned document information extraction","volume":"5","author":"Hsu","journal-title":"JAMIA Open"},{"issue":"4","key":"R12","doi-asserted-by":"publisher","first-page":"940","DOI":"10.1093\/jamia\/ocad256","article-title":"Evaluating the ChatGPT family of models for biomedical reasoning and classification","volume":"31","author":"Chen","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"R13","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbad493","article-title":"Opportunities and challenges for ChatGPT and large language models in biomedicine and health","volume":"25","author":"Tian","journal-title":"Brief Bioinformatics"},{"key":"R14","unstructured":"Hashtarkhani\/cancer_classification. GitHub. URL: https:\/\/github.com\/hashtarkhani\/Cancer_Classification [Accessed 29-09-2025]"}],"container-title":["JMIR Cancer"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/cancer.jmir.org\/2025\/1\/e72005","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:02:01Z","timestamp":1782842521000},"score":1,"resource":{"primary":{"URL":"https:\/\/cancer.jmir.org\/2025\/1\/e72005"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,2]]},"references-count":14,"URL":"https:\/\/doi.org\/10.2196\/72005","relation":{},"ISSN":["2369-1999"],"issn-type":[{"value":"2369-1999","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,2]]},"article-number":"v11i6e72005"}}