{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T23:10:37Z","timestamp":1790118637561,"version":"4.0.1"},"reference-count":51,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR AI"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Free-text clinical data are unstructured and narrative in nature, providing a rich source of patient information, but extracting research-quality clinical phenotypes from these data remains a challenge. Manually reviewing and extracting clinical phenotypes from free-text patient notes is a time-consuming process and not suitable for large-scale datasets. On the other hand, automatically extracting clinical phenotypes can be challenging because medical researchers lack gold-standard annotated references and other purpose-built resources, including software. Recent large language models (LLMs) can understand natural language instructions, which help them adapt to different domains and tasks without the need for specific training data. This makes them suitable for clinical applications, though their use in this field is limited.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n We aimed to develop an LLM pipeline based on the few-shot learning framework that could extract clinical information from free-text clinical summaries. We assessed the performance of this pipeline for classifying individuals with confirmed or suspected comorbid intellectual disability (ID) from clinical summaries of patients with severe mental illness and performed genetic validation of the results by testing whether individuals with LLM-defined ID carried more genetic variants known to confer risk of ID when compared with individuals without LLM-defined ID.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We developed novel approaches for performing classification, based on an intermediate information extraction (IE) step and human-in-the-loop techniques. We evaluated two models: Fine-Tuned Language Text-To-Text Transfer Transformer (Flan-T5) and Large Language Model Architecture (LLaMA). The dataset comprised 1144 free-text clinical summaries, of which 314 were manually annotated and used as a gold standard for evaluating automated methods. We also used published genetic data from 547 individuals to perform a genetic validation of the classification results; Firth\u2019s penalized logistic regression framework was used to test whether individuals with LLM-defined ID carry significantly more de novo variants in known developmental disorder risk genes than individuals without LLM-defined ID.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n The results demonstrate that a 2-stage approach, combining IE with manual validation, can effectively identify individuals with suspected IDs from free-text patient records, requiring only a single training example per classification label. The best-performing method based on the Flan-T5 model and incorporating the IE step achieved an\n F<\/jats:italic>\n 1<\/jats:sub>\n -score of 0.867. Individuals classified as having ID by the best performing model were significantly enriched for de novo variants in known developmental disorder risk genes (odds ratio 29.1, 95% CI 7.36-107;\n P<\/jats:italic>\n =2.1\u00d710\n \u22125<\/jats:sup>\n ).\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n LLMs and in-context learning techniques combined with human-in-the-loop approaches can be highly beneficial for extraction and categorization of information from free-text clinical data. In this proof-of-concept study, we show that LLMs can be used to identify individuals with a severe mental illness who also have suspected ID, which is a biologically and clinically meaningful subgroup of patients.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/72256","type":"journal-article","created":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T17:00:08Z","timestamp":1753894808000},"page":"e72256-e72256","source":"Crossref","is-referenced-by-count":1,"title":["Large Language Model\u2013Supported Identification of Intellectual Disabilities in Clinical Free-Text Summaries: Mixed Methods Study"],"prefix":"10.2196","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-9458-5838","authenticated-orcid":false,"given":"Aleksandra","family":"Edwards","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6845-7590","authenticated-orcid":false,"given":"Antonio F","family":"Pardi\u00f1as","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3427-3950","authenticated-orcid":false,"given":"George","family":"Kirov","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6168-9222","authenticated-orcid":false,"given":"Elliott","family":"Rees","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1618-7239","authenticated-orcid":false,"given":"Jose","family":"Camacho-Collados","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,9,18]]},"reference":[{"key":"R1","doi-asserted-by":"crossref","unstructured":"Agrawal M Hegselmann S Lang H Kim Y Sontag D . Large language models are few-shot clinical information extractors. Presented at: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing; Dec 7-11, 2022:1998-2022; Abu Dhabi, United Arab Emirates. URL: https:\/\/aclanthology.org\/2022.emnlp-main [Accessed 15-09-2025] [doi: 10.18653\/v1\/2022.emnlp-main.130]","DOI":"10.18653\/v1\/2022.emnlp-main.130"},{"issue":"6","key":"R2","doi-asserted-by":"publisher","first-page":"1113","DOI":"10.1016\/j.jbi.2011.08.006","article-title":"Coreference resolution: a review of general methodologies and applications in the clinical domain","volume":"44","author":"Zheng","journal-title":"J Biomed Inform"},{"issue":"5","key":"R3","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1006\/jbin.2001.1029","article-title":"A simple algorithm for identifying negated findings and diseases in discharge summaries","volume":"34","author":"Chapman","journal-title":"J Biomed Inform"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","DOI":"10.1038\/s41597-019-0322-0","article-title":"MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports","volume":"6","author":"Johnson","journal-title":"Sci Data"},{"issue":"10","key":"R5","doi-asserted-by":"publisher","first-page":"1529","DOI":"10.1093\/jamia\/ocaa106","article-title":"The 2019 n2c2\/UMass Lowell shared task on clinical concept normalization","volume":"27","author":"Luo","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-021-25578-4","article-title":"Automatically disambiguating medical acronyms with ontology-aware deep learning","volume":"12","author":"Skreta","journal-title":"Nat Commun"},{"key":"R7","unstructured":"Dong L Yang N Wang W et al. Unified language model pre-training for natural language understanding and generation. Presented at: NIPS\u201919: 33rd International Conference on Neural Information Processing Systems; Dec 8-14, 2019:13063-13075; Vancouver, BC, Canada. [doi: 10.5555\/3454287.3455457]"},{"key":"R8","unstructured":"Devlin J Chang MW Lee K Toutanova K . BERT: pre-training of deep bidirectional transformers for language understanding. Presented at: Proceedings of NAACL-HLT Association for Computational Linguistics; Jun 2-7, 2019:4171-4186; Minneapolis, Minnesota. URL: https:\/\/aclanthology.org\/N19-1423.pdf [Accessed 15-09-2025]"},{"key":"R9","doi-asserted-by":"crossref","unstructured":"Mosbach M Pimentel T Ravfogel S Klakow D Elazar Y . Few-shot fine-tuning vs. in-context learning: a fair comparison and evaluation. Presented at: Findings of the Association for Computational Linguistics: ACL 2023; Jul 9-14, 2023; Toronto, Canada. URL: https:\/\/aclanthology.org\/2023.findings-acl [Accessed 15-09-2025] [doi: 10.18653\/v1\/2023.findings-acl.779]","DOI":"10.18653\/v1\/2023.findings-acl.779"},{"key":"R10","unstructured":"Radford A Narasimhan K Salimans T Sutskever I . Improving language understanding by generative pre-training. OpenAI. 2018. URL: https:\/\/cdn.openai.com\/research-covers\/language-unsupervised\/language_understanding_paper.pdf [Accessed 15-09-2025]"},{"key":"R11","doi-asserted-by":"crossref","unstructured":"Viswanathan V Zhao C Bertsch A Wu T Neubig G . Prompt2Model: generating deployable models from natural language instructions. Presented at: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations; Dec 6-10, 2023:413-421; Singapore. URL: https:\/\/aclanthology.org\/2023.emnlp-demo [Accessed 15-09-2025] [doi: 10.18653\/v1\/2023.emnlp-demo.38]","DOI":"10.18653\/v1\/2023.emnlp-demo.38"},{"key":"R12","doi-asserted-by":"crossref","unstructured":"Yin W Hay J Roth D . Benchmarking zero-shot text classification: datasets, evaluation and entailment approach. Presented at: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP); Nov 3-7, 2019:3914-3923; Hong Kong, China. URL: https:\/\/www.aclweb.org\/anthology\/D19-1 [Accessed 15-09-2025] [doi: 10.18653\/v1\/D19-1404]","DOI":"10.18653\/v1\/D19-1404"},{"issue":"4","key":"R13","doi-asserted-by":"publisher","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","journal-title":"Bioinformatics"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Alsentzer E Murphy J Boag W et al. Publicly available clinical BERT embeddings. Presented at: Proceedings of the 2nd Clinical Natural Language Processing Workshop, Anna Rumshisky; Jun 7-8, 2019:72-78; Minneapolis, Minnesota, USA. [doi: 10.18653\/v1\/W19-1909]","DOI":"10.18653\/v1\/W19-1909"},{"key":"R15","unstructured":"Edwards A Ushio A Camacho-Collados J Ribaupierre H Preece A . Guiding generative language models for data augmentation in few-shot text classification. Presented at: Proceedings of the Fourth Workshop on Data Science with Human-in-the-Loop (Language Advances); Dec 8-9, 2022:51-63; Abu Dhabi, United Arab Emirates. URL: https:\/\/aclanthology.org\/2022.dash-1.8.pdf [Accessed 15-09-2025]"},{"key":"R16","doi-asserted-by":"crossref","unstructured":"Giridhara PKB Mishra C Venkataramana RKM Bukhari SS Dengel A . A study of various text augmentation techniques for relation classification in free text. Presented at: 8th International Conference on Pattern Recognition Applications and Methods; Feb 19-21, 2019:360-367; Prague, Czech Republic. [doi: 10.5220\/0007311003600367]","DOI":"10.5220\/0007311003600367"},{"key":"R17","doi-asserted-by":"crossref","unstructured":"T\u00fcrker R Zhang L Koutraki M Sack H . Knowledge-based short text categorization using entity and category embedding. Presented at: The Semantic Web: 16th International Conference, ESWC 2019; Jun 2-6, 2019:346-362; Portoro\u017e, Slovenia. [doi: 10.1007\/978-3-030-21348-0_23]","DOI":"10.1007\/978-3-030-21348-0_23"},{"key":"R18","doi-asserted-by":"crossref","unstructured":"Zhang X Wu B . Short text classification based on feature extension using the N-gram model. Presented at: 2015 12th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD); Aug 15-17, 2015:710-716; Zhangjiajie, China. [doi: 10.1109\/FSKD.2015.7382029]","DOI":"10.1109\/FSKD.2015.7382029"},{"key":"R19","unstructured":"Touvron H Martin L Stone K et al. Llama 2: open foundation and fine-tuned chat models. arXiv. Preprint posted online on Jul 18, 2023. [doi: 10.48550\/arXiv.2307.09288]"},{"key":"R20","doi-asserted-by":"crossref","unstructured":"Le Scao T Rush AM . How many data points is a prompt worth? Presented at: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; Jun 6-11, 2021:2627-2636; Online. URL: https:\/\/aclanthology.org\/2021.naacl-main [Accessed 15-09-2025] [doi: 10.18653\/v1\/2021.naacl-main.208]","DOI":"10.18653\/v1\/2021.naacl-main.208"},{"key":"R21","unstructured":"Plaza-del-Arco FM Nozza D Hovy D . Leveraging label variation in large language models for zero-shot text classification. Preprint posted online on Jul 24, 2023. [doi: 10.48550\/arXiv.2307.12973]"},{"key":"R22","unstructured":"Radford A Wu J Child R Luan D Amodei D Sutskever I et al. Language models are unsupervised multitask learners. OpenAI. 2019. URL: https:\/\/cdn.openai.com\/better-language-models\/language_models_are_unsupervised_multitask_learners.pdf [Accessed 15-09-2025]"},{"key":"R23","doi-asserted-by":"crossref","unstructured":"Schick T Sch\u00fctze H . Exploiting cloze-questions for few-shot text classification and natural language inference. Presented at: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics; Apr 19-23, 2021; Online. URL: https:\/\/aclanthology.org\/2021.eacl-main [Accessed 15-09-2025] [doi: 10.18653\/v1\/2021.eacl-main.20]","DOI":"10.18653\/v1\/2021.eacl-main.20"},{"key":"R24","unstructured":"Labrak Y Rouvier M Dufour R . A zero-shot and few-shot study of instruction-finetuned large language models applied to clinical and biomedical tasks. Presented at: Fourteenth Language Resources and Evaluation Conference (LREC-COLING 2024) Association for Computational Linguistics; May 20-25, 2024:2049-2066; Online. URL: https:\/\/aclanthology.org\/2024.lrec-main.185.pdf [Accessed 15-09-2025]"},{"key":"R25","doi-asserted-by":"crossref","unstructured":"Gao T Fisch A Chen D . Making pre-trained language models better few-shot learners. Presented at: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers); Aug 1-6, 2021:3816-3830; Online. URL: https:\/\/aclanthology.org\/2021.acl-long [Accessed 15-09-2025] [doi: 10.18653\/v1\/2021.acl-long.295]","DOI":"10.18653\/v1\/2021.acl-long.295"},{"key":"R26","unstructured":"Chung HW Hou L Longpre S et al. Scaling instruction-finetuned language models. arXiv. Preprint posted online on Oct 20, 2022. [doi: 10.48550\/arXiv.2210.11416]"},{"issue":"140","key":"R27","first-page":"1","volume":"21","author":"Raffel","journal-title":"J Mach Learn Res"},{"issue":"9","key":"R28","doi-asserted-by":"publisher","first-page":"2224","DOI":"10.1016\/j.ridd.2014.05.013","article-title":"Differences in the intellectual profile of children with intellectual vs. learning disability","volume":"35","author":"Cornoldi","journal-title":"Res Dev Disabil"},{"issue":"7956","key":"R29","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1038\/s41586-023-05881-4","article-title":"Foundation models for generalist medical artificial intelligence","volume":"616","author":"Moor","journal-title":"Nature New Biol"},{"key":"R30","doi-asserted-by":"crossref","unstructured":"Sevilla J Heim L Ho A Besiroglu T Hobbhahn M Villalobos P . Compute trends across three eras of machine learning. Presented at: 2022 International Joint Conference on Neural Networks (IJCNN); Jul 18-23, 2022:1-8; Padua, Italy. [doi: 10.1109\/IJCNN55064.2022.9891914]","DOI":"10.1109\/IJCNN55064.2022.9891914"},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3019178","article-title":"Understanding and improving disability identification in medical documents","volume":"8","author":"Fabregat","journal-title":"IEEE Access"},{"issue":"6","key":"R32","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.40895","article-title":"ChatDoctor: a medical chat model fine-tuned on a large language model meta-AI (LLaMA) using medical domain knowledge","volume":"15","author":"Li","journal-title":"Cureus"},{"key":"R33","doi-asserted-by":"crossref","unstructured":"Lu Y Zhao X Wang J . Medical knowledge-enhanced prompt learning for diagnosis classification from clinical text. Presented at: Proceedings of the 5th Clinical Natural Language Processing Workshop; Jul 14-15, 2023:278-288; Toronto, Canada. URL: https:\/\/aclanthology.org\/2023.clinicalnlp-1 [Accessed 15-09-2025] [doi: 10.18653\/v1\/2023.clinicalnlp-1.33]","DOI":"10.18653\/v1\/2023.clinicalnlp-1.33"},{"key":"R34","doi-asserted-by":"crossref","unstructured":"Meng Z Liu F Shareghi E Su Y Collins C Collier N . Rewire-then-probe: a contrastive recipe for probing biomedical knowledge of pre-trained language models. Presented at: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); May 22-27, 2022:4798-4810; Dublin, Ireland. URL: https:\/\/aclanthology.org\/2022.acl-long [Accessed 15-09-2025] [doi: 10.18653\/v1\/2022.acl-long.329]","DOI":"10.18653\/v1\/2022.acl-long.329"},{"key":"R35","doi-asserted-by":"crossref","unstructured":"Salek Faramarzi N Patel M Bandarupally SH Banerjee R . Context-aware medication event extraction from unstructured text. Presented at: Proceedings of the 5th Clinical Natural Language Processing Workshop; Jul 14-15, 2023:86-95; Toronto, Canada. URL: https:\/\/aclanthology.org\/2023.clinicalnlp-1 [Accessed 15-09-2025] [doi: 10.18653\/v1\/2023.clinicalnlp-1.11]","DOI":"10.18653\/v1\/2023.clinicalnlp-1.11"},{"issue":"9","key":"R36","doi-asserted-by":"publisher","first-page":"1821","DOI":"10.1093\/jamia\/ocae122","article-title":"BioInstruct: instruction tuning of large language models for biomedical natural language processing","volume":"31","author":"Tran","journal-title":"J Am Med Inform Assoc"},{"key":"R37","unstructured":"Wornow M Thapa R Steinberg E Fries J Shah N . EHRSHOT: an EHR benchmark for few-shot evaluation of foundation models. In: Oh A Naumann T Globerson A Saenko K Hardt M Levine S , editors. Advances in Neural Information Processing Systems 36 (NeurIPS 2023). Vol 36. NeurIPS; 2023:67125-67137."},{"issue":"6","key":"R38","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbac409","article-title":"BioGPT: generative pre-trained transformer for biomedical text generation and mining","volume":"23","author":"Luo","journal-title":"Brief Bioinformatics"},{"issue":"7","key":"R39","doi-asserted-by":"publisher","first-page":"1225","DOI":"10.1093\/jamia\/ocad091","article-title":"AI in health: keeping the human in the loop","volume":"30","author":"Bakken","journal-title":"J Am Med Inform Assoc"},{"key":"R40","doi-asserted-by":"crossref","unstructured":"Ruis F Pathak S Geerdink J Hegeman JH Seifert C Van Keulen M . Human-in-the-loop language-agnostic extraction of medication data from highly unstructured electronic health records. Presented at: 2020 International Conference on Data Mining Workshops (ICDMW); Nov 17-20, 2020:644-650; Sorrento, Italy. [doi: 10.1109\/ICDMW51313.2020.00091]","DOI":"10.1109\/ICDMW51313.2020.00091"},{"key":"R41","volume":"2020","author":"Wu","journal-title":"AMIA Annu Symp Proc"},{"issue":"9","key":"R42","doi-asserted-by":"publisher","first-page":"3638","DOI":"10.1038\/s41380-023-02293-8","article-title":"Genomic findings in schizophrenia and their implications","volume":"28","author":"Owen","journal-title":"Mol Psychiatry"},{"issue":"2","key":"R43","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1002\/wps.20624","article-title":"Nonsocial and social cognition in schizophrenia: current evidence and future directions","volume":"18","author":"Green","journal-title":"World Psychiatry"},{"issue":"5","key":"R44","doi-asserted-by":"publisher","first-page":"2081","DOI":"10.1038\/s41380-023-02013-2","article-title":"Family-based analysis of the contribution of rare and common genetic variants to school performance in schizophrenia","volume":"28","author":"Rammos","journal-title":"Mol Psychiatry"},{"issue":"6","key":"R45","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1001\/archpsyc.1990.01810180089012","article-title":"SCAN. Schedules for clinical assessment in neuropsychiatry","volume":"47","author":"Wing","journal-title":"Arch Gen Psychiatry"},{"key":"R46","unstructured":"Chapter V: mental and behavioural disorders (F00-F99). In: International Statistical Classification of Diseases and Related Health Problems 10th Revision (ICD-10). World Health Organization; 2019. URL: https:\/\/icd.who.int\/browse10\/2019\/en#\/F70-F79 [Accessed 15-09-2025]"},{"key":"R47","unstructured":"Wolf T Debut L Sanh V et al. HuggingFace\u2019s transformers: state-of-the-art natural language processing. arXiv. Preprint posted online on Oct 9, 2019. [doi: 10.48550\/arXiv.1910.03771]"},{"key":"R48","unstructured":"AleksEdwards\/JMIR-june-25. GitHub. URL: https:\/\/github.com\/AleksEdwards\/JMIR-June-25 [Accessed 15-09-2025]"},{"issue":"1-2","key":"R49","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1023\/A:1009982220290","article-title":"An evaluation of statistical approaches to text categorization","volume":"1","author":"Yang","journal-title":"Inf Retr Boston"},{"key":"R50","doi-asserted-by":"crossref","unstructured":"Elbattah M Arnaud E Ghazali DA Dequen G . Exploring the ethical challenges of large language models in emergency medicine: a comparative international review. Presented at: 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM); Dec 3-6, 2024:5750-5755; Lisbon, Portugal. [doi: 10.1109\/BIBM62325.2024.10822376]","DOI":"10.1109\/BIBM62325.2024.10822376"},{"issue":"6","key":"R51","doi-asserted-by":"publisher","first-page":"e428","DOI":"10.1016\/S2589-7500(24)00061-X","article-title":"Ethical and regulatory challenges of large language models in medicine","volume":"6","author":"Ong","journal-title":"Lancet Digit Health"}],"container-title":["JMIR AI"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ai.jmir.org\/2025\/1\/e72256","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:02:31Z","timestamp":1782842551000},"score":1,"resource":{"primary":{"URL":"https:\/\/ai.jmir.org\/2025\/1\/e72256"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,18]]},"references-count":51,"URL":"https:\/\/doi.org\/10.2196\/72256","relation":{},"ISSN":["2817-1705"],"issn-type":[{"value":"2817-1705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,18]]},"article-number":"v4i6e72256"}}