{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,27]],"date-time":"2026-10-27T04:01:09Z","timestamp":1793073669013,"version":"4.5.0"},"reference-count":28,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2026,5,25]],"date-time":"2026-05-25T00:00:00Z","timestamp":1779667200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JCM"],"abstract":"<jats:p>Background\/Objectives: Clinical phenotyping from narrative electronic health records (EHRs) often relies on multi-stage pipelines involving span-level extraction, ontology mapping, and aggregation. Large language models (LLMs) may enable direct document-level abstraction of clinically meaningful phenotype features from complete notes. We evaluated whether GPT-5.2 could approximate human annotation for note-level multiple sclerosis (MS) phenotyping and compared its performance with human annotators, a locally run open-source LLM, HPO-based extraction tools, and a supervised clinical transformer encoder. Methods: We analyzed 100 de-identified MS neurology progress notes from a single academic medical center. Each note was annotated for the presence or absence of 17 predefined neurological phenotype categories. Two human annotators independently labeled all notes using a multi-label note-level framework in Prodigy, and disagreements were adjudicated to create a reference annotation set. GPT-5.2 was evaluated in a zero-shot setting using structured JSON output. Comparator methods included Llama-3.1 8B, Doc2Hpo, ClinPhen, PhenoSnap, and BioClinical ModernBERT. Performance was assessed using agreement, precision, recall, F1, Matthews correlation coefficient, and false-positive and false-negative assignments per note. Results: Human\u2013human agreement was generally high, although lower for rare or ambiguously documented features. GPT-5.2 achieved the strongest automated performance, with macro-precision 0.734, macro-recall 0.921, macro-F1 0.801, and macro-averaged MCC 0.777, approaching human annotator performance. GPT-5.2 showed the lowest false-negative count per note but more false-positive assignments than either human annotator, reflecting a sensitive but more inclusive annotation profile. Llama-3.1 8B performed competitively among automated methods, whereas HPO-based extraction tools and BioClinical ModernBERT showed lower performance on this low-resource note-level task. Secondary review of GPT-5.2 discordant assignments found no clear hallucinations and suggested that some apparent false positives reflected phenotype evidence missed in the human-derived reference set. Conclusions: GPT-5.2 achieved near-human performance for document-level recognition of MS phenotype categories from narrative neurology notes. Direct note-level abstraction may provide a scalable approach for research and population-health phenotyping of large EHR note corpora.<\/jats:p>","DOI":"10.3390\/jcm15114092","type":"journal-article","created":{"date-parts":[[2026,5,25]],"date-time":"2026-05-25T15:05:08Z","timestamp":1779721508000},"page":"4092","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Note-Level Phenotyping of Multiple-Sclerosis Notes by a Large Language Model Achieves near Human-Level Agreement"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6179-0793","authenticated-orcid":false,"given":"Daniel B.","family":"Hier","sequence":"first","affiliation":[{"name":"College of Medicine, University of Illinois at Chicago, Chicago, IL 60612, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-0336-2360","authenticated-orcid":false,"given":"Pavankumar Y.","family":"Srinivasula","sequence":"additional","affiliation":[{"name":"College of Liberal Arts and Sciences, University of Illinois at Chicago, Chicago, IL 60607, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5337-0804","authenticated-orcid":false,"given":"Michael D.","family":"Carrithers","sequence":"additional","affiliation":[{"name":"College of Medicine, University of Illinois at Chicago, Chicago, IL 60612, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,5,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"278","DOI":"10.1212\/WNL.0000000000000560","article-title":"Defining the clinical course of multiple sclerosis: The 2013 revisions","volume":"83","author":"Lublin","year":"2014","journal-title":"Neurology"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"907","DOI":"10.1212\/WNL.46.4.907","article-title":"Defining the clinical course of multiple sclerosis: Results of an international survey","volume":"46","author":"Lublin","year":"1996","journal-title":"Neurology"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1212\/01.CON.0000293634.15851.7d","article-title":"Clinical features of multiple sclerosis","volume":"10","author":"Miller","year":"2004","journal-title":"CONTINUUM Lifelong Learn. Neurol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"380","DOI":"10.7861\/clinmed.2020-0292","article-title":"Clinical presentation and diagnosis of multiple sclerosis","volume":"20","author":"Ford","year":"2020","journal-title":"Clin. Med."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1444","DOI":"10.1212\/WNL.33.11.1444","article-title":"Rating neurologic impairment in multiple sclerosis: An expanded disability status scale (EDSS)","volume":"33","author":"Kurtzke","year":"1983","journal-title":"Neurology"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"e334","DOI":"10.1136\/amiajnl-2013-001999","article-title":"Automated extraction of clinical traits of multiple sclerosis in electronic medical records","volume":"20","author":"Davis","year":"2013","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"777","DOI":"10.1212\/WNL.0000000000005347","article-title":"Practice guideline recommendations summary: Disease-modifying therapies for adults with multiple sclerosis: Report of the Guideline Development, Dissemination, and Implementation Subcommittee of the American Academy of Neurology","volume":"90","author":"Day","year":"2018","journal-title":"Neurology"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Correia, I., Bernardes, C., Cunha, C., Nunes, C., Mac\u00e1rio, C., Sousa, L., and Batista, S. (2024). Picturing the Multiple Sclerosis patient journey: A symptomatic overview. J. Clin. Med., 13.","DOI":"10.3390\/jcm13195687"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"287","DOI":"10.7224\/1537-2073.2022-066","article-title":"Clinical and research applications of the electronic medical record in multiple sclerosis: A narrative review of current uses and future applications","volume":"24","author":"Swetlik","year":"2022","journal-title":"Int. J. MS Care"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Qi, J., Luo, L., Yang, Z., Wang, J., Zhou, H., and Lin, H. (2024, January 3\u20136). An improved method for phenotype concept recognition using rich HPO information. Proceedings of the 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Lisbon, Portugal.","DOI":"10.1109\/BIBM62325.2024.10822556"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"bay110","DOI":"10.1093\/database\/bay110","article-title":"Annotation of phenotypes using ontologies: A gold standard for the training and evaluation of natural language processing systems","volume":"2018","author":"Dahdul","year":"2018","journal-title":"Database"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Oommen, C., Howlett-Prieto, Q., Carrithers, M.D., and Hier, D.B. (2023). Inter-rater agreement for the annotation of neurologic signs and symptoms in electronic health records. Front. Digit. Health, 5.","DOI":"10.3389\/fdgth.2023.1075771"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Gehrmann, S., Dernoncourt, F., Li, Y., Carlson, E.T., Wu, J.T., Welt, J., Foote, J., Moseley, E., Grant, D.W., and Tyler, P. (2018). Comparing deep learning and concept extraction based methods for patient phenotyping from clinical narratives. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0192360"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"bav005","DOI":"10.1093\/database\/bav005","article-title":"Automatic concept recognition using the human phenotype ontology reference and test suite corpora","volume":"2015","author":"Groza","year":"2015","journal-title":"Database"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"100887","DOI":"10.1016\/j.patter.2023.100887","article-title":"Enhancing phenotype recognition in clinical notes using large language models: PhenoBCBERT and PhenoGPT","volume":"5","author":"Yang","year":"2023","journal-title":"Patterns"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales","volume":"20","author":"Cohen","year":"1960","journal-title":"Educ. Psychol. Meas."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"276","DOI":"10.11613\/BM.2012.031","article-title":"Interrater reliability: The kappa statistic","volume":"22","author":"McHugh","year":"2012","journal-title":"Biochem. Med."},{"key":"ref_18","unstructured":"OpenAI (2023). GPT-4 Technical Report. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Yang, X., PourNejatian, N., Shin, H.C., Smith, K.E., Parisien, C., Compas, C., Martin, C., Flores, M.G., Zhang, Y., and Magoc, T. (2022). Gatortron: A large language model for clinical natural language processing. MedRxiv, MedRxiv:2022.02.27.22271257.","DOI":"10.1101\/2022.02.27.22271257"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Hier, D.B., Munzir, S.I., Stahlfeld, A., Obafemi-Ajayi, T., and Carrithers, M. (2024, January 10\u201313). High-Throughput Phenotyping of Clinical Text Using Large Language Models. Proceedings of the 2024 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI), Houston, TX, USA.","DOI":"10.1109\/BHI62660.2024.10913712"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Hier, D.B., Platt, S.K., and Obafemi-Ajayi, T. (2025, January 26\u201329). Predicting Failures of LLMs to Link Biomedical Ontology Terms to Identifiers: Evidence Across Models and Ontologies. Proceedings of the 2025 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI), Atlanta, GA, USA.","DOI":"10.1109\/BHI67747.2025.11269551"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Pericharla, S., Hier, D.B., and Obafemi-Ajayi, T. (2026). From Memorization to Generalization: Fine-Tuning Large Language Models for Biomedical Term-to-Identifier Normalization. Front. Digit. Health, 8.","DOI":"10.3389\/fdgth.2026.1783907"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Bombieri, M., Fiorini, P., Ponzetto, S.P., and Rospocher, M. (2025). Do LLMs Dream of Ontologies?. ACM Trans. Intell. Syst. Technol., accepted.","DOI":"10.1145\/3725852"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"W566","DOI":"10.1093\/nar\/gkz386","article-title":"Doc2Hpo: A web application for efficient and accurate HPO concept curation","volume":"47","author":"Liu","year":"2019","journal-title":"Nucleic Acids Res."},{"key":"ref_26","unstructured":"Wang, K., and Lab, W.G. (2026, May 08). PhenoSnap: Rapid Extraction of Phenotypes and Medical Entities into Phenopackets from NATURAL Texts. Available online: https:\/\/github.com\/WGLab\/PhenoSnap."},{"key":"ref_27","unstructured":"Chen, H., Wang, Z., Nguyen, Q.M., Zhang, G., Weng, C., and Wang, K. (2026). Autonomous Agent-Orchestrated Digital Twins (AADT): Leveraging the OpenClaw Framework for State Synchronization in Rare Genetic Disorders. arXiv."},{"key":"ref_28","unstructured":"Sounack, T., Davis, J., Durieux, B., Chaffin, A., Pollard, T.J., Lehman, E., Johnson, A.E., McDermott, M., Naumann, T., and Lindvall, C. (2025). Bioclinical modernbert: A state-of-the-art long-context encoder for biomedical and clinical nlp. arXiv."}],"container-title":["Journal of Clinical Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2077-0383\/15\/11\/4092\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T11:48:54Z","timestamp":1780400934000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2077-0383\/15\/11\/4092"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,25]]},"references-count":28,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2026,6]]}},"alternative-id":["jcm15114092"],"URL":"https:\/\/doi.org\/10.3390\/jcm15114092","relation":{},"ISSN":["2077-0383"],"issn-type":[{"value":"2077-0383","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,25]]}}}