{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T19:08:44Z","timestamp":1777489724155,"version":"3.51.4"},"reference-count":40,"publisher":"SAGE Publications","issue":"4","license":[{"start":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T00:00:00Z","timestamp":1775001600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"},{"start":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T00:00:00Z","timestamp":1775001600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"DOI":"10.13039\/100010665","name":"H2020 Marie Sk\u0142odowska-Curie Actions","doi-asserted-by":"publisher","award":["101034252"],"award-info":[{"award-number":["101034252"]}],"id":[{"id":"10.13039\/100010665","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Insight Research Ireland Centre for Data Analytics","award":["12\/RC\/2289_P2"],"award-info":[{"award-number":["12\/RC\/2289_P2"]}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Cephalalgia"],"published-print":{"date-parts":[[2026,4]]},"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Accurate diagnosis of headache disorders is essential in clinical practice. Supervised machine learning models trained on structured clinical data have shown good performance, whereas the diagnostic ability of large language models (LLMs) for headache disorders has not been evaluated. This study compared a validated machine learning classifier with a general-purpose, zero-shot, non-domain-adapted LLM using the same structured patient questionnaire data, focusing on their agreement with specialist-confirmed diagnoses as the ground truth. This study was designed to reflect current real-world use scenarios, in which clinicians may apply off-the-shelf LLMs for diagnostic purposes without few-shot prompting, domain-specific fine-tuning, or adaptation, rather than to assess the theoretical upper limits of LLM capabilities.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We analyzed 1818 patients from an independent hold-out test cohort who completed a 22-item structured headache questionnaire and received specialist-confirmed diagnoses. A previously developed machine learning model and a general-purpose, non-domain-adapted LLM (GPT-4.1 with zero-shot prompting) each generated five-class International Classification of Headache Disorders, 3rd edition (ICHD-3)-based predictions: migraine and\/or medication-overuse headache (MOH), tension-type headache (TTH), trigeminal autonomic cephalalgias (TACs), other primary headache disorders, and secondary headaches. Agreement with the specialist's diagnosis and diagnostic performance metrics were calculated. Class-wise sensitivity and specificity were compared using McNemar's test.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The machine learning classifier showed significantly higher diagnostic agreement with the specialist than the LLM (Cohen's \u03ba: 0.46 vs. 0.26; 95% confidence interval of the difference: 0.15\u20130.25). Although the LLM showed slightly higher macro-averaged sensitivity (balanced accuracy) than the machine learning model, the machine learning classifier showed higher macro-averaged precision, specificity, and F-value. Class-wise analysis showed that the machine learning model demonstrated greater sensitivity for migraine and\/or MOH and secondary headaches, while the LLM showed higher sensitivity for TTH. Regarding specificity, the machine learning model outperformed the LLM in TTH, TACs, and other primary headache disorders, whereas the LLM showed higher specificity only for migraine and\/or MOH.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>A supervised machine learning model trained on real-world clinical data showed better agreement with a specialist-confirmed diagnosis than a general-purpose, zero-shot, non-domain-adapted LLM. These findings indicate that, in its current off-the-shelf configuration under this experimental setting, the diagnostic agreement between a general-purpose LLM and specialists can be limited for headache disorders.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/03331024261441574","type":"journal-article","created":{"date-parts":[[2026,4,17]],"date-time":"2026-04-17T13:57:31Z","timestamp":1776434251000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":0,"title":["Comparative diagnostic agreement of a supervised machine learning model and a general-purpose, zero-shot, non-domain-adapted large language model for classifying headache disorders using structured questionnaires"],"prefix":"10.1177","volume":"46","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0192-5430","authenticated-orcid":false,"given":"Masahito","family":"Katsuki","sequence":"first","affiliation":[{"name":"Nagaoka University of Technology"},{"name":"Dublin City University"},{"name":"Dublin City University"},{"name":"Department of Biostatistics, Graduate School of Medicine, Saitama Medical University, Saitama, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2015-8967","authenticated-orcid":false,"given":"Kieran","family":"Moran","sequence":"additional","affiliation":[{"name":"Dublin City University"},{"name":"Insight Research Ireland Centre for Data Analytics, Maynooth University, Kildare, Ireland"},{"name":"Maynooth University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2001-0746","authenticated-orcid":false,"given":"Siobh\u00e1n","family":"O\u2019Connor","sequence":"additional","affiliation":[{"name":"Dublin City University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tom\u00e1s","family":"Ward","sequence":"additional","affiliation":[{"name":"Dublin City University"},{"name":"School of Computing, Dublin City University, Dublin, Ireland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yutaro","family":"Fuse","sequence":"additional","affiliation":[{"name":"Chiba University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8182-0582","authenticated-orcid":false,"given":"Omid Kohandel","family":"Gargari","sequence":"additional","affiliation":[{"name":"Tehran University of Medical Sciences"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6016-3141","authenticated-orcid":false,"given":"Marina","family":"Romozzi","sequence":"additional","affiliation":[{"name":"Universit\u00e0 Cattolica del Sacro Cuore"},{"name":"Neurologia, Dipartimento di Scienze dell'invecchiamento, Neurologiche, Ortopediche e della Testa-Collo, Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1228-1503","authenticated-orcid":false,"given":"Alicia","family":"Gonzalez-Martinez","sequence":"additional","affiliation":[{"name":"Neurology Department, La Princesa University Hospital, Madrid, Spain"},{"name":"Translational Research Group in Multimodal Biomarkers in Neurological Diseases, IIS-Princesa, Madrid, Spain"},{"name":"Autonomous University, Madrid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2842-0085","authenticated-orcid":false,"given":"Miguel \u00c1","family":"Huerta","sequence":"additional","affiliation":[{"name":"University of Granada"},{"name":"Biosanitary Research Institute ibs.GRANADA, Granada, Spain."},{"name":"Department of 17harmacology, University of Cambridge, Cambridge, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1188-449X","authenticated-orcid":false,"given":"Woo-Seok","family":"Ha","sequence":"additional","affiliation":[{"name":"Severance Hospital, Yonsei University College of Medicine"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jackson TS","family":"Cheung","sequence":"additional","affiliation":[{"name":"UCL Faculty of Medical Sciences"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5852-0146","authenticated-orcid":false,"given":"Yasuhiko","family":"Matsumori","sequence":"additional","affiliation":[{"name":"Sendai Headache and Neurology Clinic, Sendai, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2026,4,17]]},"reference":[{"key":"e_1_3_4_2_2","first-page":"344","article-title":"Global, regional, and national burden of disorders affecting the nervous system, 1990\u20132021: a systematic analysis for the Global Burden of Disease Study 2021","volume":"23","author":"GBD 2021 Nervous System Disorders Collaborators","unstructured":"GBD 2021 Nervous System Disorders Collaborators. Global, regional, and national burden of disorders affecting the nervous system, 1990\u20132021: a systematic analysis for the Global Burden of Disease Study 2021. Lancet Neurol 2024 Apr; 23: 344\u2013381.","journal-title":"Lancet Neurol"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.1007\/s40120-021-00305-9"},{"key":"e_1_3_4_4_2","first-page":"333102425138772","article-title":"Early treatment in migraine \u2013 a call to shift prevention from attacks to disease progression: a position statement from the International Headache Society","volume":"45","author":"Pozo-Rosich P","unstructured":"Pozo-Rosich P, Caronna E, Sacco S, et al. Early treatment in migraine \u2013 a call to shift prevention from attacks to disease progression: a position statement from the International Headache Society. Cephalalgia 2025 Oct; 45: 3331024251387721.","journal-title":"Cephalalgia"},{"key":"e_1_3_4_5_2","article-title":"Understanding delays and diagnostic shifts in primary headaches: evidence from Japanese Health Insurance Claims","volume":"17","author":"Tatsuno Y","unstructured":"Tatsuno Y, Katsuki M, Kawata Y, et al. Understanding delays and diagnostic shifts in primary headaches: evidence from Japanese Health Insurance Claims. Cureus 2025 May 28; 17: e85005.","journal-title":"Cureus"},{"key":"e_1_3_4_6_2","first-page":"333102423122617","article-title":"Treatment patterns and characteristics of headache in patients in Japan: a retrospective cross-sectional and longitudinal analysis of health insurance claims data","volume":"44","author":"Katsuki M","unstructured":"Katsuki M, Matsumori Y, Ichihara T, et al. Treatment patterns and characteristics of headache in patients in Japan: a retrospective cross-sectional and longitudinal analysis of health insurance claims data. Cephalalgia 2024 Jan; 44: 3331024231226177.","journal-title":"Cephalalgia"},{"key":"e_1_3_4_7_2","first-page":"1","volume-title":"Cephalalgia","author":"Headache Classification Committee of the International Headache Society (IHS)","year":"2018","unstructured":"Headache Classification Committee of the International Headache Society (IHS). The International Classification of Headache Disorders. Cephalalgia 2018; 38: 1\u2013211."},{"key":"e_1_3_4_8_2","first-page":"333102423115692","article-title":"Developing an artificial intelligence-based headache diagnostic model and its utility for non-specialists\u2019 diagnostic accuracy","volume":"43","author":"Katsuki M","unstructured":"Katsuki M, Shimazu T, Kikui S, et al. Developing an artificial intelligence-based headache diagnostic model and its utility for non-specialists\u2019 diagnostic accuracy. Cephalalgia 2023 May; 43: 3331024231156925.","journal-title":"Cephalalgia"},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.1111\/head.14611"},{"key":"e_1_3_4_10_2","first-page":"75","article-title":"Preliminary development of a deep learning-based automated primary headache diagnosis model using Japanese natural language processing of medical questionnaire","volume":"11","author":"Katsuki M","unstructured":"Katsuki M, Narita N, Matsumori Y, et al. Preliminary development of a deep learning-based automated primary headache diagnosis model using Japanese natural language processing of medical questionnaire. Surg Neurol Int 2020 Dec 29; 11: 75.","journal-title":"Surg Neurol Int"},{"key":"e_1_3_4_11_2","first-page":"909","article-title":"The emerging clinical relevance of artificial intelligence, data science, and wearable devices in headache: a Narrative Review","volume":"15","author":"Danelakis A","unstructured":"Danelakis A, Stubberud A, Tronvik E, et al. The emerging clinical relevance of artificial intelligence, data science, and wearable devices in headache: a Narrative Review. Life (Basel) 2025 Jun 4; 15: 909.","journal-title":"Life (Basel)"},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11916-024-01297-5"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.1177\/03331024251372117"},{"key":"e_1_3_4_14_2","first-page":"51","article-title":"Influence of next-generation artificial intelligence on headache research, diagnosis and treatment: the junior editorial board members\u2019 vision - part 1","volume":"25","author":"Petru\u0161i\u0107 I","unstructured":"Petru\u0161i\u0107 I, Ha W-S, Labastida-Ramirez A, et al. Influence of next-generation artificial intelligence on headache research, diagnosis and treatment: the junior editorial board members\u2019 vision - part 1. J Headache Pain 2024 Sep 13; 25: 51.","journal-title":"J Headache Pain"},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.1186\/s10194-024-01944-7"},{"key":"e_1_3_4_16_2","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocac006"},{"key":"e_1_3_4_17_2","article-title":"Large language model influence on diagnostic reasoning: a randomized clinical trial","volume":"7","author":"Goh E","unstructured":"Goh E, Gallo R, Hom J, et al. Large language model influence on diagnostic reasoning: a randomized clinical trial. JAMA Netw Open 2024 Oct 1; 7: e2440969.","journal-title":"JAMA Netw Open"},{"key":"e_1_3_4_18_2","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare12161594"},{"key":"e_1_3_4_19_2","first-page":"4768","article-title":"A unified approach to interpreting model predictions","volume":"30","author":"Lundberg S","year":"2017","unstructured":"Lundberg S, Lee S-I. A unified approach to interpreting model predictions. Adv Neural Inf Process Syst 2017; 30: 4768\u20134777.","journal-title":"Adv Neural Inf Process Syst"},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"e_1_3_4_21_2","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-13-91"},{"key":"e_1_3_4_22_2","doi-asserted-by":"publisher","DOI":"10.1111\/head.14702"},{"key":"e_1_3_4_23_2","first-page":"333102425138768","article-title":"What does ChatGPT know about Migraine? A comparative-descriptive analysis","volume":"45","author":"Garcia LB","unstructured":"Garcia LB, Ferreira AJ, Hussein MA, et al. What does ChatGPT know about Migraine? A comparative-descriptive analysis. Cephalalgia 2025 Oct; 45: 3331024251387684.","journal-title":"Cephalalgia"},{"key":"e_1_3_4_24_2","article-title":"Reliability and accuracy of generative artificial intelligence tools in providing general information on migraine surgery","volume":"13","author":"Raposio E","unstructured":"Raposio E, Baldelli I. Reliability and accuracy of generative artificial intelligence tools in providing general information on migraine surgery. Plast Reconstr Surg Glob Open 2025 Oct 2; 13: e7176.","journal-title":"Plast Reconstr Surg Glob Open"},{"key":"e_1_3_4_25_2","article-title":"A cross-sectional comparison of patient information guides generated by chatgpt versus google gemini for Alzheimer\u2019s disease, parkinsonism, and migraine","volume":"17","author":"Chaulagain A","unstructured":"Chaulagain A, Aujla S, Priyadarsini A, et al. A cross-sectional comparison of patient information guides generated by chatgpt versus google gemini for Alzheimer\u2019s disease, parkinsonism, and migraine. Cureus 2025 May 20; 17: e84507.","journal-title":"Cureus"},{"key":"e_1_3_4_26_2","article-title":"Benchmarking state-of-the-art large language models for migraine patient education: performance comparison of responses to common queries","volume":"26","author":"Li L","unstructured":"Li L, Li P, Wang K, et al. Benchmarking state-of-the-art large language models for migraine patient education: performance comparison of responses to common queries. J Med Internet Res 2024 Jul 23; 26: e55927.","journal-title":"J Med Internet Res"},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1186\/s41512-025-00211-w"},{"key":"e_1_3_4_28_2","article-title":"CPLLM: clinical prediction with large language models","volume":"3","author":"Ben Shoham O","unstructured":"Ben Shoham O, Rappoport N. CPLLM: clinical prediction with large language models. PLOS Digit Health 2024 Dec 6; 3: e0000680.","journal-title":"PLOS Digit Health"},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1007\/s13755-025-00403-0"},{"key":"e_1_3_4_30_2","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocaf038"},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.3389\/fmed.2024.1380148"},{"key":"e_1_3_4_32_2","article-title":"Comparative analysis of large language models in clinical diagnosis: performance evaluation across common and complex medical cases","volume":"8","author":"Dinc MT","unstructured":"Dinc MT, Bardak AE, Bahar F, et al. Comparative analysis of large language models in clinical diagnosis: performance evaluation across common and complex medical cases. JAMIA Open 2025 Jun 12; 8: ooaf055.","journal-title":"JAMIA Open"},{"key":"e_1_3_4_33_2","article-title":"Large language models in medical diagnostics: scoping review with bibliometric analysis","volume":"27","author":"Su H","unstructured":"Su H, Sun Y, Li R, et al. Large language models in medical diagnostics: scoping review with bibliometric analysis. J Med Internet Res 2025 Jun 9; 27: e72062.","journal-title":"J Med Internet Res"},{"key":"e_1_3_4_34_2","first-page":"17","article-title":"A systematic review of large language model (LLM) evaluations in clinical medicine","volume":"25","author":"Shool S","unstructured":"Shool S, Adimi S, Saboori Amleshi R, et al. A systematic review of large language model (LLM) evaluations in clinical medicine. BMC Med Inform Decis Mak 2025 Mar 7; 25: 17.","journal-title":"BMC Med Inform Decis Mak"},{"key":"e_1_3_4_35_2","first-page":"237","article-title":"Towards community-based evaluation of AI in neurology: development of a headache diagnosis dataset for large language models","volume":"332","author":"Zahn A","unstructured":"Zahn A, Strauss S, Zwanzig D. Towards community-based evaluation of AI in neurology: development of a headache diagnosis dataset for large language models. Stud Health Technol Inform 2025 Oct 2; 332: 237\u2013241.","journal-title":"Stud Health Technol Inform"},{"key":"e_1_3_4_36_2","doi-asserted-by":"publisher","DOI":"10.1111\/head.14482"},{"key":"e_1_3_4_37_2","first-page":"333102423116924","article-title":"Forecasting migraine with machine learning based on mobile phone diary and wearable data","volume":"43","author":"Stubberud A","unstructured":"Stubberud A, Ingvaldsen SH, Brenner E, et al. Forecasting migraine with machine learning based on mobile phone diary and wearable data. Cephalalgia 2023 May; 43: 3331024231169244.","journal-title":"Cephalalgia"},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.1111\/ene.15037"},{"key":"e_1_3_4_39_2","first-page":"15","article-title":"Machine learning classification meets migraine: recommendations for study evaluation","volume":"25","author":"Petru\u0161i\u0107 I","unstructured":"Petru\u0161i\u0107 I, Savi\u0107 A, Mitrovi\u0107 K, et al. Machine learning classification meets migraine: recommendations for study evaluation. J Headache Pain 2024 Dec 5; 25: 15.","journal-title":"J Headache Pain"},{"key":"e_1_3_4_40_2","doi-asserted-by":"publisher","DOI":"10.1186\/s10194-025-02134-9"},{"key":"e_1_3_4_41_2","doi-asserted-by":"publisher","DOI":"10.1186\/s10194-024-01937-6"}],"container-title":["Cephalalgia"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/03331024261441574","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/03331024261441574","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/03331024261441574","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T14:44:26Z","timestamp":1777387466000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/03331024261441574"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4]]},"references-count":40,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2026,4]]}},"alternative-id":["10.1177\/03331024261441574"],"URL":"https:\/\/doi.org\/10.1177\/03331024261441574","relation":{},"ISSN":["0333-1024","1468-2982"],"issn-type":[{"value":"0333-1024","type":"print"},{"value":"1468-2982","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4]]},"article-number":"03331024261441574"}}