{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T22:57:31Z","timestamp":1791241051046,"version":"4.1.0"},"reference-count":35,"publisher":"Oxford University Press (OUP)","issue":"10","license":[{"start":{"date-parts":[[2026,6,27]],"date-time":"2026-06-27T00:00:00Z","timestamp":1782518400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["UL1 TR001857"],"award-info":[{"award-number":["UL1 TR001857"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["U24 TR004111"],"award-info":[{"award-number":["U24 TR004111"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01LM014588"],"award-info":[{"award-number":["R01LM014588"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01 LM014306"],"award-info":[{"award-number":["R01 LM014306"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,10,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>Secondary headaches require urgent recognition due to potentially devastating consequences if untreated. Despite established clinical \u201cred flag\u201d criteria, identifying patients needing immediate evaluation remains challenging in primary care. This study developed and evaluated a large language model (LLM)\u2013based multi-agent clinical decision support system for interpretable secondary headache diagnosis.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>We first established 7 clinically relevant secondary headache red flag domains through manual review and synthesis of clinical guidelines. Based on these domains, we designed an LLM-based system using an orchestrator\u2013specialist multi-agent architecture that decomposes diagnostic reasoning into 7 guideline-aligned agents corresponding to key red flag features. Each agent generates structured, evidence-grounded reasoning, coordinated by a central orchestrator. The system was evaluated on 90 expert-validated secondary headache cases and compared with a single-LLM baseline under 2 prompting strategies: question-based prompting (QPrompt) and guideline-based prompting (GPrompt). Five open-source LLMs (Qwen-8b, Qwen-14b, Qwen-30b, GPT-OSS-20b, and Llama-3.1-8b) were tested.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The orchestrated multi-agent system with GPrompt achieved the highest red flag classification performance across models, measured by F1 score. Performance gains were consistent and more pronounced in smaller LLMs, suggesting that structured reasoning improves efficiency and accuracy beyond prompt engineering alone. The framework also produced transparent and guideline-aligned intermediate reasoning.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>Decomposing clinical reasoning into specialized agents enhances interpretability and diagnostic reliability compared with monolithic LLM approaches. Multi-agent orchestration provides a clinically aligned framework for explainable decision support.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>An orchestrator\u2013specialist multi-agent LLM framework improves secondary headache diagnosis accuracy and transparency, supporting the development of explainable AI systems for time-constrained clinical decision-making in primary care.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocag111","type":"journal-article","created":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T11:45:08Z","timestamp":1781178308000},"page":"1910-1920","source":"Crossref","is-referenced-by-count":1,"title":["Orchestrator multi-agent clinical decision support system for secondary headache diagnosis in primary care"],"prefix":"10.1093","volume":"33","author":[{"given":"Xizhi","family":"Wu","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, University of Pittsburgh , Pittsburgh, PA,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hang","family":"Zhang","sequence":"additional","affiliation":[{"name":"Intelligent Systems Program, University of Pittsburgh , Pittsburgh, PA,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nelly Estefanie","family":"Garduno-Rapp","sequence":"additional","affiliation":[{"name":"Clinical Informatics Center, UT Southwestern Medical Center , Dallas, TX,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2817-9124","authenticated-orcid":false,"given":"Justin F","family":"Rousseau","sequence":"additional","affiliation":[{"name":"Clinical Informatics Center, UT Southwestern Medical Center , Dallas, TX,","place":["United States"]},{"name":"Peter O\u2019Donnell Jr. Brain Institute, UT Southwestern Medical Center , Dallas, TX,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mounika","family":"Thakkallapally","sequence":"additional","affiliation":[{"name":"Clinical Informatics Center, UT Southwestern Medical Center , Dallas, TX,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuelyu","family":"Ji","sequence":"additional","affiliation":[{"name":"Intelligent Systems Program, University of Pittsburgh , Pittsburgh, PA,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2079-8684","authenticated-orcid":false,"given":"Shyam","family":"Visweswaran","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, University of Pittsburgh , Pittsburgh, PA,","place":["United States"]},{"name":"Intelligent Systems Program, University of Pittsburgh , Pittsburgh, PA,","place":["United States"]},{"name":"Clinical and Translational Science Institute, University of Pittsburgh , Pittsburgh, PA,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9309-8331","authenticated-orcid":false,"given":"Yifan","family":"Peng","sequence":"additional","affiliation":[{"name":"Population Health Sciences, Weill Cornell Medicine , New York, NY,","place":["United States"]},{"name":"Institute of Artificial Intelligence for Digital Health, Weill Cornell Medicine , New York, NY,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4433-7839","authenticated-orcid":false,"given":"Yanshan","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, University of Pittsburgh , Pittsburgh, PA,","place":["United States"]},{"name":"Intelligent Systems Program, University of Pittsburgh , Pittsburgh, PA,","place":["United States"]},{"name":"Clinical and Translational Science Institute, University of Pittsburgh , Pittsburgh, PA,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,6,27]]},"reference":[{"key":"2026100116371928500_ocag111-B1","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1212\/WNL.0000000000006697","article-title":"Red and orange flags for secondary headaches in clinical practice: SNNOOP10 list","volume":"92","author":"Do","year":"2019","journal-title":"Neurology."},{"key":"2026100116371928500_ocag111-B2","doi-asserted-by":"crossref","first-page":"43","DOI":"10.31083\/j.jin2302043","article-title":"Headache disorders: differentiating primary and secondary etiologies","volume":"23","author":"Hernandez","year":"2024","journal-title":"J Integr Neurosci"},{"key":"2026100116371928500_ocag111-B3","volume-title":"Frontiers of Computer Science"},{"key":"2026100116371928500_ocag111-B4"},{"key":"2026100116371928500_ocag111-B5","author":"Hong","year":"2023"},{"key":"2026100116371928500_ocag111-B6","first-page":"31","article-title":"Diagnosing secondary headaches","volume":"20","author":"Juliana","year":"2020","journal-title":"Pract. Neurol"},{"key":"2026100116371928500_ocag111-B7","doi-asserted-by":"crossref","first-page":"28573","DOI":"10.1109\/ACCESS.2018.2831228","article-title":"Multi-agent systems: a survey","volume":"6","author":"Dorri","year":"2018","journal-title":"IEEE Access"},{"key":"2026100116371928500_ocag111-B8"},{"key":"2026100116371928500_ocag111-B9","first-page":"165025"},{"key":"2026100116371928500_ocag111-B10","doi-asserted-by":"crossref","first-page":"e0282237","DOI":"10.1371\/journal.pone.0282237","article-title":"A machine learning approach to support triaging of primary versus secondary headache patients using complete blood count","volume":"18","author":"Yang","year":"2023","journal-title":"PLoS One."},{"key":"2026100116371928500_ocag111-B11","doi-asserted-by":"crossref","first-page":"P5.10-002","DOI":"10.1212\/WNL.92.15_supplement.P5.10-002","article-title":"Acute headache diagnosis in the emergency department: accuracy and safety of an artificial intelligence system (P5.10-002)","volume":"92","author":"Acosta","year":"2019","journal-title":"Neurology"},{"key":"2026100116371928500_ocag111-B12","author":"Garduno Rapp","year":"2025"},{"key":"2026100116371928500_ocag111-B13","author":"Henseler"},{"key":"2026100116371928500_ocag111-B14","author":"Clifford"},{"key":"2026100116371928500_ocag111-B15","first-page":"5","article-title":"A case report of thunderclap headache with sub-arachnoid hemorrhage and negative angiography: a review of Call-Fleming syndrome and the use of transcranial dopplers in predicting morbidity","volume":"4","author":"Bittel","year":"2011","journal-title":"J Vasc Interv Neurol"},{"key":"2026100116371928500_ocag111-B16","doi-asserted-by":"crossref","first-page":"bcr2017224085","DOI":"10.1136\/bcr-2017-224085","article-title":"An unusual cause of thunderclap headache after eating the hottest pepper in the world\u2014\u201cThe Carolina Reaper\u201d","volume":"2018","author":"Boddhula","year":"2018","journal-title":"BMJ Case Rep"},{"key":"2026100116371928500_ocag111-B17","doi-asserted-by":"crossref","first-page":"001","DOI":"10.46766\/thegms.neuro.20060201","article-title":"Case report of a rare condition of thunderclap headache","volume":"1","author":"Lwin","year":"2020","journal-title":"G Med Sci"},{"key":"2026100116371928500_ocag111-B18","doi-asserted-by":"crossref","first-page":"1059","DOI":"10.3174\/ajnr.A1382","article-title":"Thunderclap headache without subarachnoid hemorrhage associated with regrowth of previously coil-occluded aneurysms","volume":"30","author":"Byrum","year":"2009","journal-title":"AJNR Am J Neuroradiol"},{"key":"2026100116371928500_ocag111-B19","doi-asserted-by":"crossref","first-page":"86","DOI":"10.5505\/jaemcr.2012.10337","article-title":"Headache and meningismus following lumbar puncture","volume":"3","author":"Bekta\u015f","journal-title":"J Acad Emerg Med Case Ref"},{"key":"2026100116371928500_ocag111-B20","doi-asserted-by":"crossref","first-page":"2038","DOI":"10.1002\/acn3.51452","article-title":"The case of a 19-year-old woman with headache, papilledema, and diplopia","volume":"8","author":"Tang","year":"2021","journal-title":"Ann Clin Transl Neurol"},{"key":"2026100116371928500_ocag111-B21","doi-asserted-by":"crossref","first-page":"79","DOI":"10.4314\/gmj.v53i1.12","article-title":"Case series of six patients diagnosed and managed for idiopathic intracranial hypertension at a tertiary institution eye centre","volume":"53","author":"Tagoe","year":"2019","journal-title":"Ghana Med J"},{"key":"2026100116371928500_ocag111-B22","doi-asserted-by":"crossref","first-page":"100165","DOI":"10.1016\/j.nerep.2023.100165","article-title":"Headache and papilledema: a case of neurosarcoidosis with cerebrovascular involvement","volume":"3","author":"Toljan","year":"2023","journal-title":"Neuroimmunol Rep"},{"key":"2026100116371928500_ocag111-B23","doi-asserted-by":"crossref","first-page":"185","DOI":"10.4103\/0972-2327.42940","article-title":"Giant cell arteritis (temporal arteritis): a report of four cases from north east India","volume":"11","author":"Laldinpuii","journal-title":"Ann Indian Acad Neurol"},{"key":"2026100116371928500_ocag111-B24","first-page":"e44107","article-title":"Giant cell arteritis masquerading as migraine: a case report","volume":"15","author":"Devi","year":"2023","journal-title":"Cureus."},{"key":"2026100116371928500_ocag111-B25","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1177\/19418744231182285","article-title":"Clinical problem solving: a 38-year-old woman with systemic lupus erythematosus presenting with headache, nausea, and vomiting","volume":"13","author":"Silverman","year":"2023","journal-title":"Neurohospitalist."},{"key":"2026100116371928500_ocag111-B26","doi-asserted-by":"crossref","first-page":"909","DOI":"10.1038\/s41597-023-02814-8","article-title":"A large-scale dataset of patient summaries for retrieval-based clinical decision support systems","volume":"10","author":"Zhao","year":"2023","journal-title":"Sci Data."},{"key":"2026100116371928500_ocag111-B27","doi-asserted-by":"crossref","first-page":"396","DOI":"10.1016\/S1474-4422(18)30085-1","article-title":"International Classification of Headache Disorders","volume":"17","author":"Olesen","year":"2018","journal-title":"Lancet Neurol."},{"key":"2026100116371928500_ocag111-B28"},{"key":"2026100116371928500_ocag111-B29"},{"key":"2026100116371928500_ocag111-B30","author":"Akram","year":"2024"},{"key":"2026100116371928500_ocag111-B31","author":"Colella","year":"2025"},{"key":"2026100116371928500_ocag111-B32","author":"Petropavlov","year":"2025"},{"key":"2026100116371928500_ocag111-B33"},{"key":"2026100116371928500_ocag111-B34"},{"key":"2026100116371928500_ocag111-B35"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocag111\/68632823\/ocag111.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/10\/1910\/68632823\/ocag111.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/10\/1910\/68632823\/ocag111.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T20:37:23Z","timestamp":1790887043000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/33\/10\/1910\/8719545"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,27]]},"references-count":35,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2026,6,27]]},"published-print":{"date-parts":[[2026,10,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocag111","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,10]]},"published":{"date-parts":[[2026,6,27]]}}}