{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,21]],"date-time":"2026-09-21T19:02:01Z","timestamp":1790017321602,"version":"4.0.1"},"reference-count":36,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Aging"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Alzheimer disease (AD) is a leading cause of dementia, and there is growing interest in scalable approaches for early screening using speech-based tasks. While prior work has demonstrated promising results using either transcript-based language features or acoustic cues, most approaches remain unimodal or rely on simple fusion strategies that do not explicitly consider interactions across modalities.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n In this study, we propose an attention-based trimodal fusion framework that integrates text, audio, and image representations of the Cookie Theft picture, which serves as the shared visual stimulus in the picture-description task.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Our method uses a new bidirectional cross-attention mechanism to achieve a unified multimodal embedding for downstream tasks. We evaluate the approach on 2 tasks: AD detection by classifying whether the participant has AD or not, and AD severity assessment by predicting Mini-Mental Status Examination cognitive scores.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n On the AD detection task, trimodal fusion achieves the best overall performance (\n F<\/jats:italic>\n 1<\/jats:sub>\n -score=0.8667, area under the receiver operating characteristic curve=0.9032), outperforming unimodal baselines, bimodal fusion, and conventional early or late fusion methods. For AD severity assessment, the proposed multimodal representation reduces prediction error of root mean squared error to about 4.20, improving over both unimodal and bimodal fusion settings. We further perform the ablation analysis to show that bidirectional cross-attention consistently outperforms conventional unidirectional cross-attention.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n These results demonstrate that attention-based multimodal fusion can enhance dementia prediction from picture-description responses and provide a strong foundation for developing multimodal cognitive screening pipelines.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/93279","type":"journal-article","created":{"date-parts":[[2026,8,31]],"date-time":"2026-08-31T10:05:07Z","timestamp":1788170707000},"page":"e93279-e93279","source":"Crossref","is-referenced-by-count":0,"title":["Multimodal Dementia Prediction With Large Language Models: Cross-Attention Over Text, Audio, and Image"],"prefix":"10.2196","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-2296-998X","authenticated-orcid":false,"given":"Felix","family":"Agbavor","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3805-1837","authenticated-orcid":false,"given":"Hualou","family":"Liang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,21]]},"reference":[{"issue":"3","key":"R1","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1016\/j.jalz.2011.03.005","article-title":"The diagnosis of dementia due to Alzheimer\u2019s disease: recommendations from the National Institute on Aging-Alzheimer\u2019s Association workgroups on diagnostic guidelines for Alzheimer\u2019s disease","volume":"7","author":"McKhann","journal-title":"Alzheimers Dement"},{"issue":"5","key":"R2","doi-asserted-by":"publisher","first-page":"501","DOI":"10.1080\/13803390701550128","article-title":"Language performance in Alzheimer\u2019s disease and mild cognitive impairment: a comparative review","volume":"30","author":"Taler","journal-title":"J Clin Exp Neuropsychol"},{"issue":"10248","key":"R3","doi-asserted-by":"publisher","first-page":"413","DOI":"10.1016\/S0140-6736(20)30367-6","article-title":"Dementia prevention, intervention, and care: 2020 report of the Lancet Commission","volume":"396","author":"Livingston","journal-title":"Lancet"},{"issue":"3","key":"R4","doi-asserted-by":"publisher","first-page":"189","DOI":"10.1016\/0022-3956(75)90026-6","article-title":"\u201cMini-mental state\u201d. A practical method for grading the cognitive state of patients for the clinician","volume":"12","author":"Folstein","journal-title":"J Psychiatr Res"},{"issue":"8","key":"R5","doi-asserted-by":"publisher","first-page":"5143","DOI":"10.1002\/alz.13859","article-title":"Revised criteria for diagnosis and staging of Alzheimer\u2019s disease: Alzheimer\u2019s Association Workgroup","volume":"20","author":"Jack","journal-title":"Alzheimers Dement"},{"issue":"12","key":"R6","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000168","article-title":"Predicting dementia from spontaneous speech using large language models","volume":"1","author":"Agbavor","journal-title":"PLOS Digit Health"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1016\/j.eclinm.2020.100583","article-title":"Linguistic markers predict onset of Alzheimer\u2019s disease","volume":"28","author":"Eyigoz","journal-title":"EClinicalMedicine"},{"key":"R8","doi-asserted-by":"crossref","unstructured":"Luz S Haider F Fuente S de la Fromm D MacWhinney B . Detecting cognitive decline using speech only: the ADReSSo challenge. Presented at: INTERSPEECH 2021; Aug 30 to Sep 3, 2021; Brno, Czechia. [doi: 10.21437\/Interspeech.2021-1220]","DOI":"10.21437\/Interspeech.2021-1220"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-13902-7","article-title":"Multimodal Siamese networks for dementia detection from speech in women","volume":"15","author":"Ksibi","journal-title":"Sci Rep"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.3389\/fnagi.2022.830943","article-title":"Multimodal deep learning models for detecting dementia from speech and transcripts","volume":"14","author":"Ilias","journal-title":"Front Aging Neurosci"},{"key":"R11","doi-asserted-by":"crossref","unstructured":"Tsai YHH Bai S Yamada M Morency LP Salakhutdinov R . Transformer dissection: an unified understanding for transformer\u2019s attention via the lens of kernel. In: Inui K Jiang J Ng V Wan X , editors. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP. Association for Computational Linguistics; 2019:4344-4353. [doi: 10.18653\/v1\/D19-1443]","DOI":"10.18653\/v1\/D19-1443"},{"issue":"6","key":"R12","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1007\/s00530-010-0182-0","article-title":"Multimodal fusion for multimedia analysis: a survey","volume":"16","author":"Atrey","journal-title":"Multimedia Systems"},{"key":"R13","unstructured":"Arevalo J Solorio T Montes-y-G\u00f3mez M Gonz\u00e1lez FA . Gated multimodal units for information fusion. arXiv. Preprint posted online on Feb 7, 2017. [doi: 10.48550\/arXiv.1702.01992]"},{"key":"R14","unstructured":"Vaswani A Shazeer N Parmar N et al. Attention is all you need. Presented at: 31st Conference on Neural Information Processing Systems (NIPS 2017); Dec 4-9, 2017; Long Beach, California, USA. URL: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2017\/file\/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf [Accessed 10-09-2026]"},{"key":"R15","doi-asserted-by":"crossref","unstructured":"Tan H Bansal M . LXMERT: learning cross-modality encoder representations from transformers. In: Inui K Jiang J Ng V Wan X , editors. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics; 2019:5100-5111. [doi: 10.18653\/v1\/D19-1514]","DOI":"10.18653\/v1\/D19-1514"},{"issue":"1","key":"R16","doi-asserted-by":"publisher","first-page":"12","DOI":"10.3390\/jdad3010012","article-title":"Dementia detection from spontaneous speech using cross-attention fusion","volume":"3","author":"Agbavor","journal-title":"J Dement Alzheimers Dis"},{"key":"R17","unstructured":"Goodglass H Kaplan E Barresi B . The Assessment of Aphasia and Related Disorders. 3rd ed. Lippincott Williams & Wilkins; 2001. ISBN: 9780683305593"},{"issue":"6","key":"R18","doi-asserted-by":"publisher","first-page":"585","DOI":"10.1001\/archneur.1994.00540180063015","article-title":"The natural history of Alzheimer\u2019s disease. Description of study cohort and accuracy of diagnosis","volume":"51","author":"Becker","journal-title":"Arch Neurol"},{"key":"R19","unstructured":"Baevski A Zhou Y Mohamed A Auli M . Wav2vec 2.0: a framework for self-supervised learning of speech representations. Presented at: 34th Conference on Neural Information Processing Systems (NeurIPS 2020); Dec 6-12, 2020; Vancouver, Canada. URL: https:\/\/proceedings.neurips.cc\/paper\/2020\/hash\/92d1e1eb1cd6f9fba3227870bb6d7f07-Abstract.html [Accessed 14-07-2022]"},{"key":"R20","unstructured":"Radford A Kim JW Xu T Brockman G McLeavey C Sutskever I . Robust speech recognition via large-scale weak supervision. Presented at: ICML\u201923: Proceedings of the 40th International Conference on Machine Learning; Jul 23-29, 2023; Honolulu, Hawaii, USA. URL: https:\/\/dl.acm.org\/doi\/10.5555\/3618408.3619590 [Accessed 10-09-2026]"},{"key":"R21","doi-asserted-by":"crossref","unstructured":"Warner B Chaffin A Clavi\u00e9 B et al. Smarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In: Che W Nabende J Shutova E Pilehvar MT , editors. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics; 2025:2526-2547. [doi: 10.18653\/v1\/2025.acl-long.127]","DOI":"10.18653\/v1\/2025.acl-long.127"},{"issue":"6","key":"R22","doi-asserted-by":"publisher","first-page":"601","DOI":"10.3109\/s10165-008-0092-6","article-title":"Logistic regression models for predicting physical and mental health-related quality of life in rheumatoid arthritis patients","volume":"18","author":"Alishiri","journal-title":"Mod Rheumatol"},{"key":"R23","volume":"13","author":"Biau","journal-title":"J Mach Learn Res"},{"key":"R24","volume":"12","author":"Pedregosa","journal-title":"J Mach Learn Res"},{"key":"R25","doi-asserted-by":"crossref","unstructured":"Sch\u00f6lkopf B Smola AJ . Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. The MIT Press; 2001. [doi: 10.7551\/mitpress\/4175.001.0001]","DOI":"10.7551\/mitpress\/4175.001.0001"},{"issue":"1","key":"R26","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1080\/00401706.1970.10488634","article-title":"Ridge regression: biased estimation for nonorthogonal problems","volume":"12","author":"Hoerl","journal-title":"Technometrics"},{"issue":"5786","key":"R27","doi-asserted-by":"publisher","first-page":"504","DOI":"10.1126\/science.1127647","article-title":"Reducing the dimensionality of data with neural networks","volume":"313","author":"Hinton","journal-title":"Science"},{"key":"R28","doi-asserted-by":"crossref","unstructured":"Wang N Cao Y Hao S Shao Z Subbalakshmi KP . Modular multi-modal attention network for Alzheimer\u2019s disease detection using patient audio and language data. Presented at: Interspeech 2021; Aug 30 to Sep 3, 2021; Brno, Czechia. [doi: 10.21437\/Interspeech.2021-2024]","DOI":"10.21437\/Interspeech.2021-2024"},{"key":"R29","doi-asserted-by":"crossref","unstructured":"Zhu Y Obyat A Liang X Batsis JA Roth RM . WavBERT: exploiting semantic and non-semantic speech using wav2vec and BERT for dementia detection. Presented at: Interspeech 2021; Aug 30 to Sep 3, 2021; Brno, Czechia. [doi: 10.21437\/Interspeech.2021-332]","DOI":"10.21437\/Interspeech.2021-332"},{"issue":"1","key":"R30","doi-asserted-by":"publisher","first-page":"96","DOI":"10.4218\/etrij.2023-0356","article-title":"Alzheimer\u2019s disease recognition from spontaneous speech using large language models","volume":"46","author":"Bang","journal-title":"ETRI Journal"},{"key":"R31","doi-asserted-by":"crossref","unstructured":"Shao Y Fang T . Alzheimer\u2019s disease detection using co-attention mechanism for acoustic and ASR-transcribed text features. Presented at: Interspeech 2025; Aug 17-21, 2025; Rotterdam, The Netherlands. [doi: 10.21437\/Interspeech.2025-219]","DOI":"10.21437\/Interspeech.2025-219"},{"issue":"Pt 12","key":"R32","doi-asserted-by":"publisher","first-page":"3727","DOI":"10.1093\/brain\/awt269","article-title":"Connected speech as a marker of disease progression in autopsy-proven Alzheimer\u2019s disease","volume":"136","author":"Ahmed","journal-title":"Brain"},{"key":"R33","doi-asserted-by":"publisher","DOI":"10.37349\/emed.2020.00028","article-title":"Identification of digital voice biomarkers for cognitive health","volume":"1","author":"Lin","journal-title":"Explor Med"},{"key":"R34","unstructured":"Radford A Kim JW Hallacy C et al. Learning transferable visual models from natural language supervision. Presented at: Proceedings of the 38th International Conference on Machine Learning; Jul 18-24, 2021. URL: https:\/\/proceedings.mlr.press\/v139\/radford21a\/radford21a.pdf [Accessed 10-09-2026]"},{"key":"R35","doi-asserted-by":"publisher","DOI":"10.1145\/3777482","article-title":"Evaluating picture description speech for dementia detection using image-text alignment","author":"Zhu","journal-title":"ACM Trans Comput Healthcare"},{"key":"R36","unstructured":"DementiaBank. TalkBank. URL: https:\/\/talkbank.org\/dementia\/ [Accessed 10-09-2026]"}],"container-title":["JMIR Aging"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,21]],"date-time":"2026-09-21T18:21:55Z","timestamp":1790014915000},"score":1,"resource":{"primary":{"URL":"https:\/\/aging.jmir.org\/2026\/1\/e93279"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,21]]},"references-count":36,"URL":"https:\/\/doi.org\/10.2196\/93279","relation":{},"ISSN":["2561-7605"],"issn-type":[{"value":"2561-7605","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,21]]},"article-number":"v9i3e93279"}}