{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T06:35:02Z","timestamp":1783578902201,"version":"3.55.0"},"reference-count":28,"publisher":"Oxford University Press (OUP)","issue":"12","license":[{"start":{"date-parts":[[2022,9,23]],"date-time":"2022-09-23T00:00:00Z","timestamp":1663891200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,11,14]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Alzheimer\u2019s disease (AD) is the most common neurodegenerative disorder with one of the most complex pathogeneses, making effective and clinically actionable decision support difficult. The objective of this study was to develop a novel multimodal deep learning framework to aid medical professionals in AD diagnosis.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We present a Multimodal Alzheimer\u2019s Disease Diagnosis framework (MADDi) to accurately detect the presence of AD and mild cognitive impairment (MCI) from imaging, genetic, and clinical data. MADDi is novel in that we use cross-modal attention, which captures interactions between modalities\u2014a method not previously explored in this domain. We perform multi-class classification, a challenging task considering the strong similarities between MCI and AD. We compare with previous state-of-the-art models, evaluate the importance of attention, and examine the contribution of each modality to the model\u2019s performance.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>MADDi classifies MCI, AD, and controls with 96.88% accuracy on a held-out test set. When examining the contribution of different attention schemes, we found that the combination of cross-modal attention with self-attention performed the best, and no attention layers in the model performed the worst, with a 7.9% difference in F1-scores.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Our experiments underlined the importance of structured clinical data to help machine learning models contextualize and interpret the remaining modalities. Extensive ablation studies showed that any multimodal mixture of input features without access to structured clinical information suffered marked performance losses.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>This study demonstrates the merit of combining multiple input modalities via cross-modal attention to deliver highly accurate AD diagnostic decision support.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocac168","type":"journal-article","created":{"date-parts":[[2022,9,23]],"date-time":"2022-09-23T19:22:39Z","timestamp":1663960959000},"page":"2014-2022","source":"Crossref","is-referenced-by-count":148,"title":["Multimodal attention-based deep learning for Alzheimer\u2019s disease diagnosis"],"prefix":"10.1093","volume":"29","author":[{"given":"Michal","family":"Golovanevsky","sequence":"first","affiliation":[{"name":"Department of Computer Science, Brown University , Providence, Rhode Island, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carsten","family":"Eickhoff","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Brown University , Providence, Rhode Island, USA"},{"name":"Center for Biomedical Informatics, Brown University , Providence, Rhode Island, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ritambhara","family":"Singh","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Brown University , Providence, Rhode Island, USA"},{"name":"Center for Computational Molecular Biology, Brown University , Providence, Rhode Island, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,9,23]]},"reference":[{"key":"2022111608003283500_ocac168-B1","author":"Naqvi","year":"2017"},{"key":"2022111608003283500_ocac168-B2","author":"Thies","year":"2013"},{"issue":"1","key":"2022111608003283500_ocac168-B3","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1186\/s40708-019-0099-0","article-title":"Predicting the course of Alzheimer\u2019s progression","volume":"6","author":"Iddi","year":"2019","journal-title":"Brain Inform"},{"issue":"2 Dementia","key":"2022111608003283500_ocac168-B4","first-page":"404","article-title":"Mild cognitive impairment","volume":"22","author":"Petersen","year":"2016","journal-title":"Continuum (Minneap Minn)"},{"issue":"1","key":"2022111608003283500_ocac168-B5","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1016\/j.jalz.2005.06.003","article-title":"Ways toward an early diagnosis in Alzheimer\u2019s disease: the Alzheimer\u2019s disease neuroimaging initiative (ADNI)","volume":"1","author":"Mueller","year":"2005","journal-title":"Alzheimer\u2019s Dement"},{"key":"2022111608003283500_ocac168-B6","doi-asserted-by":"crossref","first-page":"108544","DOI":"10.1016\/j.jneumeth.2019.108544","article-title":"A Gaussian-based model for early detection of mild cognitive impairment using multimodal neuroimaging","volume":"333","author":"Forouzannezhad","year":"2020","journal-title":"J Neurosci Methods"},{"key":"2022111608003283500_ocac168-B7","doi-asserted-by":"crossref","first-page":"108669","DOI":"10.1016\/j.jneumeth.2020.108669","article-title":"Hippocampal atrophy based Alzheimer\u2019s disease diagnosis via machine learning methods","volume":"337","author":"Uysal","year":"2020","journal-title":"J Neurosci Methods"},{"key":"2022111608003283500_ocac168-B8","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1016\/j.jneumeth.2017.12.010","article-title":"Random forest feature selection, fusion and ensemble strategy: combining multiple morphological MRI measures to discriminate among healhy elderly, MCI, cMCI and Alzheimer\u2019s disease patients: from the Alzheimer\u2019s disease neuroimaging initiative (ADNI) database","volume":"302","author":"Dimitriadis","year":"2018","journal-title":"J Neurosci Methods"},{"key":"2022111608003283500_ocac168-B9","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1016\/j.compbiomed.2017.02.011","article-title":"Classification of Alzheimer\u2019s disease and prediction of mild cognitive impairment-to-alzheimer\u2019s conversion from structural magnetic resource imaging using feature ranking and a genetic algorithm","volume":"83","author":"Beheshti","year":"2017","journal-title":"Comput Biol Med"},{"issue":"5","key":"2022111608003283500_ocac168-B10","doi-asserted-by":"crossref","first-page":"738","DOI":"10.1111\/jon.12214","article-title":"Predicting prodromal Alzheimer\u2019s disease in subjects with mild cognitive impairment using machine learning classification of multimodal multicenter diffusion-tensor and magnetic resonance imaging data","volume":"25","author":"Dyrba","year":"2015","journal-title":"J Neuroimaging"},{"key":"2022111608003283500_ocac168-B11","doi-asserted-by":"crossref","first-page":"680","DOI":"10.1016\/j.future.2020.10.005","article-title":"Alzheimer\u2019s disease progression detection model based on an early fusion of cost-effective multimodal data","volume":"115","author":"El-Sappagh","year":"2021","journal-title":"Future Gener Comput Syst"},{"key":"2022111608003283500_ocac168-B12","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1016\/j.neuroimage.2013.03.073","article-title":"Modeling disease progression via multi-task learning","volume":"78","author":"Zhou","year":"2013","journal-title":"Neuroimage"},{"key":"2022111608003283500_ocac168-B13","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1016\/j.neucom.2018.12.018","article-title":"Ensemble of 3D densely connected convolutional network for diagnosis of mild cognitive impairment and Alzheimer\u2019s disease","volume":"333","author":"Wang","year":"2019","journal-title":"Neurocomputing"},{"key":"2022111608003283500_ocac168-B14","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.imu.2018.12.003","article-title":"Multistage classifier-based approach for Alzheimer\u2019s disease prediction and retrieval","volume":"14","author":"Kruthika","year":"2019","journal-title":"Inform Med Unlocked"},{"issue":"2","key":"2022111608003283500_ocac168-B15","doi-asserted-by":"crossref","first-page":"bbab569","DOI":"10.1093\/bib\/bbab569","article-title":"Multimodal deep learning for biomedical data fusion: a review","volume":"23","author":"Stahlschmidt","year":"2022","journal-title":"Brief Bioinform"},{"key":"2022111608003283500_ocac168-B16","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1016\/j.neucom.2020.05.087","article-title":"Multimodal multitask deep learning model for Alzheimer\u2019s disease progression detection based on time series data","volume":"412","author":"El-Sappagh","year":"2020","journal-title":"Neurocomputing"},{"key":"2022111608003283500_ocac168-B17","doi-asserted-by":"crossref","first-page":"106688","DOI":"10.1016\/j.knosys.2020.106688","article-title":"Robust hybrid deep learning models for Alzheimer\u2019s progression detection","volume":"213","author":"Abuhmed","year":"2021","journal-title":"Knowl Based Syst"},{"key":"2022111608003283500_ocac168-B18","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1016\/j.eswa.2019.04.022","article-title":"A practical computerized decision support system for predicting the severity of alzheimer\u2019s disease of an individual","volume":"130","author":"Bucholc","year":"2019","journal-title":"Expert Syst Appl"},{"issue":"1","key":"2022111608003283500_ocac168-B19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-020-74399-w","article-title":"Multimodal deep learning models for early detection of alzheimer\u2019s disease stage","volume":"11","author":"Venugopalan","year":"2021","journal-title":"Sci Rep"},{"key":"2022111608003283500_ocac168-B20","author":"Ngiam","year":"28\u2013 2, 2011: 689\u201396; ,"},{"key":"2022111608003283500_ocac168-B21","doi-asserted-by":"publisher","author":"Tan","year":"2019","DOI":"10.48550\/ARXIV.1908.07490"},{"key":"2022111608003283500_ocac168-B22","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1016\/j.neuroimage.2016.06.049","article-title":"Instantiated mixed effects modeling of Alzheimer\u2019s disease markers","volume":"142","author":"Guerrero","year":"2016","journal-title":"Neuroimage"},{"issue":"3","key":"2022111608003283500_ocac168-B23","doi-asserted-by":"crossref","first-page":"1478","DOI":"10.1016\/j.neuroimage.2012.07.059","article-title":"A computational neurodegenerative disease progression score: method and results with the Alzheimer\u2019s disease neuroimaging initiative cohort","volume":"63","author":"Jedynak","year":"2012","journal-title":"Neuroimage"},{"issue":"8","key":"2022111608003283500_ocac168-B24","doi-asserted-by":"crossref","first-page":"804","DOI":"10.1016\/S1474-4422(15)00135-0","article-title":"Longitudinal assessment of neuroimaging and clinical markers in autosomal dominant Alzheimer\u2019s disease: a prospective cohort study","volume":"14","author":"Yau","year":"2015","journal-title":"Lancet Neurol"},{"key":"2022111608003283500_ocac168-B25","doi-asserted-by":"publisher","author":"Vaswani","year":"2017","DOI":"10.48550\/ARXIV.1706.03762"},{"key":"2022111608003283500_ocac168-B26","doi-asserted-by":"crossref","first-page":"108856","DOI":"10.1016\/j.jneumeth.2020.108856","article-title":"Gaussian discriminative component analysis for early detection of Alzheimer\u2019s disease: a supervised dimensionality reduction algorithm","volume":"344","author":"Fang","year":"2020","journal-title":"J Neurosci Methods"},{"issue":"7639","key":"2022111608003283500_ocac168-B27","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1038\/nature21056","article-title":"Dermatologist-level classification of skin cancer with deep neural networks","volume":"542","author":"Esteva","year":"2017","journal-title":"Nature"},{"issue":"10","key":"2022111608003283500_ocac168-B28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1117\/1.JBO.22.10.106017","article-title":"Combining deep learning and coherent anti-Stokes Raman scattering imaging for automated differential diagnosis of lung cancer","volume":"22","author":"Weng","year":"2017","journal-title":"J Biomed Opt"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/12\/2014\/47027859\/ocac168.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/12\/2014\/47027859\/ocac168.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,16]],"date-time":"2022-11-16T08:02:15Z","timestamp":1668585735000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/29\/12\/2014\/6712292"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,23]]},"references-count":28,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2022,9,23]]},"published-print":{"date-parts":[[2022,11,14]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocac168","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,12,1]]},"published":{"date-parts":[[2022,9,23]]}}}