{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,4]],"date-time":"2026-08-04T06:34:36Z","timestamp":1785825276083,"version":"3.56.0"},"reference-count":44,"publisher":"Wiley","issue":"6","license":[{"start":{"date-parts":[[2025,3,25]],"date-time":"2025-03-25T00:00:00Z","timestamp":1742860800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62402458"],"award-info":[{"award-number":["62402458"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["aapm.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Medical Physics"],"published-print":{"date-parts":[[2025,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>As one of the most prevalent neurodegenerative disorders, Alzheimer's disease (AD) severely impacts human thinking and behavior. Early and accurate prediction of cognitive decline is crucial for timely AD intervention. However, most existing prognostic methods hardly explore the underlying association among longitudinal data from different modalities in disease progression, thus the predictive ability of current models is still quite\u00a0limited.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Purpose<\/jats:title>\n                    <jats:p>\n                      We propose the unifying\n                      <jats:bold>M<\/jats:bold>\n                      ulti\u2010\n                      <jats:bold>M<\/jats:bold>\n                      odality fusion with\n                      <jats:bold>DU<\/jats:bold>\n                      al\u2010g\n                      <jats:bold>R<\/jats:bold>\n                      anularity\n                      <jats:bold>A<\/jats:bold>\n                      lignment framework (MM\u2010DURA) to simultaneously model longitudinal correlations and modalities interactions for cognitive assessment forecasting. Our proposed framework leverages temporal MRI scans, time\u2010aligned clinical diagnostics, and genomic data as inputs to forecast multiple cognitive assessment\u00a0scores.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We propose a novel coarse\u2010to\u2010fine feature representation learning approach to ascertain the congruence between modalities at both the subject and visit granularities. This method ensures the alignment of multimodal data pertaining to individual subjects and captures the temporal progression of these modalities. Additionally, we design a hierarchical multimodality fusion (HMF) block that can effectively exploit the interrelationships and dependencies among modalities. Lastly, we employ an LSTM\u2010based regression head with the fused multimodality embedding as input to forecast the future status of cognitive\u00a0ability.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We validate our method on the public Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset and investigate the optimal hierarchical structure for modality fusion. The whole dataset includes 707 subjects participating in the ADNI1, ADNIGO, and ADNI2 studies. All subjects underwent longitudinal examinations with an average study period of approximately 14 months. The subject\u2010level split for training, validation, and testing sets is 0.75:0.05:0.20. The proposed MM\u2010DURA framework demonstrates superior performance, achieving remarkable RMSE values of 1.099 for CDRSB, 5.601 for ADAS\u2010Cog, 2.051 for MMSE, 6.504 for RAVLT, and 3.447 for FAQ cognitive assessments forecasting. These results outperform all six comparison methods, including two state\u2010of\u2010the\u2010art multimodal temporal modeling approaches. Comprehensive ablation experimental results affirm the effectiveness of longitudinal modeling with temporal\u2010multimodal alignment, highlighting its clinical potential for cognitive assessment prediction. Visualizations of key brain regions and SNP significance analysis also provide substantial\u00a0interpretability.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>\n                      In this work, we proposed a novel framework that unifies multimodality fusion with dual\u2010granularity alignment for cognitive assessment forecasting. Our approach utilizes a temporal\u2010multimodal consistency alignment strategy, which effectively synchronizes various modalities within a unified latent space. Furthermore, the innovative HMF block we developed capitalizes on the inherent relationships and dependencies between modalities to optimize data integration. Extensive numerical results on five cognitive assessment scores, supported by detailed visualizations demonstrate the superior performance of our approach compared to existing methods. Our code has been released, and it is available at\n                      <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/github.com\/IcecreamArtist\/MM_DURA\">https:\/\/github.com\/IcecreamArtist\/MM_DURA<\/jats:ext-link>\n                      .\n                    <\/jats:p>\n                  <\/jats:sec>","DOI":"10.1002\/mp.17767","type":"journal-article","created":{"date-parts":[[2025,3,28]],"date-time":"2025-03-28T02:56:24Z","timestamp":1743130584000},"page":"5064-5080","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Temporal\u2010multimodal consistency alignment for Alzheimer's cognitive assessment prediction"],"prefix":"10.1002","volume":"52","author":[{"given":"Xikai","family":"Yang","sequence":"first","affiliation":[{"name":"Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xilin","family":"Dang","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jinyue","family":"Cai","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jinpeng","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xi","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong China"},{"name":"Department of Computer Science and Engineering The Hong Kong University of Science and Technology Hong Kong China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pheng\u2010Ann","family":"Heng","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong China"},{"name":"Institute of Medical Intelligence and XR The Chinese University of Hong Kong Hong Kong China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2025,3,25]]},"reference":[{"key":"e_1_2_10_2_1","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooab052"},{"key":"e_1_2_10_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2011.01.050"},{"key":"e_1_2_10_4_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-018-37769-z"},{"key":"e_1_2_10_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2023.120761"},{"key":"e_1_2_10_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jalz.2016.11.007"},{"key":"e_1_2_10_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3230668"},{"key":"e_1_2_10_8_1","doi-asserted-by":"crossref","unstructured":"OuyangJ ZhaoQ AdeliE et\u00a0al.Self\u2010supervised longitudinal neighbourhood embedding. In:Medical Image Computing and Computer Assisted Intervention\u2013MICCAI 2021: 24th International Conference Strasbourg France September 27\u2013October 1 2021 Proceedings Part II.Springer;2021:80\u201089.","DOI":"10.1007\/978-3-030-87196-3_8"},{"key":"e_1_2_10_9_1","doi-asserted-by":"crossref","unstructured":"OuyangJ ZhaoQ AdeliE PengW ZaharchukG PohlKM.LSOR: longitudinally\u2010consistent self\u2010organized representation learning. In:International Conference on Medical Image Computing and Computer\u2010Assisted Intervention.Springer;2023:279\u2010289.","DOI":"10.1007\/978-3-031-43907-0_27"},{"key":"e_1_2_10_10_1","volume-title":"Fundamentals of Brain Network Analysis","author":"Fornito A","year":"2016"},{"key":"e_1_2_10_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jneumeth.2015.06.016"},{"key":"e_1_2_10_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2022.3220220"},{"key":"e_1_2_10_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jalz.2010.03.018"},{"key":"e_1_2_10_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2022.102643"},{"key":"e_1_2_10_15_1","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_2_10_16_1","unstructured":"RadfordA KimJW HallacyC et\u00a0al.Learning transferable visual models from natural language supervision. In:International Conference on Machine Learning.PMLR;2021:8748\u20108763."},{"key":"e_1_2_10_17_1","doi-asserted-by":"crossref","unstructured":"YeK TangH DaiS et\u00a0al.Bidirectional mapping with contrastive learning on multimodal neuroimaging data. In:International Conference on Medical Image Computing and Computer\u2010Assisted Intervention.Springer;2023:138\u2010148.","DOI":"10.1007\/978-3-031-43898-1_14"},{"key":"e_1_2_10_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3344998"},{"key":"e_1_2_10_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2019.05.003"},{"key":"e_1_2_10_20_1","doi-asserted-by":"publisher","DOI":"10.1142\/S0129065716500258"},{"key":"e_1_2_10_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2020.101940"},{"key":"e_1_2_10_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.nicl.2018.101645"},{"key":"e_1_2_10_23_1","doi-asserted-by":"crossref","unstructured":"FengY ChenW GuX XuX ZhangM.Multi\u2010modal semi\u2010supervised evidential recycle framework for Alzheimer's disease classification. In:International Conference on Medical Image Computing and Computer\u2010Assisted Intervention.Springer;2023:130\u2010140.","DOI":"10.1007\/978-3-031-43907-0_13"},{"key":"e_1_2_10_24_1","doi-asserted-by":"crossref","unstructured":"Aviles\u2010RiveroAI RunkelC PapadakisN KourtziZ Sch\u00f6nliebC\u2010B.Multi\u2010modal hypergraph diffusion network with dual prior for Alzheimer classification. In:International Conference on Medical Image Computing and Computer\u2010Assisted Intervention.Springer;2022:717\u2010727.","DOI":"10.1007\/978-3-031-16437-8_69"},{"key":"e_1_2_10_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cmpb.2020.105348"},{"key":"e_1_2_10_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2020.107247"},{"key":"e_1_2_10_27_1","doi-asserted-by":"publisher","DOI":"10.3390\/s22041475"},{"key":"e_1_2_10_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2021.118143"},{"key":"e_1_2_10_29_1","doi-asserted-by":"crossref","unstructured":"ChenY ZhaoW YuL.Transformer\u2010based multimodal fusion for survival prediction by integrating whole slide images clinical and genomic data. In:2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI).IEEE;2023:1\u20105.","DOI":"10.1109\/ISBI53787.2023.10230804"},{"key":"e_1_2_10_30_1","doi-asserted-by":"crossref","unstructured":"ZhouR ZhouH ChenBY ShenL ZhangY HeL.Attentive deep canonical correlation analysis for diagnosing Alzheimer's disease using multimodal imaging genetics. In:International Conference on Medical Image Computing and Computer\u2010Assisted Intervention.Springer;2023:681\u2010691.","DOI":"10.1007\/978-3-031-43895-0_64"},{"key":"e_1_2_10_31_1","unstructured":"ZhangY JiangH MiuraY ManningCD LanglotzCP.Contrastive learning of medical visual representations from paired images and text. In:Machine Learning for Healthcare Conference.PMLR;2022:2\u201025."},{"key":"e_1_2_10_32_1","doi-asserted-by":"crossref","unstructured":"ZhouR ZhouH ShenL ChenBY ZhangY HeL.Integrating multimodal contrastive learning and cross\u2010modal attention for Alzheimer's disease prediction in brain imaging genetics. In:2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).IEEE;2023:1806\u20101811.","DOI":"10.1109\/BIBM58861.2023.10385864"},{"key":"e_1_2_10_33_1","doi-asserted-by":"publisher","DOI":"10.4316\/AECE.2020.02013"},{"key":"e_1_2_10_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/42.668698"},{"key":"e_1_2_10_35_1","doi-asserted-by":"publisher","DOI":"10.1006\/nimg.2000.0730"},{"key":"e_1_2_10_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.dadm.2019.01.005"},{"key":"e_1_2_10_37_1","unstructured":"ChenS MaK ZhengY.Med3D: Transfer Learning for 3D Medical Image Analysis. arXiv preprint arXiv:1904.00625;2019."},{"key":"e_1_2_10_38_1","doi-asserted-by":"crossref","unstructured":"TalebA KirchlerM MontiR LippertC.ContIG: self\u2010supervised multimodal contrastive learning for medical imaging with genetics. In:Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition.2022:20908\u201020921.","DOI":"10.1109\/CVPR52688.2022.02024"},{"key":"e_1_2_10_39_1","doi-asserted-by":"crossref","unstructured":"ZengA ChenM ZhangL XuQ.Are transformers effective for time series forecasting?In:Proceedings of the AAAI Conference on Artificial Intelligence. Vol37.2023:11121\u201011128.","DOI":"10.1609\/aaai.v37i9.26317"},{"key":"e_1_2_10_40_1","doi-asserted-by":"crossref","unstructured":"WangF LiuH.Understanding the behaviour of contrastive loss. In:Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition.2021:2495\u20102504.","DOI":"10.1109\/CVPR46437.2021.00252"},{"key":"e_1_2_10_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2006.01.021"},{"key":"e_1_2_10_42_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2010.06.010"},{"key":"e_1_2_10_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00429-010-0283-8"},{"key":"e_1_2_10_44_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41588-018-0311-9"},{"key":"e_1_2_10_45_1","unstructured":"KokhlikyanN MiglaniV MartinM et\u00a0al.Captum: a unified and generic model interpretability library for PyTorch. arXiv preprint arXiv:2009.07896;2020."}],"container-title":["Medical Physics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/mp.17767","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,12]],"date-time":"2025-11-12T08:04:40Z","timestamp":1762934680000},"score":1,"resource":{"primary":{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/10.1002\/mp.17767"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,25]]},"references-count":44,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,6]]}},"alternative-id":["10.1002\/mp.17767"],"URL":"https:\/\/doi.org\/10.1002\/mp.17767","archive":["Portico"],"relation":{},"ISSN":["0094-2405","2473-4209"],"issn-type":[{"value":"0094-2405","type":"print"},{"value":"2473-4209","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,25]]},"assertion":[{"value":"2024-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-02","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-25","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}