{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T19:18:34Z","timestamp":1788290314257,"version":"build-2803163510"},"reference-count":63,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T00:00:00Z","timestamp":1788220800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T00:00:00Z","timestamp":1788220800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Commun Med"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>A patient undergoes multiple examinations in each hospital stay, where each provides different facets of the health status. These assessments include temporal data with varying sampling rates, discrete single-point measurements, therapeutic interventions such as medication administration, and images. While physicians are able to process and integrate diverse modalities intuitively, neural networks need specific modeling for each modality complicating the training procedure.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>\n                      We demonstrate that this complexity can be significantly reduced by visualizing all information as images along with unstructured text and subsequently training a conventional vision-text transformer. Our approach,\n                      <jats:bold>Vi<\/jats:bold>\n                      sion\n                      <jats:bold>T<\/jats:bold>\n                      ransformer for\n                      <jats:bold>i<\/jats:bold>\n                      rregular sampled\n                      <jats:bold>M<\/jats:bold>\n                      ulti-modal\n                      <jats:bold>M<\/jats:bold>\n                      easurements (\n                      <jats:bold>ViTiMM<\/jats:bold>\n                      ), simplifies data preprocessing and modeling by unifying clinical measurements, medications, X-ray images, and electrocardiography scans into a single visual representation.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>ViTiMM outperforms current state-of-the-art methods in predicting in-hospital mortality, phenotyping, and decompensation on two datasets, the MIMIC-IV and COVID Data for Shared Learning (CDSL) dataset.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>\n                      We hope our work inspires advancements in multi-modal medical AI by reducing the training complexity to (visual) prompt engineering, thus lowering entry barriers and enabling no-code solutions for training. The source code is publicly available at\n                      <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/github.com\/Cardio-AI\/ViTiMM\" ext-link-type=\"uri\">https:\/\/github.com\/Cardio-AI\/ViTiMM<\/jats:ext-link>\n                      .\n                    <\/jats:p>\n                  <\/jats:sec>","DOI":"10.1038\/s43856-026-01817-x","type":"journal-article","created":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T18:42:21Z","timestamp":1788288141000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Visual prompt engineering for multimodal and irregularly sampled medical data"],"prefix":"10.1038","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0804-5794","authenticated-orcid":false,"given":"Malte","family":"T\u00f6lle","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohamad","family":"Scharaf","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samantha","family":"Fischer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2683-1369","authenticated-orcid":false,"given":"Christoph","family":"Reich","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Silav","family":"Zeid","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9468-6311","authenticated-orcid":false,"given":"Christoph","family":"Dieterich","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0741-2633","authenticated-orcid":false,"given":"Benjamin","family":"Meder","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7611-378X","authenticated-orcid":false,"given":"Norbert","family":"Frey","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4413-9752","authenticated-orcid":false,"given":"Philipp","family":"Wild","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sandy","family":"Engelhardt","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,9,1]]},"reference":[{"key":"1817_CR1","doi-asserted-by":"publisher","first-page":"1773","DOI":"10.1038\/s41591-022-01981-2","volume":"28","author":"JN Acosta","year":"2022","unstructured":"Acosta, J. N., Falcone, G. J., Rajpurkar, P. & Topol, E. J. Multimodal biomedical AI. Nat. Med. 28, 1773\u20131784 (2022).","journal-title":"Nat. Med."},{"key":"1817_CR2","doi-asserted-by":"crossref","unstructured":"Bluethgen, C. et al. A vision\u2013language foundation model for the generation of realistic chest X-ray images. Nat. Biomed. Eng. 9, 494\u2013506 (2024).","DOI":"10.1038\/s41551-024-01246-y"},{"key":"1817_CR3","doi-asserted-by":"publisher","first-page":"e1009224","DOI":"10.1371\/journal.pcbi.1009224","volume":"17","author":"R Duan","year":"2021","unstructured":"Duan, R. et al. Evaluation and comparison of multi-omics data integration methods for cancer subtyping. PLoS Comput. Biol. 17, e1009224 (2021).","journal-title":"PLoS Comput. Biol."},{"key":"1817_CR4","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-00341-z","volume":"3","author":"S-C Huang","year":"2020","unstructured":"Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I. & Lungren, M. P. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. npj Digit. Med. 3, 136 (2020).","journal-title":"npj Digit. Med."},{"key":"1817_CR5","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-020-78888-w","volume":"10","author":"S-C Huang","year":"2020","unstructured":"Huang, S.-C., Pareek, A., Zamanian, R., Banerjee, I. & Lungren, M. P. Multimodal fusion with deep neural networks for leveraging ct imaging and electronic health record: a case-study in pulmonary embolism detection. Sci. Rep. 10, 22147 (2020).","journal-title":"Sci. Rep."},{"key":"1817_CR6","doi-asserted-by":"publisher","first-page":"1060","DOI":"10.1093\/jamia\/ocac030","volume":"29","author":"S Jabbour","year":"2022","unstructured":"Jabbour, S., Fouhey, D., Kazerooni, E., Wiens, J. & Sjoding, M. W. Combining chest x-rays and electronic health record (EHR) data using machine learning to diagnose acute respiratory failure. J. Am. Med. Inform. Assoc. 29, 1060\u20131068 (2022).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"1817_CR7","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1038\/s41586-023-05881-4","volume":"616","author":"M Moor","year":"2023","unstructured":"Moor, M. et al. Foundation models for generalist medical artificial intelligence. Nature 616, 259\u2013265 (2023).","journal-title":"Nature"},{"key":"1817_CR8","doi-asserted-by":"publisher","unstructured":"Touvron, H. et al. LLaMA: open and efficient foundation language models. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2302.13971 (2023).","DOI":"10.48550\/arXiv.2302.13971"},{"key":"1817_CR9","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-022-00712-8","volume":"5","author":"A Kline","year":"2022","unstructured":"Kline, A. et al. Multimodal machine learning in precision health: a scoping review. npj Digit. Med. 5, 171 (2022).","journal-title":"npj Digit. Med."},{"key":"1817_CR10","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1016\/j.inffus.2014.10.004","volume":"25","author":"M Nejati","year":"2015","unstructured":"Nejati, M., Samavi, S. & Shirani, S. Multi-focus image fusion using dictionary-based sparse representation. Inf. Fusion 25, 72\u201384 (2015).","journal-title":"Inf. Fusion"},{"key":"1817_CR11","doi-asserted-by":"publisher","first-page":"1308","DOI":"10.1007\/s10278-021-00554-y","volume":"35","author":"N Tawfik","year":"2022","unstructured":"Tawfik, N., Elnemr, H. A., Fakhr, M., Dessouky, M. I. & El-Samie, F. E. A. Multimodal medical image fusion using stacked auto-encoder in nsct domain. J. Digit. Imaging 35, 1308\u20131325 (2022).","journal-title":"J. Digit. Imaging"},{"key":"1817_CR12","doi-asserted-by":"publisher","first-page":"956","DOI":"10.1097\/RLU.0000000000002810","volume":"44","author":"SH Hyun","year":"2019","unstructured":"Hyun, S. H., Ahn, M. S., Koh, Y. W. & Lee, S. J. A machine-learning approach using pet-based radiomics to predict the histological subtypes of lung cancer. Clin. Nucl. Med. 44, 956\u2013960 (2019).","journal-title":"Clin. Nucl. Med."},{"key":"1817_CR13","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-018-37387-9","volume":"9","author":"D Nie","year":"2019","unstructured":"Nie, D. et al. Multi-channel 3d deep feature learning for survival time prediction of brain tumor patients using multi-modal neuroimages. Sci. Rep. 9, 1103 (2019).","journal-title":"Sci. Rep."},{"key":"1817_CR14","doi-asserted-by":"crossref","unstructured":"McKeen, K. et al. ECG-FM: an open electrocardiogram foundation model. JAMIA Open 8, ooaf122 (2024).","DOI":"10.1093\/jamiaopen\/ooaf122"},{"key":"1817_CR15","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-020-15432-4","volume":"11","author":"AH Ribeiro","year":"2020","unstructured":"Ribeiro, A. H. et al. Automatic diagnosis of the 12-lead ECG using a deep neural network. Nat. Commun. 11, 1760 (2020).","journal-title":"Nat. Commun."},{"key":"1817_CR16","unstructured":"Vaswani, A. et al. Attention is all you need. In Advances in Neural Information Processing Systems (NeuRIPS) vol. 30 (2017)."},{"key":"1817_CR17","unstructured":"Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Proc. International Conference on Learning Representations https:\/\/openreview.net\/forum?id=YicbFdNTTy (2021)."},{"key":"1817_CR18","unstructured":"Jaegle, A. et al. Perceiver IO: a general architecture for structured inputs and outputs. In Proc. International Conference on Learning Representations (ICLR) (2021)."},{"key":"1817_CR19","unstructured":"Jaegle, A. et al. Perceiver: general perception with iterative attention. In Proc. International Conference on Machine Learning (ICML) (PMLR, 2021)."},{"key":"1817_CR20","doi-asserted-by":"crossref","unstructured":"Dar, S. U. H. et al. Unconditional latent diffusion models memorize patient imaging data: implications for openly sharing synthetic data. Nat. Biomed. Eng. 10, 458\u2013472(2024).","DOI":"10.1038\/s41551-025-01468-8"},{"key":"1817_CR21","unstructured":"Radford, A. et al. Learning transferable visual models from natural language supervision. In Proc. International Conference on Machine Learning, 8748\u20138763 (PMLR, 2021)."},{"key":"1817_CR22","doi-asserted-by":"publisher","DOI":"10.1038\/s41597-022-01899-x","volume":"10","author":"AEW Johnson","year":"2023","unstructured":"Johnson, A. E. W. et al. Mimic-iv, a freely accessible electronic health record dataset. Sci. Data 10, 1 (2023).","journal-title":"Sci. Data"},{"key":"1817_CR23","doi-asserted-by":"crossref","unstructured":"Li, Z., Li, S. & Yan, X. Time Series as Images: Vision Transformer for Irregularly Sampled Time Series. In Proc. 37th Conference on Neural Information Processing Systems (NeurIPS)https:\/\/openreview.net\/forum?id=ZmeAoWQqe0 (2023).","DOI":"10.52202\/075280-2139"},{"key":"1817_CR24","unstructured":"Lipton, Z. C., Kale, D. & Wetzel, R. Directly modeling missing data in sequences with RNNs: improved classification of clinical time series. In Proc. 1st Machine Learning for Healthcare Conference, vol. 56, 253\u2013270 https:\/\/proceedings.mlr.press\/v56\/Lipton16.html (2016)."},{"key":"1817_CR25","unstructured":"Zhang, X., Zeman, M., Tsiligkaridis, T. & Zitnik, M. Graph-guided network for irregularly sampled multivariate time series. In Proc. International Conference on Learning Representations (ICLR) (PMLR, 2022)."},{"key":"1817_CR26","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Yang, X., Ivy, J. & Chi, M. Attain: attention-based time-aware LSTM networks for disease progression modeling. In Proc. International Joint Conferences on Artificial Intelligence Organization (IJCAI), 4369\u20134375 (ACM, 2019).","DOI":"10.24963\/ijcai.2019\/607"},{"key":"1817_CR27","doi-asserted-by":"crossref","unstructured":"Liu, J. et al. CLIP-driven universal model for organ segmentation and tumor detection. In Proc. International Conference on Computer Vision (ICCV) (IEEE, 2023).","DOI":"10.1109\/ICCV51070.2023.01934"},{"key":"1817_CR28","first-page":"10666","volume":"13","author":"F Khader","year":"2023","unstructured":"Khader, F. et al. Medical transformer for multimodal survival prediction in intensive care: integration of imaging and non-imaging data. Nat. Sci. Rep. 13, 10666 (2023).","journal-title":"Nat. Sci. Rep."},{"key":"1817_CR29","unstructured":"Hayat, N., Geras, K. J. & Shamout, F. E. MedFuse: multi-modal fusion with clinical time-series data and chest X-ray images. In Proc. 7th Machine Learning for Healthcare Conference, vol. 182, 479\u2013503 https:\/\/proceedings.mlr.press\/v182\/hayat22a.html (2022)."},{"key":"1817_CR30","doi-asserted-by":"publisher","first-page":"104796","DOI":"10.1016\/j.jbi.2025.104796","volume":"164","author":"M Wang","year":"2025","unstructured":"Wang, M., Fan, S., Li, Y., Xie, Z. & Chen, H. Missing-modality enabled multi-modal fusion architecture for medical data. J. Biomed. Inform. 164, 104796 (2025).","journal-title":"J. Biomed. Inform."},{"key":"1817_CR31","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K. Q. Densely connected convolutional networks. In Proc. Conference on Computer Vision and Pattern Recognition (CVPR), 4700\u20134708 (Scientific Research, 2017).","DOI":"10.1109\/CVPR.2017.243"},{"key":"1817_CR32","first-page":"1","volume":"3","author":"Y Gu","year":"2021","unstructured":"Gu, Y. et al. Domain-specific language model pretraining for biomedical natural language processing. Trans. Comput. Healthc. 3, 1\u201323 (2021).","journal-title":"Trans. Comput. Healthc."},{"key":"1817_CR33","doi-asserted-by":"publisher","unstructured":"Ritor\u00e9, \u00c1., Oprescu, A. M., Estirado Bronchalo, A. & Armengol de la Hoz, M. \u00c1. COVID data for shared learning (CDSL): a comprehensive, multimodal COVID-19 dataset from HM hospitales (version 1.0.0). PhysioNet https:\/\/doi.org\/10.13026\/1176-6c44 (2024).","DOI":"10.13026\/1176-6c44"},{"key":"1817_CR34","doi-asserted-by":"publisher","first-page":"e215","DOI":"10.1161\/01.CIR.101.23.e215","volume":"101","author":"AL Goldberger","year":"2000","unstructured":"Goldberger, A. L. et al. PhysioBank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. Circulation 101, e215\u2013e220 (2000).","journal-title":"Circulation"},{"key":"1817_CR35","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1038\/s41597-019-0322-0","volume":"6","author":"AEW Johnson","year":"2019","unstructured":"Johnson, A. E. W. et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Nat. Sci. Data 6, 317 (2019).","journal-title":"Nat. Sci. Data"},{"key":"1817_CR36","doi-asserted-by":"publisher","unstructured":"Gow, B. et al. MIMIC-IV-ECG: diagnostic electrocardiogram matched subset (version 1.0). PhysioNet https:\/\/doi.org\/10.13026\/4nqg-sb35 (2023).","DOI":"10.13026\/4nqg-sb35"},{"key":"1817_CR37","doi-asserted-by":"publisher","unstructured":"Johnson, A., Pollard, T., Mark, R., Berkowitz, S. & Horng, S. MIMIC-CXR Database (version 2.1.0). PhysioNet https:\/\/doi.org\/10.13026\/4jqj-jw95 (2024).","DOI":"10.13026\/4jqj-jw95"},{"key":"1817_CR38","unstructured":"Gupta, M. et al. An extensive data processing pipeline for MIMIC-IV. In Machine Learning for Health symposium, vol. 193, 311\u2013325 https:\/\/proceedings.mlr.press\/v193\/gupta22a.html (2022)."},{"key":"1817_CR39","doi-asserted-by":"publisher","first-page":"707","DOI":"10.1007\/BF01709751","volume":"22","author":"J-L Vincent","year":"1996","unstructured":"Vincent, J.-L. et al. The sofa (sepsis-related organ failure assessment) score to describe organ dysfunction\/failure. Intensive Care Med. 22, 707\u2013710 (1996).","journal-title":"Intensive Care Med."},{"key":"1817_CR40","doi-asserted-by":"publisher","first-page":"1297","DOI":"10.1097\/01.CCM.0000215112.84523.F0","volume":"34","author":"JE Zimmerman","year":"2006","unstructured":"Zimmerman, J. E., Kramer, A. A., McNair, D. S. & Malila, F. M. Acute physiology and chronic health evaluation (APACHE) IV: hospital mortality assessment for today\u2019s critically ill patients. Crit. Care Med. 34, 1297\u20131310 (2006).","journal-title":"Crit. Care Med."},{"key":"1817_CR41","doi-asserted-by":"crossref","first-page":"S453","DOI":"10.1016\/S0735-1097(22)04181-X","volume":"79","author":"J Rubin","year":"2022","unstructured":"Rubin, J., Parvaneh, S., Rahman, A., Conroy, B. & Babaeizadeh, S. Cardiac arrhythmia detection and classification using electrocardiogram signals. J. Am. Coll. Cardiol. 79, S453 (2022).","journal-title":"J. Am. Coll. Cardiol."},{"key":"1817_CR42","doi-asserted-by":"publisher","unstructured":"Rajpurkar, P. et al. Chexnet: radiologist-level pneumonia detection on chest X-rays with deep learning. Preprint at https:\/\/doi.org\/10.48550\/arXiv.1711.05225 (2017).","DOI":"10.48550\/arXiv.1711.05225"},{"key":"1817_CR43","first-page":"2187","volume":"34","author":"JA Russell","year":"2008","unstructured":"Russell, J. A. Vasopressor therapy in critically ill patients with shock. Intensive Care Med. 34, 2187\u20132200 (2008).","journal-title":"Intensive Care Med."},{"key":"1817_CR44","doi-asserted-by":"publisher","DOI":"10.1038\/s41597-019-0103-9","volume":"6","author":"H Harutyunyan","year":"2019","unstructured":"Harutyunyan, H., Khachatrian, H., Kale, D. C., Ver Steeg, G. & Galstyan, A. Multitask learning and benchmarking with clinical time series data. Sci. Data 6, 96 (2019).","journal-title":"Sci. Data"},{"key":"1817_CR45","unstructured":"Lipton, Z. C., Kale, D. C., Elkan, C. & Wetzel, R. Learning to diagnose with LSTM recurrent neural networks. In Proc. 4th International Conference on Learning Representations (ICLR, 2016)."},{"key":"1817_CR46","doi-asserted-by":"publisher","first-page":"368","DOI":"10.1097\/CCM.0000000000001571","volume":"44","author":"MM Churpek","year":"2016","unstructured":"Churpek, M. M. et al. Multicenter comparison of machine learning methods and conventional regression for predicting clinical deterioration on the wards. Crit. Care Med. 44, 368\u2013374 (2016).","journal-title":"Crit. Care Med."},{"key":"1817_CR47","doi-asserted-by":"publisher","first-page":"299ra122","DOI":"10.1126\/scitranslmed.aab3719","volume":"7","author":"KE Henry","year":"2015","unstructured":"Henry, K. E., Hager, D. N., Pronovost, P. J. & Saria, S. A targeted real-time early warning score (TREWScore) for septic shock. Sci. Transl. Med. 7, 299ra122 (2015).","journal-title":"Sci. Transl. Med."},{"key":"1817_CR48","doi-asserted-by":"publisher","unstructured":"Liu, Y. et al. RoBERTa: a robustly optimized BERT pretraining approach. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.1907.11692 (2019).","DOI":"10.48550\/arXiv.1907.11692"},{"key":"1817_CR49","doi-asserted-by":"crossref","unstructured":"Liu, Z. et al. Swin transformer: hierarchical vision transformer using shifted windows. In Proceedings of the IEEE\/CVF International Conference on Computer Vision, 10012\u201310022 (2021).","DOI":"10.1109\/ICCV48922.2021.00986"},{"key":"1817_CR50","doi-asserted-by":"crossref","unstructured":"T\u00f6lle, M. et al. Real world federated learning with a knowledge distilled transformer for cardiac CT imaging. npj Digit.Med. 8, 88 (2025).","DOI":"10.1038\/s41746-025-01434-3"},{"key":"1817_CR51","doi-asserted-by":"crossref","unstructured":"Deng, J. et al. ImageNet: a large-scale hierarchical image database. In Proc. Conference on Computer Vision and Pattern Recognition (CVPR), 248\u2013255 (IEEE, 2009).","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"1817_CR52","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proc. Conference on Computer Vision and Pattern Recognition (CVPR), 770\u2013778 (IEEE, 2016).","DOI":"10.1109\/CVPR.2016.90"},{"key":"1817_CR53","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","volume":"9","author":"S Hochreiter","year":"1997","unstructured":"Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735\u20131780 (1997).","journal-title":"Neural Comput."},{"key":"1817_CR54","doi-asserted-by":"publisher","unstructured":"Anthony, L. F. W., Kanding, B. & Selvan, R. Carbontracker: tracking and predicting the carbon footprint of training deep learning models. Presented at ICML Workshop on Challenges in Deploying and Monitoring Machine Learning Systems Preprint at https:\/\/doi.org\/10.48550\/arXiv.2007.03051 (2020).","DOI":"10.48550\/arXiv.2007.03051"},{"key":"1817_CR55","doi-asserted-by":"crossref","first-page":"2518","DOI":"10.1097\/CCM.0b013e3181fb7b1c","volume":"39","author":"M Hravnak","year":"2011","unstructured":"Hravnak, M. et al. Real alerts and artifact in the intensive care unit: observational study of patient physiological variables and monitor alarms. Crit. Care Med. 39, 2518\u20132526 (2011).","journal-title":"Crit. Care Med."},{"key":"1817_CR56","unstructured":"Lipton, Z. C., Kale, D. & Wetzel, R. Directly modeling missing data in sequences with RNNs: Improved classification of clinical time series. In Proc. Machine Learning for Healthcare Conference 253\u2013270 (PMLR, 2016)."},{"key":"1817_CR57","doi-asserted-by":"crossref","unstructured":"Marlin, B. M., Kale, D. C., Khemani, R. G. & Wetzel, R. C. Unsupervised pattern discovery in electronic health care data using probabilistic clustering models. In Proc. 2nd ACM SIGHIT International Health Informatics Symposium 389\u2013398 (ACM, 2012).","DOI":"10.1145\/2110363.2110408"},{"key":"1817_CR58","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-018-24271-9","volume":"8","author":"Z Che","year":"2018","unstructured":"Che, Z., Purushotham, S., Cho, K., Sontag, D. & Liu, Y. Recurrent neural networks for multivariate time series with missing values. Sci. Rep. 8, 6085 (2018).","journal-title":"Sci. Rep."},{"key":"1817_CR59","doi-asserted-by":"crossref","unstructured":"McDermott, M. B. A., Nestor, B., Argaw, P. & Kohane, I. Event stream GPT: a data pre-processing and modeling library for generative, pre-trained transformers over continuous-time sequences of complex events. In Advances in Neural Information Processing Systems (NeuRIPS) (2023).","DOI":"10.52202\/075280-1057"},{"key":"1817_CR60","doi-asserted-by":"publisher","first-page":"837","DOI":"10.2307\/2531595","volume":"44","author":"ER DeLong","year":"1988","unstructured":"DeLong, E. R., DeLong, D. M. & Clarke-Pearson, D. L. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44, 837\u2013845 (1988).","journal-title":"Biometrics"},{"key":"1817_CR61","doi-asserted-by":"publisher","first-page":"174107","DOI":"10.1109\/ACCESS.2024.3497668","volume":"12","author":"A Rasekh","year":"2024","unstructured":"Rasekh, A. et al. Robust fusion of time series and image data for improved multimodal clinical prediction. IEEE ACCESS 12, 174107\u2013174121 (2024).","journal-title":"IEEE ACCESS"},{"key":"1817_CR62","doi-asserted-by":"publisher","first-page":"e26772","DOI":"10.1016\/j.heliyon.2024.e26772","volume":"10","author":"Y Wang","year":"2024","unstructured":"Wang, Y., Yin, C. & Zhang, P. Multimodal risk prediction with physiological signals, medical images and clinical notes. Heliyon 10, e26772 (2024).","journal-title":"Heliyon"},{"key":"1817_CR63","doi-asserted-by":"publisher","unstructured":"Gow, B. et al. MIMIC-IV-ECHO: Echocardiogram Matched Subset (version 0.1). PhysioNet https:\/\/doi.org\/10.13026\/ef48-v217 (2023).","DOI":"10.13026\/ef48-v217"}],"container-title":["Communications Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s43856-026-01817-x","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s43856-026-01817-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s43856-026-01817-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T19:02:30Z","timestamp":1788289350000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s43856-026-01817-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,1]]},"references-count":63,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["1817"],"URL":"https:\/\/doi.org\/10.1038\/s43856-026-01817-x","relation":{},"ISSN":["2730-664X"],"issn-type":[{"value":"2730-664X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,1]]},"assertion":[{"value":"13 March 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 July 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 September 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"N.F. reports speaker honoraria, presentations or advisory board consultations from AstraZeneca, Bayer AG, Boehringer Ingelheim, Novartis, Pfizer, Daiichi Sankyo Deutschland. S.E. reports speaker honorarium from Boehringer Ingelheim and Abbott. P.W. reports grants from Bayer AG, non-financial grants from Philips Medical Systems, grants and consulting fees from Boehringer Ingelheim, grants and consulting fees from Novartis Pharma, grants and consulting fees from Sanofi-Aventis, grants, consulting and lecturing fees from Bayer Health Care, grants and consulting fees from Daiichi Sankyo Europe, lecturing fees from Pfizer Pharma, lecturing fees from Bristol Myers Squibb, consulting fees from Astra Zeneca, consulting fees and non-financial support from Diasorin and non-financial support from IEM. P.W. is funded by the European Regional Development Fund and the Ministry of Science and Health of the State of Rhineland-Palatinate (EFRE\/REACT-EU, Grant Nos. 84007232 and 84009735), the Federal Ministry of Education and Research (BMBF 01EO1503, 81Z0210103, 81Z0210102, 03ZU1202AA, 03ZU1202CD, 03ZU1202DB, 03ZU1202JC, 03ZU1202KB, 03ZU1202LB, 03ZU1202MB, 03ZU1202OA), Ministry of Science and Health of the State of Rhineland Palatinate (MWG RLP 724-0010#2021\/0030-1501), the ReALity Initiative of the Life Sciences of the Johannes Gutenberg University Mainz for the establishment of a cell bank, the National Research Network University Medicine on Covid-19 (\"NaFoUniMedCovid19\u201d, Grant No. 01KX2021) B-FAST, and the Federal Institute for Occupational Safety and Health (BAuA, F2447\/537727) outside the present work. P.S.W. is principal investigator of the German Center for Cardiovascular Research (DZHK), principal investigator of the DIASyM research core (BMBF DIASyM research core (BMBF 161L0217A, 031L0217A, 16LW0241K), of the future cluster curATime (BMBF 03ZU1202AA, 03ZU1202CD, 03ZU1202DB, 03ZU1202JC, 03ZU1202KB, 03ZU1202LB, 03ZU1202MB, and 03ZU1202OA) and principal investigator of EPIC-AI (BMBF 01EQ2401A). None are related to the content of the manuscript. The other authors declare no competing interests.","order":1,"name":"Ethics","label":"Competing interests","group":{"name":"EthicsHeading","label":"Ethics"}}],"article-number":"474"}}