{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T00:29:50Z","timestamp":1782865790110,"version":"3.54.5"},"reference-count":33,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2025,6,1]],"date-time":"2025-06-01T00:00:00Z","timestamp":1748736000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,6,6]],"date-time":"2025-06-06T00:00:00Z","timestamp":1749168000000},"content-version":"vor","delay-in-days":5,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004252","name":"Qatar University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004252","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Cogn Comput"],"published-print":{"date-parts":[[2025,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>The ability to predict breast cancer metastases is essential for making effective clinical decisions and managing patients. Traditional models predominantly rely on structured clinical data, which often lacks essential contextual details, limiting their predictive accuracy. In order to address this limitation, a multi-modal approach is introduced in which structured data is transformed into unstructured text, while contextual richness is preserved. Using this text, synthetic images are generated across three key diagnostic modalities, histopathology, mammography, and ultrasound, to enhance predictive capabilities. Based on converted text data, a pre-trained diffusion model was used to generate synthetic medical images in histopathology, mammography, and ultrasound modalities. The impact of a variety of text description variants on image quality and metastasis prediction was assessed. Comprehensive tumor descriptions or a combination of histological type and differentiation status were the most effective generation strategies. A comparison was conducted between three prediction approaches: a unimodal approach, an early fusion approach based on concatenation, and the Multi Co-Guided Attention (MCGA) approach. Through mutual attention, MCGA enhances feature alignment by addressing inter- and intra-modal heterogeneity and capturing complex relationships. Unimodal and multi-modal methods were evaluated with the application of SMOTE to mitigate the impact of data imbalance. Multi-modal fusion significantly outperforms unimodal methods, especially when class imbalances are mitigated by using SMOTE. When SMOTE was applied, Ultrasound+BERT achieved the highest level of accuracy (0.90), followed by Histopathology+BERT (0.88), and Mammogram+BERT (0.88). As compared to early fusion, MCGA demonstrated better class balance and improved minority class detection. Incorporating unstructured text with synthetic imaging modalities improves the accuracy of metastasis prediction by preserving contextual information. A MCGA fusion is particularly effective in ensuring balanced class performance, particularly for rectifying class imbalances. Through this approach, the complementary strengths of textual and visual data are leveraged to overcome limitations in multi-modality integration. These results demonstrate the potential for advancing the prediction of breast cancer metastasis, offering a more robust and context-aware framework for clinical decision-making.<\/jats:p>","DOI":"10.1007\/s12559-025-10474-6","type":"journal-article","created":{"date-parts":[[2025,6,6]],"date-time":"2025-06-06T00:18:26Z","timestamp":1749169106000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Enhancing the Prediction of Breast Cancer Progression Through Multi-modal Data Transformation"],"prefix":"10.1007","volume":"17","author":[{"given":"Faseela","family":"Abdullakutty","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Younes","family":"Akbari","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Somaya","family":"Al-Maadeed","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ahmed","family":"Bouridane","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rifat","family":"Rifat Hamoudi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,6,6]]},"reference":[{"issue":"3","key":"10474_CR1","doi-asserted-by":"publisher","first-page":"2127","DOI":"10.32604\/cmes.2023.025484","volume":"136","author":"Z Zhu","year":"2023","unstructured":"Zhu Z, Wang S-H, Zhang Y-D. A survey of convolutional neural network in breast cancer. Computer modeling in engineering & sciences: CMES. 2023;136(3):2127.","journal-title":"Computer modeling in engineering & sciences: CMES."},{"key":"10474_CR2","doi-asserted-by":"publisher","first-page":"283","DOI":"10.1007\/s41666-019-00046-3","volume":"3","author":"Z Zeng","year":"2019","unstructured":"Zeng Z, Yao L, Roy A, Li X, Espino S, Clare SE, Khan SA, Luo Y. Identifying breast cancer distant recurrences from electronic health records using machine learning. Journal of Healthcare Informatics Research. 2019;3:283\u201399.","journal-title":"Journal of Healthcare Informatics Research."},{"issue":"8","key":"10474_CR3","doi-asserted-by":"publisher","first-page":"1848","DOI":"10.3390\/electronics12081848","volume":"12","author":"C Mugisha","year":"2023","unstructured":"Mugisha C, Paik I. Bridging the gap between medical tabular data and NLP predictive models: a fuzzy-logic-based textualization approach. Electronics. 2023;12(8):1848.","journal-title":"Electronics"},{"key":"10474_CR4","doi-asserted-by":"crossref","unstructured":"Abdullakutty F, Akbari Y, Al-Maadeed S, Bouridane A, Hamoudi R. Transforming tabular data for multi-modality: enhancing breast cancer metastasis prediction through data conversion. In: 2024 IEEE International Conference on Image Processing Challenges and Workshops (ICIPCW); 2024. pp. 4149\u20134155. IEEE","DOI":"10.1109\/ICIPCW64161.2024.10769174"},{"issue":"8","key":"10474_CR5","doi-asserted-by":"publisher","first-page":"848","DOI":"10.3390\/diagnostics14080848","volume":"14","author":"A Carriero","year":"2024","unstructured":"Carriero A, Groenhoff L, Vologina E, Basile P, Albera M. Deep learning in breast cancer imaging: state of the art and recent advancements in early 2024. Diagnostics. 2024;14(8):848.","journal-title":"Diagnostics."},{"key":"10474_CR6","doi-asserted-by":"publisher","first-page":"1450103","DOI":"10.3389\/fmed.2024.1450103","volume":"11","author":"F Abdullakutty","year":"2024","unstructured":"Abdullakutty F, Akbari Y, Al-Maadeed S, Bouridane A, Hamoudi R, Talaat IM. Histopathology in focus: a review on explainable multi-modal approaches for breast cancer diagnosis. Front Med. 2024;11:1450103.","journal-title":"Front. Med."},{"key":"10474_CR7","unstructured":"Devlin J, Chang M-W, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding. 2018. arXiv:1810.04805."},{"key":"10474_CR8","unstructured":"Gonz\u00e1lez-Carvajal, S., Garrido-Merch\u00e1n, E.C.: Comparing BERT against traditional machine learning text classification. 2020. arXiv:2005.13012."},{"key":"10474_CR9","doi-asserted-by":"crossref","unstructured":"Pelka, O., Koitka, S., R\u00fcckert, J., Nensa, F., Friedrich, C.M.: Radiology objects in context (ROCO): a multimodal image dataset. In: Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis: 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 3, pp. 180\u2013189. 2018. Springer","DOI":"10.1007\/978-3-030-01364-6_20"},{"issue":"12","key":"10474_CR10","doi-asserted-by":"publisher","first-page":"3321","DOI":"10.1109\/TBME.2023.3282840","volume":"70","author":"I Maouche","year":"2023","unstructured":"Maouche I, Terrissa LS, Benmohammed K, Zerhouni N. An explainable ai approach for breast cancer metastasis prediction based on clinicopathological data. IEEE Trans Biomed Eng. 2023;70(12):3321\u20139.","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"10474_CR11","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2020.104206","volume":"130","author":"Y-W Lee","year":"2021","unstructured":"Lee Y-W, Huang C-S, Shih C-C, Chang R-F. Axillary lymph node metastasis status prediction of early-stage breast cancer using convolutional neural networks. Comput Biol Med. 2021;130: 104206.","journal-title":"Comput. Biol. Med."},{"issue":"9","key":"10474_CR12","doi-asserted-by":"publisher","first-page":"13787","DOI":"10.1007\/s11042-020-10448-w","volume":"80","author":"A Alzu\u2019bi","year":"2021","unstructured":"Alzu\u2019bi A, Najadat H, Doulat W, Al-Shari O, Zhou L. Predicting the recurrence of breast cancer using machine learning algorithms. Multimedia Tools and Applications. 2021;80(9):13787\u2013800.","journal-title":"Multimedia Tools and Applications."},{"key":"10474_CR13","doi-asserted-by":"publisher","first-page":"664","DOI":"10.1007\/s12282-020-01202-z","volume":"28","author":"B-I Song","year":"2021","unstructured":"Song B-I. A machine learning-based radiomics model for the prediction of axillary lymph-node metastasis in breast cancer. Breast Cancer. 2021;28:664\u201371.","journal-title":"Breast Cancer"},{"issue":"1","key":"10474_CR14","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1186\/s12911-023-02166-8","volume":"23","author":"X Zhao","year":"2023","unstructured":"Zhao X, Jiang C. The prediction of distant metastasis risk for male breast cancer patients based on an interpretable machine learning model. BMC Med Inform Decis Mak. 2023;23(1):74.","journal-title":"BMC Med. Inform. Decis. Mak."},{"issue":"2","key":"10474_CR15","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1049\/ipr2.12652","volume":"17","author":"Y Lu","year":"2023","unstructured":"Lu Y, Zhang J, Liu X, Zhang Z, Li W, Zhou X, Li R. Prediction of breast cancer metastasis by deep learning pathology. IET Image Proc. 2023;17(2):533\u201343.","journal-title":"IET Image Proc."},{"key":"10474_CR16","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2023.106649","volume":"155","author":"E Hossain","year":"2023","unstructured":"Hossain E, Rana R, Higgins N, Soar J, Barua PD, Pisani AR, Turner K. Natural language processing in electronic health records in relation to healthcare decision-making: a systematic review. Comput Biol Med. 2023;155: 106649.","journal-title":"Comput. Biol. Med."},{"issue":"7","key":"10474_CR17","doi-asserted-by":"publisher","first-page":"1208","DOI":"10.1093\/jamia\/ocac040","volume":"29","author":"S Zhou","year":"2022","unstructured":"Zhou S, Wang N, Wang L, Liu H, Zhang R. CancerBERT: a cancer domain-specific language model for extracting breast cancer phenotypes from electronic health records. J Am Med Inform Assoc. 2022;29(7):1208\u201316.","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"10474_CR18","first-page":"77","volume-title":"Medical image generation using generative adversarial networks: a review","author":"NK Singh","year":"2021","unstructured":"Singh NK, Raza K. Medical image generation using generative adversarial networks: a review. Health informatics: A computational perspective in healthcare; 2021. p. 77\u201396."},{"key":"10474_CR19","doi-asserted-by":"crossref","unstructured":"Kazerouni A, Aghdam EK, Heidari M, Azad R, Fayyaz M, Hacihaliloglu I, Merhof D. Diffusion models in medical imaging: a comprehensive survey. Med Image Anal. 2023;88.","DOI":"10.1016\/j.media.2023.102846"},{"key":"10474_CR20","doi-asserted-by":"crossref","unstructured":"Moghadam PA, Van\u00a0Dalen S, Martin KC, Lennerz J, Yip S, Farahani H, Bashashati A. A morphology focused diffusion probabilistic model for synthesis of histopathology images. In: Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision, pp. 2000\u20132009. 2023.","DOI":"10.1109\/WACV56688.2023.00204"},{"key":"10474_CR21","doi-asserted-by":"crossref","unstructured":"Han Y, Lei Y, Shkolnikov V, Xin D, Auduong A, Barcelo S, Allebach J, Delp EJ. An ensemble method with edge awareness for abnormally shaped nuclei segmentation. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 4315\u20134325. 2023","DOI":"10.1109\/CVPRW59228.2023.00454"},{"key":"10474_CR22","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2023.122093","volume":"238","author":"Q Jin","year":"2024","unstructured":"Jin Q, Cui H, Sun C, Song Y, Zheng J, Cao L, Wei L, Su R. Inter-and intra-uncertainty based feature aggregation model for semi-supervised histopathology image segmentation. Expert Syst Appl. 2024;238: 122093.","journal-title":"Expert Syst. Appl."},{"key":"10474_CR23","doi-asserted-by":"crossref","unstructured":"Li Z, Tang Z, Hu J, Wang X, Jia D, Zhang Y. NST: a nuclei segmentation method based on transformer for gastrointestinal cancer pathological images. Biomed Signal Process Control. 2023;84.","DOI":"10.1016\/j.bspc.2023.104785"},{"key":"10474_CR24","doi-asserted-by":"crossref","unstructured":"Imtiaz T, Fattah SA, Kung S-Y. BAWGNet: boundary aware wavelet guided network for the nuclei segmentation in histopathology images. Comput Biol Med. 2023;165.","DOI":"10.1016\/j.compbiomed.2023.107378"},{"key":"10474_CR25","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2020.106881","volume":"99","author":"Q Jin","year":"2021","unstructured":"Jin Q, Cui H, Sun C, Meng Z, Su R. Cascade knowledge diffusion network for skin lesion diagnosis and segmentation. Appl Soft Comput. 2021;99: 106881.","journal-title":"Appl. Soft Comput."},{"key":"10474_CR26","doi-asserted-by":"crossref","unstructured":"Long S, Yang Z, Li Y, Qian X, Zeng K, Hao T. MAMF: a multi-level attention-based multimodal fusion model for medical visual question answering. In: International Conference on Neural Computing for Advanced Applications, pp. 202\u2013214. 2023. Springer","DOI":"10.1007\/978-981-99-5847-4_15"},{"key":"10474_CR27","doi-asserted-by":"crossref","unstructured":"Zhang Z, Wang J, Zhu R, Gong X. Efficient bilinear attention-based fusion for medical visual question answering. 2024. arXiv:2410.21000.","DOI":"10.1109\/IJCNN64981.2025.11228425"},{"key":"10474_CR28","unstructured":"Hayat N, Geras KJ, Shamout FE. MedFuse: multi-modal fusion with clinical time-series data and chest X-ray images. In: Machine Learning for Healthcare Conference, pp. 479\u2013503. 2022. PMLR"},{"key":"10474_CR29","doi-asserted-by":"crossref","unstructured":"Schouten D, Nicoletti G, Dille B, Chia C, Vendittelli P, Schuurmans M, Litjens G, Khalili N. Navigating the landscape of multimodal ai in medicine: a scoping review on technical challenges and clinical applications. 2024. arXiv:2411.03782.","DOI":"10.1016\/j.media.2025.103621"},{"key":"10474_CR30","doi-asserted-by":"crossref","unstructured":"Dhar J, Zaidi N, Haghighat M, Goyal P, Roy S, Alavi A, Kumar V. Multimodal fusion learning with dual attention for medical imaging. 2024. arXiv:2412.01248.","DOI":"10.1109\/WACV61041.2025.00428"},{"key":"10474_CR31","unstructured":"Haghighat M. Multimodal fusion learning with dual attention for medical imaging. In: IEEE\/CVF Winter Conference on Applications of Computer Vision WACV 2025. 2025. Institute of Electrical and Electronics Engineers Inc."},{"key":"10474_CR32","doi-asserted-by":"crossref","unstructured":"Slaoui M, Mouh FZ, Ghanname I, Razine R, El Mzibri M, Amrani M. Outcome of breast cancer in Moroccan young women correlated to clinic-pathological features, risk factors and treatment: a comparative study of 716 cases in a single institution. PLoS ONE. 2016;11(10):0164841.","DOI":"10.1371\/journal.pone.0164841"},{"key":"10474_CR33","doi-asserted-by":"crossref","unstructured":"Ding S, Li J, Wang J, Ying S, Shi J. Multimodal co-attention fusion network with online data augmentation for cancer subtype classification. IEEE Transactions on Medical Imaging. 2024.","DOI":"10.1109\/TMI.2024.3405535"}],"container-title":["Cognitive Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12559-025-10474-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s12559-025-10474-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12559-025-10474-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,20]],"date-time":"2025-11-20T10:57:38Z","timestamp":1763636258000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s12559-025-10474-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6]]},"references-count":33,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,6]]}},"alternative-id":["10474"],"URL":"https:\/\/doi.org\/10.1007\/s12559-025-10474-6","relation":{},"ISSN":["1866-9956","1866-9964"],"issn-type":[{"value":"1866-9956","type":"print"},{"value":"1866-9964","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6]]},"assertion":[{"value":"8 October 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 May 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 June 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This article does not contain any studies with human participants or animals performed by any of the authors. The research used publicly available datasets with suitable references.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical Approval"}},{"value":"The authors declare no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}}],"article-number":"114"}}