{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,20]],"date-time":"2026-05-20T16:24:05Z","timestamp":1779294245733,"version":"3.51.4"},"reference-count":64,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,8,18]],"date-time":"2025-08-18T00:00:00Z","timestamp":1755475200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:sec><jats:title>Background\/Introduction<\/jats:title><jats:p>Skin lesion classification poses a critical diagnostic challenge in dermatology, where early and accurate identification has a direct impact on patient outcomes. While deep learning approaches have shown promise using dermatoscopic images alone, the integration of clinical metadata remains underexplored despite its potential to enhance diagnostic accuracy.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>We developed a novel multimodal data fusion framework that systematically integrates dermatoscopic images with clinical metadata for the classification of skin lesions. Using the HAM10000 dataset, we evaluated multiple fusion strategies, including simple concatenation, weighted concatenation, self-attention mechanisms, and cross-attention fusion. Clinical features were processed through a customized Multi-Layer Perceptron (MLP), while images were analyzed using a modified Residual Networks (ResNet) architecture. Model interpretability was enhanced using Gradient-weighted Class Activation Mapping (Grad-CAM) visualization to identify the contribution of clinical attributes to classification decisions.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Cross-attention fusion achieved the highest classification accuracy, demonstrating superior performance compared to unimodal approaches and simpler fusion techniques. The multimodal framework significantly outperformed image-only baselines, with cross-attention effectively capturing inter-modal dependencies and contextual relationships between visual and clinical data modalities.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion\/Conclusions<\/jats:title><jats:p>Our findings demonstrate that integrating clinical metadata with dermatoscopic images substantially improves the accuracy of skin lesion classification. However, challenges, including class imbalance and the computational complexity of advanced fusion methods, require further investigation.<\/jats:p><\/jats:sec>","DOI":"10.3389\/frai.2025.1608837","type":"journal-article","created":{"date-parts":[[2025,8,18]],"date-time":"2025-08-18T05:26:41Z","timestamp":1755494801000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Comparative analysis of multimodal architectures for effective skin lesion detection using clinical and image data"],"prefix":"10.3389","volume":"8","author":[{"given":"Adriteyo","family":"Das","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vedant","family":"Agarwal","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nisha P.","family":"Shetty","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2025,8,18]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"1774","DOI":"10.1038\/s41598-024-52185-2","article-title":"Multimodal classification of alzheimer's disease and mild cognitive impairment using custom mkscddl kernel over CNN with transparent decision-making for explainable diagnosis","volume":"14","author":"Adarsh","year":"2024","journal-title":"Sci. Rep"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.1101\/2024.05.30.24308213","article-title":"Accurate skin lesion classification using multimodal learning on the ham10000 dataset","author":"Adebiyi","year":"2024","journal-title":"medRxiv. Preprint"},{"key":"B3","doi-asserted-by":"publisher","first-page":"541","DOI":"10.3390\/healthcare10030541","article-title":"Machine-learning-based disease diagnosis: a comprehensive review","volume":"10","author":"Ahsan","year":"2022","journal-title":"Healthcare"},{"key":"B4","doi-asserted-by":"publisher","first-page":"310","DOI":"10.1186\/s12911-020-01332-6","article-title":"Explainability for artificial intelligence in healthcare: a multidisciplinary perspective","volume":"20","author":"Amann","year":"2020","journal-title":"BMC Med. Inform. Decis. Mak"},{"key":"B5","doi-asserted-by":"publisher","first-page":"10209","DOI":"10.1007\/s00521-019-04559-1","article-title":"Gated multimodal networks","volume":"32","author":"Arevalo","year":"2020","journal-title":"Neural Comput. Appl"},{"key":"B6","doi-asserted-by":"publisher","first-page":"9619079","DOI":"10.1155\/2021\/9619079","article-title":"A computer-aided diagnosis system using deep learning for multiclass skin lesion classification","volume":"2021","author":"Arshad","year":"2021","journal-title":"Comput. Intell. Neurosci"},{"key":"B7","doi-asserted-by":"publisher","first-page":"972","DOI":"10.1109\/TIP.2002.802531","article-title":"A comparison of computational color constancy algorithms. I: Methodology and experiments with synthesized data","volume":"11","author":"Barnard","year":"2002","journal-title":"IEEE Trans. Image Proc"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01249","article-title":"\u201cQuasi-unsupervised color constancy,\u201d","author":"Bianco","year":"2019","journal-title":"2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"B9","first-page":"13","article-title":"\u201cSoft-attention improves skin cancer classification performance,\u201d","volume-title":"International Workshop on Interpretability of Machine Intelligence in Medical Image Computing","author":"Datta","year":"2021"},{"key":"B10","doi-asserted-by":"publisher","first-page":"CD013194","DOI":"10.1002\/14651858.CD013194","article-title":"Visual inspection for diagnosing cutaneous melanoma in adults","volume":"12","author":"Dinnes","year":"2018","journal-title":"Cochr. Datab. System. Rev"},{"key":"B11","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1038\/nature21056","article-title":"Dermatologist-level classification of skin cancer with deep neural networks","volume":"542","author":"Esteva","year":"2017","journal-title":"Nature"},{"key":"B12","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1016\/j.neucom.2018.09.013","article-title":"Gan-based synthetic medical image augmentation for increased cnn performance in liver lesion classification","volume":"321","author":"Frid-Adar","year":"2018","journal-title":"Neurocomputing"},{"key":"B13","doi-asserted-by":"publisher","first-page":"457","DOI":"10.18653\/v1\/D16-1044","article-title":"\u201cMultimodal compact bilinear pooling for visual question answering and visual grounding,\u201d","author":"Fukui","year":"2016","journal-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing"},{"key":"B14","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1016\/j.ejca.2021.04.029","article-title":"Epidemiology of cutaneous melanoma and keratinocyte cancer in white populations 1943\u20132036","volume":"152","author":"Garbe","year":"2021","journal-title":"Eur. J. Cancer"},{"key":"B15","doi-asserted-by":"publisher","first-page":"100864","DOI":"10.1016\/j.mex.2020.100864","article-title":"Skin lesion classification using ensembles of multi-resolution efficientNets with meta data","volume":"7","author":"Gessert","year":"2020","journal-title":"MethodsX"},{"key":"B16","doi-asserted-by":"publisher","first-page":"1836","DOI":"10.1093\/annonc\/mdy166","article-title":"Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists","volume":"29","author":"Haenssle","year":"2018","journal-title":"Ann. Oncol"},{"key":"B17","doi-asserted-by":"publisher","first-page":"1529","DOI":"10.1016\/j.jid.2018.01.028","article-title":"Classification of the clinical images for benign and malignant cutaneous tumors using a deep learning algorithm","volume":"138","author":"Han","year":"2018","journal-title":"J. Invest. Dermatol"},{"key":"B18","doi-asserted-by":"publisher","first-page":"185","DOI":"10.5021\/ad.2011.23.2.185","article-title":"A clinical and histopathological study of 122 cases of dermatofibroma (benign fibrous histiocytoma)","volume":"23","author":"Han","year":"2011","journal-title":"Ann. Dermatol"},{"key":"B19","doi-asserted-by":"publisher","first-page":"168","DOI":"10.1186\/s12943-023-01854-3","article-title":"Skin cancer: understanding the journey of transformation from conventional to advanced treatment approaches","volume":"22","author":"Hasan","year":"2023","journal-title":"Mol. Cancer"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90","article-title":"\u201cDeep residual learning for image recognition,\u201d","author":"He","year":"2016","journal-title":"2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"B21","volume-title":"Andrews' Diseases of the Skin: Clinical Dermatology","author":"James","year":"2006"},{"key":"B22","doi-asserted-by":"publisher","first-page":"6048","DOI":"10.3390\/cancers13236048","article-title":"Interpretability of a deep learning based approach for the classification of skin lesions into main anatomic body sites","volume":"13","author":"Jaworek-Korjakowska","year":"2021","journal-title":"Cancers"},{"key":"B23","first-page":"357","article-title":"Early detection and treatment of skin cancer","volume":"62","author":"Jerant","year":"2000","journal-title":"Am Fam Phys"},{"key":"B24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1155\/2022\/9322937","article-title":"Multimodal imaging of target detection algorithm under artificial intelligence in the diagnosis of early breast cancer","volume":"2022","author":"Jiang","year":"2022","journal-title":"J. Healthc. Eng"},{"key":"B25","doi-asserted-by":"publisher","first-page":"421","DOI":"10.25259\/IJDVL_978_20","article-title":"The burden of skin diseases in india: Global burden of disease study 2017","volume":"89","author":"Kavita","year":"2023","journal-title":"Indian J. Dermatol. Venereol. Leprol"},{"key":"B26","doi-asserted-by":"publisher","first-page":"408","DOI":"10.18653\/v1\/N18-1038","article-title":"\u201cLearning visually grounded sentence representations,\u201d","author":"Kiela","year":"2018","journal-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)"},{"key":"B27","article-title":"Hadamard product for low-rank bilinear pooling","author":"Kim","year":"2017","journal-title":"arXiv preprint arXiv:1610.04325"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.00371","article-title":"\u201cSegment anything,\u201d","author":"Kirillov","year":"2023","journal-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision"},{"key":"B29","doi-asserted-by":"publisher","first-page":"108396","DOI":"10.1016\/j.compeleceng.2022.108396","article-title":"Ensemble multimodal deep learning for early diagnosis and accurate classification of covid-19","volume":"103","author":"Kumar","year":"2022","journal-title":"Comput. Electr. Eng"},{"key":"B30","doi-asserted-by":"publisher","first-page":"76261","DOI":"10.1109\/ACCESS.2022.3181225","article-title":"Fixcaps: An improved capsules network for diagnosis of skin cancer","volume":"10","author":"Lan","year":"2022","journal-title":"IEEE Access"},{"key":"B31","doi-asserted-by":"publisher","first-page":"318","DOI":"10.1109\/TPAMI.2018.2858826","article-title":"Focal loss for dense object detection","volume":"42","author":"Lin","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell"},{"key":"B32","doi-asserted-by":"publisher","first-page":"170","DOI":"10.1016\/j.soncn.2013.06.003","article-title":"Advances in skin cancer early detection and diagnosis","volume":"29","author":"Loescher","year":"2013","journal-title":"Semin. Oncol. Nurs"},{"key":"B33","article-title":"\u201cVilbert: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,\u201d","volume-title":"Proceedings of the 33rd International Conference on Neural Information Processing Systems","author":"Lu","year":"2019"},{"key":"B34","first-page":"167","article-title":"Basal cell carcinoma: pathogenesis, epidemiology, clinical features, diagnosis, histopathology, and management","volume":"88","author":"Marzuka","year":"2015","journal-title":"Yale J. Biol. Med"},{"key":"B35","doi-asserted-by":"publisher","first-page":"20130392","DOI":"10.1259\/bjr.20130392","article-title":"Vascular anomalies: classification, imaging characteristics and implications for interventional radiology treatment approaches","volume":"87","author":"Mulligan","year":"2014","journal-title":"Br. J. Radiol"},{"key":"B36","doi-asserted-by":"publisher","first-page":"237","DOI":"10.4103\/jmss.JMSS_53_20","article-title":"Generative adversarial network image synthesis method for skin lesion generation and classification","volume":"11","author":"Mutepfe","year":"2021","journal-title":"J. Med. Signals Sens"},{"key":"B37","article-title":"\u201cDermatofibroma,\u201d","author":"Myers","year":"2024"},{"key":"B38","unstructured":"Melanoma treatment (pdq"},{"key":"B39","unstructured":"Skin cancer treatment (pdq"},{"key":"B40","first-page":"689","article-title":"\u201cMultimodal deep learning,\u201d","volume-title":"Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML'11","author":"Ngiam","year":"2011"},{"key":"B41","doi-asserted-by":"publisher","first-page":"1029991","DOI":"10.3389\/fsurg.2022.1029991","article-title":"A deep learning based multimodal fusion model for skin lesion diagnosis using smartphone collected clinical images and metadata","volume":"9","author":"Ou","year":"2022","journal-title":"Front. Surg"},{"key":"B42","author":"Puckett","year":"2025","journal-title":"Melanoma Pathology"},{"key":"B43","doi-asserted-by":"publisher","first-page":"adv00136","DOI":"10.2340\/00015555-3491","article-title":"Melanoma epidemiology and sun exposure","volume":"100","author":"Raimondi","year":"2020","journal-title":"Acta Dermato-Venereol"},{"key":"B44","doi-asserted-by":"publisher","first-page":"354","DOI":"10.1002\/ima.22616","article-title":"An accurate and noninvasive skin cancer screening based on imaging technique","volume":"32","author":"Rajput","year":"2021","journal-title":"Int. J. Imaging Syst. Technol"},{"key":"B45","doi-asserted-by":"publisher","first-page":"637","DOI":"10.1016\/j.abd.2019.10.004","article-title":"Actinic keratoses: review of clinical, dermoscopic, and therapeutic aspects","volume":"94","author":"Reinehr","year":"2019","journal-title":"An. Bras. Dermatol"},{"key":"B46","doi-asserted-by":"publisher","DOI":"10.1101\/2024.06.03.24308401","article-title":"Multimodal deep learning for low-resource settings: a vector embedding alignment approach for healthcare applications","author":"Restrepo","year":"2024","journal-title":"medRxiv"},{"key":"B47","doi-asserted-by":"publisher","first-page":"1081","DOI":"10.1001\/jamadermatol.2015.1187","article-title":"Incidence estimate of nonmelanoma skin cancer (keratinocyte carcinomas) in the us population, 2012","volume":"151","author":"Rogers","year":"2015","journal-title":"JAMA Dermatol"},{"key":"B48","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1038\/323533a0","article-title":"Learning representations by back-propagating errors","volume":"323","author":"Rumelhart","year":"1986","journal-title":"Nature"},{"key":"B49","unstructured":"Scott Mader\n              K.\n            \n          \n          39891245\n          Skin cancer mnist: Ham10000. Kaggle\n          \n          2018"},{"key":"B50","doi-asserted-by":"publisher","first-page":"e2023115","DOI":"10.5826\/dpc.1302a115","article-title":"Benign keratosis: a useful term?","volume":"13","author":"Scott","year":"2023","journal-title":"Dermatol. Pract. Concept"},{"key":"B51","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.74","article-title":"\u201cGrad-cam: visual explanations from deep networks via gradient-based localization,\u201d","author":"Selvaraju","year":"2017","journal-title":"2017 IEEE International Conference on Computer Vision (ICCV)"},{"key":"B52","doi-asserted-by":"publisher","first-page":"120","DOI":"10.3390\/curroncol32030120","article-title":"Basal cell carcinoma in patients over 80 years presenting for surgical excision: Clinical characteristics and surgical outcomes","volume":"32","author":"Seretis","year":"2025","journal-title":"Curr. Oncol"},{"key":"B53","doi-asserted-by":"publisher","first-page":"12039","DOI":"10.1007\/s00521-021-05929-4","article-title":"A deep convolutional neural network-based pigmented skin lesion classification application and experts evaluation","volume":"33","author":"Sevli","year":"2021","journal-title":"Neural Comput. Applic"},{"key":"B54","article-title":"Very deep convolutional networks for large-scale image recognition","author":"Simonyan","year":"2015","journal-title":"arXiv preprint arXiv:1409.1556"},{"key":"B55","doi-asserted-by":"publisher","first-page":"108259","DOI":"10.1016\/j.compeleceng.2022.108259","article-title":"A median based quadrilateral local quantized ternary pattern technique for the classification of dermatoscopic images of skin cancer","volume":"102","author":"Srivastava","year":"2022","journal-title":"Comput. Electr. Eng"},{"key":"B56","doi-asserted-by":"publisher","first-page":"225","DOI":"10.1055\/s-0037-1604295","article-title":"Classification of vascular anomalies: an update","volume":"34","author":"Steiner","year":"2017","journal-title":"Semin. Intervent. Radiol"},{"key":"B57","unstructured":"\u201cAxiomatic attribution for deep networks,\u201d\n          \n          3319\n          3328\n          \n            \n              Sundararajan\n              M.\n            \n            \n              Taly\n              A.\n            \n            \n              Yan\n              Q.\n            \n          \n          Proceedings of the 34th International Conference on Machine Learning\n          \n          2017"},{"key":"B58","doi-asserted-by":"publisher","first-page":"938","DOI":"10.1016\/S1470-2045(19)30333-X","article-title":"Comparison of the accuracy of human readers versus machine-learning algorithms for pigmented skin lesion classification: an open, web-based, international, diagnostic study","volume":"20","author":"Tschandl","year":"2019","journal-title":"Lancet Oncol"},{"key":"B59","doi-asserted-by":"publisher","first-page":"180161","DOI":"10.1038\/sdata.2018.161","article-title":"The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions","volume":"5","author":"Tschandl","year":"2018","journal-title":"Sci. Data"},{"key":"B60","article-title":"\u201cAttention is all you need,\u201d","author":"Vaswani","year":"2017","journal-title":"Proceedings of NeurIPS 2017"},{"key":"B61","doi-asserted-by":"publisher","first-page":"5146543","DOI":"10.1155\/2023\/5146543","article-title":"Multiscale feature fusion for skin lesion classification","volume":"2023","author":"Wang","year":"2023","journal-title":"Biomed Res. Int"},{"key":"B62","article-title":"Development of skip connection in deep neural networks for computer vision and medical image analysis: A survey","author":"Xu","year":"2024","journal-title":"ArXiv, abs\/2405.01725"},{"key":"B63","doi-asserted-by":"publisher","first-page":"1103","DOI":"10.18653\/v1\/D17-1115","article-title":"\u201cTensor fusion network for multimodal sentiment analysis,\u201d","author":"Zadeh","year":"2017","journal-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing"},{"key":"B64","doi-asserted-by":"publisher","first-page":"e1002683","DOI":"10.1371\/journal.pmed.1002683","article-title":"Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study","volume":"15","author":"Zech","year":"2018","journal-title":"PLoS Med"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2025.1608837\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,11]],"date-time":"2025-09-11T10:46:59Z","timestamp":1757587619000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2025.1608837\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,18]]},"references-count":64,"alternative-id":["10.3389\/frai.2025.1608837"],"URL":"https:\/\/doi.org\/10.3389\/frai.2025.1608837","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,18]]},"article-number":"1608837"}}