{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T08:03:39Z","timestamp":1789027419065,"version":"build-2803163510"},"reference-count":41,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,4,23]],"date-time":"2025-04-23T00:00:00Z","timestamp":1745366400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Physiol."],"abstract":"<jats:sec><jats:title>Introduction<\/jats:title><jats:p>Tongue diagnosis is a fundamental technique in traditional Chinese medicine (TCM), where clinicians evaluate the tongue\u2019s appearance to infer the condition of pathological organs. However, most existing research on intelligent tongue diagnosis primarily focuses on analyzing tongue images, often neglecting the important descriptive text that accompanies these images. This text is an essential component of clinical diagnosis. To overcome this gap, we propose a novel Cross-Modal Pathological Organ Diagnosis Model that integrates tongue images and textual descriptions for more accurate pathological classification<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>Our model extracts features from both the tongue images and the corresponding textual descriptions. These features are then fused using a cross-modal attention mechanism to enhance the classification of pathological organs. The cross-modal attention mechanism enables the model to effectively combine visual and textual information, addressing the limitations of using either modality alone<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>We conducted experiments using a self-constructed dataset to evaluate our model\u2019s performance. The results demonstrate that our model outperforms common models regarding overall accuracy. Additionally, ablation studies, where either tongue images or textual descriptions were used alone, confirmed the significant benefit of multimodal fusion in improving diagnostic accuracy.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>This study introduces a new perspective on intelligent tongue diagnosis in TCM by incorporating visual and textual data. The experimental findings highlight the importance of cross-modal feature fusion for improving the accuracy of pathological diagnosis. Our approach not only contributes to the development of more effective diagnostic systems but also paves the way for future advancements in the automation of TCM diagnosis.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fphys.2025.1580985","type":"journal-article","created":{"date-parts":[[2025,4,23]],"date-time":"2025-04-23T04:10:35Z","timestamp":1745381435000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Cross-modal attention model integrating tongue images and descriptions: a novel intelligent TCM approach for pathological organ diagnosis"],"prefix":"10.3389","volume":"16","author":[{"given":"Quan","family":"Gan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chen","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhaoman","family":"Zhong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiaying","family":"Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qiwei","family":"Ge","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lei","family":"Shi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiaqing","family":"Shang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chuanxia","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2025,4,23]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"208","DOI":"10.1016\/j.inffus.2020.06.008","article-title":"A smart healthcare monitoring system for heart disease prediction based on ensemble deep learning and feature fusion","volume":"63","author":"Ali","year":"2020","journal-title":"Inf. Fusion"},{"key":"B2","first-page":"357","article-title":"Crossvit: cross-attention multi-scale vision transformer for image classification","volume-title":"Proceedings of the IEEE\/CVF international conference on computer vision","author":"Chen","year":""},{"key":"B3","doi-asserted-by":"publisher","first-page":"108827","DOI":"10.1016\/j.patcog.2022.108827","article-title":"Gashis-transformer: a multi-scale visual transformer approach for gastric histopathological image detection","volume":"130","author":"Chen","year":"2022","journal-title":"Pattern Recognit."},{"key":"B4","volume-title":"Tongue-watching formula","author":"Chen","year":""},{"key":"B5","first-page":"4171","article-title":"Bert: pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers)","author":"Devlin","year":"2019"},{"key":"B6","article-title":"An image is worth 16x16 words: transformers for image recognition at scale","author":"Dosovitskiy","year":"2020"},{"key":"B7","first-page":"290","article-title":"Automatic diagnosis model of gastrointestinal diseases based on tongue images","volume-title":"International conference on pattern recognition","author":"Fu","year":"2024"},{"key":"B8","first-page":"85","article-title":"Construction of a human body model of traditional Chinese medicine by taking into account of qi, blood and fluid","volume":"20","author":"Gan","year":"2023","journal-title":"J. East Asian Stud."},{"key":"B9","doi-asserted-by":"publisher","first-page":"104526","DOI":"10.1016\/j.biosystems.2021.104526","article-title":"Construction of a human body model for acupuncture and moxibustion treatment by colored petri nets","volume":"210","author":"Gan","year":"","journal-title":"BioSystems"},{"key":"B10","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1016\/j.biosystems.2021.104526","article-title":"Construction of human body model by colored petri nets based on traditional Chinese medicine","author":"Gan","year":"","journal-title":"IEICE Trans. Inf. and Syst. J104-D"},{"key":"B11","first-page":"770","article-title":"Deep residual learning for image recognition","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition","author":"He","year":"2016"},{"key":"B12","doi-asserted-by":"publisher","first-page":"102110","DOI":"10.1016\/j.artmed.2021.102110","article-title":"Fully-channel regional attention network for disease-location recognition with tongue images","volume":"118","author":"Hu","year":"2021","journal-title":"Artif. Intell. Med."},{"key":"B13","volume-title":"Look at the tongue to identify disease atlas","author":"Jia","year":"2008"},{"key":"B14","doi-asserted-by":"publisher","first-page":"252","DOI":"10.1016\/j.compeleceng.2018.03.037","article-title":"Medical images fusion by using weighted least squares filter and sparse representation","volume":"67","author":"Jiang","year":"2018","journal-title":"Comput. and Electr. Eng."},{"key":"B15","doi-asserted-by":"crossref","first-page":"288","DOI":"10.1109\/BIBMW.2012.6470318","article-title":"Zheng classification in traditional Chinese medicine based on modified specular-free tongue images","volume-title":"2012 IEEE international conference on bioinformatics and biomedicine workshops","author":"Kanawong","year":"2012"},{"key":"B16","volume-title":"Illustrated tongue diagnosis disease","author":"Lai","year":"2017"},{"key":"B17","doi-asserted-by":"crossref","first-page":"633","DOI":"10.1109\/ITOEC49072.2020.9141557","article-title":"Research about tongue image of traditional Chinese medicine (tcm) based on artificial intelligence technology","volume-title":"2020 IEEE 5th information technology and mechatronics engineering conference (ITOEC)","author":"Li","year":"2020"},{"key":"B18","doi-asserted-by":"publisher","first-page":"895","DOI":"10.11842\/wst.20211103006","article-title":"Discussion on the viscera partition method","volume":"29","author":"Li","year":"2014","journal-title":"Guangming J. Chin. Med."},{"key":"B19","first-page":"1345","article-title":"Theoretical origin and clinical application of traditional Chinese medicine inspection","volume":"22","author":"Liu","year":"2013","journal-title":"J. Emerg. Traditional Chin. Med."},{"key":"B20","doi-asserted-by":"publisher","first-page":"1195","DOI":"10.1109\/TBME.2018.2869989","article-title":"Joint classification and regression via deep multi-task multi-channel learning for alzheimer\u2019s disease diagnosis","volume":"66","author":"Liu","year":"2018","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"B21","doi-asserted-by":"publisher","first-page":"52","DOI":"10.1109\/TBME.2017.2688919","article-title":"Joint magnetic calibration and localization based on expectation maximization for tongue tracking","volume":"65","author":"Lu","year":"2017","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"B22","volume-title":"Graphic tongue examination","author":"Luo","year":"2018"},{"key":"B23","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1016\/j.artmed.2019.03.008","article-title":"Complexity perception classification method for tongue constitution recognition","volume":"96","author":"Ma","year":"2019","journal-title":"Artif. Intell. Med."},{"key":"B24","doi-asserted-by":"publisher","first-page":"94769","DOI":"10.1109\/ACCESS.2021.3094226","article-title":"Internet of things and synergic deep learning based biomedical tongue color image analysis for disease diagnosis and classification","volume":"9","author":"Mansour","year":"2021","journal-title":"IEEE Access"},{"key":"B25","doi-asserted-by":"publisher","first-page":"254","DOI":"10.1111\/j.1468-3083.2004.00769.x","article-title":"The tongue in clinical diagnosis","volume":"18","author":"Rogers","year":"2004","journal-title":"J. Eur. Acad. Dermatology Venereol."},{"key":"B26","doi-asserted-by":"publisher","first-page":"1337558","DOI":"10.1155\/2021\/1337558","article-title":"A new method for syndrome classification of non-small-cell lung cancer based on data of tongue and pulse with machine learning","volume":"2021","author":"Shi","year":"2021","journal-title":"BioMed Res. Int."},{"key":"B27","volume-title":"Holography of tongue diagnosis","author":"Song","year":"2022"},{"key":"B28","first-page":"104","article-title":"Multi-view analysis of unregistered medical images using cross-view transformers","volume-title":"Medical image computing and computer assisted intervention\u2013MICCAI 2021: 24th international conference, strasbourg, France, september 27\u2013october 1, 2021, proceedings, Part III 24","author":"Van Tulder","year":"2021"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1706.03762v1","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. neural Inf. Process. Syst."},{"key":"B30","doi-asserted-by":"publisher","first-page":"381","DOI":"10.1109\/TITB.2012.2226736","article-title":"A new tongue colorchecker design by space representation for precise correction","volume":"17","author":"Wang","year":"2013","journal-title":"IEEE J. Biomed. health Inf."},{"key":"B31","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1007\/s44196-024-00402-1","article-title":"Interpretable tongue constitution recognition via reshaped wavelet attention","volume":"17","author":"Wen","year":"2024","journal-title":"Int. J. Comput. Intell. Syst."},{"key":"B32","volume-title":"TCM tongue diagnosis clinical diagram","author":"Xu","year":"2017"},{"key":"B33","doi-asserted-by":"publisher","first-page":"4574","DOI":"10.11842\/wst.20211103006","article-title":"The present and future of the research on the objectification of tcm tongue diagnosis based on the microscopic angle","volume":"24","author":"Zhang","year":"2022","journal-title":"World Sci. Technology-Modernization Traditional Chin. Med."},{"key":"B34","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1109\/ICMB.2014.25","article-title":"Disease detection using tongue geometry features with sparse representation classifier","volume-title":"2014 International conference on medical biometrics","author":"Zhang","year":"2014"},{"key":"B35","volume-title":"Clinical tongue record","author":"Zhang","year":"2017"},{"key":"B36","volume-title":"Tongue diagnosis ten lectures","author":"Zhang","year":"2021"},{"key":"B37","first-page":"256","article-title":"Summary of computer-assisted tongue diagnosis solutions for key problems","volume":"48","author":"Zhang","year":"2021","journal-title":"Comput. Sci."},{"key":"B38","doi-asserted-by":"publisher","first-page":"170","DOI":"10.13862\/j.cn43-1446\/r.2023.06.033","article-title":"Research progress on objectification and intelligence of the four diagnosis methods of traditional Chinese medicine","volume":"29","author":"Zhang","year":"2023","journal-title":"Guid. J. Traditional Chin. Med. Pharm."},{"key":"B39","doi-asserted-by":"crossref","first-page":"904","DOI":"10.1145\/3644116.3644271","article-title":"Swin-resnet: research and implementation of a tooth-marked tongue classification method combining resnet-50 and swin transformer","volume-title":"Proceedings of the 2023 4th international symposium on artificial intelligence for medicine science","author":"Zhao","year":"2023"},{"key":"B40","volume-title":"Shanghanzhizhang","author":"Zhen","year":"2016"},{"key":"B41","doi-asserted-by":"publisher","first-page":"3128","DOI":"10.3390\/app9153128","article-title":"Tonguenet: a precise and fast tongue segmentation system using u-net with a morphological processing layer","volume":"9","author":"Zhou","year":"2019","journal-title":"Appl. Sci."}],"container-title":["Frontiers in Physiology"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fphys.2025.1580985\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,4,24]],"date-time":"2025-04-24T08:03:11Z","timestamp":1745481791000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fphys.2025.1580985\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,23]]},"references-count":41,"alternative-id":["10.3389\/fphys.2025.1580985"],"URL":"https:\/\/doi.org\/10.3389\/fphys.2025.1580985","relation":{},"ISSN":["1664-042X"],"issn-type":[{"value":"1664-042X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,23]]},"article-number":"1580985"}}