{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T06:40:07Z","timestamp":1778740807196,"version":"3.51.4"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,8,19]],"date-time":"2023-08-19T00:00:00Z","timestamp":1692403200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,8,19]],"date-time":"2023-08-19T00:00:00Z","timestamp":1692403200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"the Major Projects of Technological Innovation in Hubei Province","award":["2019AEA170"],"award-info":[{"award-number":["2019AEA170"]}]},{"name":"the Frontier Projects of Wuhan for Application Foundation","award":["2019010701011381"],"award-info":[{"award-number":["2019010701011381"]}]},{"name":"the Translational Medicine and Interdisciplinary Research Joint Fund of Zhongnan Hospital of Wuhan University","award":["ZNJC202226"],"award-info":[{"award-number":["ZNJC202226"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Bioinformatics"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>Two types of non-invasive, radiation-free, and inexpensive imaging technologies that are widely employed in medical applications are ultrasound (US) and infrared thermography (IRT). The ultrasound image obtained by ultrasound imaging primarily expresses the size, shape, contour boundary, echo, and other morphological information of the lesion, while the infrared thermal image obtained by infrared thermography imaging primarily describes its thermodynamic function information. Although distinguishing between benign and malignant thyroid nodules requires both morphological and functional information, present deep learning models are only based on US images, making it possible that some malignant nodules with insignificant morphological changes but significant functional changes will go undetected.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>Given the US and IRT images present thyroid nodules through distinct modalities, we proposed an Adaptive multi-modal Hybrid (AmmH) classification model that can leverage the amalgamation of these two image types to achieve superior classification performance. The AmmH approach involves the construction of a hybrid single-modal encoder module for each modal data, which facilitates the extraction of both local and global features by integrating a CNN module and a Transformer module. The extracted features from the two modalities are then weighted adaptively using an adaptive modality-weight generation network and fused using an adaptive cross-modal encoder module. The fused features are subsequently utilized for the classification of thyroid nodules through the use of MLP. On the collected dataset, our AmmH model respectively achieved 97.17% and 97.38% of F1 and F2 scores, which significantly outperformed the single-modal models. The results of four ablation experiments further show the superiority of our proposed method.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>The proposed multi-modal model extracts features from various modal images, thereby enhancing the comprehensiveness of thyroid nodules descriptions. The adaptive modality-weight generation network enables adaptive attention to different modalities, facilitating the fusion of features using adaptive weights through the adaptive cross-modal encoder. Consequently, the model has demonstrated promising classification performance, indicating its potential as a non-invasive, radiation-free, and cost-effective screening tool for distinguishing between benign and malignant thyroid nodules. The source code is available at <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/wuliZN2020\/AmmH\">https:\/\/github.com\/wuliZN2020\/AmmH<\/jats:ext-link>.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12859-023-05446-2","type":"journal-article","created":{"date-parts":[[2023,8,19]],"date-time":"2023-08-19T15:01:44Z","timestamp":1692457304000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["An adaptive multi-modal hybrid model for classifying thyroid nodules by combining ultrasound and infrared thermal images"],"prefix":"10.1186","volume":"24","author":[{"given":"Na","family":"Zhang","sequence":"first","affiliation":[]},{"given":"Juan","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Yu","family":"Jin","sequence":"additional","affiliation":[]},{"given":"Wensi","family":"Duan","sequence":"additional","affiliation":[]},{"given":"Ziling","family":"Wu","sequence":"additional","affiliation":[]},{"given":"Zhaohui","family":"Cai","sequence":"additional","affiliation":[]},{"given":"Meng","family":"Wu","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,8,19]]},"reference":[{"issue":"3","key":"5446_CR1","doi-asserted-by":"publisher","first-page":"629","DOI":"10.1002\/jum.14731","volume":"38","author":"A Abbasian Ardakani","year":"2019","unstructured":"Abbasian Ardakani A, Bitarafan-Rajabi A, Mohammadzadeh A, Mohammadi A, Riazi R, Abolghasemi J, Homayoun Jafari A, Bagher Shiran M. A hybrid multilayer filtering approach for thyroid nodule segmentation on ultrasound images. J Ultrasound Med. 2019;38(3):629\u201340.","journal-title":"J Ultrasound Med"},{"issue":"24","key":"5446_CR2","doi-asserted-by":"publisher","first-page":"2347","DOI":"10.1056\/NEJMcp1415786","volume":"373","author":"KD Burman","year":"2015","unstructured":"Burman KD, Wartofsky L. Thyroid nodules. N Engl J Med. 2015;373(24):2347\u201356.","journal-title":"N Engl J Med"},{"issue":"1","key":"5446_CR3","doi-asserted-by":"publisher","first-page":"25","DOI":"10.14366\/usg.15027","volume":"35","author":"G Russ","year":"2016","unstructured":"Russ G. Risk stratification of thyroid nodules on ultrasonography with the french ti-rads: description and reflections. Ultrasonography. 2016;35(1):25.","journal-title":"Ultrasonography"},{"issue":"11","key":"5446_CR4","doi-asserted-by":"publisher","first-page":"5364","DOI":"10.1109\/JBHI.2022.3197910","volume":"26","author":"TK Dash","year":"2022","unstructured":"Dash TK, Chakraborty C, Mahapatra S, Panda G. Gradient boosting machine and efficient combination of features for speech-based detection of covid-19. IEEE J Biomed Health Inform. 2022;26(11):5364\u201371.","journal-title":"IEEE J Biomed Health Inform"},{"key":"5446_CR5","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2019.116137","volume":"202","author":"J-D Tournier","year":"2019","unstructured":"Tournier J-D, Smith R, Raffelt D, Tabbara R, Dhollander T, Pietsch M, Christiaens D, Jeurissen B, Yeh C-H, Connelly A. Mrtrix3: a fast, flexible and open software framework for medical image processing and visualisation. Neuroimage. 2019;202: 116137.","journal-title":"Neuroimage"},{"issue":"3","key":"5446_CR6","doi-asserted-by":"publisher","first-page":"12776","DOI":"10.1111\/exsy.12776","volume":"39","author":"HK Bhuyan","year":"2022","unstructured":"Bhuyan HK, Chakraborty C, Shelke Y, Pani SK. Covid-19 diagnosis system by deep learning approaches. Expert Syst. 2022;39(3):12776.","journal-title":"Expert Syst"},{"issue":"5","key":"5446_CR7","doi-asserted-by":"publisher","first-page":"1202","DOI":"10.1007\/s10278-020-00362-w","volume":"33","author":"SW Kwon","year":"2020","unstructured":"Kwon SW, Choi IJ, Kang JY, Jang WI, Lee G-H, Lee M-C. Ultrasonographic thyroid nodule classification using a deep convolutional neural network with surgical pathology. J Digit Imaging. 2020;33(5):1202\u20138.","journal-title":"J Digit Imaging"},{"issue":"7","key":"5446_CR8","doi-asserted-by":"publisher","first-page":"137","DOI":"10.21037\/atm.2019.04.34","volume":"7","author":"Q Guan","year":"2019","unstructured":"Guan Q, Wang Y, Du J, Qin Y, Lu H, Xiang J, Wang F. Deep learning based classification of ultrasound images for thyroid nodules: a large scale of pilot study. Ann Transl Med. 2019;7(7):137.","journal-title":"Ann Transl Med"},{"issue":"3","key":"5446_CR9","doi-asserted-by":"publisher","first-page":"1168","DOI":"10.1109\/TBME.2008.915731","volume":"55","author":"A Helmy","year":"2008","unstructured":"Helmy A, Holdmann M, Rizkalla M. Application of thermography for non-invasive diagnosis of thyroid gland disease. IEEE Trans Biomed Eng. 2008;55(3):1168\u201375.","journal-title":"IEEE Trans Biomed Eng"},{"key":"5446_CR10","doi-asserted-by":"publisher","DOI":"10.1016\/j.infrared.2019.103126","volume":"104","author":"F Bahramian","year":"2020","unstructured":"Bahramian F, Mojra A. Thyroid cancer estimation using infrared thermography data. Infrared Phys Technol. 2020;104: 103126.","journal-title":"Infrared Phys Technol"},{"issue":"1","key":"5446_CR11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12938-022-01009-3","volume":"21","author":"VMB de Camargo","year":"2022","unstructured":"de Camargo VMB, Ulbricht L, Coninck JCP, Ripka WL, Gamba HR. Thermography as an aid for the complementary diagnosis of nodules in the thyroid gland. Biomed Eng Online. 2022;21(1):1\u201315.","journal-title":"Biomed Eng Online"},{"key":"5446_CR12","doi-asserted-by":"publisher","first-page":"5134","DOI":"10.1109\/TIP.2022.3193288","volume":"31","author":"W Tang","year":"2022","unstructured":"Tang W, He F, Liu Y, Duan Y. Matr: multimodal medical image fusion via multiscale adaptive transformer. IEEE Trans Image Process. 2022;31:5134\u201349.","journal-title":"IEEE Trans Image Process"},{"issue":"4","key":"5446_CR13","doi-asserted-by":"publisher","first-page":"1401","DOI":"10.1007\/s00530-021-00826-1","volume":"28","author":"V Ravi","year":"2022","unstructured":"Ravi V, Narasimhan H, Chakraborty C, Pham TD. Deep learning-based meta-classifier approach for covid-19 classification using ct scan and chest x-ray images. Multimedia Syst. 2022;28(4):1401\u201315.","journal-title":"Multimedia Syst"},{"key":"5446_CR14","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2022.109631","volume":"129","author":"P Razzaghi","year":"2022","unstructured":"Razzaghi P, Abbasi K, Shirazi M, Rashidi S. Multimodal brain tumor detection using multimodal deep transfer learning. Appl Soft Comput. 2022;129: 109631.","journal-title":"Appl Soft Comput"},{"key":"5446_CR15","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2022.106339","volume":"151","author":"B Li","year":"2022","unstructured":"Li B, Hwang J-N, Liu Z, Li C, Wang Z. Pet and mri image fusion based on a dense convolutional network with dual attention. Comput Biol Med. 2022;151: 106339.","journal-title":"Comput Biol Med"},{"key":"5446_CR16","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2023.106657","volume":"155","author":"F Wang","year":"2023","unstructured":"Wang F, Cheng C, Cao W, Wu Z, Wang H, Wei W, Yan Z, Liu Z. Mfcnet: a multi-modal fusion and calibration networks for 3d pancreas tumor segmentation on pet-ct images. Comput Biol Med. 2023;155: 106657.","journal-title":"Comput Biol Med"},{"key":"5446_CR17","doi-asserted-by":"crossref","unstructured":"Huang W, Wang X, Huang Y, Lin F, Tang X (2022) Multi-parametric magnetic resonance imaging fusion for automatic classification of prostate cancer. In: 2022 44th Annual international conference of the ieee engineering in medicine & biology Society (EMBC), pp. 471\u2013474. IEEE.","DOI":"10.1109\/EMBC48229.2022.9871334"},{"key":"5446_CR18","doi-asserted-by":"publisher","first-page":"52","DOI":"10.1016\/j.ymeth.2019.06.014","volume":"173","author":"R Yan","year":"2020","unstructured":"Yan R, Ren F, Wang Z, Wang L, Zhang T, Liu Y, Rao X, Zheng C, Zhang F. Breast cancer histopathological image classification using a hybrid deep neural network. Methods. 2020;173:52\u201360.","journal-title":"Methods"},{"key":"5446_CR19","doi-asserted-by":"publisher","first-page":"645","DOI":"10.1007\/s00500-021-06490-x","volume":"26","author":"S Ketu","year":"2022","unstructured":"Ketu S, Mishra PK. India perspective: Cnn-lstm hybrid deep learning model-based covid-19 prediction and current status of medical resource availability. Soft Comput. 2022;26:645\u201364.","journal-title":"Soft Comput"},{"key":"5446_CR20","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Guyon, I., Luxburg, U., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in neural information processing systems 30 (NIPS 2017). Advances in Neural Information Processing Systems, vol. 30 (2017). In: 31st Annual Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, DEC 04-09."},{"key":"5446_CR21","unstructured":"Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S et al: (2020) An image is worth 16x16 words: transformers for image recognition at scale. In: International Conference on Learning Representations."},{"key":"5446_CR22","unstructured":"Touvron H, Cord M, Douze M, Massa F, Sablayrolles A, J\u00e9gou H (2021) Training data-efficient image transformers & distillation through attention. In: International conference on machine learning, pp. 10347\u201310357 . PMLR."},{"key":"5446_CR23","doi-asserted-by":"crossref","unstructured":"Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE\/CVF international conference on computer vision, pp. 10012\u201310022.","DOI":"10.1109\/ICCV48922.2021.00986"},{"issue":"13","key":"5446_CR24","doi-asserted-by":"publisher","first-page":"2970","DOI":"10.1073\/pnas.1717139115","volume":"115","author":"P Mobadersany","year":"2018","unstructured":"Mobadersany P, Yousefi S, Amgad M, Gutman DA, Barnholtz-Sloan JS, Vel\u00e1zquez Vega JE, Brat DJ, Cooper LA. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc Natl Acad Sci. 2018;115(13):2970\u20139.","journal-title":"Proc Natl Acad Sci"},{"key":"5446_CR25","first-page":"754","volume":"41","author":"RJ Chen","year":"2020","unstructured":"Chen RJ, Lu MY, Wang J, Williamson DF, Rodig SJ, Lindeman NI, Mahmood F. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Trans Med Imag. 2020;41:754.","journal-title":"IEEE Trans Med Imag"},{"key":"5446_CR26","doi-asserted-by":"crossref","unstructured":"Braman N, Gordon JW, Goossens ET, Willis C, Stumpe MC, Venkataraman J (2021) Deep orthogonal fusion: Multimodal prognostic biomarker discovery integrating radiology, pathology, genomic, and clinical data. In: International conference on medical image computing and computer-assisted intervention, pp. 667\u2013677. Springer.","DOI":"10.1007\/978-3-030-87240-3_64"},{"key":"5446_CR27","doi-asserted-by":"publisher","DOI":"10.1016\/j.bspc.2022.104524","volume":"81","author":"T Zhou","year":"2023","unstructured":"Zhou T. Modality-level cross-connection and attentional feature fusion based deep neural network for multi-modal brain tumor segmentation. Biomed Signal Process Control. 2023;81: 104524.","journal-title":"Biomed Signal Process Control"},{"key":"5446_CR28","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770\u2013778.","DOI":"10.1109\/CVPR.2016.90"},{"key":"5446_CR29","unstructured":"Ba JL, Kiros JR, Hinton GE. Layer normalization stat. 2016;1050:21."},{"key":"5446_CR30","unstructured":"Hendrycks D, Gimpel K (2016) Bridging nonlinearities and stochastic regularizers with gaussian error linear units."},{"key":"5446_CR31","unstructured":"Kenton JDM-WC, Toutanova LK (2019) Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT, pp. 4171\u20134186."},{"key":"5446_CR32","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2019.112866","volume":"140","author":"D Devarriya","year":"2020","unstructured":"Devarriya D, Gulati C, Mansharamani V, Sakalle A, Bhardwaj A. Unbalanced breast cancer data classification using novel fitness functions in genetic programming. Expert Syst Appl. 2020;140: 112866. https:\/\/doi.org\/10.1016\/j.eswa.2019.112866.","journal-title":"Expert Syst Appl"}],"container-title":["BMC Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-023-05446-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12859-023-05446-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-023-05446-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,19]],"date-time":"2023-11-19T14:04:00Z","timestamp":1700402640000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/s12859-023-05446-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,19]]},"references-count":32,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["5446"],"URL":"https:\/\/doi.org\/10.1186\/s12859-023-05446-2","relation":{},"ISSN":["1471-2105"],"issn-type":[{"value":"1471-2105","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,19]]},"assertion":[{"value":"18 November 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 August 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 August 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"All procedures performed in this study are in accordance with the tenets of the Declaration of Helsinki and are approved by the ethics committee of the Zhongnan Hospital of Wuhan University (2022215). Informed consent was obtained from all the participants.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"315"}}