{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,11]],"date-time":"2025-09-11T19:40:32Z","timestamp":1757619632291,"version":"3.44.0"},"reference-count":48,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,7,26]],"date-time":"2025-07-26T00:00:00Z","timestamp":1753488000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,7,26]],"date-time":"2025-07-26T00:00:00Z","timestamp":1753488000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Ulcerative colitis (UC) is a chronic inflammatory disorder necessitating precise severity stratification to facilitate optimal therapeutic interventions. This study harnesses a triple-pronged deep learning methodology\u2014including multimodal inference pipelines that eliminate domain-specific training, few-shot meta-learning, and Vision Transformer (ViT)-based ensembling\u2014to classify UC severity within the HyperKvasir dataset. We systematically evaluate multiple vision transformer architectures, discovering that a Swin-Base model achieves an accuracy of 90%, while a soft-voting ensemble of diverse ViT backbones boosts performance to 93%. In parallel, we leverage multimodal pre-trained frameworks (e.g., CLIP, BLIP, FLAVA) integrated with conventional machine learning algorithms, yielding an accuracy of 83%. To address limited annotated data, we deploy few-shot meta-learning approaches (e.g., Matching Networks), attaining 83% accuracy in a 5-shot context. Furthermore, interpretability is enhanced via SHapley Additive exPlanations (SHAP), which interpret both local and global model behaviors, thereby fostering clinical trust in the model\u2019s inferences. These findings underscore the potential of contemporary representation learning and ensemble strategies for robust UC severity classification, highlighting the pivotal role of model transparency in facilitating medical image analysis.<\/jats:p>","DOI":"10.1038\/s41598-025-12827-5","type":"journal-article","created":{"date-parts":[[2025,7,26]],"date-time":"2025-07-26T05:38:10Z","timestamp":1753508290000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["A triple pronged approach for ulcerative colitis severity classification using multimodal, meta, and transformer based learning"],"prefix":"10.1038","volume":"15","author":[{"given":"Md. Nasim","family":"Ahmed","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dipta","family":"Neogi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Muhammad Rafsan","family":"Kabir","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shafin","family":"Rahman","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sifat","family":"Momen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nabeel","family":"Mohammed","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,7,26]]},"reference":[{"key":"12827_CR1","doi-asserted-by":"publisher","first-page":"2769","DOI":"10.1016\/S0140-6736(17)32448-0","volume":"390","author":"SC Ng","year":"2017","unstructured":"Ng, S. C. et al. Worldwide incidence and prevalence of inflammatory bowel disease in the 21st century: A systematic review of population-based studies. Lancet 390, 2769\u20132778 (2017).","journal-title":"Lancet"},{"key":"12827_CR2","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1016\/S2468-1253(16)30077-2","volume":"1","author":"GG Kaplan","year":"2016","unstructured":"Kaplan, G. G. & Ng, S. C. Globalisation of inflammatory bowel disease: Perspectives from the evolution of inflammatory bowel disease in the UK and China. Lancet Gastroenterol. Hepatol. 1, 307\u2013316 (2016).","journal-title":"Lancet Gastroenterol. Hepatol."},{"key":"12827_CR3","doi-asserted-by":"publisher","first-page":"2027","DOI":"10.1053\/j.gastro.2018.04.003","volume":"154","author":"M Misawa","year":"2018","unstructured":"Misawa, M. et al. Artificial intelligence-assisted polyp detection for colonoscopy: Initial experience. Gastroenterology 154, 2027\u20132029 (2018).","journal-title":"Gastroenterology"},{"key":"12827_CR4","doi-asserted-by":"crossref","unstructured":"Mohapatra, S. et al. Severity grading of ulcerative colitis using endoscopy images: An ensembled deep learning and transfer learning approach. J. Inst. Eng. (India) Ser. B 1\u201320 (2024).","DOI":"10.1007\/s40031-024-01099-8"},{"key":"12827_CR5","doi-asserted-by":"publisher","first-page":"283","DOI":"10.1038\/s41597-020-00622-y","volume":"7","author":"H Borgli","year":"2020","unstructured":"Borgli, H. et al. Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Sci. Data 7, 283. https:\/\/doi.org\/10.1038\/s41597-020-00622-y (2020).","journal-title":"Sci. Data"},{"key":"12827_CR6","doi-asserted-by":"publisher","first-page":"1431","DOI":"10.1093\/ibd\/izac226","volume":"29","author":"G Polat","year":"2023","unstructured":"Polat, G. et al. Improving the computer-aided estimation of ulcerative colitis severity according to mayo endoscopic score by using regression-based deep learning. Inflamm. Bowel Dis. 29, 1431\u20131439 (2023).","journal-title":"Inflamm. Bowel Dis."},{"key":"12827_CR7","doi-asserted-by":"publisher","first-page":"2748","DOI":"10.1038\/s41598-022-06726-2","volume":"12","author":"RT Sutton","year":"2022","unstructured":"Sutton, R. T., Zaiane, O. R., Goebel, R. & Baumgart, D. C. Artificial intelligence enabled automated diagnosis and grading of ulcerative colitis endoscopy images. Sci. Rep. 12, 2748 (2022).","journal-title":"Sci. Rep."},{"key":"12827_CR8","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2818\u20132826 (2016).","DOI":"10.1109\/CVPR.2016.308"},{"key":"12827_CR9","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770\u2013778 (2016).","DOI":"10.1109\/CVPR.2016.90"},{"key":"12827_CR10","unstructured":"Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (2015)."},{"key":"12827_CR11","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K.\u00a0Q. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4700\u20134708 (2017).","DOI":"10.1109\/CVPR.2017.243"},{"key":"12827_CR12","doi-asserted-by":"crossref","unstructured":"Szegedy, C. et al. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1\u20139 (2015).","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"12827_CR13","doi-asserted-by":"crossref","unstructured":"Zhang, X., Zhou, X., Lin, M. & Sun, J. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6848\u20136856 (2018).","DOI":"10.1109\/CVPR.2018.00716"},{"key":"12827_CR14","doi-asserted-by":"crossref","unstructured":"Uparkar, O., Bharti, J., Pateriya, R., Gupta, R.\u00a0K. & Sharma, A. Vision transformer outperforms deep convolutional neural network-based model in classifying X-ray images. Proc. Comput. Sci. 218, 2338\u20132349. 10.1016\/j.procs.2023.01.209 (2023). International Conference on Machine Learning and Data Engineering.","DOI":"10.1016\/j.procs.2023.01.209"},{"key":"12827_CR15","doi-asserted-by":"publisher","first-page":"3453","DOI":"10.3390\/app13063453","volume":"13","author":"H Shin","year":"2023","unstructured":"Shin, H., Jeon, S., Seol, Y., Kim, S. & Kang, D. Vision transformer approach for classification of Alzheimer\u2019s disease using 18F-florbetaben brain images. Appl. Sci. 13, 3453 (2023).","journal-title":"Appl. Sci."},{"key":"12827_CR16","doi-asserted-by":"publisher","first-page":"1282200","DOI":"10.3389\/fmed.2023.1282200","volume":"10","author":"M Waseem Sabir","year":"2023","unstructured":"Waseem Sabir, M., Farhan, M., Almalki, N. S., Alnfiai, M. M. & Sampedro, G. A. Fibrovit-vision transformer-based framework for detection and classification of pulmonary fibrosis from chest CT images. Front. Med. 10, 1282200 (2023).","journal-title":"Front. Med."},{"key":"12827_CR17","unstructured":"Uden, C.\u00a0V. et al. Exploring the versatility of zero-shot clip for interstitial lung disease classification. CoRR arXiv:2306.01111 (2023)."},{"key":"12827_CR18","unstructured":"Radford, A. et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. 8748\u20138763 (PMLR, 2021)."},{"key":"12827_CR19","unstructured":"Phan, T.\u00a0V., Nguyen, T.\u00a0K., Hoang, Q.\u00a0A., Phan, Q.\u00a0T. & Nguyen-Tat, T.\u00a0B. Uit-2q2t at imageclefmedical 2024 caption: Multimodal medical image captioning using bootstrapping language-image pre-training. In CLEF 2024: Conference and Labs of the Evaluation Forum (2024)."},{"key":"12827_CR20","unstructured":"Xu, K. et al. Show, attend and tell: Neural image caption generation with visual attention. In (Bach, F. & Blei, D. eds.) Proceedings of the 32nd International Conference on Machine Learning. Vol.\u00a037. Proceedings of Machine Learning Research. 2048\u20132057 (PMLR, 2015)."},{"key":"12827_CR21","first-page":"1877","volume":"33","author":"T Brown","year":"2020","unstructured":"Brown, T. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877\u20131901 (2020).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"12827_CR22","unstructured":"Li, J., Li, D., Xiong, C. & Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning. 12888\u201312900 (PMLR, 2022)."},{"key":"12827_CR23","doi-asserted-by":"crossref","unstructured":"Liu, J. et al. Clip-driven universal model for organ segmentation and tumor detection. In Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV). 21152\u201321164 (2023).","DOI":"10.1109\/ICCV51070.2023.01934"},{"key":"12827_CR24","doi-asserted-by":"crossref","unstructured":"Lu, L., Cui, X., Tan, Z. & Wu, Y. Medoptnet: Meta-learning framework for few-shot medical image classification. In IEEE\/ACM Transactions on Computational Biology and Bioinformatics (2023).","DOI":"10.1109\/TCBB.2023.3284846"},{"key":"12827_CR25","doi-asserted-by":"publisher","first-page":"4979","DOI":"10.1007\/s11042-021-11472-0","volume":"81","author":"X-J Liu","year":"2022","unstructured":"Liu, X.-J., Li, K.-L., Luan, H.-Y., Wang, W.-H. & Chen, Z.-Y. Few-shot learning for skin lesion image classification. Multimed. Tools Appl. 81, 4979\u20134990 (2022).","journal-title":"Multimed. Tools Appl."},{"key":"12827_CR26","doi-asserted-by":"crossref","unstructured":"Codella, N. C.\u00a0F. et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), hosted by the International Skin Imaging Collaboration (ISIC). In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). 168\u2013172. 10.1109\/ISBI.2018.8363547 (2018).","DOI":"10.1109\/ISBI.2018.8363547"},{"key":"12827_CR27","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1186\/s12880-023-01089-0","volume":"23","author":"X Cheng","year":"2023","unstructured":"Cheng, X. et al. Predicting response to CCRT for esophageal squamous carcinoma by a radiomics-clinical Shap model. BMC Med. Imaging 23, 145 (2023).","journal-title":"BMC Med. Imaging"},{"key":"12827_CR28","doi-asserted-by":"crossref","unstructured":"Singh, A. et al. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 15638\u201315650 (2022).","DOI":"10.1109\/CVPR52688.2022.01519"},{"key":"12827_CR29","doi-asserted-by":"publisher","first-page":"18391","DOI":"10.1007\/s00521-022-07451-7","volume":"34","author":"MU Salur","year":"2022","unstructured":"Salur, M. U. & Ayd\u0131n, \u0130. A soft voting ensemble learning-based approach for multimodal sentiment analysis. Neural Comput. Appl. 34, 18391\u201318406 (2022).","journal-title":"Neural Comput. Appl."},{"key":"12827_CR30","doi-asserted-by":"crossref","unstructured":"Chen, Y., Liu, Z., Xu, H., Darrell, T. & Wang, X. Meta-baseline: Exploring simple meta-learning for few-shot learning. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 9062\u20139071 (2021).","DOI":"10.1109\/ICCV48922.2021.00893"},{"key":"12827_CR31","unstructured":"Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D. et al. Matching networks for one shot learning. Adv. Neural Inf. Process. Syst. 29 (2016)."},{"key":"12827_CR32","unstructured":"Snell, J., Swersky, K. & Zemel, R. Prototypical networks for few-shot learning. Adv. Neural Inf. Process. Syst. 30, 122 (2017)."},{"key":"12827_CR33","doi-asserted-by":"crossref","unstructured":"Lin, Z. & Wang, S. Optimization of expression recognition by resnet18 based on meta-learning. In Proceedings of the 4th International Conference on Artificial Intelligence and Computer Engineering. 580\u2013585 (2023).","DOI":"10.1145\/3652628.3652724"},{"key":"12827_CR34","doi-asserted-by":"publisher","first-page":"15394","DOI":"10.1109\/TPAMI.2023.3320731","volume":"45","author":"S Nag","year":"2023","unstructured":"Nag, S., Raychaudhuri, D. S., Paul, S. & Roy-Chowdhury, A. K. Reconstruction guided meta-learning for few shot open set recognition. IEEE Trans. Pattern Anal. Mach. Intell. 45, 15394\u201315405 (2023).","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"12827_CR35","doi-asserted-by":"crossref","unstructured":"Ye, H.-J., Hu, H., Zhan, D.-C. & Sha, F. Few-shot learning via embedding adaptation with set-to-set functions. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 8808\u20138817 (2020).","DOI":"10.1109\/CVPR42600.2020.00883"},{"key":"12827_CR36","doi-asserted-by":"publisher","first-page":"108111","DOI":"10.1016\/j.patcog.2021.108111","volume":"120","author":"R Singh","year":"2021","unstructured":"Singh, R. et al. Metamed: Few-shot medical image classification using gradient-based meta-learning. Pattern Recognit. 120, 108111 (2021).","journal-title":"Pattern Recognit."},{"key":"12827_CR37","first-page":"24581","volume":"34","author":"S Laenen","year":"2021","unstructured":"Laenen, S. & Bertinetto, L. On episodes, prototypical networks, and few-shot learning. Adv. Neural Inf. Process. Syst. 34, 24581\u201324592 (2021).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"12827_CR38","unstructured":"Finn, C., Abbeel, P. & Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning. 1126\u20131135 (PMLR, 2017)."},{"key":"12827_CR39","unstructured":"Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (2021)."},{"key":"12827_CR40","doi-asserted-by":"crossref","unstructured":"Touvron, H. et al. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning. 10347\u201310357 (PMLR, 2021).","DOI":"10.1109\/ICCV48922.2021.00010"},{"key":"12827_CR41","doi-asserted-by":"crossref","unstructured":"Liu, Z. et al. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 10012\u201310022 (2021).","DOI":"10.1109\/ICCV48922.2021.00986"},{"key":"12827_CR42","unstructured":"Hinton, G. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)."},{"key":"12827_CR43","unstructured":"Lundberg, S.\u00a0M. & Lee, S.-I. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS\u201917, 4768\u20134777 (Curran Associates Inc., 2017)."},{"key":"12827_CR44","unstructured":"Kvasir. kvasir-Segmentation Dataset. https:\/\/universe.roboflow.com\/kvasir-kzzsf\/kvasir-segmentation (2024). Accessed 28 Apr 2025."},{"key":"12827_CR45","unstructured":"Liu, H., Li, C., Wu, Q. & Lee, Y.\u00a0J. Visual instruction tuning. In Thirty-Seventh Conference on Neural Information Processing Systems (2023)."},{"key":"12827_CR46","unstructured":"Liu, H. et al. Llava-next: Improved reasoning, OCR, and world knowledge (2024)."},{"key":"12827_CR47","first-page":"23716","volume":"35","author":"J-B Alayrac","year":"2022","unstructured":"Alayrac, J.-B. et al. Flamingo: A visual language model for few-shot learning. Adv. Neural Inf. Process. Syst. 35, 23716\u201323736 (2022).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"12827_CR48","doi-asserted-by":"publisher","first-page":"1112","DOI":"10.1109\/TNSE.2024.3524077","volume":"12","author":"Y Yang","year":"2025","unstructured":"Yang, Y. et al. Integrating fuzzy clustering and graph convolution network to accurately identify clusters from attributed graph. IEEE Trans. Netw. Sci. Eng. 12, 1112\u20131125. https:\/\/doi.org\/10.1109\/TNSE.2024.3524077 (2025).","journal-title":"IEEE Trans. Netw. Sci. Eng."}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-025-12827-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-025-12827-5","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-025-12827-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,8]],"date-time":"2025-09-08T01:06:54Z","timestamp":1757293614000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-025-12827-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,26]]},"references-count":48,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["12827"],"URL":"https:\/\/doi.org\/10.1038\/s41598-025-12827-5","relation":{},"ISSN":["2045-2322"],"issn-type":[{"type":"electronic","value":"2045-2322"}],"subject":[],"published":{"date-parts":[[2025,7,26]]},"assertion":[{"value":"5 March 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 July 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 July 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"27205"}}