{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T16:38:20Z","timestamp":1791995900675,"version":"4.4.0"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,10,21]],"date-time":"2024-10-21T00:00:00Z","timestamp":1729468800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,10,21]],"date-time":"2024-10-21T00:00:00Z","timestamp":1729468800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"NTNU Norwegian University of Science and Technology"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Ulcerative Colitis is an Inflammatory Bowel disease caused by a variety of factors that lead to a serious impact on the quality of life of the patients if left untreated. Due to complexities in the identification procedures of this disease, the treatment timeline and quality can be severely affected, leading to further consequences for the sufferer. The difficulties in identification are due to high patients to healthcare professionals ratio. Researchers have proposed variety of machine\/deep learning methods for automated detection of ulcerative colitis, however, several challenges exists including class imbalance problem, comprehensive feature extraction and accurate classification. We propose a novel method for accurate detection of ulcerative colitis with augmentation techniques to overcome class imbalance issue, a comprehensive feature vector extraction using custom architecture of Vision Transformer (ViT) and accurate classification using customized Convolutional Neural Network (CNN). We used the TMC-UCM and LIMUC datasets in this research for training and testing of proposed method and achieved accuracy of 90% with AUC-ROC scores of 0.91, 0.81, 0.94, and 0.94 for the endoscopic classes of Mayo 0, Mayo 1, Mayo 2, and Mayo 3 respectively. We have compared the proposed method with existing state of the art methods and conclude that the proposed method outperforms the existing methods.<\/jats:p>","DOI":"10.1038\/s41598-024-75901-4","type":"journal-article","created":{"date-parts":[[2024,10,21]],"date-time":"2024-10-21T16:03:25Z","timestamp":1729526605000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":22,"title":["A hybrid approach of vision transformers and CNNs for detection of ulcerative colitis"],"prefix":"10.1038","volume":"14","author":[{"given":"Syed Abdullah","family":"Shah","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Imran","family":"Taj","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Syed Muhammad","family":"Usman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Syed Nehal","family":"Hassan Shah","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ali Shariq","family":"Imran","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shehzad","family":"Khalid","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,10,21]]},"reference":[{"key":"75901_CR1","doi-asserted-by":"crossref","unstructured":"Polat, G. et al. Class distance weighted cross-entropy loss for ulcerative colitis severity estimation. In Annual Conference on Medical Image Understanding and Analysis 157\u2013171 (Springer, 2022).","DOI":"10.1007\/978-3-031-12053-4_12"},{"key":"75901_CR2","first-page":"56","volume":"2022","author":"F Jiang","year":"2022","unstructured":"Jiang, F. et al. Artificial intelligence algorithm-based differential diagnosis of crohns disease and ulcerative colitis by ct image. Comput. Math. Methods Med. 2022, 56 (2022).","journal-title":"Comput. Math. Methods Med."},{"key":"75901_CR3","unstructured":"Penman, I.\u00a0D., Ralston, S.\u00a0H., Strachan, M.\u00a0W. & Hobson, R. Davidson\u2019s Principles and Practice of Medicine E-Book: Davidson\u2019s Principles and Practice of Medicine E-Book (Elsevier Health Sciences, 2022)."},{"key":"75901_CR4","doi-asserted-by":"publisher","first-page":"1421","DOI":"10.1111\/cts.13542","volume":"16","author":"G Wang","year":"2023","unstructured":"Wang, G. et al. Cb-hrnet: A class-balanced high-resolution network for the evaluation of endoscopic activity in patients with ulcerative colitis. Clin. Transl. Sci. 16, 1421\u20131430 (2023).","journal-title":"Clin. Transl. Sci."},{"key":"75901_CR5","doi-asserted-by":"crossref","unstructured":"Carreras, J. Artificial intelligence analysis of ulcerative colitis using an autoimmune discovery transcriptomic panel. In Healthcare, vol.\u00a010 1476 (MDPI, 2022).","DOI":"10.3390\/healthcare10081476"},{"key":"75901_CR6","doi-asserted-by":"publisher","first-page":"889","DOI":"10.1136\/gutjnl-2021-326376","volume":"71","author":"X Gui","year":"2022","unstructured":"Gui, X. et al. Picasso histologic remission index (phri) in ulcerative colitis: development of a novel simplified histological score for monitoring mucosal healing and predicting clinical outcomes and its applicability in an artificial intelligence system. Gut 71, 889\u2013898 (2022).","journal-title":"Gut"},{"key":"75901_CR7","doi-asserted-by":"publisher","first-page":"1180","DOI":"10.1053\/j.gastro.2023.02.031","volume":"164","author":"M Iacucci","year":"2023","unstructured":"Iacucci, M. et al. Artificial intelligence enabled histological prediction of remission or activity and clinical outcomes in ulcerative colitis. Gastroenterology 164, 1180\u20131188 (2023).","journal-title":"Gastroenterology"},{"key":"75901_CR8","doi-asserted-by":"publisher","first-page":"14133","DOI":"10.1007\/s00521-021-06055-x","volume":"33","author":"IV Popa","year":"2021","unstructured":"Popa, I. V. et al. A new approach to predict ulcerative colitis activity through standard clinical-biological parameters using a robust neural network model. Neural Comput. Appl. 33, 14133\u201314146 (2021).","journal-title":"Neural Comput. Appl."},{"key":"75901_CR9","doi-asserted-by":"publisher","first-page":"710","DOI":"10.1053\/j.gastro.2020.10.024","volume":"160","author":"K Gottlieb","year":"2021","unstructured":"Gottlieb, K. et al. Central reading of ulcerative colitis clinical trial videos using neural networks. Gastroenterology 160, 710\u2013719 (2021).","journal-title":"Gastroenterology"},{"key":"75901_CR10","doi-asserted-by":"publisher","first-page":"634","DOI":"10.3390\/s23020634","volume":"23","author":"A Chaddad","year":"2023","unstructured":"Chaddad, A., Peng, J., Xu, J. & Bouridane, A. Survey of explainable ai techniques in healthcare. Sensors 23, 634 (2023).","journal-title":"Sensors"},{"key":"75901_CR11","doi-asserted-by":"publisher","first-page":"263177452199062","DOI":"10.1177\/2631774521990623","volume":"14","author":"B Gutierrez Becker","year":"2021","unstructured":"Gutierrez Becker, B. et al. Training and deploying a deep learning model for endoscopic severity grading in ulcerative colitis using multicenter clinical trial data. Therapeut. Adv. Gastrointest. Endosc. 14, 2631774521990623 (2021).","journal-title":"Therapeut. Adv. Gastrointest. Endosc."},{"key":"75901_CR12","doi-asserted-by":"publisher","first-page":"11351","DOI":"10.1038\/s41598-023-38206-6","volume":"13","author":"JE Kim","year":"2023","unstructured":"Kim, J. E. et al. Deep learning model for distinguishing mayo endoscopic subscore 0 and 1 in patients with ulcerative colitis. Sci. Rep. 13, 11351 (2023).","journal-title":"Sci. Rep."},{"key":"75901_CR13","doi-asserted-by":"publisher","first-page":"e193963","DOI":"10.1001\/jamanetworkopen.2019.3963","volume":"2","author":"RW Stidham","year":"2019","unstructured":"Stidham, R. W. et al. Performance of a deep learning model vs human reviewers in grading endoscopic disease severity of patients with ulcerative colitis. JAMA Netw. Open 2, e193963\u2013e193963 (2019).","journal-title":"JAMA Netw. Open"},{"key":"75901_CR14","doi-asserted-by":"publisher","first-page":"678","DOI":"10.1097\/JCMA.0000000000000559","volume":"84","author":"T-Y Huang","year":"2021","unstructured":"Huang, T.-Y., Zhan, S.-Q., Chen, P.-J., Yang, C.-W. & Lu, H.H.-S. Accurate diagnosis of endoscopic mucosal healing in ulcerative colitis using deep learning and machine learning. J. Chin. Med. Assoc. 84, 678\u2013681 (2021).","journal-title":"J. Chin. Med. Assoc."},{"key":"75901_CR15","doi-asserted-by":"publisher","first-page":"E171","DOI":"10.1055\/a-1675-1941","volume":"10","author":"M Mascarenhas","year":"2022","unstructured":"Mascarenhas, M. et al. Deep learning and colon capsule endoscopy: automatic detection of blood and colonic mucosal lesions using a convolutional neural network. Endosc. Int. Open 10, E171\u2013E177 (2022).","journal-title":"Endosc. Int. Open"},{"key":"75901_CR16","doi-asserted-by":"publisher","first-page":"854677","DOI":"10.3389\/fmed.2022.854677","volume":"9","author":"G Ruan","year":"2022","unstructured":"Ruan, G. et al. Development and validation of a deep neural network for accurate identification of endoscopic images from patients with ulcerative colitis and crohns disease. Front. Med. 9, 854677 (2022).","journal-title":"Front. Med."},{"key":"75901_CR17","doi-asserted-by":"publisher","first-page":"300","DOI":"10.1186\/s12911-022-02043-w","volume":"22","author":"M Chierici","year":"2022","unstructured":"Chierici, M. et al. Automatically detecting crohn\u2019s disease and ulcerative colitis from endoscopic imaging. BMC Med. Inform. Decis. Mak. 22, 300 (2022).","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"75901_CR18","doi-asserted-by":"publisher","first-page":"13744","DOI":"10.1038\/s41598-020-70583-0","volume":"10","author":"HM Khorasani","year":"2020","unstructured":"Khorasani, H. M., Usefi, H. & Pena-Castillo, L. Detecting ulcerative colitis from colon samples using efficient feature selection and machine learning. Sci. Rep. 10, 13744 (2020).","journal-title":"Sci. Rep."},{"key":"75901_CR19","doi-asserted-by":"publisher","first-page":"283","DOI":"10.1038\/s41597-020-00622-y","volume":"7","author":"H Borgli","year":"2020","unstructured":"Borgli, H. et al. Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Sci. Data 7, 283 (2020).","journal-title":"Sci. Data"},{"key":"75901_CR20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10044-023-01206-3","volume":"27","author":"J Maur\u00edcio","year":"2024","unstructured":"Maur\u00edcio, J. & Domingues, I. Distinguishing between crohn\u2019s disease and ulcerative colitis using deep learning models with interpretability. Pattern Anal. Appl. 27, 1 (2024).","journal-title":"Pattern Anal. Appl."},{"key":"75901_CR21","doi-asserted-by":"publisher","first-page":"416","DOI":"10.1016\/j.gie.2018.10.020","volume":"89","author":"T Ozawa","year":"2019","unstructured":"Ozawa, T. et al. Novel computer-assisted diagnosis system for endoscopic disease activity in patients with ulcerative colitis. Gastrointest. Endosc. 89, 416\u2013421 (2019).","journal-title":"Gastrointest. Endosc."},{"key":"75901_CR22","doi-asserted-by":"publisher","first-page":"1584","DOI":"10.3390\/jpm13111584","volume":"13","author":"J-H Kim","year":"2023","unstructured":"Kim, J.-H. et al. Using a deep learning model to address interobserver variability in the evaluation of ulcerative colitis (uc) severity. J. Personal. Med. 13, 1584 (2023).","journal-title":"J. Personal. Med."},{"key":"75901_CR23","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1016\/j.gie.2022.08.015","volume":"97","author":"Y Fan","year":"2023","unstructured":"Fan, Y. et al. Novel deep learning-based computer-aided diagnosis system for predicting inflammatory activity in ulcerative colitis. Gastrointest. Endosc. 97, 335\u2013346 (2023).","journal-title":"Gastrointest. Endosc."},{"key":"75901_CR24","doi-asserted-by":"publisher","first-page":"101987","DOI":"10.1016\/j.artmed.2020.101987","volume":"111","author":"G Vandewiele","year":"2021","unstructured":"Vandewiele, G. et al. Overly optimistic prediction results on imbalanced data: a case study of flaws and benefits when applying over-sampling. Artif. Intell. Med. 111, 101987 (2021).","journal-title":"Artif. Intell. Med."},{"key":"75901_CR25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40537-019-0192-5","volume":"6","author":"JM Johnson","year":"2019","unstructured":"Johnson, J. M. & Khoshgoftaar, T. M. Survey on deep learning with class imbalance. J. Big Data 6, 1\u201354 (2019).","journal-title":"J. Big Data"},{"key":"75901_CR26","doi-asserted-by":"crossref","unstructured":"Zhang, P. et al. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. In Proceedings of the IEEE\/CVF International Conference on Computer Vision 2998\u20133008 (2021).","DOI":"10.1109\/ICCV48922.2021.00299"},{"key":"75901_CR27","doi-asserted-by":"crossref","unstructured":"Chen, H. et al. Pre-trained image processing transformer. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition 12299\u201312310 (2021).","DOI":"10.1109\/CVPR46437.2021.01212"},{"key":"75901_CR28","first-page":"11960","volume":"34","author":"Y Wang","year":"2021","unstructured":"Wang, Y., Huang, R., Song, S., Huang, Z. & Huang, G. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition. Adv. Neural. Inf. Process. Syst. 34, 11960\u201311973 (2021).","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"75901_CR29","doi-asserted-by":"crossref","unstructured":"Wang, W. et al. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE\/CVF International Conference on Computer Vision 568\u2013578 (2021).","DOI":"10.1109\/ICCV48922.2021.00061"},{"key":"75901_CR30","unstructured":"Zhou, D. et al. Deepvit: Towards deeper vision transformer. arXiv:2103.11886 (2021)."},{"key":"75901_CR31","doi-asserted-by":"crossref","unstructured":"Liu, Z. et al. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE\/CVF International Conference on Computer Vision 10012\u201310022 (2021).","DOI":"10.1109\/ICCV48922.2021.00986"},{"key":"75901_CR32","unstructured":"Lee, S.\u00a0H., Lee, S. & Song, B.\u00a0C. Vision transformer for small-size datasets. arXiv:2112.13492 (2021)."},{"key":"75901_CR33","doi-asserted-by":"publisher","first-page":"18962","DOI":"10.1109\/ACCESS.2021.3054735","volume":"9","author":"I Ali","year":"2021","unstructured":"Ali, I., Muzammil, M., Haq, I. U., Amir, M. & Abdullah, S. Deep feature selection and decision level fusion for lungs nodule classification. IEEE Access 9, 18962\u201318973 (2021).","journal-title":"IEEE Access"},{"key":"75901_CR34","first-page":"1","volume":"2021","author":"X Cheng","year":"2021","unstructured":"Cheng, X., Tan, L. & Ming, F. Feature fusion based on convolutional neural network for breast cancer auxiliary diagnosis. Math. Probl. Eng. 2021, 1\u201310 (2021).","journal-title":"Math. Probl. Eng."}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-024-75901-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-024-75901-4","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-024-75901-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,21]],"date-time":"2024-10-21T16:08:55Z","timestamp":1729526935000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-024-75901-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,21]]},"references-count":34,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["75901"],"URL":"https:\/\/doi.org\/10.1038\/s41598-024-75901-4","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,21]]},"assertion":[{"value":"1 July 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 October 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 October 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"24771"}}