{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,26]],"date-time":"2026-09-26T05:14:16Z","timestamp":1790399656936,"version":"4.1.0"},"reference-count":51,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2022,4,22]],"date-time":"2022-04-22T00:00:00Z","timestamp":1650585600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["GM123558"],"award-info":[{"award-number":["GM123558"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["NRT1633631"],"award-info":[{"award-number":["NRT1633631"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>Colorectal cancer (CRC) is a leading cause of mortality worldwide, and preventive screening modalities such as colonoscopy have been shown to noticeably decrease CRC incidence and mortality. Improving colonoscopy quality remains a challenging task due to limiting factors including the training levels of colonoscopists and the variability in polyp sizes, morphologies, and locations. Deep learning methods have led to state-of-the-art systems for the identification of polyps in colonoscopy videos. In this study, we show that deep learning can also be applied to the segmentation of polyps in real time, and the underlying models can be trained using mostly weakly labeled data, in the form of bounding box annotations that do not contain precise contour information. A novel dataset, Polyp-Box-Seg of 4070 colonoscopy images with polyps from over 2000 patients, is collected, and a subset of 1300 images is manually annotated with segmentation masks. A series of models is trained to evaluate various strategies that utilize bounding box annotations for segmentation tasks. A model trained on the 1300 polyp images with segmentation masks achieves a dice coefficient of 81.52%, which improves significantly to 85.53% when using a weakly supervised strategy leveraging bounding box images. The Polyp-Box-Seg dataset, together with a real-time video demonstration of the segmentation system, are publicly available.<\/jats:p>","DOI":"10.3390\/jimaging8050121","type":"journal-article","created":{"date-parts":[[2022,4,23]],"date-time":"2022-04-23T08:14:06Z","timestamp":1650701646000},"page":"121","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Weakly Supervised Polyp Segmentation in Colonoscopy Images Using Deep Neural Networks"],"prefix":"10.3390","volume":"8","author":[{"given":"Siwei","family":"Chen","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of California, Irvine, CA 92697, USA"},{"name":"Institute for Genomics and Bioinformatics, University of California, Irvine, CA 92697, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gregor","family":"Urban","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of California, Irvine, CA 92697, USA"},{"name":"Institute for Genomics and Bioinformatics, University of California, Irvine, CA 92697, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8752-4664","authenticated-orcid":false,"given":"Pierre","family":"Baldi","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of California, Irvine, CA 92697, USA"},{"name":"Institute for Genomics and Bioinformatics, University of California, Irvine, CA 92697, USA"},{"name":"Center for Machine Learning and Intelligent Systems, University of California, Irvine, CA 92697, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,4,22]]},"reference":[{"key":"ref_1","unstructured":"(2022, April 18). Colorectal Cancer Statistics|Center of Disease Control and Prevention, Available online: https:\/\/www.cdc.gov\/cancer\/colorectal\/statistics\/."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"40950","DOI":"10.1109\/ACCESS.2018.2856402","article-title":"Automatic Colon Polyp Detection Using Region Based Deep CNN and Post Learning Approaches","volume":"6","author":"Shin","year":"2018","journal-title":"IEEE Access"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"e77","DOI":"10.1038\/ctg.2015.5","article-title":"Colonoscopy: Quality Indicators","volume":"6","author":"Anderson","year":"2015","journal-title":"Clin. Transl. Gastroenterol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"e7468","DOI":"10.1097\/MD.0000000000007468","article-title":"Risk factors of missed colorectal lesions after colonoscopy","volume":"96","author":"Lee","year":"2017","journal-title":"Medicine"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"254","DOI":"10.5946\/ce.2016.115","article-title":"Clinical and Biological Features of Interval Colorectal Cancer","volume":"50","author":"Lee","year":"2017","journal-title":"Clin. Endosc."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Troelsen, F.S., S\u00f8rensen, H.T., Crockett, S.D., Pedersen, L., and Erichsen, R. (2021). Characteristics and Survival of Patients with Inflammatory Bowel Disease and Postcolonoscopy Colorectal Cancers. Clin. Gastroenterol. Hepatol.","DOI":"10.1016\/j.cgh.2021.05.039"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"957","DOI":"10.1136\/gutjnl-2013-304880","article-title":"Postcolonoscopy colorectal cancers are preventable: A population-based study","volume":"63","author":"Bouwens","year":"2014","journal-title":"Gut"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2724","DOI":"10.1016\/j.cgh.2020.06.010","article-title":"Postcolonoscopy Colorectal Cancer in Sweden From 2003 to 2012: Survival, Tumor Characteristics, and Risk Factors","volume":"18","author":"Forsberg","year":"2020","journal-title":"Clin. Gastroenterol. Hepatol."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Macken, E., Dongen, S.V., Brabander, I.D., Francque, S., Driessen, A., and Hal, G.V. (2019). Post-colonoscopy colorectal cancer in Belgium: Characteristics and influencing factors. Endosc. Int. Open.","DOI":"10.1055\/a-0751-2660"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Baldi, P. (2021). Deep Learning in Science, Cambridge University Press.","DOI":"10.1017\/9781108955652"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_12","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Girshick, R.B. (2015). Fast R-CNN. arXiv.","DOI":"10.1109\/ICCV.2015.169"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Long, J., Shelhamer, E., and Darrell, T. (2014). Fully Convolutional Networks for Semantic Segmentation. arXiv.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1002\/lsm.23324","article-title":"Combining Deep Learning with Optical Coherence Tomography Imaging to Determine Scalp Hair and Follicle Counts","volume":"53","author":"Urban","year":"2021","journal-title":"Lasers Surg. Med."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1201","DOI":"10.3174\/ajnr.A5667","article-title":"Deep-Learning Convolutional Neural Networks Accurately Classify Genetic Mutations in Gliomas","volume":"39","author":"Chang","year":"2018","journal-title":"AJNR Am. J. Neuroradiol."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1029","DOI":"10.1109\/TCBB.2018.2841396","article-title":"Deep learning for drug discovery and cancer research: Automated analysis of vascularization images","volume":"16","author":"Urban","year":"2018","journal-title":"IEEE\/ACM Trans. Comput. Biol. Bioinform."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1172","DOI":"10.1109\/TMI.2017.2655486","article-title":"Detecting Cardiovascular Disease from Mammograms with Deep Learning","volume":"36","author":"Wang","year":"2017","journal-title":"EEE Trans. Med. Imaging"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1016\/j.compbiomed.2017.03.024","article-title":"A multi-resolution approach for spinal metastasis detection using deep Siamese neural networks","volume":"84","author":"Wang","year":"2017","journal-title":"Comput. Biol. Med."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"105123","DOI":"10.1016\/j.compbiomed.2021.105123","article-title":"Review and classification of AI-enabled COVID-19 CT imaging models based on computer vision tasks","volume":"141","author":"Hassan","year":"2022","journal-title":"Comput. Biol. Med."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Li, R., Xiao, C., Huang, Y., Hassan, H., and Huang, B. (2022). Deep Learning Applications in Computed Tomography Images for Pulmonary Nodule Detection and Diagnosis: A Review. Diagnostics, 12.","DOI":"10.3390\/diagnostics12020298"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"P1069","DOI":"10.1053\/j.gastro.2018.06.037","article-title":"Deep Learning Localizes and Identifies Polyps in Real Time with 96% Accuracy in Screening Colonoscopy","volume":"155","author":"Urban","year":"2018","journal-title":"Gastroenterology"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2169","DOI":"10.1053\/j.gastro.2020.02.036","article-title":"Improved Accuracy in Optical Diagnosis of Colorectal Polyps Using Convolutional Neural Networks with Visual Explanations","volume":"158","author":"Jin","year":"2020","journal-title":"Gastroenterology"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1756284820910659","DOI":"10.1177\/1756284820910659","article-title":"Automated endoscopic detection and classification of colorectal polyps using convolutional neural networks","volume":"13","author":"Ozawa","year":"2020","journal-title":"Ther. Adv. Gastroenterol."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3079765","article-title":"From Annotation to Computer-Aided Diagnosis: Detailed Evaluation of a Medical Multimedia System","volume":"13","author":"Riegler","year":"2017","journal-title":"ACM Trans. Multimed. Comput. Commun. Appl."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Tavanapong, W., Oh, J., Kijkul, G., Pratt, J., Wong, J., and de Groen, P. (2020, January 28\u201330). Real-Time Feedback for Colonoscopy in a Multicenter Clinical Trial. Proceedings of the 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems (CBMS), Rochester, MN, USA.","DOI":"10.1109\/CBMS49503.2020.00010"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1813","DOI":"10.1136\/gutjnl-2018-317500","article-title":"Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: A prospective randomised controlled study","volume":"68","author":"Wang","year":"2019","journal-title":"Gut"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"415","DOI":"10.1016\/j.gie.2019.08.026","article-title":"Impact of real-time automatic quality control system on colorectal polyp and adenoma detection: A prospective randomized controlled study (with video)","volume":"91","author":"Su","year":"2019","journal-title":"Gastrointest. Endosc."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"AB482","DOI":"10.1016\/j.gie.2016.03.671","article-title":"Mo1979 The Next Endoscopic Frontier: A Novel Computer Vision Program Accurately Identifies Colonoscopic Colorectal Adenomas","volume":"83","author":"Li","year":"2016","journal-title":"Gastrointest. Endosc."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"AB364","DOI":"10.1016\/j.gie.2017.03.843","article-title":"Su1614 Artificial Intelligence (AI) in Endoscopy\u2013Deep Learning for Optical Biopsy of Colorectal Polyps in Real-Time on Unaltered Endoscopic Videos","volume":"85","author":"Byrne","year":"2017","journal-title":"Gastrointest. Endosc."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Girshick, R.B., Donahue, J., Darrell, T., and Malik, J. (2013). Rich feature hierarchies for accurate object detection and semantic segmentation. arXiv.","DOI":"10.1109\/CVPR.2014.81"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S.K., Girshick, R.B., and Farhadi, A. (2015). You Only Look Once: Unified, Real-Time Object Detection. arXiv.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Brandao, P., Mazomenos, E., Ciuti, G., Cali\u00f2, R., Bianchi, F., Menciassi, A., Dario, P., Koulaouzidis, A., Arezzo, A., and Stoyanov, D. (2017, January 11\u201316). Fully convolutional neural networks for polyp segmentation in colonoscopy. Proceedings of the Medical Imaging 2017: Computer-Aided Diagnosis, Orlando, FL, USA.","DOI":"10.1117\/12.2254361"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"V\u00e1zquez, D., Bernal, J., S\u00e1nchez, F.J., Fern\u00e1ndez-Esparrach, G., L\u00f3pez, A.M., Romero, A., Drozdzal, M., and Courville, A.C. (2016). A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Images. arXiv.","DOI":"10.1155\/2017\/4037190"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Doll\u00e1r, P., and Girshick, R.B. (2017). Mask R-CNN. arXiv.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"588","DOI":"10.1515\/biol-2020-0055","article-title":"Automatic detection and segmentation of adenomatous colorectal polyps during colonoscopy using Mask R-CNN","volume":"15","author":"Meng","year":"2020","journal-title":"Open Life Sci."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1016\/j.compmedimag.2015.02.007","article-title":"WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians","volume":"43","author":"Bernal","year":"2015","journal-title":"Comput. Med. Imaging Graph."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Guo, Y., Bernal, J., and Matuszewski, B.J. (2020). Polyp Segmentation with Fully Convolutional Deep Neural Networks\u2014Extended Evaluation Study. J. Imaging, 6.","DOI":"10.3390\/jimaging6070069"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Jha, D., Smedsrud, P.H., Riegler, M.A., Johansen, D., de Lange, T., Halvorsen, P., and Johansen, H.D. (2019, January 9\u201311). ResUNet++: An Advanced Architecture for Medical Image Segmentation. Proceedings of the 2019 IEEE International Symposium on Multimedia (ISM), San Diego, CA, USA.","DOI":"10.1109\/ISM46123.2019.00049"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"99495","DOI":"10.1109\/ACCESS.2020.2995630","article-title":"Contour-Aware Polyp Segmentation in Colonoscopy Images Using Detailed Upsampling Encoder-Decoder Networks","volume":"8","author":"Nguyen","year":"2020","journal-title":"IEEE Access"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Jha, D., Smedsrud, P.H., Riegler, M.A., Halvorsen, P., de Lange, T., Johansen, D., and Johansen, H.D. (2019, January 8\u201311). Kvasir-SEG: A Segmented Polyp Dataset. Proceedings of the International Conference on Multimedia Modeling, Thessaloniki, Greece.","DOI":"10.1007\/978-3-030-37734-2_37"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"104119","DOI":"10.1016\/j.compbiomed.2020.104119","article-title":"PolypSegNet: A modified encoder-decoder architecture for automated polyp segmentation from colonoscopy images","volume":"128","author":"Mahmud","year":"2021","journal-title":"Comput. Biol. Med."},{"key":"ref_43","unstructured":"Tomar, N.K., Jha, D., Riegler, M.A., Johansen, H.D., Johansen, D., Rittscher, J., Halvorsen, P., and Ali, S. (2021). FANet: A Feedback Attention Network for Improved Biomedical Image Segmentation. arXiv."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Wu, H., Chen, G., Wen, Z., and Qin, J. (2021, January 11\u201317). Collaborative and Adversarial Learning of Focused and Dispersive Representations for Semi-supervised Polyp Segmentation. Proceedings of the 2021 IEEE\/CVF International Conference on Computer Vision (ICCV), Virtual.","DOI":"10.1109\/ICCV48922.2021.00347"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1122","DOI":"10.1136\/gut.52.8.1122","article-title":"Wireless capsule endoscopy: A comparison with push enteroscopy in patients with gastroscopy and colonoscopy negative gastrointestinal bleeding","volume":"52","author":"Mylonaki","year":"2003","journal-title":"Gut"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Dai, J., He, K., and Sun, J. (2015). BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation. arXiv.","DOI":"10.1109\/ICCV.2015.191"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"3412","DOI":"10.1093\/bioinformatics\/16.5.412","article-title":"Assessing the accuracy of prediction algorithms for classification: An overview","volume":"16","author":"Baldi","year":"2000","journal-title":"Bioinformatics"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"ImageNet Large Scale Visual Recognition Challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis."},{"key":"ref_50","unstructured":"Han, X., Laga, H., and Bennamoun, M. (2019). Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era. arXiv."},{"key":"ref_51","unstructured":"Seitz, S., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R. (2006, January 17\u201322). A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms. Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201906), New York, NY, USA."}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/8\/5\/121\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:59:04Z","timestamp":1760137144000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/8\/5\/121"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,22]]},"references-count":51,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,5]]}},"alternative-id":["jimaging8050121"],"URL":"https:\/\/doi.org\/10.3390\/jimaging8050121","relation":{},"ISSN":["2313-433X"],"issn-type":[{"value":"2313-433X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,22]]}}}