{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,5]],"date-time":"2026-07-05T09:05:37Z","timestamp":1783242337855,"version":"3.54.6"},"reference-count":49,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2021,6,2]],"date-time":"2021-06-02T00:00:00Z","timestamp":1622592000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>Medical image annotation is a major hurdle for developing precise and robust machine-learning models. Annotation is expensive, time-consuming, and often requires expert knowledge, particularly in the medical field. Here, we suggest using minimal user interaction in the form of extreme point clicks to train a segmentation model which, in effect, can be used to speed up medical image annotation. An initial segmentation is generated based on the extreme points using the random walker algorithm. This initial segmentation is then used as a noisy supervision signal to train a fully convolutional network that can segment the organ of interest, based on the provided user clicks. Through experimentation on several medical imaging datasets, we show that the predictions of the network can be refined using several rounds of training with the prediction from the same weakly annotated data. Further improvements are shown using the clicked points within a custom-designed loss and attention mechanism. Our approach has the potential to speed up the process of generating new training datasets for the development of new machine-learning and deep-learning-based models for, but not exclusively, medical image analysis.<\/jats:p>","DOI":"10.3390\/make3020026","type":"journal-article","created":{"date-parts":[[2021,6,2]],"date-time":"2021-06-02T21:23:41Z","timestamp":1622669021000},"page":"507-524","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":64,"title":["Going to Extremes: Weakly Supervised Medical Image Segmentation"],"prefix":"10.3390","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3662-8743","authenticated-orcid":false,"given":"Holger R.","family":"Roth","sequence":"first","affiliation":[{"name":"NVIDIA Corporation, Bethesda, MD 20814, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5031-4337","authenticated-orcid":false,"given":"Dong","family":"Yang","sequence":"additional","affiliation":[{"name":"NVIDIA Corporation, Bethesda, MD 20814, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5728-6869","authenticated-orcid":false,"given":"Ziyue","family":"Xu","sequence":"additional","affiliation":[{"name":"NVIDIA Corporation, Bethesda, MD 20814, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaosong","family":"Wang","sequence":"additional","affiliation":[{"name":"NVIDIA Corporation, Bethesda, MD 20814, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daguang","family":"Xu","sequence":"additional","affiliation":[{"name":"NVIDIA Corporation, Bethesda, MD 20814, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,6,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"630","DOI":"10.1148\/radiol.2017151022","article-title":"Use of Volumetry for Lung Nodule Management: Theory and Practice","volume":"284","author":"Devaraj","year":"2017","journal-title":"Radiology"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1116","DOI":"10.1016\/j.neuroimage.2006.01.015","article-title":"User-Guided 3D Active Contour Segmentation of Anatomical Structures: Significantly Improved Efficiency and Reliability","volume":"31","author":"Yushkevich","year":"2006","journal-title":"Neuroimage"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1768","DOI":"10.1109\/TPAMI.2006.233","article-title":"Random walks for image segmentation","volume":"28","author":"Grady","year":"2006","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Long, J., Shelhamer, E., and Darrell, T. (2015, January 7\u201312). Fully convolutional networks for semantic segmentation. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5\u20139). U-net: Convolutional Networks for Biomedical Image Segmentation. Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention\u2014MICCAI 2015, Munich, Germany.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Milletari, F., Navab, N., and Ahmadi, S.A. (2016, January 25\u201328). V-net: Fully convolutional neural networks for volumetric medical image segmentation. Proceedings of the 2016 Fourth International Conference on 3D Vision (3DV), Stanford, CA, USA.","DOI":"10.1109\/3DV.2016.79"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"\u00c7i\u00e7ek, \u00d6., Abdulkadir, A., Lienkamp, S.S., Brox, T., and Ronneberger, O. (2016, January 17\u201321). 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation. Proceedings of the 19th International Conference on Medical Image Computing and Computer Assisted Intervention, Athens, Greece.","DOI":"10.1007\/978-3-319-46723-8_49"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Liu, S., Xu, D., Zhou, S.K., Pauly, O., Grbic, S., Mertelmeier, T., Wicklein, J., Jerebko, A., Cai, W., and Comaniciu, D. (2018, January 16\u201320). 3D Anisotropic Hybrid Network: Transferring Convolutional Features from 2d Images to 3d Anisotropic Volumes. Proceedings of the International Conference on Medical Image Computing & Computer Assisted Intervention, Granada, Spain.","DOI":"10.1007\/978-3-030-00934-2_94"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Myronenko, A. (2018). 3D MRI brain tumor segmentation using autoencoder regularization. International MICCAI Brainlesion Workshop, Springer.","DOI":"10.1007\/978-3-030-11726-9_28"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1007\/s13735-017-0141-z","article-title":"A review of semantic segmentation using deep neural networks","volume":"7","author":"Guo","year":"2018","journal-title":"Int. J. Multimed. Inf. Retr."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"101693","DOI":"10.1016\/j.media.2020.101693","article-title":"Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation","volume":"63","author":"Tajbakhsh","year":"2020","journal-title":"Med. Image Anal."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1016\/j.aanat.2016.11.009","article-title":"Accuracy and efficiency of computer-aided anatomical analysis using 3D visualization software based on semi-automated and automated segmentations","volume":"210","author":"An","year":"2017","journal-title":"Ann. Anat. Anat. Anz."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1007\/s11263-006-7934-5","article-title":"Graph cuts and efficient ND image segmentation","volume":"70","author":"Boykov","year":"2006","journal-title":"IJCV"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1206","DOI":"10.1117\/12.480165","article-title":"Interactive shape models","volume":"5032","author":"Loog","year":"2003","journal-title":"Med. Imaging 2003 Image Process. Int. Soc. Opt. Photonics"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Schwarz, T., Heimann, T., Wolf, I., and Meinzer, H.P. (October, January 30). 3D heart segmentation and volumetry using deformable shape models. Proceedings of the 2007 Computers in Cardiology, Durham, NC, USA.","DOI":"10.1109\/CIC.2007.4745592"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Dougherty, G. (2011). Medical Image Processing: Techniques and Applications, Springer Science & Business Media.","DOI":"10.1007\/978-1-4419-9779-1"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1016\/j.media.2016.04.009","article-title":"Slic-Seg: A minimally interactive segmentation of the placenta from sparse and motion-corrupted fetal MRI in multiple views","volume":"34","author":"Wang","year":"2016","journal-title":"Med. Image Anal."},{"key":"ref_18","unstructured":"Amrehn, M., Gaube, S., Unberath, M., Schebesch, F., Horz, T., Strumia, M., Steidl, S., Kowarschik, M., and Maier, A. (2017). UI-Net: Interactive artificial neural networks for iterative image segmentation based on a user model. Eurographics Workshop Vis. Comput. Biol. Med."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1559","DOI":"10.1109\/TPAMI.2018.2840695","article-title":"DeepIGeoS: A deep interactive geodesic framework for medical image segmentation","volume":"41","author":"Wang","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1562","DOI":"10.1109\/TMI.2018.2791721","article-title":"Interactive medical image segmentation using deep learning with image-specific fine tuning","volume":"37","author":"Wang","year":"2018","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Can, Y.B., Chaitanya, K., Mustafa, B., Koch, L.M., Konukoglu, E., and Baumgartner, C.F. (2018). Learning to Segment Medical Images with Scribble-Supervision Alone. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer.","DOI":"10.1007\/978-3-030-00889-5_27"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Dias, P.A., Shen, Z., Tabb, A., and Medeiros, H. (2019, January 7\u201311). FreeLabel: A Publicly Available Annotation Tool Based on Freehand Traces. Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.","DOI":"10.1109\/WACV.2019.00010"},{"key":"ref_23","unstructured":"Sakinis, T., Milletari, F., Roth, H., Korfiatis, P., Kostandy, P., Philbrick, K., Akkus, Z., Xu, Z., Xu, D., and Erickson, B.J. (2019). Interactive segmentation of medical images through fully convolutional neural networks. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Khan, S., Shahin, A.H., Villafruela, J., Shen, J., and Shao, L. (2019). Extreme Points Derived Confidence Map as a Cue for Class-Agnostic Interactive Segmentation Using Deep Neural Network. Medical Image Computing and Computer Assisted Intervention\u2014MICCAI 2019, Springer International Publishing.","DOI":"10.1007\/978-3-030-32245-8_8"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Majumder, S., and Yao, A. (2019, January 15\u201320). Content-Aware Multi-Level Guidance for Interactive Instance Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.01187"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Ling, H., Gao, J., Kar, A., Chen, W., and Fidler, S. (2019, January 15\u201320). Fast Interactive Object Annotation With Curve-GCN. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00540"},{"key":"ref_27","unstructured":"Jawahar, C.V., Li, H., Mori, G., and Schindler, K. (2019). Semantic Segmentation Refinement by Monte Carlo Region Growing of High Confidence Detections, Springer International Publishing. Computer Vision\u2014ACCV 2018."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Cerrone, L., Zeilmann, A., and Hamprecht, F.A. (2019, January 15\u201320). End-To-End Learned Random Walker for Seeded Image Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.01284"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"674","DOI":"10.1109\/TMI.2016.2621185","article-title":"Deepcut: Object segmentation from bounding box annotations using convolutional neural networks","volume":"36","author":"Rajchl","year":"2017","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Zhang, L., Gopalakrishnan, V., Lu, L., Summers, R.M., Moss, J., and Yao, J. (2018, January 4\u20137). Self-learning to detect and segment cysts in lung CT images without manual annotation. Proceedings of the 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), Washington, DC, USA.","DOI":"10.1109\/ISBI.2018.8363763"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Nie, D., Gao, Y., Wang, L., and Shen, D. (2018). ASDNet: Attention Based Semi-supervised Deep Networks for Medical Image Segmentation. Medical Image Computing and Computer Assisted Intervention\u2014MICCAI 2018, Springer International Publishing.","DOI":"10.1007\/978-3-030-00937-3_43"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1016\/j.media.2019.02.009","article-title":"Constrained-CNN losses for weakly supervised segmentation","volume":"54","author":"Kervadec","year":"2019","journal-title":"Med Image Anal."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Cai, J., Tang, Y., Lu, L., Harrison, A.P., Yan, K., Xiao, J., Yang, L., and Summers, R.M. (2018). Accurate weakly-supervised deep lesion segmentation using large-scale clinical annotations: Slice-propagated 3D mask generation from 2D RECIST. International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.","DOI":"10.1007\/978-3-030-00937-3_46"},{"key":"ref_34","unstructured":"Rajchl, M., Lee, M.C., Schrans, F., Davidson, A., Passerat-Palmbach, J., Tarroni, G., Alansary, A., Oktay, O., Kainz, B., and Rueckert, D. (2016). Learning under distributed weak supervision. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Roth, H., Zhang, L., Yang, D., Milletari, F., Xu, Z., Wang, X., and Xu, D. (2019). Weakly supervised segmentation from extreme points. Large-Scale Annotation of Biomedical Data and Expert Label Synthesis (LABELS) and Hardware Aware Learning (HAL) for Medical Imaging and Computer Assisted Intervention (MICCAI), Springer.","DOI":"10.1007\/978-3-030-33642-4_5"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Maninis, K.K., Caelles, S., Pont-Tuset, J., and Van Gool, L. (2018, January 18\u201323). Deep Extreme Cut: From Extreme Points to Object Segmentation. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00071"},{"key":"ref_37","unstructured":"Oktay, O., Schlemper, J., Folgoc, L.L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N.Y., and Kainz, B. (2018, January 4\u20136). Attention u-net: Learning where to look for the pancreas. Proceedings of the 1st Conference on Medical Imaging with Deep Learning (MIDL), Amsterdam, The Netherlands."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1016\/j.media.2018.01.006","article-title":"Spatial aggregation of holistically-nested convolutional neural networks for automated pancreas localization and segmentation","volume":"45","author":"Roth","year":"2018","journal-title":"Med Image Anal."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Papadopoulos, D.P., Uijlings, J.R., Keller, F., and Ferrari, V. (2017, January 22\u201329). Extreme clicking for efficient object annotation. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.528"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1007\/BF01386390","article-title":"A note on two problems in connexion with graphs","volume":"1","author":"Dijkstra","year":"1959","journal-title":"Numer. Math."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Wu, Y., and He, K. (2018, January 8\u201314). Group normalization. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01261-8_1"},{"key":"ref_43","unstructured":"Ioffe, S., and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1822","DOI":"10.1109\/TMI.2018.2806309","article-title":"Automatic multi-organ segmentation on abdominal CT with dense v-networks","volume":"37","author":"Gibson","year":"2018","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Roth, H.R., Lu, L., Farag, A., Shin, H.C., Liu, J., Turkbey, E.B., and Summers, R.M. (2015, January 5\u20139). Deeporgan: Multi-level deep convolutional networks for automated pancreas segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.","DOI":"10.1007\/978-3-319-24553-9_68"},{"key":"ref_46","unstructured":"BTCV (2021, May 28). Multi-Atlas Labeling Beyond the Cranial Vault\u2014MICCAI Workshop and Challenge. Available online: https:\/\/www.synapse.org\/#!Synapse:syn3193805."},{"key":"ref_47","unstructured":"Simpson, A.L., Antonelli, M., Bakas, S., Bilello, M., Farahani, K., van Ginneken, B., Kopp-Schneider, A., Landman, B.A., Litjens, G., and Menze, B. (2019). A large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41597-020-00715-8","article-title":"CT-ORG, a new dataset for multiple organ segmentation in computed tomography","volume":"7","author":"Rister","year":"2020","journal-title":"Sci. Data"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Raju, A., Ji, Z., Cheng, C.T., Cai, J., Huang, J., Xiao, J., Lu, L., Liao, C., and Harrison, A.P. (2020, January 4\u20138). User-Guided Domain Adaptation for Rapid Annotation from User Interactions: A Study on Pathological Liver Segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Lima, Peru.","DOI":"10.1007\/978-3-030-59710-8_45"}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/3\/2\/26\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:10:06Z","timestamp":1760163006000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/3\/2\/26"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,2]]},"references-count":49,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2021,6]]}},"alternative-id":["make3020026"],"URL":"https:\/\/doi.org\/10.3390\/make3020026","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,6,2]]}}}