{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T02:35:42Z","timestamp":1760236542381,"version":"build-2065373602"},"reference-count":55,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2021,12,2]],"date-time":"2021-12-02T00:00:00Z","timestamp":1638403200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004663","name":"Ministry of Science and Technology","doi-asserted-by":"publisher","award":["MOST 109-2221-E-011-141"],"award-info":[{"award-number":["MOST 109-2221-E-011-141"]}],"id":[{"id":"10.13039\/501100004663","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>As the techniques of autonomous driving become increasingly valued and universal, real-time semantic segmentation has become very popular and challenging in the field of deep learning and computer vision in recent years. However, in order to apply the deep learning model to edge devices accompanying sensors on vehicles, we need to design a structure that has the best trade-off between accuracy and inference time. In previous works, several methods sacrificed accuracy to obtain a faster inference time, while others aimed to find the best accuracy under the condition of real time. Nevertheless, the accuracies of previous real-time semantic segmentation methods still have a large gap compared to general semantic segmentation methods. As a result, we propose a network architecture based on a dual encoder and a self-attention mechanism. Compared with preceding works, we achieved a 78.6% mIoU with a speed of 39.4 FPS with a 1024 \u00d7 2048 resolution on a Cityscapes test submission.<\/jats:p>","DOI":"10.3390\/s21238072","type":"journal-article","created":{"date-parts":[[2021,12,6]],"date-time":"2021-12-06T03:10:38Z","timestamp":1638760238000},"page":"8072","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Real-Time Semantic Segmentation with Dual Encoder and Self-Attention Mechanism for Autonomous Driving"],"prefix":"10.3390","volume":"21","author":[{"given":"Yu-Bang","family":"Chang","sequence":"first","affiliation":[{"name":"Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei City 106, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chieh","family":"Tsai","sequence":"additional","affiliation":[{"name":"Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei City 106, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3646-3261","authenticated-orcid":false,"given":"Chang-Hong","family":"Lin","sequence":"additional","affiliation":[{"name":"Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei City 106, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Poki","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei City 106, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,12,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"834","DOI":"10.1109\/TPAMI.2017.2699184","article-title":"DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs","volume":"40","author":"Chen","year":"2018","journal-title":"Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","unstructured":"Zhu, Z., Xu, M., Bai, S., Huang, T., and Bai, X. (November, January 27). Asymmetric Non-Local Neural Networks for Semantic Segmentation. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Korea."},{"key":"ref_3","unstructured":"Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., and Liu, W. (November, January 27). CCNet: Criss-Cross Attention for Semantic Segmentation. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Korea."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Zhu, Y., Sapra, K., Reda, F.A., Shih, K.J., Newsam, S., Tao, A., and Catanzaro, B. (2019, January 15\u201320). Improving Semantic Segmentation via Video Propagation and Label Relaxation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00906"},{"key":"ref_5","unstructured":"Tao, A., Sapra, K., and Catanzaro, B. (2020). Hierarchical multi-scale attention for semantic segmentation. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., and Sang, N. (2018, January 8\u201314). Bisenet: Bilateral segmentation network for real-time semantic segmentation. Proceedings of the European conference on computer vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01261-8_20"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Li, H., Xiong, P., Fan, H., and Sun, J. (2019, January 15\u201320). DFANet: Deep Feature Aggregation for Real-Time Semantic Segmentation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00975"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Or\u0161ic, M., Kre\u0161o, I., Bevandic, P., and \u0160egvic, S. (2019, January 15\u201320). Defense of Pre-Trained ImageNet Architectures for Real-Time Semantic Segmentation of Road-Driving Images. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.01289"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1109\/LRA.2020.3039744","article-title":"Real-Time Semantic Segmentation with Fast Attention","volume":"6","author":"Hu","year":"2021","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Lin, P., Sun, P., Cheng, G., Xie, S., Li, X., and Shi, J. (2020, January 13\u201319). Graph-Guided Architecture Search for Real-Time Semantic Segmentation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00426"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1175","DOI":"10.1007\/s11042-020-09425-0","article-title":"Human segmentation in surveillance video with deep learning","volume":"80","author":"Gruosso","year":"2020","journal-title":"Multimed. Tools Appl."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Cane, T., and Ferryman, J. (2018, January 27\u201330). Evaluating deep semantic segmentation networks for object detection in maritime surveillance. Proceedings of the 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Auckland, New Zealand.","DOI":"10.1109\/AVSS.2018.8639077"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1109\/TGRS.2016.2612821","article-title":"Convolutional Neural Networks for Large-Scale Remote-Sensing Image Classification","volume":"55","author":"Maggiori","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1155","DOI":"10.1109\/TGRS.2018.2864987","article-title":"Scene Classification with Recurrent Attention of VHR Remote Sensing Images","volume":"57","author":"Wang","year":"2019","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"760","DOI":"10.1109\/TNSRE.2019.2896269","article-title":"Deep Learning for Electromyographic Hand Gesture Signal Classification Using Transfer Learning","volume":"27","author":"Fall","year":"2019","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"911","DOI":"10.1109\/TGRS.2018.2862899","article-title":"Locality and Structure Regularized Low Rank Representation for Hyperspectral Image Classification","volume":"57","author":"Wang","year":"2019","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27\u201330). You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks","volume":"39","author":"Ren","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Lin, T., Doll\u00e1r, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21\u201326). Feature Pyramid Networks for Object Detection. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.106"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Yang, M., Yu, K., Zhang, C., Li, Z., and Yang, K. (2018, January 18\u201323). DenseASPP for Semantic Segmentation in Street Scenes. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00388"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Mehta, S., Rastegari, M., Caspi, A., Shapiro, L., and Hajishirzi, H. (2018, January 8\u201314). Espnet: Efficient spatial pyramid of dilated convolutions for semantic segmentation. Proceedings of the European conference on computer vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01249-6_34"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Mehta, S., Rastegari, M., Shapiro, L., and Hajishirzi, H. (2019, January 15\u201320). ESPNetv2: A Light-Weight, Power Efficient, and General Purpose Convolutional Neural Network. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00941"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (2016, January 27\u201330). The Cityscapes Dataset for Semantic Urban Scene Understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.350"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018, January 8\u201314). Encoder-decoder with atrous separable convolution for semantic image segmentation. Proceedings of the Proceedings of the European conference on computer vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01234-2_49"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Long, J., Shelhamer, E., and Darrell, T. (2015, January 7\u201312). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2017, January 21\u201326). Pyramid Scene Parsing Network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.660"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Yuan, Y., Chen, X., and Wang, J. (2020, January 23\u201328). Object-Contextual Representations for Semantic Segmentation. Proceedings of the European conference on computer vision (ECCV), Glasgow, UK.","DOI":"10.1007\/978-3-030-58539-6_11"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1109\/TITS.2017.2750080","article-title":"ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation","volume":"19","author":"Romera","year":"2018","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Zhao, H., Qi, X., Shen, X., Shi, J., and Jia, J. (2018, January 8\u201314). ICNet for real-time semantic segmentation on high-resolution images. Proceedings of the European conference on computer vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01219-9_25"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5\u20139). U-Net: Convolutional Networks for Biomedical Image Segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention (MICCAI), Munich, Germany.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Roy, A.G., Navab, N., and Wachinger, C. (2018). Concurrent spatial and channel squeeze & excitation in fully convolutional networks. Comput. Sci., 421\u2013429.","DOI":"10.1007\/978-3-030-00928-1_48"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"540","DOI":"10.1109\/TMI.2018.2867261","article-title":"Recalibrating Fully Convolutional Networks with Spatial and Channel \u201cSqueeze and Excitation\u201d Blocks","volume":"38","author":"Roy","year":"2019","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_34","unstructured":"Ioffe, S., and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv."},{"key":"ref_35","unstructured":"Glorot, X., Bordes, A., and Bengio, Y. (2011, January 11\u201313). Deep Sparse Rectifier Neural Networks. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Ft. Lauderdale, FL, USA."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., and Wei, Y. (2017, January 22\u201329). Deformable Convolutional Networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.89"},{"key":"ref_37","unstructured":"Chao, P., Kao, C., Ruan, Y., Huang, C., and Lin, Y. (November, January 27). HarDNet: A Low Memory Traffic Network. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Korea."},{"key":"ref_38","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27\u201330). Rethinking the Inception Architecture for Computer Vision. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.308"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Maaten, L.V.D., and Weinberger, K.Q. (2017, January 21\u201326). Densely Connected Convolutional Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.243"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Lin, G., Milan, A., Shen, C., and Reid, I. (2017, January 21\u201326). Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.549"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Wang, P., Chen, P., Yuan, Y., Liu, D., Huang, Z., Hou, X., and Cottrell, G. (2018, January 12\u201315). Understanding convolution for semantic segmentation. Proceedings of the IEEE winter conference on applications of computer vision (WACV), Lake Tahoe, NV, USA.","DOI":"10.1109\/WACV.2018.00163"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"Imagenet large scale visual recognition challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis."},{"key":"ref_44","first-page":"225","article-title":"Coarse-to-Fine Semantic Segmentation from Image-Level Labels","volume":"29","author":"Jing","year":"2020","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Hou, Q., Zhang, L., Cheng, M.M., and Feng, J. (2020, January 13\u201319). Strip Pooling: Rethinking Spatial Pooling for Scene Parsing. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00406"},{"key":"ref_46","unstructured":"Zhao, S., Wang, Y., Yang, Z., and Cai, D. (2019). Region mutual information loss for semantic segmentation. arXiv."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Shrivastava, A., Gupta, A., and Girshick, R. (2016, January 27\u201330). Training Region-Based Object Detectors with Online Hard Example Mining. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.89"},{"key":"ref_48","unstructured":"Zha, H., Liu, R., Zhou, D., Yang, X., Zhang, Q., and Wei, X. (2020, January 14\u201316). Efficient Attention Calibration Network for Real-Time Semantic Segmentation. Proceedings of the Asian Conference on Machine Learning, Bangkok, Thailand."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"55630","DOI":"10.1109\/ACCESS.2021.3071866","article-title":"Lightweight Asymmetric Dilation Network for Real-Time Semantic Segmentation","volume":"9","author":"Hu","year":"2021","journal-title":"IEEE Access"},{"key":"ref_50","unstructured":"Chen, P., Liu, S., Zhao, H., and Jia, J. (2020). GridMask Data Augmentation. arXiv."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1007\/s11263-014-0733-5","article-title":"The pascal visual object classes challenge: A retrospective","volume":"111","author":"Everingham","year":"2015","journal-title":"Int. J. Comput. Vis."},{"key":"ref_52","unstructured":"NVIDIA (2019, June 10). TensorRT. Available online: https:\/\/developer.nvidia.com\/tensorrt."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"2481","DOI":"10.1109\/TPAMI.2016.2644615","article-title":"SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation","volume":"39","author":"Badrinarayanan","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Fan, M., Lai, S., Huang, J., Wei, X., Chai, Z., Luo, J., and Wei, X. (2021, January 20\u201325). Rethinking BiSeNet For Real-time Semantic Segmentation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00959"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Nirkin, Y., Wolf, L., and Hassner, T. (2021, January 20\u201325). Hyperseg: Patch-wise hypernetwork for real-time semantic segmentation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00405"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/23\/8072\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:38:56Z","timestamp":1760168336000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/23\/8072"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,2]]},"references-count":55,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["s21238072"],"URL":"https:\/\/doi.org\/10.3390\/s21238072","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2021,12,2]]}}}