{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T16:21:24Z","timestamp":1781108484516,"version":"3.54.1"},"reference-count":62,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2023,6,6]],"date-time":"2023-06-06T00:00:00Z","timestamp":1686009600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Wuhan University\u2013Huawei Geoinformatics Innovation Laboratory"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>The detection and recognition of traffic signs is an essential component of intelligent vehicle perception systems, which use on-board cameras to sense traffic sign information. Unfortunately, issues such as long-tailed distribution, occlusion, and deformation greatly decrease the detector\u2019s performance. In this research, YOLOv5 is used as a single classification detector for traffic sign localization. Afterwards, we propose a hierarchical classification model (HCM) for the specific classification, which significantly reduces the degree of imbalance between classes without changing the sample size. To cope with the shortcomings of a single image, a training-free multi-frame information integration module (MIM) was constructed, which can extract the detection sequence of traffic signs based on the embedding generated by the HCM. The extracted temporal detection information is used for the redefinition of categories and confidence. At last, this research performed detection and recognition of the full class on two publicly available datasets, TT100K and ONCE. Experimental results show that the HCM-improved YOLOv5 has a mAP of 79.0 in full classes, which exceeds that of state-of-the-art methods, and achieves an inference speed of 22.7 FPS. In addition, MIM further improves model performance by integrating multi-frame information while only slightly increasing computational resource consumption.<\/jats:p>","DOI":"10.3390\/rs15122959","type":"journal-article","created":{"date-parts":[[2023,6,7]],"date-time":"2023-06-07T01:38:41Z","timestamp":1686101921000},"page":"2959","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Traffic Sign Detection and Recognition Using Multi-Frame Embedding of Video-Log Images"],"prefix":"10.3390","volume":"15","author":[{"given":"Jian","family":"Xu","sequence":"first","affiliation":[{"name":"School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5497-9897","authenticated-orcid":false,"given":"Yuchun","family":"Huang","sequence":"additional","affiliation":[{"name":"School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dakan","family":"Ying","sequence":"additional","affiliation":[{"name":"School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,6,6]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2229","DOI":"10.1049\/iet-ipr.2018.5424","article-title":"Efficient coarser-to-fine holistic traffic sign detection for occlusion handling","volume":"12","author":"Rehman","year":"2018","journal-title":"IET Image Process."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1016\/j.future.2018.11.027","article-title":"Smart data driven traffic sign detection method based on adaptive color threshold and shape symmetry","volume":"94","author":"Xu","year":"2019","journal-title":"Future Gener. Comput. Syst.-Int. J. Escience"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2022","DOI":"10.1109\/TITS.2015.2482461","article-title":"Towards Real-Time Traffic Sign Detection and Classification","volume":"17","author":"Yang","year":"2016","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Cao, J., Song, C., Peng, S., Xiao, F., and Song, S. (2019). Improved Traffic Sign Detection and Recognition Algorithm for Intelligent Vehicles. Sensors, 19.","DOI":"10.3390\/s19184021"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1149","DOI":"10.1515\/phys-2018-0135","article-title":"Fast recognition algorithm for static traffic sign information","volume":"16","author":"Guo","year":"2018","journal-title":"Open Phys."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2161","DOI":"10.3390\/s150102161","article-title":"Fast Traffic Sign Recognition with a Rotation Invariant Binary Pattern Based Feature","volume":"15","author":"Yin","year":"2015","journal-title":"Sensors"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"939","DOI":"10.1049\/iet-ipr.2019.0634","article-title":"Two-stage traffic sign detection and recognition based on SVM and convolutional neural networks","volume":"14","author":"Hechri","year":"2020","journal-title":"Iet Image Process."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"6721","DOI":"10.1007\/s00500-019-04307-6","article-title":"A robust system for road sign detection and classification using LeNet architecture based on convolutional neural network","volume":"24","author":"Bouti","year":"2020","journal-title":"Soft Comput."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2807","DOI":"10.1007\/s00521-017-2887-x","article-title":"Traffic sign recognition based on color, shape, and pictogram classification using support vector machines","volume":"30","author":"Madani","year":"2018","journal-title":"Neural Comput. Appl."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1016\/j.neucom.2014.11.026","article-title":"Traffic sign segmentation and classification using statistical learning methods","volume":"153","year":"2015","journal-title":"Neurocomputing"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1016\/j.neucom.2014.12.111","article-title":"A novel traffic sign detection method via color segmentation and robust shape matching","volume":"169","author":"Li","year":"2015","journal-title":"Neurocomputing"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"5005","DOI":"10.1007\/s00521-018-03994-w","article-title":"Speed limit sign detection and recognition system using SVM and MNIST datasets","volume":"31","author":"Saadna","year":"2019","journal-title":"Neural Comput. Appl."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1016\/j.eswa.2015.11.018","article-title":"On circular traffic sign detection and recognition","volume":"48","author":"Berkaya","year":"2016","journal-title":"Expert Syst. Appl."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1016\/j.isprsjprs.2022.08.011","article-title":"SignHRNet: Street-level traffic signs recognition with an attentive semi-anchoring guided high-resolution network","volume":"192","author":"Yu","year":"2022","journal-title":"Isprs J. Photogramm. Remote Sens."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"56416","DOI":"10.1109\/ACCESS.2021.3072211","article-title":"Small-Object Detection Based on YOLO and Dense Block via Image Super-Resolution","volume":"9","author":"Wang","year":"2021","journal-title":"IEEE Access"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"7567","DOI":"10.1007\/s11042-022-13251-x","article-title":"Attention-YOLOV4: A real-time and high-accurate traffic sign detection algorithm","volume":"82","author":"Li","year":"2023","journal-title":"Multimed. Tools Appl."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"238","DOI":"10.1007\/s10489-022-03459-7","article-title":"MTSDet: Multi-scale traffic sign detection with attention and path aggregation","volume":"53","author":"Wei","year":"2023","journal-title":"Appl. Intell."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Wang, X., Guo, J., Yi, J., Song, Y., Xu, J., Yan, W., and Fu, X. (2022). Real-Time and Efficient Multi-Scale Traffic Sign Detection Method for Driverless Cars. Sensors, 22.","DOI":"10.3390\/s22186930"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Hu, J., Wang, Z., Chang, M., Xie, L., Xu, W., and Chen, N. (2022). PSG-Yolov5: A Paradigm for Traffic Sign Detection and Recognition Algorithm Based on Deep Learning. Symmetry, 14.","DOI":"10.3390\/sym14112262"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Triki, N., Karray, M., and Ksantini, M. (2023). A Real-Time Traffic Sign Recognition Method Using a New Attention-Based Deep Convolutional Neural Network for Smart Vehicles. Appl. Sci., 13.","DOI":"10.3390\/app13084793"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Gao, X., Chen, L., Wang, K., Xiong, X., Wang, H., and Li, Y. (2022). Improved Traffic Sign Detection Algorithm Based on Faster R-CNN. Appl. Sci., 12.","DOI":"10.3390\/app12188948"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1712","DOI":"10.1049\/iet-its.2020.0217","article-title":"Scale-aware limited deformable convolutional neural networks for traffic sign detection and classification","volume":"14","author":"Liu","year":"2020","journal-title":"IET Intell. Transp. Syst."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2137","DOI":"10.1007\/s11227-022-04670-6","article-title":"Traffic sign detection based on multi-scale feature extraction and cascade feature fusion","volume":"79","author":"Zhang","year":"2023","journal-title":"J. Supercomput."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"7853","DOI":"10.1007\/s00521-022-08077-5","article-title":"Improved YOLOv5 network for real-time multi-scale traffic sign detection","volume":"35","author":"Wang","year":"2022","journal-title":"Neural Comput. Appl."},{"key":"ref_25","first-page":"4285436","article-title":"Traffic Sign Detection Based on SSD Combined with Receptive Field Module and Path Aggregation Network","volume":"2022","author":"Wu","year":"2022","journal-title":"Comput. Intell. Neurosci."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"116783","DOI":"10.1016\/j.image.2022.116783","article-title":"Traffic sign detection algorithm based on improved YOLOv4-Tiny","volume":"107","author":"Yao","year":"2022","journal-title":"Signal Process.-Image Commun."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/j.neucom.2021.03.049","article-title":"TSingNet: Scale-aware and context-rich feature learning for traffic sign detection and recognition in the wild","volume":"447","author":"Liu","year":"2021","journal-title":"Neurocomputing"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"6533","DOI":"10.1007\/s00521-019-04086-z","article-title":"Traffic sign detection and recognition based on pyramidal convolutional networks","volume":"32","author":"Liang","year":"2020","journal-title":"Neural Comput. Appl."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"3423","DOI":"10.1109\/TIP.2019.2896952","article-title":"VSSA-NET: Vertical Spatial Sequence Attention Network for Traffic Sign Detection","volume":"28","author":"Yuan","year":"2019","journal-title":"IEEE Trans. Image Process."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"178798","DOI":"10.1109\/ACCESS.2019.2959015","article-title":"FAMN: Feature Aggregation Multipath Network for Small Traffic Sign Detection","volume":"7","author":"Ou","year":"2019","journal-title":"IEEE Access"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Suto, J. (2022). An Improved Image Enhancement Method for Traffic Sign Detection. Electronics, 11.","DOI":"10.3390\/electronics11060871"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"20545","DOI":"10.1007\/s11042-020-08848-z","article-title":"Performance enhancement techniques for traffic sign recognition using a deep neural network","volume":"79","author":"Khan","year":"2020","journal-title":"Multimed. Tools Appl."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Khan, J.A., Yeo, D., and Shin, H. (2018). New Dark Area Sensitive Tone Mapping for Deep Learning Based Traffic Sign Recognition. Sensors, 18.","DOI":"10.3390\/s18113776"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1121","DOI":"10.1109\/TITS.2020.3020556","article-title":"Traffic Sign Recognition with Lightweight Two-Stage Model in Complex Scenes","volume":"23","author":"Wang","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"171170","DOI":"10.1109\/ACCESS.2020.3024583","article-title":"Focus First: Coarse-to-Fine Traffic Sign Detection with Stepwise Learning","volume":"8","author":"Liu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1007\/s41095-019-0152-1","article-title":"A three-stage real-time detector for traffic signs in large panoramas","volume":"5","author":"Song","year":"2019","journal-title":"Comput. Vis. Media"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"15794","DOI":"10.1109\/TITS.2022.3145467","article-title":"Traffic Sign Recognition Based on Semantic Scene Understanding and Structural Traffic Sign Location","volume":"23","author":"Min","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"4466","DOI":"10.1109\/TITS.2018.2886283","article-title":"Traffic Sign Detection Using a Multi-Scale Recurrent Attention Network","volume":"20","author":"Tian","year":"2019","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"272","DOI":"10.1016\/j.neucom.2022.06.055","article-title":"Encrypted internet traffic classification using a supervised spiking neural network","volume":"503","author":"Rasteh","year":"2022","journal-title":"Neurocomputing"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"106232","DOI":"10.1016\/j.engappai.2023.106232","article-title":"A storage-efficient SNN-CNN hybrid network with RRAM-implemented weights for traffic signs recognition","volume":"123","author":"Zhang","year":"2023","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"9980","DOI":"10.1109\/TVT.2022.3178808","article-title":"Efficient Federated Learning with Spike Neural Networks for Traffic Sign Recognition","volume":"71","author":"Xie","year":"2022","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1186\/s40537-019-0192-5","article-title":"Survey on deep learning with class imbalance","volume":"6","author":"Johnson","year":"2019","journal-title":"J. Big Data"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"16632","DOI":"10.1109\/TITS.2022.3170354","article-title":"Traffic Sign Detection and Recognition in Multiimages Using a Fusion Model with YOLO and VGG Network","volume":"23","author":"Yu","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Atif, M., Zoppi, T., Gharib, M., and Bondavalli, A. (2022). Towards Enhancing Traffic Sign Recognition through Sliding Windows. Sensors, 22.","DOI":"10.3390\/s22072683"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"8226","DOI":"10.1007\/s10489-021-02838-w","article-title":"Detection-by-tracking of traffic signs in videos","volume":"52","author":"Zhang","year":"2022","journal-title":"Appl. Intell."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Song, S., Li, Y., Huang, Q., and Li, G. (2021). A New Real-Time Detection and Tracking Method in Videos for Small Target Traffic Signs. Appl. Sci., 11.","DOI":"10.3390\/app11073061"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Girshick, R., Donahue, J., Darrell, T., Malik, J., and IEEE (2014, January 23\u201328). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the 27th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.81"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"3212","DOI":"10.1109\/TNNLS.2018.2876865","article-title":"Object detection with deep learning: A review","volume":"30","author":"Zhao","year":"2019","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P., and Zitnick, C.L. (2014, January 6\u201312). Microsoft COCO: Common Objects in Context. Proceedings of the 13th European Conference on Computer Vision (ECCV), Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"103910","DOI":"10.1016\/j.imavis.2020.103910","article-title":"Recent advances in small object detection based on deep learning: A review","volume":"97","author":"Tong","year":"2020","journal-title":"Image Vis. Comput."},{"key":"ref_51","unstructured":"Ren, S., He, K., Girshick, R., and Sun, J. (2015, January 7\u201312). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Proceedings of the 29th Annual Conference on Neural Information Processing Systems (NIPS), Montreal, QC, Canada."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Dollar, P., Girshick, R., and IEEE (2017, January 22\u201329). Mask R-CNN. Proceedings of the 16th IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Houben, S., Stallkamp, J., Salmen, J., Schlipsing, M., Igel, C., and IEEE (2013, January 4\u20139). Detection of Traffic Signs in Real-World Images: The German Traffic Sign Detection Benchmark. Proceedings of the International Joint Conference on Neural Networks (IJCNN), Dallas, TX, USA.","DOI":"10.1109\/IJCNN.2013.6706807"},{"key":"ref_54","first-page":"23","article-title":"CCTSDB 2021: A More Comprehensive Traffic Sign Detection Benchmark","volume":"12","author":"Zhang","year":"2022","journal-title":"Hum.-Cent. Comput. Inf. Sci."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Zhu, Z., Liang, D., Zhang, S., Huang, X., Li, B., Hu, S., and IEEE (2016, January 27\u201330). Traffic-Sign Detection and Classification in the Wild. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR.2016.232"},{"key":"ref_56","unstructured":"Mao, J., Niu, M., Jiang, C., Liang, H., Chen, J., Liang, X., Li, Y., Ye, C., Zhang, W., and Li, Z. (2021). One million scenes for autonomous driving: ONCE dataset. arXiv."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Chu, J., Zhang, C., Yan, M., Zhang, H., and Ge, T. (2023). TRD-YOLO: A Real-Time, High-Performance Small Traffic Sign Detection Algorithm. Sensors, 23.","DOI":"10.3390\/s23083871"},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"103774","DOI":"10.1016\/j.jvcir.2023.103774","article-title":"Improved traffic sign recognition algorithm based on YOLOv4-tiny","volume":"91","author":"Sharma","year":"2023","journal-title":"J. Vis. Commun. Image Represent."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"643","DOI":"10.1007\/s00521-022-07782-5","article-title":"CDFF: A fast and highly accurate method for recognizing traffic signs","volume":"35","author":"Wang","year":"2023","journal-title":"Neural Comput. Appl."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"19395","DOI":"10.1109\/ACCESS.2022.3204818","article-title":"Faster Light Detection Algorithm of Traffic Signs Based on YOLOv5s-A2","volume":"11","author":"Yuan","year":"2023","journal-title":"IEEE Access"},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"122774","DOI":"10.1109\/ACCESS.2021.3109606","article-title":"A Traffic-Sign Detection Algorithm Based on Improved Sparse R-CNN","volume":"9","author":"Cao","year":"2021","journal-title":"IEEE Access"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"24105","DOI":"10.1109\/TITS.2022.3200737","article-title":"Long-Tailed Traffic Sign Detection Using Attentive Fusion and Hierarchical Group Softmax","volume":"23","author":"Gao","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/12\/2959\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:49:27Z","timestamp":1760125767000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/12\/2959"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,6]]},"references-count":62,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2023,6]]}},"alternative-id":["rs15122959"],"URL":"https:\/\/doi.org\/10.3390\/rs15122959","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,6]]}}}