{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T02:26:01Z","timestamp":1760235961862,"version":"build-2065373602"},"reference-count":57,"publisher":"MDPI AG","issue":"20","license":[{"start":{"date-parts":[[2021,10,15]],"date-time":"2021-10-15T00:00:00Z","timestamp":1634256000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Video analytics and computer vision applications face challenges when using video sequences with low visibility. The visibility of a video sequence is degraded when the sequence is affected by atmospheric interference like rain. Many approaches have been proposed to remove rain streaks from video sequences. Some approaches are based on physical features, and some are based on data-driven (i.e., deep-learning) models. Although the physical features-based approaches have better rain interpretability, the challenges are extracting the appropriate features and fusing them for meaningful rain removal, as the rain streaks and moving objects have dynamic physical characteristics and are difficult to distinguish. Additionally, the outcome of the data-driven models mostly depends on variations relating to the training dataset. It is difficult to include datasets with all possible variations in model training. This paper addresses both issues and proposes a novel hybrid technique where we extract novel physical features and data-driven features and then combine them to create an effective rain-streak removal strategy. The performance of the proposed algorithm has been tested in comparison to several relevant and contemporary methods using benchmark datasets. The experimental result shows that the proposed method outperforms the other methods in terms of subjective, objective, and object detection comparisons for both synthetic and real rain scenarios by removing rain streaks and retaining the moving objects more effectively.<\/jats:p>","DOI":"10.3390\/s21206856","type":"journal-article","created":{"date-parts":[[2021,10,17]],"date-time":"2021-10-17T23:25:15Z","timestamp":1634513115000},"page":"6856","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Video Rain-Streaks Removal by Combining Data-Driven and Feature-Based Models"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1760-1577","authenticated-orcid":false,"given":"Muhammad Rafiqul","family":"Islam","sequence":"first","affiliation":[{"name":"School of Computing, Mathematics and Engineering, Charles Sturt University, Panorama Ave, Bathurst, NSW 2795, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6870-5056","authenticated-orcid":false,"given":"Manoranjan","family":"Paul","sequence":"additional","affiliation":[{"name":"School of Computing, Mathematics and Engineering, Charles Sturt University, Panorama Ave, Bathurst, NSW 2795, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,10,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"40892","DOI":"10.1109\/ACCESS.2020.2976761","article-title":"GAN-based rain noise removal from single-image considering rain composite models","volume":"8","author":"Matsui","year":"2020","journal-title":"IEEE Access"},{"key":"ref_2","unstructured":"Li, Y., Tan, R.T., Guo, X., Lu, J., and Brown, M.S. (July, January 26). Rain streak removal using layer priors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_3","first-page":"1","article-title":"Learning aggregated transmission propagation networks for haze removal and beyond","volume":"99","author":"Liu","year":"2018","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Li, R., Cheong, L.-F., and Tan, R.T. (2019, January 16\u201320). Heavy rain image restoration: Integrating physics model and conditional adversarial learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00173"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"4059","DOI":"10.1109\/TPAMI.2020.2995190","article-title":"Single image deraining: From model-based to data-driven and beyond","volume":"43","author":"Yang","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Jiang, K., Wang, Z., Yi, P., Chen, C., Huang, B., Luo, Y., Ma, J., and Jiang, J. (2020, January 14\u201319). Multi-scale progressive fusion network for single image deraining. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00837"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"564","DOI":"10.1109\/TPAMI.2003.1195991","article-title":"Kernel-based object tracking","volume":"25","author":"Comaniciu","year":"2003","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1254","DOI":"10.1109\/34.730558","article-title":"A model of saliency-based visual attention for rapid scene analysis","volume":"20","author":"Itti","year":"1998","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Farenzena, M., Bazzani, L., Perina, A., Murino, V., and Cristani, M. (2010, January 13\u201318). Person re-identification by symmetry-driven accumulation of local features. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA.","DOI":"10.1109\/CVPR.2010.5539926"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Islam, M.R., and Paul, M. (2019, January 2\u20134). Rain streak removal from video sequence using spatiotemporal appearance. Proceedings of the 2019 Digital Image Computing: Techniques and Applications (DICTA), Perth, WA, Australia.","DOI":"10.1109\/DICTA47822.2019.8946080"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/s11263-006-0028-6","article-title":"Vision and rain","volume":"75","author":"Garg","year":"2007","journal-title":"Int. J. Comput. Vis."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1661","DOI":"10.1109\/TMM.2016.2568748","article-title":"Democratic diffusion aggregation for image retrieval","volume":"18","author":"Gao","year":"2016","journal-title":"IEEE Trans. Multimed."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1109\/TMM.2015.2510326","article-title":"Guided image contrast enhancement based on retrieved images in cloud","volume":"18","author":"Wang","year":"2015","journal-title":"IEEE Trans. Multimed."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Zhang, X., Li, H., Qi, Y., Leow, W.K., and Ng, T.K. (2006, January 9\u201312). Rain removal in video by combining temporal and chromatic properties. Proceedings of the 2006 IEEE International Conference on Multimedia and Expo, Toronto, ON, Canada.","DOI":"10.1109\/ICME.2006.262572"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Luo, Y., Xu, Y., and Ji, H. (2015, January 7\u201313). Removing rain from a single image via discriminative sparse coding. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.388"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Gu, S., Meng, D., Zuo, W., and Zhang, L. (2017, January 22\u201329). Joint convolutional analysis and synthesis sparse representation for single image layer separation. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.189"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2944","DOI":"10.1109\/TIP.2017.2691802","article-title":"Clearing the skies: A deep network architecture for single-image rain removal","volume":"26","author":"Fu","year":"2017","journal-title":"IEEE Trans. Image Process."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Zhang, H., and Patel, V.M. (2018, January 18\u201322). Density-aware single image de-raining using a multi-stream dense network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00079"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., and Paisley, J. (2017, January 21\u201326). Removing rain from single images via a deep detail network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.186"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Li, X., Wu, J., Lin, Z., Liu, H., and Zha, H. (2018, January 8\u201314). Recurrent squeeze-and-excitation context aggregation net for single image deraining. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01234-2_16"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Wei, W., Yi, L., Xie, Q., Zhao, Q., Meng, D., and Xu, Z. (2017, January 22\u201329). Should we encode rain streaks in video as deterministic or stochastic?. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.275"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Liu, J., Yang, W., Yang, S., and Guo, Z. (2018, January 18\u201322). Erase or fill? deep joint recurrent rain removal and reconstruction in videos. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00341"},{"key":"ref_23","first-page":"664","article-title":"Utilizing two-phase processing with fbls for single image deraining","volume":"26","author":"Lin","year":"2020","journal-title":"IEEE Trans. Multimed."},{"key":"ref_24","unstructured":"Garg, K., and Nayar, S.K. (July2004, January 27). Detection and removal of rain from videos. Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, DC, USA."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1421","DOI":"10.1007\/s11760-012-0373-6","article-title":"Removal of rain from videos: A review","volume":"8","author":"Tripathi","year":"2014","journal-title":"Signal Image Video Process."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1097","DOI":"10.1109\/TIP.2013.2290595","article-title":"A rain pixel recovery algorithm for videos with highly dynamic scenes","volume":"23","author":"Chen","year":"2014","journal-title":"IEEE Trans. Image Process."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"2658","DOI":"10.1109\/TIP.2015.2428933","article-title":"Video deraining and desnowing using temporal correlation and low-rank matrix completion","volume":"24","author":"Kim","year":"2015","journal-title":"IEEE Trans. Image Process."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1007\/s11263-014-0759-8","article-title":"Utilizing local phase information to remove rain from video","volume":"112","author":"Santhaseelan","year":"2015","journal-title":"Int. J. Comput. Vis."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1721","DOI":"10.1109\/TPAMI.2015.2491937","article-title":"Adherent raindrop modeling, detectionand removal in video","volume":"38","author":"You","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Jiang, T.X., Huang, T.Z., Zhao, X.L., Deng, L.J., and Wang, Y. (2017, January 21\u201326). A novel tensor-based video rain streaks removal approach via utilizing discriminatively intrinsic priors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.301"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Ren, W., Tian, J., Han, Z., Chan, A., and Tang, Y. (2017, January 21\u201326). Video desnowing and deraining based on matrix decomposition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.303"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Li, M., Xie, Q., Zhao, Q., Wei, W., Gu, S., Tao, J., and Meng, D. (2018, January 18\u201322). Video rain streak removal by multiscale convolutional sparse coding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00695"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Wei, W., Meng, D., Zhao, Q., Xu, Z., and Wu, Y. (2019, January 16\u201320). Semi-supervised transfer learning for image rain removal. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00400"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Yang, C., Liu, R., Ma, L., Fan, X., Li, H., and Zhang, M. (2018, January 13\u201316). Unrolled optimization with deep priors for intrinsic image decomposition. Proceedings of the 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), Xi\u2019an, China.","DOI":"10.1109\/BigMM.2018.8499478"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Chen, J., Tan, C.H., Hou, J., Chau, L.P., and Li, H. (2018, January 18\u201322). Robust video content alignment and compensation for rain removal in a CNN framework. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00658"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"699","DOI":"10.1109\/TIP.2018.2869722","article-title":"D3r-net: Dynamic routing residue recurrent network for video rain removal","volume":"28","author":"Liu","year":"2019","journal-title":"IEEE Trans. Image Process."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Yang, W., Liu, J., and Feng, J. (2019, January 16\u201320). Frame-consistent recurrent video deraining with dual-level flow. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00176"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Xue, X., Ding, Y., Mu, P., Ma, L., Liu, R., and Fan, X. (2020, January 4\u20138). Sequential deep unrolling with flow priors for robust video deraining. Proceedings of the ICASSP 2020\u20132020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.","DOI":"10.1109\/ICASSP40776.2020.9052969"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1377","DOI":"10.1109\/TPAMI.2019.2895793","article-title":"Joint rain detection and removal from a single image with contextualized deep networks","volume":"42","author":"Yang","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Dollar, P., and Girshick, R. (2017, January 22\u201329). Mask r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Girshick, R. (2015, January 7\u201313). Fast r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.169"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Haque, M., Murshed, M., and Paul, M. (2008, January 8\u201311). Improved Gaussian mixtures for robust object detection by adaptive multi-background generation. Proceedings of the 2008 19th International Conference on Pattern Recognition, Tampa, FL, USA.","DOI":"10.1109\/ICPR.2008.4761496"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1016\/j.cosrev.2014.04.001","article-title":"Traditional and recent approaches in background modeling for foreground detection: An overview","volume":"11","author":"Bouwmans","year":"2014","journal-title":"Comput. Sci. Rev."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"827","DOI":"10.1109\/TPAMI.2005.102","article-title":"Effective Gaussian mixture learning for video background subtraction","volume":"27","author":"Lee","year":"2005","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1109\/TCE.2021.3077241","article-title":"A full-process optimization-based background subtraction for moving object detection on general-purpose embedded devices","volume":"67","author":"Li","year":"2021","journal-title":"IEEE Trans. Consum. Electron."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1190","DOI":"10.1109\/TIP.2017.2772858","article-title":"Virtual view synthesis for free viewpoint video and multiview video compression using Gaussian mixture modelling","volume":"27","author":"Rahaman","year":"2017","journal-title":"IEEE Trans. Image Process."},{"key":"ref_47","first-page":"1","article-title":"Video coding with dynamic background","volume":"11","author":"Manoranjan","year":"2013","journal-title":"EURASIP J. Adv. Signal Process."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"1161","DOI":"10.1109\/TCE.2007.4341600","article-title":"Efficient spatio-temporal segmentation for extracting moving objects in video sequences","volume":"53","author":"Li","year":"2007","journal-title":"IEEE Trans. Consum. Electron."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"4677","DOI":"10.1109\/TIP.2016.2593343","article-title":"Robust low-rank matrix factorization under general mixture noise distributions","volume":"25","author":"Cao","year":"2016","journal-title":"IEEE Trans. Image Process."},{"key":"ref_50","unstructured":"Zhao, Q., Meng, D., Xu, Z., Zuo, W., and Zhang, L. (2014, January 21\u201326). Robust principal component analysis with complex noise. Proceedings of the International Conference on Machine Learning, Beijing, China."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Meng, D., and de la Torre, F. (2013, January 1\u20138). Robust matrix factorization with unknown noise. Proceedings of the IEEE International Conference on Computer Vision, Sydney, NSW, Australia.","DOI":"10.1109\/ICCV.2013.169"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"825","DOI":"10.1109\/TNNLS.2014.2387376","article-title":"L1 -norm low-rank matrix factorization by variational Bayesian method","volume":"26","author":"Zhao","year":"2015","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"1054","DOI":"10.1109\/TCYB.2017.2677944","article-title":"Denoising hyperspectral image with non-IID noise structure","volume":"48","author":"Chen","year":"2017","journal-title":"IEEE Trans. Cybern."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Haque, M., Murshed, M., and Paul, M. (2008, January 8\u201310). A hybrid object detection technique from dynamic background using Gaussian mixture models. Proceedings of the 2008 IEEE 10th Workshop on Multimedia Signal Processing, Cairns, QLD, Australia.","DOI":"10.1109\/MMSP.2008.4665205"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Haque, M., Murshed, M., and Paul, M. (2008, January 1\u20133). On stable dynamic background generation technique using Gaussian mixture models for robust object detection. Proceedings of the 2008 IEEE Fifth International Conference on Advanced Video and Signal Based Surveillance, Santa Fe, NM, USA.","DOI":"10.1109\/AVSS.2008.12"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Goyette, N., Jodoin, P.M., Porikli, F., Konrad, J., and Ishwar, P. (2012, January 16\u201321). Changedetection. net: A new change detection benchmark dataset. Proceedings of the 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Providence, RI, USA.","DOI":"10.1109\/CVPRW.2012.6238919"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"3943","DOI":"10.1109\/TCSVT.2019.2920407","article-title":"Image de-raining using a conditional generative adversarial network","volume":"30","author":"Zhang","year":"2019","journal-title":"IEEE Trans. Circuits Syst. Video Technol."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/20\/6856\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:15:22Z","timestamp":1760166922000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/20\/6856"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,15]]},"references-count":57,"journal-issue":{"issue":"20","published-online":{"date-parts":[[2021,10]]}},"alternative-id":["s21206856"],"URL":"https:\/\/doi.org\/10.3390\/s21206856","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2021,10,15]]}}}