Abstract
Rain can cause performance degradation of outdoor computer vision tasks. Thus, the exploration of rain removal from videos or a single image has drawn considerable attention in the field of image processing. Recently, various deraining methodologies have been proposed. However, no comprehensive survey work has yet been conducted to summarize existing deraining algorithms and quantitatively compare their generalization ability, and especially, no off-the-shelf toolkit exists for accumulating and categorizing recent representative methods for easy performance reproduction and deraining capability evaluation. In this regard, herein, we present a comprehensive overview of existing video and single image deraining methods as well as reproduce and evaluate current state-of-the-art deraining methods. In particular, these approaches are mainly classified into model- and deep-learning-based methods, and more elaborate branches of each method are presented. Inherent abilities, especially generalization performance, of the state-of-the-art methods have been both quantitatively and visually analyzed through thorough experiments conducted on synthetic and real benchmark datasets. Moreover, to facilitate the reproduction of existing deraining methods for general users, we present a comprehensive repository with detailed classification, including direct links to 85 deraining papers, 24 relevant project pages, source codes of 12 and 25 algorithms for video and single image deraining, respectively, 5 and 10 real and synthesized datasets, respectively, and 7 frequently used image quality evaluation metrics, along with the corresponding computation codes. Research limitations worthy of further exploration have also been discussed for future research along this direction.
Similar content being viewed by others
References
Yang W, Tan R T, Feng J, et al. Deep joint rain detection and removal from a single image. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2017. 1357–1366
Yang W, Tan R T, Feng J, et al. Joint rain detection and removal from a single image with contextualized deep networks. IEEE Trans Pattern Anal Mach Intell, 2020, 42: 1377–1393
Li S, Araujo I B, Ren W, et al. Single image deraining: a comprehensive benchmark analysis. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019. 3838–3847
Bahnsen C H, Moeslund T B. Rain removal in traffic surveillance: does it matter?. IEEE Trans Intell Transp Syst, 2019, 20: 2802–2819
Comaniciu D, Ramesh V, Meer P. Kernel-based object tracking. IEEE Trans Pattern Anal Mach Intell, 2003, 25: 564–577
Itti L, Koch C, Niebur E. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell, 1998, 11: 1254–1259
Farenzena M, Bazzani L, Perina A, et al. Person re-identification by symmetry-driven accumulation of local features. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010. 2360–2367
Shehata M S, Cai J, Badawy W M, et al. Video-based automatic incident detection for smart roads: the outdoor environmental challenges regarding false alarms. IEEE Trans Intell Transp Syst, 2008, 9: 349–360
Pang Y W, Xie J, Li X L. Visual haze removal by a unified generative adversarial network. IEEE Trans Circ Syst Video Technol, 2019, 29: 3211–3221
Sun H Q, Pang Y W. GlanceNets-efficient convolutional neural networks with adaptive hard example mining. Sci China Inf Sci, 2018, 61: 109101
Zhang Z J, Pang Y W. CGNet: cross-guidance network for semantic segmentation. Sci China Inf Sci, 2020, 63: 120104
Ma S, Pang Y W, Pan J, et al. Preserving details in semantics-aware context for scene parsing. Sci China Inf Sci, 2020, 63: 120106
Garg K, Nayar S K. Vision and rain. Int J Comput Vis, 2007, 75: 3–27
Yu W J, Huang Z, Zhang W, et al. Gradual network for single image de-raining. In: Proceedings of the 27th ACM International Conference on Multimedia, 2019. 1795–1804
Zhang X P, Li H, Qi Y Y, et al. Rain removal in video by combining temporal and chromatic properties. In: Proceedings of IEEE International Conference on Multimedia and Expo, 2006. 461–464
Luo Y, Xu Y, Ji H. Removing rain from a single image via discriminative sparse coding. In: Proceedings of IEEE International Conference on Computer Vision, 2015. 3397–3405
Li Y, Tan R T, Guo X, et al. Rain streak removal using layer priors. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016. 2736–2744
Gu S, Meng D, Zuo W, et al. Joint convolutional analysis and synthesis sparse representation for single image layer separation. In: Proceedings of IEEE International Conference on Computer Vision, 2017. 1708–1716
Yasarla R, Patel V M. Confidence measure guided single image de-raining. IEEE Trans Image Process, 2020, 29: 4544–4555
Wang X F, Chen J, Jiang K, et al. Single image de-raining via clique recursive feedback mechanism. Neurocomputing, 2020, 417: 142–154
Fu X Y, Huang J B, Ding X H, et al. Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans Image Process, 2017, 26: 2944–2956
Zhang H, Patel V M. Density-aware single image de-raining using a multi-stream dense network. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2018. 695–704
Fu X Y, Huang J B, Zeng D L, et al. Removing rain from single images via a deep detail network. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2017. 3855–3863
Li X, Wu J L, Lin Z C, et al. Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Proceedings of European Conference on Computer Vision (ECCV), 2018. 254–269
Wei W, Yi L X, Xie Q, et al. Should we encode rain streaks in video as deterministic or stochastic? In: Proceedings of IEEE International Conference on Computer Vision, 2017. 2516–2525
Li M H, Xie Q, Zhao Q, et al. Video rain streak removal by multiscale convolutional sparse coding. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2018. 6644–6653
Liu J Y, Yang W H, Yang S, et al. Erase or fill? Deep joint recurrent rain removal and reconstruction in videos. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2018. 3233–3242
Liu J Y, Yang W H, Yang S, et al. D3R-Net: dynamic routing residue recurrent network for video rain removal. IEEE Trans Image Process, 2019, 28: 699–712
Jiang K, Wang Z Y, Yi P, et al. Decomposition makes better rain removal: an improved attention-guided deraining network. IEEE Trans Circ Syst Video Technol, 2020. doi: https://doi.org/10.1109/TCSVT.2020.3044887
Pang B, Zhai D M, Jiang J J, et al. Single image deraining via scale-space invariant attention neural network. In: Proceedings of the 28th ACM International Conference on Multimedia, 2020. 375–383
Shao M W, Li L, Wang H, et al. Selective generative adversarial network for raindrop removal from a single image. Neurocomputing, 2021, 426: 265–273
Ren D W, Zuo W M, Hu Q H, et al. Progressive image deraining networks: a better and simpler baseline. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019. 3937–3946
Wang T Y, Yang X, Xu K, et al. Spatial attentive single-image deraining with a high quality real rain dataset. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019. 12270–12279
Pan J S, Liu S F, Sun D Q, et al. Learning dual convolutional neural networks for low-level vision. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2018. 3070–3079
Wei W, Meng D Y, Zhao Q, et al. Semi-supervised transfer learning for image rain removal. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019. 3877–3886
Yang C, Liu R S, Ma L, et al. Unrolled optimization with deep priors for intrinsic image decomposition. In: Proceedings of the 4th International Conference on Multimedia Big Data (BigMM), 2018. 1–7
Yang W, Tan R T, Wang S, et al. Single image deraining: from model-based to data-driven and beyond. IEEE Trans Pattern Anal Mach Intell, 2020. doi: https://doi.org/10.1109/TPAMI.2020.2995190
Beard K V, Chuang C. A new model for the equilibrium shape of raindrops. J Atmos Sci, 1987, 44: 1509–1524
Tripathi A K, Mukhopadhyay S. Removal of rain from videos: a review. Signal Image Video Process, 2014, 8: 1421–1430
Tripathi A K, Mukhopadhyay S. A probabilistic approach for detection and removal of rain from videos. IETE J Res, 2011, 57: 82–91
Garg K, Nayar S K. When does a camera see rain? In: Proceedings of the 10th IEEE International Conference on Computer Vision (ICCV’05), 2005. 1067–1074
Foote G B, Toit P S D. Terminal velocity of raindrops aloft. J Appl Meteor, 1969, 8: 249–253
Garg K, Nayar S K. Photometric Model of A Rain Drop. CMU Technical Report, 2003
Liu P, Xu J, Liu J F, et al. Pixel based temporal analysis using chromatic property for removing rain from videos. Comput Inform Sci, 2009, 2: 53–60
Garg K, Nayar S K. Detection and removal of rain from videos. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004
Tripathi A K, Mukhopadhyay S. Video post processing: low-latency spatiotemporal approach for detection and removal of rain. IET Image Process, 2012, 6: 181–196
Park W J, Lee K H. Rain removal using kalman filter in video. In: Proceedings of International Conference on Smart Manufacturing Application, 2008. 494–497
Brewer N, Liu N. Using the shape characteristics of rain to identify and remove rain from video. In: Proceedings of Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR), 2008. 451–458
Zhao X D, Liu P, Liu J F, et al. The application of histogram on rain detection in video. In: Proceedings of the 11th Joint International Conference on Information Sciences, 2008
Bossu J, Hautiére N, Tarel J P. Rain or snow detection in image sequences through use of a histogram of orientation of streaks. Int J Comput Vis, 2011, 93: 348–367
Barnum P, Kanade T, Narasimhan S. Spatio-temporal frequency analysis for removing rain and snow from videos. In: Proceedings of International Workshop on Photometric Analysis for Computer Vision, 2007
Barnum P C, Narasimhan S, Kanade T. Analysis of rain and snow in frequency space. Int J Comput Vis, 2010, 86: 256–274
Chen Y L, Hsu C T. A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: Proceedings of IEEE International Conference on Computer Vision, 2013. 1968–1975
Chen J, Chau L P. A rain pixel recovery algorithm for videos with highly dynamic scenes. IEEE Trans Image Process, 2014, 23: 1097–1104
Ren W H, Tian J D, Han Z, et al. Video desnowing and deraining based on matrix decomposition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2017. 4210–4219
Kim J H, Sim J Y, Kim C S. Video deraining and desnowing using temporal correlation and low-rank matrix completion. IEEE Trans Image Process, 2015, 24: 2658–2670
Jiang T X, Huang T Z, Zhao X L, et al. A novel tensor-based video rain streaks removal approach via utilizing discriminatively intrinsic priors. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2017. 4057–4066
Jiang T X, Huang T Z, Zhao X L, et al. FastDeRain: a novel video rain streak removal method using directional gradient priors. IEEE Trans Image Process, 2019, 28: 2089–2102
Chen J, Tan C H, Hou J, et al. Robust video content alignment and compensation for rain removal in a CNN framework. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2018. 6286–6295
Yang W H, Liu J Y, Feng J S. Frame-consistent recurrent video deraining with dual-level flow. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020
Yang W, Tan R T, Wang S, et al. Self-learning video rain streak removal: when cyclic consistency meets temporal correspondence. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 1720–1729
Xu J, Zhao W, Liu P, et al. Removing rain and snow in a single image using guided filter. In: Proceedings of IEEE International Conference on Computer Science and Automation Engineering (CSAE), 2012. 304–307
He K M, Sun J, Tang X O. Guided image filtering. IEEE Trans Pattern Anal Mach Intell, 2013, 35: 1397–1409
Xu J, Zhao W, Liu P, et al. An improved guidance image based method to remove rain and snow in a single image. Comput Inform Sci, 2012, 5: 49
Zheng X H, Liao Y H, Guo W, et al. Single-image-based rain and snow removal using multi-guided filter. In: Proceedings of International Conference on Neural Information Processing, 2013. 258–265
Ding X H, Chen L Q, Zheng X H, et al. Single image rain and snow removal via guided L0 smoothing filter. Multimed Tools Appl, 2016, 75: 2697–2712
Kim J H, Lee C, Sim J Y, et al. Single-image deraining using an adaptive nonlocal means filter. In: Proceedings of IEEE International Conference on Image Processing, 2013. 914–917
Mu P, Chen J, Liu R S, et al. Learning bilevel layer priors for single image rain streaks removal. IEEE Signal Process Lett, 2019, 26: 307–311
Gauvain J L, Lee C H. Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains. IEEE Trans Speech Audio Process, 1994, 2: 291–298
Zhang K, Zuo W M, Gu S H, et al. Learning deep CNN denoiser prior for image restoration. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2017. 3929–3938
Fu Y H, Kang L W, Lin C W, et al. Single-frame-based rain removal via image decomposition. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2011. 1453–1456
Chen D Y, Chen C C, Kang L W. Visual depth guided color image rain streaks removal using sparse coding. IEEE Trans Circ Syst Video Technol, 2014, 24: 1430–1455
Kang L W, Lin C W, Fu Y H. Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans Image Process, 2012, 21: 1742–1755
Kang L W, Lin C W, Lin C T, et al. Self-learning-based rain streak removal for image/video. In: Proceedings of IEEE International Symposium on Circuits and Systems, 2012. 1871–1874
Wang Y L, Liu S C, Chen C, et al. A hierarchical approach for rain or snow removing in a single color image. IEEE Trans Image Process, 2017, 26: 3936–3950
Sun S H, Fan S P, Wang Y C F. Exploiting image structural similarity for single image rain removal. In: Proceedings of IEEE International Conference on Image Processing (ICIP), 2014. 4482–4486
Zhu L, Fu C W, Lischinski D, et al. Joint bi-layer optimization for single-image rain streak removal. In: Proceedings of IEEE International Conference on Computer Vision, 2017. 2526–2534
Eigen D, Krishnan D, Fergus R. Restoring an image taken through a window covered with dirt or rain. In: Proceedings of IEEE International Conference on Computer Vision, 2013. 633–640
He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016. 770–778
Fan Z W, Wu H F, Fu X Y, et al. Residual-guide feature fusion network for single image deraining. 2018. ArXiv:1804.07493
Qian R, Tan R T, Yang W, et al. Attentive generative adversarial network for raindrop removal from a single image. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2018. 2482–2491
Zhang H, Sindagi V, Patel V M. Image de-raining using a conditional generative adversarial network. IEEE Trans Circ Syst Video Technol, 2020, 30: 3943–3956
Yu F, Koltun V. Multi-scale context aggregation by dilated convolutions. 2015. ArXiv:1511.07122
Ren D W, Shang W, Zhu P F, et al. Single image deraining using bilateral recurrent network. IEEE Trans Image Process, 2020, 29: 6852–6863
Fu X, Liang B, Huang Y, et al. Lightweight pyramid networks for image deraining. IEEE Trans Neural Netw Learn Syst, 2020, 31: 1794–1807
Wang H, Xie Q, Wu Y C, et al. Single image rain streaks removal: a review and an exploration. Int J Mach Learn Cyber, 2020, 11: 853–872
Yasarla R, Patel V M. Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019. 8405–8414
Jiang K, Wang Z Y, Yi P, et al. Multi-scale progressive fusion network for single image deraining. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 8346–8355
Li G B, He X, Zhang W, et al. Non-locally enhanced encoder-decoder network for single image de-raining. 2018. ArXiv:1808.01491
Wang G Q, Sun C M, Sowmya A. ERL-Net: entangled representation learning for single image de-raining. In: Proceedings of IEEE International Conference on Computer Vision, 2019. 5644–5652
Hu X, Fu C W, Zhu L, et al. Depth-attentional features for single-image rain removal. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019. 8022–8031
Du Y J, Xu J, Qiu Q, et al. Variational image deraining. In: Proceedings of IEEE Winter Conference on Applications of Computer Vision, 2020. 2406–2415
Wang H, Wu Y C, Xie Q, et al. Structural residual learning for single image rain removal. Knowl-Based Syst, 2021, 213: 106595
Wang H, Xie Q, Zhao Q, et al. A model-driven deep neural network for single image rain removal. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 3103–3112
Yasarla R, Sindagi V A, Patel V M. Syn2real transfer learning for image deraining using gaussian processes. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 2726–2736
Cho J, Jang H, Ha N, et al. Deep unsupervised learning for rain streak removal using time-varying rain streak scene. J Korea Multi Soc, 2019, 22: 1–9
Jin X, Chen Z B, Lin J X, et al. Unsupervised single image deraining with self-supervised constraints. In: Proceedings of IEEE International Conference on Image Processing (ICIP), 2019. 2761–2765
Li S Y, Ren W Q, Zhang J W, et al. Single image rain removal via a deep decomposition-composition network. Comput Vision Image Understand, 2019, 186: 48–57
Li R, Cheong L F, Tan R T. Heavy rain image restoration: integrating physics model and conditional adversarial learning. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019. 1633–1642
Liu R S, Jiang Z Y, Fan X, et al. Knowledge-driven deep unrolling for robust image layer separation. IEEE Trans Neural Netw Learn Syst, 2020, 31: 1653–1666
Yang W H, Liu J Y, Yang S, et al. Scale-free single image deraining via visibility-enhanced recurrent wavelet learning. IEEE Trans Image Process, 2019, 28: 2948–2961
Deng S, Wei M Q, Wang J, et al. Detail-recovery image deraining via context aggregation networks. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 14560–14569
Wang H, Yue Z S, Xie Q, et al. From rain removal to rain generation. 2020. ArXiv:2008.03580
Garg K, Nayar S K. Photorealistic rendering of rain streaks. ACM Trans Graph, 2006, 25: 996–1002
Zhang L, Zhang L, Mou X Q, et al. FSIM: a feature similarity index for image quality assessment. IEEE Trans Image Process, 2011, 20: 2378–2386
Sheikh H R, Bovik A C. Image information and visual quality. IEEE Trans Image Process, 2006, 15: 430–444
Wang Z, Bovik A C, Sheikh H R, et al. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process, 2004, 13: 600–612
Huynh-Thu Q, Ghanbari M. Scope of validity of PSNR in image/video quality assessment. Electron Lett, 2008, 44: 800–801
Mittal A, Soundararajan R, Bovik A C. Making a “completely blind” image quality analyzer. IEEE Signal Process Lett, 2013, 20: 209–212
Mittal A, Moorthy A K, Bovik A C. No-reference image quality assessment in the spatial domain. IEEE Trans Image Process, 2012, 21: 4695–4708
Liu L X, Liu B, Huang H, et al. No-reference image quality assessment based on spatial and spectral entropies. Signal Process-Image Commun, 2014, 29: 856–863
Paszke A, Gross S, Chintala S, et al. Automatic differentiation in PyTorch. In: Proceedings of the 31st Conference on Neural Information Processing System, 2017
Goyette N, Jodoin P M, Porikli F, et al. Changedetection.net: a new change detection benchmark dataset. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2012. 1–8
Martin D, Fowlkes C, Tal D, et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of IEEE International Conference on Computer Vision, 2002
Liu R S, Cheng S H, Ma L, et al. Deep proximal unrolling: algorithmic framework, convergence analysis and applications. IEEE Trans Image Process, 2019, 28: 5013–5026
Acknowledgements
This work was supported by the National Key R&D Program of China (Grant No. 2020YFA0713900) and National Natural Science Foundation of China (Grant Nos. 11690011, 61721002, U1811461).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Wang, H., Wu, Y., Li, M. et al. Survey on rain removal from videos or a single image. Sci. China Inf. Sci. 65, 111101 (2022). https://doi.org/10.1007/s11432-020-3225-9
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11432-020-3225-9