The task of removing rain is of great significance for outdoor vision systems such as video surveillance, vision based navigation and so on. Rain produces complex time varying intensity fluctuations in images or videos, which seriously reduce the performance of outdoor vision systems. Due to similar visual appearances of rain and moving objects, the current rain removal algorithms cannot easily distinguish between the two. In this paper, we propose a novel algorithm for rain detection and removal based on the rain image formation model and edge-preserving filtering. The effectiveness of our algorithm is demonstrated in comparison with the existing approaches, by experimenting on videos of intricate scenes with moving objects or time-varying textures.