In this paper, we develop a unified image deblurring framework that consists of both blur kernel estimation and non-blind image deconvolution. For blind kernel estimation, we propose a patch selection procedure and integrate it with a coarse-to-fine kernel estimation algorithm to develop a robust blur kernel estimation algorithm. For the non-blind image deconvolution, we modify the traditional Richardson-Lucy (RL) image restoration algorithm to suppress the notorious ringing artifact in the regions around strong edges. Experimental results on some real blurred images are shown to demonstrate the improved efficiency and image restoration by using the proposed algorithm.