Convolutional neural networks have become ubiquitous in image classification tasks. The state-of-the-art models for image classifications use convolutional layers in one way or another. There is a need for deploying deep learning models, especially the real-time vision models, in the edge devices to get better latency. But deploying such models in edge devices are becoming critical as the networks are becoming deeper and more dense. An overparameterized network is not necessarily required in many of the use cases of such deployment. This led researcher to develop technique for optimizing smaller and shallower networks, network architecture search techniques, and deep learning model compression techniques. In this research, we proposed a framework that utilizes deep determinisitic policy gradient, a class of deep reinforcement learning algorithm, to the learn the best set of filters considering the intrinsic dimensionality of the dataset, feature of each layer and the criteria based on which the filters of a convolutional layer will be ranked. By learning this relationship, we can prune off unnecessary filters which will reduce both computational and memory requirement for the model without losing too much accuracy. Our method showed that the model can prune off 66% filters overall.