In this paper we address the 3D reconstruction of points, on a non-central catadioptric system, composed by a mirror, a projector, and a perspective camera. The goal of the paper is to propose a framework to build an omnidirectional depth camera, towards an omnidirectional RGB-D camera system. The main contributions are: an efficient technique to project 3D points from the world to an image of a general non-central catadioptric camera; the definition of the template pattern (for both the projector and camera's images); and the matching between the projection of these features to the world and its respective images. The 3D depth is directly recovered using the template matching approach. In conclusion, we apply some filtering techniques to improve the results. To evaluate the proposed framework, we test the method using synthetic data, under different levels and types of noises, proving that the framework is robust to noise and, thus, can be put into practice.