Outliers seriously affect the accuracy in geometric model fitting. Previous works in coping with outliers involve threshold selection and scale estimation. However most scale estimators suppose the inlier distribution as a Gaussian model, which usually poorly meets the cases in geometric model fitting. Outliers, considered as the points with big residuals to all the true models, share common items with big values in quantized residual preferences, thus making the outliers gather away from the inliers in quantized residual preference space. In this paper we make use of this outlier consensus in quantized residual preference space by extending the usage of energy minimization to combine the model error and the spatial smoothness in quantized residual preference space for outlier detection. The energy minimization based outlier detection process follows an alternate sampling and labeling framework. After the outlier detection, an ordinary energy minimization method is employed to optimize the inlier labels, which also follows the framework of alternate sampling and labeling. The experimental results in this study show that the energy minimization based outlier detection method can detect most of the outliers in the data. Furthermore, the proposed energy minimization based inlier segmentation process segments the inliers into different models quite accurately. Overall, the performance of the proposed method is better than most of the state-of-the-art methods.