Summary: This paper presents a new approach for reconstructing realistic 3D models of buildings from uncalibrated image sequences taken by a hand-held camera. Firstly, correspondences between image pairs are established by using various computer vision tools, and then the fundamental matrix is estimated to high accuracy. Meanwhile, homography constraints are exploited to find more correspondences, to avoid degenerate cases and to obtain more accurate results. Secondly, rectified image pairs are resampled by using epipolar geometry constraints, where epipolar lines coincide with image scan-lines and disparities between the images are in the \(x\)-direction only. This allows subsequent stereoscopic analysis algorithms to easily take advantage of the epipolar constraint and reduce the search space to one dimension, namely along the horizontal row of the rectified images. Furthermore, dense stereo matching of the original image pairs is simple and low computational cost. Finally, the 3D model can be built through self-calibration, matching and Delaunay triangulation. The self-calibration method uses prior knowledge of orthogonal planes (lines) and parallel planes (lines) to act as constraints on the absolute quadric. A large number of experimental results show that this method improves the speed and accuracy of reconstructed 3D models and the 3D models obtained are more realistic.