In recent years, with the rapid development of computer technology and the popularity of four-rotor unmanned aerial vehicles equipped with high-definition cameras, three-dimensional reconstruction of buildings based on passive scanning has become a hot issue in the research field of computer vision and graphics. However, current passive scanning methods for modeling are characterized with such flaws as time-consuming reconstruction and poor quality in the reconstruction of local details. In view of these shortcomings, a three-dimensional model reconstruction method has thus been proposed based on UAV aerial sequences. First, the SFM method is adopted to recover the sparse point cloud of the surrounding scene from the aerial image collection of the target building. Next, by adopting the RANSAC method, an effort has been made to extract the ground and the building facades. And then the point clouds are clustered and extracted to obtain the columnar model of the building. Finally, the 3D model of the building is obtained by smoothing the columnar model. The experimental results show that the 3D model of the building can be obtained quickly by adopting the above-mentioned method, with the contour details of the top of the building well preserved.