The general aim of this paper is to study the spatio-temporal modeling techniques which can efficiently represent multiple moving objects’ in video databases. The traditional schemes only consider direction property, time interval property, and spatial relationship property for modeling moving objects’ trajectories. But, our scheme also takes into account on distance property, conceptual location information, and related object information so that we may improve a retrieval accuracy to measure a similarity between two moving objects as well as them. As its application, we implement the Content- and Semantic-based Soccer Video Retrieval (CS²VR) system by using MS Visual C++ and DirectX for indexing and searching on soccer video data. The CS²VR helps users to easily extract the trajectory information of soccer ball form soccer video data semi-automatically as well as to conveniently retrieve the results acquired by sketching query trajectory with mouse button.