Diffusion tensor imaging (DTI) is an advanced magnetic resonance technology that describes subtle brain structures using a diffusion tensor at each point. The obtained DTI image is always degraded since diffusion-weighted imaging sequences, which are used to estimate DTI images, are corrupted by noise. In this paper, we propose an approach called geometric invariant nonlocal means on the tensor manifold (GINLM-TM) to reduce undesired components in the degraded DTI image. We transform the diffusion tensor into a positive definite matrix (called a tensor) to measure the intrinsic property of the diffusion tensor. Then, we directly regularise DTI images in the tensor manifold endowed with an affine invariant metric. Finally, geometrically invariant measures of patches of tensors are used to define the similarity function of patches to ensure the similarity between patches is more accurate and robust. It is experimentally demonstrated that the proposed method performs adequately in reducing undesired components without blurring the boundaries of DTI images. The results of fractional anisotropy (FA) images and fibre tracking of our restored data indicate that our method performs well in denoising the DTI image.