Brain age prediction is important for understanding brain development and aging. Currently, researchers can predict brain age using resting-state functional MRI (rs-fMRI) data. However, there are differences in brain age development among different subjects, and the same subject also has different development at different ages. So far, how to accurately estimate brain age using rs-fMRI efficiently remains a challenging problem. Therefore, a brain age prediction model with the similarity metric convolutional neural network is proposed in this paper. Specifically, this paper first introduces a siamese convolutional neural network, which includes convolution, batch normalization, and pooling steps simultaneously learns the features of two groups of rs-fMRI, and designs a similarity measurement network. Subsequently, fMRI images of two groups of different subjects are input into the network, and a similarity metirc module is designed to calculate the similarity between the two groups of images. Then the network is optimized by a loss function, and finally, the average value of the three groups of sample labels with the greatest similarity is taken. The absolute mean error and correlation coefficients obtained from the model are 5.337 and 0.6279, respectively. Experimental results show that this method has low mean absolute error and high correlation coefficient on the longitudinal imaging data set of Southwest University.