The deep neural network (DNN) has made significant progre-ss in the single remote sensing image super-resolution (SRSISR). The success of DNN-based SRSISR methods mainly stems from the use of the global information and the fusion of shallow features and the deep features, which fits the non-local self-similarity characteristic of the remote sensing image very well. However, for the fusion of different depth (level) features, most DNN-based SRSISR methods always use the simple skip-connection, e.g. the element-wise addition or concatenation, to transform the feature coming from preceding layers to later layers directly. To achieve sufficient complementation between different levels and capture more informative features, in this paper, we propose a stage-mutual-affine network (SMAN) for high-quality SRSISR. First, for the use of the global information, we construct a convolution-transformer dual-branch module (CTDM), in which we propose an adaptive multi-head attention (AMHA) strategy to dynamically rescale the head-wise features of the transformer for more effective global information extraction. Then, the global information is fused with the local structure information extracted by the convolution branch for more accurate recurrence information reconstruction. Second, a novel hierarchical feature aggregation module (HFAM) is proposed to effectively fuse shallow features and deep features by using a mutual affine convolution operation. The superiority of the proposed HFAM is that it achieves sufficient complementation and enhances the representational capacity of the network by extracting the global information and exploiting the interdependencies between different levels of features, effectively. Extensive experiments demonstrate the superior performance of our SMAN over the state-of-the-art methods in terms of both qualitative evaluation and quantitative metrics.