Single image super-resolution plays an important role in the field of computer vision. This technology aims to reconstruct high-resolution images from low-resolution images. In recent years, deep neural networks make performance in SISR task significantly improved. However, recently works based on convolutional neural network equally treat high-frequency and low-frequency features, which makes the reconstruction of high-frequency details poor, the output too smooth and the texture information lack. On the other hand, very deep convolutional network is not easy to converge, and as the depth of the neural network grows, the long-term information from the former layer can easily be weakened or lost in the latter layer, which makes the benefit not proportional to the depth of the network and the computational complexity. To solve these above problems, it proposes a spatial attention module and a channel attention module as the basic block of convolutional neural network for SISR. Firstly, in the same channel, the information of different locations is given different weights by the spatial attention module. Secondly, the weights between different channels are determined by the channel attention module, which makes the high-frequency information gain a higher position in the reconstruction task. The reconstruction performance is improved. It further proposes a short-term and long-term feature modulation module to transform the layer depth of the network into the block depth, which greatly reduces the depth of the network, in order to solve the problem of long-term information loss in the front layer. Compared with other methods based on deep convolution neural network, the Peak Signal-to-Noise Ratio（PSNR） on several benchmark datasets are better, which proves the effectiveness of the proposed method.