A CNN is usually composed of several convolutional layers, pooling layers, and fully connected layers which result in a deep network architecture. Therefore, a CNN is able to deal with more complex classification and recognition problems and achieve excellent results.

Specifically, the training sample set is assumed to be X={x(1),x(2),⋯,x(m),⋯,x(M)} and its corresponding labeled sample set. For each convolutional layer l , all feature maps are summed by the convolution operation of the previous layer’s feature map with a convolutional kernel. The calculation of the feature map is shown in Eq. (1):

(1) x(l,s)=f(∑t=1Nl−1x(l−1,t)∙k(l,s,t)+b(l,s))

where x(l,s) is the sth feature map of the lth layer, Nl−1 is the number of the previous layer’s feature map, and k(l,s,t) b(l,s) are the convolutional kernel and corresponding bias terms, respectively.

The input feature map is down-sampled by the pooling layer to realize scale-invariance. The number of feature maps is unchanged. The down-sampling operation is shown in Eq. (2):

(2) x(l,s)=g(β(l,s)∙down(x(l−1,s))+b(l,s))

where down(∙) is the pooling function, β is the multiplicative bias, and b is the additive bias. According to this formula, each output feature map of the pooling layer is the down-sampling of its corresponding input feature map.

The mean square error is the energy function of the whole network, as shown in Eq. (3):

(3) J(W,b)=12M∑m=1M‖y(m)−z(m)‖22

where z(m) is the actual output.

In practice, the overall accuracy of the convolutional neural network is related to the depth of the network. In general, the accuracy of the model is improved by increasing the network depth, but at a certain point the overall accuracy will decrease if the network depth continues to increase. The main reason is that the deeper the network, the more likely it is to encounter the vanishing gradient problem, and it is easy to fall into a local minimum. Therefore, it is difficult to make full use of the feature extraction ability of the deep network by directly stacking shallow layers into a deep network.

To address the gradient degeneration problem, He proposed the Residual Neural Network (ResNet) [19]. The residual block is the basic architecture of ResNet; a residual neural network is composed of several residual blocks, as shown in Fig. 1:

Here, x represents the input data. For a network with no short connections, the output is F(x) ; for a residual block with short connections, the output is H(x) , where H(x)=F(x)+x . Experimental results show that it is much easier to optimize the residual mapping F(x) than the original function mapping H(x) in the residual block. H(x) can be understood as the sum of the residual mapping F(x) and the identity mapping x in the network. Identity mapping neither increases the number of parameters nor affects the complexity of the original network. In the figure, Conv represents the convolution operation, BN represents batch normalization, and ReLU represents the activation function.

ResNet has one more shortcut connection than a traditional neural network. From the perspective of feature flow, it enables features to be transferred directly to the next layer. When the layers of the neural network are very deep, there are still lower features that enhance the higher features, so that the features can be introduced deeper. From the perspective of backpropagation, when the output changes a small amount, the gradient ∂H(x)∂x will be very small, which is extremely difficult for directly learning H(x) . However, because the difference is calculated in ResNet training F(x) , which amplifies the slight change, the gradient becomes ∂H(x)∂x−1 . The absolute value of the gradient becomes larger, the training process continues, and the degradation problem is solved.

Recently, He et al. built a random depth architecture based on a 1202-layer ResNet [19]. However, they found that randomly discarding the ResNet layer did not change the convergence in training. This phenomenon indicates that ResNet does not make full use of the output feature by each convolutional layer in the residual block, and also ignores the connection between any two convolutional layers. Meanwhile, the mode of adding layers is not conducive to the transmission of features in the network.

Huang et al. [20] proposed the DenseNet model. DenseNet is able to connect any two convolutional layers in a dense cell, realizing feature reuse and feature transfer. DenseNet is based on a residual dense block, which is composed of several convolutional layers and activation layers, and plays the role of feature extraction. The output of each block will establish a short connection with the output of each convolutional layer of the next block, realizing continuous feature transmission. The structure of a residual network is shown in Fig. 2:

Assume that the input and output of the dth block are Fd−1 and Fd , respectively. The number of input and output feature maps is G0 . The output of the cth convolutional layer can be represented as in Eq. (4):

(4) Fd,c=H{[Fd−1,Fd,1,⋯,Fd,c−1]}

where H(∙) is a nonlinear operation of the convolutional layer, including convolution and ReLU functions. Let Fd,c output G feature maps representing the connection between the feature maps output by the previous block and the feature graph output by the c−1 convolutional layer before the block, containing G0+(c−1)×G feature maps in total.

Since full connection is adopted between the input layer of the block and the convolutional layer, it is necessary to compress the feature maps at the end of the block. Therefore, 1 × 1 convolution is adopted to control the number of feature maps, which can be represented as in Eq. (5):

(5) Fd,LF=HLEFd{[Fd−1,Fd,1,⋯,Fd,c]}

where HLEFd represents the 1×1 convolution operation. The final output of the block is shown in Eq. (6):

(6) Fd=Fd−1+Fd,LF

Local residual learning is calculated by adding the output and input of the block, which further preserves a large amount of image detail and improves the feature extraction performance of the residual dense block.

HSI is a 3D cube with rich spectral-spatial features. As a result, the 3D convolution operation [21] is adopted to extract spectral and spatial features. The 3D convolution operation is shown in Fig. 3.

As we can see from this figure, the input data is a 3D image composed of spectral and spatial dimensions. Therefore, the convolution kernel performs the convolution operation on both spectral and spatial dimensions of the input 3D image. One pixel at a time is obtained in the 3D image by the convolutional operation, and a new 3D feature map is obtained after the processing of the whole image. The calculation is shown in Eq. (7).

(7) xi,jx,y,z=σ(∑m∑p=0Pi−1∑q=0Qi−1∑r=0Ri−1Wi,j,mp,q,rX(i−1),mx+p,y+q,z+r+bij)

Here, xi,jx,y,z is the output value of the jth feature map at position (x,y,z) of the ith layer. m is the set of feature maps connected to the current feature graph at the (i−1)th layer, Wi,j,mp,q,r is the weight of the position (p,q,r) of the 3D convolution kernel in the mth feature map, and bi,j is the bias. σ(∙) is the activation function. Pi , Qi , Ri is the length, width, and height of the convolutional kernel, respectively.

HSI is characterized by large data volumes but with limited data for training. The features learned by the convolution kernel with a fixed scale are not conducive to the training of the model. Therefore, a multi-scale network is used to learn features at different scales, extract more discriminative features, and improve feature extraction for small sample data. HSI classification by the 3D-multiscale network can alleviate the problem of low accuracy caused by limited training samples.

HSI is three-dimensional data, including one-dimensional spectral data and two-dimensional spatial data. Although HSI contains abundant spectral information, there are many bands with high correlation between adjacent bands and data redundancy. Since spectral and spatial information play important roles in HSI classification, spectral and spatial dimensions should be considered in feature extraction. HSI features are extracted by using a 3D convolutional kernel [22,23]. Although these methods improve the classification accuracy, they do not fully extract discriminative spectral-spatial features.

In order to predict the category of ground objects, we propose 3D-Multiscale Spectral-Spatial DenseNet. A convolution kernel of 1×1×5 and 1×1×7 is chosen to extract spectral features, and a convolution kernel of 3×3×1 and 5×5×1 is chosen to extract spatial features. In the network, spectral and spatial features are extracted continuously, and more discriminative spectral-spatial features are used for classification. The application of multi-scale networks can alleviate the problem of limited training samples. In addition, the feature fusion module is embedded in the multi-scale network. The 3DMSS approach shares feature information of different scales to enhance the information flow of the network, which is conducive to the extraction of spectral-spatial features and improves the classification accuracy. The model of the network is shown in Fig. 4.

3D-Multiscale Spectral DenseNet is shown in Fig. 5. In the training process, for the purpose of dimensionality reduction, the convolution operation is carried out using 24 convolutional kernels with a step size of 2 for the original HSI. The 3D feature map after dimensionality reduction is used as the input to the spectral feature extraction channel.

In 3D-Multiscale Spectral DenseNet, the multi-scale features of the spectral domain are extracted by using the K convolution kernels with of size p and q, respectively. This is shown in Eqs. (8) and (9):

(8) M1=σ(w1×1×p1⊗On−1+b1×1×p1)

(9) N1=σ(w1×1×q⊗On−1+b1×1×q1)

where On−1 is the input feature map of the 3D-Multiscale Spectral DenseNet. ⊗ is the convolutional operation, w is the weight of the convolution kernel, and b is the bias. The superscripts of w and b are the number of convolutional layers and the subscripts are the size of the convolutional kernel. σ(∙) is the activation function.

Shallow spectral features at two scales were extracted, and On was obtained by fusing the K feature maps (a total of 2×K feature maps) learned at each scale and the original input. This is shown in Eq. (10).

(10) On=On−1+M1+N1

Then, K spectral convolutional kernels of different scales are used to carry out the multi-scale convolution operation on On , as shown in Eqs. (11) and (12):

(11) M2=σ(w1×1×p2⊗On+b1×1×p2)

(12) N2=σ(w1×1×q2⊗On+b1×1×q2)

where the meanings of each variable are the same as in formula (8) and formula (9). The discriminant spectral feature diagram O will be learned after the extraction of spectral features, as shown in Eq. (13).

(13) On+1=On+On−1+M2+N2

3D-Multiscale Spatial DenseNet is shown in Fig. 6. In the training process, to achieve the purposes of the dimension reduction, the convolution operation is carried out by using 24 convolutions kernels with a step size of 2 to the original HSI. The 3D feature map after dimension reduction is used as the input data of spatial feature extraction channel.

In 3D-Multiscale Spatial DenseNet, the multi-scale features of spatial domain are extracted by using the K′ convolution kernels with sizes of p′ and q′ , respectively. As shown in Eqs. (14) and (15):

(14) M1′=σ(wp′×p′×11⊗On−1′+bp′×p′×11)

(15) N1′=σ(wq′×q′×11⊗On−1′+bq′×q′×11)

where, On−1′ is the input feature map of the 3D-Multiscale Spectral DenseNet. “ ⊗ ” is the convolutional operation. w is the weight of the convolution kernel. b is the bias. The superscript of w and b is the number of convolutional layers, and the subscript is the size of the convolutional kernel. σ(∙) is the activation function.

The shallow spatial features at two scales were extracted, and the On′ is obtained by fusing the K′ feature maps (a total of 2×K′ feature maps) learned at each scale and the original input. As shown in Eq. (16):

(16) On′=On−1′+M1′+N1′

Then, K′ spatial convolution kernels of different scales are used to carry out multi-scale convolution operation on On′ , as shown in Eqs. (17) and (18):

(17) M2′=σ(w5×5×12⊗On′+b5×5×12)

(18) N2′=σ(w3×3×12⊗On′+b3×3×12)

The meanings of each variable are the same as formula (14) and formula (15).

The discriminant spectral feature diagram O will be learned after the extraction of spectral features, as shown in Eq. (19):

(19) On+1′=On′+On−1′+M2′+N2′

As shown in Fig. 7, the results of spectral and spatial learning are concatenated as input followed by a BN, RELU, and convolution layer block, which is the same as the process for Block 2.

At the end of the block, global average pooling layers are inserted. It was originally designed to replace the traditional FC layer in CNNs. The global average pooling layer contains a much smaller number of parameters than FC layers and can retain good localization ability for a network. It is important to consider two main problems in HSI classification: the overfitting phenomenon caused by the large model scale with limited training data, and the effective extraction of both spectral and spatial features. After the FC layer, a softmax layer is used to obtain the final classification result.

Three datasets were used: Indian Pines (IN), Kennedy Space Center (KSC), and Salinas (SA). The Indian Pines dataset contains 220 spectral channels and the spatial resolution is 20 m. Each band contains 145 × 145 pixels. The sample size is shown in Tab. 1. The KSC dataset contains 224 spectral channels and 13 land cover categories; the sample size is shown in Tab. 2. The Salinas dataset contains 224 spectral channels, and the spatial resolution is 3.7 m. The sample size is shown in Tab. 3.

To set the parameters of 3DMSS, we determined the optimal parameters through a series of experiments, which included the number of convolution kernels, the convolution kernels’ depth of spectral feature channels, the convolution kernels’ size of spatial feature channels, and the number of training samples in each batch.

In order to verify the classification performance of 3DMSS proposed in this paper, we compared this with five other classical HSI classification methods on the basis of the OA, AA, and Kappa coefficients. These five methods include: A Support Vector Machine method [6], a Gabor-based method (GABOR) [24], the Image Fusion and Recursive Filtering method [25], 3D-CNN [17], and MS3FE [26]. Tabs. 6–8 show the test results of each method for three datasets. Figs. 9–11 are the visual maps of the different methods on the three datasets.

From Tabs. 6–8, it can be seen that the proposed method attains the best classification performance for the three datasets. The OA for the three datasets is 99.36%, 99.86%, and 99.99%, respectively. Figs. 9–11 are the visual maps of the different methods for the three datasets. It can be seen from the figures that the visual maps for SVM, GABOR, and 3D-CNN have noise and fuzzy classification. The visual maps for RF and MS3FE have clear classification boundaries, but there is still a small amount of noise. The visual maps for 3DMSS are the clearest, and the classification result is the closest to the real object label.

Analyzing the above experimental results, we can draw the following conclusions:

The higher the spatial resolution of HSI, the better the classification performance is achieved for the larger convolutional kernel size. The spatial resolution of IN, KSC, and SA is 145 × 145, 512 × 614, and 512 × 217, respectively. Since the resolution of IN is the smallest, IN achieves the best classification accuracy for the convolution kernel size of 3 × 3. The spatial resolution of KSC and SA is greater than IN, so they achieve the best classification accuracy for the slightly larger convolution kernel size 5 × 5.