Single-channel and dual-channel CNNs are the two most common network models used in fine-grained recognition applications. A variation of the single-channel CNN, the dual-channel CNN consists of two CNN networks, or feature extractors. It extracts the information about interactions between various regions from the feature maps that the two CNN networks extracted using bilinear pooling procedures [7]. In 2015, Lin et al. [8] proposed a novel dual-channel CNN model for extracting high-order features from images. This model multiplied the features outputted by two parallel networks, M-Net and D-Net, and pooled them to obtain an image descriptor, thereby enhancing feature representation capability. In 2017, Lin et al. utilized the pooling outputs of the dual-channel CNN for feature extraction and located feature interactions under certain rules to obtain second-order features. However, the results of the experiments indicated that the bilinear CNN needed two feature extraction rounds, which resulted in lengthy training periods. Furthermore, a significant amount of redundant data was included in the second-order features that were produced following feature interaction. The authors made an effort to lessen superfluous features without compromising identification accuracy, however the results were not very pleasing. Although the dual-channel CNN can capture interaction information between different regions in feature maps, it requires two rounds of feature extraction, resulting in prolonged training time. In this study, we apply dimensionality reduction and down sampling to the features outputted by the fourth convolutional group of ResNet-50, ensuring that the feature dimensions from the fourth and fifth convolutional groups are consistent. Then, we encode the features from the fourth and fifth convolutional groups using the efficient second-order feature encoding method proposed in this study. By employing a single CNN for one round of feature extraction, we effectively shorten training time. For further details, please refer to Section 3.1 in this paper.

Data augmentation is a method that takes original samples and applies a few adjustments to create new ones. Its goal is to increase the dataset’s size to mitigate the overfitting issue brought on by a lack of training data. Tran et al. [9] analyzed data augmentation strategies from the perspective of Bayesian methods, learning missing data from the distribution of the training set to perform data augmentation. However, this method does not specifically focus on discriminative regions. Hu et al. [10] employed a single channel to guide the generation of augmented samples through cropping and erasing, treating discriminative and background regions equally. Cropping and erasing may unintentionally eliminate discriminative regions when the fraction of objects to be identified is low, making it impossible to increase the number of samples containing discriminative regions in a targeted manner. Chen et al. [11] randomly sampled a batch and applied two types of data augmentation to each image in the batch, producing two views. They aimed to bring different views of the same image closer together in latent space and keep views of different images far apart. This method could not guarantee that augmented samples must contain discriminative regions since it did not distinguish between discriminative and background regions, even though restricting augmented samples with loss functions made them more rational and realistic. Current data augmentation methods in fine-grained recognition research increase sample numbers but primarily rely on simple geometric transformations. These methods treat both target object regions and background regions equally, without focusing on discriminative regions. As a result, some augmented samples may consist solely of background regions, generating ineffective samples and weakening the model’s generalization ability. To address this issue, this study proposes an importance-ranking algorithm for localizing discriminative regions. This algorithm leverages the weight matrix of the fully connected layer to rank the importance of each channel in the feature map. Since each channel corresponds to a specific part of the image, selecting and combing channels with higher importance allows for precise localization of discriminative regions. This ensures that the augmented samples generated through data augmentation contain meaningful discriminative regions, thereby enhancing the effectiveness of the new samples.

The distinctions between object categories in fine-grained recognition tasks are often so slight that they cannot always be made out, not even with convolutional features. Several techniques for computing second-order features have emerged to capture the minor distinctions between features of different categories. Second-order feature encoding methods combine first-order features, such as by taking the outer product of features [12,13]. Second-order features can capture interactions between features and express nonlinear relationships. Lin et al. [8] extracted features from images using two separate CNNs. To acquire second-order features, they multiplied the feature values in distinct channels at the same place in the two feature maps. The capacity to represent features is improved by these second-order features, which are the sum of the features from two CNN paths. Gao et al. proposed a compact bilinear pooling method, which effectively reduces feature dimensions compared to conventional bilinear pooling methods, albeit with a slight decrease in performance. Kong and Fowlkes [14] introduced the LRBP model, which addresses the issue of excessively high dimensionality of fused second-order features and the need for large parameter quantities in linear classifiers through two successive approximation operations: low-rank approximation of parameter matrices and shared projection. Cui et al. [15] proposed the Kernel Pooling framework, which captures second-order information between features and is versatile. Paschali et al. [16] proposed the bilinear attention pooling method, which randomly selects a channel from the features outputted by the fourth convolutional group of ResNet-50 and multiplies it channel-wise with features from the fifth convolutional group to obtain second-order feature [17]. Nevertheless, the second-order features could not always improve discriminative region features because of the randomness of the selection process. Although second-order features can capture local structures and texture information, offering stronger feature representation and improved classification performance, traditional second-order feature encoding methods require two CNNs to extract first-order features from images and combine them. This leads to longer training time and high-dimensional fused features. Therefore, this study applies dimensionality reduction, down sampling, and filtering to the features outputted by the fourth convolutional group of ResNet-50 to generate an attention map. This attention map is then multiplied channel-wise and pixel-wise with the features from the fifth convolutional group to produce second-order features. By performing feature extraction only once and reducing the dimensionality and complexity of the features, we shorten training time and alleviate the issue of excessively high dimensionality in fused second-order features.

In order to reduce training time and mitigate the difficulties caused by small object proportions and complex backgrounds in fine-grained images, sfeModel uses ResNet-50 as the backbone network and integrates a data augmentation module and an effective second-order feature encoding module, as shown in Fig. 1. The efficient second-order feature encoding module utilizes a single CNN for one round of feature extraction. It reduces and down samples the convolutional features outputted by the fourth convolutional group to generate an attention map, then performs elementwise multiplication between the attention map and the convolutional features of the fifth convolutional group at corresponding positions to obtain second-order features. Rather than treating background and discriminative regions equally, the data augmentation module uses the significance ranking method to localize discriminative regions, guaranteeing that supplemented samples contain discriminative regions.

The sfeModel reduces training time and increases identification accuracy by combining the effective second-order feature encoding module and the data augmentation module. This helps to efficiently handle the obstacles presented by complicated background and small object proportions in images.

Previous studies have shown that second-order feature encoding can effectively capture local structural and textural information, thereby enhancing feature representation and classification performance. In contrast, traditional aggregation methods such as summation or averaging rely solely on first-order statistics. As illustrated in Fig. 2, let the convolutional features outputted by the fourth convolutional group and the fifth convolutional group be denoted as X4 and X5, respectively. We perform global average pooling on X4, resulting in a 1×1×c vector. After sorting this vector in descending order, we select the top m values to form an attention map Am, where each value corresponds to a channel in X4. Next, a max pooling with a 2×2 pooling kernel and stride 2 on Am was done to down sample it, resulting in Am′. At this point, Am′ and X5 have the same height and width. Finally, element-wise multiplication between Am′ and the corresponding elements of X5 was done at each position to obtain second-order features F, as shown in Eq. (1).

(1)F=Am′×X5

In this context, F represents second-order features, Am′ represents the downsampled attention map after pooling, and X5 represents the output features from the fifth convolutional group of ResNet-50. The fourth convolutional group’s output features are used to create attention maps. The attention map’s channels each represent a distinct feature of the object, such as the head of a bird, an airplane’s wings, or an automobile’s registration plate. The element-wise multiplication between the attention map Am′ and the feature map X5 is designed to efficiently capture second-order feature interactions. Unlike the full outer product in classical bilinear pooling, which computes interactions between all channel pairs, our method computes a selective interaction. The attention map Am′, derived from the most salient channels of X4, acts as a condensed representation of discriminative shallow features (e.g., edges, textures). The operation F=Am′×X5 calculates, for each spatial location, the Hadamard product between this shallow feature representation and the deep semantic features of X5. This captures the local co-activation or covariance between these two distinct sets of features. The resulting feature map F thus encodes how discriminative low-level patterns and high-level semantic concepts co-vary across the image, which is a fundamental characteristic of second-order statistics. This approach provides a powerful yet computationally efficient alternative to full bilinear pooling. The sfeModel only needs one round of feature extraction, saving computational costs and complexity as compared to bilinear networks. To mitigate the issue of excessively high dimensionality in the fused second-order features, attention maps are generated in order to down sample the features produced by the fourth convolutional group.

Through the second-order feature encoding module described in Section 3.1, we obtain a second-order feature map F={F1,F2,F3,…Fi...Fp}, where i∈[1,P]. Here, P represents the number of channels. We then reduce the dimensionality of the second-order feature map F by applying convolutional layers, aligning the number of channels in F with that of the output feature map of the fifth convolutional group of ResNet-50. Suppose the dimension-reduced feature map is denoted as F′={F1,F2,F3…Fi...Fc}, where i∈[1,C], and C represents the number of channels in the dimension-reduced feature map. The dimension-reduced feature map F′ is flattened into a one-dimensional vector f∈R(CHW)×1×1. Here, H, and W respectively represent height and width of the second-order feature map. However, due to the high dimensionality of those features, which leads to excessive parameter calculations and a waste of computational resources and memory, global average pooling is employed to further reduce the dimensionality of each channel of the dimension-reduced feature map. After dimension reduction, a feature vector f∈RC×1×1, is obtained which is then normalized in terms of its magnitude. Finally, the normalized feature vector f∈RC×1×1 is fed into a classifier for classification.

The classifier is implemented by fully connected layers and a soft max layer. The fully connected layers use a weight matrix to map the dimension-reduced feature vector f∈RC×1×1 to probability scores for each class. To be more precise, the weight matrix is used to weight and sum each member of the feature vector in order to get the probability scores for each class. As a result, the weight matrix may be used to determine the significance of each channel in the feature map for a particular class. Let N be the number of classes, and the weight matrix of the fully connected layers be denoted as w∈RCXN. Each element wi,j in the weight matrix represents the connection weight between the ith element of the feature vector and the score for class j, where i∈[1,C] and j∈[1,N]. Since each element of the feature vector represents the global information of the corresponding channel in the feature map F, wi,j can be interpreted as the contribution of the ith channel in the feature map F to class j.

As shown in Fig. 3, based on the weight matrix w∈RCXN, the importance of each channel in the feature map can be determined for each class. While importance sorting is based on the categories with the greatest probability scores during testing, it is carried out during training using the categories corresponding to the ground truth labels. The elements Wi,j are sorted in descending order, where i∈[1,C], and j represents the selected classes. This importance sorting algorithm selects the top k values, where these k values correspond to the most important channels, providing an approximate localization of the discriminative regions in the object. The remaining C−k channels are considered relatively less important. To ensure diversity in the cropped regions, m (m<C−k) channels are randomly selected from these C−k channels. The weighted sum of these selected k channels and m channels is computed to generate the cropped enhancement image, as shown in Eq. (2). Furthermore, t channels are randomly selected from the aforementioned k channels, and the weighted sum of these t channels is computed to generate the erased enhancement image, as shown in Eq. (3). Finally, normalization is performed on the cropped enhancement image and erased enhancement image according to Eq. (4).

(2)Ac(i,j)=∑i=1k+mWi,jFi (3)Ad(i,j)=∑i=1tWi,jFi (4)A∗′(i,j)=A∗−min(A∗)max(A∗)−min(A∗)where Ac represents the cropped enhancement image, Wi,j denotes the weight value at position (i,j) in the weight matrix, Fi represents the i-th feature channel, and Ad represents the erased enhancement image. A∗ denotes both the cropped enhancement image Ac and the erased enhancement image Ad, while A∗′ refers to the normalized cropped enhancement image and erased enhancement image. Each feature channel can locate certain specific regions in the original image. Note that this method prevents the problem of overly small, cropped regions that arises from randomly choosing only one channel each time, while also ensuring sample diversity. The network cannot acquire useful information when the cropped region is too tiny because it contains fewer features. The number of feature channels selected in this study is limited by parameters, preventing the cropped region from becoming excessively large. An excessively large cropped region contains too many non-discriminative regions, which defeats the purpose of excluding interference. The k channels identified locate the discriminative regions of the object. Therefore, the subsequent selection of t channels can pinpoint relatively important discriminative sub-regions. By removing these sub-regions, the network is encouraged to look for other discriminative features instead than depending only on one particular class of discriminative features. The network will rely on other discriminative features for recognition when a discriminative region in the original sample image is obscured, strengthening the model’s resilience.

This study conducts experiments on three datasets including CUB-200-2011, Stanford Cars, and FGVC Aircraft. For region cropping and region erasure, the threshold θ was randomly sampled uniformly between 0.2 and 0.4.

For the objects that need to be classified, each of the three datasets has part annotation points, bounding box annotations, and category labels. All trials, however, simply use the object’s category labels. For the objects that need to be classified, each of the three datasets provides part annotation points and bounding box annotations in addition to category labels. It is important to emphasize that in all our experiments, only the image data and category labels were used for both training and testing; no bounding boxes, part annotations, or other localization signals were utilized at any stage. This ensures a fair comparison with other weakly-supervised methods and demonstrates our model’s capability to autonomously locate discriminative regions.

Table 1 displays the specifics of the three datasets. CUB-200-2011 dataset comprises 11,788 images of 200 bird species, with a nearly 1:1 ratio between the training and test sets. Because birds usually have small bodies, the bird item takes up a small amount of the image. Furthermore, birds frequently rest on elaborate tree branches or structures, creating background that are varied and complicated. Furthermore, birds differ significantly in appearance while they are in various stances, including flying, standing, and spreading their wings. As a result, this dataset is generally acknowledged to be difficult for tasks involving recognition. Stanford Cars dataset comprises 16,185 images of 196 car models, with a nearly 1:1 ratio between the training and test sets. The images are captured from multiple angles, leading to significant variations in the proportions of the objects across different images.

FGVC Aircraft dataset comprises 10,000 images of aircraft. It can be categorized into four levels of granularity: Manufacturer, Family, Variant, and Model. This research employs the Variant level, where the 10,000 aircraft images are divided into 100 different categories. The ratio of training set to test set is approximately 2:1.

This study uses ACC (Accuracy) as the evaluation metric. ACC is calculated as the proportion of correctly predicted samples to the total number of testing samples, as shown in Eq. (7): (7)ACC=ncorrectnwhere ncorrect represents the number of samples predicted correctly, and n represents the total number of samples. If the predicted label for each sample matches the ground truth label, the sample is deemed to have been accurately predicted; if not, the sample was predicted wrongly.

PyTorch 1.0 was utilized to train this study. An i7-10700 CPU and an NVIDIA GeForce RTX 2070 GPU were part of the experimental configuration. ResNet-50, which was initially trained using pre-trained weights from ImageNet, was used as the backbone network. During the training process, the momentum coefficient was set to 0.9, the weight decay parameter was set to 1e−5, and the initial learning rate of the network was set to 1e−3. The learning rate was multiplied by 0.9 every two epochs. The batch size for training samples was set to 10. For region cropping and region erasure, the thresholds θc and θd were randomly sampled uniformly from the range [0.2,0.4] for each training sample. The other key hyperparameters for discriminative region localization were set as follows: the number of top channels k was set to 0.3C, the number of random channels m was set to 0.2C, and the number of channels to erase t was set to 0.5k, where C is the number of channels in the dimension-reduced second-order feature map F′. These settings were consistent across all datasets.

The loss function is the cross-entropy loss, as shown in Eq. (8). Although this study rarely introduces noise during data augmentation, a small weight is still assigned to augmented samples through a series of comparative experiments. In Eq. (8), when β1 and β2 are both set to 0.15, a higher validation accuracy is achieved. (8)L(xraw,xc,xd,y)=L(xraw,y)+β1L(xc,y)+β2L(xd,y)here, xraw,xc,xd represent the original sample, cropped augmented sample, and erased augmented sample, respectively, while y represents the label of the sample. L(xraw,y), L(xc,y), and L(xd,y) represent the loss values of the original sample, cropped augmented sample, and erased augmented sample, respectively. L(xraw,xc,xd,y) represents the total loss value of the network. f represents the feature modulus.

Since normalizing the modulus reduces the value of the cross-entropy loss and slows down the convergence speed of the model, this study adopts the approach of multiplying the modulus by a hyperparameter s to normalize the modulus to a larger value. Here, s is set to 100, as shown in Eq. (9). (9)f~=sf||f||2

To reduce random error, this study trains for 80 epochs on each dataset and conducts five tests. The average of the five test results is taken as the result.