The detail description of ResNet50 is formed from five stages of convolutional layers presented in the left three columns of Table 1. The first layer in each stage performs a down-sampling process by striding. We define Sij to denote the ith layer in stage j, where j =1,…,5. The output of Sij of the stage in ResNet50 network is formulated as

(1)(Sij∗fij)(p)=∑a+b=pSij(a)fij(b)where fij represents the filter combined with layer Sij and the domain of p denotes the feature map in Sij.

To increase the resolution of output feature maps without reducing the receptive field of each output unit, dilated convolution is adopted in the final two stages, S4 and S5, in ResNet50 to construct the dilated residual network, which is shown in the right two columns of Table 1.

For S4, we remove the striding in S14 and substitute the convolution operators with dilated convolutions with a dilated factor of 2, which is represented as:

(2)(Si4∗2fi4)(p)=∑a+2b=pSi4(a)fi4(b)where all i≥1. For the layer of S5, the striding in S15 is also removed. And the dilated convolutions with a dilated factor of 4 are applied to compensating for the reduction in the receptive field due to removing subsampling, which can be formulated as

(3)(Si5∗4fi5)(p)=∑a+4b=pSi5(a)fi5(b)where all i≥1.

However, employing dilated convolution may cause gridding artifacts, especially when a feature map has higher-frequency content than the sampling rate of the dilated convolution. Additionally, the max pooling layer after the initial 7 × 7 convolution, which leads to high-amplitude high-frequency activations, will also eventually aggravate gridding artifacts. In this case, we further modify the dilated residual network and replace the maximum pooling with convolution layers. In addition, we add two convolution layers with progressively decreasing dilation to the final group of the network, adopting filters of appropriate frequency, so that gridding artifacts are better eliminated. Compared to ResNet50 which has an output resolution of 7 × 7, the proposed dilated residual network with an output size of 28 × 28 can obtain more spatial information and identify small polyps without wasting unnecessary computational power to improve the classification performance of the network.

With the attention mechanism, the deep learning network can divert attention to the most important areas of an image while ignoring irrelevant parts, which can be treated as a dynamic weight adjustment process according to the features of the input image. To bring performance gain further for our proposed network and avoid high model complexity, we apply ECA [26], a lightweight channel attention module, to all the residual blocks of IDE and ConvE in our network. The residual block structures of IDE Block and ConvE Block with the incorporation of the ECA module are shown in Figs. 2a and 2b, respectively. As an improved attention mechanism based on Squeeze-and-Excitation Networks (SE) [27], ECA can effectively capture cross-channel interaction information without dimensionality reduction. The structure of the ECA module is presented in Fig. 2c.

As illustrated in Fig. 2c, after adopting channel-level GAP to aggregate features without dimensionality reduction, a 1D convolution is implemented to capture local cross-channel interactions between each channel and the corresponding k neighbors. Additionally, the 1D convolution with the kernel size of k can be selected adaptively, determining coverage of local cross-channel interaction. The parameter k is determined as

(4)k=ψ(C)=|log2(C)γ+bγ|oddwhere γ and b are hyperparameters, C is the channel dimension, and |x|odd represents the nearest odd number of x.

In this study, the collected polyp datasets are unbalanced, and the amount of image data in some categories is relatively small, which is prone to over-fitting in the training process of the proposed network. Therefore, we adopt a regularization method DropBlock after convolution layers (ConvE-layer) and residual modules (IDE block) in stages 4 and 5 of the proposed network to improve generalization ability and reduce overfitting, which is shown in Fig. 1b.

DropBlock has two hyperparameters, γ and m, with the former controlling denoting the numbers of features to be dropped and the latter representing the size of the contiguous area to be dropped. During network training, the block size m is fixed as 7, and γ is calculated by the following formula:

(5)γ=(1−keep_prob)×n2m2×(n−m+1)2where keep_prob denotes the probability of keeping an activation unit in traditional dropout, n represents the size of a feature map, and (n−m+1)2 denotes the size of the valid seed region. However, DropBlock with a fixed keep_prob has been found to be ineffective [15]. In this paper, a linear reduction of keep_prob parameter is adopted, that is, with the increase of training steps, keep_prob gradually decreases from 1 to 0.9.

Colonoscopy images and pathological information were collected retrospectively from the Affiliated Hospital of Hebei University, from June 1, 2016 to June 1, 2019. All the specimens were examined by certified pathologists and histologically confirmed. Patients who had been histologically confirmed with conventional adenoma (tubular, villioustublar, villous), sessile serrated polyp/adenoma (SSAP), hyperplastic polyp and normal (no polyps), were included in this study. Only the unamplified images observed in conventional white-light or NBI mode were selected. In this study, insufficiently insufflated colorectal images and unclear images with stool residue, halation, or bleeding were excluded. Finally, we collected 2132 images from 436 patients that contain six types of endoscopic colorectal disease images, which are shown in Table 2.

In order to build the classification models, the above datasets were classified by patients and randomly divided into the training set and test set in a ratio of 4:1. To develop our deep learning network more effectively, the training set was increased by a factor of 4 with data augmentation methods of horizontal flip, vertical flip, noise, and rotation to 6823 images. Considering that the collected colonoscopy images ranged from 424 × 368 to 1920 × 1080 pixels, they were uniformly resized to 224 × 224 pixels according to the network training requirements.

The Polyp-DedNet was pre-trained using the ImageNet Large Scale Visual Recognition Challenge datasets, that is, using transfer learning to improve the classification performance of the network. The network parameters were optimized by the Adam algorithm with an initial learning rate of 0.0001. All programs were run on an Ubuntu 18.04.5 LTS PC with one RTX 2080Ti GPU and Intel (R) Core i7-7800X 3.5-GHz CPU.

Cross-entropy (CE) loss function is often used for deep learning network training. In this paper, to address the problem of training on imbalanced data, we adopted CB sigmoid cross-entropy loss by introducing a weighted factor that was inversely proportional to the effective number of samples. The CB sigmoid cross-entropy loss is formulated as:

(6)CBsigmoid(z,y)=−1−β1−βny∑i=1Clog⁡(11+exp⁡(−zit))where z=[z1,z2,…,zC]T represents the predicted value for all classes from Polyp-DedNet, and C is the total number of polyp types. y is the sample whose category is labeled as y. β indicates the effective sample growth rate, βϵ[0,1).

To evaluate the classification performance of the Polyp-DedNet network, we introduced four metrics including accuracy, recall, precision, and F1 Score [28], which can be calculated by the following formulas:

(7)Accuracy=TP+TNTP+TN+FP+FN

(8)Precision =TPTP+FP

(9)Recall=TPTP+FN

(10)F1Score=2×Recall×PrecisionRecall+Precisionwhere TP, FP, TN and FN denote the numbers of true positives, false positives, true negatives, and false negatives, respectively.

In this study, 5-fold cross-validation was used to analyze the network performance for four- and six-category classifications. The difference between the proposed model Polyp-DedNet and comparative models were calculated for each performance metric using the paired t-test. The p-value of <0.05 was considered to indicate a statistically significant difference between groups.

To evaluate the effect of loss function on data imbalance during training, we compared two specific methods’ performance, including CE loss and CB sigmoid cross-entropy loss. As shown in Fig. 3, while the train loss and the test loss gradually converged to 0.20 ± 0.03 and 0.65 ± 0.03 when training with the proposed Polyp-DedNet with CE loss, they gradually converged to 0.04 ± 0.01 and 0.10 ± 0.01 with CB sigmoid cross-entropy loss. In this case, our Polyp-DedNet had a faster convergence rate and a lower value for final loss with the implementation of CB sigmoid cross-entropy loss for network training compared with the use of CE loss. To a certain extent, the performance degradation caused by data imbalance was alleviated, especially with CB sigmoid cross-entropy loss.

The four- and six-category performance tests on a test set of 426 images were performed on the ResNet50 and Polyp-DedNet, respectively. Results of the five-fold cross-validation for the classifications were listed in Table 3, in which the best for each metric was highlighted. For the four-category classification task, although Polyp-DedNet achieved insignificant increment in recall compared with the baseline ResNet50 (p = 0.068), the remaining metrics including accuracy (89.91 ± 0.92 vs. 88.46 ± 0.79, p = 0.001), precision (82.35 ± 2.71 vs. 79.37 ± 2.28, p = 0.009) and F1-score (82.29 ± 1.75 vs. 79.41 ± 1.73, p = 0.006) were significantly improved. Therefore, the overall performance of Polyp-DedNet for colorectal polyp classification was superior to that of ResNet50. For the six-category classification task, the cross-validation results illustrated the performance of the developed network was significantly improved in all evaluation metrics to varying degrees compared with the baseline network (p < 0.05). The above results confirmed that extracting more spatial features from the input colonoscopy images can facilitate the network’s effect on multi-classification.

In order to further compare the classification ability between ResNet50 and Polyp-DedNet, heat maps of the confusion matrix for the four- and six-category classifications were shown in Fig. 4, in which the row and column represented the actual category and predicted category, respectively.

In the four-classification task, the proposed Polyp-DedNet outperformed ResNet50 in classifying colorectal polyp images of adenomas and normal, with similar performance in classifying hyperplastic polyps and sessile serrated adenoma/polyp (SSAP). In addition, both networks provided more than 90% achievable performance for the classifications of adenomas and normal. In contrast, the performances of classifying other categories of polyps were relatively lower with the fact that most of them were misclassified as adenomas (see Figs. 4a and 4b). It is worth mentioning that the training data for hyperplastic polyps and sessile serrated adenoma/polyp (SSAP) were much less than that of adenomas, and SSAP was more difficult to be distinguished than other polyps in clinical practice. In the six-category classification task, although our proposed framework provided comparable performance for the classifications of normal and villous adenomas with ResNet50, it achieved higher performance consistently for the classifications of hyperplasia, sessile serrated adenoma/polyp (SSAP), tubular adenoma and villoustubular adenoma with the improvements varied from 2% to 6%. In particular, the network Polyp-DedNet correctly classified more than 90% of normal and tubular adenoma images due to its increased focus on high-dimensional spatial information.

We further measured the standalone performance of the networks by the overall and class-specific ROC curves and the area under ROC curve (AUC) of four and six pathological types with an epoch set of 100, respectively. The larger the value of AUC, the better the performance of the network. The metric changes tend to level off after an average of 70 epochs. For the four-category classification, as shown in Figs. 5a and 5b, the Polyp-DedNet was almost at a higher classification level in adenoma, normal and SSAP. The AUCs were as follows: adenoma, 0.9645 vs. 0.9742, normal, 0.9949 vs. 0.9966, SSAP, 0.9576 vs. 0.9587. For the classification of hyperplastic polyps, Polyp-DedNet performed slightly worse than ResNet50 (AUC, 0.9073 vs. 0.9062).

In the six-category classification, the AUCs of Polyp-DedNet were significantly higher than those of ResNet50 for the hyperplastic polyps, tubular, villioustublar and villious adenomas. However, the Polyp-DedNet performed slightly worse for normal and SSAP with lower AUCs than the pre-improved ResNet50. In addition, the improved network Polyp-DedNet showed overall competitive multi-classification performance with a mean AUC of 0.9593.

To display the area of interest in the colonoscopy image more intuitively, representative class activation map (CAM) images of ResNet50 and Polyp-DedNet for multi-classification of polyps were generated, including hyperplastic polyps, (tubular, villioustublar and villious) adenomas and SSAP. As shown in Fig. 6, while both the networks ResNet50 and Polyp-DedNet could accurately localize hyperplastic polyps and villous adenomas (see the column (a) and (c) of Fig. 6), our proposed network Polyp-DedNet captured villioustublar adenomas (see the column (b) of Fig. 6) and tubular adenomas (see the column (e) of Fig. 6) more accurately. It can be also observed that when the point of interest contained distinct categories of targets (i.e., a SSAP indicated by the red arrow and a non-SSAP polyp indicated by the yellow dashed box in the column (d) of Fig. 6), the attended areas of ResNet50 were broadly distributed on both of them, whereas the attention focus of Polyp-DedNet was more precise on the characteristic area of SSAP, thereby extracting pathological features of colorectal polyps more effectively. Therefore, the extracted classification discrimination area of the proposed Polyp-DedNet was more consistent with the cognition of experts, which partially enhanced the applicability of deep learning.

In order to further verify the colorectal polyp classification performance of Polyp-DedNet, we performed comparative experiments with several state-of-the-art methods in terms of accuracy, precision, recall and F1-score, in which all the deep learning networks were built based on pre-trained ImageNet. In order to ensure the fairness of test results comparison, each network was trained under the same condition and tested on the same datasets. The six-category test results from five-fold cross-validation for each network were presented in Table 4. It can be seen that Polyp-DedNet achieved the best performance in terms of selected evaluation indicators, which yielded an accuracy of 85.13% ± 1.10%, a precision of 84.30% ± 2.22%, a recall of 80.91% ± 3.48% and an F1-score of 81.98% ± 2.60%. As shown in Fig. 7, Polyp-DedNet was remarkably superior to MobileNetV3 according to all four indicators (p < 0.01, paired t test). Besides, the precision of Polyp-DedNet was significantly higher than that of EfficientNetV2, and the metrics of recall and F1-score were consistently higher than those of RegNet (p < 0.01). In addition, compared with EfficientNetV2, Polyp-DedNet enhanced in various indicators and achieved a statistically significant improvement in the average precision (1.64%; p < 0.05) of multi-classification. In summary, with the more effective ability of feature extraction, the proposed Polyp-DedNet has excellent recognition ability for colorectal polyps with varying pathological types.