In deep learning, the first method specifically designed for semantic segmentation was FCN [5]. This method differs from traditional approaches that treat semantic segmentation as a region classification problem by framing it as a pixel-level classification problem. Badrinarayanan et al. [6] improved upon FCN by introducing the SegNet. Compared to the FCN model, SegNet uses the corresponding max-pooling layer indices to restore the resolution of feature maps, showing better performance on low-resolution images. DeepLab introduced atrous convolution and atrous spatial pyramid pooling (ASPP) into the segmentation network, effectively expanding the receptive field. Multi-level feature fusion modules were created by PSPNet [7] to handle context information at different scales. Lin et al. [8] employed residual connections and multi-scale fusion techniques in the RefineNet network, and used deconvolution to achieve higher resolution in the output segmentation results.

Early real-time semantic segmentation methods explored many lightweight network architectures, mainly reducing inference costs through channel compression and fast downsampling. For instance, Zhao et al. [9] proposed a novel image cascade network in ICNet, which refines segmentation predictions by utilizing low-resolution semantic information and high-resolution image details. Yu et al. proposed BiSeNetV1 [10] and BiSeNetV2 [11], which combine shallow feature details with deep feature semantics. To reduce time-consuming auxiliary paths, Fan et al. presented STDC [12], a bilateral-branch network based on BiSeNet that encodes spatial information using a guidance module. DDRNet extracts high and low-resolution features independently using a multi-resolution network in order to balance context information during the quick downsampling process. Some real-time semantic segmentation methods also employ transformers to enhance performance. Zhang et al. [13] introduced the self-attention mechanism in the Top-Former, but using transformers on low-resolution feature maps led to lower accuracy. RTformer proposed a more GPU-friendly attention mechanism. SeaFormer [14] is a lightweight transformer model that compresses the spatial dimensions of the input feature map to lower computing cost.

Knowledge distillation is a lightweight method that maintains high model performance, helping achieve high-performance real-time semantic segmentation. Some logic-based knowledge distillation methods have been improved through model ensembles, contrastive learning, and other techniques. Touvron et al. [15] introduced a novel distillation process based on distillation tokens when training transformer students. To narrow the large capacity gap between teacher and student models, Huang et al. [16] proposed relaxing the precise matching based on KL divergence. Furthermore, Romero et al. [17] first proposed a hint-based distillation method, in which the student features are projected into the teacher’s feature space through convolutional layers. To adapt to the characteristics of dense prediction tasks, Xie et al. [18] computed the Euclidean distance between the central pixel and its 8-neighboring pixels, constructing a local similarity graph. Liu et al. [19] suggested a method to capture structured information between pixels and global correlation. To focus the learning process on intra-class feature variations, Wang et al. [20] employed the cosine distance between pixel features and corresponding class prototypes to learn structural knowledge. Shu et al. [21] proposed a distillation loss function that pays more attention to the most salient regions across channels. Currently, knowledge distillation is widely applied in semantic segmentation tasks [22–25]. TriKD [26] offers a three-view knowledge distillation framework for semi-supervised semantic segmentation. For few-shot unsupervised semantic segmentation tasks, Li et al. [27] designed a semi-supervised semantic segmentation framework. Xu et al. [28] developed a single-branch real-time semantic segmentation model.

As illustrated in Fig. 2, the overall architecture comprises a CNN backbone for efficient inference and a Transformer branch for semantic feature extraction. Within the CNN backbone, the attention mechanism embedded in the FCB module mitigates the semantic gap between the two branches. SIDM serves as a bridge between the two branches, enabling the backbone network to extract semantic information from the pre-trained Transformer. The decoder of the backbone network incorporates a DASP module for multi-scale feature extraction, followed by a segmentation head. Specifically, features from the fourth stage are first fed into the DASP to capture multi-scale features, which are then fused with features from the second stage to obtain rich contextual information. The resulting features are subsequently passed to the segmentation head, and finally processed by a 1×1 convolutional classifier to obtain accurate segmentation results.

In the designed network architecture, both an efficient CNN and a Transformer capable of extracting high-quality contextual information are used. However, structural differences between the networks can hinder knowledge distillation, especially when the teacher’s features exceed the processing capacity of the student [15].

As seen in Fig. 3 (right), an FCB is designed to reduce the discrepancy between the information learned by the Transformer branch and the CNN branch. In FCB, the DEC Attention module employs convolution operations to compute attention, enabling the CNN branch to capture Transformer-like features. The structure of FCB is derived from the typical Transformer encoder structure, and it can be described as follows: (1)xm=Norm(xi+DECAttention(xi)),xo=Norm(xm+FFN(xm)),

where Norm(⋅) denotes batch normalization, and xi, xm, and xo represent the input features, intermediate features, and output features, respectively.

The proposed Detail Enhancing Convolutional Attention (DEC) method is designed for real-time semantic segmentation, which requires low latency and efficient feature extraction. Its specific structure is shown in Fig. 3 (left). It linearly combines the channel values of each pixel using convolution kernels, operating only along the channel dimension without involving spatial position interactions. For the input feature map X∈RH×W×Cin and convolution kernel W∈R1×1×Cin×Cout, Y∈RH×W×Cout is the output of pixel-by-pixel convolution. The detailed calculation process can be described as follows: (2)Yi,j,Cout=∑Cin=1CinXi,j,Cin⋅W1,1,Cin,Cout,

where xi,j,Cin denotes the value of the input feature at position (i,j) with channel index Cin, H and W represent the height and width of the input feature, and Cin, Cout denote the numbers of channels in the input and output feature maps, respectively.

The calculation of attention is highly sensitive to the size of the input feature map. To address this, Grouped Double Normalization (GDN) is adopted to compute attention weights in different ways along two dimensions. Specifically, the softmax is applied across the spatial dimension to generate spatial attention weights, and grouped L2 regularization is employed along the channel dimension. This approach processes parameters in groups to improve efficiency while balancing the weights across different groups, preventing instability caused by excessively large weights in a single group. It increases the diversity among the attention maps of different query points, and thus captures richer semantic representation.

To reduce computational complexity, convolution kernels of 1×k and k×1 are used in the horizontal and vertical directions to replace the standard k×k convolution. After that, attention is then computed separately in both directions. While this approach is efficient, it lacks local information. Therefore, a detail enhancement branch is introduced in the attention mechanism. The initially computed q, k, and v are concatenated along the channel dimension, and a 3×3 depthwise convolution is applied to aggregate auxiliary local details. The output is then processed by a linear projection with an activation function and batch normalization. This process compresses the channel dimension and produces the detail enhancement weights. Finally, the detail-enhanced features are fused with the attention features, and the process can be described as follows: (3)y=(σ(X⊗Kh)⊗KhT+σ(X⊗Kv)⊗KvT)∗Fd,

where X∈RC×H×W, K∈RC×H×k×k and KT∈RN×C×k×k represent the input image, query, and key, respectively, and C, H, W denote the number of channels, height, and width of the feature map. N represents the number of learnable parameters, and k denotes the size of the convolution kernel. σ refers to the grouped double normalization. Fd represents the detailed features obtained from the detail enhancement branch.

In the FCB, computationally expensive matrix operations for attention are replaced by per-pixel convolutions, which preserve the original spatial structure and are beneficial for feature extraction. DEC Attention computes attention using stripe convolutions in both horizontal and vertical directions, which reduces computational cost compared to standard convolutions. By adding local information to the extracted attention features, the detail enhancement branch may enhance the model’s overall performance. A more efficient FFN (Fig. 3, right) is employed, in which depthwise separable convolutions are used to perform convolution independently on each channel, significantly reducing computational cost and better preserving channel-wise independence. In FFN, residual connections are incorporated to mitigate the vanishing gradient problem [29].

Previous knowledge distillation methods [30] have mostly been used for learning between similar models, whereas our model needs to learn from two different types of models. To enable the lightweight CNN backbone to efficiently extract features, SIDM is designed to extract semantic information from a pre-trained Transformer branch. As shown in Fig. 2, this module can be divided into intermediate feature alignment and logits alignment.

Intermediate Feature Alignment: During knowledge distillation, differences in model structures are often reflected in their feature spaces, with features preserved in different latent spaces [4]. The use of basic similarity measurement functions, such as Mean Squared Error (MSE) [15] loss, for information extraction does not ensure effective alignment of learned features and may negatively impact model performance. Relying on the designed Feature Conversion Block (FCB), attention is computed via convolution operations, allowing features from different models to be represented similarly. This results in intermediate features with structures similar to those of the Transformer features. During feature alignment, the student features derived from the CNN are first projected onto the teacher feature dimensions of the Transformer. Adjust the resolution using upsampling or downsampling to avoid directly aligning the features. Then, adjust the CNN student features to ensure their statistical properties are consistent with the teacher features. Finally, the semantic loss is computed between the adjusted CNN student features and the Transformer teacher features. To ensure that the student model focuses on semantic rather than spatial information, CWD Loss is employed as the alignment loss, with its computation process can be summarized as follows: (4)ϕ(xc)=exp⁡(xc,i𝒯)∑i=1W⋅Hexp⁡(xc,i𝒯),where c=1,2,...,C is the channel index, i=1,2,... is the index of the channel space, and xT, xS represent the feature maps of the Transformer teacher and CNN student, respectively. 𝒯 is a hyperparameter called temperature; the larger its value, the softer the probability distribution, allowing a wider region of each channel to be considered. ϕ computes a channel-level probability distribution from the feature activation map, mitigating the impact of scale differences between the two models.

The process of computing the channel discrepancy between the student and teacher models can be formulated as: (5)φ(xT,xS)=𝒯2C∑c=1C∑i=1H⋅Wϕ(xTc,i)⋅log⁡[ϕ(xTc,i)ϕ(xSc,i)],

where φ(⋅) is used to evaluate the difference between the two networks. To minimize φ(⋅), as the value of ϕ(xTc,i) increases, the the value of ϕ(xSc,i) also increases, and conversely, when ϕ(xTc,i) decreases, the influence of ϕ(xSc,i) on KL divergence becomes smaller. This asymmetric learning mechanism makes the KL divergence focus more on the foreground salient regions with high probabilities predicted by the teacher model, while reducing attention to the background areas.

Logits Alignment: When features are passed through the segmentation head to obtain logits, they do not contain model-specific information like intermediate features do. Therefore, different models can be aligned directly in the logits space. However, despite sharing the same learning objectives in the logits space, the different inductive biases of the models often lead to different results. Their outcomes are influenced by these biases, resulting in different prediction distributions. For example, CNN excels at capturing shared local information across different categories, while Transformers are better at learning global features using attention mechanisms. Given these learning biases, OFA Loss is used to align different models in the logits space. It employs an adaptive target information enhancement method, which adds a term related only to the target class, guiding the student model to learn from a more confident teacher. This process can be formulated as: (6)ℒKD=−log⁡pc^S−Ec∼y[pcTlog⁡pcS]=−(1+pc^T)log⁡pc^S−Ec∼y/{c^}[pcTlog⁡pcS],

where c and c^ represent the target class and the predicted class, respectively, and pc^S, pc^T denote the predicted probability distributions of the student model and teacher model, respectively.

To regulate the relationship between the teacher and student model distributions and encourage better alignment of the target class in the student model, a parameter θ is added to the term 1+pc^T to enhance the target class. The distillation loss function can be described as follows: (7)ℒOFA=−(1+pc^T)θlog⁡pc^S−Ec∼y/{c^}[pcTlog⁡pcS]=ℒKD+−(∑k=1θ(θk)(pc^T)k−pc^T)log⁡pc^S,

where the term added with the parameter θ in ℒOFA is a positive term that is only related to the target class. If the teacher model is confident in the target class, the higher-order term with the parameter θ decays slowly. Conversely, if the teacher model is less confident, the decay accelerates to hinder the learning of the target class.

By adjusting this parameter, adaptive enhancement of the target class information learning is achieved, mitigating the influence of soft labels when the teacher model provides suboptimal predictions. This enables models with different structures to discount biases in learning capabilities, thereby enhancing the overall distillation effect.

Accurate segmentation of street scene images requires balancing features across multiple scales. This allows the model to simultaneously recognize both large and small objects while enhancing its ability to perceive objects of varying sizes. To achieve this, a novel DASP is proposed to efficiently and accurately extract features at different scales. Its structure is shown in Fig. 4.

To address the conflict between multi-scale inference and full-resolution dense prediction, a common approach [3] is to obtain a global view through downsampling layers and use repeated upsampling to restore lost resolution. However, this method leads to a decrease in resolution, necessitating upsampling to restore resolution when concatenating features at different scales. The extensive use of upsampling operations significantly increases the computational cost. Moreover, the information and object lost during the downsampling process cannot be fully recovered through upsampling, which causes suboptimal performance in semantic segmentation tasks. Atrous convolutions can expand the receptive field without reducing resolution, thereby eliminating the need for costly upsampling operations. Therefore, atrous convolutions with different dilation rates are used instead of standard convolutions to obtain multi-scale features in this method. Three dilated convolution layers with kernel size 3×3 are assigned dilation rates of r=[6,12,18], following the empirically effective configuration in prior works like DeepLab [4]. This design also draws on the principle of Hybrid Dilated Convolution [31], which avoids using identical dilation rates across layers to ensure a complete receptive field. In multiple layers using atrous convolutions, neighboring pixels across layers are convolved from mutually independent subsets, leading to a lack of dependency between them. To mitigate this, a deep aggregation approach is adopted, which can be formulated as: (8)fi={C1×1(x),i=1;C3×3(D3×3(x)+fi−1),1<i<n;C3×3(U(C1×1(Pglobal(x))+fi−1)),i=n,

where by taking x as input, the features of each layer can be represented as fi. C1×1, C1×1 represents a 1×1 convolution, C3×3 represents a 3×3 convolution, D3×3 denotes a 3×3 atrous convolution, U represents the upsampling operation, and Pglobal stands for global adaptive average pooling. n represents the number of feature extraction layers. To enable the network to capture the overall feature of each channel from a global perspective, a 1×1 adaptive average pooling layer is also employed.

After multi-scale feature extraction, adjacent atrous convolution layers that lack correlation are concatenated and passed through a 3×3 standard convolution layer to extract more local information. This process produces multiple feature maps that are both correlated and of different scales. These feature maps are then concatenated and passed through a 1×1 convolution layer to compress the channels to the expected dimension. Additionally, to mitigate gradient vanishing and explosion, a skip connection is introduced to retain the semantic information from the input before entering the module. Although our module employs multiple convolution layers and complex feature fusion methods, the input resolution of the DASP is only 1/32 of the original image resolution. Even at an input size of 1024×1024, the largest feature map resolution is only 32×32, indicating that DASP imposes a limited impact on inference speed.

Cityscapes is a dataset that focuses on analyzing street scenes. It is divided into three parts: training, validation, and test sets, containing 2975, 500, and 1525 images, respectively. We adopt 19 common categories (such as roads, cars, and pedestrians) for the semantic segmentation task. For model training, AdamW is chosen as the optimizer, with an initial learning rate set to 0.0004 and a weight decay of 0.0125. A poly learning rate strategy with a power of 0.9 is used to reduce the learning rate, and linear warm-up is applied at the beginning of training. The random scaling range is set between 0.25 and 1.5, and random cropping of sizes 1024×512 or 1024×1024 is applied to both the images and their corresponding ground truth annotations. Additionally, images are randomly flipped horizontally with a probability of 0.5.

CamVid is a dataset designed for street scene understanding, including categories such as roads, cars, bicycles, and others. It contains 701 densely annotated frames with a resolution of 960×720. The images are split into 367 training, 101 validation, and 233 test images. The training is conducted on the training and validation datasets. During training, the initial learning rate is set to 0.001, and the images are randomly cropped to 960×720. All other training configurations are kept consistent with those for Cityscapes.

ADE20K is a scene parsing dataset containing 150 semantic categories across a wide range of indoor and outdoor environments, such as buildings, furniture, animals, and roads. It is divided into 20K, 2K, and 3K images for training, validation, and testing. The initial learning rate is set to 0.0005, weight decay is set to 0.01. Images are randomly cropped to 512×512 for data augmentation. The remaining training settings follow those used for Cityscapes.

To validate the effectiveness of the method, a base model of comparable size to RTFormer-B/DDRNet-23 called BSDNet-B is constructed, along with a smaller variant called BSDNet-S. The CNN backbone is first pre-trained on ImageNet, with SegFormer chosen as the transformer branch, followed by fine-tuning the model on semantic segmentation datasets. All our models use Cross-Entropy Loss (CE Loss) [38] to compute the loss between predictions and ground truth.

For performance evaluation, Mean Intersection over Union (mIoU) and Frames Per Second (FPS) are adopted as metrics to evaluate accuracy and inference speed. Experiments are conducted on an NVIDIA A6000 with 48 GB of memory and an Intel® Xeon® Gold 6226R CPU @ 2.9 GHz, using Python 3.8 and PyTorch 1.11.0. To ensure fair comparisons, the inference speed of all proposed methods is measured on an NVIDIA A6000, with the FPS values reported based on identical input resolutions.

Results on Cityscapes: As shown in Table 1, underlining indicates the best mIoU result, while bold formatting highlights our best mIoU and FPS scores. BSDNet-B-Seg100 achieves 81.7% mIoU at 70.6 FPS, representing our best performance on Cityscapes. The FPS improvement primarily stems from the dilated convolutions in DASP, which effectively reduce computational complexity. Meanwhile, the semantic information captured by the Transformer branch significantly enhances BSDNet’s accuracy. Additionally, Fig. 5 presents visualization results on the Cityscapes dataset. Compared to DDRNet and SCTNet, BSDNet not only provides more accurate predictions for large-area categories such as road and vegetation (in yellow boxes) but also preserves finer details for small objects like traffic lights and traffic signs (in white boxes). This demonstrates that BSDNet effectively captures high-quality long-range context while retaining fine details of small-scale objects.

Results on CamVid: Due to the lower pixel resolution in CamVid, the inference speed is generally higher than on Cityscapes. The results on the dataset are shown in Table 2. With an input resolution of 720×960, BSDNet-B achieves the highest mIoU of 84.7% at 135.1 FPS, outperforming RTFormer-B by 2.2%. Although DDRNet-23-S achieves the second-highest FPS at 253.0 FPS, it sacrifices segmentation accuracy by omitting the pretraining process to achieve higher inference speed. This further demonstrates that our method strikes a better balance between speed and accuracy.

Results on ADE20K: To further demonstrate the generalization ability and effectiveness of the BSDNet method, experiments are conducted on the ADE20K. As shown in Table 3, BSDNet-B achieved the best 44.6% mIoU and 180.7 FPS. The result indicates that BSDNet not only performs excellently in street scenes but also has outstanding generalization ability in broader scene types. Furthermore, while the mIoU of BSDNet is close to that of SeaFormer-B, its 189.4 FPS is twice as high as that of SeaFormer-B. This demonstrates that BSDNet has excellent real-time performance. Visualization results on ADE20K are presented in Fig. 6. Compared to SCTNet-B, BSDNet-B yields more accurate segmentation along object boundaries and more coherent predictions for large-area objects, further confirming its generalization capability in complex scenes.

This part verifies the effectiveness of FCB, SIDM, and DASP, and conducts ablation experiments on the proposed modules.