This section details the architecture of the proposed 3D-ERVSNet. As illustrated in Fig. 1, 3D-ERVSNet comprises four key modules: the Forward and Backward Bidirectional Propagation Module (FBBPM), the 3D Convolution Module (3DCM), the Deep Feature Aggregation Module (DFAM), and the Pixel-Upsampling Module (PUM). The FBBPM is specifically designed to extract features in parallel along both forward and backward temporal directions. It consists of three sub-modules: the Optical Flow Alignment Block (OFAB), the Feature Alignment Block (FAB), and the Enhanced Residual Structure (ERS). A 3D convolution layer is applied subsequent to the Backward Propagation Module (BPM). The DFAM then fuses the output features from the Forward Propagation Module (FPM) with those from the 3DCM. The fused result is subsequently fed into the PUM to generate the final high-resolution (HR) video frame.

Recognizing the unique characteristic of the VSR task (i.e., the input comprises image sequences rather than individual frames as in SISR), it is essential to model both the spatial features within each frame and the temporal relationships between frames. Consequently, 3D-ERVSNet incorporates two parallel feature propagation pathways. As depicted in Fig. 1, the Forward Propagation Module (FPM, Ff) and the Backward Propagation Module (BPM, Fb) operate concurrently, processing video sequences in opposite temporal directions. While sharing an identical internal structure, these modules differ solely in their direction of feature propagation. The operation of these modules is mathematically expressed as follows: (1)Of=forward(ILR) (2)Ob=backward(ILR)where forward and backward denote the FPM and BPM modules, respectively. ILR represents the input LR video sequence, and Of and Ob denote the outputs of the corresponding modules. Each propagation module primarily consists of three components including the OFAB, the FAB and the ERS, described in detail below.

Motivated by the proven effectiveness of 3D convolutions for capturing spatio-temporal information in various computer vision tasks, such as face restoration using plug-and-play 3D facial priors [29], we incorporate a 3×3×3 3D convolution layer after the BPM. This module performs implicit spatio-temporal alignment and fusion, formulated as:(7)O3D=3D(Ob)where O3D denotes the output feature map. Since a 3×3×3 convolution kernel reduces the temporal length of the sequence, we replicate the first and last frames of the BPM’s output sequence. This results in an extended feature volume of dimensions (T+2)×C×H×W, effectively padding the sequence temporally. Additionally, spatial padding of size (0,1,1) is applied to maintain consistent spatial dimensions. The 3DCM serves to mitigate inaccuracies arising from unreliable optical flow estimations and effectively addresses occlusion challenges. We strategically apply the 3DCM only after the backward propagation path. The rationale stems from the observation that features processed by the BPM have already aggregated information from future frames, thereby providing a richer temporal context. This context is particularly suitable for the implicit spatio-temporal fusion capabilities of 3D convolutions. This asymmetric design choice [30] is empirically validated to be both effective and computationally more efficient than symmetric application on both paths. Unlike explicit alignment methods, 3D convolution implicitly captures spatio-temporal dependencies between frames. It dynamically leverages reliable information from neighboring frames to compensate for and rectify unreliable regions within the current frame. This implicit mechanism significantly enhances the robustness of the feature representations and consequently improves video restoration quality, particularly in challenging scenarios involving complex motion or occlusions.

To synthesize a more comprehensive feature representation, the DFAM aggregates the deep features generated by the preceding modules. The DFAM consists of a single 1×1 convolutional layer. Its inputs are the output features Of from the FPM and O3D from the 3DCM. The module concatenates these two feature maps along the channel dimension and subsequently adjusts the channel dimensionality through the 1×1 convolution. This aggregation process is defined as:(8)Oc=fusion(Of,O3D)where fusion represents the function of the Deep Feature Aggregation Module, and Oc denotes the fused output feature map.

The Pixel-Upsampling Module (PUM) reconstructs the final high-resolution (HR) image from the aggregated deep features Oc. It employs a dual-path architecture with a residual connection. The primary feature path, denoted as PU1, learns the high-frequency residual details. This path processes Oc through a sequence of two 3×3 convolution layers, each followed by a pixel shuffle operation with a upscaling factor of 2, effectively upsampling the features to the target resolution. A LeakyReLU activation is applied after each convolution. While ReLU outputs zero for negative inputs, LeakyReLU retains a small gradient in such cases. This ensures that the gradient can continue to flow within the network even with a negative input, which is beneficial for model convergence. The secondary residual path, denoted as PU2, provides a coarse, structurally-sound baseline image. It directly upsamples the original low-resolution input ILR to the target resolution using a single bilinear interpolation operation. The full-resolution feature map from PU1 is added element-wise to the full-resolution upsampled image from PU2. This design allows the network to focus on learning the high-frequency residual information needed to transform the coarse bilinear upsampling into a high-quality HR image. This final fusion is described by the equation:(9)Ooutput=PU1(Oc)⊕PU2(ILR)where Ooutput denotes the final high-resolution output frame of the 3D-ERVSNet, PU1 represents the function of the primary up-sampling path processing the DFAM output, PU2 represents the function of the secondary path up-sampling the original LR input frame, and ⨁ denotes element-wise addition.

To ensure experimental fairness, the proposed 3D-ERVSNet model was trained on the REDS dataset [31]. Evaluation was conducted on three benchmark test sets: Vid4 [32], UDM10 [33], and Vim4 (comprising four sequences randomly selected from Vim90k [19]). The REDS dataset is partitioned into a training set (240 videos), a validation set (30 videos), and a test set (30 videos). The Vid4 test set contains four video sequences: ‘calendar’, ‘city’, ‘foliage’, and ‘walk’. The UDM10 dataset includes 10 video sequences, each consisting of 32 consecutive frames. The Vim4 dataset consists of 4 video sequences extracted from Vim90k, each containing 7 consecutive frames. Quantitative performance evaluation employs the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) metrics.

Following the training methodology of LapSRN [34], we employ the Charbonnier loss [35] to optimize the network parameters. This loss function measures the difference between the ground-truth high-resolution (HR) video frames and the predicted HR frames. For effective training, an input sequence length of 15 frames is utilized. Optimization is performed using the Adam optimizer with a cosine annealing learning rate schedule. The initial learning rates are configured as follows: 1×10−4 for the feature extraction components, 2.5×10−5 for the optical flow estimation module (SPyNet, kept frozen), and 2×10−4 for the remaining modules. Training proceeds for a total of 30,000 iterations. Crucially, the weights of the BPM and FAB sub-modules are frozen during the initial 5000 iterations to stabilize early training. The batch size is set to 8. Within each ERB, Kaiming initialization is applied to the convolutional layers, with the initial weights scaled by a factor of 0.1. All experiments focus on the challenging and prevalent case of ×4 video super-resolution [36]. The computational platform comprises an AMD EPYC 7502P Processor (64 cores), an NVIDIA GeForce RTX 3090 GPU, and 128 GB of RAM.

An ablation study is conducted to evaluate the individual contributions of the Enhanced Residual Structure (ERS) and the 3D Convolution Module (3DCM) within the proposed 3D-ERVSNet framework. Tables 1 and 2 present the quantitative results on the UDM10 and Vid4 (BD degradation) datasets, respectively.

The baseline model employs a standard residual structure, lacking both the ERS and 3DCM. As evidenced by Tables 1 and 2, integrating the ERS yields statistically significant improvements in both PSNR and SSIM metrics (e.g., approximately +1.40 dB PSNR on UDM10). This substantial gain underscores the critical role of the ERS in enhancing the feature extraction capability. The results suggest that the ERS successfully facilitates the extraction of richer features while effectively preserving vital shallow texture information present in the video frames. The subsequent integration of the 3DCM further enhances performance consistently across datasets (e.g., approximately +0.46 dB PSNR on Vid4). This improvement confirms the efficacy of the 3DCM in refining spatio-temporal details and implicitly capturing inter-frame dependencies. Consequently, the synergistic combination of both the ERS and 3DCM leads to the most accurate and detailed reconstruction performance.

Table 3 summarizes the quantitative comparison against other methods. The proposed 3D-ERVSNet demonstrates competitive or superior performance across all three benchmark datasets. Notably, it achieves the highest PSNR and SSIM values on the REDS (30.95 dB/0.8822) and Vim4 (32.78 dB/0.8987) datasets. On the challenging Vid4 benchmark with BD degradation, 3D-ERVSNet (27.02 dB/0.8224) also performs favorably compared to strong methods like RBPN (27.12 dB/0.8180) and EDVR-M (27.10 dB/0.8186), achieving comparable PSNR with a higher SSIM score.

Table 4 compares model complexity and inference speed. The proposed 3D-ERVSNet achieves an excellent balance, offering competitive performance (Table 3) with moderate parameter complexity (6.3M parameters, inclusive of the frozen SPyNet) and significantly faster inference time (77 ms per frame) compared to many methods like RBPN (1507 ms) and DUF (974 ms). This efficiency makes 3D-ERVSNet highly practical.

Visual comparisons further substantiate the effectiveness of 3D-ERVSNet. Fig. 3 illustrates that 3D-ERVSNet reconstructs finer texture details on the Vid4 dataset compared to other methods. In the zoomed upper-left region (marked area), our model reconstructs the structural texture of the mural with clear visibility, while competing methods fail to restore these patterns entirely. Similarly, Fig. 4 demonstrates superior sharpness and clarity in the marked region for results on the UDM10 dataset. Upon magnification, our result exhibits the closest resemblance to the HR ground truth in texture lines. Other methods either omit these lines or produce textures that significantly deviate from the HR reference. Fig. 5 highlights the realism achieved by 3D-ERVSNet on the Vim4 dataset, particularly evident in the accurate reconstruction of intricate elements like the door handle within the marked area. Vertical stripes on the truck are faithfully preserved without tilt (unlike the skewed outputs of other models). And our method avoids generating false patterns (e.g., MuCAN erroneously reconstructs the pickup truck’s door handle as a vertical line, while ours matches the GT structure). These qualitative observations align with the quantitative metrics, confirming that 3D-ERVSNet excels in recovering authentic spatial details and robustly modeling temporal information.