Classical image processing methods include histogram equalization [3–5,26,27] and gamma correction [6–8]. Cheng et al. [3] proposed a multi-peak generalized histogram equalization method that improves global histogram equalization by using multi-peak histogram equalization combined with local information. Lee et al. [4] proposed a novel contrast enhancement algorithm based on the layered difference representation of 2D histograms to enhance image contrast by amplifying the gray-level differences between adjacent pixels. Wang et al. [6] proposed a new method combining dynamic contrast ratio enhancement and inverse gamma correction for alternating current plasma display panel (AC PDP), and both are realized simultaneously. Huang et al. [7] proposed an automatic transformation technique is presented which improves the brightness of dimmed images via gamma correction and luminance pixel probability distribution and uses temporal information for video enhancement to reduce computational complexity. Rahman et al. [8] proposed an adaptive gamma correction method where parameters are set dynamically based on image information to appropriately enhance the contrast of the image. These methods enhance images by adjusting brightness and contrast. These methods are simple and highly interpretable. However, they do not account for complex real-world lighting scenarios. As a result, enhanced images often lack naturalness and realism.

According to the Retinex theory [9], light illuminates different object surfaces, and objects with different materials have distinct reflective properties. These reflections form images on the retina. The Retinex theory decomposes an image into illumination and reflectance components. It enhances low-light images by adjusting the illumination component [28–32]. Fu et al. [28] proposed a fusion-based method for enhancing weakly illuminated images using multiple techniques, including decomposing, deriving, designing, fusing, and compensating to obtain an enhanced image for different weak illumination conditions. Fu et al. [29] proposed a weighted variational model to estimate the reflectance and illumination from an observed image. Guo et al. [30] proposed a simple yet effective low-light image enhancement (LIME) method. Wang et al. [32] made three major contributions including proposing a lightness-order-error measure, a bright-pass filter for image decomposition and a bi-log transformation for mapping illumination. However, these methods require manual setting of illumination priors. Inaccurate priors can lead to artifacts and color distortions in the enhanced results. Additionally, traditional methods often neglect the impact of noise. Simple brightness enhancement can retain and amplify noise.

In 2017, Lore et al. [33] proposed a deep learning-based low-light image enhancement algorithm. Convolutional neural network (CNN) methods [33–37] have been widely applied in low-light image enhancement. Hong et al. [34] proposed a novel unsupervised low-light image enhancement network named LE-GAN based on generative adversarial networks and trained with unpaired low-light images. Lore et al. [33] proposed a deep autoencoder-based approach to identify signal features from low-light images and adaptively brighten images without over-amplifying lighter parts in high dynamic range images. Sharma et al. [35] proposed a method involving estimating the camera response function, decomposing the linearized image into LF and HF feature maps, processing them separately, and combining them to generate an output with increased dynamic range and suppressed light effects. Jiang et al. [36] introduced a highly effective unsupervised generative adversarial network that can be trained without low-light image pairs and generalizes well on real-world test images. Wei et al. [38] and subsequent studies [20,39] combined deep learning methods with the Retinex theory. They adjusted and optimized the Retinex model parameters through convolutional networks. These studies showed improvements over traditional cognitive methods. However, these methods often employ multi-stage training processes. This can be cumbersome. In 2019, Wang et al. [12] proposed a single-stage CNN method to directly predict illumination maps. However, this approach can cause color distortions and noise amplification while enhancing the brightness of low-light images. CNN-based methods still have limitations in expressing long-range spatial dependencies.

In 2017, Vaswani et al. [40] presented the Transformer, a neural network architecture relying on self-attention to process sequential data rather than traditional recurrent or convolutional layers. Vision Transformer converts images into sequence data by splitting them into patches and mapping each patch to a vector for processing. Recently, Vision Transformers and their variants have been applied to low-light image enhancement. In 2022, Xu et al. [41] proposed SNR-Net, a CNN-Transformer hybrid model for low-light image enhancement. Due to the computational complexity of Transformers being proportional to the square of the spatial size, SNR-Net integrates a single global Transformer only at the lowest resolution of the U-shaped network. In 2023, Retinexformer [14], based on the Retinex theory, designed a single-stage Transformer network. It concatenates the height and width of the image into HW tokens and performs self-attention calculations in the channel direction. Spatial position information is extracted through two layers of 3 × 3 convolutional networks. Retinexformer achieved SOTA results that year. However, there is still room for improvement in extracting long-range spatial dependency features. The application of deep learning methods has significantly improved low-light image enhancement. However, further research is needed to enhance the extraction of long-range spatial dependency features under acceptable computational complexity.

The Retinex image enhancement algorithm decomposes a low-light image I∈RH×W×3. It splits it into a reflection component R∈RH×W×3 and an illumination component L∈LH×W, as follows: (1)I=R⊙L

The reflection component R expresses the inherent reflective properties of objects. The illumination component L describes the distribution of light in the scene. In real low-light scenarios, image acquisition often includes noise and artifacts. Reflection disturbances are expressed as R^∈RH×W×3. The illumination distribution is uneven, with weaker illumination in shadowed areas or multiple light sources in low-light environments. Illumination disturbances are expressed as L^∈LH×W. Considering the multi-source light field in real scenes, the illumination component can be decomposed as L=∑i=1NLi. The illumination disturbance can be decomposed as L^=∑i=1NLi^. By introducing reflection disturbances and multi-source light field disturbances, the low-light enhancement function can be expressed as: (2)I=(∑i=1NLi+∑i=1NL^i)⊙(R+R^)=∑i=1NLi⊙R+∑i=1NL^i⊙R+(∑i=1NLi+∑i=1NL^i)⊙R^

Multiplying both sides of the equation by the illumination map L¯=(∑i=1NLi)−1, we get: (3)L¯⊙I=R+∑i=1NL^i⊙L¯⊙R+(∑i=1NLi+∑i=1NL^i)⊙L¯⊙R^

In this equation, L¯⊙I is the illuminated image, denoted as Ilu. R is the normally exposed image. ∑i=1NL^i⊙L¯⊙R is the color distortion part after illumination. (∑i=1NLi+∑i=1NL^i)⊙L¯⊙R^ is the noise and artifacts enhanced during illumination.

In the implementation of [14], each pixel of the input image I is averaged along the channel dimension to obtain the illumination prior Lp. The input image I and Lp are concatenated into a four-channel input. A 5 × 5 convolution outputs the 40-channel light-up features Llu. A 1×1 convolution converts the 40 channels into three channels, generating the light-up map L¯=(∑i=1NLi)−1. The 40-channel light-up features Flu describe various illumination characteristics. However, a 5 × 5 convolution kernel is insufficient to extract the spatial distribution features of multiple light sources.

To better extract multi-source distribution features, the convolution network design should consider two aspects. These are expressing long-range spatial semantic relationships and expressing semantic weights between pixels. The number of convolution layers and the convolution kernel size are crucial to achieving these goals. Multi-scale dilated convolution, with different dilation coefficients, can expand the receptive field. This method controls computational complexity and captures multi-scale contextual information. The HDC [37] rule should be followed in designing dilated convolutions. This ensures continuous coverage of the focus area and avoids holes or missing edges. The HDC rules are as follows: 1.

The maximum distance between two non-zero elements in the second layer should be less than the size of the convolution kernel of that layer.

To balance computational complexity, the convolution kernel size is set to 3. This follows the HDC rule, where the maximum distance between non-zero elements in the second layer is less than 3. We use six layers of convolution with a sawtooth dilation coefficient pattern r = 9,3,1,9,3,1. This generates a convolution receptive field of size 53 × 53.

When the dilation coefficient r = 9, as shown in Fig. 2a. When when r = 9,3, as shown in Fig. 2b, the maximum distance between non-zero elements is less than 3. When r = 9,3,1, as shown in Fig. 2c, the convolution is equivalent to a normal convolution with a kernel size of 27 × 27. When r = 9,3,1,9, as shown in Fig. 2d. When r = 9,3,1,9,3, as shown in Fig. 2e. When r = 9,3,1,9,3,1, as shown in Fig. 2f, the semantic weights between pixels decrease with increasing distance from the central pixel.

Eq. (3) shows that spatial weight disturbances of multi-source illumination on the enhanced image cause noise and color distortion. To address this issue, a three-scale Unet++ structure is designed, as shown in Fig. 1b. This structure uses multi-level skip connections. It can extract and fuse more semantic information at different levels. This reduces information loss and improves segmentation accuracy. It better expresses the spatial distribution features of multi-source illumination.

As shown in Fig. 1b, the IFU is designed as a three-layer Unet++ [42] structure. Unet++ is a deeply supervised encoder-decoder network. The encoder and decoder sub-networks are connected through a series of nested and dense skip pathways. Compared with the Unet structure, it has enhanced feature extraction ability and better performance. In the encoding part, the illuminated image Ilu undergoes a 3 × 3 convolution with a stride of 2, an IFAB encoding, a 4 × 4 convolution with a stride of 2 for down-sampling, two IFAB encodings, and another 4 × 4 convolution with a stride of 2 to generate multi-scale encoded features Fi∈RH2i×W2i×2iC where i = 0,1,2. Subsequently, F2 undergoes two IFAB encodings. The decoding part features a symmetric up-sampling branch, with up-sampling using 2 × 2 deconv after two IFAB encodings of F1 and F2, adding skip connections at the same level to reduce information loss during encoding and decoding. Finally, the decoding part outputs the residual image Ire, enhancing the illuminated image through Ien=Ilu+Ire.

Fig. 3 shows the process flow of obtaining output features by fusing Light-up Feature and Input Feature through the IFAB. Fig. 3a shows the structure of the IFAB module, which consists of a Layer Normalization (LN) layer, an Illumination-Fused Dual Self-Attention (IF-DSA) module, and a Feed-Forward Neural Network (FNN). The structure of the IF-DSA module is shown in Fig. 3b.

By observing the input image, it is clear that different regions are illuminated by different light sources. Therefore, it is necessary to extract semantic information from different regions of the image to encode the illumination features and achieve brightness enhancement. Retinexformer [14] reshapes the input features Fin∈RH×W×C into X∈RHW×C and uses three bias-free fully connected layers to linearly map X to Q=XWQT,K=XWKTandV=XWVT∈RHW×C (WQ,WK,WV are learnable parameters of the linear mapping layers, and T represents matrix transpose). The input light-up feature Flu is reshaped into Y∈RHW×C, then Q,K,VandY are each split into k heads:Q=[Q1,Q2,…,Qk],K=[K1,K2,…,Kk],V=[V1,V2,…,Vk],Y=[Y1,Y2,…,Yk],whereQi,Ki,Vi,Yi∈RHW×dk,dk=Ck,andi=1,2,…,k. The channel self-attention for each head is represented as: (4)Channel_Attention(Qi,Ki,Vi,Yi)=(Yi⊙Vi)⊙softmax(KiTQi(α_ci))where (α_ci)∈Rdk×dk is a learnable adaptive scaling parameter matrix. The spatial context information is extracted using two 3 × 3 convolutions to obtain the spatial position information P∈RC×HW, which is then connected to the channel multi-head self-attention. Finally, after reconstruction, the output feature Fout∈RH×W×C is obtained.

Retinexformer [14] cleverly compresses the spatial HW dimensions in the self-attention computation, making the computational complexity of the attention calculation only the square of the single-head channel size. This ensures that the computational complexity remains within an acceptable range while effectively obtaining channel attention to express different light-up features. However, using only two 3 × 3 convolutions to extract spatial semantic information is insufficient to express long-range dependencies in the space. Directly performing self-attention calculations on the spatial dimensions would result in a computational and memory resource overhead proportional to the square of the spatial size H×W, which is undeniably substantial. Therefore, expressing long-range dependencies in space while controlling computational complexity is a worthwhile research problem.

Retinexformer+ reshapes the input features Fin∈RH×W×C into X∈RC×H×W. It uses bias-free fully connected layers to linearly map X to Q=XWQT and V=XWVT (WQ and WV are learnable parameters of the linear mapping layers, and T represents matrix transpose). After X undergoes 3 × 3 convolution operations with dilation coefficients r = 9,3,1,9,3,1, KC×H×W is obtained. Q and K are concatenated, followed by two layers of 1 × 1 convolutions for fusion, achieving spatial position self-attention calculation. The input light-up feature Flu is reshaped into Y∈RHW×C, and the spatial self-attention calculation formula is expressed as: (5)Spatial_Attention(Q,K,V,Y)=(Y⊙V)⊙softmax(conv(K)⊙Qα_s)where α_s∈RH×W is a learnable adaptive scaling parameter matrix. Finally, the channel self-attention in Retinexformer [14] is reconstructed into Channel_Attention∈RC×H×W and connected to the spatial self-attention Spatial_Attention∈RC×H×W, ultimately reconstructing the output feature Fout∈RH×W×C.

The computational complexity of the IF-DSA module includes the complexity of convolution operations and pointwise multiplication operations between matrices. The computational complexity for six convolution operations with a kernel size of 3 × 3 is O(Conv)=H×W×C2×32×6=54HWC2. The complexity for pointwise multiplication operations between matrices is O(Pointwise_Multiply)=HWC. Therefore, the complexity of the IF-DSA module can be expressed as: (6)O(IF−DSA)=O(Conv)+O(Pointwise_Multiply)=(54C2+C)HW

In contrast, the computational complexity of traditional spatial self-attention methods, like the global MSA used in SNR-Net, is: (7)O(G−MSA)=2C(HW)2

Comparing Eqs. (6) and (7) show that the IF-DSA module expresses long-range semantic relationships in space. It reduces the computational complexity of spatial self-attention methods from the square of the spatial size to a linear function of the spatial size.

We conducted experiments on the LOL dataset. The LOL dataset is divided into LOL_v1 [38] and LOL_v2 [19]. LOL_v1 contains 485 pairs of training data. It contains 15 pairs of testing data. LOL_v2 is further divided into LOL_v2_real and LOL_v2_synthetic. LOL_v2_real consists of real-scene photographs, containing 689 pairs of training data and 100 pairs of testing data. LOL_v2_synthetic consists of synthetic data. It contains 900 pairs of training data and 100 pairs of testing data. Each pair of data in all datasets includes one standard reference image. Each pair also includes one corresponding low-light image.

We implemented the Retinexformer+ model based on PyTorch. Training and testing were conducted on a Linux server equipped with a 3090 24 GB GPU. The system environment included CUDA 11.8, Python 3.7, and PyTorch 1.13. During training, the data size for LOL_v1 and LOL_v2_synthetic was set to 128 × 128, with a batch size of 8. For LOL_v2_real, the data size was 256 × 256, with a batch size of 8. We used random rotation and flipping to augment the training data. The training was performed using the Adam optimizer, with momentum terms of 0.9 and control parameters of 0.999. The aim was to minimize the Mean Absolute Error (MAE) between the enhanced image and the ground truth image. The initial learning rate was set to 2 × 10⁻⁴. It was gradually decreased to 1 × 10⁻⁶ using a cosine annealing strategy.

In the quantitative evaluation, we used Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Root Mean Square Error (RMSE) as metrics. Higher PSNR indicates better image enhancement. Higher SSIM indicates better retention of high-frequency details and structures. Lower RMSE indicates better model performance. As shown in Table 1, our method improves the PSNR by 1.15 on the LOL_v1 dataset. Our method improves the PSNR by 0.42 on the LOL_v2_real dataset. The RMSE decreases by 0.39 on the LOL_v1 dataset. The RMSE decreases by 0.18 on the LOL_ v2_real dataset.

Table 1 compares the evaluation metrics of various supervised and unsupervised state-of-the-art (SOTA) methods. The comparison is based on the publicly available LOL_v1 and LOL_v2_real datasets. The data in the table were directly quoted from these papers.This demonstrates the superior performance of our method.

Our method has FLOPs and Params that are 3.31 times and 2.20 times larger than Retinexformer, and 1.21 times and 0.77 times larger than RetinexMamba. The computational complexity and parameter size are acceptable.

Figs. 4 and 5 show the qualitative comparison results of Retinexformer+ with other SOTA algorithms on the LOL_v1 and LOL_v2_real datasets, respectively. Upon zooming in, it can be observed that the images processed by the Retinex method exhibit significant noise and artifacts. KinD and Uretinex-Net show overexposure and underexposure in different regions. Retinexformer+ restores the color of the stapler more realistically than the Retinexformer and Retinexmamba methods. Furthermore, to visually demonstrate the effects of our method, we compared the HSV color space images corresponding to the images on the LOL_v1 and LOL_v2_real datasets in Fig. 6. It can be observed that the color distribution of the images processed by our method is closer to the Ground Truth images.

In order to evaluate the preservation of semantic information by enhancement methods, we followed the evaluation method in References [48,49]. We conducted brightness enhancement on underexposed images from real scenes. Subsequently, we verified the effects using existing object detection models. The Exdark dataset, containing 7363 underexposed images with annotations for 12 object categories, was chosen for the experiment. After applying the brightness enhancement method described in Table 2, we used the recommended weights of the YOLOv3 algorithm to detect the objects. The COCO dataset provides annotations for 80 different object categories found in real scenes. All 12 object categories from Exdark have corresponding types in COCO. We only needed to change the annotations for “People” and “Table” to “Person” and “Dining Table” in COCO. Finally, we evaluated the performance using the mAP metric.

The results in Table 2 show that images enhanced by our method achieved the highest average precision: 63.67 AP in the YOLOv3 model. This was a 0.56 AP improvement over the second highest. Furthermore, our method also had the best results in detecting categories such as Boat, Bottle, Car, Cup, Dog, Person, and Dining Table. It is worth noting that the detection accuracy for Retinex and KinD enhanced images did not improve.

Fig. 7 shows target detection results of original and enhanced images. Our method suppresses noise and has generalization for real scene image enhancement in Exdark dataset. Using the recommended weights of YOLOv3 to perform target detection on pictures, original image has high detection confidence but missed detections. Retinex-enhanced image reduces confidence and has false detections. The KinD-enhanced image detects a car on the right side, but confidence decreases. Uretinex-Net misdetects car as person. Retinexformer slightly reduces person detection confidence. Retinexmamba significantly reduces person detection confidence. Our method reduces missed and false detection rates while maintaining confidence.

We conducted ablation studies on three datasets: LOL_v1, LOL_v2_real, and LOL_v2_synthetic. We set up four different framework models to verify the contribution of each designed component to the improvement of the algorithm’s performance.

“Dilated conv” indicates using a six-layer dilated convolution structure with dilation rates of r = 9,3,1,9,3,1 during the illumination estimation phase. This structure expands the receptive field. It also captures how semantic dependencies between pixels weaken with increasing distance.

“Unet++” indicates using Unet++ with multi-level skip connections for damage restoration. This approach better integrates semantic information from different levels and reduces information loss.

“Conv-transformer” indicates adding multi-scale dilated convolution spatial attention to the IF-DSA module. This addition helps express long-range spatial semantic relationships.

“Transformer+” indicates the experimental results after incorporating all components into the network.

Under the same local configuration environment, we tested the PSNR, SSIM, and RMSE values for each framework model. The improvements of each component enhance the algorithm’s performance to varying degrees, as shown in Table 3. The data in the table are from the code testing results. Retinexformer+ performs the best in parameter evaluation metrics. This confirms that our network design is reasonable and effective.