Over the past decade, deep learning has driven remarkable advancements in image caption generation. In 2014, Kiros introduced an encoder-decoder framework [2], allowing the encoder to rank images and sentences, while the decoder initiated description generation. Anderson, in 2017, implemented an image caption generation technique leveraging Convolutional Neural Networks (CNN) and Recurrent Neural Networks, where CNN extracted visual features and RNN formed sequences [3]. In 2018, Yao introduced a novel approach that combines bottom-up and top-down mechanisms for calculating attention specifically at the level of prominent image regions [4]. Zhu introduced the Captioning Transformer model, featuring a stacked attention module that replaces the LSTM decoder with a transformer decoder, thereby addressing the cross-time series issue inherent in LSTM [5]. More recently, Liu’s 2021 Caption Transformer model [6] and Devlin’s 2018 BERT-based image caption generation system [7] represented emerging research directions. Yan introduced the zero-shot novel object captioning task, which generates descriptions for novel objects without the need for paired image-caption data and extra training sentences [8]. Daneshfar provided an exhaustive examination of image-to-text diffusion models within the landscape of artificial intelligence and generative computing, filling a critical void in the literature [9]. Wang presented ClipCap++, an enhanced version of ClipCap that integrates key-value pair and residual connection modules [10].

To sum up, these studies have significantly improved the performance of image caption generation models. However, as models become increasingly sophisticated, their applicability in environments with constrained resources diminishes accordingly.

Researchers have been exploring model compression techniques to bridge the gap between the demands of large models and the constraints posed by resource-limited devices. In the realm of image captioning, researchers have made strides in developing efficient models. In 2020, Dai improved the accuracy of LSTM models while reducing their parameter count by incorporating a hidden layer [11]. Furthermore, in 2021, Sharif addressed the challenge of uncommon word representation by representing titles as sequences of subwords, which also helped in reducing the training vocabulary size [12]. More recently, in 2022, Tan proposed three innovative methods to decrease the number of parameters in transformer models: radix coding, cross-layer parameter sharing, and attention parameter sharing [13]. Liu et al. investigated a refined CA-MobileNet V3 model, which was incorporated into cutting-edge microscope products to augment the microscope’s feature extraction capabilities and mitigate misclassification errors during diagnostic processes [14]. Saqib et al. proposed MobileNet V1 and MobileNet V2, which were based on an optimized architecture specifically designed to construct lightweight deep neural networks by utilizing depth-wise separable convolutions [15]. Wang developed a lightweight captioning method with a collaborative mechanism, LCM-Captioner, which balances high efficiency with high performance [16].

Govindharaj et al. introduced a novel approach that integrated Generative Adversarial Networks with the pretrained MobileNet V2 architecture for diagnosing glaucoma [17]. This method addressed the imbalance in medical imaging associated with small datasets by utilizing GANs to enhance the quality of fundus images.

Despite these recent advancements in lightweight image caption generation, there are still numerous challenges that need to be tackled to further improve the efficiency and performance of these models.

Wavelet transform has been integrated into neural network architectures for multiple tasks. Finder et al. [18] employed wavelets to compress feature maps, thereby enhancing the efficiency of Convolutional Neural Networks (CNNs). Saragadam et al. [19] used wavelets as activation functions for implicit neural representations. Liu et al. [20] and Alaba et al. [21] utilized the wavelet transform in a modified U-Net architecture for down-sampling and the inverse wavelet transform for up-sampling. Fujieda et al. [22] put forward a DenseNet-style architecture that leverages wavelets to reinject lower frequencies from the input into subsequent layers. However, the methods in [20–22] were highly tailored architectures, unable to be smoothly used in other CNN architectures. By contrast, Chen concentrated on computational efficiency [23].

Researchers have undertaken efforts to enlarge the kernel size of Convolutional Neural Networks (CNNs), aiming to emulate the global receptive field characteristic of self-attention blocks in Vision Transformers. To attain a global receptive field, it is necessary to conduct spatial mixing operations within the frequency domain following a Fourier transform [24]. Nevertheless, when the Fourier transform is applied, it converts the input into a representation solely in the frequency domain, which prevents it from capturing the local interactions between neighboring pixels. By contrast, the wavelet transform is capable of successfully retaining certain local information while decomposing the image into separate frequency bands, which makes it possible to carry out operations at various decomposition levels. A concurrent study employed neural implicit functions to facilitate efficient mixing in the frequency domain [25]. Based on WTC, Finder et al. introduced a more lightweight, user-friendly, and linear layer structure. This layer can serve as a seamless drop-in replacement for depth-wise convolutions, leading to an enhanced receptive field. Moreover, their research has made a significant further advance [26].

In this work, we employ wavelet transform as it is efficient and straightforward [18,28]. Note that fLL is a low-pass filter, and fLH,fHL,fHH are a set of high-pass filters. Given an image X, WT(X) is obtained by the following convolution: (1)Y=WT(X)=[XLL,XLH,XHL,XHH]=Conv([fLL,fLH,fHL,fHH],X),where XLL is the low-frequency component of X, while XLH,XHL,XHH are its horizontal, vertical, and diagonal high-frequency components, the inverse wavelet transform (IWT) is obtained by the transposed convolution: (2)X=IWT(Y)=Transposed(Conv([fLL,fLH,fHL,fHH],[XLL,XLH,XHL,XHH])).

The cascade wavelet decomposition is then given by recursively decomposing the low-frequency component. Each level of the decomposition is given by: (3)XLL(i),XLH(i),XHL(i),XHH(i)=WT(XLL(i−1)),where XLL(0)=X and i is the current level. This results in an increased frequency resolution and a reduced spatial resolution for the lower frequencies.

Expanding the kernel size of a convolutional layer leads to parameters increasing quadratically. To mitigate that, first, use the wavelet transform to filter and downscale the lower- and higher-frequency content of the input. Then, perform a small-kernel depth-wise convolution on the different frequency maps before using the IWT to construct the output. In other words, the process is given by (4)Y=IWT(Conv(W,WT(X))),where X is the input tensor, and W is the weight tensor of a k×k depth-wise kernel. The operation not only separates the convolution between the frequency components but also allows a smaller kernel to operate in a larger area of the original input.

We take this l-level combined operation and increase it further by using the same cascade principle from Eq. (5). The process is given by (5)XLL(i),XH(i)=WT(XLL(i−1)),YLL(i),YH(i)=Conv(W(i),(XLL(i),XH(i))),where XLL(0) is the input of the layer, and XH(i) represents all three high-frequency maps of level i. To combine the outputs of the different frequencies, performing (6)Z(i)=IWT(YLL(i)+Z(i+1),YH(i)),where Z(i) is the aggregated outputs from leveli onward.

Unlike the WTC architecture mentioned in [26] (Algorithm 1), which involves an additional convolutional step on the source image but is less time-efficient, we have made improvements by replacing function return value Z(0) in the literature [26] with Z(1), and algorithm IMWTC is proposed as follows (Algorithm 2):

For MobileNet V3, depth-wise separable convolution [29] constitutes a fundamental element within the MobileNet series, serving as a highly effective method for diminishing model complexity. Formula (7) delineates the mechanics of depth-wise convolution: (7)G^k,l,m=∑i,jK^i,j,m⋅Fk+i−1,l+j−1,m,here K^ signifies the depth-wise convolutional kernel, DK×DK×M represents the kernel size, and F denotes the feature map possessing dimensions of DF×DF×M.

Subsequently, the feature map undergoes pointwise convolution to adjust its dimensionality, either increasing or decreasing it. Depth-wise separable convolution boasts approximately one-third of the parameters compared to traditional convolution, rendering it a more streamlined and potent technique. The particular methodology for this calculation is outlined in Formula (8): (8)DK⋅DK⋅M⋅DF⋅DF+M⋅N⋅DF⋅DFDK⋅DK⋅M⋅N⋅DF⋅DF=1N+1DK2,where DK×DK denotes the kernel dimensions, M stands for the number of input channels, N represents the number of output channels, and DF×DF signifies the size of the feature map. Subsequently, DK⋅DK⋅M⋅DF⋅DF indicates the computational overhead of depth-wise convolution, while DK⋅DK⋅M⋅DF⋅DF+M⋅N⋅DF⋅DF denotes computational cost of depth-wise separable convolution, and DK⋅DK⋅M⋅N⋅DF⋅DF means computational expense of standard convolution.

MobileNet V3 introduces the Squeeze-and-Excitation block (SE block) [30] into its partial inverted residual structure to elevate the semantic richness of the feature map. Within the SE block, the squeeze component computes the feature vector for each channel of the feature map by employing Global Average Pooling (GAP). This process involves averaging the feature values across the entire feature map for each individual channel: (9)zc=Fsq(uc)=1WIDTH×HIGHT∑i=1WIDTH∑j=1HIGHTuc(i,j),here, u denotes the input to the squeeze component, possessing dimensions of WIDTH×HIGHT×CHANNEL. Specifically, WIDTH×HIGHT represents the spatial dimensions of the feature map, for each channel indexed by c, where c∈{1,2,…,C}, C stands for the total number of channels, and uc signifies the c-th channel of the input feature map.

The excitation block utilizes a gate mechanism composed of two fully connected layers to learn the weights for each of the C channels from z. This gating mechanism is computed according to Formula (10), effectively assigning a weight to each channel based on its relative importance. (10)s=Fex(z,W)=σ(g(z,W))=σ(g(W2δ(W1z))),where δ represents the ReLU activation function, σ denotes the sigmoid activation function, and g stands for gate function. W1∈RCr×C, W2∈RCr×C are the weight matrices for the two fully connected layers, respectively. Here, r signifies the number of hidden layer nodes in the intermediate layer. The ultimate output is computed using Formula (11). (11)xc~=Fscale(uc,dc)=sc⋅uc,where xc~ denotes a feature map corresponding to a feature channel of X~, and sc represents a scalar value within the gated units.

MobileNet V3 adopts h-swish activation function to enhance the network’s nonlinear expression capabilities. The mathematical formulation of the h-swish function is given by Eqs. (12) and (13). (12)h−swish[x]=x⋅ReLU6(x+3)6, (13)ReLU6(x)=min(6,max(0,x)).

Notably, MobileNet V3 can be trained and fine-tuned using a wide range of image caption datasets, allowing it to adapt effectively to various domains and scenarios.

The implementation of the aforementioned models requires the following hardware & software configuration: CPU (Intel^® Core™ i7-10875H), GPU (NVIDIA GeForce GTX1650 (4G)), Windows 10 & Pytorch1.9.0 & Python3.7, CUDA11.1 + cuDNN8.2.1. Fig. 2 is a visualization of the IMWTC and attention changes in our proposed model, Mob-IMWTC.

The datasets used are MS COCO (328,000 images), Flickr 8k (8000 images), and Flickr 30k (31,783 images). The datasets are divided into three sets—training, validation, and testing—in a ratio of 80:10:10, respectively. For the above datasets, in the phase of preprocessing, images are resized to 64 × 64 pixels and normalized; texts are removed, leading “A” or “The” from captions, converted to lowercase, and filtered out captions with a length of less than 5 words.

In this section, our methods (Mob-WTC & Mob-IMWTC) are compared with three CNN architectures (CNN-LSTM [31], CNN-Att-LSTM [32], CNN-Tran [33]), two mainstream methods (LCM-Captioner [16], ClipCap [10]), and our previous work (Mob-Tran [1] based on a lightweight model). The models’ training employs the Adam optimizer with an initial learning rate of 1.5 × 10⁻⁵. The training process consists of 50 epochs, each utilizing a batch size of 50 samples.

We present the sentences generated by the eight models utilized in the experiments. Following this, we list the scores obtained by each model in both objective and subjective evaluations, see Sections 4.2.1 and 4.2.2.