Ayush Kumar et al. [4] addressed small vehicle detection in aerial imagery using PCA for dimensionality reduction, improving accuracy by reducing misclassifications. Among deep models, ResNet50 outperformed MobileNetv1 (85% vs. 76.25%), emphasizing model selection. However, ambiguity in detecting small objects remained a challenge. Qureshi et al. [5] proposed vehicle detection via channel differencing, blob detection, and shape matching, but the method lacked robustness against varying angles and dataset diversity. Lin et al. [6] introduced the VAID dataset with 6000 aerial images and achieved 88% accuracy using a similar approach, though its performance dropped under non-uniform conditions. Chen et al. [7] combined a Deformable Part Model (DPM) with a CNN to enhance structural feature capture and robustness, especially in occluded or angled views.

Zhuang et al. [8] addressed inefficiencies in UAV detection caused by patch-based image splitting and proposed a lightweight multi-task classification (MTC) network to enhance speed. However, its performance in dense scenes and accurate trade-offs remains unexamined. Zuraimi et al. [9] combined YOLOv4 with DeepSORT, achieving 82.08% precision, but lacked solutions for occlusion and lighting issues. Yang et al. [10] introduced BFEN with SLPN and PNW, reaching 88.71% mAP on DETRAC, though it wasn’t tested in rural settings and had unclear complexity. Rani et al. [11] merged Faster R-CNN and SSD, with performance dropping from 85.22% (KITTI) to 64.83% (PASCAL2007), revealing limitations under varied conditions. Basak and Suresh [12] achieved 87.56% on CityCam using CNNs in low-res traffic scenes, but environmental variability was not assessed. Lyu et al. [13] used a CNN-based segmentation model on UAVID, achieving 85% accuracy, though they struggled with small object tracking and temporal consistency.

Unlike these methods, our proposed approach addresses small object detection, occlusions, and scene diversity. It employs MSR for illumination enhancement, EM for accurate segmentation, and YOLOv10 for robust detection. MSER, Dense SIFT, and Zernike Moments ensure rich feature extraction, optimized via Hybrid Swarm-based selection. Finally, the Vision Transformer handles scale variance and background clutter through self-attention, delivering high accuracy and generalization in UAV-based vehicle monitoring. While existing works have explored individual use of descriptors or CNN-based architectures, few have combined handcrafted multi-view features with transformer-based classification. To the best of our knowledge, this is the first approach that fuses MSER, Dense SIFT, and Zernike features; applies a hybrid PSO–ACO optimization for redundancy reduction; and leverages a Vision Transformer for final recognition in UAV-based traffic monitoring.

Preprocessing is crucial in aerial vehicle detection due to common issues like illumination variation, shadows, and low contrast from atmospheric or sensor limitations. To address this, Multiscale Retinex (MSR) is applied as an advanced enhancement technique that normalizes illumination while preserving details [14]. Unlike conventional methods, MSR enhances both dark and bright regions by estimating reflectance across multiple Gaussian scales, making vehicles more visible in challenging backgrounds. This makes MSR highly suitable for aerial imagery. The mathematical formulation is shown in Eq. (1): (1)R(x,y)=∑s=1sws[logI(x,y)−log(I(x,y)∗Gs(x,y))]here, R(x, y) is the enhanced reflectance, I(x, y) is the input image, and Gs(x,y) is the Gaussian function at scale s with standard deviation σs denotes convolution, S is the number of scales, and ws are scale weights summing to 1. MSR’s multi-scale design enhances details at various levels while suppressing noise. A normalization step, shown in Eq. (2), balances contrast across the image: (2)IMSR′(x,y)=αR(x,y)−min(R)max(R)−min(R)+βwhere IMSR′(x,y) is the final enhanced image, α and β are scaling parameters to control brightness and contrast adaptation, and min(R) and max(R) define the minimum and maximum reflectance values, respectively. This transformation ensures that the enhanced image remains within the valid intensity range, preventing overexposure or excessive darkening. The output of the preprocessing can be seen in Fig. 2.

Segmentation is vital in aerial vehicle detection to separate vehicles from complex backgrounds, enabling accurate feature extraction and classification. Given challenges like lighting variation and occlusions, the Expectation-Maximization (EM) algorithm is applied as a robust probabilistic segmentation method [15]. Unlike traditional thresholding or edge-based techniques, EM uses Gaussian mixtures to model pixel intensities and iteratively refines clusters, ensuring adaptive and precise segmentation, as shown in Eq. (3): (3)P(I(x,y)∣θ)=∑k=1KπkN(I(x,y)∣μk,σk2where, P(I(x,y)∣θ) represents the probability of pixel intensity I(x,y) given the model parameters θ, πk is the weight of the k-th Gaussian component, ensuring ∑k=1Kπk=1, N(I(x,y)∣μk,σk2 denotes a Gaussian distribution with mean μk and variance σk2 for the k-th cluster. The EM algorithm iteratively refines the segmentation through the posterior probability that each pixel belongs to a specific Gaussian component using Bayes’ theorem as mentioned below in Eq. (4). (4)γk(x,y)=πkN(I(x,y)∣μk,σk2∑k=1KπjN(I(x,y)∣μj,σj2where γk(x,y) represents the probability that pixel I(x,y) belongs to cluster k. By leveraging the EM algorithm, aerial images are effectively partitioned into meaningful regions, isolating vehicles from complex backgrounds with high accuracy. The results of the segmentation can be depicted in Fig. 3.

Comparative Evaluation of Segmentation Methods

To validate the effectiveness of EM segmentation in aerial scenes, we compared it against two commonly used unsupervised segmentation methods: Otsu’s thresholding and K-means clustering. All three methods were evaluated on UAVDT frames, using Intersection over Union (IoU) and Dice Similarity Coefficient (DSC) to measure performance. Table 1 shows the quantative comparison of the segmentation strategies.

The results demonstrate that the EM algorithm consistently provides better foreground object separation, particularly in complex aerial backgrounds with varied lighting and occlusion. This strong performance supports its integration in our proposed vehicle monitoring pipeline.

Accurate vehicle detection is essential for aerial surveillance applications, including traffic monitoring and urban planning. However, challenges such as scale variation, occlusion, and dense traffic complicate the task of aerial detection. This study employs YOLOv10, a cutting-edge model designed for fast and precise detection in complex scenes. Unlike region-based methods, YOLO processes the entire image in one pass, enhancing efficiency for large-scale UAV imagery. YOLOv10 boosts performance through an upgraded backbone, an improved Feature Pyramid Network (FPN), and an anchor-free design, making it particularly effective at detecting small vehicles. The detection pipeline consists of feature extraction, bounding box regression, and classification, utilizing a lightweight convolutional backbone: (5)Fl=ϕ(Wl∗I+bl)where, Fl represents the feature maps at layer l, Wl and bl denote the weights and biases of the convolutional layer, ϕ is the activation function (e.g., SiLU or Leaky ReLU). Following feature extraction, the detection head generates bounding box predictions by regressing the object coordinates (x, y, w, h) along with the confidence score C. The bounding box regression is defined as in Eq. (6): (6)B^=σ(tx)+cx,σ(ty)+cy,Pwetw,Phethwhere, x^,y,^w^,h^ are the predicted bounding box center coordinates, width, and height, (cx,cy) denote the coordinates of the cell in the feature grid, (pw, ph) represent prior anchor dimensions, tx,ty,tw,th are the predicted offset values, and σ(⋅) is the sigmoid function ensuring the output remains in a valid range. The final classification stage assigns a confidence score C to each detected object using the SoftMax activation function as defined in Eq. (7): (7)C=ezi∑j=1Neziwhere zi represents the raw class score for a specific category (i.e., vehicle) and N is the total number of classes. This probability distribution ensures accurate classification of detected vehicles. Table 2 shows the parameters used for the YOLOv10 algorithm. The outcome of YOLOv10 can be seen on Fig. 4.

Feature extraction plays a vital role in vehicle classification by transforming raw images into meaningful representations. This study employs MSER, Dense SIFT, and Zernike Moments to boost detection accuracy. MSER detects stable regions under lighting changes, Dense SIFT captures multi-scale textures, and Zernike Moments offer rotation-invariant shape descriptors. Their integration enhances recognition by addressing scale variation, clutter, and occlusion in aerial imagery.

Feature optimization improves aerial vehicle detection by selecting relevant features and reducing redundancy. This study uses Hybrid Swarm-Based Optimization (HSO), combining PSO’s global search with ACO’s efficient pathfinding. HSO enhances classification accuracy and avoids premature convergence by refining features from MSER, Dense SIFT, and Zernike Moments. PSO updates positions using personal and global bests to optimize feature subsets: (13)vit+1=wvit+c1r1(pbest,i−xit)+c2r2(gbest,i−xit) (14)xit+1=xit+vit+1where vit and xit represent the velocity and position of the i^th particle at iteration t, w is the inertia weight, c1 and c2 are cognitive and social coefficients, and r1, r2 are random numbers between 0 and 1. The ACO component refines feature selection by constructing an optimal subset through pheromone-guided probabilistic selection.(15)Pi=τiαηiβ∑j∈Sτjαηjβwhere Pi is the probability of selecting feature i, τi is the pheromone level, ηi is the heuristic information (feature importance score), and α, β control the influence of pheromone and heuristic information. 3D graph for the feature optimization for the vehicle classes can be seen in Fig. 8.

Table 3 lists the key parameter values used in the hybrid PSO–ACO optimization process, selected based on experimental tuning and literature guidance.

The optimization terminates when either the maximum number of iterations is reached or no improvement in classification accuracy is observed over 10 consecutive iterations. The final subset is chosen based on the highest validation accuracy achieved using the Vision Transformer classifier. Algorithm 1 shows the flow of Hybrid swarm optimization below.

The Vision Transformer (ViT) is used for classifying detected vehicles in aerial imagery, utilizing self-attention to capture long-range spatial dependencies and patterns. Unlike CNNs, ViT learns global relationships across the entire image, making it effective for handling scale, occlusion, and perspective variations [18]. The process starts by dividing the image into non-overlapping patches, which are linearly embedded and passed, along with a classification token, into the transformer encoder. Multi-head self-attention (MHSA) then models dependencies between all patches, formulated as:(16)Aij=exp((WQXi)(WKXj)Tdk)∑j=1Nexp((WQXi)(WKXj)Tdk)where Xi represents the embedded feature of the i^th patch, WQ, WK, and WV are the trainable query, key, and value projection matrices, and dk is the key dimensionality for normalization. This attention mechanism assigns a dynamic weight Aij to each patch, allowing the network to focus on the most relevant features. To further refine feature representation, each transformed patch undergoes a feed-forward transformation incorporating non-linearity and residual connections, formulated as follows:(17)Zi(l+1)=σ(W1Zi(l)+b1)+W2+b2+Zi(l)where Zi(l) is the transformed feature vector at layer l, W1 and W2 are weight matrices, b1 and b2 are biases, and σ(⋅) represents a non-linear activation function. Residual connection ensures gradient stability and preserves crucial low-level information while allowing deep feature learning. The final classification is performed by extracting the learned class token and mapping it to output probabilities via a SoftMax function. To enhance robustness, second-order moment-based entropy regularization is introduced to penalize uncertain predictions, defined as:(18)Lreg=−λ∑i=1Cpilogpi+α∑i=1Cpi2where pi represents the predicted probability for class i, λ controls entropy regularization, and α adjusts the impact of second-order probability smoothing. This technique ensures confident predictions while mitigating the impact of ambiguous instances.

Table 4 provides a structured overview of the critical hyperparameters and configurations used in the Vision Transformer-based vehicle classification model. To train the Vision Transformer classifier, we initialized the model with pretrained weights from the ImageNet-21k dataset. The transformer was then fine-tuned on the optimized feature vectors obtained from our feature extraction and Hybrid Swarm Optimization pipeline. We used the AdamW optimizer with an initial learning rate of 3 × 10⁻⁴ and applied a cosine annealing rate schedule to gradually reduce the learning rate to 1 × 10⁻⁶ over the training process.

The model was trained for a maximum of 100 epochs with a batch size of 32, and early stopping was implemented with a patience of 10 epochs, monitoring the validation F1-score. Training was conducted using PyTorch 1.12 on an NVIDIA RTX 3060 GPU (6 GB VRAM). The data augmentation included feature-level mixing and standardization. This configuration enabled stable convergence while preventing overfitting and allowed the Vision Transformer to generalize effectively across both UAVDT and UAVID datasets.

Vanishing Gradient Consideration

A potential concern in deep learning models is the vanishing gradient problem, especially in architectures with multiple stacked layers. However, our classification module specifically designs the Vision Transformer (ViT) to mitigate such issues. ViT employs residual connections and layer normalization, which facilitate better gradient flow across layers. Furthermore, the self-attention mechanism in ViT does not rely on deep convolutional stacks, thereby reducing gradient attenuation risks. The use of the AdamW optimizer with adaptive learning rates also helps stabilize training and maintain effective backpropagation. As a result, our approach does not suffer from vanishing gradients during training, and stable convergence was observed across all experiments.

Algorithm 2 presents the full vehicle detection and classification pipeline, integrating all stages from aerial video input through preprocessing, segmentation, detection, feature extraction, optimization, and final classification. This unified approach ensures high accuracy by leveraging advanced techniques at each step. Key advantages include:

Integrated Pipeline: Combines enhancement, segmentation, detection, feature selection, and classification in one cohesive framework.

This algorithm offers a validated and practical solution for intelligent aerial vehicle monitoring through end-to-end integration and advanced feature optimization. Alogrithm 2 shows the complete proposed methodolgy below.

This subsection introduces the benchmark datasets used to evaluate the proposed framework: UAVDT and UAVID, both widely recognized for aerial object recognition. Sections 4.1.1 and 4.1.2 detail their characteristics and challenges. These datasets validate the robustness and generalization of our approach across diverse conditions.

The k-fold cross-validation strategy was used to assess the proposed model’s performance on UAVDT and UAVID datasets, ensuring robust generalization across diverse scenarios. The evaluation demonstrated high effectiveness in vehicle detection and classification, with UAVDT achieving 94.40% accuracy, as shown in Fig. 9, while Precision, Recall, and F1-scores are detailed in Table 5.

We calculate Precision, Recall, and F1-score (F-measure) using the Vision Transformer. These metrics are then visualized through curves that depict their respective values.

For the UAVID dataset, the confusion matrix results are provided in Fig. 10, showing the confusion matrix, with the Precision, Recall, and F1-scores in Table 6. While Table 7 shows the comparison with SOTA methods.

The comparison table highlights that the proposed method achieves the highest accuracy on both UAVDT (94.40%) and UAVID (93.57%) datasets. Prior methods generally show lower performance, with most results ranging between 75%–92%. Several existing works reports results on only one dataset, limiting their generalization.

The complexity of preprocessing and segmentation depends on pixel count per frame, while YOLOv10 detection scales with frames and bounding boxes, showing quadratic growth. Feature extraction relies on keypoints, and HSO is influenced by feature vectors and iterations. Vision Transformer classification, based on embeddings, generally has cubic complexity. Table 8 summarizes execution times and module-wise efficiency.

Real-Time System Feasibility

To assess the system’s practical usability in real-world traffic monitoring, we evaluated the average processing time of each module on an NVIDIA RTX 3060 GPU (6 GB VRAM). Table 9 presents the breakdown of runtime:

The system achieves an average speed of ~24 FPS, confirming its suitability for real-time traffic monitoring. Since HSO is performed offline, it does not affect inference. Feature extraction supports CPU parallelization, and ViT inference benefits from GPU acceleration and batching, with future deployment feasible on edge devices via model pruning and quantization.

Although the proposed hybrid framework demonstrates promising results in aerial vehicle recognition, there remain several avenues for improvement and further exploration. First, while the current system performs well on UAVDT and UAVID datasets, future work should focus on evaluating the model’s generalizability across additional datasets that cover more extreme variations, such as occlusion-heavy or low-resolution nighttime traffic footage. Second, the computational cost associated with multi-stage feature extraction and optimization, although mitigated through HSO, can be further reduced by exploring lightweight descriptor alternatives or model pruning strategies for real-time edge deployment. Third, integrating a self-adaptive feature selection mechanism or reinforcement learning-based optimization may help further reduce redundancy and dynamically adjust to diverse environments. Lastly, incorporating spatiotemporal information, such as object tracking and temporal attention modules, could expand the system’s capabilities toward video-based behavior analysis and anomaly detection in UAV-based surveillance. Fig. 11 shows some limitation of the proposed system.

To evaluate the individual contribution of each module within the proposed framework, an ablation study was conducted by systematically removing or replacing specific components, and the results are summarized in Table 10.

The ablation study evaluates the individual and combined impact of each module in the proposed framework on classification accuracy across UAVDT and UAVID datasets. Removing key components like MSR preprocessing, EM segmentation, or HSO optimization leads to a noticeable performance drop, confirming their contribution. Replacing YOLOv10 or ViT with alternate models (e.g., YOLOv8, ResNet50) also results in decreased accuracy, highlighting the effectiveness of the chosen architecture. The full model achieves the highest accuracy (94.40% on UAVDT), validating the synergy of all components. Overall, the study demonstrates that the integration of all modules is crucial for optimal performance.