Our contributions are as follows:

Object-Aware Augmentation Pipeline: We present a novel framework to incorporate ground-truth bounding box information directly into transformation parameter computation, creating semantically plausible variations for object detection tasks.

Understanding why adversarial perturbations transfer from one model to another is fundamental to designing effective attacks. Ilyas et al. [15] demonstrated that adversarial examples originate from models learning highly predictive but brittle features inherent to the data distribution—termed “non-robust features.” These features prove useful for standard accuracy but remain vulnerable to small perturbations. Transferability occurs when perturbations exploit such features that are common across various model architectures [20,21].

Subsequent work has examined the geometry of adversarial perturbations [22] and demonstrated that transferable attacks tend to occupy flatter loss basins, which represent generalization across models. Researchers have also analyzed neural network decision boundaries, demonstrating that adversarial examples transfer when they push inputs across shared decision boundaries. These theoretical insights provide a foundation: true transferability requires perturbations that exploit fundamental shared vulnerabilities rather than model-specific artifacts. The challenge lies in designing optimization procedures that enforce this property, which motivates our regularization-based approach.

We establish a theoretical framework grounded in the non-robust features hypothesis [15] explaining how semantic awareness enhances transferability.

Definition 1 (Semantic Plausibility): A transformation 𝒯:𝒳→𝒳 is semantically plausible for image x with annotations ℬ={b1,…,bn} if p(𝒯(x)|ℬ)>τ, where p(⋅|ℬ) represents the conditional distribution of natural images given object configurations.

Intuitive Interpretation: The conditional probability p(𝒯(x)|ℬ)>τ measures whether a transformed image 𝒯(x) remains statistically consistent with natural images containing similar object configurations ℬ. Consider a concrete example: if an image contains small distant objects (e.g., cars 50 pixels wide at 100 meters), applying aggressive 2× scaling would produce unrealistic 100-pixel objects at the same perceived distance—such distortions yield low p(𝒯(x)|ℬ) and violate the threshold τ. Conversely, moderate scaling proportional to object size (e.g., 1.2× for large objects, 1.05× for small objects) maintains p(𝒯(x)|ℬ)>τ, keeping transformations within the natural image manifold—the subspace of statistically plausible images observed in real-world scenes.

Contrasting Transformation Distributions: The distinction between Φsemantic and Φgeneric is critical for understanding our contribution. Content-agnostic methods sample transformations from fixed, image-independent distributions (e.g., uniform random scaling s∼𝒰(0.8,1.2) applied identically to all images), which generate semantically implausible variations—such as distorting small objects beyond realistic aspect ratios or creating scale mismatches inconsistent with scene depth. In contrast, Φsemantic conditions transformation parameters on actual scene content: scaling intensity adapts to object size (s=1+robj⋅ρ where robj reflects true object scale, Section 3.3.3), and rotation centers align with object positions rather than being uniformly random. This conditional sampling ensures all transformations remain within the natural manifold, forcing perturbations to exploit fundamental architectural vulnerabilities rather than artifacts of unrealistic distortions.

Justification: Content-agnostic transformations enable shortcuts: when 𝒯 ignores semantics, 𝒯(x) may leave the natural manifold, allowing perturbations to exploit transformation artifacts. Semantic constraints force targeting of features that are: (i) scale-invariant via s∝robj, (ii) viewpoint-invariant through object-centric rotation, and (iii) architecture-agnostic by construction.

Information-Theoretic View: Semantic-aware augmentation maximizes I(δ;ℱshared) while minimizing I(δ;ℱmodel), ensuring Φsemantic covers architecture-common variations.

Practical Implementation Note:¹1

While τ is not explicitly computed in practice (as p(⋅|B) lacks closed form), our design implicitly enforces high plausibility: by computing transformation parameters as functions of ground-truth object statistics (Section 3.3.3), all generated variations satisfy τ>0.9 based on empirical validation against COCO image statistics.

Object detectors fall into three general architecture families, each with distinct attack surfaces requiring different vulnerability exploitation strategies.

Two-stage detectors such as Faster R-CNN [10] and Cascade R-CNN [23] operate in sequential stages using a Region Proposal Network (RPN) and detection head. Attacks can target either stage, preventing proposal generation [7,24] or causing misclassification in the detection head [25,26]. This sequential architecture creates multiple vulnerability points across the pipeline stages—region information, intermediate results, or final classifications.

One-stage detectors including YOLO [3,12], SSD [11], RetinaNet [27], and FCOS [28] directly predict dense feature maps to reduce inference time. These models are traditionally attacked by maximizing the loss over spatial locations [9,29,30]. Their dense prediction approach renders them both efficient and potentially more vulnerable to spatially distributed perturbations.

Transformer-based detectors, pioneered by DETR [13] and advanced by Deformable DETR [31], DINO [14], and Conditional DETR [32], represent a paradigm shift. By treating detection as set prediction through global self-attention, these models eliminate hand-designed components such as anchors and NMS. Unlike CNNs that process images through hierarchical local features, transformers employ global self-attention mechanisms from early layers, fundamentally altering the vulnerability landscape. The architectural heterogeneity of these three families creates the primary challenge for transferable attacks: perturbations must prove effective across fundamentally different processing pipelines, from local convolution-based feature extraction to global attention mechanisms.

A natural approach to enhancing transferability involves optimizing perturbations across multiple source models simultaneously. Tramèr et al. [8] introduced ensemble adversarial training, demonstrating that attacks effective against multiple models transfer more readily to unseen models. This occurs because the perturbation must exploit vulnerabilities common to all ensemble members, thereby eliminating model-specific artifacts. Recent work extends this principle to object detection [33–35], demonstrating improved black-box performance through attack generation across various detector ensembles. Liu et al. [36] showed that ensemble diversity—measured by architecture, training procedure, and objective function differences—correlates positively with attack transferability. Dong et al. [37] introduced momentum-based iterative attacks coupled with ensemble optimization, achieving state-of-the-art transfer performance.

However, ensemble methods suffer from a critical limitation: computational cost scales linearly with ensemble size. Each optimization iteration involves a forward-backward pass through all source models, rendering the approach prohibitively expensive for large ensembles or resource-constrained settings. This limitation has motivated exploration of single-source alternatives that achieve comparable transferability through other regularization mechanisms, such as input-space transformations and gradient variance reduction schemes [38]. Our work operates in this single-source paradigm, demonstrating that semantically aware augmentation can approach ensemble-level transferability without the computational overhead.

Input transformation has emerged as an efficient alternative to ensemble methods for enhancing transferability. Xie et al. [39] proposed Diverse Input (DI-FGSM), applying random resizing and padding during optimization to improve transferability for image classification. This work adapted the Expectation over Transformation (EOT) framework [16], originally developed for physical-world robustness, to enhance digital transferability. The key insight is that optimizing attacks to remain effective over a distribution of transformed inputs forces perturbations to target more fundamental, transformation-invariant features.

Transformation diversity has been extended along multiple dimensions. Dong et al. [37,40] proposed translation-invariant attacks using momentum and diverse inputs, demonstrating that spatial shift invariance improves black-box performance. Lin et al. [41] applied Nesterov accelerated gradient for improved optimization dynamics in the presence of transformations. Wang and He [42] introduced scale-invariant attacks through multi-scale feature aggregation, demonstrating that scale invariance proves particularly important for object detection where objects appear at various sizes.

For object detection specifically, T-SEA [9] combined self-ensemble augmentation with ShakeDrop regularization, demonstrating the value of transformation-based regularization in detection tasks.

Critical Differentiation from DI-FGSM and T-SEA: While our approach builds upon the EOT framework shared with DI-FGSM [39] and T-SEA [9], our fundamental design philosophy differs substantially from these content-agnostic methods:

Comparison with DI-FGSM: DI-FGSM applies uniform random transformations (resizing, padding) sampled from fixed distributions regardless of image content. For instance, all images receive identical scale factor distributions s∼𝒰(0.875,1.00) and padding ranges, treating a close-up portrait identically to a landscape with distant objects. This task-agnostic approach, originally designed for image classification, ignores the spatial structure and object-level semantics critical to detection tasks.

Comparison with T-SEA: Although T-SEA targets object detection, it retains content-agnostic transformations: random resizing samples scale factors from fixed distributions (e.g., s∼𝒰(0.8,1.2)) regardless of whether images contain large dominant objects or small distant ones. Similarly, random rotation applies identical angle distributions without considering object positions.

Our Innovation: In contrast, AugTrans introduces content-adaptive transformations where augmentation parameters are computed as functions of actual scene characteristics: (1)

Multi-box aware resizing: We compute scale factors as s=1+robj⋅ρ where robj represents the normalized size of the Kobj largest objects. Large-object scenes receive aggressive scaling (robj→1⇒ wider range), while small-object scenes receive conservative variations (robj→0⇒ narrower range), ensuring semantic plausibility.

This content-awareness ensures transformations remain semantically plausible—preserving the statistical regularities that define shared non-robust features across architectures—rather than introducing artificial distortions that enable shortcut learning. Our experiments demonstrate that our method achieves substantially lower AP on YOLOv5s (2.06%) compared to T-SEA (8.6%), demonstrating that semantic awareness constitutes the critical factor enabling superior transferability.

Apart from our approach, other contemporary studies have focused on optimization refinements. Shi et al. [30] improved transferability through better augmentation scheduling and loss formulations. Recent advances (2023–2025) have further extended the boundaries of transferable attacks. Wang et al. [43] proposed DIB-UAP, which leverages deep information bottleneck (DIB) theory to extract and disrupt crucial intermediate features. By identifying features that significantly impact detection through mutual information analysis, DIB-UAP achieves notable cross-architecture transferability. Additionally, they introduce a Scale & Tile augmentation strategy to improve attack effectiveness on medium and large objects. Xue et al. [44] proposed an object feature-wise attention mechanism combined with perturbation extraction using generative adversarial networks, demonstrating improved stealthiness while maintaining attack transferability. Chen et al. [45] improved ensemble transferability through dynamic model diversity weighting, achieving enhanced cross-architecture performance. That validates exploiting shared structural properties—similar to our semantic-aware approach. Zhang et al. [46] demonstrated that path-augmented methods improve transferability by optimizing perturbations along multiple transformation pathways, achieving 18% improvement over baseline methods.

However, these methods either require ensemble optimization (computationally expensive), architectural knowledge (violating black-box constraints), or focus on frequency-domain manipulations without semantic awareness. Despite these advances, existing methods employ statistical feature selection (DIB-UAP) or require complex generative models (Xue et al.), which may not optimally generalize across diverse detector architectures. Our approach differs fundamentally by incorporating ground-truth object information directly into transformation design within a single-source paradigm, forcing attacks to target semantically meaningful invariances inherently shared across architectures—including the emerging CNN-to-transformer transfer challenge that these recent works largely overlook.

Despite these advances, existing methods employ generic augmentations (random resize, padding, rotation) without regard to task-specific semantics. A significant gap therefore remains: for object detection, transformations that ignore object locations, perspectives, scene composition, and object sizes may cause unrealistic distortions that allow the optimization to find shortcuts. Random resizing, for example, applies identical scale factors regardless of whether images contain large dominant objects or small distant ones. This content-agnostic approach fails to leverage opportunities to regularize the attack toward semantically realistic transformations that better capture real-world variations across architectures. Our work addresses this gap through content-aware augmentation design.

Data augmentation has emerged as a cornerstone of adversarial robustness during model training [47–49]. The intuition is straightforward: if models learn features invariant to augmentations, they become more robust to both natural corruptions and adversarial perturbations. Methods such as AugMix [50], AutoAugment [51], and RandAugment [48,52] demonstrate that diverse augmentation strategies encourage learning of robust, generalizable features. Xie et al. [53] demonstrated that adversarial examples can improve the performance of standard models when used as deep augmentation, implying deep connections between adversarial robustness and augmentation-invariant features.

We propose an inversion of this reasoning: if augmentation-invariant features constitute robust features during training, then augmentation-invariant non-robust features should be shared across architectures during attack optimization. By tuning perturbations to remain effective across semantically rich transformations, we force the attack to exploit fundamental vulnerabilities shared across models rather than model-specific artifacts. This insight connects robustness research with transferability. Prior studies suggest that EOT can be modified for digital portability when augmentations are designed with task-specific awareness. However, an important question remains: should attack augmentations mirror those used for defense training, or should they be application-specific? For object detection, the semantics of object scale, location, and spatial arrangement are absent from generic image augmentations, motivating our object-aware augmentation design.

Transformer-based detectors introduce unique challenges for cross-architecture transferability. Recent work has developed specialized attacks targeting transformer-specific vulnerabilities. Methods such as BFDA [54] achieve strong performance by perturbing attention maps and query-key interactions characteristic of transformer architectures. Shao et al. [55] and Fu et al. [56] conducted comprehensive analyses demonstrating that vision transformers exhibit distinct adversarial vulnerability patterns compared to CNNs: perturbations exploit self-attention mechanisms and global feature aggregation from early layers through fundamentally different pathways than convolutional feature hierarchies.

However, these specialized approaches require architectural knowledge of the target model and employ different strategies for CNN vs. transformer targets. This limits their applicability in black-box attack scenarios where the target architecture remains unknown. The fundamental architectural difference between CNNs (local and hierarchical) and transformers (global and attention-based) presents a significant challenge: perturbations exploiting CNN-specific inductive biases may fail against attention mechanisms, while transformer-specific attacks may not transfer to convolutional models. This motivates investigation of general augmentation strategies capable of achieving cross-paradigm transferability.

Despite considerable effort, current transfer attack methods exhibit severe shortcomings. Current EOT-based approaches employ task-agnostic transformations without considering object detection semantics—all images receive identical processing regardless of object location, size, or scene composition. Random resizing applies the same scale factors for large and small objects, creating artificial distortions that facilitate shortcut learning rather than forcing robust feature targeting. Static augmentation schedules fail to exploit curriculum-based progression, and content-independent transformations cannot capture semantically plausible variations that drive true cross-architecture transferability.

Our framework addresses these gaps through semantically aware augmentation: (1) multi-box aware resizing couples transformation intensity directly to actual object content, ensuring plausible scale variations proportional to scene characteristics; (2) dynamic object-centric rotation incorporates ground-truth object positions into transformation design, forcing invariance to semantically relevant geometric shifts; (3) adaptive scheduling implements curriculum learning from conservative to aggressive transformations. By elevating augmentation from generic preprocessing to task-aware regularization, we achieve transferability rivaling ensemble methods while maintaining single-source computational efficiency—particularly valuable in true black-box scenarios where the target architecture remains unknown.

Notation

Before detailing our methodology pipeline, we establish the key mathematical notation used throughout this section. Table 1 summarizes the primary symbols, with comprehensive definitions provided in Supplementary Materials Section S7.

Setup: We consider a transfer-based black-box attack where the attacker has white-box access to a source detector 𝒟source (full architecture, parameters, and gradients) but only black-box query access to victim detectors 𝒟victim (unknown architecture, no gradients).

Goal: Craft imperceptible perturbations δ (‖δ‖∞≤5/255) optimized on 𝒟source that transfer to degrade 𝒟victim without gradient access.

Constraints: No knowledge of victim architecture, defenses, or training procedures; perturbations must remain visually imperceptible.

Let D denote an object detector that processes an input image x∈RH×W×Nc with ground-truth annotations y to produce detection predictions D(x), where Nc denotes the number of image channels. In a transfer-based adversarial attack scenario, we aim to craft a perturbation δ such that the adversarial example xadv=x+δ causes an unknown victim detector Dvictim to fail.

The perturbation is generated using white-box access to a source detector Dsource, meaning we have access to its architecture, parameters, and gradients throughout the optimization process. To ensure visual imperceptibility, the perturbation is constrained by an L∞-norm: ‖δ‖∞≤ε, where ε represents a small constant (typically ε =5/255 in our experiments).

The primary challenge in transfer attacks is that perturbations tailored to a specific source model often exploit model-specific artifacts rather than fundamental vulnerabilities. When applied to architecturally distinct target models, these overfitted perturbations frequently fail because they do not target features common across multiple architectures.

To address this challenge, we implement the Expectation over Transformation (EOT) framework with a novel approach. Instead of generic augmentations that ignore image content, we construct a transformation pipeline based on actual information about object locations and sizes. This forces our attack to function across realistic variations, making it robust enough to transfer between different detector architectures. The key idea involves creating a semantically informed transformation distribution Φ, where each sampled transformation combines multiple content-aware operations that produce diverse, realistic variations. Unlike previous work that applies identical transformations regardless of image content, our pipeline adapts to scene properties such as object positions, sizes, and spatial distribution. During attack optimization, a fresh transformation sequence comprising four complementary components is assigned to each EOT sample.

While our core methodology leverages ground-truth bounding boxes for object-aware transformations during attack optimization, we address the practical concern of black-box scenarios where ground-truth annotations are unavailable. In realistic attack settings, adversaries have access only to the source model’s predictions rather than true annotations.

To validate the practical applicability of our approach, we evaluate a variant that uses predicted bounding boxes (PB) from the source detector instead of ground-truth (GT) annotations. Specifically:

During attack generation, we obtain bounding box predictions from the source detector (e.g., Faster R-CNN ResNet50) on the clean image.

This modification enables our method to operate in true black-box conditions without requiring dataset annotations. Section 4.2.1 presents comprehensive experimental validation demonstrating that the predicted-box variant maintains comparable—and in some cases superior—attack effectiveness compared to the ground-truth variant.

Having defined our semantically aware transformation pipeline, we now specify the attack objective and optimization procedure that leverages this pipeline to generate transferable adversarial perturbations.

Datasets. We evaluate on PASCAL VOC 2012 (5823 validation images, 20 classes) [57] and MS COCO 2017 (5000 validation images, 80 classes) [58]. We report mean Average Precision (mAP): AP50 denotes mAP at IoU threshold 0.5, while AP denotes average mAP over IoU thresholds 0.5–0.95 (step 0.05). Lower values indicate more effective attacks.

Models. We evaluate three detector families: (1) Two-stage: Faster R-CNN with ResNet50 (FR-R50), ResNet50v2 (FR-R50v2), and MobileNetv3 (FR-M3) backbones; (2) One-stage: SSD-VGG16 and YOLOv5s; (3) Transformer-based: DETR and DINO. We use FR-R50 as the primary source model unless stated otherwise.

Baselines. We compare against state-of-the-art transfer attacks: DAG [7], RAP [24], CAP [26], TOG [29], and Shi et al. [30]. For recent methods (2023–2024), we compare against G-UAP, GD-UAP, Daedalus, SU, PG-UAP, SGA-UAP, FD-UAP, and DIB-UAP [43].

Hyperparameters. All experiments use: perturbation budget ε =5/255, iterations Kmax=160, step size η=0.0004, EOT samples NEOT=10 per iteration. Augmentation parameters: rotation range θ∈[−8∘,8∘], curriculum factor c=0.5, resizing modulation ρ∈[−0.2,0.5], aspect ratio jitter ζar=0.1, Gaussian noise σ=0.02, salt-and-pepper probability p=0.01. Loss weights: αcls=1.0, αbox=1.0, αobj=2.0, αrpn_box=1.0, with gradient regularization γcls=γobj=0.8.

Implementation. We use PyTorch 1.13.0 with CUDA 11.7 on an NVIDIA RTX 3090 GPU. The random seed is set to 42 for reproducibility. Pretrained weights are obtained from PyTorch Vision Model Zoo²2

https://pytorch.org/vision/stable/models.html

Code Availability. The source code for implementing the AugTrans framework will be made publicly available at https://github.com/sudhirpandey243/LLM-Model.

Predicted Bounding Boxes. For black-box scenarios, we obtain predictions from the source detector on clean images using confidence threshold 0.5 and NMS IoU threshold 0.5, replacing ground-truth annotations while maintaining all other hyperparameters.

Note on Baseline Performance. Clean model performance varies across datasets and architectures due to inherent characteristics: MS COCO’s complexity (80 classes, dense scenes) yields lower AP than PASCAL VOC (20 classes, simpler scenes). Transformer-based detectors (DETR, DINO) achieve higher accuracy through global attention, while one-stage detectors (SSD, YOLO) prioritize inference speed. All models use official pretrained weights evaluated on respective validation sets.

Reproducibility Note. All experiments use seed = 42 with single-run evaluation. This approach aligns with established practice in adversarial attack research [9,30,43,54], where single-seed reporting is standard for computationally intensive transfer experiments. Once perturbations are generated, victim model evaluation is deterministic—the same adversarial example produces identical outputs. Our convergence analysis (Section 4.7) and consistent performance patterns across architectures and datasets validate that results represent systematic transferability rather than seed-specific artifacts. Multi-seed evaluation would require 3–5× computational cost with minimal additional insight given deterministic evaluation. Code will be released for full reproducibility.

We investigate performance against related attacks on PASCAL VOC 2012 and MS COCO 2017 datasets. Results demonstrate consistent, significant improvements in black-box transferability.

Table 2 presents transferability results on the MS COCO 2017 dataset, featuring more categories and complex scenes. Using FR-R50 as the source, our method demonstrates effective transferability across diverse architectures, with particularly strong performance on one-stage detectors. For YOLOv5s, our method degrades AP to 2.06%, a substantial reduction compared to 8.6% achieved by Shi et al. [30].

Table 3 presents PASCAL VOC 2012 results. Our attack reduces YOLOv5s AP to 8.2%, compared to 18.0% for the nearest competitor. Consistent results across benchmarks demonstrate that our augmentation strategy effectively targets vulnerabilities common across diverse detectors.

Table 4 compares against recent methods (2023–2024), showing mAP degradation (higher is better). Our method achieves average degradation of 53.0% on VOC and 35.8% on COCO across three one-stage detectors, demonstrating superior or competitive performance against all baselines. Notably, we achieve 73.2% degradation on RetinaNet (VOC) and 42.5% on RetinaNet (COCO)—the highest values among all compared methods. These results validate that semantic-aware augmentation enables strong transferability across diverse detector architectures and datasets.

Table 5 presents transfer results on PASCAL VOC 2012 using the lightweight FR-M3 as the source. Our method achieves strong transferability, degrading YOLOv5s to 8.1% AP compared to 18.0% for Shi et al. [30]—a 55% improvement. Similar gains are observed across all targets: SSD_VGG16 (26.1% vs. 28.4%), FR-R50 (16.1% vs. 19.4%), and FR-R50v2 (30.2% vs. 41.6%). These results demonstrate that semantically aware augmentation remains effective even with mobile architectures as source models.

Table 6 provides a per-category breakdown on PASCAL VOC. This compares our method against baselines in challenging black-box transfer attacking YOLOv5s from FR-R50. Results highlight the comprehensive nature of the attack. For challenging categories such as bicycle, bird, and diningtable, our method reduces AP to 0.0%, meaning the detector completely fails to recognize these objects. This contrasts with Shi et al., which achieves only 37.1%, 12.1%, and 10.7%, respectively.

Category-Specific Analysis. Table 6 reveals interesting patterns in attack transferability across object categories. Categories achieving complete failure (0% AP) such as bicycle, bird, chair, diningtable, and train share distinctive characteristics: they possess relatively rigid geometric structures with consistent appearance patterns that our content-aware transformations effectively disrupt. The box-aware resizing and object-centric rotation particularly destabilize the spatial features these detectors rely upon for these structurally simple objects.

Conversely, categories maintaining residual detection performance (sheep: 66.7% AP50, cow: 27.5%, horse: 28.7%) exhibit natural robustness factors: (1) high intra-class appearance variation due to texture, pose, and lighting diversity; (2) deformable body structures that preserve discriminative features under perturbation; and (3) complex natural textures that inherently mask adversarial noise. This aligns with findings by Ilyas et al. [15] that classes with greater natural variation learn more robust features.

Despite these variations, our method achieves substantial degradation across all 20 categories (average AP reduction: 89.7%), demonstrating comprehensive attack coverage while revealing the relationship between object complexity and adversarial vulnerability.

Transfer attack generalizability requires fooling fundamentally different architectures. CNN-based detectors share inductive biases through local receptive fields and hierarchical features. In contrast, transformer-based detectors such as DETR [13] and DINO [14] employ radically different approaches based on global self-attention and set prediction. These models eliminate hand-crafted components like anchors and NMS, instead learning to directly predict object sets through learned queries.

Our experiments demonstrate that AugTrans achieves transferability to transformer architectures despite using only CNN-based source models. Tables 2 and 3 present results when attacking from Faster R-CNN ResNet50:

On MS COCO 2017: Our method reduces DETR-R50 AP from 42.5% to 2.8%, compared to 9.9% achieved by Shi et al. [30]. This represents approximately 72% reduction in the performance gap. For the advanced DINO-R50, we achieve AP of 5.4% compared to 15.1% for the next best method, representing a 64% improvement in attack efficacy. These results prove striking given that DINO represents one of the most robust transformer architectures, employing denoising training and contrastive learning.

On PASCAL VOC 2012: The trend continues with DETR-R50 AP reduced to 9.5% (vs. 20.4% for Shi et al.) and DINO-R50 AP reduced to 13.2% (vs. 29.1%). Consistent results across benchmarks confirm that the performance gain reflects a fundamental capability to target shared vulnerabilities.

This cross-paradigm transferability proves noteworthy because transformers process information through entirely different mechanisms. CNNs rely on local convolutions and spatial hierarchies, whereas transformers employ global self-attention to model long-range dependencies. The successful transfer of a perturbation optimized against CNNs to attention-based models supports our hypothesis: regularization through semantically aware transformations forces targeting of deeper non-robust features that transcend architectural choices.

Theoretical Justification for Cross-Paradigm Transfer. The successful transfer from CNN-based source models to transformer-based detectors requires theoretical grounding. Recent work by Bai et al. [59] demonstrates that despite architectural differences, CNNs and Vision Transformers learn remarkably similar intermediate representations when trained on the same tasks. Their analysis reveals that both architectures converge to comparable feature manifolds in high-dimensional space, with centered kernel alignment (CKA) scores exceeding 0.7 between corresponding layers. This representational similarity provides the foundation for our observed transferability.

Furthermore, Mao et al. [60] establish that adversarial perturbations crafted on CNNs remain effective against transformers precisely because both architectures rely on similar low-level texture statistics and mid-level shape features for object recognition. Their gradient attribution analysis shows that adversarial examples exploit shared non-robust features (in the sense of Ilyas et al. [15]) that exist independent of the attention mechanism. Specifically, they demonstrate that perturbations targeting frequency-domain vulnerabilities transfer with 73% efficacy from ResNet to ViT architectures.

Our results align with these findings: the 72% AP reduction on DETR-R50 and 64% on DINO-R50 (compared to baselines) can be attributed to our semantic-aware augmentation forcing perturbations to target these architecture-invariant features. By optimizing over content-adaptive transformations that both CNNs and transformers must handle (scale, rotation, noise), we implicitly select for vulnerabilities in shared representational spaces rather than architecture-specific inductive biases.

Comparative Analysis and Scope: Table 9 provides important context for our claims. While specialized transformer-aware methods such as BFDA [54] achieve superior performance on transformer targets through architecture-specific mechanisms (DETR: 0.012 vs. our 0.058; DINO: 0.018 vs. our 0.068), our contribution lies in achieving robust cross-paradigm transfer using only CNN-source optimization without requiring target architecture knowledge—a critical advantage in true black-box scenarios where the target architecture remains unknown. The complementary strengths are evident: our method demonstrates superior effectiveness against CNN-based detectors (MRCNN: 0.110 vs. BFDA’s 0.162; YOLOX: 0.214 vs. BFDA’s 0.222), while BFDA excels against transformers through transformer-specific vulnerability exploitation. Our contribution demonstrates that semantically aware augmentation enables strong cross-paradigm transfer from a single source model without architecture-specific tuning. This offers practical benefits in terms of computational efficiency and generalizability compared to ensemble-based or architecture-specific approaches.

To evaluate generalization beyond dataset-specific patterns, we conduct cross-dataset transfer experiments where attacks are generated on PASCAL VOC but tested on MS COCO-trained detectors (VOC→COCO). This scenario poses a significant challenge due to substantial distributional differences: PASCAL VOC contains 20 object categories with relatively simple scenes, while MS COCO encompasses 80 categories with complex multi-object compositions averaging 7.7 objects per image.

Fig. 3 presents the results. Despite these fundamental differences in object categories, image distributions, and scene complexity, our method achieves a transfer rate of 91.0%±7.8%. Specifically, attacks generated on VOC (reducing source detector mAP@50 from ∼90% to ∼8%) successfully transfer to COCO-trained detectors, degrading target YOLOv5s performance from 75% mAP@50 (clean) to near 0% (attacked). The mAP@50:95 metric shows similar trends, with convergence occurring around 80 iterations on both source and target domains.

This cross-dataset transferability validates our central hypothesis: semantically aware augmentation forces attacks to target fundamental, dataset-agnostic vulnerabilities rather than dataset-specific artifacts. By optimizing perturbations under content-adaptive transformations (box-aware resizing, object-centric rotation) that remain semantically plausible across both VOC’s simple scenes and COCO’s complex compositions, our method learns to exploit shared non-robust features [15] that generalize across data distributions.

The 91% transfer rate demonstrates that our object-aware augmentation strategy successfully decouples attack effectiveness from training data characteristics—a critical requirement for real-world black-box scenarios where the victim model’s training distribution remains unknown.

Beyond mAP reduction, we examine how our attacks affect detector behavior through multiple complementary metrics including precision, recall, F1-score, false detection rate (FDR), and misclassification rate (MCR) across all victim models. Results demonstrate consistent attack success: FDR increases from baseline 34.3% to 94.8% (Faster R-CNN), 85.6% (Faster R-CNN V2), and 61.9% (YOLOv5), while MCR rises from 1.8% to 17.6%–22.6%. Precision drops by 92%–98% and recall by 70%–82% across architectures, with detection count distributions shifting from balanced true/false positives to extreme false positive dominance (8–12× amplification). Comprehensive quantitative analysis including per-detector breakdowns, detection count distributions, and cross-architecture robustness comparisons are provided in Supplementary Materials Section S6.

Fig. 4 illustrates our attack’s cross-architecture impact. Row 1 shows correct chair detections across diverse detectors (Faster R-CNN, SSD-VGG16, MobileNetV3, Mask R-CNN) on benign images. Row 2 demonstrates that a single adversarial perturbation causes all detectors to fail, producing either cascades of false positives or missed detections. Rows 3 and 4 reveal the underlying mechanism through Grad-CAM visualizations: the attack corrupts clean, focused feature activations (Row 3) into noisy, disrupted patterns (Row 4), validating that our method targets shared vulnerabilities across architectures.

Fig. 5 provides qualitative visualizations to elucidate the attack transferability mechanisms. The top two rows display detection outputs on sample images before and after the attack. On benign images, all detectors correctly identify the potted plants with single accurate boxes. After our attack, detector outputs become chaotic. Two-stage detectors such as Faster R-CNN generate numerous false positives, while one-stage models such as SSD-VGG16 and RetinaNet either fail to detect objects or generate highly inaccurate boxes. This visual evidence supports the severe performance degradation demonstrated quantitatively.

The bottom two rows visualize the internal feature maps generated by models using Grad-CAM [61]. This provides graphical support for our main hypothesis. The causal chain is clear: attacks interfere with feature maps, causing detection failure. In benign cases, all models exhibit concentrated activations over salient objects, which can be interpreted as similar learned representations. Adversarial perturbations consistently disrupt the common activation patterns across all architectures tested, causing activation maps to become diffuse and scattered. This provides strong visual support for our hypothesis: attacks successfully target general transferable non-robust features rather than model-specific artifacts. This constitutes the primary mechanism for superior black-box transferability.

Fig. 6 demonstrates cross-architecture robustness, presenting detection outputs across six diverse architectures on benign vs. adversarial images. Original images show consistent, correct single-object detection across all models. DETR, FCOS, YOLOv3, DINO, YOLOX, and VFNet each accurately localize and classify cats. When our adversarial perturbation is applied, all detectors experience catastrophic failure, generating numerous false positives with spurious class predictions and scattered boxes. This synchronized breakdown across fundamentally different paradigms corroborates our quantitative findings and validates that our augmentation-driven approach generates perturbations exploiting universally shared vulnerabilities.

Computationally prohibitive attacks have limited practical utility [62]. We analyze attack convergence to assess efficiency. Figs. 7 and 8 plot black-box transfer performance (mAP against YOLOv5s) vs. optimization iterations on PASCAL VOC and MS COCO.

In both cases, attacks demonstrate rapid convergence, with mAP (both AP50 and AP) drops precipitously within the first 40 iterations on both datasets. Performance continues decreasing until plateauing around 80–100 iterations. Our method does not require excessive iterations to achieve effectiveness. The hyperparameter Kmax=160 guarantees powerful attacks without additional overhead. This efficiency proves important for practical applications and large-scale experiments.

Table 10 summarizes the computational cost comparison. We evaluate computational efficiency on MS COCO 2017 using an NVIDIA RTX 3090 GPU. Our method requires 16.7 s per image (2.9× slower than the baseline’s 5.6 s) but achieves superior effectiveness-per-computation: 10× fewer FLOPs (2292 vs. 25,208 GFLOPs), 4.2× better attack success (2.06% vs. 8.6% AP), and 10.6× higher efficiency score. The time overhead stems from semantic augmentation (27.9% of computation) but enables GPU-optimized operations with better computational efficiency. Acceleration strategies reduce generation time to ∼5 s per image with minimal performance degradation (comprehensive analysis in Section S4, Supplementary Materials).

We conduct rigorous ablation studies over the augmentation pipeline, loss function components, curriculum scheduling analysis, and hyperparameter sensitivity. These provide systematic evidence that each component contributes meaningfully to the final performance.

To evaluate practical robustness, we assess effectiveness against common input transformation defenses (JPEG compression, bit-depth reduction, Gaussian smoothing, median filtering, random resizing).

Key Findings: Our attack maintains effectiveness across all defenses with only marginal degradation. The best-performing defense (JPEG-50) reduces average detections from 89.31 to 84.91 per image on MS COCO, representing less than 5% improvement. This limited effectiveness occurs.

Why Defenses Fail: (1) Transformation-aware optimization during attack generation defends against similar defensive transformations, (2) our attack targets fundamental vulnerabilities in learned representations rather than superficial artifacts, and (3) content-adaptive perturbations resist content-agnostic preprocessing.

Comprehensive defense evaluation including detailed results, visualizations, and analysis of JPEG compression, bit-depth reduction, spatial filtering, and comparisons with clean images are provided in Supplementary Materials Section S5.