Depending on the design, one-stage detectors can be anchor-based or anchor-free. Anchor-based detectors rely on a set of predefined anchor boxes (priors) with different scales and aspect ratios as regression references, which are often designed using statistics (e.g., clustering) over the training set. Anchor-free methods dispense with explicit anchors and instead predict bounding boxes from points or keypoints on feature maps, leading to simpler designs and often better robustness to extreme aspect ratios and small objects. OverFeat [8] is among the earliest deep learning-based one-stage detectors, introducing a unified framework for joint classification, localization, and detection. YOLO [5] formulates object detection as a single regression problem: it partitions the final feature map into a 7×7 grid, where each cell predicts class scores and bounding-box coordinates. SSD [6] further extends this paradigm by exploiting multi-scale feature maps and a set of default boxes with diverse aspect ratios, thereby discretizing the bounding-box space and enabling multi-scale detection within a single forward pass.

Anchor-free detectors further reduce reliance on handcrafted priors. CornerNet [9] casts detection as paired keypoint prediction by producing heatmaps for the top-left and bottom-right corners. FCOS [3] treats each pixel on feature maps as a candidate location and regresses distances to the four box sides, while an additional centerness branch suppresses low-quality predictions. DETR [10] introduces Transformers and reformulates detection as a set-prediction problem, using Hungarian matching between a fixed set of object queries and ground-truth boxes, thereby removing the need for anchor design and non-maximum suppression.

These developments have significantly improved the accuracy and simplicity of one-stage detectors, enabling their benchmarking and application across diverse dense detection scenarios. However, they also highlight a key bottleneck: how to effectively assign labels to the large number of dense candidates.

During training, each candidate (anchor or point) must be assigned to a ground-truth instance or to the background prior to loss computation; this positive/negative assignment shapes optimization and largely determines detector performance.

Early detectors such as Faster R-CNN [11] and RetinaNet mainly rely on anchor ground-truth Intersection over Union (IoU): candidates above a preset threshold are labeled as positives, whereas those below a lower threshold are labeled as negatives. In contrast, FCOS [3] and YOLO [5] incorporate spatial constraints. YOLO assigns responsibility to anchors whose centers fall in the grid cell containing the ground-truth center, while FCOS and FoveaBox [12] expand the positive set by treating points within a region around each ground truth as positives.

Despite their differences, these approaches share a key limitation: positives and negatives are separated by a single hand-crafted criterion (e.g., IoU threshold, scale, or spatial rule). Such fixed heuristics can yield noisy or ambiguous supervision and fail to exploit richer context among candidates, thereby limiting adaptivity and attainable performance.

To improve adaptivity, a range of dynamic assignment strategies has been proposed. ATSS [13] derives instance-specific IoU thresholds from the mean and standard deviation of candidate IoUs. FreeAnchor [14] casts assignment as maximum-likelihood estimation, allowing anchors to select ground truths via learned likelihood scores. Zhang et al. [15] model quality scores with a Gaussian mixture and use EM to probabilistically separate positives from negatives. AutoAssign [16] introduces instance-wise labeling via central/confidence weighting modules, and DW [17] assigns task-aware weights to both positive and negative samples.

More recently, several methods explicitly combine classification confidence and localization quality for matching and/or weighting. SimOTA (and OTA) [18] performs dynamic-k selection and minimum-cost matching based on classification and regression losses. TOOD [19] designs a task-aligned score to guide sample selection and weighting, while GFL/GFLv2 [20,21] injects localization quality into classification/regression via quality-aware objectives and distributional box modeling. However, while improving task consistency, current methods typically treat label assignment as an instantaneous matching or alignment problem. They often fail to account for the scarcity of truly optimal samples in complex scenes and overlook the potential of explicitly modeling the abundant suboptimal positives to drive a progressive evolution toward optimality.

Intuitively, each ground-truth object should correspond to one and only one optimal sample. Therefore, preserving a meaningful ranking among suboptimal samples is crucial: only when their task rankings are properly ordered can the truly optimal sample emerge during training.

In FSS, we explicitly focus on the suboptimal samples of each instance and aim to “excavate” the optimal one from these suboptimal candidates. To this end, the probability associated with each suboptimal sample is used to guide the allocation of its learning weight, which in turn reweights the classification and localization losses to encourage consistent task rankings.

FSS mitigates cross-task inconsistency among suboptimal samples by assigning instance-aware weights and gradually transforming high-quality suboptimal samples into optimal ones. The weighting design follows three principles: 1.

Preserve intra-instance ranking. For suboptimal samples of the same instance, a larger score s should lead to a larger learning weight and predictions closer to the ground truth.

Formally, for the j-th ground-truth object, let ψ(j) denote the index set of its assigned suboptimal samples. To normalize the ranking among these samples, we derive their learning weights from the probability scores:(8)wi=(sis¯)1u(α+β),wheres¯=∑i∈ψ(j)si|ψ(j)|where i is the sample index. However, when α,β take the larger values, the variance of s tends to increase, which can lead to overly sharp or unstable weights. To moderate this effect, we introduce a reweighting factor u as a hyperparameter that smooths the weight differences between low-ranked and high-ranked suboptimal samples within the same group. By dividing the exponent by u(α+β), we effectively constrain the variance of wi, allowing the detector to focus more on high-quality suboptimal samples while avoiding excessive emphasis on extremely hard cases.

Eq. (8) can be viewed as a mean-normalized power transform of the optimality score. It satisfies the desired properties: (i) monotonicity and intra-instance ranking preservation: for any fixed instance j, wi is strictly increasing in si (when si>0), and for any two suboptimal samples a,b∈ψ(j) we have wawb=(sasb)1u(α+β), so the ranking induced by s is preserved; (ii) inter-instance fairness: dividing by s¯ makes the weights invariant to a global rescaling of scores within the same instance (e.g., due to instance difficulty), keeping weight magnitudes comparable across instances; (iii) controlled sharpness: taking logarithms yields log⁡wi=1u(α+β)(log⁡si−log⁡s¯), showing that u acts as a temperature that smooths weight differences and prevents overly peaked weights when the score distribution becomes sharp (e.g., for larger α,β).

Finally, we apply the learned weights to both classification and localization losses to promote the emergence of optimal samples. The overall FSS loss is defined as:(9)cls=1Npos∑i=1Nposwi⋅ℒcls(picls,IoUi)obj=1Npos∑i=1Nposwi⋅ℒcls(piobj,1)+1Nneg∑q=1Nnegℒcls(pqobj,0)box=1Npos∑i=1Nposwi⋅ℒreg(pibox,GTibox)where i indexes positive samples and q indexes negative samples. ℒcls denotes the binary cross entropy (BCE) loss for classification, while ℒreg denotes the bounding-box regression loss. In our implementation, ℒreg is instantiated as an IoU-based regression loss (e.g., α-CIoU loss [22]) computed between the decoded predicted box and the assigned ground truth; for brevity, the decoding operation is omitted in the notation. We use the foreground IoU as a soft target for classification, thereby integrating localization quality into the classification task under the unified weighting scheme of FSS. Concretely, for a positive sample i assigned to ground truth gtj with class cj, we decode its predicted box B^i from pibox and define IoUi≜IoU(B^i,gtj).

(1) MS-COCO dataset

The MS-COCO benchmark [23] contains approximately 118k training images, 5k validation images, and 20k test-dev images. Following standard practice, we adopt the trainval135k/minival split: models are trained on trainval135k (the union of the train set and a 35k subset of the val set, totaling 135k images), and validated on the remaining 5k images (minival). Final results are reported on test-dev by submitting predictions to the official MS-COCO evaluation server, ensuring fair comparisons with state-of-the-art (SOTA) detectors.

(2) DOTA dataset (a large-scale dataset of object detection in aerial images)

The DOTA dataset [24] is an open-source benchmark for object detection in remote-sensing imagery. Unlike natural-image datasets, objects in aerial images appear with arbitrary orientations due to the overhead viewing geometry. DOTA-v1.5 extends DOTA-v1.0 by expanding the label space from 10 to 16 categories. It contains over 2800 images collected from diverse platforms and online sources, each with a resolution of approximately 4000 × 4000 pixels. Using oriented bounding boxes to capture objects with varying orientations, scales, and shapes, DOTA provides annotations for 16 categories with 188,282 instances.

Given the extremely high resolution of DOTA images, we adopt a tiling-based preprocessing strategy inspired by YOLT (You Only Look Twice) [25], which splits each image into overlapping tiles and merges tile-level predictions with non-maximum suppression (NMS) to remove duplicates. While tiling is effective for small, densely packed objects, performing it online during inference substantially increases runtime because each high-resolution image must be processed into many patches. To avoid additional inference cost, we apply tiling offline as a data-augmentation procedure, thereby improving sensitivity to small/occluded objects without sacrificing inference efficiency. After extensive experiments, we crop each image into 1024 × 1024 patches with a stride of 800 pixels. This setting largely preserves boundary cues for large objects while increasing the effective number of small-object instances. Statistics on the tiled set indicate that small objects dominate across categories and that the dataset exhibits pronounced imbalance both between large and small objects and among categories, making it a challenging testbed for object detection. After tiling, the training and validation sets contain 11,046 and 3615 patches, respectively.

The experiments are trained on Ubuntu 18.04.3 with an NVIDIA Tesla T4 GPU (16 GB) and an Intel(R) Xeon(R) Silver 4110 CPU @ 2.10 GHz (16 cores). We use SGD with an initial learning rate of 0.01, momentum of 0.937, a 3-epoch learning-rate warmup, and a weight decay of 0.0005, and we further adopt an exponential moving average (EMA) of model parameters. The test environment is Microsoft Windows 10 (19043.1348) with an NVIDIA GeForce RTX 3060 Laptop GPU (80 W) and an 11th Gen Intel(R) Core(TM) i7-11800H CPU @ 2.30 GHz (8 cores).

We evaluate both accuracy and efficiency. For accuracy, we report the standard metrics, with Average Precision (AP) as the primary measure. The model resolution input is 1024 × 1024 pixels for DOTA dataset images and 640 × 640 for MS-COCO dataset images, with the remaining regions padded with zero if necessary. For the COCO dataset, all test-dev accuracy results are obtained by submitting predictions to the official COCO evaluation server. For efficiency, we report inference throughput (FPS) with a batch size of 1, and inference latency to reflect resource-constrained deployment.

We use stochastic gradient descent (SGD) with a momentum of 0.937 and a weight decay of 5e–4. The base learning rate is set to 0.01 with a cosine annealing schedule, preceded by 3 warmup epochs. Unless otherwise specified, models are trained from scratch for 200 epochs on 4 GPUs with a mini-batch size of 16 per GPU, using a strong data augmentation pipeline (without pre-training). The hyperparameter u in the weighting function is set to 2 by default.

Since FSS is a detector-agnostic label assignment strategy, it can be seamlessly integrated into a wide range of modern detectors. In our experiments, we instantiate two representative architectures based on commonly used backbones: ResNet-101 [26] and CSPDarkNet [27]. In line with mainstream object detection frameworks, both architectures comprise a backbone, a neck, and a detection head. For the CSPDarkNet backbone, we adopt a PANet-style neck and a decoupled detection head; for the ResNet-101 backbone, we use an FPN neck coupled with the same decoupled head. The primary difference between the two instantiations thus lies in the choice of backbone and neck, while the head and the proposed FSS label assignment remain identical.

Although FSS introduces no additional inference overhead, it adds extra computation during training due to the Gaussian-prior dynamic k selection and probability-to-weight conversion. To quantify this cost, we report the training-time overhead and memory footprint of the training stage in Table 1. All measurements were conducted under identical hardware and training settings. As shown in Table 1, integrating FSS incurs only a minor training-time overhead on both detector instantiations. Specifically, the per-iteration latency increases by ∼2.1%–3.1% and the per-epoch time increases by ∼2.0%–2.3%, while the peak GPU memory rises by less than 5% (∼3.8%–4.2%). These results indicate that the additional computations introduced by the Gaussian-prior dynamic k and the probability-to-weight conversion are lightweight, making FSS a practical drop-in label assignment strategy with negligible impact on training efficiency and no inference-time cost.

All ablation experiments are conducted on the COCO minival set. Unless otherwise specified, we use CSPDarkNet [27] as the backbone, train for 42 epochs under the same settings as in Section 4.2, and keep all other implementation details identical to ensure fair comparisons.

The comparative quantitative results of FSS with representative label assignment strategies on detectors on the MS-COCO test-dev dataset are summarized in Table 5. The compared strategies include ATSS [13], which calculates adaptive IoU thresholds based on statistical properties of object fits; GFL [20], which optimizes a generalized focal loss to jointly model classification and localization quality; PAA [15], which separates positive and negative anchors using a probabilistic Gaussian mixture model; TOOD [19], which aligns tasks via an explicit task-aligned head and learning mechanism; and DW [17], which introduces dual weighting to dynamically refine label importance. FSS, in contrast, focuses on progressively promoting high-quality suboptimal samples through unified probability scoring.

Since detection accuracy is strongly influenced by model capacity (e.g., the backbone/neck), we primarily draw conclusions from backbone-matched comparisons and report results under multiple backbone settings to demonstrate generality. As shown in Table 5, FSS consistently improves performance across diverse detector architectures (YOLOv3, Faster R-CNN, RetinaNet, and modern Decoupled detection head (DDH)-based models) while maintaining high inference efficiency.

Improvements on Standard Detectors. On the classic YOLOv3 (DarkNet-53), FSS boosts the baseline from 33.0 to 36.3 AP (+3.3 AP), outperforming advanced assignment methods such as TOOD [19] (36.0 AP) and PAA [15] (36.0 AP) without the speed drop observed in TOOD (35 vs. 37 FPS). Similarly, on the two-stage Faster R-CNN (ResNet-50), FSS achieves the highest AP of 44.5, surpassing GFL (44.2) and TOOD (43.9). Notably, on the anchor-based RetinaNet (ResNet-101), FSS delivers a substantial gain of +5.0 AP (39.1 → 44.1 AP), significantly outperforming DW (42.9 AP) and ATSS (42.4 AP).

Validation on Modern Decoupled Heads. To verify effectiveness on stronger baselines, we apply FSS to detectors with Decoupled Detection Heads (DDH). With a ResNet-101 backbone, FSS achieves 49.0 AP, matching the top-performing TOOD but with a distinct speed advantage (37.6 FPS vs. 34.4 FPS). More importantly, FSS yields superior localization quality, achieving higher AP75 (53.5 vs. 52.4 for TOOD) and AP50 (67.8 vs. 65.9). When instantiated with a CSPDarkNet backbone (YOLOX-style), FSS reaches 50.8 AP, equaling the best competitor, but again demonstrates a superior accuracy-speed trade-off (37.1 FPS) compared to TOOD (33.4 FPS) and higher AP50/AP75 metrics.

These results confirm that (i) the proposed suboptimal-sample–focused label assignment can be effectively integrated into different detector architectures (ResNet-101+FPN and CSPDarkNet+PANet), and (ii) FSS provides a competitive or superior accuracy–speed trade-off compared with strong SOTA detectors, without introducing any additional inference overhead.

To further evaluate the generalization of FSS in scenarios dominated by tiny, densely packed, and arbitrarily oriented objects, we conduct additional experiments on the DOTA dataset. DOTA is a large-scale aerial-image benchmark featuring extreme aspect ratios, cluttered backgrounds, and dense object layouts, where “perfectly aligned” optimal samples are often scarce. This makes DOTA a particularly suitable testbed for assessing whether our strategy can reliably mine high-quality suboptimal positives and improve label assignment under challenging conditions.

The quantitative results are summarized in Table 6. Since the table contains multiple detectors and assignment variants, we highlight the main takeaway here: FSS consistently improves AP and AP50 across diverse detector families while keeping model complexity essentially unchanged. In particular, FSS yields +4.2 AP/+3.0 AP50 on YOLOX-L (47.4 → 51.6 AP; 71.3 → 74.3 AP50) at the same Params/FLOPs/FPS, +6.2 AP/+4.0 AP50 on Mobile-Former (46.5/70.3 → 52.7/74.3), and +6.0 AP/+3.4 AP50 on SSD512 (38.6/62.2 → 44.6/65.6). These results indicate that focusing on high-quality suboptimal positives is particularly beneficial for remote-sensing scenes with dense layouts and large scale variations, and they provide direct evidence of the detector-agnostic property of FSS. We observe that DOTA images contain many densely packed, tiny instances, frequent partial occlusions, and complex backgrounds. Under these conditions, many candidates around an object exhibit inconsistent classification confidence and localization quality, making “clean” positive assignments difficult and causing noisy gradients, especially for lightweight/compact detectors (e.g., Mobile-Former) or older architectures with weaker feature representations (e.g., SSD512). FSS explicitly mines high-quality suboptimal positives and forms a spatially coherent positive set (Gaussian-prior dynamic-k) around the potentially optimal candidate, thereby reducing ambiguity in crowded regions. Moreover, the probability-derived weighting encourages consistent ranking between classification and localization, providing more reliable supervision for small and adjacent objects; this effect is amplified when the baseline model capacity is limited, leading to the larger relative gains observed on Mobile-Former and SSD512.

Furthermore, by integrating our proposed FSS strategy (denoted as SADet+FSS), the detection performance is boosted to 48.2 AP and 71.8 AP50, representing a consistent improvement of 5.4 AP and 4.2 AP50. Furthermore, the high inference speed of 95 FPS remains. This indicates that mining high-quality suboptimal samples is particularly effective for small and crowded objects.

To intuitively demonstrate this capability, Fig. 5 presents the qualitative visualization of FSS on the DOTA dataset compared with ground-truth annotations. As observed, even in extreme scenarios with densely packed vehicles and varying-scale ships, FSS maintains exceptional localization precision. The predicted bounding boxes closely align with the ground truth, effectively distinguishing adjacent tiny instances without introducing significant false positives. The successful application on DOTA confirms that FSS is not limited to natural scenes but generalizes well to complex remote sensing tasks, further validating its cross-domain robustness.

Conceptual comparison with related methods. Although several recent methods also couple classification confidence with localization quality, their primary focus differs from ours. ATSS [13] adaptively sets an IoU threshold by exploiting the statistics of candidate IoUs, whereas FSS explicitly models the learning value of suboptimal candidates via a unified probability score and then uses a Gaussian-prior-guided dynamic k to form instance-adaptive positives around the potentially optimal sample. PAA [15] assigns labels by fitting a mixture model to separate positives and negatives based on training signals, while FSS treats label assignment as a progressive promotion process: it first selects high-quality suboptimal candidates and then reweights them by optimality probability to preserve ranking consistency across classification and localization. SimOTA/OTA [18] performs dynamic matching by minimizing a cost composed of classification and regression losses, whereas FSS directly models candidate optimality with a probability score and uses this score consistently for both selection and transformation. AutoAssign [16] generates soft labels with center-based priors and learned confidence weighting, whereas FSS centers the selection on the potentially optimal candidate and enforces ranking consistency between classification and localization within each instance through probability-derived weights. GFL [20] (and its variants) improves localization-quality estimation via distributional modeling and quality-aware classification, whereas FSS is orthogonal to head design and can be integrated as a training-time assignment/reweighting strategy to reduce the mismatch between classification confidence and localization quality. TOOD [19] improves task alignment through a dedicated head and learning objective, while FSS addresses alignment from the perspective of supervision generation: by enforcing consistent ranking through instance-wise weights, it encourages the emergence of truly optimal samples from informative suboptimal ones.

Limitations and applicability. FSS leverages the correlation between classification confidence and localization quality to construct the probability score. In the early epochs, both classification confidence and IoU can be poorly calibrated, which may lead to unstable score rankings and fluctuating selected samples/weights. To mitigate this early-stage instability, the score is used only for sample selection/weighting via stop-gradient, and it is further stabilized by instance-wise normalization and smoothing, thereby bounding its impact on optimization. In extremely sparse scenes, the pool of informative suboptimal candidates may be limited, and in extremely dense/crowded scenes, multiple nearby instances can produce highly ambiguous candidates, potentially increasing assignment uncertainty. In practice, we find FSS is stable across a wide range of α and β (Section 4.2). We compute Eq. (3) with stop-gradient (detach) so that gradients do not propagate through the IoU term during score computation, avoiding a noisy feedback loop between localization quality and the assignment weights. We further improve stability in early training via standard learning-rate warmup and EMA, together with instance-wise normalization/smoothing in the weighting function.

Performance-efficiency trade-off and deployment feasibility. FSS modifies only the training-time label assignment and reweighting procedure, and it does not introduce any additional layers or computations in the inference graph. Therefore, the deployed detector preserves the same model complexity (Params/FLOPs) as the corresponding baseline, and the inference throughput/latency reported in our experiments reflects practical deployment behavior under fixed hardware and precision settings (e.g., batch size 1 and FP32 as stated in the table captions). From the perspective of computational and energy constraints, this means that inference-time compute and the associated energy cost remain essentially unchanged when adopting FSS. The only extra cost occurs during training; as quantified in Table 1, the added overhead is minor, indicating that FSS is feasible for practical training and deployment pipelines.

Complexity and scaling. The dominant additional computation in FSS comes from constructing the instance-wise score/matching statistics between candidate locations and ground-truth instances. Let N denote the number of candidates (which grows with input resolution and feature-map density) and M denote the number of ground-truth objects; the associated matrix-style operations scale roughly with N×M, similar in order to many dynamic assignment methods that build per-instance matching costs. In practice, these computations are fully vectorized on the GPU and account for only a small fraction of training time in our settings (Table 1). For extremely high-resolution inputs or extreme object counts, the cost increases proportionally with N and M, and it can be controlled in implementation by restricting candidates using spatial priors and/or computing the score matrix in chunks, without affecting inference-time efficiency.