This detection module utilizes an optimized YOLOv12 architecture, refined through an integrated multi-stage pipeline designed for precision, efficiency, and scalability.

[Stage 1: Foundation Model Construction and High-Precision Teacher Model]. In the foundation model construction phase (Stage 1), we implemented a “Target-Exclusive” annotation strategy. This approach involved exclusively labeling the target species (macaques) while deliberately retaining distinct but confounding entities—such as humans, cats, and dogs—as unlabeled data, effectively treating them as “implicit background samples”. Fundamentally, this strategy functions as a Semantic High-Pass Filter: by compelling the model to penalize feature activations associated with non-target entities as errors, the network is forced to sharpen its decision boundaries. This mechanism fundamentally suppresses false positives arising from morphologically similar biological features. Furthermore, the incorporation of approximately 10% pure background images into the training set further bolstered the model’s noise immunity. Consequently, this mechanism yielded a high Precision of 0.924. Crucially, this metric serves not merely as an indicator of immediate detection performance but as the direct determinant of the Signal-to-Noise Ratio (SNR) for the pseudo-labels generated in the subsequent Semi-Supervised Learning phase (Stage 2). It was empirically verified that the error rate of this teacher model in high-confidence predictions converged to 0.0%, thereby providing a near noise-free supervision signal for Stage 2 and establishing the foundational causal factor that ensures the stability of the semi-supervised learning process.

[Stage 2: SSL & Robustness Enhancement]. To scale data diversity efficiently, a Semi-Supervised Learning (SSL) framework is deployed. The S1 teacher model creates pseudo-labels for unlabeled data, leveraging its discriminative power and a strict confidence threshold (confidence≥0.80) to filter for quality. This ensures a maximal Signal-to-Noise Ratio (SNR) in the training set for the “student model”. Consequently, the injection of noise-free pseudo-labels optimizes decision boundaries and repairs the recall weaknesses observed in S1, resulting in state-of-the-art performance.

[Stage 3: IFT & Functional Expansion]. The final stage employs Incremental Fine-Tuning (IFT) to incorporate the “Laser Dot” class without triggering “Catastrophic Forgetting”. IFT is engineered to preserve the perceptual integrity established in SSL, based on the following theoretical principles:

Hierarchical Features: Leveraging the distinction between shallow (general) and deep (semantic) feature extraction.

By reconciling functional expansion with stability, this multi-stage approach delivers a robust perception subsystem. It provides reliable state space inputs—specifically target trajectory and velocity—to the Deep Deterministic Policy Gradient (DDPG) decision-making subsystem, serving as the cornerstone for the system’s adaptive anti-habituation capabilities.

To overcome the limitations of a single sensing modality under adverse weather (rain, fog) and variable lighting (low-light, backlight), the system integrates RGB, Near-Infrared (NIR), and Time-of-Flight (ToF) cameras. Data streams from these sensors are fused and processed with image enhancement algorithms, such as automatic exposure compensation and dehazing, ensuring stable and reliable all-weather detection performance.

To maintain individual animal identity continuity (crucial for behavioral analysis), this study employs advanced Multi-Object Tracking (MOT) algorithms. The system is designed to compare ByteTrack and BoT-SORT to address rapid movement and mutual occlusion in natural environments. ByteTrack excels at handling large numbers of fast-moving and briefly occluded targets, while BoT-SORT, by integrating appearance features, provides stronger robustness in scenarios involving long-term occlusion and appearance ambiguity. This dual-track strategy ensures the system can establish stable and coherent spatiotemporal trajectories.

Non-Lethal Deterrence Principle: The system employs a Class 2 safety laser module (output ≤5 mW) compliant with the IEC 60825-1 international standard. This mechanism utilizes the strong contrast between the laser beam and the low-light environment during early morning hours to trigger an avoidance response in macaques, ensuring safe and effective physical intervention.

Anti-Habituation for Highly Intelligent Species: To prevent the rapid habituation caused by traditional methods (e.g., sound, firecrackers) in highly intelligent species like Formosan macaques, the system combines AI tracking with RL decision-making. The goal is to generate unpredictable and target-specific stimulus sequences to maintain long-term deterrent effects.

The robustness of the DDPG decision policy is strictly predicated on the temporal coherence of the input state vector (St). The reinforcement learning agent relies on accurate history-dependent variables—specifically target velocity and dwell time—to compute the Rhabituation penalty and optimize the anti-habituation strategy. The superior tracking stability of Bot-SORT (avg. tracklet lifetime 1.88 s) serves as a critical noise-suppression mechanism. By minimizing Identity Switches, Bot-SORT prevents the fragmentation of the state history, ensuring that the DDPG agent receives a high-fidelity representation of the macaque’s behavioral response. This causal dependency explains why simpler trackers like ByteTrack, despite acceptable detection accuracy, fail to support the long-term strategic learning required for habituation suppression.

We adopt a hybrid framework combining Reinforcement Learning (RL) and Bayesian Decision Theory. The system utilizes a Deep Deterministic Policy Gradient (DDPG) agent for learning.

State and Action Space:

State Space: Defined by the target attributes provided by the perception subsystem, including the target’s real-time trajectory and velocity, distance to the orchard boundary, and target group density.

This study defines the RL agent’s optimization goal as a composite reward function (R). The function balances multiple objectives (long-term efficacy, safety, anti-habituation) through the adjustment of weights (w).

Rationale and Sensitivity Analysis for Weight Setting: –

Primary Consideration (Deterrence Success Rdeter): The wdeter is set to the highest value (e.g., 1.0), prioritizing the maximization of protection against crop loss risk (i.e., deterrence success). Rdeter must dominate the optimization process, ensuring the strategy remains focused on driving macaques away.

Unit and Normalization of the Reward Function: To ensure the mathematical rigor of R, all reward components (Rdeter, Rsafety, Rhabituation) are quantified and normalized before weighting, preventing a single component’s magnitude from dominating the optimization. Values from different physical units are converted into Unitless values and scaled to a unified range (e.g., [0,1] or [−1,1]).

Component Definition: –

Rdeter: Quantified and normalized based on the percentage change in macaque dwell time reduction or exit from the ROI.

The structure of the composite reward function is defined as: (1)R=wdeter⋅Rdeter+wsafety⋅Rsafety−whabituation⋅Rhabituation

Table 3 listed below presents the DDPG reward function’s various components and their design principles.

Through the optimization of this composite function, the DDPG agent can formulate complex strategies in millisecond timeframes, achieving unpredictable and target-specific stimulus sequences, thereby fundamentally suppressing the habituation effect in highly intelligent species.

The experimental site was chosen in an orchard located in Tou Cheng, Yilan County, Taiwan (as shown in Fig. 6). This location was selected due to its frequent records of Formosan macaque invasions and its representative terrain and environmental characteristics, including proximity to mountainous areas, vegetative cover, and diversity of fruit tree species. These factors make it an ideal platform for system validation. The orchard’s scale and geographical setting effectively simulate typical agricultural damage scenarios in northeastern Taiwan, ensuring the generalizability of the experimental results.

The specific equipment configuration installed around the orchard boundary, as shown in Fig. 7 and Table 4, is as follows:

Image Acquisition Unit: Four high-resolution cameras are deployed, providing 360-degree surveillance coverage. To meet the environmental adaptability requirements of the perception subsystem, the cameras support both RGB and Near-Infrared (NIR) dual-mode functionality. Time-of-Flight (ToF) cameras are also configured at key locations to provide depth information, enhancing target separation capabilities in complex backgrounds.

This study adopts a multidimensional set of metrics to comprehensively evaluate the system’s performance:

Deterrence Efficiency (DE): Includes intrusion frequency, intrusion duration, and crop damage reduction rate.

Experimental Timeline: Covers the deployment and calibration phase, the official experiment phase (three weeks), and the data analysis and model optimization phase.

To ensure the methodological rigor of our performance benchmarking, we conducted empirical comparisons under identical dataset specifications and experimental conditions. This standardized protocol facilitates an objective evaluation of the proposed lightweight YOLOv12n architecture against contemporary State-of-the-Art (SOTA) baselines, with a specific focus on robustness and computational efficiency within a three-stage Incremental Fine-Tuning (IFT) framework. The experimental pipeline comprises three distinct phases: Stage 1 (S1) Supervised Training (establishing teacher model baselines); Stage 2 (S2) Semi-Supervised Learning (optimizing student models); and Stage 3 (S3) Incremental Fine-Tuning (an IFT stress test aimed at integrating the novel “Laser Dot” class while mitigating catastrophic forgetting). The benchmarks for this empirical comparison are defined as follows:

All models were inference-tested on the same edge computing unit (i.e., the NVIDIA Jetson AGX Orin platform).

The primary objective of Stage 1 was to establish initial performance baselines across all models and to identify the optimal “Teacher Model” for the semi-supervised learning in Stage 2 (S2), as illustrated in the baseline comparison in Fig. 8. During the initial supervised training phase, YOLOv12n demonstrated a decisive advantage, with its Precision (P) significantly outperforming all competing models. This high-precision attribute is a critical prerequisite for the efficacy of the subsequent semi-supervised learning phase. Specifically, the ability of YOLOv12n to minimize False Positives allows it to generate high-fidelity supervisory signals, thereby substantiating its role as a reliable Teacher Model. Comprehensive empirical metrics for Stage 1 are provided in Table A1 of Appendix A.

The success of Stage 2 (S2) hinges on the quality of pseudo-labels generated by the Stage 1 (S1) Teacher Model across 536 unlabeled images. This study quantified the “Signal-to-Noise Ratio” (SNR) of the model architectures by analyzing the retention rates of high-confidence pseudo-labels.

Data analysis reveals that the YOLOv12n Teacher Model, trained using the “implicit background” strategy, exhibits the highest discriminative capability, with a high-confidence retention rate that significantly outperforms all competing models. This empirical result confirms that its high-precision architecture effectively translates into a pseudo-label set characterized by the highest quality and SNR. To ensure methodological rigor, a manual audit was performed on all retained high-confidence pseudo-labels. The results strongly confirm that YOLOv12n achieved a precision of 100.0% (0.0% error rate), yielding a noise-free pseudo-label set. Conversely, competing models exhibited higher error rates, posing a potential risk of data contamination during student model training. (For detailed statistics on pseudo-label generation and human-verified error analysis, please refer to Tables A2 and A3 in Appendix A.)

As illustrated in Fig. 9, the blue and orange bars denote the counts of candidate (Conf ≥ 0.5) and retained (Conf ≥ 0.8) pseudo-labels, respectively, while the red line indicates the high-confidence retention rate (%). YOLOv12n achieved a retention rate of 47.1%, significantly surpassing other models, thereby demonstrating the superior quality and confidence of its generated pseudo-labels.

Stage 2 (S2) was designed to quantify the specific impact of the pseudo-labels generated in Stage 1 on the performance of the Stage 2 student models. The results unequivocally demonstrate that the quality of these supervision signals is the determinant factor for student model performance. Fig. 10 visualizes this empirical comparison, where blue and orange bars denote the mAP@0.5 for Stage 1 and Stage 2, respectively, while the overlaid lines track the F1-Score evolution. The annotated ±Δ values quantify the magnitude of post-SSL improvement, with green indicating performance lift and red indicating regression.

Notably, YOLOv12n emerged as the primary beneficiary of the semi-supervised pipeline, achieving the most significant performance gains (+0.010 mAP, +0.023 F1), followed by YOLOv10n (+0.008 mAP, +0.031 F1). This success is directly attributable to the injection of the highest Signal-to-Noise Ratio (SNR) pseudo-labels. The high-fidelity supervision effectively optimized YOLOv12n’s decision boundary, significantly repairing its primary weakness in Recall observed during Stage 1.

In stark contrast, both the Stage 1 leader (YOLOv11n) and the SOTA benchmark (YOLOv8n) suffered performance regression in the semi-supervised phase. These divergent performance trajectories provide empirical validation of the Noise Propagation theory. The degradation observed in YOLOv8n (−0.011 mAP) is causally linked to its low-SNR pseudo-labels (33.3% error rate), which introduced confirmation bias into the student model. Conversely, the performance gain of YOLOv12n verifies that high-SNR pseudo-labels allow the student model to effectively learn from the unlabeled domain distribution without fitting to noise. This outcome strongly reinforces the causal chain proposed in our training strategy: “Implicit Background → High SNR → Effective SSL” (for detailed numerical evolution from S1 to S2, please refer to Table A4 in Appendix A).

Stage 3 served as a stress test for Incremental Fine-Tuning (IFT), designed to evaluate the models’ ability to learn a new class (“Laser Dot”) while mitigating “Catastrophic Forgetting” of previously learned classes.

Results indicate that YOLOv10n suffered a catastrophic failure during this phase, with a precipitous drop in mAP@0.5. This strongly suggests a fundamental incompatibility between its architecture and the applied IFT strategy. In stark contrast, YOLOv12n once again demonstrated superior robustness, experiencing only a minimal decline in mAP. By concluding the test with the highest mAP@0.5 and F1-Score, YOLOv12n solidified its leadership throughout the entire IFT process. This validates that the IFT strategy designed and employed in this study offers the highest protective efficacy for the YOLOv12n architecture. (For detailed data on IFT performance robustness, please refer to Table A5 in Appendix A.)

The empirical comparison is presented in Fig. 11. The blue and orange paired bars correspond to mAP@0.5 at the start of S2 and the end of S3, respectively; the red and blue lines illustrate the F1-Score and Recall during Stage 3. The ±Δ annotations indicate the change in mAP during the IFT phase (green denotes improvement; red denotes decline). It is evident that YOLOv12n exhibited the minimal performance degradation (−0.019), demonstrating the highest training robustness, whereas YOLOv10n showed the most pronounced regression (−0.233), revealing its structural sensitivity to incremental training.

This section aims to quantify the models’ capacity to integrate established (“Macaque”) and novel (“Laser Dot”) classes. This assessment is critical for ensuring the reliability of the “Laser Dot” serving as a visual feedback signal within the closed-loop control system.

As indicated in Fig. 12, regarding the established class (“Macaque”), both YOLOv12n and YOLOv11n exhibited excellent F1-Scores, indicating that the IFT strategy successfully preserved previously acquired knowledge. More significantly, YOLOv12n demonstrated an overwhelming superiority in acquiring the novel class (“Laser Dot”). Its exceptionally high Recall for this class provides evidence that YOLOv12n’s feature extractor adapts most effectively to the detection of transient light sources and minute targets. The stability of this high Recall is vital for the “dynamic tracking” scenarios inherent to the closed-loop control system proposed in this study. In contrast, the catastrophic failure of YOLOv10n was primarily manifested in its severe incapacity to detect the new class. (For detailed class-specific performance decomposition in Stage 3, please refer to Table A6 in Appendix A.)

The final model selection was driven not only by precision but also by real-time inference speed (FPS) and computational resource efficiency (GFLOPs, Parameters). Based on the outcomes of the multi-stage training, empirical data unequivocally support YOLOv12n as the superior choice among competitors for deployment scenarios demanding high precision, efficiency, and IFT robustness.

This conclusion is substantiated by the comprehensive analysis presented in Fig. 13. In this figure, the blue and orange paired bars represent the mAP@0.5 and F1-Score at the final IFT stage, respectively, while the red line plots the inference speed (FPS) measured on a Jetson AGX Orin (FP16) edge computing unit. Results demonstrate that YOLOv12n achieves a synergy of the highest precision (0.923) and peak performance (480 FPS), coupled with the lowest theoretical complexity (6.3 GFLOPs), positioning it as the optimal solution for deployment.

Throughout the critical stages of the IFT workflow, YOLOv12n consistently outperformed other models, leading with a final mAP@0.5 of 0.923 and an F1-Score of 0.886. Conversely, these findings offer a strong cautionary recommendation for future deployments utilizing IFT strategies: the use of YOLOv10n must be explicitly avoided. The catastrophic collapse observed in YOLOv10n during Stage 3 (a precipitous mAP drop of −0.233) exposes a severe incompatibility with incremental learning paradigms. Consequently, for applications requiring long-term functional expansion and knowledge retention, the high robustness of YOLOv12n renders it the premier choice as the core of the closed-loop perception subsystem. (For detailed empirical data on comprehensive model performance and edge deployment efficiency, please refer to Table A7 in Appendix A.)

The system design in this study targets edge deployment, ensuring real-time performance and precision through TensorRT optimization and multiple filtering mechanisms. The empirical inference results and data when deploying the YOLOv12 model on the edge device (NVIDIA Jetson AGX Orin) are shown in Table 5, including key energy efficiency and latency metrics to comprehensively evaluate system feasibility. Our results indicate that while the system maintains a high average frame rate of 35.0 Frames/s, the single-frame inference latency is only 28.6 ms, ensuring the immediacy of deterrence commands. More importantly, the system’s total average power consumption is 12.5 W, and the energy efficiency ratio reaches 2.8 FPS/W, providing critical empirical support for the system’s sustainability in edge and remote deployment scenarios.

This study integrates the theoretical underpinnings of the three critical weights (β0^,β1^,β2^) within the quadratic polynomial regression model. We analyze the sensitivity and robustness of these weights through experiments conducted with varying calibration point counts (N = 50, 100, 181), thereby demonstrating the scientific validity of the model structure and the reliability of the estimated parameters.

Impact of Calibration Data Quality on Robustness: To ensure the stability of weight estimation, we analyzed the influence of the number and distribution of data points on model performance. Experimental results, presented in Table A8 of Appendix A, unequivocally show that using N = 181 calibration points leads to the most robust weight estimates (lowest standard errors) and achieves the optimal RMSE of 1.2 pixels. This demonstrates that a sufficient number of data points is a necessary prerequisite for obtaining stable and precise weights.

This study provides a visual comparison of two key tracking scenarios, as illustrated in Fig. 14. The upper section, “Scene 1”, demonstrates tracking performance in a densely vegetated environment where occlusions among targets are likely. The lower section, “Scene 2”, presents a relatively open environment. Both scenarios visually depict the tracking outcomes of the two algorithms, with blue bounding boxes indicating detected macaques and their corresponding tracking IDs.

Total Unique Tracks: In both scenarios, BoT-SORT generated significantly fewer unique tracks (Scenario 1: 16, Scenario 2: 27) compared to ByteTrack (Scenario 1: 45, Scenario 2: 49). A lower track count typically indicates superior performance, as it suggests fewer instances of identity switches—misassignments of object IDs due to occlusion or appearance changes. These results imply that ByteTrack is more prone to fragmented trajectories in this application.

The primary driver of tracker performance differences lies in the discriminative power of their re-identification (Re-ID) features. The intra-class cosine distance measures the dissimilarity among feature embeddings of different individuals within the same class; a higher distance value indicates stronger capability of the model to distinguish between different identities. The comparative results for different trackers are shown in Table A10 of Appendix A.

Bot-SORT’s success is attributed to its high-quality Re-ID feature extraction capability, which enables it to overcome the challenge of high appearance similarity in wildlife group tracking.

To empirically validate the long-term adaptive advantages of the DDPG strategy (L5), we conducted a three-week (21-day) field experiment specifically to observe its deterrent effect on macaque troops. The data in this section (as shown in Table 7) links the quantified behavioral responses with the three core objectives of the reinforcement learning (RL) reward function.

This study conducted a 3-week (21-day) longitudinal research protocol to quantitatively validate whether the DDPG agent’s (L5) adaptive strategy can effectively counter the target’s biological learning instinct (i.e., “habituation”). •

Quantification Method: Intrusion Frequency Reduction Rate (IFRR):

The primary efficacy metric is the Intrusion Frequency Reduction Rate (IFRR): –

Baseline Definition (B): During the 7-day pre-treatment phase before the formal experiment, the detection system was active, but the deterrence actuators were disabled. The recorded average daily intrusions were 50 intrusions/day.

The L5-DDPG algorithm’s performance demonstrated high stability; over the 21-day experiment, the average Intrusion Frequency Reduction Rate (IFRR) reached 88%, and the daily IFRR never dropped below 80%. This efficacy is critically attributed to its “anti-habituation” reward function design. A habituation penalty term (Rhabituation_penalty) explicitly punishes repetitive behavior, forcing the DDPG agent to constantly explore its “infinite action space” (as DDPG outputs continuous parameter vectors). This mechanism eliminates the prerequisite of repetition required for habituation, thereby proactively resetting the animal’s habituation curve and maintaining long-term deterrent effects.

Statistical Validation

I. Between-Subjects Factor: Algorithm (3 levels: L1, L2, L5).

II. Within-Subjects Factor: Time (21 levels: Day 1 to Day 21). –

Core Statistical Endpoint: The “Algorithm × Time” Interaction

The central statistical question was: Did the performance of L5 change over time differently from that of L1 and L2?

I. The RM-ANOVA analysis revealed a highly significant interaction effect between algorithm and time.

II. Statistical Result: F(40,1170)=88.42,p<0.001.

III. Statistical Conclusion: This result statistically confirms that the performance trajectories of the three algorithms diverged significantly, proving that L1, L2, and L5 exhibited different rates of habituation. –

Paired Superiority Validation at Study Endpoint

To validate the specific advantage of L5 at the endpoint (Day 21), Post-Hoc Tests (using Tukey’s HSD) were conducted.

I. L5 (91%) vs. L1 (8%): Difference of 83% (p<0.001).

II. L5 (91%) vs. L2 (35%): Difference of 56% (p<0.001).

This pairwise analysis validates that by Day 21, the L5-DDPG algorithm was both statistically and substantively superior to the traditional L1 (Fixed) and L2 (Random) deterrence methods.

In summary, these quantitative results and statistical validations collectively support the study’s hypothesis: DDPG, as an adaptive strategy, successfully re-framed the deterrence problem as a dynamic predator-prey game and achieved decisive superiority in the long-term performance comparison.

The comparison of efficacy and habituation for the three algorithms is shown in Fig. 15.

According to the experimental equipment configuration list, a single site is equipped with hardware including a Jetson Xavier NX edge computing unit, IP cameras, ToF cameras, and Class 2 safety laser modules.

The estimate shown in Table 9 indicates that deploying a four-unit regional defense system incurs an initial hardware cost of approximately US$4120. Compared with traditional full-perimeter electric fencing (which may cost tens of thousands of dollars and requires substantial ongoing maintenance costs) or long-term employment of human deterrence personnel (which involves high annual labor costs), this demonstrates the low-cost advantage of this system in terms of hardware.

The primary economic benefit of this system lies in its AI-assisted, precise and adaptive deterrence, which effectively reduces wildlife damage to crops. Based on the empirical deterrence data obtained in this study, and using a 75% reduction in crop damage as the comparative benchmark, an orchard of medium scale—whose annual uninsured crop loss is estimated at US$16,000—yields economic benefits as calculated below (see Table 10):

The payback period for the system can be calculated as follows: (8)Payback Period=Initial Deployment CostAnnual Net Economic Benefit=US$ 4120US$ 12,000≈0.34 years

This analysis indicates that, under the assumption of an annual crop-loss cost of US$16,000, the system can recover its initial deployment cost in approximately four months (0.34 years). These results quantitatively demonstrate the very high economic viability of our intelligent deterrence system—with effective deterrence (88% reduction in intrusion frequency) while delivering a rapid and significant economic benefit to the grower—strongly supporting the claim of a “low-cost” solution. Additionally, by employing a DDPG-driven strategy for low-power, intermittent deterrence, the system further reduces long-term energy and maintenance costs.