We model cooperative multi-UAV search as a discounted Markov decision process on a discretized workspace. Let the two-dimensional workspace be discretized as (1)𝒟={(x,y)∣x=1,…,DX, y=1,…,DY},where each cell (x,y) either contains a target or is empty, and |𝒟|=DXDY denotes the total number of grid cells. Time is discrete t=0,1,2,…. The cellwise occupancy posterior is (2)bx,y(t)=P(cell (x,y) contains a target|𝒵1:t),with 𝒵1:t the σ-algebra generated by all measurements up to t. The initial prior bx,y(0) is specified from domain knowledge (uniform). Denote by b(t)={bx,y(t)}(x,y)∈𝒟 the full occupancy field.

We now specify the agent set, action space, kinematics, joint control, and state representation. Let 𝒰={1,…,N} index the UAVs. UAV u has planar position ptu∈R2 and heading ψtu∈(−π,π]. Each UAV selects a discrete yaw–increment action (3)atu∈𝒜={−α,0,+α}  (degrees),where α=45, and the action evolves under constant–speed kinematics with sampling step Δt>0 and speed v>0.

Given a chosen yaw–increment action atu and constant-speed motion, the heading and planar position of UAV u evolve according to (4){ψt+1u=ψtu+π180atu,pt+1u=ptu+vΔt[cos⁡ψt+1usin⁡ψt+1u],where v is the constant forward speed and Δt is the sampling interval.

Collecting all per-agent yaw–increment actions yields the joint action vector (5)at=(at1,…,atN)∈𝒜N.

The MDP state collects agent poses and the occupancy field, namely (6)St=({ptu,ψtu}u∈𝒰, b(t)).

Each UAV carries an omnidirectional sensing disc of radius Rsen>0. A grid cell (x,y) is classified as fully observed by UAV u at time t if all four vertices ∂𝒟(x,y) lie within the disc centered at ptu: (7)Covu(x,y;t)={1,max(x′,y′)∈∂𝒟(x,y)‖ptu−(x′,y′)‖≤Rsen,0,otherwise.

The instantaneous covered set is defined by (8)𝒱t={(x,y)∈𝒟: ∑u=1NCovu(x,y;t)≥1},where the coverage ratio at time t is given by CovRate(t)=|𝒱t||𝒟|. The cumulative exploration status of a cell is recorded by (9)Visited(x,y;t)=I{∃τ≤t: ∑u=1NCovu(x,y;τ)≥1}.

Let Ztu denote the measurement field collected by UAV u at time t. For any covered cell (x,y) with Covu(x,y;t)=1, the binary detector on UAV u obeys (10)P(Zx,yu(t)=1|Tx,y)={PDu,if Tx,y=1,PFAu,if Tx,y=0,where Zx,yu(t)∈{0,1} is the binary measurement at cell (x,y) and time t, Tx,y∈{0,1} denotes the hidden occupancy of cell (x,y), and PDu, PFAu are the detection and false–alarm probabilities of UAV u, respectively. The set of UAVs that cover (x,y) at time t is (11)𝒰x,y(t)={u∈𝒰: Covu(x,y;t)=1}.

Under conditional independence given Tx,y, the joint likelihoods for the measurements on (x,y) at time t are (12){ℒ1=∏u∈𝒰x,y(t)(PDu)Zx,yu(t)(1−PDu)1−Zx,yu(t),ℒ0=∏u∈𝒰x,y(t)(PFAu)Zx,yu(t)(1−PFAu)1−Zx,yu(t).

The cellwise Bayesian update is then (13)bx,y(t+1)={bx,y(t)ℒ1bx,y(t)ℒ1+(1−bx,y(t))ℒ0,𝒰x,y(t)≠∅,bx,y(t),𝒰x,y(t)=∅.

The transition kernel induced by the deterministic kinematics and the cellwise Bayesian update is (14)P(St+1∣St,at)=∏u=1Nδ(pt+1u−ptu−vΔt[cos⁡ψt+1u,sin⁡ψt+1u]⊤)∏(x,y)∈𝒟δ(bx,y(t+1)−Bayes(bx,y(t),Zx,y(t))).

The team reward adopts a scale-calibrated saturated form, first define analytic normalizers (15)Umax=NπRsen2,Emax=c0NΔt+cψπ180Nα,with fixed coefficients c0=1.0×10−3 and cψ=1.0×10−2, and construct dimensionless components (16)ℐ^t=H(b(t))−H(b(t+1))Umaxln⁡2,𝒞^t=𝒞t,ℰ^t=ℰtEmax.

The saturated team reward is then (17)Rt=λℐtanh(ℐ^t)+λ𝒞tanh(𝒞^t)−λℰtanh(ℰ^t),where λℐ, λ𝒞, and λℰ are positive weighting coefficients that control the relative importance of information gain, coverage efficiency, and energy–time cost, respectively.

The reward components are defined as follows. The entropy of the belief field is (18)H(b(t))=∑(x,y)∈𝒟[−bx,y(t)ln⁡bx,y(t)−(1−bx,y(t))ln⁡(1−bx,y(t))],the coverage efficiency term is (19)𝒞t=1|𝒟|∑(x,y)∈𝒟bx,y(t)ΔVx,y(t)⋅∑(x,y)∈𝒟I{∑u=1NCovu(x,y;t)≥1}bx,y(t)max{1, ∑(x,y)∈𝒟∑u=1NCovu(x,y;t)bx,y(t)},where the energy–time cost term is defined as (20)ℰt=c0NΔt + cψπ180∑u=1N|atu|,with ΔVx,y(t)=Visited(x,y;t)−Visited(x,y;t−1) denoting the incremental visitation indicator at cell (x,y), and I{⋅} the indicator function. The coefficients c0>0 and cψ>0 control the time-related and turning-related energy costs, respectively.

The planning objective is the discounted return (21)J=E[∑t=0∞γtRt],γ∈(0,1),which fully specifies the discounted MDP on the state St and underpins the two-timescale latent–skill policy in Section 3.2.

On the discounted MDP of Section 3.1, each agent is equipped with a slow latent skill that governs behaviour over a fixed segment of K steps, while fast per-step actions are generated conditionally on the current skill and an autoregressive hidden state.

Let segment boundaries be tk=kK with K∈N. At each segment start tk, agent u samples a categorical latent skill from a skill head conditioned on a summary of global and local information, (22)ztku∼πϕu(z∣Gtku),ztu≡ztku,where z∈{1,…,M} denotes the discrete intent index, and the skill remains fixed within the current segment t∈[tk,tk+K−1]. The summary Gtu aggregates a spatial feature of the belief field with local encodings and team statistics, (23)Gtu=Agg(gtmap,gtloc,u,{ptv}v∈𝒰),where gtmap=CNN(b(t)) encodes the global belief map, and gtloc,u=enc(otu) encodes the local observation of UAV u. Agg(⋅) is a permutation-invariant aggregation function over team states, and Emb(z)∈Rdz denotes a learnable embedding associated with intent z.

For each agent u, a hidden state evolves within the segment via a GRU driven by local features and the current skill embedding, (24)htu=GRUω(ht−1u, [gtloc,u ⊕ Emb(ztu)]),with parameters ω and concatenation ⊕. The resulting state, coupled with the belief feature, parameterizes the action head: (25)ℓtu=MLPθ([htu ⊕ gtmap])∈R3,πθu(atu=k∣htu,Emb(ztu),gtmap)=exp⁡(ℓt,ku)∑k′∈{−α,0,+α}exp⁡(ℓt,k′u),with parameters θ.

Collecting parameters Θ=(ϕ,θ,ω), the joint policy over a segment factorizes into skill selections at tk and per-step action selections conditioned on the fixed skill and the evolving hidden state, (26)ΠΘ(ztk,atk:tk+K−1∣Stk:tk+K−1)=∏u=1Nπϕu(ztku∣Gtku) ∏t=tktk+K−1 ∏u=1Nπθu(atu∣htu,Emb(ztku),gtmap),where htu evolves according to (24).

Learning objectives are matched to these timescales. The segment return starting at tk is (27)Gtk(K)=∑τ=0K−1γτRtk+τ,where γ∈(0,1) is the discount factor, and the stepwise (infinite-horizon) return is given by Gt=∑τ=0∞γτRt+τ. A centralized value function on the belief state augmented by latent and hidden summaries is introduced as (28)Ξt=(St, {ztku}u∈𝒰, {htu}u∈𝒰),Vψ(Ξt)≈E[Gt∣Ξt],which supports low-variance advantage estimates at both levels. With temporal-difference residuals and generalized advantage estimation, (29)δt=Rt+γVψ(Ξt+1)−Vψ(Ξt),A^t=∑ℓ=0∞(γλ)ℓδt+ℓ,λ∈[0,1],the segment-level advantage at tk is (30)A^tkskill=Gtk(K)−b¯φ(Υtk),Υtk=(Stk,{Gtku}u∈𝒰),where b¯φ is a K-step baseline with parameters φ.

Optimization is carried out by proximal policy updates at both timescales. For the action head, define the probability ratio (31)rtu(θ)=πθu(atu∣htu,Emb(ztku),gtmap)πθoldu(atu∣htu,Emb(ztku),gtmap),and minimize the clipped surrogate aggregated over agents, (32)ℒPPO-act(θ,ψ)=−∑u=1NE[min(rtuA^t, clip(rtu,1±ϵ)A^t)]+cvE[(Vψ(Ξt)−V^t)2]−cent∑u=1NE[ℋ(πθu)],with clip parameter ϵ>0, weights cv,cent>0, and bootstrap target V^t. For the skill head, the segment–start ratio (33)rtk(z),u(ϕ)=πϕu(ztku∣Gtku)πϕoldu(ztku∣Gtku)leads to the segment–level surrogate (34)ℒPPO-skill(ϕ)=−∑u=1NE[min(rtk(z),uA^tkskill, clip(rtk(z),u,1±ϵ)A^tkskill)]−cent-z∑u=1NE[ℋ(πϕu)],with cent-z>0. Combining both timescales yields the overall objective (35)minθ,ϕ,ψ ℒtotal=ℒPPO-act(θ,ψ)+ℒPPO-skill(ϕ).

All experiments strictly follow the discounted MDP and sensing specification described above, and adopt the conventional benchmark setting used in prior cooperative search studies [28,34], with stationary targets and ideal, latency-free inter-UAV communication. The basic setup consists of a 50×50 grid with three UAVs and ten stationary targets. The sensor footprint is set to Rsen=0.8 (in grid-cell units), which determines per-step observable area and thereby the rate of uncertainty reduction. Optimization proceeds with PPO using γ=0.99, λ=0.95, and clipping coefficient ϵ=0.20; actor and critic learning rates are both 3×10−4. Reward shaping follows the three-parameter saturated design in Eq. (17), with (λℐ,λ𝒞,λℰ)=(1.0,1.0,0.1) to balance information gain and coverage against energy–time penalization on a common scale.

All baseline networks are configured with comparable capacity and are trained using similar batch sizes. For the PPO-based SCLI–CMUS, we use on-policy trajectory batches without long-term replay, whereas the MADDPG-based methods rely on a replay buffer of fixed capacity with minibatch sampling.

1.

DQN method [35]: Each UAV runs an independent Deep Q-Network on its local observation to estimate action values for the discrete yaw increments. The absence of explicit coordination limits information sharing and typically degrades scalability in larger or cluttered maps.

We report task performance through spatial coverage and target discovery, and we quantify search efficiency via convergence times to fixed performance levels. Let 𝒟 denote the grid, 𝒱t⊆𝒟 the set of cells visited at least once by time t, N⋆ the total number of targets, and Ndet(t) the number detected by time t. Define the instantaneous fractions (36)κ(t)=|𝒱t||𝒟|,δ(t)=Ndet(t)N⋆.

To capture the speed at which operational effectiveness is achieved, introduce coverage and discovery convergence times as first hitting times of prescribed thresholds ρcov,ρdet∈(0,1]: (37)τcov=inf{t∈N0: κ(t)≥ρcov},τdet=inf{t∈N0: δ(t)≥ρdet}.

In all experiments we set ρcov=ρdet=0.85. Smaller values of τcov and τdet indicate faster attainment of wide-area exploration and target acquisition, respectively, and correlate with reduced flight time and energy expenditure.

This section quantifies learning efficiency and asymptotic performance of the proposed method relative to a strong baseline. Fig. 2 reports episode–wise learning curves, where the horizontal axis denotes the training episode index and the vertical axis denotes the total episode reward computed under the reward design in Section 3.1. Curves correspond to the mean over repeated runs, and shaded bands depict variability across runs.

A consistent pattern emerges across all subplots. The proposed SCLI–CMUS (red) rises sharply at early episodes and reaches a high plateau with markedly reduced dispersion, whereas MADDPG (blue) exhibits slower ascent, a lower steady level, and wider fluctuations. This behaviour is most pronounced in Fig. 2a with three agents, where SCLI–CMUS achieves a visibly higher steady reward and converges in substantially fewer episodes. The gap persists in Fig. 2b, indicating that the advantage is robust when scaling to five agents. The reduced variance of SCLI–CMUS is consistent with the segment–conditioned intent mechanism and the saturated, scale–balanced reward, which together suppress redundant exploration and stabilize gradient updates.

The scaling trend with agent count is also informative. Moving from three to seven agents, both methods display a gradual reduction in asymptotic reward, which is consistent with fixed–horizon evaluation: faster attainment of high coverage leaves a longer terminal phase dominated by energy–time penalization. Despite this shift in absolute level, SCLI–CMUS maintains a persistent margin and tighter confidence bands in Fig. 2c, indicating improved coordination under higher platform density.

In Table 2 (Search Area = 50 × 50), the proposed SCLI–CMUS achieves the best coverage and discovery convergence across all team sizes. For N=3, SCLI–CMUS reduces τcov to 1102 and τdet to 1232, yielding improvements of approximately 36% and 29% relative to MADDPG (1718/1723) and 32% and 23% relative to ME–RL (1611/1598). Against the strongest hierarchical baseline (DTH–MADDPG), SCLI–CMUS still provides 21% faster coverage and 9% faster discovery (1397/1348 vs. 1102/1232). As the team scales, the margins persist. For N=5, τcov and τdet fall to 860/1137, improving over MADDPG by 43%/20% and over DTH–MADDPG by 13%/10%. At N=7, SCLI–CMUS attains 710/997, exceeding MADDPG by 48%/40% and DTH–MADDPG by 11%/6%. The aggregate “Total” column confirms the trend: 6038 for SCLI–CMUS vs. 6853 for DTH–MADDPG (≈12% gain) and 9400 for MADDPG (≈36% gain). These gains are attributable to segment-conditioned intent selection and scale-calibrated reward saturation, which jointly suppress redundant footprint overlap, prioritize high-entropy regions, and stabilize critic estimates under CTDE.

The scaling behavior with N is also consistent and informative. All methods exhibit decreasing τcov and τdet as the team grows, reflecting the intrinsic parallelism of multi-UAV coverage. However, SCLI–CMUS shows the steepest decline, indicating that additional agents are efficiently utilized rather than inducing interference. In particular, the improvement from N=3 to N=7 is 36% for coverage (1102→710) and 19% for discovery (1232→997), whereas MADDPG improves by 21% and 3% over the same range. The hierarchical baseline DTH–MADDPG narrows the gap relative to flat actor–critic learners, yet it remains consistently behind SCLI–CMUS, suggesting that segment-consistent skill conditioning and the three-parameter saturated reward yield more effective division of labor and faster attainment of operational performance.

To further probe this behaviour, we extended the comparison between SCLI–CMUS and the strongest hierarchical baseline (DTH–MADDPG) from the 50×50 workspace to larger search areas of 60×60 and 70×70. The 50×50 case already shows that DTH–MADDPG is the closest competitor in terms of coverage and discovery convergence, so these larger maps provide a more stringent test of scalability. As the search area grows to 60×60, the advantage of SCLI–CMUS becomes most pronounced: the total convergence-score gap between the two methods increases to 6797 vs. 7698, i.e., an absolute difference of 901, larger than the corresponding gap on the 50×50 grid (6038 vs. 6853). This indicates that on moderately larger workspaces, segment-conditioned intents and belief-based reward shaping yield more efficient spatial partitioning and reduce redundant coverage more effectively than the dual-timescale controller in DTH–MADDPG.

We next examine the sensitivity of SCLI–CMUS to the three reward weights λℐ, λ𝒞, and λℰ in (17). Since information gain is the primary driver of target discovery in belief-based search, we fix λℐ=1 throughout and vary λ𝒞 and λℰ around the nominal setting (λ𝒞,λℰ)=(1.0,0.1) used in Section 4.1.

Representative results for N=5 are summarized in Table 3. The coverage weight λ𝒞 controls how strongly the policy prioritizes expanding the visited set: a low value (λ𝒞=0.5) yields markedly larger τcov and τdet (up to 1150/1450), whereas a high value (λ𝒞=1.5) achieves the fastest coverage (851 steps at λℰ=0.05) but consistently slower target discovery (τdet≈1230–1280). The energy–time coefficient λℰ regulates motion aggressiveness: with λ𝒞=1.0, the setting (λ𝒞,λℰ)=(1.0,0.10) attains the best overall trade-off, with τcov=860 and the globally minimal τdet=1137, while both smaller and larger λℰ slightly degrade either coverage or discovery.

This case study provides a qualitative examination of cooperative behaviour under the proposed policy in a 50×50 workspace with three UAVs and ten fixed targets. Fig. 3 depicts colour–coded trajectories at three representative time stamps. At t=300 (Fig. 3a), the team has already established a clear spatial allocation: trajectories exhibit strong inter-agent separation and limited crossovers. Large uncovered areas are partitioned implicitly, and each UAV conducts frontier-seeking sweeps within its assigned sector. The resulting footprints cover disjoint corridors with small overlap, which accelerates global coverage while preventing early concentration around the same cells.

At t=1000 (Fig. 3b), the belief map has concentrated around multiple target locations, and paths become denser in those neighbourhoods. Agents maintain sector integrity while adapting their local loops to repeatedly interrogate high-probability cells. Boundary incursions are rare and occur only where adjacent sectors meet, indicating stable intent selection and limited handover cost. The joint pattern reflects a balanced exploration–exploitation regime: residual unexplored pockets are swept, and detected vicinities receive increased sampling frequency.

At t=1500 (Fig. 3c), the team enters a persistent monitoring phase. Each UAV continues to patrol its sector with short, recurrent loops centred on previously informative regions. The path overlap remains low and the blank areas show no redundant revisits, which is consistent with the energy–time penalization in the reward and the segment-consistent action generation.