PEFT has emerged as a mainstream paradigm for adapting pre-trained LLMs to new tasks with minimal cost, preserving their strong generalization capabilities [23,24]. The core idea is to freeze the vast majority of the LLM’s parameters and update only a small, manageable subset.

Prominent PEFT techniques achieve this in various ways. For instance, Adapter Tuning inserts small, trainable modules between existing Transformer layers [25,26], while Prefix-Tuning prepends trainable vectors to the input to steer the model’s attention [27]. Other methods, like BitFit, fine-tune an extremely small fraction of existing parameters, such as the bias terms alone, yet achieve competitive performance [28]. Among the most widely adopted methods is Low-Rank Adaptation (LoRA) [29], which utilizes a low-rank approximation for weight updates to ensure high parameter efficiency.

While these methods effectively reduce the number of trainable parameters, they fail to resolve the critical memory bottleneck during on-device training. This is because backpropagation still requires loading the entire model and storing intermediate activations, resulting in a memory footprint as high as 70% of full fine-tuning [30,31]. This memory requirement is prohibitive for most edge devices, revealing a critical gap: parameter efficiency does not equate to training efficiency. This fundamental limitation motivates architectural-level solutions. Therefore, our proposed LESFT framework leverages model splitting to overcome the memory barrier that traditional PEFT cannot, enabling efficient learning at the edge.

To fine-tune LLMs on distributed edge data while preserving privacy, research has centered on two main approaches: Federated Learning (FL) and Split Learning (SL).

FL is a key paradigm for edge AI [32–34], but applying it to LLMs faces prohibitive communication and computation costs. A solution is Federated PEFT [35], where clients train and aggregate only lightweight modules. For example, FedLoRA requires exchanging only the LoRA adapters [36]. While this reduces communication, it fails to solve the local training bottleneck, as clients must still load the entire LLM. In contrast, SL directly tackles the local resource issue by partitioning the model. This property makes SL an attractive option for deploying LLMs on resource-constrained devices [37]. A natural extension is to combine SL with PEFT in methods like SplitLoRA, which mitigates the local bottleneck while keeping communication low.

The evolution from FL to SplitLoRA has progressively solved system-level bottlenecks related to communication, memory, and computation. However, these methods share a deeper limitation: their universal reliance on the conventional SFT paradigm. SFT is notoriously inefficient for complex reasoning tasks, often requiring extensive training samples and communication rounds for convergence. With system-level hurdles now largely addressed, this paradigm-level inefficiency emerges as the next critical barrier. Our work, the LESFT framework, is motivated by the need to overcome this very challenge by introducing a more data-efficient learning paradigm for LLMs at the edge.

Conventional response-level fine-tuning, or outcome-supervised learning, provides only sparse feedback from the final output, creating severe credit assignment challenges in complex reasoning tasks [38]. The model often fails to identify which reasoning steps are correct, and may even reach the right answer through flawed reasoning, a weakness reinforced by outcome-level supervision. In contrast, process supervision offers feedback to intermediate steps, yielding more reliable reasoning capabilities [39].

Building on this idea, token-level fine-tuning further refines supervision granularity by directly optimizing generated tokens. This enables more precise and stable learning and has been widely applied in SFT. However, studies show that even in high-quality datasets, many tokens are redundant or detrimental [40], diluting gradients and hindering performance. To address this, selective token optimization strategies estimate token contributions or use perplexity-based weighting to mask or down-weight non-informative tokens, thereby improving the signal-to-noise ratio and robustness [40,41].

Token-level techniques have also been extended to preference alignment. While Direct Preference Optimization (DPO) simplifies training, it ignores token-level variation in preference signals [42,43]. Token-Level DPO (TDPO) addresses this by modeling alignment as a token-wise Markov Decision Process, improving credit assignment and better matching the autoregressive nature of LLMs [44]. Further, Reinforced Token Optimization (RTO) integrates DPO with PPO, using dense token-level rewards from preference data to enhance policy learning and performance [45]. These advances highlight the effectiveness of token-level methods in improving accuracy and consistency for complex reasoning tasks.

Complementary to weighting and alignment approaches, Natural Language Fine-Tuning (NLFT) [41] strengthens reasoning by contrastively training on correct and incorrect reasoning chains. In this work, we adopt and extend this paradigm for few-shot reasoning in edge environments, simplifying the process to rely only on correct reasoning chains. This positive-sample strategy allows more efficient modeling of high-quality reasoning paths under limited data, improving both robustness and adaptability of LLMs.

As shown in Fig. 1, LESFT splits a traditional LLM architecture into three parts: an embedding layer, Transformer blocks, and a language modeling head. During inference, the input text is converted into a sequence of token IDs by the client-side tokenizer, and the embedding layer maps these IDs to continuous representations. The intermediate Transformer layers are deployed on the server to perform feature transformation, and finally, the language modeling head outputs a sequence of tokens in an auto-regressive manner.

To alleviate the memory bottleneck on edge devices, LESFT deploys the majority of its parameters in the cloud. The client retains only the embedding layer and some lightweight components. Specifically, after the input text is tokenized and embedded locally, it is sent to the server as a high-dimensional tensor. The server then performs the forward pass to output tokens. Since the server does not have a tokenizer, it cannot access the raw data. This mechanism structurally prevents raw data from leaving the local device, thereby ensuring user privacy while achieving computational offloading.

To further reduce the training overhead on the client, our framework introduces the LoRA technique. We partition the overall parameters into pre-trained base parameters, θ0, and lightweight, trainable adaptation parameters, Δθ. LoRA assumes that the parameter update ΔW during fine-tuning has a low intrinsic rank and can be decomposed into the product of two low-rank matrices, U and V. Therefore, only these low-rank parameters need to be updated during training, significantly reducing the memory and computational resources required by the client.

In this section, we define the problem and introduce the LESFT framework, which will be detailed in Section 3.3. The aim is to describe the system model of LESFT, laying a theoretical foundation for the subsequent sections. As illustrated in Fig. 2, we construct a typical scenario where edge users with limited computational resources fine-tune a model for complex reasoning tasks. This system comprises three core components:

Client Server: In this framework, a client is abstracted as an edge node with independent computing capabilities. We denote the set of all participating clients as 𝒩={1,2,…,N}, where N is the total number of clients. For any client i∈𝒩, its local private dataset is denoted as 𝒟i={(xi,k,yi,k,x~i,k,y~i,k)}k=1|𝒟i|, where |𝒟i| is the total number of samples. In this context, xi,k denotes the user’s prompt input, while yi,k represents the corresponding desired output from the model, which contains the Chain-of-Thought (CoT) reasoning [46]. x~i,k is a reference input formed by combining the user’s prompt with the ground-truth answer; its purpose is to provide a constraining signal in contrastive learning, thereby helping the model generate the target reasoning yi,k. Meanwhile, y~i,k is identical to yi,k. We assume that each edge node can independently perform local model forward propagation and backward propagation operations. The local model weights on each client are denoted as Wc, within which only a set of trainable LoRA adapter parameters is designated as the trainable part. This parameter set is denoted as Θi={(Ui(n),Vi(n))}n=1Mc, where Mc is the number of trainable LoRA adapters on the client side, and Ui(n) and Vi(n) are the low-rank decomposition matrices of the n-th LoRA adapter on the edge device i.

We represent the overall model parameters as W=[Ws;Wc], which consists of the server-side parameters Ws and the client-side parameters Wc. The training objective is to learn the optimal client adapter parameters {Θi}i=1N and server-side adapter parameters Φ by minimizing the weighted average of local losses across all clients. This optimization process can be formulated as: (1)min{Θi}i=1N,Φ ∑i=1N|𝒟i||𝒟|ℒi(W;Θi,Φ)where 𝒟=⋃i=1N𝒟i is the union of all local datasets. The term ℒi(⋅) represents the loss for client i on its local data 𝒟i, computed by the model configured with the base parameters W and the adapters Θi and Φ. The weighted aggregation provides a unified global objective that integrates updates from multiple clients.

This section details the overall workflow of the proposed LESFT framework. LESFT’s core innovation, which sets it apart from existing methods, is a fine-grained, token-level fine-tuning mechanism. We integrate this mechanism into a hierarchical split collaborative fine-tuning architecture. This approach not only enhances the model’s expressive power but also effectively reduces resource overhead. Simultaneously, LESFT incorporates the PEFT technique, LoRA, to further enhance the adaptability and efficiency of the training process.

During the training initialization phase, the central server first processes the base weights of the large model to be fine-tuned and partitions them into a server-side sub-model and a client-side sub-model to accommodate the distributed computational resources. Subsequently, LESFT performs distributed fine-tuning over I consecutive local training rounds. In each round, the clients update only their local adapter parameters, while the server updates its assigned model parameters. After completing I iterations, the aggregation server integrates the sets of LoRA adapter parameters, Θi, uploaded by each client to obtain the global adapter parameters, Θ¯, and distributes them to all clients to serve as the initialization for subsequent training. This process is repeated until the global model converges or the maximum number of training rounds, R, is reached.

Overall, the training process of LESFT consists of two main phase: (i) the split natural language fine-tuning stage, which is executed in every training round; and (ii) the client adapter aggregation stage, which is triggered every I rounds. Fig. 2 illustrates the overall workflow of LESFT, where the training round index is r∈ℛ={1,2,…,R}.

Phase 1. Split Natural Language Fine-Tuning Phase: The split natural language fine-tuning phase involves client-side and server-side fine-tuning performed by the participating clients and the central server in each training round. This phase consists of the following seven steps.

1. Client-Side forward Propagation: In this step, all participating clients i∈𝒩 perform forward propagation on their local sub-models in parallel. Specifically, each client i randomly samples a mini-batch of size B, denoted as ℬi(r)={(xi,j(r),yi,j(r),x~i,j(r),y~i,j(r))}j=1B, from its local dataset 𝒟i. This batch contains two sets of input data: the base input Xi(r)={xi,j(r)}j=1B and a reference input for contrastive learning, X~i(r)={x~i,j(r)}j=1B.

The client sub-model consists of fixed pre-trained weights Wc and trainable LoRA adapters Θi(r)={(Ui(n,r),Vi(n,r))}n=1Mc specific to client i in round r. After feeding both sets of inputs, Xi(r) and X~i(r), to the client sub-model, activations are generated at the cut layer. The computation is as follows: (2)Si(r)=ϕ(Wc|Θi(r−1),Xi(r)) (3)S~i(r)=ϕ(Wc|Θi(r−1),X~i(r))

here, ϕ(W|Θ,X) and ϕ(W|Θ,X~) represent the mapping from the joint processing of the base input X and the reference input X~ to the predicted values at the cut layer, given the model parameters W and the set of trainable LoRA adapters Θ. Finally, these activations, Si(r) and S~i(r), are transmitted to the central server for subsequent computations.

2. Upload of Intermediate Activations and Supervision Signals: After the client completes its local forward propagation, each client i uploads its two generated sets of activations, Si(r) and S~i(r), along with the corresponding mini-batch labels Yi(r) and Y~i(r), to the central server via a communication link. The server uses these collected activations as input for its server-side sub-model to proceed with the subsequent training steps.

3. Central Server Forward Computation: Upon receiving the activations and labels from all participating clients, the central server feeds these activations into its server-side model to perform the server-side forward pass. The concatenated activation matrices, 𝒮(r) and 𝒮~(r), are represented as: 𝒮(r)=[S1(r);S2(r);…;SN(r)] and 𝒮~(r)=[S~1(r);S~2(r);…;S~N(r)]. Subsequently, the server inputs these two aggregated activation matrices into its server-side sub-model. The server sub-model consists of fixed pre-trained weights Ws and trainable LoRA adapters Φ(r)={(U(m,r),V(m,r))}m=1Ms for round r. Using the server-side mapping function ψ(⋅), the final predictions are calculated: (4)Y^(r)=ψ(Ws(r)|Φ(r−1),𝒮(r)), (5)Y~^(r)=ψ(Ws(r)|Φ(r−1),𝒮~(r)),where Y^(r) and Y~^(r) are the model’s reasoning results corresponding to the client base inputs and reference inputs, respectively.

4. Token-Level Loss Computation: After the server-side model computes the reasoning results, the model base predictions Y^(r) and reference predictions Y~^(r) and the ground-truth labels Y(r) and Y~(r) are used to calculate an improved token-level loss function, ℒ(r). This loss function assigns different weights to each token to guide the model in learning reasoning logic. Its specific formulation will be detailed in Section 3.4.

5. Central Server Gradient Computation and Parameter Update: After computing the token-level loss ℒ(r), the central server performs backward propagation to calculate the gradients for the server-side LoRA adapter parameters. Specifically, for the m-th server-side LoRA adapter, the gradients for its decomposition matrices U(m) and V(m) are denoted as GU,s(m,r) and GV,s(m,r), respectively. These parameters are then updated using a gradient descent algorithm: (6)U(m,r)←U(m,r−1)−γs⋅GU,s(m,r), (7)V(m,r)←V(m,r−1)−γs⋅GV,s(m,r),where γs is the learning rate on the server side. This update process ensures that the server-side adapters can effectively learn from the aggregated information from all clients.

6. Backward Transmission of Gradients to Clients: After the server completes its backward propagation and updates its LoRA adapter parameters, it computes the gradients of the loss function ℒ(r) with respect to the input activations 𝒮(r) and 𝒮~(r), which are ∇𝒮(r)ℒ(r) and ∇𝒮~(r)ℒ(r). Subsequently, the server partitions these gradients according to their client origin and transmits them to the corresponding participating clients. Specifically, each client i receives its corresponding gradient components ∇Si(r)ℒ(r) and ∇S~i(r)ℒ(r), which will serve as the input for the continued backward propagation on the client’s local model.

7. Client-Side Local Parameter Update: In this step, each client, based on the received activation gradients, continues the backward propagation process on its local sub-model to update the client-side LoRA adapter parameters. For a client i, the decomposition matrices Ui(n) and Vi(n) of its n-th LoRA adapter are updated via gradient descent: (8)Ui(n,r)←Ui(n,r−1)−γc⋅GU,i(n,r), (9)Vi(n,r)←Vi(n,r−1)−γc⋅GV,i(n,r),where GU,i(n,r) and GV,i(n,r) are the gradients computed in the current training round r for the matrices Ui(n) and Vi(n) of the n-th LoRA adapter of client i, and γc is the local learning rate for the client. This process allows clients to optimize their local adapter parameters under the guidance of the global loss signal, achieving a more collaborative and efficient distributed fine-tuning.

Phase 2. Periodic Global Adapter Fusion Cycle: The client adapter aggregation phase is primarily executed by the aggregation server. Its core objective is to integrate and fuse the local LoRA adapter parameters uploaded by each client to achieve global knowledge sharing and model performance improvement. This phase is executed once every I training rounds and consists of the following three steps:

8. Upload of Local Adapter Parameters: In this step, all participating clients i∈𝒩 upload their current client-side LoRA adapter parameter sets, Θi(r)={(Ui(n,r),Vi(n,r))}n=1Mc, to the aggregation server via wireless or wired links. This process only transmits the adapter parameters and does not involve any raw data, thereby ensuring that user privacy is protected.

9. Weighted Federated Aggregation: Upon receiving the adapter parameters from all clients, the aggregation server performs a weighted average of their LoRA parameters based on the size of each client’s local dataset to generate a globally unified client LoRA adapter, Θ¯={(U¯(n),V¯(n))}n=1Mc. Specifically, the decomposition matrices of the n-th LoRA adapter are aggregated as follows: (10)U¯(n)=∑i=1N|𝒟i||𝒟|Ui(n,r), (11)V¯(n)=∑i=1N|𝒟i||𝒟|Vi(n,r).

This weighted aggregation strategy accounts for the differences in data distribution among clients, which helps to improve the generalization ability and convergence stability of the global model.

10. Distribution of Global Adapter Parameters: After the aggregation is complete, the aggregation server distributes the global adapter parameters Θ¯ back to each participating client. Upon receiving Θ¯, each client uses it as the initial parameters for its local adapters in the next training phase. This step ensures that all clients proceed with fine-tuning from a consistent global starting point in subsequent training, thereby promoting the overall consistency and efficiency of the federated training.

The overall training process of LESFT is summarized in Algorithm 1.

Through the algorithmic process outlined above, the LESFT framework effectively coordinates the computational resources of edge devices and cloud servers while preserving data privacy, providing a systematic solution for the efficient fine-tuning of LLMs in edge environments. The framework not only reduces the computational burden on the edge side but also significantly enhances training efficiency in few-shot scenarios through its natural language fine-tuning mechanism.

Traditional SFT optimizes by minimizing the cross-entropy loss between predictions and ground-truth labels. It treats all tokens in a reasoning chain equally, which can easily lead the model to rely on pattern memorization while neglecting logical deduction. Natural Language Fine-Tuning (NLFT) builds upon this by introducing dynamic weighting and language feedback. On one hand, it identifies salient tokens from positive and negative samples’ reasoning chains to enhance the model’s sensitivity to reasoning. On the other hand, it uses natural language feedback to locate critical steps and penalize erroneous ones, shifting the optimization objective from simple answer fitting to capability-guided learning.

NLFT assumes that answers generated by a large model can be categorized as either correct or incorrect, and it identifies salient tokens by comparing different prompts. However, in practical edge fine-tuning, model outputs are often partially correct and difficult to classify as entirely wrong. In a translation task, for example, multiple results may be semantically correct but differ in fluency and style, making it hard to determine a clear winner. Furthermore, NLFT’s approach, which involves constructing positive and two distinct negative reasoning paths, effectively triples the processing load per sample and significantly increasing computational power requirements and latency.

To address this, we propose an improved scheme that constructs only two paths: a base input and a reference input. This design reduces redundant computation. The base input provides minimal context, while the reference input serves as a quality comparator to guide the model’s learning. This introduces soft prior knowledge and forms a directional constraint on the original task, maintaining the effectiveness of contrastive learning while significantly reducing latency and computational overhead.

For a mini-batch of samples ℬi(r)={(xi,j(r),yi,j(r),x~i,j(r),y~i,j(r))}j=1B drawn by client i in round r, the server receives activations Si,j(r) and S~i,j(r) from the client and generates two sets of predictions: (12)Y^i,j(r)=ψ(Ws(r)|Φ(r−1),Si,j(r)), (13)Y~^i,j(r)=ψ(Ws(r)|Φ(r−1),S~i,j(r)).

Let the target output sequence be yi,j(r)={yi,j,1(r),…,yi,j,Ti,j(r)}. For a token at any position t, we define the conditional probabilities under the two paths: (14)Pi,j,t(r)=P(yi,j,t(r)∣xi,j(r),yi,j,<t(r); Wc,Θi(r−1),Ws(r),Φ(r−1)), (15)P~i,j,t(r)=P(yi,j,t(r)∣x~i,j(r),yi,j,<t(r); Wc,Θi(r−1),Ws(r),Φ(r−1)).

When analyzing the reasoning chain generated by the model, we first identify Salient Tokens based on the conditional probability of the reference path, P~i,j,t(r). Specifically, if the probability P~i,j,t(r) of a token yi,j,t(r) exceeds a predefined threshold psal, it is classified into the salient token set, Yi,j,sal(r). Next, using these salient tokens as cluster centers, we apply cosine similarity–based semantic clustering to group semantically related tokens, which are then labeled as Sub-salient Tokens and placed in the set Yi,j,subsal(r). Finally, tokens not assigned to either category are regarded as Irrelevant Tokens and are collected in the set Yi,j,irrel(r).

For these three categories of tokens, we define a scaling weight S(yi,j,t(r)) based on the reference path probability: (16)S(yi,j,t(r))={1+(P~i,j,t(r)−psal1−psal)γ1,yi,j,t(r)∈Yi,j,sal(r),(P~i,j,t(r)psal)γ2,yi,j,t(r)∈Yi,j,subsal(r),(P~i,j,t(r)psal)γ3,yi,j,t(r)∈Yi,j,irrel(r),where γ1,γ2,γ3 are tunable hyperparameters, typically set with γ2<γ3 so that sub-salient tokens receive higher weights than irrelevant tokens. For salient tokens, the formulation includes an additive constant 1, which ensures that their scaling value is always greater than 1, thereby amplifying their influence in the gradient computation during loss optimization.

After obtaining the scaling weight for each token, we use a weighted cross-entropy on the predicted probabilities Pi,j,t(r) of the model’s reasoning chain as the final loss: (17)ℒi,j(r) = −1|yi,j(r)|∑t=1|yi,j(r)|S(yi,j,t(r))log⁡Pi,j,t(r).

This weighted design preserves the emphasis on logically critical tokens from NLFT while using only two input paths, thereby significantly reducing the additional computational and communication overhead in edge scenarios.

The loss calculation process of LESFT is summarized in Algorithm 2.

This section has detailed the overall architecture, training process, and core token-level loss calculation mechanism of the LESFT framework. By introducing natural language fine-tuning into a split federated fine-tuning architecture, the framework provides a systematic solution for the efficient fine-tuning of large models in edge environments. To validate the practical performance and effectiveness of our proposed framework, the next chapter will present a series of comprehensive comparative experiments and analyses conducted on public benchmark datasets.

To evaluate the generalizability and performance of the proposed framework, this study utilizes three distinct reasoning benchmarks. To simulate data-constrained edge scenarios and test few-shot learning efficiency, we uniformly and randomly sampled 800 instances from the official training set of each dataset.

GSM8K dataset. This dataset contains high quality elementary school level math word problems. The official training split comprises 7473 instances and the official test split comprises 1319 instances.

CommonsenseQA dataset. This dataset is designed for commonsense question answering and consists of multiple choice questions. The official training split comprises 9741 instances and the official test split comprises 1140 instances.

AQUA_RAT dataset. This dataset targets algebraic question answering with rationale annotations and consists of programmatic and textual solutions. The official training split comprises 97467 instances and the official test split comprises 254 instances.

Considering that the base models used in our experiments, such as Qwen2.5-3B, do not possess robust mathematical reasoning capabilities before fine-tuning, it is not feasible to have them directly generate the reasoning chains required for training. Therefore, we adopt a data distillation strategy, using a more powerful teacher model to generate high-quality fine-tuning data. Specifically, we utilize the Llama-3-8B-Instruct model, which achieves an accuracy of 77% on GSM8K, 79% on CommonsenseQA, and 81% on AQUA_RAT, to ensure the reliability of the generated supervision signals. For each dataset, we randomly select 800 samples correctly answered by the teacher model to construct the fine-tuning subset. The selected responses, combined with Chain-of-Thought prompting, provide detailed reasoning chains that serve as the ideal user outputs the student model needs to learn. Concurrently, the original solution steps from each dataset are used as the reference input to provide the constraint signal, thereby satisfying the dual-path input requirement of the LESFT framework.

To comprehensively evaluate our framework, we compare it against four representative fine-tuning methods, spanning centralized to distributed paradigms.

Centralized Supervised Fine-Tuning (Centralized SFT). This baseline reflects the upper-bound performance in a non-privacy-preserving setting, where a central server directly fine-tunes the LLM with LoRA on the entire training set, comprising 800 GSM8K samples in our study.

Federated Averaging (FedAvg). The canonical algorithm in federated learning, adapted here for LoRA-based fine-tuning. In each round, clients train local adapters on their data and upload updates to the server, which aggregates them via weighted averaging. Raw data remain local, ensuring privacy.

Split Learning (SL). A vertical partitioning strategy in which clients hold the front layers and the server holds the remaining ones. Clients send intermediate activations to the server, which completes the forward pass, computes the loss, and returns gradients. No aggregation across clients is performed, making SL suitable for evaluating performance without horizontal knowledge sharing.

SplitLoRA. An advanced variant combining split learning with federated aggregation. Adapter parameters are decomposed into shared and private parts: shared layers are collaboratively trained across clients, while private layers are updated locally to preserve personalization. This design balances knowledge sharing and data heterogeneity.

SplitFrozen [47]. This method is a split learning framework where the initial model layers deployed on client devices are frozen. Clients execute only a forward pass and transmit the resulting activations to a central server. The server holds the remaining layers and manages all training updates, applying parameter-efficient fine-tuning via LoRA exclusively to its portion of the model. This configuration avoids client-side backward propagation and can be combined with pipeline parallelism to reduce device idle time.

We conducted experiments on three models from the Qwen2.5 series with different parameter scales: Qwen2.5-0.5B, Qwen2.5-1.5B, and Qwen2.5-3B. These cover lightweight to medium-size configurations and enable validation under different resource constraints. All experiments used the original pre-trained versions to avoid potential bias introduced by instruction tuning.

A vertical splitting strategy was adopted to simulate edge devices with limited resources. The client was assigned the tokenizer, the embedding layer, and the first quarter of the Transformer layers, while the server processed the remaining layers and the language model head. This allocation reduced the computational burden on the client and enhanced data privacy. The same splitting strategy was applied to all baselines to ensure fairness in comparison.

All experiments were performed on a unified high-performance computing platform equipped with an NVIDIA RTX 4090 GPU, PyTorch 2.3.0, and CUDA 12.1. To guarantee consistency, we used AdamW as the optimizer with a learning rate of 5e−5. The batch size per client was set to 1 with gradient accumulation of 4, resulting in an effective batch size of 4. Local training was conducted for 10 epochs on a dataset of 800 samples. Four clients were simulated, and evaluation was performed using identical prompts and validation protocols.

The LoRA configuration was kept uniform across all experiments. The rank was set to 8, lora_alpha to 16, and the dropout rate to 0.2. LoRA was applied to the gate_proj, down_proj, and up_proj linear layers. The salient token threshold was fixed at 0.95, with hyperparameters γ1=5, γ2=0.3, and γ3=0.6.