FL is a distributed machine learning paradigm that enables clients to collaboratively train a global model without sharing raw data, thereby balancing privacy protection and model performance. FL has been widely applied in financial risk control, medical diagnosis, and personalized recommendation. With ongoing research progress, efforts have focused on key techniques such as optimization algorithms, communication compression, and privacy enhancement, and have introduced contrastive learning, transfer learning, and data augmentation to cope with complex data scenarios. For example, Ref. [20] improves system robustness and interpretability via visual digital twins, but it relies on complicated data assimilation and simulation algorithms and is difficult to generalize to cross-domain and few-shot settings. Communication-compressed FL methods [21] reduce communication by structured pruning. However, when dealing with multi-source heterogeneous images, sparsification may discard a small amount of critical discriminative features, limiting final classification accuracy. Image clustering and joint representation learning frameworks [22] alleviate the impact of Non-IID data on classifier training to some extent, but they lack explicit identification of key discriminative subspaces, which constrains their ability to distinguish scarce classes.

To address these issues, a more robust form of locally discriminative yet shareable knowledge is needed. This paper proposes the FKD-RTM method, which adopts an intuitive architectural design and focuses on learning local features under complex data conditions caused by feature heterogeneity, scarce classes, and few-shot data, where distilled knowledge serves as additional guidance. Therefore, the FKD-RTM is expected to achieve stronger fitting ability in complex-data scenarios and better generalization performance for classification tasks.

GBDTs are a strongly nonlinear ensemble model composed of multiple decision trees, which improves overall predictive capability by continuously fitting the errors (residuals) from previous iterations. In FL, the presence of few samples and missing classes often leads to limited and inconsistent client datasets, making conventional neural networks prone to oscillation and overfitting during local training, thereby restricting performance. For instance, a global-attention method [23] improves few-shot classification focus through global attention and key localization, but it often ignores fine-grained local neighborhood relationships and neighborhood consistency, resulting in limited performance in local or noisy scenarios. A multi-channel local-description and context-enhancement network [24] improves few-shot discrimination by strengthening joint local–context representations, yet it remains limited in explicitly selecting key discriminative subspaces and in efficient computation, making it difficult to balance accurate neighborhood modeling and inference overhead. Although lightweight models [25,26] reduce parameters and computation via model compression and multi-scale fusion, they often sacrifice neighborhood information fusion and cross-scale consistency modeling, which limits classification performance in application scenarios with missing classes or subtle inter-class differences.

To alleviate these issues, this paper introduces GBDT in the FKD-RTM method and use it as the local teacher model. Compared with conventional neural networks, GBDT can maintain stable discriminative capability under few-shot, class-missing, and distribution-biased conditions (i.e., complex data scenarios). As a result, the method adopts GBDT as the teacher because it can still preserve strong fitting ability in complex environments, improving model adaptability and generalization, and enhancing practicality for deployment.

MLP is a lightweight neural network model composed only of linear layers, activation layers, and normalization layers. In FL, to reduce communication, researchers introduce knowledge distillation and typically adopt neural networks as teacher and student models. Knowledge distillation uses soft labels to extend supervision from hard labels to soft labels, enhancing learnability under few-shot settings; meanwhile, communication still relies on parameter aggregation, which can greatly reduce communication cost with only minor loss of performance. However, in complex data scenarios, distillation with neural-network students has clear limitations. For example, a distillation-based method [27] enhances generalization under few-shot classification and class imbalance by aligning the probability distributions of the teacher and student. Yet, when the feature space is sparse or local classes are missing, the student may fail to learn the teacher’s distribution and local neighborhood relationships in these sparse regions, leading to alignment failure and degraded performance.

Compared with conventional convolutional neural network (CNN) students, MLPs are lighter, train faster, and have stronger fitting ability. More importantly, MLPs are easier to integrate with partial fine-tuning, and are thus more suitable for cross-domain knowledge distillation in the feature space. GBDT is a piecewise strong decision model, whereas the convolutional structure of CNNs tends to be locally smooth and spatially biased, making it less compatible with fixed feature representations in this setting. Accordingly, FKD-RTM adopts an MLP as the student: the mapping of an MLP can approximate the piecewise function of GBDT and mimic its logical decision process, enabling the student to better match the teacher’s discriminative capability and alleviating possible alignment failure and performance degradation during distillation.

The feature extractor is responsible for transforming image data from all clients into a shared representation space. However, in FL, client data often exhibit complex conditions such as missing classes and limited samples. If the feature extractor is fully frozen, the shared representation may not fit extreme clients, preventing the student model from adequately fitting local data. Therefore, we freeze most parameters of the feature extractor and only allow the last few layers to be fine-tuned, updating them with each client’s local data to adapt to its own distribution. The shared representation mitigates cross-client representation misalignment, while partial fine-tuning addresses the mismatch for extreme clients [7]. Recent studies on personalized federated learning have also shown that relying only on a unified shared representation is often not enough to handle distribution differences across clients. Targeted local adaptation based on client feature distributions is still needed. For example, pFedFDA [28] combines shared feature representation with personalized classifier adaptation from the perspective of feature distribution adaptation. It achieves strong performance under covariate shift and local data scarcity. This further suggests that combining shared feature representation with local personalized adaptation is an effective direction. Accordingly, the FKD-RTM adopt this design so that partial fine-tuning of the feature extractor remains a core advantage of the distillation method and can achieve stronger adaptability and higher personalization performance while preserving communication efficiency and privacy protection [29].

Residual enhancement incorporates a model’s own residual information during prediction, enabling the model to maintain stability and plasticity during knowledge distillation and feature adaptation [30]. During distillation, the global MLP student can only passively smooth toward soft labels. In addition, partial fine-tuning of the feature extractor can easily introduce shifts, which may cause large fluctuations in client features or soft labels and lead to overfitting to local distributions, thereby undermining the advantages brought by GBDT-based distillation. To further strengthen client-side local adaptation while preserving global consistency, we introduce residual enhancement to prevent overfitting and improve local adaptability [31]. For feature representations after partial fine-tuning or for student logits (pre-normalization scores), the model does not directly replace the global output; instead, it treats them as residual supplement to the global model. Accordingly, the FKD-RTM design is highly compatible with the GBDT-to-MLP distillation structure and the partial fine-tuning strategy, and serves as an important component of our method.

By combining tree models and MLPs, we integrate the advantages of tree ensembles in structured modeling with the strong nonlinear expressiveness of MLPs into the federated knowledge distillation method. The key principle is to decouple robust local teaching from globally aggregatable student learning. The FKD-RTM design can significantly improve the training performance of cross-model distillation. It reduces the reliance on homogeneous teacher–student architectures and provides an efficient solution for heterogeneous knowledge distillation.

Specifically, under limited and heterogeneous client data, tree models can quickly fit local data and provide stable probability estimates even in few-shot and class-missing conditions. Their nonlinear partitioning in the feature space captures local decision boundaries, producing informative and stable soft labels for the student. Meanwhile, MLPs can reliably align to the soft labels produced by an ensemble teacher, remain lightweight for efficient communication and local fine-tuning, and approximate the piecewise decision boundaries of gradient-boosted trees in a shared feature space. With partial fine-tuning, the MLP student can further adapt to local distributions. Overall, this framework improves generalization under complex data conditions and supports robust FL.

First, we train a GBDT on the shared feature representations to better handle few-shot settings and locally missing classes. GBDT is trained in a stage-wise manner by iteratively fitting the residuals from the previous iteration. On client k, we learn a local GBDT ensemble model Tk using the embedded local data. After training, the teacher outputs a per-class score (or probability) for each sample:(1)Tk(hi)=∑mMRmTm(h),

Here, for a given embedded sample h on client k, Tm:Rd→Rc denotes the vector output of the m-th tree (typically producing one score for each class), and Rm is the learning rate. During training, we apply a probability transformation function f(∙)=softmax(∙) to convert the raw tree outputs in Eq. (1) into a probability distribution in Eq. (2):(2)pk(hi)=softmax(Tk(hi))∈ΔC−1,pk(hi)∈RC,

Although GBDT provides more robust local supervision than neural teachers under complex heterogeneous settings, its output probabilities may still be miscalibrated and over-confident. Calibration aims to align teacher confidence with empirical correctness, making the predicted probabilities more reliable, i.e., a prediction with confidence p should be correct approximately p proportion of the time. If the probabilities produced by the tree model in Eq. (2) are used without calibration, they may become over-confident and skew toward extreme values, which can amplify training discrepancies and lead to poor convergence or even training failure. Therefore, probability calibration is necessary for the teacher outputs. Each client fits a calibration function Ck(∙) using its local validation set Dkval, implemented as Platt scaling in our method, and the resulting calibrated probabilities are further transformed as in Eq. (3). We further compare FKD-RTM with and without calibration in the ablation study, showing that local calibration improves the stability and effectiveness of teacher supervision under heterogeneous data settings. Temperature scaling, in contrast, controls the sharpness of the calibrated distribution and exposes richer inter-class relations, so that the student can better absorb dark knowledge from the teacher. To obtain a smoother and more informative target distribution, we further apply temperature scaling to the calibrated teacher outputs by transforming the original predicted probabilities and then performing temperature-based rescaling. Therefore, in FKD-RTM, calibration improves the correctness of the teacher probabilities, whereas temperature scaling improves their learnability for distillation. This yields the temperature-smoothed probabilities in Eq. (3):(3)p~k(hi)=exp⁡(log⁡(p^k(hi)T))∑cexp⁡(log⁡(p^c(hi)T)),p^k(hi)=Ck(pk(hi)),p^c(hi)=Cc(pc(hi)),

Second, on each client we obtain teacher soft targets from the trained teacher model and distill them into a lightweight MLP student model Sϕ(∙). The student network consists of multiple fully connected layers. Given the feature representations of local training samples, the student is trained to minimize the Kullback-Leibler (KL) divergence between the student predictive distribution and the tree teacher’s soft targets, thereby measuring and maximizing their distributional similarity. In this way, the student can efficiently align to the ensemble teacher’s soft-label distribution, as formulated in Eq. (4). (4)LKD(ϕ)=1M∑mMKL(softmax(Sϕk(hm)T)‖∑kp~k(hm)),

Here, p~k(h) denotes the teacher’s temperature-smoothed soft targets, and T is the temperature hyperparameter. Moreover, Sϕk(h) represents the student’s local predictive distribution on client k. To ensure that training remains faithful to the ground-truth labels, we further incorporate a cross-entropy loss term:(5)LCE(ϕ)=−1M∑m=1M∑c=1Cym,clog⁡(softmax(Sϕk(hm)))c,

The overall objective for training the student model is a weighted combination of the hard-label loss in Eq. (5) and the soft-label KL-divergence loss in Eq. (4), resulting in the total loss in Eq. (6). Here, α denotes a weight coefficient that is linearly adjusted with respect to local training rounds, and is used to balance the contributions of the hard-label supervision and the temperature-smoothed distillation term. (6)L(ϕ)=(1−α)LCE(ϕ)+αLKD(ϕ),

Finally, we apply a channel compression strategy in the last layer to reduce the feature dimensionality to the target output dimension, thereby lowering model complexity and improving inference efficiency.

Comparison with Conventional Distillation-Based Approaches: Unlike conventional federated distillation methods that align soft targets at the server side, the proposed FKD-RTM approach performs the entire distillation process locally on each client and only uploads the parameters of the student model and the residual model. As a result, the server only needs to conduct parameter-level aggregation, which preserves the benefits of distillation while avoiding the additional communication overhead and potential privacy risks associated with transmitting predictive distributions. Conventional distillation typically assumes identical or similar neural network architectures for the teacher and student. In such settings, the teacher essentially learns a continuous and differentiable distribution optimized via backpropagation, which often requires substantial data to fit high-quality decision boundaries and is less robust under few-shot or class-imbalanced conditions. Moreover, the teacher’s soft targets can be over-confident and become unstable in Non-IID or noisy scenarios. In contrast, tree-based models are naturally effective for few-shot learning, class imbalance, and other forms of complex data, and can provide more robust probability estimates than convolutional neural network (CNN) teachers, yielding more reliable distillation signals for the student. The outputs of tree teachers exhibit an inherent piecewise decision structure, making them well suited as a source of soft supervision in federated environments. Meanwhile, MLPs have strong nonlinear expressive power and can efficiently approximate the piecewise decision boundaries of gradient-boosted trees, leading to high compatibility with the soft targets produced by tree ensembles.

Novelty and Advantages: The proposed cross-architecture distillation framework FKD-RTM, which employs a tree-based teacher and an MLP student, constitutes a form of heterogeneous knowledge distillation under a cross-model paradigm. The piecewise partitioning structure of tree models complements the continuously differentiable representation capacity of MLPs, making the distillation signal more expressive and more amenable to learning. Moreover, the inherent properties of GBDT enable each client to generate complementary and diverse teacher knowledge. Through knowledge distillation, when student models are aggregated at the server, the global student can capture global decision structures that are difficult to obtain via conventional Federated Averaging (FedAvg) [1] or conventional neural-teacher distillation. In addition, by aligning predictive probabilities on a shared feature representation space, our method alleviates failures of feature-space alignment, while benefiting from more stable teacher outputs and more easily absorbable supervision for the student.

The feature extractor serves as the shared backbone network of the overall distillation framework, mapping the raw image input x into a stable feature representation space h that is shared across all clients. Neither the tree teacher nor the MLP student operates directly on raw images; instead, both models rely on this shared representation space for training and inference.

First, given a client sample x, we extract its representation using the feature extractor fθ(∙) with shared parameters θ, the feature extractor is globally shared across all clients. obtaining an embedding that can be directly used to train the tree model, as shown in Eq. (7). During the first-stage distillation, the feature extractor is fully frozen to preserve a consistent representation space and to avoid additional communication of extractor updates; it does not participate in client-side local optimization. Although client data distributions differ, all clients share the same feature extractor, which ensures that they operate within a consistent semantic embedding space:(7)hk,i=fθ(xk,i),hk,i∈Rd,∇θ,

The tree models trained on clients can fit their local data within a consistent representation space, producing teacher soft targets with the same dimensionality across clients. Accordingly, the student model can be trained directly in this representation space under the supervision of the teacher soft targets, enabling knowledge sharing after aligning client representations.

Local tree training is performed on the feature representations Tk:Rd→Rc. This allows the piecewise, nonlinear, and locally adaptive strengths of tree ensembles to be more effectively exploited in the representation space. The stability of the learned representations also prevents the tree model from collapsing due to noise in raw pixel inputs.

During client-side distillation of the student model, the same feature representations extracted by the shared extractor are used as inputs. Since the student does not need to process raw images, the computational cost of knowledge distillation is substantially reduced, as indicated in Eq. (8):(8)Sϕ(hk,i)=Sϕ(fθ(xk,i)),

Second, in federated settings with highly Non-IID data, missing classes, or long-tailed samples, single-stage distillation often exhibits systematic bias in regions containing hard examples. To further compensate for the limitations of the student model, we introduce a residual enhancement module. During the residual and personalization stage, a partial fine-tuning module is applied when the global residual model is broadcast to clients and used to assist student training. This enables lightweight local adaptation of the residual signals, making the model better fit each client’s complex data environment and improving local performance. To adapt the client-personalized model to the local distribution, partial fine-tuning unfreezes the last several layers of the feature extractor, only the last few high-level layers are unfrozen for local adaptation, while the lower layers remain fixed. These locally fine-tuned parameters are not uploaded to the server, which enables lightweight personalization without affecting the global model.

We unfreeze the last several layers of the feature extractor and the MLP, while freezing all remaining layers. This can be expressed as θ={θfrozen,θadapt}, where θfrozen denotes the lower-level feature parameters that remain fixed, and θadapt denotes the trainable high-level feature layers together with the classification head. Partial fine-tuning is performed using local ground-truth labels via the supervised objective in Eq. (9), without requiring an additional distillation step:(9)LFT=−∑iyk,ilog⁡(softmax(Sϕ(fθ(xk,i)))),

To prevent personalization updates from contaminating the global model, partial fine-tuning is performed only on the client side, and the fine-tuned parameters are excluded from federated aggregation. Without altering the global model, this strategy allows different clients to update in different directions for personalized adaptation, while also reducing computational overhead.

Novelty and Advantages: Unlike conventional federated distillation methods that directly rely on raw features or raw model outputs, the proposed FKD-RTM approach introduces a globally shared feature extractor that maps heterogeneous client data into a unified representation space, as shown in Eq. (7). This alleviates a core issue in conventional federated distillation—namely, the misalignment of representations produced by different client teachers—thereby ensuring global consistency. The feature extractor remains fully frozen during most of the training process, which avoids synchronizing extractor parameters and thus reduces communication. Partial fine-tuning preserves globally shared semantic information while substantially lowering local training cost and preventing the global model from being dominated by highly heterogeneous clients. As a finer-grained and more stable personalization strategy, it is more lightweight than existing approaches such as [7,32]. Finally, partial fine-tuning is conducted entirely on-device, which improves security and further reduces communication overhead.

During the client-side heterogeneous distillation stage, the student model Sϕ(∙) has already learned the major decision boundaries under the supervision of the tree teacher Tk(∙) via its soft targets. However, under highly Non-IID data, missing classes, or long-tailed distributions, single-stage distillation often exhibits systematic bias in regions containing hard examples. To further capture the complementary knowledge not yet absorbed by the student, we introduce a second-stage complementary knowledge correction mechanism, namely a residual enhancement module Rg implemented by class-wise local GBDT learners. It explicitly models the teacher-student discrepancy in the form of logit residuals, enabling residual-based compensation and local personalization in the second stage.

Using the features extracted in the representation space in Eq. (7), the teacher produces soft targets, which are further converted into probabilistic outputs as in Eqs. (1)–(3). After the first-stage distillation, both the teacher soft targets and the student outputs are mapped into the logit-residual form. The residual in FKD-RTM is defined in the pre-normalization score domain rather than directly in the probability space. In the probability space, vectors are constrained on the simplex, so changes in one class are strongly coupled with changes in others. This makes additive correction less stable. By contrast, the pre-normalization score domain is more flexible for linear residual compensation. It allows the residual model to explicitly capture the teacher–student discrepancy before normalization. Specifically, the teacher soft targets are obtained from Eq. (3), while the student outputs logits on the shared representations and applies a softmax function to obtain the predictive distribution ps(hk,i) as in Eq. (4). Based on these quantities, we define a global residual learning objective to capture the remaining knowledge that the student has not absorbed:(10)r(hk,i)=log⁡(p~k(hk,i)+ε)−((Sϕ(hk,i))+ε),

To improve training stability, we apply temperature scaling to the raw residual in Eq. (10) and introduce a small stabilization constant ε to avoid numerical issues caused by extremely small probabilities when transforming teacher outputs. We analyze the parameter sensitivity of the stabilization constant ε. The results show only minor fluctuations, indicating that FKD-RTM is not sensitive to ε within a reasonable range. This yields the temperature-smoothed residual in Eq. (11). This temperature-based modeling makes residual learning more stable and controllable:(11)r~(hk,i)=r(hk,i)/Tr,

In FKD-RTM, the local residual model is implemented as a class-wise GBDT residual learner. Specifically, for each client, one residual regressor is trained for each class in the pre-normalization score domain, and these regressors jointly form a multi-output correction module. Therefore, the residual learner does not replace the first-stage teacher, but instead captures the complementary corrective knowledge that remains after the student has absorbed the principal teacher knowledge. Since the local residual learner is tree-based, its parameters are not directly compatible with weighted parameter averaging. Therefore, FKD-RTM aggregates residual knowledge Rg(hk,i)≈r(hk,i) in the function space rather than in the parameter space. Let A ={xk,i}j=1m denote a small public anchor set maintained at the server, and let hk,i=fϕ(xk,i) be the shared representation of anchor sample xk,i. Each client evaluates its local residual learner gk(hk,i)≈r~(hk.i) on these anchor features and uploads the corresponding residual predictions R^k,i=gk(hk,i). The server then performs weighted fusion of the uploaded residual predictions:(12)R¯g=∑k−1kωkR^k,i,ωk=nk∑l=1knl,where nk denotes the sample size of client k, and ωk denotes the aggregation weight defined based on the sample size in Eq. (12). In this way, FKD-RTM does not directly average heterogeneous tree parameters, but instead fuses client-specific residual knowledge in the shared function space. The aggregated residual prediction R¯ serves as the global residual knowledge for anchor sample.

By combining conventional distillation learning regime Sϕ(hk,i)≈pT(hk,i) with residual enhancement regime ΔSϕ(hk,i)≈R¯g, the resulting model can be viewed as learning in a two-stage manner. The two stages in FKD-RTM play different but complementary roles. Specifically, stage-1 distillation is responsible for learning the principal knowledge of the local teacher, i.e., the major decision structure that can be transferred reliably to the global student. Stage-2 residual learning is designed to capture complementary corrective knowledge, namely the prediction bias that remains after Stage-1, especially in hard regions induced by class imbalance, local class missing, or severe distribution shift. This two-stage learning process yields a model with stronger expressive capability, as described in Eq. (13):(13)S~ϕnew(hk,i)=Sϕ(hk,i)+ΔSϕ(hk,i),

The aggregated global residual knowledge is then broadcast back to all clients and used as the second-stage corrective signal after local distillation. It is combined with partial fine-tuning of the feature extractor and the student model to enhance local learning on difficult regions.

Novelty and Advantages: Conventional distillation typically performs knowledge aggregation only once. In contrast, this paper proposes FKD-RTM method aggregates the residual model parameters on the server and broadcasts the resulting model to clients, introducing it as an auxiliary learning signal in the second stage. By explicitly constructing the residual and applying temperature smoothing (as in the corresponding equation), the proposed residual enhancement strengthens the student’s ability to learn the portion of teacher knowledge that remains unknown after the first-stage distillation. This design fundamentally changes the distillation process: the student not only imitates the teacher but also actively compensates for its own deficiencies, focusing more on hard regions. Moreover, residuals are computed from each client’s teacher soft targets, yielding discrepancy signals that reflect complementary gaps across clients. Aggregating these cross-client discrepancy signals provides an explicit training objective, enabling the student to capture shared regularities across multiple data sources as well as complex structures that are missed in the first-stage distillation, including complementary knowledge among client teachers. Overall, this module implements an incremental distillation strategy in which the student learns principal knowledge first and then absorbs the remaining knowledge, constituting a novel and effective distillation paradigm.

In the FKD-RTM method, we introduce tree-based models and MLPs, employ a unified feature-extraction space together with a partial fine-tuning module, and further optimize the learning process via residual enhancement. These designs enable strong fitting ability and high privacy preservation under complex data conditions caused by nonlinear feature structures, feature heterogeneity, and limited samples, while improving generalization under Non-IID settings without sacrificing global consistency. Fig. 1 illustrates the overall federated distillation framework.

First, before the t-th communication round starts, the server maintains a globally shared feature extractor fθ(∙). For client k, each local sample x is mapped into the shared representation space as in Eq. (7). This ensures that samples from different clients are represented in a unified semantic space and provides an aligned input basis for subsequent tree-teacher training and student distillation, thereby substantially alleviating representation inconsistency caused by Non-IID data.

Subsequently, each client k trains a local GBDT teacher model on its feature representations {(hk,i,yk,i)} in the shared space, as in Eq. (1), and outputs a class-probability distribution that is further calibrated. The tree teacher exhibits stronger local fitting ability under few-shot and Non-IID conditions. Moreover, it does not rely on gradient-based optimization and does not require parameter synchronization, making it well suited for federated settings. In addition, it can fit client-specific patterns and thus provides diverse and complementary knowledge across clients.

In FKD-RTM, the student and residual models are aggregated through different operators: the student uses weighted parameter averaging, while the residual GBDT branch is aggregated in the function space via weighted fusion of anchor-based residual predictions. The first stage, the server collects the student model parameters wi uploaded by clients and performs global weighted aggregation across clients, as in Eq. (14). Note that Eq. (14) is used only for aggregating student parameters. The residual branch is not aggregated in the parameter space because the local residual model is GBDT-based; instead, it is aggregated through weighted fusion of anchor-based residual predictions, as defined in Eq. (12). The updated global model Wg is then broadcast back to all clients. Under the supervision of the teacher’s soft targets, the student is iteratively optimized to learn the principal knowledge from clients and to approximate the teacher’s decision structure in the shared representation space, thereby forming a unified global model:(14)Wt+1=∑k∈stαkWit+1,

Here, αk denotes the weight determined by the client’s data volume and or reliability proportion. On the server, αk is computed based on an estimated teacher reliability score and is used to perform weighted aggregation for student parameters in Eq. (15). This weighting scheme enables more personalized aggregation and simultaneously provides reliability scores to filter out abnormal clients, thereby achieving robust aggregation.

To improve robustness under heterogeneous client quality, FKD-RTM adopts a reliability-aware aggregation strategy, where nk denote the number of local training samples on client k and rk denote the reliability score of the local teacher. To jointly consider data size and teacher reliability, we define the pre-normalized aggregation weight as α~k, and the final normalized aggregation weight is computed as:(15)rk=Acc(Tk,Dkval),α~k=nk∙max(rk,τr),αk=α~k∑j∈Stα~j,where Acc(⋅) denotes validation accuracy of the local teacher model. Here, τr is a lower clipping threshold used to avoid extremely small or unstable weights. We further include an ablation comparison with and without reliability-aware weighting aggregation in the experimental section. Second, after the first-stage distillation, we further enhance client-side adaptability while maintaining global consistency. The second stage, we compute the residual ri between the student and teacher soft targets, and train a residual model according to Eqs. (10) and (11). Unlike the student model, the residual learner is tree-based and therefore is not aggregated by directly averaging model parameters in Eq. (14). Instead, each client evaluates its residual learner on a server-side public anchor set and uploads the corresponding anchor-based residual predictions. The server then performs weighted aggregation of these residual predictions in Eq. (12), thereby constructing a global residual knowledge representation in the function space. The resulting global residual knowledge is broadcast to clients and used as a second-stage corrective signal. Combined with partial fine-tuning of the feature extractor and the student model, this mechanism strengthens the student’s performance in disadvantaged regions and improves its ability to fit complex decision boundaries and long-tail classes. In this way, the server explicitly models such discrepancies, constructs a global residual enhancement model, and applies temperature smoothing. The resulting residual model is R¯g then broadcast to clients, where it is combined with partial fine-tuning of the feature extractor and the student model to conduct second-stage learning. This module strengthens the student’s performance in its disadvantaged regions and improves its ability to fit complex decision boundaries and long-tail classes. On each client, during the residual and personalization stage, partial fine-tuning of the feature extractor and the student facilitates enhanced distillation between the residual model and the student, and further helps absorb complementary knowledge across clients, thereby improving the overall capacity for knowledge extraction.

The core workflow of this FKD-RTM mechanism is summarized as follows. First, we construct a unified feature space to enable robust local teacher training. Next, the student learns the global principal knowledge and incorporates residual enhancement to absorb the remaining knowledge. Finally, the server aggregates the student parameters and residual model parameters and broadcasts the updates; on each client, feature-level fine-tuning is performed for personalized adaptation. This procedure is repeated in every federated communication round, enabling the student model to progressively approach an optimal solution corresponding to the multi-client ensemble. The overall federated algorithm is summarized in Algorithm 1.

Training setups. All experiments are conducted on a computing platform equipped with an NVIDIA P100 GPU. On all evaluation datasets, we use the AdamW optimizer with a learning rate of 0.001 and an L2 regularization coefficient of 3×10−4. In the federated learning setting, each client performs 5 local steps per communication round, and the total number of communication rounds is 100. The batch size is fixed to 64 for all experiments. With the total data volume fixed, the default number of clients participating per round is 10. In the number of clients study, we set the total number of clients to 20, 40, and 60, respectively. To ensure the statistical reliability of the reported results, all experiments are repeated over three independent runs with different random seeds, and the reported results are averaged across these runs, and all values are reported as mean ± standard deviation, with standard deviations shown in parentheses. In all result tables, the best value in each column is highlighted in bold. The distillation temperature is set to 2 by default, and all methods use the same client sampling ratio C = 0.7. Similar to other distillation-based methods, we simulate data heterogeneity using a Dirichlet distribution over labels, Dir(α). We allocate all training samples to user models (clients) and evaluate performance using all test samples. For the classifier in all methods, ResNet18 is adopted as a strong feature extractor module. The depth of the gradient-boosted decision trees is 6 with a learning rate of 0.05. The student network contains four linear layers; each hidden layer is followed by a fully connected layer with BatchNorm and GELU. The dropout rate is initialized to 0.4 and then decreases layer by layer. More detailed implementation settings and reproducibility-related configurations are provided in Appendix A.

Experimental datasets. The FKD-RTM study uses four widely adopted real-world image classification datasets, each following its official split: CIFAR-10 (a natural image classification benchmark) and CIFAR-100 (a more challenging multi-class dataset with more categories). To evaluate performance under high-resolution and few-shot settings, we choose the semi-supervised dataset STL-10. To evaluate performance on specialized medical imaging, we use PathMNIST. These datasets cover different difficulty levels and validate adaptability under different data scales or resolutions as well as cross-domain generalization, ensuring that experimental conclusions in federated learning are comprehensive and credible.

Baselines. We compare the proposed FKD-RTM method with the following mainstream baselines: FedAvg [1] update the global model by weighted averaging of clients’ local model parameters without sharing raw data; FedProx [10] additionally introduces a proximal regularization term to constrain local updates from drifting too far from the global model; FedMD [13] exchanges clients’ predictions on a shared public or reference dataset to perform inter-model knowledge distillation; FedDF [18] first obtains an initial global model via parameter aggregation, then further improves it by distillation using client models as teachers; FedMKD [33] combines knowledge from multiple teachers and uses confidence-based weighting to make distillation more robust; FedTKD [34] designed for heterogeneous model architectures and introduces an adaptive weighting mechanism; DaFKD [35] introduces domain awareness by considering distribution differences across clients and reweighting soft labels during distillation. FedGKD [14] and FedICT [36] are also closely related to our work. FedGKD [14] studies heterogeneous federated learning through distillation from historical global models. Our method instead relies on client-specific GBDT teachers and second-stage residual correction. FedICT [36] is developed for bidirectional distillation in multi-task edge-computing scenarios. In contrast, FKD-RTM focuses on complex image classification with shared feature representations and tree-to-MLP heterogeneous distillation. Therefore, we discuss them as relevant recent methods rather than direct baselines.