Modern Natural Language Processing (NLP) has advanced dramatically with the advent of Transformer-based Language Models (TbLMs) [1], in particular the Autoencoding and Autoregressive architectures. Because the Transformer architecture comprises millions to billions of parameters, these models necessitate an extensive corpus of training examples to learn diverse patterns and contextual relationships. Such rich and varied data distributions are essential for the self-attention mechanism to capture complex inter-token interactions and enhance generalization performance. In regimes of data scarcity, models are prone to over-fitting, leading to a precipitous decline in predictive accuracy on novel sentences or unseen domains.

Historically, to meet the demand for large-scale training data, it has been common practice to collect and centrally process user data on a dedicated server. However, this paradigm conflicts with strengthened privacy regulations such as the General Data Protection Regulation (GDPR) [2] and confronts significant practical limitations. In particular, the GDPR’s purpose-limitation and data-minimization principles, which entered into force in May 2018, explicitly prohibit the large-scale aggregation and retention of raw user data, thereby engendering a fundamental tension with centralized model training.

Under these constraints, federated learning (FL), a distributed optimization framework [3], emerges as an effective solution to address this challenge. In FL, each client trains a model locally on its raw data and shares only the resulting model parameters with a central server, thereby preserving data privacy while collaboratively improving a global model. This approach has proven useful in pre-trained FL settings [4], particularly for TbLMs.

However, the distributed nature of FL gives rise to several security issues by providing malicious or compromised participants with opportunities for attack [5]. A representative threat is the backdoor attack, where a malicious client uses an update trained on poisoned data to cause the global model to output a specific, incorrect prediction for a certain input. Such attacks can severely undermine model reliability.

Notably, backdoor attack techniques validated in standalone TbLMs environments are theoretically effective in federated learning contexts as well. In this study, we empirically evaluate these techniques under various attack modalities and defense mechanisms in practical FL scenarios.

In this way, Ensuring robustness is an essential research topic for the safe and widespread adoption of FL-based language models. Indeed, various defense mechanisms have been proposed to counter threats such as the backdoor attacks described previously. However, there is a lack of systematic research comparing the performance of these existing defenses under key attack scenarios, specifically when applied to autoencoding and autoregressive-based language models.

Bridging this gap is essential, as the inability to defend against such attacks undermines the trustworthiness of FL as a privacy-preserving framework. This challenge is extends beyond NLP; it is a posing critical implications in other privacy-sensitive domains like IoT and edge computing, where resource constraints are also paramount [6]. Likewise, in the healthcare domain, where sensitive patient data and strict privacy regulations further accentuate the necessity for secure and reliable FL frameworks [7]. Collectively, these applications highlight the urgent need for robust and secure FL systems.

The key contributions of this work are summarized as follows:

We provide a systematic evaluation of both classical and advanced backdoor attacks in FL environments with transformer-based language models.

This paper aims to contribute to the development of trustworthy FL-based NLP systems. To this end, we experimentally evaluate both classical and advanced backdoor attack methods, as well as representative defense mechanisms in FL environments, across two paradigms—autoencoding (BERT) and autoregressive (GPT-2). Based on these results, we analyze and compare the performance of defense mechanisms against major backdoor attacks, thereby providing an empirical basis for selecting appropriate defense strategies for specific threats. To the best of our knowledge, no prior work has systematically compared backdoor defense effectiveness in FL-based NLP systems employing TbLMs.

The remainder of the paper is organized as follows. Section 2 presents the background of our work. Section 3 introduces representative backdoor attack methods for TbLMs. In Section 4, we describe the aggregation methods designed for robust FL-based NLP, and Section 5 presents experimental results under representative attack scenarios. Section 6 discusses key findings and their implications, and Section 7 concludes with a summary and directions for future research.

Backdoor attacks are an insidious threat that embed malicious behaviors into a model, which are activated only by specific triggers without degrading general performance. These attacks become a particularly severe security vulnerability in the FL environment, where the central server cannot validate client data to preserve privacy. This section, therefore, systematically analyzes backdoor attack methodologies, categorizing them into traditional and advanced approaches. These attack methods are summarized in Table 1.

In the federated learning process, since each client trains its own local model, the defense mechanisms originally designed for a single centralized model are unlikely to be effective. Consequently, federated learning environments require the adoption of defense mechanisms specifically tailored to the federated learning paradigm. We analyze defense mechanisms in federated learning environments by classifying them according to their mechanisms into perturbation-based and screening-based Approach. These defense mechanisms are summarized in Table 2.

The experimental evaluation is designed to assess the effectiveness of the aforementioned defense strategies against various attack scenarios. To quantitatively evaluate performance, we adopt the Attack Success Rate (ASR) as the key metric. ASR measures the effectiveness of a backdoor and is defined according to the model architecture as follows:

For autoencoding models (e.g., BERT), ASR denotes the proportion of trigger-injected inputs that are misclassified into the attacker’s target label.

Figs. 3 and 4 demonstrate a clear performance disparity between defense strategies. Perturbation-based defenses, such as norm clipping and weak DP, largely failed to suppress the ASR, with their ineffectiveness becoming more pronounced as attack sophistication increased. In contrast, screening-based defenses like Multi-Krum proved highly effective, particularly against rare trigger-based attacks where they maintained a 0% ASR. However, this superior security comes at a significant cost. Focusing on the aggregation phase for the 355-million-parameter GPT-2 Medium model, a screening-based method required 27.03 s, while a perturbation-based method took only 3.9 s—a nearly seven-fold difference in computational overhead. We anticipate this overhead will grow exponentially for state-of-the-art models with billions of parameters, highlighting a critical trade-off between security and efficiency.

Thus, while screening-based defense methods deliver superior protection, we confirmed that they incur relatively high overhead in aggregation computations. Moreover, as attacks become more sophisticated, some malicious vectors may still survive the screening stage. To fully suppress these residual threats, it may be beneficial to adopt a hybrid defense that follows the initial screening phase with a lightweight perturbation based mechanism. The FLAME framework, as previously described, likewise employs a three-stage hybrid defense—comprising clustering, adaptive norm clipping, and adaptive noise injection—but incurs, on average, an aggregation time of 63.20 s—approximately 2.3 times the overhead of alternative approaches. Consequently, to realize a more lightweight hybrid solution, it is worth investigating a scheme founded on Multi-Krum.

However, it is crucial to acknowledge that even robust screening-based defenses like Multi-Krum and FLAME can be circumvented under certain sophisticated attack scenarios. As highlighted in prior work [13], when the attack vector itself is composed of sentences containing profanity and hate speech, the resulting malicious updates may not be markedly different from benign ones. This statistical similarity makes it significantly easier for adversaries to evade detection, posing an ongoing challenge for even advanced defense mechanisms.

This paper presents a critical evaluation of applying existing defense strategies to TbLMs within FL environments, representing a crucial step in understanding their practical security implications. Our comprehensive analysis of perturbation-based and screening-based defense mechanisms revealed a significant trade-off between defensive performance and communication efficiency.

Perturbation-based methods, which introduce noise or constraints, were consistently characterized by their low computational overhead across both models. This makes them easily integrable into existing systems. However, our evaluation uniformly showed that they struggle as a robust defense, with the ASR remaining considerably high.

In contrast, screening-based methods demonstrated a considerably higher rate of defense in our experiments. By actively identifying and excluding suspicious components, these techniques effectively neutralized backdoor threats in both model environments. Despite this high efficacy, however, there was a clear drawback of high overhead, consuming intensive computation and resources that were more than double those of perturbation-based methods.

Ultimately, overcoming this defensive performance-communication efficiency trade-off is a key challenge for future work. Specifically, research is urgently needed to achieve the high security performance of screening-based defenses at a realistic computational cost. Reducing the computational overhead while maintaining a high level of defense is the essential next step toward deploying truly trustworthy machine learning systems in a wider range of environments. This challenge is not limited to text-based FL systems but also extends to multimodal AI models, where adversarial robustness remains an open research problem [20].

Future research needs to be concretized along two complementary directions. First, perturbation-based defenses should maintain their inherently low overhead while improving their currently modest defense rate. Second, screening-based defenses must focus on reducing overhead while sustaining strong robustness. Since most of the overhead arises from pairwise comparisons across client updates, strategies to reduce the number and cost of these comparisons are essential. Finally, Neurotoxin [13], examined in this study, serves as a representative example of an adaptive adversary, underscoring the need for future defenses to address not only traditional backdoors but also adaptive and evolving attack strategies. In this regard, exploring more advanced and durable attacks such as SDBA [21], as well as recent defense mechanisms applied in image-based FL models [22], will be crucial for assessing their applicability and effectiveness in TbLMs.

While this study primarily focuses on the technical robustness of federated learning (FL)-based language models, we acknowledge that the deployment of FL systems in sensitive domains (e.g., healthcare, finance, education) raises important ethical considerations. These include issues of fairness, accountability, potential misuse, and the unintended amplification of biases. Although these concerns fall outside the direct scope of our experiments, addressing them is essential for safe and trustworthy real-world adoption of FL systems. We encourage future research to incorporate both technical defenses and ethical safeguards.