LLMs, exemplified by ChatGPT [24], have demonstrated remarkable performance across various Natural Language Processing (NLP) tasks. Currently, LLMs exhibit significant potential across diverse fields. In the legal domain, ChatLaw [25], a multi-agent system based on LLMs, has shown robust capabilities in providing legal consultations. In finance, BloombergGPT [26], trained on extensive financial data, has achieved high performance on various financial tasks. In the medical domain, models such as BioMedGPT [27] and Huatuo [28] support analysis and research efforts. Additionally, LLMs have proven reliable and useful in fields like psychology [29] and human-computer interaction [30,31].

Beyond addressing specific tasks, LLMs have demonstrated effectiveness in data generation. Sahu et al. [32] evaluated GPT-3 for intent classification in data augmentation, showing that GPT-3-generated data significantly boosts downstream classifier performance, particularly in low-data scenarios. However, the lack of human alignment in GPT-3 presents challenges in enhancing data diversity through prompt engineering, sometimes resulting in lower-quality outputs. Ye et al. [33] proposed the LLM-DA model with 14 rewriting strategies that effectively bolstered Named Entity Recognition performance in resource-limited contexts, highlighting the value of multi-strategy prompting for high-quality data synthesis with LLMs. Similarly, Wu et al. [34] introduced CALLM, capable of generating medical datasets by simulating roles like patients or doctors, thus advancing low- and zero-shot tasks. Lai et al. [35] developed RumorLLM for rumor detection, transforming existing rumors into new variations while preserving style and semantic consistency, achieving data augmentation. However, these studies predominantly concentrate on data processing for English NLP tasks, leaving the enhancement effects of LLMs on Chinese-language tasks largely unverified.

Text classification is one of the core challenges in NLP. While many NLP models utilize sequential deep learning techniques, graph-based models can directly handle complex structured text data and leverage global information. The TextGCN [36] model represents text data as a graph structure, where nodes correspond to words and documents, and edges are constructed based on word co-occurrence information and TF-IDF weights. TG-Transformer [37] introduces heterogeneity into this graph structure, assigning different weights to document nodes and word nodes. To manage large-scale corpora, the model employs the PageRank algorithm for subgraph sampling.

To harness the knowledge within pre-trained models, BertGCN [38] initializes document nodes using the Classification Token ([CLS]) from BERT, while assigning zero values to the inputs of word nodes, enhancing TextGCN’s representational capacity. It also combines TextGCN and BERT outputs via interpolation for joint training. To further enhance multi-view information extraction, TensorGCN [39] constructs three independent graphs: a semantic-based, a syntax-based, and a sequence-based, and then integrates them into a tensor, where word-document edges across all graphs share the same TF-IDF values.

While these approaches provide valuable insights into the application of graph techniques within NLP, the majority are deductive models with constrained generalization capabilities. Additionally, the process of transforming text into a graph structure often relies on statistical information such as word frequency. A key challenge in this field remains constructing graph structures that incorporate advanced semantic features.

To develop a more accurate detection system for Chinese malicious SMS, this paper proposes a model that integrates LLMs enhancement and multi-view multi-task optimization based on a Chinese pre-trained model. The overall framework structure of the model is illustrated in Fig. 2, and it primarily includes LLMs-based data augmentation, dual-view input, a pre-trained backbone network with graph-based feature fusion, and a multi-task optimization output.

Specifically, during the data processing stage, LLMs with billions of parameters are used for semantic-level data augmentation. In the input module, both word-level and character-level features of SMS are used as model inputs. In the feature analysis stage, a dual-tower Transformer backbone facilitates robust feature extraction and interaction, while a GAT fuses multi-view features.

In the output stage, both supervised classification cross-entropy loss and supervised category contrastive learning loss are used as optimization objectives. This helps the model further learn intrinsic association features within the text, ultimately achieving effective analysis and detection of malicious SMS.

LLMs are trained on massive datasets and aligned with human preferences, enabling them to possess extensive world knowledge. Specifically, the training of LLMs involves three main steps:

(1) Unsupervised Pre-training: During the pre-training phase, the model learns rich semantic and syntactic knowledge from large-scale data without relying on manually labeled data. Instead, the model adopts Next Token Prediction (NTP) as its training objective, as shown in Eq. (1). In this equation, xt represents the target word at time step t, x1:t−1 denotes the context sequence preceding this time step, and θ refers to the model parameters. The goal of this phase is to minimize the negative log-likelihood of the model when predicting the next word. (1)ℒNTP(θ)=−∑t=1nlog⁡P(xt|x1:t−1;θ)

(2) Supervised Fine-tuning: The supervised fine-tuning process leverages high-quality, manually annotated instruction-response datasets to train the model. This approach enhances the model’s ability to understand human instructions and activate stored knowledge, aligning outputs more closely with human needs.

(3) Reinforcement Learning with Human Feedback (RLHF) [40]: In this phase, a reward model is introduced to assess the quality of the model’s outputs, enabling performance optimization through interactions with human feedback. This further enhances the accuracy and fluency of the model’s outputs. Typically, the Proximal Policy Optimization (PPO) algorithm is used for the specific training in RLHF. The optimization objective is detailed in Eq. (2), where πθ represents the current policy, πθold denotes the old policy, x is the model input, A^(x) is the estimated advantage function, r(x,θ) refers to the policy probability ratio, while clip and ε denote the clipping function and its associated parameter, respectively. (2)ℒPPO(θ)=Ex∼πθold[min(r(x,θ)A^(x), clip(r(x,θ), 1−ε, 1+ε)A^(x))]

LLMs have been proven to possess human-level capabilities in text annotation tasks and can be used for various natural language generation tasks, such as text summarization and question answering. This implies that LLMs can be used for semantic-level data augmentation rather than traditional token-level methods [41].

To efficiently leverage LLMs for high-quality data augmentation while mitigating the risk of overfitting, we have designed a flexible prompt engineering framework aimed at enhancing the diversity of generated data. The detailed workflow is illustrated in Fig. 3. Initially, a persona library was introduced to enhance LLMs with diverse character identities. These personas encompass roles simulating potential adversaries (e.g., social engineering hackers, advertisers, scammers, and phishing attackers) as well as neutral roles (e.g., editors and writers), enabling the generation of data in varied rephrasing styles.

Next, we constructed a task description library encompassing a range of directives, such as rewriting, expansion, and summarization. By using a variety of prompt templates, we enable flexible prompt generation through random combinations of personas and task descriptions.

For model selection, we employed three representative Chinese LLMs: ChatGLM [42], MiniMax, and Baichuan [43], to enhance the quality of generated samples. Additionally, by adjusting hyperparameters such as the temperature coefficient and Top-K, we further minimized data redundancy in the outputs, ensuring both diversity and novelty in the generated content.

We also accounted for the proportion of original samples during the generation process, producing additional samples for underrepresented categories to ensure a more balanced training dataset. Finally, a comparison of data distribution before and after LLMs enhancement is shown in Fig. 4.

In the multi-view feature input phase, we introduce both word-level and character-level features to enhance the model’s representation capability while minimizing information loss.

For word-level feature extraction, we utilize a word-level tokenizer to process SMS text, mapping each word to its corresponding token. This method captures the overall semantic information of words, particularly preserving the inherent semantic structure and expression of longer words or fixed phrases. For instance, certain compound words or phrases may convey critical category information in SMS classification tasks.

To further capture fine-grained semantic information within SMS text, we process the text at the character level, treating each character as an independent token. Character-level processing enables the model to capture subtle relationships and semantic nuances between characters, especially in languages like Chinese, where individual characters and their order of combination can convey different meanings. For example, malicious SMS may obscure their true intent through subtle character modifications, and character-level processing can effectively detect such changes, ensuring the fidelity of textual information.

Additionally, to improve the model’s ability to recognize specific types of SMS, we expanded the vocabulary of the Transformer. Specifically, we added high-frequency words unique to different SMS categories by statistically analyzing the distinctive high-frequency terms in the training data.

We conducted a systematic investigation of the current mainstream Chinese pre-trained Transformer models, with a particular focus on their tokenization methods and vocabulary size (as detailed in Table 1).

To effectively extract word-level features, we selected Word-Roformer [44] as a component of the backbone network. To maintain structural symmetry and optimize computational efficiency, Char-Roformer was employed for character-level feature analysis. Compared to other models, Char-Roformer enhances operational speed without compromising model performance by removing bias terms and normalization operations. RoFormer is a variant based on the Transformer [45] architecture, with the core module being the Transformer Encoder, as shown in Fig. 5.

RoFormer utilizes Rotary Position Embedding (RoPE) during the input processing stage, injecting positional information into the input tokens to preserve sequential relationships when the model processes the sequence. The process is outlined in Eq. (3), where βi represents the rotational angle computed based on the token’s position in the sequence, and xi⊥ denotes the vector orthogonal to xi. Compared to traditional positional encoding methods, RoPE demonstrates superior extrapolation capabilities. This enables the model, after being trained on shorter SMS sequences, to generalize effectively to detect malicious text of greater length without the need for retraining. (3)PE(xi)=xi⋅cos⁡(βi)+xi⊥⋅sin⁡(βi) (4){Attention(Q,K,V)=softmax(QKTdk)VMultiHead(Q,K,V)=Concat(head1,…,headh)WOwhereheadi=Attention(QWiQ,KWiK,VWiV) (5)FFN(x)=max(0,xW1+b1)W2+b2

During feature processing, RoFormer alternates between Multi-Head Self-Attention (MHSA) and Feed-Forward Networks (FFN) to extract deep-level features, as described in Eqs. (4) and (5), where Q, K, and V represent the query, key, and value vectors, respectively, and dk is the normalization factor.

To more effectively fuse dual-view features, we propose a feature fusion method based on graph processing. Since the output vector sequence of the dual-stream Transformer model cannot be directly processed by a GAT [46], it is necessary to transform the vector sequence into a graph structure. The current mainstream approach is to construct a KNN graph. However, this method has some limitations.

Firstly, the connections between the central node and its neighbors are typically determined solely by similarity metrics, which provide limited information and restrict the expressive capacity of the generated graph structure. Secondly, this approach often causes the central node to dominate the feature fusion process, thereby weakening the contribution of other nodes and resulting in an imbalanced flow of information. To overcome these challenges, we propose a Skip Central Node (SCN) method, as detailed in Algorithm 1.

The core concept of SCN is as follows: first, retrieve the most relevant neighbor vectors based on the central node; then, during graph construction, exclude the central node and only establish edges between these related neighboring nodes. Compared to the traditional star-shaped graph structure (as shown in Fig. 6), this method effectively reduces the direct influence of the central node on the graph structure, transforming the central node into a bridge for relational retrieval, indirectly connecting potentially related neighboring nodes. This strategy not only reduces the interference of redundant features but also enhances the diversity of the graph structure, thereby facilitating better aggregation of deep semantic information.

The SCN method aims to enhance feature fusion by refining the traditional KNN graph construction process. Understanding computational complexity is crucial for assessing the scalability and efficiency of the proposed approach, particularly when handling large datasets or long input sequences. The time complexity of SCN primarily stems from the following steps:

(1) Sample Iteration (Outer Loop): For a batch of samples of size N, the loop executes N times.

(2) Node Set Construction: For each sample, the time complexity to build the node set is O(T).

(3) Edge Set Construction: First, for each node A, similarity with the remaining T−1 nodes is computed (via inner product), with a computation complexity of O(d) per inner product. Thus, calculating similarity for a single node has a complexity of O((T−1)d)≈O(Td). Next, sorting the similarities requires O(Tlog⁡T) time. Following this, for each node, selecting the top K nearest neighbors has a time complexity of O(K), which is negligible. Finally, in edge construction (excluding the central node), edges are established between each node’s K neighbors, leading to a time complexity of O(K2) per node.

Consequently, the time complexity of SCN for processing a single sample is O(T2d+T2log⁡T+TK2). Given that O(T2d) and O(T2log⁡T) are the leading terms, with O(T2d) generally dominating, the overall time complexity is primarily driven by O(T2d). Notably, with the incorporation of computational acceleration tools like the Faiss library, the time complexity of SCN-based graph construction can be reduced from O(T2d) to O(T⋅logk⁡T⋅d), where k represents a parameter of the indexing structure (e.g., the branching factor in tree-based methods).

After constructing the SCN graph, the GAT is employed to process the graph structure. In GAT, the feature representation of each node is determined not only by its own attributes but also by the features of its neighboring nodes and the attention weights assigned to these connections. The self-attention mechanism assigns different attention coefficients to the neighboring nodes, highlighting the contributions of important nodes while diminishing the impact of less significant ones. This enables the network to extract more useful information from complex structures, as illustrated in Eqs. (6) and (7). Consequently, after processing by GAT, each node’s features are thoroughly fused and optimized, allowing the model to capture multi-scale contextual information. (6)αij=exp⁡(LeakyReLU(aT[Whi‖Whj]))∑k∈𝒩(i)exp⁡(LeakyReLU(aT[Whi‖Whk])) (7)hi′′=σ(∑j∈𝒩(i)αijWhj)

Upon completing the GAT processing of the graph structure, we perform global pooling on the generated node features to aggregate the local node information. Additionally, to further enhance the expressive power of the global features, we concatenate the [CLS] vectors generated from the two branches of the dual-stream Transformer with the globally pooled graph node features. This results in a comprehensive sentence representation vector that integrates both local semantic relationships between words in the sentence and global semantic features, thereby offering higher semantic completeness and expressive capability.

To further enhance the model’s representation capability and classification performance, we introduce two optimization objectives to guide the training process:

(1) Categorical Cross-Entropy Loss: Categorical cross-entropy loss is a widely adopted objective function in supervised classification tasks. It measures the divergence between the model’s predicted probability distribution and the ground truth label distribution, directly guiding classification performance. The model’s output represents a probability distribution across classes, and this loss minimizes the negative log-likelihood of the true labels, thereby encouraging the model to maximize the probability assigned to the correct class. This is formally defined in Eq. (8), where yi denotes the true label of the sample, pi represents the model’s predicted probability for the corresponding class, and C is the set of possible labels. (8)ℒCE=−∑i=1Cyilog⁡(pi)

(2) Supervised Contrastive Loss [47]: This loss function enhances the model’s ability to cluster same-class samples and separate different-class samples by bringing feature vectors of the same class closer together and pushing those of different classes further apart. The process is illustrated in Eq. (9), where zi and zp represent the feature vectors of samples i and p, respectively, τ is a temperature parameter, P(i) denotes the set of positive samples belonging to the same class as sample i, and A(i) refers to the set of all positive and negative samples. (9)ℒSC=∑i=1N−1|P(i)|∑p∈P(i)log⁡exp⁡(zi⋅zp/τ)∑a∈A(i)exp⁡(zi⋅za/τ)

The ultimate optimization objective is a weighted average of the two loss functions, as shown in Eq. (10). By combining the categorical cross-entropy loss with the supervised contrastive loss, the model is not only able to directly optimize classification performance but also to enhance inter-class differences and intra-class consistency within the feature space. This dual optimization strategy helps the model better capture the complex features, thereby enhancing its performance in fine-grained analysis tasks. (10)ℒ=αℒCE+(1−α)ℒSC

To evaluate the performance of the LEGF-DST model in the malicious SMS detection task, we conducted experiments based on the publicly available fine-grained dataset FBS_SMS_Dataset [48], which contains 14 malicious categories, including gambling promotions, fake bank frauds, retail advertisements, and more. Additionally, we added normal SMS samples to the dataset to simulate real-world scenarios where regular and malicious SMS coexist. In the experiments, the training set, augmented training set (used by LEGF-DST), validation set, and test set contained 8000, 27,900, 2000, and 7500 samples, respectively.

The hardware configuration and software environment of the server used in the experiment are shown in Table 2. During the training process, the AdamW optimizer was used, with the batch size and learning rate set to 256 and 5e-5, respectively. In the LEGF-DST model, the K is set to 2 when constructing the SCN graph.

The evaluation of model detection performance in this study is conducted using four key metrics: accuracy, weight-precision, weight-recall, and weight-F1 Score, as defined in Eqs. (11)–(13). TP and TN represent the numbers of true positives and true negatives, while FP and FN represent false positives and false negatives. (11)Accuracy=Weight-Recall=∑i=1CNiN×TPiTPi+FNi (12)Weight-Precision=∑i=1CNiN×TPiTPi+FPi (13)Weight-F1=∑i=1CNiN×2×Precisioni×RecalliPrecisioni+Recalli

In the experiments, we selected methods from both statistical machine learning and deep neural networks as baselines. On one hand, we used Naive Bayes (NB), Decision Tree (DT), Random Forest (RF), SVM, KNN, and CatBoost [49] based on TF-IDF features as comparison models. On the other hand, we selected TextCNN [50], DPCNN [51], BiLSTM, and BiGRU [52] for comparison. Additionally, we included various pre-trained models as baselines, including ALBERT [15], TinyBERT [18], RoBERTa [16], ERNIE [17], Char-Roformer, Word-Roformer, and GF-DST (LEGF-DST without LLMs-based enhancement) model. We evaluated the performance of the model by conducting experiments on both a binary classification task for malicious SMS detection and a fine-grained multi-class classification task. The experimental results are presented in Tables 3 and 4.

An analysis of the experimental results indicates that, in the binary classification task, the performance differences among various models are negligible, with all deep learning-based approaches achieving near-perfect accuracy, close to 100%.

However, in the fine-grained classification task, model performance exhibited considerable variation. Overall, deep learning methods outperform traditional statistical machine learning methods overall, exhibiting stronger representational capabilities and higher classification accuracy. Among the statistical machine learning models, SVM and Random Forest performed relatively well, but they still fell short of deep learning models. This indicates that deep neural networks are more effective at handling complex textual features.

Further analysis of the experimental results indicates that pre-trained language models significantly outperform deep learning models trained from scratch in classification tasks. Models such as RoBERTa, Char-RoFormer, and Word-RoFormer all achieved accuracy rates exceeding 96%, demonstrating that the prior knowledge learned from large-scale corpora in pre-trained models can effectively enhance the performance of malicious SMS classification.

The LEGF-DST model proposed in this paper demonstrated superior performance compared to all baseline models, achieving an accuracy of 97.79% and delivering optimal results across all evaluated metrics. This indicates that the data generated by LLMs can significantly improve the robustness and generalization ability of classification models, leading to excellent performance in malicious SMS detection tasks. In comparison, despite not incorporating LLMs-based augmentation, GF-DST achieved an impressive accuracy of 97.29%, underscoring the advantages of our model design in representing malicious SMS features.

We conducted a comprehensive visual analysis of the accuracy degradation observed as models transitioned from binary classification tasks to multi-class classification tasks, as shown in Fig. 7. Most models exhibited a pronounced decline in accuracy, indicating certain limitations in their ability to capture fine-grained deep features. In contrast, the LEGF-DST model exhibited only a 2.08% decrease in accuracy, demonstrating its strong capability in feature representation.

To analyze the classification reliability, we selected LEGF-DST and RoBERTa (the closest performance counterpart to our method), TextCNN, and SVM as representatives of deep learning and machine learning models. We visualized their PR curves on both datasets, as shown in Fig. 8. It can be seen that LEGF-DST has a superior Weighted Average Precision (Weighted-AP). This demonstrates that the proposed method exhibits superior robustness in SMS classification tasks and is better suited for handling complex data in real-world scenarios.

We also plotted the confusion matrices of the LEGF-DST model before and after augmentation, as shown in Fig. 9. It can be seen that the augmented LEGF-DST shows a significant improvement in recognition accuracy for the weaker categories AD_Other, FR_Financial, and Other.

To conduct a more detailed analysis of the robustness of our models, we manually examined samples predicted by the LEGF-DST and RoBERTa, as illustrated in Table 5. On one hand, RoBERTa struggled with samples containing adversarial noise, such as traditional Chinese or visually similar characters. This limitation likely stems from the model’s reliance on single-perspective features, which results in insufficient redundancy and compensatory capacity when encountering specific types of noise interference. Consequently, RoBERTa is particularly sensitive to minor character-level variations and is prone to misclassification. Furthermore, samples containing deceptive terms such as (legitimate) or (reasonable) are often misclassified by RoBERTa as benign, underscoring the advantage of the dual-perspective feature analysis employed by the LEGF-DST. By integrating character-level and word-level features, LEGF-DST effectively captures subtle semantic distinctions. Specifically, character-level features are adept at identifying fine-grained variations, while word-level features enhance the understanding of broader semantic context. This synergy enables the LEGF-DST model to exhibit heightened robustness in identifying samples with deceptive terminology. Furthermore, RoBERTa exhibits limited accuracy in identifying high-risk samples, such as cult propaganda, due to limited domain-specific pre-training and insufficient labeled data for fine-tuning. In contrast, LEGF-DST, enhanced by LLMs, addresses this issue more effectively, demonstrating an improved capacity to identify such challenging samples. For ethical reasons, malicious content associated with cult propaganda is not displayed in this paper.

Additionally, we compared the LEGF-DST model with current mainstream methods for Chinese malicious SMS detection, and the results are presented in Table 6. It can be observed that the LEGF-DST model demonstrates distinct advantages in both coarse-grained detection and fine-grained analysis.

We conducted a comprehensive evaluation of the efficacy of LLM-augmented data for this task, focusing on its diversity and validity. To facilitate a meaningful comparison, we sampled and visualized a subset of the data generated by different methods. Specifically, we compared the distribution of LLM-augmented data with that produced by two established techniques: Easy Data Augmentation (EDA) [41] and a Markov Chain-based method [53], as shown in Fig. 10. Data from EDA and the Markov Chain-based method exhibits overlaps with the original data, indicating limited capacity to introduce new features. This redundancy may not only restrict the model’s generalization ability but also increase the risk of overfitting. In contrast, LLMs-generated data shows a distinctly different distribution while preserving original semantics. This variation suggests that LLMs can create text with greater semantic and structural diversity without altering the SMS category. Such diversified training data enables the model to better capture complex features of malicious SMS, enhancing generalization and reducing overfitting. Consequently, LLMs-augmented data introduces novel feature patterns during training, significantly boosting classification performance.

Additionally, we conducted a detailed investigation into the effect of the scale of data generated by LLMs on model performance. Our analysis specifically focused on the model’s overall performance, as well as its performance across high-frequency and low-frequency classes. The results are shown in Fig. 11. Overall, as the amount of generated data increases, the classification model’s performance consistently improves, and this improvement is observed in both high-frequency and low-frequency categories. This finding suggests that LLMs-generated data not only effectively augments the size of the training set but also enhances semantic diversity, reducing data redundancy and minimizing information repetition. Consequently, these factors collectively contribute to the model’s improved performance.

To validate the contribution of each component within the LEGF-DST model, we conducted ablation experiments. Using the complete LEGF-DST model as the baseline, we attempted to remove different components to observe their impact on detection performance. The specific configurations tested include:

(1) LEGF-DST w EDA: Substituting LLMs-based data augmentation with an EDA method.

(2) LEGF-DST w OS: Substituting LLMs-based data augmentation with an oversampling method.

(3) LEGF-DST w MC: Substituting LLMs-based data augmentation with a Markov Chain-based method.

(4) LEGF-DST w/o character-level features: Substituting the Char-Word multi-view backbone network with a dual-stream Word-Roformer model.

(5) LEGF-DST w/o word-level features: Substituting the Char-Word multi-view backbone network with a dual-stream Char-Roformer model.

(6) LEGF-DST w/o SC loss: Removing the contrastive learning optimization objective.

(7) LEGF-DST w concat fusion: Substituting GAT-based feature fusion with an additive method.

(8) LEGF-DST w add fusion: Substituting GAT-based feature fusion with a concatenation method.

(9) LEGF-DST w attention fusion: Substituting GAT-based feature fusion with an attention-based method.

(10) LEGF-DST w KNN graph: Substituting SCN-based graph with a KNN-based graph.

The experimental results, presented in Table 7, demonstrate that each component contributes positively to the performance of the LEGF-DST. Models trained with LLMs-generated data perform better than traditional methods, such as EDA or oversampling. Simultaneously using character-level and word-level features helps the model extract high-level features of malicious SMS, thereby improving detection performance.

Additionally, the value of K plays a critical role in determining the time complexity of the LEGF-DST. To evaluate whether increasing K improves model performance, we analyzed the variation in Laplacian spectral entropy of the SCN graph for different K values relative to K = 2, as shown in Fig. 12. The results indicate that as K increases, the growth in Laplacian spectral entropy across input texts of varying lengths remains under 10%. This finding suggests that a smaller K value is sufficient to construct graph structures rich in information while significantly reducing computational costs.