Static analysis tools detect memory issues by modeling source code or intermediate representations before execution. While they scale well, they lack runtime awareness. For example, Zhang et al. [18] proposed UAFX, which uses alias analysis and partial-order constraints to detect complex UAF vulnerabilities in the Linux kernel, addressing some limitations of traditional tools. However, UAFX heavily depends on path modeling and alias resolution, which face scalability issues like path explosion. Common tools like Cppcheck [19] and Flawfinder [20] perform efficient rule-based scans but cannot capture deep semantic relationships, particularly in multi-threaded or non-deterministic paths. In summary, while static analysis is useful for early-stage scanning, it lacks the precision and granularity needed for real-time detection, especially for vulnerabilities dependent on timing or thread interleaving.

Dynamic detection monitors actual memory behavior during execution using tools like Valgrind [21] and AddressSanitizer (ASan) [22], which detect UAFs and buffer overflows with red-zone techniques, though at a high performance cost (2-5x runtime overhead). Several specialized mechanisms have been proposed to improve efficiency. For example, Lee et al. [13] introduced Dangling Pointer Nullification (DPN), which nullifies references during object deallocation to prevent UAF exploits, but it is ineffective against pointer aliasing. Van Der Kouwe et al. [23] proposed DangSan, which reclaims dangling pointers using background threads, though it suffers from delayed detection. Gorter et al. [24] presented DangZero, employing a One-Time Allocation (OTA) strategy for high-precision detection by directly accessing user-space page tables, but it requires significant deployment support. Ahn et al. [25] introduced BUDAlloc, which combines user-space and kernel-level cooperation to reduce overhead, improving adaptability. UAF-GUARD [14] uses fine-grained memory permissions to block illegal memory accesses with high precision but introduces additional runtime overhead, making it less suitable for lightweight or performance-sensitive applications. Despite their high accuracy, most dynamic detection methods rely on privileged instructions, kernel modifications, or custom allocators, complicating deployment and limiting generalizability across software ecosystems.

In conclusion, both static and dynamic approaches face inherent trade-offs: Static analysis offers high efficiency but lacks temporal and concurrency awareness; Dynamic detection provides better precision but suffers from significant performance costs and deployment complexity. This motivates the need for a lightweight, real-time, and semantically expressive alternative, which we explore in subsequent sections.

Hybrid analysis aims to balance detection coverage and behavioral precision by combining static and dynamic techniques. Although platforms like Angr [26] and Driller [27] demonstrate strong capability through symbolic execution and path exploration, their reliance on SMT solvers and complex analysis pipelines significantly limits real-time deployment and scalability in high-frequency online systems.

Recent advances in deep learning [28] have fueled interest in anomaly detection based on runtime logs. LSTM-based models, such as DeepLog [9], effectively learn normal log sequences and identify deviations. However, these models are limited by their inability to capture structural relationships between events. To enhance contextual awareness, LogAnomaly [10] introduces event embeddings and log templates, yet it still relies on binary classification and lacks multi-class granularity.

To address these limitations, recent studies have integrated graph neural networks (GNNs) into log-based frameworks. Logs2Graphs [11] transforms log windows into weighted graphs for GCN-based detection, but suffers from limited classification accuracy. GLAD-PAW [29] and DeepIraLog [30] enhance structural analysis with attention mechanisms and heterogeneous graph modeling, improving interpretability but introducing significant pipeline complexity, hindering real-time use.

Despite growing interest in structure-aware models, most log-driven methods remain constrained to offline settings, binary classification, and single-stage inference, falling short of the demands for real-time, multi-class heap anomaly detection.

In summary, the current methods for detecting heap vulnerabilities face several common limitations:

Offline Operation: Many existing systems operate offline, failing to meet the real-time response requirements that are critical for modern memory anomaly detection.

These limitations highlight a clear gap for a deployable, real-time, multi-class detection framework. The next section introduces MemHookNet, our proposed approach that addresses these challenges via hybrid temporal-structural deep modeling.

MemHookNet adopts a four-stage pipeline: log collection, temporal filtering, graph construction, and multi-class classification (Fig. 1). At runtime, a HOOK module intercepts memory operations (e.g., malloc, free) and generates structured logs. A sliding-window LSTM model pre-filters suspicious sequences. Those exceeding a threshold are transformed into memory access graphs and classified by a GAT-based network across four categories: Normal, UAF, Double-Free, and Memory Leak.

Algorithm 1 illustrates the real-time workflow of log collection, filtering, and classification.

To provide a clear execution overview, Algorithm 1 illustrates the system’s runtime flow, detailing how memory logs are collected, filtered, transformed, and ultimately classified in a real-time processing loop.

Steps 1–2: The system is initialized by loading the HOOK module, which intercepts memory operations and logs them in a structured format.

The HOOK module non-intrusively intercepts and logs heap memory operations (e.g., malloc, free) in real time, providing high-quality temporal input for downstream analysis.

To achieve transparent interception, the module leverages the LD_PRELOAD [31] mechanism to inject logging hooks at the user level. It uses thread-safe logging strategies to preserve ordering, and includes lightweight pointer lifecycle tracking for anomalies such as UAF and Double-Free.

Each operation is recorded in structured fields (e.g., operation type, pointer ID, thread ID, memory size, timestamp), enabling accurate sequence modeling and graph construction.

Overall, the HOOK module is lightweight, stable, and incurs negligible runtime overhead, making it suitable for high-frequency heap monitoring in practical systems.

To enhance detection recall while minimizing computational overhead, MemHookNet integrates a lightweight temporal pre-filtering mechanism within the log stream processing pipeline. This module utilizes state-preserving LSTM [32,33] networks to efficiently process sliding windows of log data, enabling rapid filtering of potential anomalies from continuous memory operation sequences. The LSTM model has been enhanced to ensure memory retention across different windows, allowing for better temporal context and dependency modeling.

In addition, the model employs a multi-scale sliding window mechanism, simultaneously training models with window sizes of 10, 20, and 30. This multi-scale approach helps capture different patterns of anomalies that may span across different window lengths, improving the robustness of the detection process. The log sequences are segmented into fixed-length operation fragments, with each fragment containing five key fields: operation type (op_type), pointer identifier (ptr_id), thread identifier (tid_id), allocated memory size (size), and the time interval between adjacent operations (time_delta). These fields are discretized and normalized before being input into a multi-class LSTM network.

The LSTM architecture, MulticlassLSTM, includes three embedding layers [34], each encoding operation type, pointer ID, and thread ID, with an embedding dimension of 16. The core of the model consists of a two-layer bidirectional LSTM with 64 units per layer and Dropout [35] (set to 0.3) to prevent overfitting. This design allows the model to capture long-term dependencies and generalize across the various temporal patterns in memory access logs. The output is passed through a fully connected layer and processed by a Softmax function to generate predictions for identifying normal and suspicious behaviors (e.g., UAF, Double-Free, Memory Leak [36]). The training uses a Label Smoothing Cross-Entropy loss function with Early Stopping, evaluated using macro F1-score and accuracy [37] to ensure robustness and generalization.

During inference, the system computes the probability distribution for each log window across the four behavior categories and applies an adjustable confidence threshold (default = 0.6). If the prediction probability for any anomaly category in a given window exceeds this threshold, the segment is flagged as suspicious behavior and proceeds to the graph modeling stage. If the threshold is not met, the segment is discarded, conserving computational resources for subsequent processing stages. This design reduces GNN invocation frequency while preserving high recall, offering an efficient front-end filter.

Fig. 2 illustrates the architecture of the Log Window and LSTM Pre-filtering Module, showing how log windows are processed by the LSTM model, followed by threshold-based classification to identify suspicious behavior.

MemHookNet captures structural dependencies by transforming suspicious logs into memory behavior graphs, built on LSTM-filtered segments. These graphs represent pointer access paths, thread collaboration, and temporal logic, providing structured input for the downstream graph neural network. Each unique pointer identifier (ptr_id) within a sliding window is treated as a node in the graph. Each node is assigned a 14-dimensional feature vector, including: a 6-dimensional one-hot encoding of the operation type, normalized allocation size, thread ID, operation frequency, relative position to the most recent malloc, and time intervals between operations. These features characterize the behavior and context of the pointer object within the current log segment.

For edge construction, the system defines three types of semantically meaningful relationships:

Access edges: Represent multiple accesses to the same pointer, reflecting the usage trajectory during its lifecycle.

To enhance the graph’s learnability, each edge is also annotated with a 6-dimensional edge feature vector, which includes: one-hot encoded edge type, pointer consistency, thread consistency, and normalized time differences. These edge attributes improve the model’s ability to perceive operational semantics and relationships.

Finally, the graph samples are exported in a standard JSON triplet format, containing node features (nodes.json), edge relationships (edges.json), and corresponding labels (label.txt). These graph objects can be directly utilized by the graph neural network module for training and inference. Through this module, linear log sequences are effectively transformed into structurally expressive graph representations, laying the foundation for fine-grained classification of complex heap memory behaviors.

The MemGATClassifier, as the core discriminative module of the MemHookNet framework, is responsible for performing multi-class classification of memory access behaviors that are represented in graph structures. This module takes advantage of the structural modeling capabilities of GNNs and integrates an attention mechanism to capture the operational semantics and thread dependencies between nodes in the graph. This enables accurate classification of four behavior categories: Normal, UAF, Double-Free, and Memory Leak.

To enhance the model’s ability to capture structural semantics, MemHookNet adopts the GATv2 [38–40] as its backbone architecture. GATv2 differs from traditional GCNs by introducing learnable attention mechanisms during the feature aggregation phase. This allows the model to adaptively identify the most critical adjacent nodes for classification, enhancing its capacity to model complex dependency patterns in pointer behavior graphs. The architecture of the MemGATClassifier is detailed in Table 1, encompassing input feature encoding, graph convolution layers, graph-level pooling, fully connected classifiers, and the design of the loss function. The overall structural workflow is illustrated in Fig. 3.

During the inference process, the model aggregates neighboring features for each node based on attention weights and generates a unified semantic representation through graph-level pooling. It then outputs the probability distribution for each graph sample across the four behavior categories. If the prediction probability for any behavior category exceeds a specified threshold (e.g., UAF category > 0.7), the system triggers a series of security response mechanisms, including:

Anomaly Log Archiving: High-risk graph samples and their classification results are recorded locally for subsequent auditing and analysis.

By integrating graph structure discrimination with risk response mechanisms, MemGATClassifier not only performs multi-class classification of heap memory behaviors but also supports immediate disposal strategies based on model outputs. This creates a feedback loop from detection to defense, providing solid potential for practical deployment.

To evaluate the adaptability of various GNNs for memory vulnerability detection, we compared mainstream models (see Table 2). Given the sparse nature of log graphs, asymmetric access dependencies, and varying importance of critical nodes, we chose the Graph Attention Network (GAT) for its ability to automatically identify key access nodes, distinguish pointer/thread IDs in different access paths (e.g., malloc vs. free vs. access), and provide a simple structure that is easy to train and deploy. Additionally, GAT’s decoupling from the LSTM module offers flexibility in model combinations, enhancing its suitability for this task.

In contrast, traditional GCN treats neighbors equally, while GraphSAGE lacks sufficient expressive power. Gated Graph Neural Network(GGNN) and Spatio-Temporal Graph Neural Network(ST-GNN) have excessively high construction and training costs, and GAE/VGAE are not suitable for supervised classification tasks. Therefore, GAT is the optimal choice, balancing accuracy and engineering feasibility.

To summarize, the MemGATClassifier not only achieves high-accuracy classification of heap memory anomalies but also enables system-level behavior auditing and defense linkage through its highly interpretable attention mechanism. This, when combined with the LSTM module that captures temporal features, forms a robust “time-structure” dual modeling capability for memory anomaly detection, providing a solid foundation for system deployment and model generalization.

All experiments were conducted in a unified local hardware and software platform environment, with specific configurations shown in Table 3. The system model implementation and training were carried out using the PyTorch and PyTorch Geometric frameworks, and the experiments were deployed on a workstation environment with Graphics Processing Unit (GPU) acceleration to ensure efficient training and inference performance.

To facilitate reproducible experimentation and ensure consistency across different processing stages, we designed a standardized logging procedure to structure intercepted memory operations into training-ready sequences. In our experiments, we extracted memory operation logs by executing vulnerable programs from the SARD dataset, using the LD_PRELOAD technique to intercept heap memory operations such as malloc, free, and memcpy during runtime. The resulting structured logs were used as the primary input for training and evaluating the MemHookNet framework.

This interception process is implemented via the LD_PRELOAD + dlsym mechanism, which transparently hijacks key memory management functions and monitors heap activity without modifying the original source code. The pseudocode for the memory log collection and structuring process is presented in Algorithm 2, which outlines how each intercepted memory operation is formatted into a consistent structure. Specifically, the system dynamically resolves the address of each target function, extracts the operation type, pointer address, thread ID, allocation size, and timestamp, and encodes this information into a unified log entry. These structured logs serve as the foundational input for the downstream sequence modeling and graph construction modules.

To ensure consistency in input formatting across both sequence modeling and graph modeling, each memory operation log entry is formalized as the following quintuple: To ensure consistency in input formatting across both sequence modeling and graph modeling, each memory operation log entry is formalized as the following quintuple. See Eq. (1): (1)Lt=⟨p_t,ptr_t,tid_t,size_t,Δt⟩where:

op_type: Operation type;

The final complete log stream is represented as L={L1,L2,…,LT}.

This structured definition provides mathematical consistency to support sliding window segmentation, sequence encoding, and graph modeling.

This section provides a detailed explanation of the two core models in the MemHookNet system: the LSTM sequence classifier and the GNN graph classifier. It covers their network architectures, training strategies, and loss functions, aiming to demonstrate the design rationale and implementation details of each module in addressing the multi-class heap memory behavior detection task.

To objectively evaluate the performance of MemHookNet for multi-class heap memory anomaly detection, we selected four representative baselines covering time-series, structural, and hybrid modeling paradigms: DeepLog, Logs2Graphs, LogAnomaly, and GLAD-PAW, as illustrated in Fig. 5. These models were re-implemented using official code and trained under identical conditions (dataset, hardware, and hyperparameter settings) to ensure fair comparison.

DeepLog: A classic log sequence prediction model based on LSTM, widely used in general anomaly detection scenarios;

Fig. 5 compares their input processing pipelines, model architectures, and output strategies. We observe that MemHookNet uniquely combines temporal filtering and structural modeling, enabling superior detection of complex heap anomalies.

This comparative analysis highlights MemHookNet’s advantages in multi-class recognition, real-time response, and adaptability. Its ability to effectively detect Use-After-Free, Double-Free, and Memory Leak anomalies with minimal computational overhead makes it a competitive solution for practical deployment in security-critical environments.

To further examine the individual contributions of each core component in the MemHookNet framework, we conducted a series of ablation experiments. These experiments aim to isolate the impact of three key modules—LSTM-based sequence filtering, graph construction with attention, and HOOK-based log interception—on the overall anomaly detection performance. The study includes four experimental variants and the full model configuration, as detailed in Table 4.

Experimental results demonstrate that each module—HOOK, LSTM, and GATv2—plays a critical role in achieving high detection accuracy and robustness. The exclusion of any single component leads to significant performance degradation. In contrast, the full MemHookNet framework consistently yields the highest precision and recall, confirming the complementary effect of temporal and structural modeling.

In summary, this section evaluates all key modules within a unified experimental platform. By covering the entire pipeline—from log collection and normalization to sequence filtering and graph classification—the study ensures both reproducibility and engineering feasibility. The combination of multiple baselines and carefully designed variants establishes a strong foundation for the comparative performance evaluation presented in Section 5.

While MemHookNet demonstrates promising performance in detecting multi-class heap memory anomalies, several threats to validity should be acknowledged:

Generalization across Platforms: Our implementation was tested on Linux-based environments using LD_PRELOAD for function interception. Portability to other operating systems (e.g., Windows, macOS) or to kernel-level anomalies may require architectural changes.

Future work will address these threats by validating the framework on more diverse and large-scale real-world applications, and by exploring adaptive threshold tuning and cross-platform generalization.

To evaluate the computational efficiency of MemHookNet, we analyze the time complexity of its key modules.

LSTM Pre-Filtering: Given a sliding window of size N and embedding dimension d, the two-layer bidirectional LSTM has a time complexity of 𝒪(N⋅d2). Since only recent logs are filtered in real time, this stage is lightweight.

To contextualize these results, we compare MemHookNet’s complexity with representative baselines in Table 5.

Compared with baselines like DeepLog and Logs2Graphs, MemHookNet achieves a balance between accuracy and efficiency by filtering low-risk data early and applying graph-based analysis only when necessary. This makes it suitable for real-time deployment.

We evaluate the detection performance of MemHookNet and compare it with four representative baselines: DeepLog, LogAnomaly, Logs2Graphs, and GLAD-PAW. These methods differ in their modeling strategies, ranging from sequence learning to graph-based classification.

As shown in Table 6, MemHookNet achieves the best performance across all metrics, including 85.5% precision, 82.2% accuracy, 81.5% recall, 81.7% F1-score, and 94.7% AUC, with the lowest inference latency (15 ms). These results validate the model’s practical effectiveness and stability in anomaly detection.

To further contextualize these findings, we conduct comparative experiments under identical datasets and training conditions. The selected baselines span LSTM-based sequence models, template-based log analyzers, and graph neural networks. Table 6 presents the quantitative performance metrics, while Table 7 summarizes the functional capabilities of each method in terms of multi-class support, online processing, and deployment suitability.

Unlike binary classifiers such as DeepLog and Logs2Graphs, which only distinguish between normal and anomalous behavior, MemHookNet supports real-time multi-class detection. It enables fine-grained classification of UAF, Double-Free, and Memory Leak anomalies. This design advantage contributes to its superior performance in detailed anomaly recognition, as shown in Tables 6 and 7.

According to the Tables 6, 7 and Fig. 6, MemHookNet significantly outperforms existing baselines in both detection accuracy and inference efficiency, achieving a 47.9% improvement in accuracy and a 92.5% reduction in latency compared to Logs2Graphs. These gains highlight the benefit of combining HOOK-based runtime logging with temporal–structural modeling.

Even when compared to strong models like GLAD-PAW and DeepLog, MemHookNet delivers higher classification accuracy with substantially lower computational cost. As shown in Fig. 7, it converges faster and reaches lower final loss values, demonstrating more stable and efficient learning. It also consistently maintains the highest macro-F1 scores throughout training, especially in early-stage groups, validating its ability to capture representative patterns from runtime memory logs. In contrast, Logs2Graphs struggles with convergence, while DeepLog and LogAnomaly exhibit slower learning and weaker generalization.

In summary, MemHookNet not only achieves high detection precision but also supports real-time multi-class anomaly detection with minimal overhead, making it a compelling solution for practical deployment in dynamic memory security scenarios.

To evaluate the individual contributions of the modules (HOOK, LSTM, GNN) to the final performance, we designed three ablation experiments, each removing one of the modules to observe the performance changes. The experimental configurations are shown in Table 8:

As shown in the Table 8 and Fig. 8, MemHookNet outperforms the other comparison models on all key performance indicators (Accuracy, Precision, Recall, F1-Score), demonstrating its clear overall advantage. Notably, MemHookNet shows significant improvements in Precision and F1 score. Specifically, MemHookNet improves over:

GATv2-noHook(B3) by 31.3% (85.5 → 54.2) in Precision and 35.5% (85.7 → 50.2) in F1 score;

This result validates the effectiveness and advancement of the HOOK + LSTM + GNN hybrid architecture in multi-class recognition, structural modeling, and online processing tasks. In addition, a complexity analysis in Section 4.7 demonstrates that MemHookNet achieves a favorable trade-off between accuracy and efficiency, making it suitable for real-time deployment in practical systems.

To assess the practical efficiency of MemHookNet in real deployment scenarios, we evaluate both subsystem-level runtime latency and system-wide capability across multiple performance dimensions.

We first measure the average execution latency of each key module in the inference pipeline, including memory operation logging, LSTM-based filtering, graph construction, and GNN-based classification. The detailed latency breakdown is shown in Table 9.

These results demonstrate that MemHookNet achieves efficient online processing with an average end-to-end inference latency of approximately 12 ms per log fragment, which supports sub-second responsiveness for heap anomaly monitoring. This performance makes the system well-suited for high-frequency memory event surveillance in embedded or cloud-based environments.

To further contextualize the runtime efficiency and deployment suitability of MemHookNet, we conducted a high-level comparison against mainstream static, dynamic, and hybrid detection frameworks. As shown in Fig. 9 and Table 10, we evaluate five core dimensions that reflect practical deployment needs: Detection Granularity, Multi-class Support, Real-time Capability, Deployment Overhead, Extensibility.

From the radar chart, we observe that MemHookNet consistently outperforms other methods across all dimensions: Compared to static analysis tools (e.g., Cppcheck), MemHookNet supports runtime and semantic modeling while maintaining low deployment cost. Compared to dynamic analysis frameworks (e.g., Valgrind, ASan), it dramatically reduces runtime latency (seconds vs. milliseconds). Compared to hybrid tools (e.g., Angr, Driller), MemHookNet avoids symbolic execution bottlenecks and enables end-to-end online processing.

The combined analysis confirms that MemHookNet strikes a highly favorable balance between detection capability and real-world usability, enabling real-time, fine-grained, and scalable heap anomaly detection that outperforms traditional tools in both responsiveness and deployment flexibility.

To assess the robustness of MemHookNet under varying runtime conditions, we conducted a series of experiments simulating real-world variations in data noise, anomaly frequency, and thread-level concurrency. Specifically, we evaluated three robustness scenarios: Noise Injection: Random non-anomalous log entries were inserted into normal and abnormal sequences to simulate log noise or benign interference. We tested with noise rates of 0%, 10%, and 20%. Anomaly Rate Variation: The proportion of anomalous log windows was varied across 1%, 5%, 10%, 20% to assess the model’s sensitivity under class imbalance. Thread Mixing: Memory operations from multiple threads were interleaved and randomly ordered to test the temporal resilience of the LSTM and GAT modules.

Fig. 10 shows the accuracy degradation across scenarios. MemHookNet demonstrates strong robustness in all settings, with less than 3.5% performance drop under 20% noise, and less than 2% reduction under severe class imbalance. This confirms the model’s ability to retain high detection fidelity even in noisy or skewed environments, which are common in production systems.

While MemHookNet is designed to classify memory behaviors into four categories—Normal, UAF, Double-Free, and Memory Leak—the boundaries between these classes can sometimes be subtle. To further understand classification behavior, we conducted a confusion matrix analysis on the validation set, along with per-class performance breakdown.

Fig. 11 shows the confusion matrix, highlighting that most misclassifications occur between UAF and Double-Free, due to their similar deallocation patterns. Memory Leaks are the most separable class, with high precision and minimal confusion.

To better understand how MemHookNet performs across different anomaly categories, we provide a class-wise breakdown in Table 11 and Fig. 10. The model achieves the highest F1-score for Use-After-Free (0.88), followed by Memory Leak (0.79) and Double-Free (0.78), showing balanced performance across categories. Double-Free detection is slightly less accurate due to overlap with UAF behaviors. In contrast, Memory Leaks are the most separable, with minimal confusion. These results validate the model’s robustness in handling diverse heap anomaly types.

These results confirm that MemHookNet maintains strong discriminative capability across categories, particularly in distinguishing memory leaks. Additional feature tuning and graph relation encoding may further improve separation between UAF and Double-Free instances.