Ransomware encrypts digital data and alters system login credentials to restrict access to victims’ resources. The attacker claims a ransom from the target for acquiring access to its resources. In 2017, the WannaCry cyberattack infected over 200,000 systems across 150 countries [1]. Ransomware is a prominent threat in the malware landscape, consistently ranked as one of the top concerns for cybersecurity experts. While infection rates have declined, ransomware remains highly cost-effective, as attackers focus on targeted internal communications and demand larger payouts. This profitability continues to attract cybercriminals, solidifying ransomware’s position as a major cybersecurity challenge.

The proliferation of ransomware consists of multiple factors, including the widespread Internet usage enabling cross-border cyberattacks and the anonymity of cryptocurrencies facilitating untraceable ransom payments. Moreover, the cost-effectiveness of digital storage creates vulnerabilities that ransomware exploits, particularly through encryption. The availability of ransomware kits on the Dark Web and the rise of ransomware-as-a-service schemes makes it increasingly accessible, ensuring its sustained growth as a cyber threat. The types of ransomwares can be shown in Fig. 1.

Attack Vector

Ransomware is often propagated through phishing emails, in which cybercriminals distribute malicious attachments or embed links that direct victims to counterfeit documents or seemingly legitimate websites. When a victim clicks on these attachments, the ransomware activates on their device and begins spreading across the system. Advanced ransomware variants further execute reconnaissance on the infected device, gathering system information before establishing communication with their Command and Control (C & C) server. Through this connection, the ransomware transmits the collected data and requests cryptographic keys. Using these keys, it encrypts critical files, such as Microsoft Office documents, images, media files, and metadata. The ransomware only reveals its presence after encrypting all essential data, leaving the victim locked out of their files [2]. Almost 63.3% of ransomware infections are attributable to emails, with 35.7% relating to opening an attachment and 28.6% to clicking on a link, as illustrated in Fig. 2. Moreover, 21.4% of victims report being unaware of how they became infected with ransomware. Additionally, ransomware typically provides its victims with a limited timeframe to pay the ransom before the amount increases.

Another significant vector for ransomware attacks involves victims visiting malicious or compromised websites, either intentionally or inadvertently. This type of attack, known as malvertising, leverages redirection strategies to exploit vulnerabilities in unsuspecting users. In a typical malvertising scenario, attackers embed exploit kits within an inline frame, or iframe, on a legitimate website. When users access the compromised site, they are redirected from the secure page to a malicious one, triggering the download and execution of ransomware. This sophisticated tactic highlights the increasing risks associated with online advertising and website vulnerabilities [3]. Windows OS is the most widely used operating system, making it a primary target for ransomware attacks. A critical aspect of ransomware detection within Windows lies in its Application Programming Interface (API). APIs serve as the primary interaction point with the operating system, and all programs rely on these interfaces for execution. Consequently, analyzing API behavior is a highly effective approach for identifying ransomware [4].

This study focuses on crypto ransomware, a particularly destructive variant that encrypts victims’ data and files using robust techniques such as hybrid encryption. The damage caused by crypto ransomware is often severe and irreversible, especially if recovery mechanisms are unavailable post-encryption. The pre-encryption phase is therefore crucial for detection and intervention, as early identification can significantly reduce the impact of an attack. Developing an early detection model is vital for mitigating these threats and safeguarding user data.

Existing research has introduced numerous models for ransomware detection, leveraging classical and advanced machine learning techniques. Advanced ML techniques, such as deep learning and convolutional neural networks (CNNs), require extensive data for effective training/classification, where classical machine learning algorithms can deliver reliable performance with fewer samples, making them an ideal choice. However, most of the classical ML-based ransomware detection models also rely on an extensive number of features, complex architectures, and are based on a single detection approach, which may lead them to be less effective against sophisticated ransomware variants.

Therefore, the proposed model addresses these limitations by employing a smaller, optimized set of features (API calls) and integrating both signature-based and behavior-based crypto ransomware detection mechanisms through classical machine learning. Furthermore, we optimized the detection process by reducing the required number of features, improving the model’s efficiency compared to existing machine learning-based approaches. This approach serves as proof of concept for a lightweight and effective ransomware detection system. In short, the proposed approach enhances robustness and adaptability, offering a significant contribution to the advancement of ransomware detection methodologies. The core contribution of the study is as follows:

Behavioral Analysis: Identified key ransomware behaviors through API analysis and optimized features for efficient detection.

This paper conducts a comprehensive review of existing studies and presents a detailed discussion on the design, implementation, and analysis of the experimental framework. The findings derived from the experiments are thoroughly analyzed, leading to significant conclusions that contribute to the field. Additionally, the paper offers valuable recommendations to guide future research and advance the understanding of the subject.

Numerous studies and technical reports have highlighted the escalating threat of malware and its substantial impact on organizational security. The proliferation of advanced technologies, such as the Internet of Things (IoT) and cloud computing, which enhance global connectivity, has been accompanied by a corresponding rise in malware attacks [5]. This increase presents significant risks, especially when end-users lack awareness of the threats associated with these emerging technologies [6]. Among malware families, ransomware stands out as the most prevalent, targeting organizations and individual users on a large scale. Ransomware’s effectiveness stems from its direct interaction between attackers and victims, where attackers demand ransoms in exchange for decryption keys. Alarmingly, data shows that 37% of affected organizations fail to recover their data even after paying the ransom [7]. Once ransomware infiltrates a system, it encrypts files using advanced encryption algorithms, withholding the decryption key and rendering data inaccessible [8]. While encryption is fundamentally used to ensure data confidentiality, attackers maliciously exploit this technology by deploying crypto-ransomware to deny users access to their own data. In particular, the work in [9] highlights the challenges and vulnerabilities associated with encryption schemes when faced with such attacks, discussing how encryption, although essential for security, can be turned against users when misused by malicious actors. The study also outlines the limitations of current encryption techniques in big data environments, where the massive scale and distributed nature of data increase the impact of ransomware and emphasize the need for robust, attack-resistant public key encryption schemes.

Current anti-malware solutions, such as antivirus software and intrusion detection systems (IDS), which rely on signature-based detection methods, are ineffective against zero-day attacks. This limitation arises because new malware signatures are not immediately added to the signature databases, leaving systems vulnerable to emerging threats [10]. Ransomware remains a favored tool for cybercriminals, with new variants continually developed to evade antivirus programs [11]. Moreover, the emergence of Ransomware-as-a-Service (RaaS) has lowered the technical barrier, enabling non-technical individuals with malicious intent to deploy ransomware attacks [12]. To address this growing threat, strong protection is urgently needed for organizations and users. Significant research efforts are focused on detecting ransomware at an early stage, employing techniques such as static and dynamic analysis to improve detection accuracy and mitigate damage [13].

Static analysis involves examining the source code without executing the ransomware, making it a safer approach for initial evaluation. One common technique involves extracting opcodes from the source code and generating n-gram representations, which are then used to train machine learning models for ransomware prediction [14]. Similarly, researchers have proposed rule-based detection methods derived from source code analysis [15]. In addition, analyzing the headers of executable files has also proven effective for ransomware detection. Malware apps targeting mobile devices exploit Android’s open-source nature, unofficial app stores, and lack of verification processes. PerDRaML [16] is a permissions-based malware detection system that uses features such as permissions and smali sizes from a dataset of 10,000 apps. Machine learning models demonstrated high detection accuracy, with Random Forest achieving 89.96%. The approach optimized approximately 77% of the feature set, enhancing precision, sensitivity, and F-measure compared to existing techniques. Another study [17] evaluates four classifiers using a real-device hybrid analysis dataset (2008–2020). Random Forest achieved the highest accuracy (97.86%) and demonstrated sustainability, while Logistic Regression was the most efficient, with GPU-based setups significantly reducing training time. The results demonstrate notable improvements in detection accuracy, efficiency, and long-term performance. However, a major limitation of static analysis lies in its susceptibility to obfuscation techniques, which can mask malicious code and potentially result in flawed or incomplete evaluations.

Dynamic analysis of ransomware requires executing the malware in a controlled environment, such as a sandbox, to monitor its behavior. This approach is effective due to the direct interaction it enables, allowing researchers to observe the ransomware’s activities. However, the primary risk of dynamic analysis is the potential spread of malware within the lab environment. Additionally, advanced ransomware may detect the sandbox environment and alter its behavior, remaining dormant to evade detection. Some studies have proposed various detection methods, such as analyzing the encryption process, which involves repetitive patterns that can be examined at the file system level to identify ransomware behavior [18]. Tracking the frequency of encryption is another technique to distinguish between malicious attacks and legitimate encryption processes. A dynamic analysis approach involves monitoring system API calls to track ransomware interactions with the operating system [19].

In [20], the authors introduced a similar framework for detecting crypto-ransomware, named RENTAKA. This framework used the RISS dataset as a reference and successfully reduced the number of features to 80. However, this reduction caused a 3% decrease in detection accuracy, lowering the detection rate to 97.03% when using the Support Vector Machine (SVM) classifier, compared to [21]. The authors also employed several classification models, including SVM, Random Forest, Naive Bayes, KNN, and J48. Despite these advancements, the study lacks detailed information about the proposed model, and it does not clarify which encryption APIs were targeted during the analysis. A study on API-based ransomware detection was proposed by [22], using a feature importance technique. The authors analyzed API calls to identify specific patterns of ransomware behavior. They collected 653 executable files, including both benign and malicious applications. After conducting API analysis, they performed feature ranking and created a dataset for further use. The dataset was then provided to machine learning models, which were trained using four classifiers: Random Forest, C5.0, AdaBoost, and SVM. The highest detection accuracy achieved was 95.38% using the AdaBoost classifier. However, the primary contribution of this study lies in the application of feature ranking.

In [21], the authors analyze crypto-ransomware behaviors through function calls and Application Programming Interfaces (APIs). They propose a two-level detection model: the first level uses signature-based detection, while the second level employs machine learning algorithms to detect ransomware based on API calls. The authors employed the RISS dataset [23], which contains 30,967 applications, including both benign and malicious ones. After analyzing the dataset, they extracted ransomware-specific features, resulting in a new dataset with 232 API features and 1800 entries. This new dataset was specifically focused on pre-encryption functions. Using machine learning, the authors achieved 100% detection accuracy with the Random Forest classifier. After detecting ransomware, the model stores its signature for future detections. The experiments were conducted using Cuckoo Sandbox (V2.0.6), Windows 10 Pro, and Ubuntu 18.04 platforms. However, there is potential for further optimization by reducing insignificant features while maintaining similar detection accuracy. Another recent study [24] proposed a similar model to Kok et al. [21], focusing on API-based detection, but reduced the number of features from 232 to 206. However, this reduction led to a 1% decrease in detection accuracy when using the K-NN classifier. While both models focus on detecting known ransomware. Table 1 presents a comprehensive overview of commonly employed techniques for ransomware detection.

It is observed that the above methods achieved high detection accuracies, however, they also employed a larger number of features, i.e., 323, 80, 100, 120. That further can be reduced, or only effective API calls can be employed for real-time detection. Secondly, the above methods are totally dependent on ML-based detection models, which can be optimized with signature matching for existing and future ransomware. Finally, the existing methods employed the custom or proprietary ransomware sample dataset, which is inaccessible to researchers in this domain.

This study proposes an optimal approach for detecting ransomware before it is executed, thereby minimizing potential damage to an organization. A dynamic solution is introduced to enable early detection through signature matching through an updated database, as well as the use of API calls and machine learning techniques. The proposed model identified only 26 features and can detect both known and unknown ransomware variants on the available sample dataset.

In this section, we discuss the proposed pre-encryption detection model of ransomware, which consists of two phases. The first phase focuses on identifying known ransomware using a signature database. The second phase involves analyzing system calls or APIs to identify ransomware behavior (discussed in Section 4), aided by a trained machine learning algorithm. This approach allows for the detection of previously unknown ransomware. The overview of the proposed model is illustrated in Fig. 3.

In the first phase, the proposed approach allows users to inspect files before opening them to identify potential ransomware threats. The user selects a file for analysis, and the model generates its SHA-256 hash signature, a unique 256-bit fingerprint based on the file’s contents. This hash is then compared against known ransomware hashes stored in the Signature Archive. Hash comparison is more efficient than analyzing the entire file and requires minimal storage. If a match is found, the pre-encryption approach alerts the user and isolates the file. However, this method is limited to detecting known ransomware.

In the case of an unmatched file, the second phase of the proposed model is executed by sending the files to the Cuckoo Sandbox for further analysis of potential unknown malware. The file is executed in a secure virtual environment, where its behavior is monitored. Once the processing is complete, Cuckoo generates an analysis report for the file. In this study, we focus on the APIs that are employed for opening and accessing files. The pre-encryption model identifies these APIs before the encryption function call occurs.

The model searches for the keyword “crypt” within the APIs. If the keyword is found, the API is recorded, and the process proceeds to the next stage. After the APIs are converted into a suitable data format in a comma-separated values (CSV) file. The machine learning model is then applied to the extracted API for classification. The pseudocode is as follows (Pseudocode I).

In this section, we present the proposed methodology for ransomware detection. The study is organized into four key components shown in Fig. 4: Sample Collection, Cuckoo Sandbox, Data Extraction, Machine Learning, and Database. Further details are as follows.

In this section, we evaluate and compare the performance of the proposed and existing techniques using metrics such as accuracy, precision, recall, True Positive Rate (TP Rate), False Positive Rate (FP Rate), F-measure, and cross-validation. The TP Rate measures how effectively the model correctly predicts positive cases, while the FP Rate indicates how often it incorrectly classifies negative cases as positive. Accuracy assesses the overall effectiveness of the model in identifying ransomware, whereas precision evaluates the correctness of positive predictions, and recall measures how well the model detects all relevant positive instances.

The F-measure balances precision and recall, providing a comprehensive assessment of the model’s performance. Additionally, the Receiver Operating Characteristic (ROC) curve represents the TP Rate vs. FP Rate, while the Precision-Recall Curve (PRC) plots precision against recall.

From Fig. 7, it is notable that to achieve 100% accuracy, 232 API features were employed. However, from our experiments, we identified only 46 API calls that are sufficient for maintaining a detection accuracy of 99%. The feature importance based on the random forest classifier highlights the features that contribute the most to the detection mechanism. We further categorized the 46 APIs into reading and writing functions/APIs to identify their significant contributions. The identified read-and-write APIs in the proposed model include 20 read functions and 26 write functions, as detailed in Tables 6 and 7.

Table 8 shows the accuracy of machine learning models along with precision, recall, F1 score, and their cross-validation checks. We achieved 100% accuracy using the Random Forest model, 99.81% using Support Vector Machine, 99.63% using the Decision Tree, 98.09% using K-Nearest Neighbors, 97.29% using Naive Bayes, and 99.18% with the Ensemble Classifier for 46 features. Additionally, when analyzing the reading functions, we obtained the following accuracies: 96.20% using Random Forest, 96.38% using Support Vector Machine, 95.84% using Decision Tree, 96.38% using K-Nearest Neighbors, 92.23% using Naive Bayes, and 96.48% with the Ensemble Classifier.

We focused on the write functions, as they play a crucial role in ransomware activity, especially in the initial stage when files are encrypted, making it more significant than read functions. We achieved 99.45% accuracy using Random Forest, 99.27% with Support Vector Machine, 99.45% with Decision Tree, 99.27% with K-Nearest Neighbors, 97.47% with Naive Bayes, and 99.72% with the Ensemble Classifier, which averages all classifiers. This means that we can achieve the same accuracy as presented previously in [21], using only 26 features of the write function APIs. Our collected dataset is also verified using other machine learning models, Support Vector Machine, Decision Tree, K-Nearest Neighbors, Naive Bayes, and Ensemble Classifier with Random Forest, which showed better results with fewer API functions.

Similarly, for evaluating other evaluation matrices, such as ROC curves, that are used to measure the probability of classification models at different levels. The curve has a true positive rate along the x-axis and a false positive rate along the y-axis. Fig. 8 represents ROC curves of different machine learning models.

The performance of the proposed method is evaluated and compared with recent ransomware detection models, as shown in Table 9. In [32], the authors employed system call data analyzed with various machine learning algorithms, achieving a detection accuracy of up to 91%. Similarly, Kok et al. [21] reported 100% detection accuracy on a custom sample dataset. However, their approach utilized a significantly larger feature set (232 features) compared to the 46 features employed in our method, and their dataset was proprietary. Zhang et al. [26] achieved a comparable 97% accuracy using a minimal feature set of 25 features. However, their dataset was synthetic, and they relied on advanced and complex machine learning models, such as convolutional neural networks (CNNs). Additionally, Akbar et al. [16] achieved 90% accuracy using only 5 features. However, their study focused on general Android malware, including ransomware, without specifying the exact number of ransomware samples analyzed.

In contrast, the proposed method achieved 98% detection accuracy using only 46 features, evaluated on 1846 ransomware-specific samples. These results demonstrate that the proposed approach outperforms existing methods in terms of achieving high detection accuracy with a minimal number of features, offering a promising solution for effective ransomware detection.

This study has presented an early detection approach that effectively identified ransomware activity before encryption occurred. It integrated the two-tiered detection mechanism that combined signature-based hashing techniques with dynamic behavior analysis using Cuckoo Sandbox and machine learning. The first stage used hashing to compare file signatures against known ransomware, enabling rapid detection without execution. The second stage employed a trained machine learning model to analyze pre-encryption API calls, effectively identifying both known and unknown ransomware. Newly detected signatures were stored for future use, enhancing the model’s effectiveness over time. Additionally, the inclusion of critical API call monitoring further strengthened system-level threat analysis, enabling the identification of malicious activity with high precision. Experimental results demonstrated that the proposed approach achieved a 98% detection accuracy using 46 key ransomware attributes, compared to over 250 attributes used in other methods. The results were validated through an 80:20 training-to-testing ratio and 10-fold cross-validation. When compared to existing solutions, the proposed method could optimize the feature selection while maintaining robust performance, ensuring efficiency and lightweight ransomware detection.