Malware is defined as any program that has a malicious intent (malicious software). They are designed to disrupt normal device operation, display unwanted advertising, steal sensitive data, or remotely take control of the user’s device. Viruses, worms, Trojan horses, ransomware, rootkits, and botnets are all types of malwares [1]. Malware systems have evolved to become more sophisticated over the years. Malware uses polymorphic and metamorphic techniques to evade detection by conventional malware detection techniques [2–5]. Extensive static analysis is required to detect newly developed malware that is too sophisticated to evade detection by emulators. Malware can also be propagated through metamorphosis techniques such as multi-packing, registry modification, encryption, anti-debugging, code transformation, virtual machines, and instruction permutation.

The evasion form of malware is intelligent, as it changes its infection routines to include an initial step that carefully inspects the environment in which it runs, allowing it to select the most appropriate time to execute the payload [4,6–9]. By utilizing reusable development modules and automated development, new variants of malware can be created [10–12]. For the purpose of creating new malware variants, malware developers frequently alter small sections of the original source code [11,13,14]. As a result, it is a challenging task to distinguish between different variants of the same family of malware [15,16]. Android apps that are malicious in nature, on the other hand, can infiltrate smartphones and mine cryptocurrency without the user’s knowledge. Mobile malware is expected to continue to grow and proliferate, allowing criminals to commit a wide range of cybercrimes on mobile devices. There was a total of 35 million malware attacks, but according to the mobile threat report [17] published by McAfee in the first quarter of 2020, there were approximately 800,000 new malware attacks that were detected in the fourth quarter of 2019, surpassing the total number of malware attacks. Recently, LeifAccess emerged as a detritus malware with capability of creating and posting fake reviews on Google Play Store by exploiting the OAuth leveraging accessibility advantage. After the installation, it operates in the background without displaying a shortcut or an icon. The emerging of new malware families and increasing number of mobile malwares posed a serious threat to the Android ecosystem’s security. Several studies shaded the light on combating the security concerns faced Android ecosystem by conducing number of research related to the detection and classification of Android malware samples [18,19]. Despite the progress that made in Android malware detection and classification, a new challenge arisen related to the changes on malware malicious behavior over time. Thus, it makes it challengeable task to the machine learning-based malware classifiers to avoid performance deteriorations.

Static analysis and dynamic analysis are the techniques that are considered most important for identifying malicious software. Malware can be identified through a combination of signature and behavioral techniques. Static analysis can be done in situations where a piece of source code is under suspicion without running the program. The source code must be disassembled to extract features [20–22]. This technique is not resistant to obscure code and loading dynamic code [23–26]. On the other hand, the dynamic analysis examines the characteristics and traces of suspect use during the implementation [27–32]. Dynamic analysis is a promising technique, but it is time-and resource-intensive [23,33]. Intelligent malware uses dynamic analysis with anti-emulation technology [34–36]. In addition, it takes a lot of manual effort or human intervention to use static and dynamic techniques on such files. This requires knowledge of the domain to analyze the application or to reverse engineer it [37–41]. For Android malware family classification, the time required to create features manually throughout the Android Application Package (APK) structure is significantly high [4,6,42–45]. These safety mechanisms require high computer resources, and it is challenging to deploy them in a restricted smartphone environment [34]. Android malware traces are being investigated using Classes.dex, resources, manifests, and Android application certificate files [46,47].

Android malware detection has received a great deal of attention in the academic and commercial communities due to the prevalence of attacks against the Android mobile operating system. There is, however, a significant discrepancy between the amount of work that has been done and the number of malicious applications that are published daily.

Recent efforts have been made to reduce the number of malware attacks by testing deep learning and optimization algorithms. The proposed work converts Android malware applications to malware images. A CNN was developed to automatically extract the rich features of malware images using a proven and widely accepted method for classifying images by the research community. As a result, these characteristics were used to classify malicious applications according to their families. This methodology recommends that binary data extracted from Android files be converted into images. Analysts can see through malware binary images without executing them using visualization-based techniques. The performance of feature representation can be accomplished by deep learning algorithms without the need for any assumptions or configuration of parameters. In addition, deep learning models can learn complex patterns and solve the dimensionality problem with little guidance. The following is a list of the primary contributions:

We propose a stacked architecture consisting of ResNet-50 and non-linear Support Vector Machine (SVM) for identification of malware Android APK converted to graylevel images.

This paper is organized as follows. Section 2 reviews the methods used to detect Android malware. Section 3 describes the use of the dataset for Android malware detection and Android package conversion to images. The proposed approach is presented in Section 4. The detailed results are presented and discussed in Sections 5 and 6 respectively. Finally, conclusions are drawn, and future directions are discussed in Section 7.

Researchers have performed visualization-based malware analyses [10,48–50]. The structure of malware images is directly affected by visualization techniques [11,51–53]. In contrast to static and dynamic approaches, visualization-based analyses allow for faster malware sample classification since no disassembly or execution is needed. Thus, the task was to provide a method for classifying many malware samples that was superior to that achieved with conventional technologies. In [54], APK files were converted using Hue–Saturation–Lightness (HSL), Red–Green–Blue (RGB), grayscale, and Cyan–Magenta–Yellow–Black (CMYK). To decide whether an application was benign or malicious, three machine learning classifiers (decision trees, random forest, and k-nearest neighbors) were trained on each image representation using Global Image Descriptors (GISTs). On the grey image, the random forest classification had a high accuracy of 91 percent. A visualization tool was used to fine-grained classify Portable Executable (PE) files [11]. The malware was disguised as an RGB image file. The dataset included 7,087 malware samples from 15 families. By merging global and local malware classification functions, the model was developed. The data and code portions of the file were processed to create feature vectors for local attributes. The RGB image’s global characteristics were deleted. To train the model, random tree, support vector, and k-nearest neighbor classifications were used to train the model. According to the malware classification trial results, the random forest classification had a 97.47% accuracy score. The approach did not function with non-PE files due to the complexity of generating RGB-colored images and extracting valid local features for these malicious codes.

In [55], only the coding for the APK files was considered. As a result, the DEX files were converted into Java Achive (JAR) files using the dex2jar package. A Java Application Descriptor (JAD) tool was used to translate other JAR files into Java files. Each APK file was supplemented with code from different text files. The Word Frequency-Inverse Document Frequency (TF–IDF) technique was used to differentiate keywords in the text records. The TF–IDF weight is a mathematical measure that simplifies evaluating a word’s value in many text files. The cumulative number of words in a text is multiplied by the number of times the word gets the normalized TF. The IDF decides the worth of a word. The number of documents in the corpus is divided by the logarithm of the number of documents in the corpus. This increases the scarcity of the words while reducing their frequency. These files were grouped after the related words were extracted from the text files. The SimHash [56] and the djb2 [57] algorithms were used to process the classes into images. A convolutional neural network was used in the analysis, and it was graded with 92 percent accuracy. META-INF, resources, and AndroidManifest. XML, among other APK file building blocks, was overlooked. Reference [58] included a total of 12,000 malware images from 32 malware families. A vector sustainable machine and the efficiency of customizers including Convolutional Neural Network (CNN) and k-nearest neighbors, Local Binary Patterns [LBPs] and GIST). 93.92% of the chosen dataset had high precision using the six-layer conversion-trained neural network model with LBP functionality. Grayscale and Red–Green–Alpha (RGBA) graphics were used to represent the malware. The LBP-trained CNN model’s productivity was investigated. According to the report, malware may cause essential features of a color image to be lost. In essential malware detection, the decision on which subset of features to include is difficult in machine learning. The design of the correct feature set is required to produce an effective malware detection or analysis model. Reference [59] developed C language visualization tools to investigate the internal configuration (anomalies or patterns) of malware executable files. They mapped the.dex bytes to the image pixels to reveal a set of features for malware classification. Legitimate Android developers use a variety of obfuscation tools to protect their intellectual property. Malware authors exploit and abuse these tools and techniques to make malware versions for Android more resistant. The authors used the visualization-based methodology of [60] to fingerprint the obfuscation tools used in the Android application development period. An image is used to display the malware binary. They used an image to quantify two types of statistical properties, which were then combined to retrieve data from the application developer’s obfuscation software. The accuracy rates for fingerprinting the obfuscation method and classifying the obfuscated and original implementations, respectively, were calculated to be 73 percent and 86 percent, according to the researchers. While the review of the literature establishes that an APK file is a sequence of bits and thus a binary image, there is no clear consensus among researchers on the type of analysis and prominent APK parameters appropriate for malware classification. Traditional malware classification methods rely on the extraction of dynamic and static features. These approaches typically employ code analysis to address the problem of malware classification. Existing malware classification techniques make use of both signature-based and feature-based classification. Unfortunately, these systems have several drawbacks, including excessive resource usage, code obfuscation, and code disassembly. Furthermore, researchers discovered that these methods require a lot of time and space. The era of deep learning-infused visualization approaches is beginning in Android security. The proposed methodology solves the multiclass malware classification problem by combining the power of visualization and deep learning approaches. A deep learning architecture reduces the requirement to gather features such as meta-data information, permissions, API calls, and other dynamic variables to produce a high-quality malware classification model. Recent research in security and privacy has demonstrated the value of solutions combining deep learning and visualization-based analysis [61,62]. For Windows malware, the majority of solutions [10,11,16] had high classification accuracy [10,16]. As a result of only being able to conduct their experiments on Windows systems, the researchers had to use PE files. Windows’s hardware architecture differs greatly from that of mobile devices based on the lightweight Android operating system, which is why Windows is the most popular desktop operating system. Consequently, it is not suitable to classify Android malware families using tools designed for Windows platform applications. For Android malware detection, the reference [63] suggested Integrating Neural Architecture and Visualization Technologies (SARVOTAM). The machine creates fingerprint images from the malware’s non-intuitive features in order to collect high-quality data. The study’s precision was 92.59% when using the DREBIN dataset. As reported in [64], 17 different datasets are used to detect Android malware. It is challenging to perform benchmark tests due to the lack of standard benchmark datasets. Thus, machine learning model performance may fluctuate on some datasets. Furthermore, when obfuscation attacks are used, some static machine learning methods fail to function properly. There is no guarantee that a classifier model based on different datasets is still effective for new malicious applications. The authors of [65] created a model based on feature fusion. In which features derived from deep levels of CNN layers were combined with handmade features such as LBP, GIST and Gray Level Co-occurrence Matrix (GLCM) to create a feature fusion for the classification of Android malware images. They achieved 93.24% accuracy using the malware image combination of CR + AM using the feature fusion-SVM classifier.

In this study, we utilized the transfer learning ability of a complex architecture, ResNet-50, in combination with a non-linear SVM, to identify malicious files. ResNet-50 is a 50-layer architecture that can learn complex patterns in data more easily than other CNN-based architectures. Furthermore, the softmax layer was replaced with a non-linear SVM, which serves as a supplement to the identification task. The proposed stacked architecture is trained and tested on the DEBRIN datasets consisting of malicious Android APKs after converting them to gray-level images. Different combinations were used to test the model’s performance, and results associated with different groups were reported.

Transfer learning uses the previous pre-trained model with some or no modification for another problem. A machine uses the knowledge gained from a previous task to enhance its generalization ability for a subsequent task. This way, it enables us to build a more robust architecture in the most cost-effective way instead of training and fine-tuning from scratch. Various models can be used for transfer learning, such as AlexNet, GooLeNet, and VGG. They stacked many convolutional layers, leading to difficulty optimizing the networks, vanishing gradient problems, and degradation problems. For Android malware detection, various CNN models are used, which require end-to-end training. In this work, the pre-trained ResNet-50 model is utilized instead of end-to-end training. This reduces the execution time and improves the classification results. ResNet-50 is a widely accepted architecture that is beneficial for solving complicated tasks and improving detection performance. It tries to solve the optimization, vanishing gradient, and degradation problems found in other networks by incorporating the data fed to previous layers into the following layers [66]. In this work, we incorporated the previous preprocessing method described in [63], where the malware’s non-intuitive features were converted to fingerprint images in order to extract useful information. For identification, the ResNet-50 is fine-tuned where the softmax layer is replaced with the SVM model with a Gaussian kernel. The ResNet-50 architecture consists of five stages, each consisting of a convolution block and an identity block, as illustrated in Fig. 4.

Each convolution block consists of 3 connected convolution, batch normalization and ReLU units as shown in Fig. 5.

Nevertheless, the identity block is reduced to two units, as depicted in Fig. 6. The softmax layer is replaced with an SVM integrated with a Gaussian kernel. This architecture has over 23 million trainable parameters. Various numbers of layers in the back-end of the model have been tested and evaluated, and it has been concluded that employing a smaller number of layers reduces the classification measures. Nonetheless, the increment in the number of layers has no significant effect on performance and only increases the time to train and execute the model. Furthermore, the classification performance of the proposed model is better than all other state-of-the-art models. However, a major limitation is the complexity of the model during training, which increases the Multiply-Accumulate (MAC) and Floating Point Operations (FLOPs).

As is illustrated in Table 3, the architecture of the proposed model consists of the following elements: The first layer contains a convolution with a kernel size of 7 × 7 and 64 kernels with a stride of size 2. Next, conv2 is a 1 × 1 convolution, 64 kernels following 3 × 3, 64 kernels, and a 1 × 1, 256 kernels. These three layers are repeated three times for a total of nine layers. In the conv3 convolution block, there are 1 × 1, 128 kernels. Next is a kernel of 3 × 3, 128 kernels following 1 × 1, 512 kernel convolutions, repeated four times for 12 layers. Similarly, the next convolution block, conv5, consists of 1 × 1, 256 kernels following 3 × 3, 256 kernels, leading up to 1 × 1, 1,024 kernels. This process was repeated six times for a total of 18 layers. In the last convolution block, there were 1 × 1, 512 kernels followed by 3 × 3, 512 kernels and 1 × 1, 2,048 kernels. This was repeated three times for a total of nine layers in this block. The last layer, which consists of 1,000 nodes with softmax entropy, is replaced with nonlinear kernels to identify malicious images. The simulations were run on a system with 20 GB of RAM and an Intel® CoreTM i3 processor, as well as an NVIDIA GeForce GTX 1080ti graphics processing unit with a frame buffer of 11 GB.

To demonstrate the detection performance of the proposed model, the confusion matrix is generated, as illustrated in Table 4. Furthermore, we computed precision, recall, accuracy, and F1 measures from the information presented in Table 5.

Accuracy is computed by dividing the total number of true positives (TP) and true negatives (TN) by the total number of predictions, i.e., all entries in the confusion matrix. Recall, which is also known as the true positive rate (TPR), is calculated by dividing the total number of TP by the sum of TP and TN. It shows the number of instances correctly classified by the model. Similarly, precision (PR) is another measure that considers the number of instances that were considered malicious, but they were non-malicious, also known as false positives (FP). All these quantitative measures are shown in Table 6, which demonstrates a 97% accuracy, while the other measures are at 95%. The performance of the proposed model is higher on malware families such as certificates and Android manifests. This represents that the model learned and perceived the actual behaviors of these files.

However, the performance is lower on ExploitLinuxLootor, MobileTx, Gappusin, and BadeBridge due to fewer samples relative to other malware types. Furthermore, the performance is the lowest of the DroidDream, MobileTx, Gappusin, and ExploitLinuxLooter malware families. These malware families exhibit a significantly lower number of samples in the training dataset, and it seems to affect rooted Android devices. There is a high certainty that these types alter their signatures after getting root access to the device. This statement can be analyzed in the future, which opens a new dimension of research to evaluate existing algorithms on rooted and non-rooted devices. It can also be seen from the results that DroidDream, Imlog, and DroidKungFu have a lower recall rate. However, their precision is high. Such contrasting results illustrate that the nature of these files learned by the model is different from other families. In Fig. 7, we show the number of instances of each class misclassified by the proposed model. During training, the classification models try to extract discriminative features. If the input data contains a lot of information and relevant features, the classification task of the model becomes more accurate. As shown in this study, various combinations of the images were used in classification to increase the accuracy. We have seen that the combination of CR and AM has produced the maximum classification results. As a result, time and effort can be saved in inspecting the entire APK structure for Android malware classification. Misclassification occurs as a result of similarity between images of different classes. In order to resolve this issue in future work, we need to propose a common framework based on frequency transformation, which could be distinguished in similar images.

In our line of work, the proposed model performed admirably for approximately 100 epochs. The simulation results were recorded for the DREBIN dataset after converting malicious Android apps into fingerprint images and utilizing AM and CR images. These results were recorded after using AM and CR images. Our model’s performance was evaluated in comparison to that of several other models considered to be state-of-the-art, and the results showed that our model’s performance was superior. As can be seen in Fig. 8, the evaluation is compared for a number of different possible combinations of the image types. Table 7 shows the highest level of accuracy achieved using the combination of AM and CR.

Both the observations and the results of the simulations indicate that both files contain the maximum amount of relevant information about different types of malicious software, which results in satisfactory classification performance. In addition to measures of classification, Table 8 provides a time-based comparison for each combination that was utilized in the research, as well as the number of images that were processed per second and belonged to the appropriate class. Consequently, once the model has been trained with high-quality classification metrics, it can be utilized for testing in a wide variety of different applications. Table 8 shows the average processing time required to process a single image, which is comparable to that of earlier studies [63]. Therefore, once the model is integrated and utilized in software systems, its execution performance will be identical to that of the state-of-the-art method.

In this study, we proposed a classification model for Android malware that uses ResNet-50 and SVM. The ResNet-50 was used because it has transferable learning abilities. The first step was to use substrings from many binary malware files in the DREBIN dataset to generate vectors in the 8-bit range. The next step is to convert these vectors into grayscale images. The ResNet-50 model’s softmax layer for classification is replaced by a support vector machine (SVM), which uses a non-linear kernel to improve detection performance. In addition, various combinations of the images were used to fine-tune the model in search of the files that had the greatest impact on the model. From simulation results, it can be concluded that the certificate and Android manifest (CR + AM) are the most suitable features for identifying and classifying malware, as they contain sufficient information. Using the CR and AM images, we reported the highest accuracy, recall, precision, and F1 measures. When using the DREBIN dataset, the highest level of accuracy achieved was 97%. In the future, we intend to extend the evaluation to include malware on additional platforms to evaluate the effectiveness of our model.