Our dataset consists of chest X-ray images with Posterior Anterior (PA) view as shown in Fig. 2 (more detailed information is provided in Section 3.). We obtained the datasets from reliable online sources such as Kaggle and NIH databases [21–23]. The datasets contain 5 classes which are Atelectasis, COVID-19, Normal, Pneumonia, and Pneumothorax.

Chest X-ray images were first in grayscale format and then converted into red-green-blue (RGB) format for processing with a deep learning model and feature extraction method. The images were 224 × 224 with 3 color channels in RGB format. For each class, we mapped the images to their labels. For binary-class classifications, 0 and 1 are used as our output. For categorical or multi-class classification, one-hot encoding is used to transform the numeric label into values that represent each class. Normal means having no lung disease.

Deep learning is one of machine learning techniques that uses artificial neural network as its core. Convolutional Neural Network (CNN) is a well-known technique that is used to classify images. Deep learning usually consists of many layers. The network uses a back-propagation learning algorithm to adapt the weights linking neurons in the network to one another. The weights determine the final output of the network when given an input to classify. Deep learning already contains feature extraction, unlike classical machine learning that requires an additional feature extraction process before classification as shown in Fig. 3.

Convolutional neural networks (CNN) [24] are specialized for handling spatially distributed data such as images. They contain convolutional layers in which the output from a particular neuron or unit depends not only on its activation value but also on its position in an array, with respect to other units. The convolutional layers of a CNN are primarily responsible for learning spatial patterns, that is, features. To work well, CNNs also include other types of layers, including pooling layers, drop-out layers, and fully connected layers.

The convolutional layer is used to collect information from image inputs and recombine them into new images or features that will be fed into the deeper layer. The convolution layers are the core of the convolutional neural network. The convolution operation is mathematically defined in Eq. (1), where G is feature mapping, h is a kernel, and f is an input image. Pooling layers are used to reduce the dimensionality of the network and summarize the output from convolutional layers. Dropout layers randomly discard some of the output weights from the layer. Dropout layers slow the rate of learning but help make the network generalize better by reducing overfitting. The fully interconnected layer (FC) is the last part of the network. It is used to classify the input and gives predicted output. The input will be converted from two-dimensional arrays to a one-dimensional vector (called “flattened”) before entering this layer.(1) G[m,n]=(f∗h)[m,n]=∑j⁡∑k⁡h[j,k]f[m−j,n−k]

A popular method used in training a deep learning model is called “transfer learning” [25]. The method adapts the knowledge of an already trained model for a new training goal. For classification, the transfer learning freezes the trained capability of a previously trained model and removes the classification layers which are fully connected layers. Then, we replace them with new fully connected layers and train the model further to reach our goal. The transfer learning uses the lower-level features learned by the original model, allowing them to be refined and applied in a new model to solve a new problem. Many successful architectures have used the ImageNet training dataset [26] and gained high detection performance. VGG is one of them. It has 2 main variation models which are VGG16 and VGG19 [27]. The VGG16 model has 16 layers while the VGG19 model has 19 layers as shown in Fig. 4. The layer counting does not include max pooling layers and fully connected layers.

Unlike CNN, classical machine learning techniques require the researcher to define and extract features (dimensions or attributes) from the input before training the model. Feature extraction calculates values from the input that represent the important characteristics of the data. In this work, we use pre-trained features to extract the feature from our data.

The pre-trained feature approach borrows the idea from the transfer learning model with trained weights. Usually, transfer learning is applied to a new CNN model, but in our case, the deep-learning-trained features are used with classical ML approaches. We start with the VGG19 model trained with the ImageNet dataset. The fully connected layers in the model are removed, so that the last layer is the max-pooling layer, as shown in Fig. 5. The model receives input size 224 × 224 × 3 and gives an output size of 7 × 7 × 512. Them, the output features are flattened to produce 25,088 features and will be later used as the input to the classical machine learning techniques.

Like deep learning models, supervised classical machine learning techniques must be trained with labeled samples of the input data. However, unlike deep learning, these techniques do not learn the best features in the input dataset by themselves. Instead, the input data must be transformed into a set of selected features, for each training and testing data item. In this work, we present two effective machine learning techniques which are Support Vector Machine (SVM) and Random Forest (RF).

SVM [28] is a machine learning technique that separates data items (input data items, represented as vectors of feature values) into classes in high dimensional space by using a hyperplane. The data items in the space can be handled with a selected kernel function that maps the data into a different form. There are 3 common kernel functions which are linear, polynomial, and Gaussian functions.

RF [29] is an ensemble technique that consists of multiple decision trees [30]. The RF method uses a randomly selected subset of data features to create a tree and creates many trees instead of one. The data is classified based on the result of many trees, using some aggregation algorithm.

The chest X-ray dataset used in our experiments consists of 5 classes which are normal, COVID-19, pneumonia, pneumothorax, and atelectasis. These diseases are well-known and commonly found all around the world. The input images were obtained from online databases at Kaggle (kaggle.com) and the National Institutes of Health (nihcc.app.box.com). Details of the datasets are shown in Tab. 2.

The parameter setting for deep learning and classical machine learning are presented in Tabs. 3 and 4, respectively. The deep learning models that we consider are VGG 16 and VGG19, while the machine learning models are SVM and RF.

We considered both binary (2 classes) and categorical (3–5 classes) classification tasks. For the binary classification, two pairwise discriminations: normal vs. COVID-19 (case 1) and COVID-19 vs. non-COVID-19 (case 2) were examined. For the categorical classification, 3 classes (normal, COVID-19 and others) and 5 classes (normal, COVID-19, pneumonia, pneumothorax, and atelectasis) were considered. Tab. 5 shows the detailed experimental design and input dataset for each experimental case.

For evaluating our detection/classification models, we use two methods to separate the training and testing data which are a hold-out method and a k-fold cross-validation method. In the deep learning experiments, we used only the hold-out method. The method splits the training, testing, and validation data into three non-overlapping sets. Validation data is only used to observe the result from model training to find the best time to stop training to avoid over-trained or overfitting. K-fold cross-validation splits the data into k subsets. Then it tests each subset while using the model trained from the remaining subsets. In our experiment, the k-fold method is only used in classical machine learning. The detailed information of our evaluation is shown in Tab. 6.

Performance evaluation metrics are calculated from the confusion matrix including accuracy, precision, recall, and F1-score. The definitions of the confusion matrix and performance evaluation metrics are shown in Fig. 6 and Tab. 7, respectively, where TP is true positive, FP is false positive, FN is false negative, and TN is True negative.

In this paper, all detection/classification models were developed/trained using a personal computer with i5-9500F 3.00 GHz Processor, 16 GB RAM, and GTX 1060 Graphics card running on the operating system Windows 10, 64-bit Pro. We conducted the experiments using Python 3 with Keras deep learning library and Scikit learn classical machine learning library.

Our first part of the experiments focused on pure deep learning methods using transfer learning with ImageNet weight. The experiment contained 4 cases which are Normal vs. COVID-19, COVID-19 vs. non-COVID-19, Normal vs. COVID-19 vs. Others, and all 5 classes. As discussed in Section 3.4, the experimental cases split the train-test-validation in the ratio of 70:20:10 using the hold-out method. The training/validation accuracy and training/validation loss results of VGG19 are graphically displayed in Fig. 7.

From Fig. 7, we see that the accuracy of both training and validation for all models increase rapidly during the first 10 iterations/epochs and then gradually increase. The first row presents result obtained from the first experimental case, while the 2^nd-4^th rows present results obtained from the 2^nd-4^th cases, respectively. During this training time, each classification model learns to adjust itself so that the difference between the two detection/accuracy rates (training and validation) is small and becomes stable. Then, the models are ready for testing with separate testing datasets, where results are presented with confusion matrices as shown in Fig. 8, and the corresponding performance evaluation metrics are shown in Tabs. 8 and 9.

From Fig. 8, we can see that the classification results obtained from the VGG16 and VGG19 in each experimental case are similar. COVID-19 images can be mostly detected/classified. In particular, in cases 1–2 with binary-class classification, less than 50 X-ray images out of 1400 images (700 COVID-19 images and 700 non-COVID-19 images) were misclassified. Normal images, shown in case 1, can be identified mostly as well. Considering multi-class classification, cases 3–4, more misclassified images were obtained. However, we were still able to identify COVID-19 correctly mostly, apart from normal and other diseases, as shown in case 3. Then, with more output classes to classify, in case 4, we had most of the misclassified images belong to Pneumothorax and Atelectasis classes. The performance of the VGG16 and VGG19 can be considered in detail as presented in the following tables.

From Tabs. 8 and 9, experimental cases 1 and 2 which are binary class classification gave the most promising results with over 0.95 on the accuracy, recall, precision, and F1-score. For experimental cases 3 and 4, we consider more lung diseases for classification with 3 and 5 classes, respectively. Both of the experiments give good results on COVID-19 detection/classification. In experimental case 4 with 5-class classification, the performance in COVID-19 classification is 0.934 accuracy, while pneumonia classification is the best, with 0.968 accuracy.

In summary, both VGG16 and VGG19 models give a similar performance at binary-class classification. However, VGG19 performs slightly better than VGG16 at multi-class classification.

Next, we considered classical ML techniques with pre-trained features of VGG19 as input data. All experimental cases use 5-fold cross-validation method. The input images are in red-green-blue format. The performance evaluation metrics for SVM and RF are presented in Tabs. 10 and 11, respectively, where SVM was found to have superior performance among all classification techniques that we considered in all experiments.

From Tab. 10, the SVM model gave very high accuracy and overall performance in all experimental cases, with 0.99 accuracy for COVID-19 classification, while less accuracy was obtained for normal or other disease classification. Other measures which are recall, precision and F1-score were obtained with similar results as the accuracy as shown in the table.

Tab. 11 presented the results obtained with the RF model, where COVID-19 detection was obtained with 0.96–0.98 accuracy and similar values for recall, precision, and F1-score. The RF is not as good as the SVM in solving these classification problems.

In our experiments, deep learning and classical machine learning models were trained to classify the X-ray images. We wanted to compare our computation time between using deep learning (VGG19) and classical machine learning (SVM and RF). The computation time is separated into two parts which are image pre-processing time and training time. The pre-processing time includes the time when images are resized, reformatted, encoded, and feature extracted. This is shown in Tab. 12. The computation time used for model training is shown in Tab. 13.

The image pre-processing time for the deep learning model is less than the time required for image pre-processing for classical machine learning because the deep learning model does not require feature extraction. However, training each deep learning model takes a significantly longer time than training a classical machine learning model, as shown in Tab. 13. In addition, for experimental case 3, the amount of data used in training each model is a lot more than in other cases. Thus, the SVM used more time with an average of 1871.2 s which is close to the deep learning training time. However, RF is the model that requires the lowest training time in all experimental cases.

To further demonstrate the effectiveness of our SVM model to identify COVID-19 infected images apart from many other lung-infected/abnormal and normal images, we consider a well-known NIH dataset that consists of X-ray images of many lung diseases and abnormalities. The dataset does not have COVID-19 X-ray images, so we include in this last experiment the set of COVID-19 images that were used in our previous experimental cases. So, this last experiment consists of 3500 COVID-19 images and 3500 non-COVID-19 images from the NIH database.

We sampled 3500 images from the dataset consisting of normal (no finding), multi-finding, and many lung diseases and abnormalities as shown in Tab. 14. We want to perform COVID-19 detection against other diseases as well as normal. This is a binary class classification with 5-fold cross-validation. In this case, the VGG19 pre-trained features are used as input and the classification result is shown in Tab. 15.

In this experiment with our SVM model, COVID-19 images can be detected with 0.984 accuracy on average of 5 folds, and sometimes as high as 0.992 accuracy in a fold which is quite impressive. The other performance metrics which are recall, precision and F1-score also have very high values ranging between 0.98–0.99 on average, and sometimes as high as 0.997, as shown in Tab. 15. This means the SVM model can detect COVID-19 accurately against many other findings in the chest X-ray images.