Several CX-R images were carefully selected from different sources to create a relatively large-scale COVID-19 CX-R image dataset of confirmed infected cases. This dataset was named as COVID19-vs.-Normal and then mixed with some CX-R images of normal cases for a reliable diagnosis of COVID-19 virus. The sources of the COVID19-vs.-Normal dataset are as follows:

A set of 200 CX-R images with confirmed COVID-19 infection of Cohen’s GitHub repository [20].

Samples of the COVID-19 and normal cases of the large-scale COVID-19 CX-R images are shown in Fig. 3. The established COVID19-vs.-Normal dataset will have a constantly updated number of the COVID-19 cases depending on the availability of new CX-R images with confirmed COVID-19 infection and is available publicly at https://github.com/AlaaSulaiman/COVID19-vs.-Normal-dataset. Data augmentation was applied to prevent overfitting and enhance the generalization ability of the last trained model. First, the size of the original image was rescaled to (224×224) pixels, and five random image regions of size (128×128) pixels were then extracted from each image. Horizontal flip and rotation of 5 degrees (e.g., clockwise and counter-clockwise) were then conducted for every single image in the dataset. A total of 24,000 CX-R images of size (128×128) pixels were extracted from both classes (e.g., COVID-19 and normal images). Data augmentation was implemented after dividing the COVID19-vs.-Normal dataset into three mutually exclusive sets (e.g., training, validation, and testing set) to avoid generating biased prediction results.

A raw CX-R image obtained by an electronic detector usually has poor quality and thus may be unsuitable for detection and diagnosis. Image enhancement methods should be applied to enhance the quality of CX-R images. Furthermore, training the DNNs on the top of pre-processed images instead of using raw images data can substantially reduce the generalization error of the DNNs and their training time. Hence, an effective image enhancement procedure was proposed to enhance the CX-R image’s poor quality prior to feeding to the proposed approaches (e.g., DBN and CDBN). First, the small details, textures, and low contrast of the CX-R image was enhanced through adaptive contrast enhancement based on CLAHE [36]. CLAHE is different from the original histogram equalization method that computes several histograms (e.g., each one corresponding to a distinct part of an image) to redistribute the lightness values of the input image, as depicted in Fig. 4b. Hence, this method can improve the image local contrast and enhance the visibility of the edges and curves in each part of an image. Second, the Butterworth bandpass filter was employed to reduce the noise in the image produced from the previous step, as shown in Fig. 4c. The Butterworth Bandpass filter was calculated by multiplying the low and high pass filters as follows:

(1) HLP(u,v)=11+[F(u,v)/FL]2n,

(2) HHP(u,v)=1-11+[F(u,v)/FH]2n,

where FL and FH are the cut frequencies of the low and high pass filters set as 15 and 30, respectively; n=3 is the filter order; and is F(u,v) the distance from the origin.

A hybrid deep learning detection system based on the incorporation of two discriminative deep learning approaches (e.g., DBN and CDBN) was proposed to detect COVID-19 infection in CX-R images. To the authors’ best knowledge, the possibility of using DBN and CDBN in a unified system to detect the COVID-19 virus in the CX-R images has not been previously investigated. Fig. 2 shows that the enhanced image was fed into the visible units of the proposed deep learning approaches to learn high-level feature representations. DBN is a new generative probabilistic model developed by Hinton et al. [37]. Different from other conventional deep neural networks (DNNs), DBN has one visible layer and several hidden layers that can learn the statistical correlations of the neurons in the previous layer [38]. Similar to other deep learning approaches, DBNs are directly applied to raw image data. Although DBNs have been effectively applied to solve many challenging problems (e.g., face recognition [39] and audio classification [40]), scaling them to high dimensional images is challenging for two reasons. First, the input image with a high dimensionality can increase the complexity of the learning process and require a long-time for convergence. Second, the features learned by DBNs are highly sensitive to image translations, especially when the raw image data are assigned directly to the visible layer. This phenomenon can lead to discarding most of the fine details in the input image, thus seriously affecting their performance. As a solution, the proposed DBN model was trained on the top of pre-processed images rather than raw image data to remarkably reduce the training time and learn additional discriminative feature representations [41]. Assigning pre-processed CX-R images to the input layers of DBN and CDBN can remarkably improve their ability to learn essential and prominent feature representations with less time require to obtain the last trained models. As depicted in Fig. 5, the main architecture of the proposed DBN is composed of stacking five RBMs as hidden layers. The first four RBMs can be viewed as non-linear features descriptors trained sequentially using the CD learning algorithm in an unsupervised greedy layer-wised manner to learn a multi-layer non-linear generative model. The last RBM is a discriminative RBM (DRBM) trained as a non-linear classifier associated with SoftMax function to produce the probability distribution of each class label. DRBM comprises two layers of visible units to represent the input vector and a softmax label unit to represent the predicted class. During RBM training, the stochastic gradient descent algorithm was applied to maximize the log-likelihood of the training data. Hence, the updating rules for the weights can be defined as follows:

(4) Δwi,j=ϵ(〈1σi2vihj〉data-〈1σi2vihj〉model),

(5) Δbi=ϵ(〈1σi2vi〉data-〈1σi2vi〉model),

where ϵ refers to the learning rate, 〈⋅〉data and 〈⋅〉model denote the positive stage and the negative stage, respectively. Finally, bi and ci and represent biases for visible and hidden units, respectively. Calculating the 〈vihj〉model in Eq. (4) is difficult. Thus, the CD algorithm [42] was used to update the parameters of a given RBM by applying k steps Gibbs sampling from the probability distribution to compute the second term in Eq. (4). The single-step of the CD algorithm can be implemented as follows:

Initially, the training data are given to the visible units (vi) to compute the probabilities of the hidden units. A hidden activation (hj) vector is then sampled from the same probability distribution.

Herein, the k parameter of the CD learning algorithm was set to 1, and all the weights were randomly set with small values computed from a normal distribution of zero mean and SD of 0.02. CDBN is a hierarchical generative representation developed by Lee et al. [43] and is composed of several convolutional RBMs (CRBMs) stacked on each other as building blocks. CRBM is an expansion of the traditional RBM. Different from RBM, the weights between the visible and hidden units in the CRBM are locally shared among all positions in the input image. This scheme of sharing parameters (weights) introduces a form of translational invariance that uses the same-trained filter to detect specific useful features at different locations in an image. The proposed CDBN consists of three stacked CRBMs associated with probabilistic max pooling. CRBM uses Gaussian-valued visible units and binary-valued hidden units. The first and second CRBMs consist of 32 and 64 trainable filters (K1=32, and K2=64) of (5×5) pixels, respectively, and the last CRBM consists of 128 trainable filters (K3=128) of (6×6) pixels. The max-pooling ratio is fixed to two for each pooling layer. The output of the last CRBM is fed into one fully connected layer composed of 256 units, followed by the application of a SoftMax function to produce the probability distribution of each class label. CDBM models are highly overcomplete because each CRBM (hidden layer) has K trainable filters (e.g., groups of units) with sizes roughly equal to that of the input image. In general, the overcomplete model runs the risk of learning trivial feature representations (e.g., single-pixel detectors). As a solution, a sparsity penalty term was added to the objective function to obtain a small part of the fired output. In practice, the following simple update process (applied before weight updates) can be employed:

(7) Δpksparsity∝p-1NH2∑i,jP(hijk=1∣v),

where p is referred to as the target sparsity. In this work, the target sparsity was set as 0.005 for all the CRBMs. The proposed training methodology for DBN and CDBN models is composed of three phases: unsupervised pre-training, supervised, and fine-tuning.

Similar to other deep learning networks, DBN and CDBN need a massive amount of training data to prevent overfitting during the training process, reduce the generalization error of the last obtained model, and achieve satisfactory predictions. Thus, simple data augmentation was implemented to artificially increase the number of training samples in the COVID19-vs.-Normal dataset (see Subsection 3.1).

In the prediction phase, the average values of seven quantitative performance measures, namely, detection accuracy rate (DAR), sensitivity, specificity, precision, F1-score, mean squared error (MSE), and root Mean Squared Error (RMSE) were computed to measure the accuracy and the efficiency of the proposed COVID-DeepNet model by using the testing set. These seven quantitative performance measures are calculated as follows:

(8) DAR=(TP+TN)(TP+TN+FP+FN),

(9) Sensitivity(Recall)=(TP)(TP+FN),‘

where TP, TN, FP, and FN stand for true positives, true negatives, false positives, and false negatives, respectively.

(13) MSE=1n∑i=1n(Yi-Y^i)2,

(14) RMSE=1n∑i=1n(Yi-Y^i)2,

where n refers to the total number of data samples, Y is the vector of observed values of the variable being predicted, and Y^ is the vector of n predicted values.

The main architecture of the COVID-DeepNet system is based on the combined output scores produced from two discriminative deep learning methods (e.g., DBN and CDBN). The main challenging task of using deep learning models is the huge number of structures and hyper-parameters to be assessed (e.g., number of hidden layers, filter size, number of epochs, and learning rate). Herein, several experiments were performed to find the best model’s architecture for DBN and CDBN. The influence of different values of the hyper-parameters on the performance of the proposed approaches was also analyzed. Two different training set configurations from the COVID19-vs.-Normal dataset were created and evaluated to reveal the important contribution of the proposed image enhancement procedure in guiding the learning of proposed approaches and improving their performance compared with the use of raw images as input data. The two training set configurations were as follows: (i) TrainingSet_1 consists of raw images and those produced using the proposed data augmentation procedure, and (ii) TraningSet_2 consists of the pre-processed and those produced using the proposed data augmentation procedure.

In DBNs and CDBMs, the values of their hyper-parameters mainly depend on each other. Furthermore, the hyper-parameters values used in a specific RBM may be affected by the hyper-parameter values used in other RBMs. Thus, the hyper-parameter fine-tuning task in these two approaches requires a large cost. Therefore, a coarse search procedure was implemented to find the best hyper-parameter values. As shown in Tab. 1, the proposed DBN model composed of stacking five RBMs (hidden layers) was trained in a bottom-up way using the proposed training methodology presented in (Subsection 3.3). The number of the hidden units in the first two layers was fixed to 4096 units, and the different numbers of hidden units were evaluated in the last three layers to find the best network configuration. When the unsupervised training of the first RBM (the first hidden layer) was completed, the weight matrix of the hidden layer was frozen and was used as an input data for the training of the second RBM (the second hidden layer) in the stack. With the CD learning algorithm (e.g., one step of Gibbs sampling), the first four RBMs were trained separately in an unsupervised greedily manner. Each RBM (hidden layer) was trained for 100 epochs with a mini-batch size of 100, a weight decay value of 0.0002, a learning rate of 10⁻², and a momentum value of 0.9. The weights were randomly initialized with small values computed from a normal distribution of zero mean and SD of 0.02. The last layer was trained as a non-linear DRBM classifier with SoftMax units to produce the final probability scores. The last DRBM was trained using the same hyper-parameters values of the first four RBMs. Finally, the back-propagation algorithm equipped with the dropout method was applied for the parameter fine-tuning of the whole DBN model in a top-down manner to avoid overfitting and achieve satisfactory predictions. The dropout ratio was 0.5. Initially, the whole DBN was trained in a top-down manner for 100 epochs; however, the model can be further improved by increasing the number of epochs. Therefore, the number of epochs was set to approximately 500 epochs using the early stopping procedure. As revealed in Tab. 1, five DBN models were trained using two different training sets (e.g., TrainingSet_1 and TrainingSet_2), and the highest accuracy was obtained using the fourth DBN (4096-4096-3000-2048-1024) model. The training time of all the five trained models was substantially decreased by training them on the top of the TrainingSet_2 containing only the processed images data. This finding confirms our argument that training the proposed DNNs on the top of the pre-processed images can remarkably improve their ability to rapidly learn useful feature representations with less time required to obtain the last trained model. Therefore, TrainingSet_2 was used for all subsequent experiments. Additional information on the DBN and its hyper-parameters are given in Tab. 2. For an initial CDBN architecture, only two CRBMs were greedily trained using the same proposed trained methodology described in (Subsection 3.3). This initial CDBN architecture was referred to as a CDBN-A in the subsequent experiments. The number of filters was initially set to 32 filters at each CRBM layer, and the size of the filter was set to (5×5) pixels. The CD learning algorithm (e.g., one step of Gibbs sampling) was used to train all the CRBMs in an unsupervised greedily manner. Each CRBM was trained separately for 100 epochs with a mini-batch size of 100, target sparsity of 0.005, learning rate of 10⁻², a weight decay value of 0.0005, and a momentum value of 0.95. The weights were randomly initialized with small values computed from a normal distribution of zero mean and SD of 0.02.

The dropout technique with the dropout ratio of 0.5 was applied only for the fully-connected layer. In the fine-tuning phase, the weights of the whole CDBN-A model were optimized using the back-propagation algorithm in a top-down manner to achieve satisfactory predictions. First, the CDBN-A was trained for 500 epochs with a mini-batch size of 100. However, the performance of the CDBN-A model using the validation data declined when 500 epochs were evaluated because the last trained model started overfitting the training set (see Fig. 6).

For the CDBN model, this hyper-parameter was determined empirically by varying its value from 100 epochs to 500 epochs in steps of 10. The highest validation accuracy rate was obtained from the validation set by training the CDBN-A model in a top-down manner for 300 epochs. Moreover, a high accuracy rate can be achieved by adding a new CRBM with 128 trainable filters of size (6×6) pixels and changing the number of filters in the second CRBM from 32 trainable filters to 64 trainable filters. This newly added CRBM was trained using the same hyper-parameters of the other CRBM. This new architecture was denoted as a CDBN-B model and used for all remaining experiments instead of the CDBN-A model. Additional information on the CDBN-B and its hyper-parameters are presented in Tab. 2. The ROC curves of the CDBN-A and CDBN-B models are shown in Fig. 7 to visualize their performances on the testing set of the COVID19-vs.-Normal dataset. Fig. 8 shows the learned high-level feature representations from the last hidden layers in CDBN-A and CDBN-B model after training.

The proposed COVID-DeepNet system makes the final decision by integrating the results produced from two different deep learning models (e.g., DBN and CDBN). Every time a CX-R image is assigned to the proposed COVID-DeepNet system, two predicted probability scores are computed, and the highest probability score is used to assign the input image to one of two classes (e.g., either normal or COVID-19 class). In this section, the results obtained from the DBN and CDBN models were combined and evaluated using different fusion rules in the score-level fusion (e.g., using sum, weighted sum, product, max, and min rule) and decision-level fusion (e.g., using AND OR rule). Additional information on how these fusion rules are implemented in both levels can be found in [44].

Parallel architecture, which provides radiologists a high degree of confidence to make their final decision and to accurately distinguish between healthy and COVID-19 infected subjects, was considered in the proposed COVID-DeepNet system. During the implementation of the weighted sum rule (WSR) at the score-level, a slightly higher weight value was given to the CDBN-B model than to the DBN model due to the better performance of the former. Moreover, normalization is not required prior to applying the score fusion rules because both classifiers generate the same probability scores and within the same numeric range [0,1]. Herein, the average values of seven quantitative performance measures using various fusion rules at the score- and decision-level fusion are presented in Tabs. 3 and 4, respectively.

The accuracy of the proposed hybrid COVID-DeepNet system was remarkably enhanced compared with that of DBN or CDBN alone. The highest values of the adopted seven quantitative measures were obtained using the WSR and the OR rule in the score- and decision-level fusion, respectively. The proposed COVID-DeepNet system correctly and accurately diagnosed the patients with COVID-19 in the score-level fusion with a DAR of 99.93%, sensitivity of 99.90%, specificity of 100%, precision of 100%, F1-score of 99.93%, MSE of 0.021%, and RMSE of 0.016% using the WSR and in the decision-level fusion using the OR rule with a DAR of 99.35%, sensitivity of 99.33%, specificity of 99.37%, precision of 99.37%, F1-score of 99.35%, MSE of 0.064%, and RMSE of 0.036%. The high precision value of 100% achieved in the score-level fusion using the WSR is essential in reducing the number of misclassified healthy cases as COVID-19 cases. Finally, the two confusion matrices of COVID-19 infected and normal test results using the WSR and OR rule are shown in Fig. 9. Under WSR rule, only one COVID-19 infected image was misidentified as a healthy image, and three healthy images were misclassified as COVID-19. Under OR rule in the decision-level fusion, 19 COVID-19 infected images were misclassified as healthy images, and 20 healthy images were misclassified as COVID-19. Thus, WSR was used in the performance comparison of the proposed COVID-DeepNet system with current state-of-the-art systems due to its effectiveness in exploiting the strength of each classifier. These results further strengthened the possibility of employing the proposed COVID-DeepNet system in real-world settings to seriously moderate the workload of radiologists and help them accurately detect COVID-19 infection by using CX-R images.

The efficiency and reliability of the proposed COVID-DeepNet system were compared with those of the most current state-of-the-art COVID-19 detection systems. The first three COVID-19 detection systems were evaluated on the COVIDx dataset containing only 76 CX-R images with confirmed COVID-19. The first system was developed by Wang et al. [19] using a deep tailored designed model based on a CNN termed as a COVID-Net. The second system was proposed by Farooq et al. [27] by employing a pre-trained ResNet-50 model termed as a COVID-ResNet. The third system was proposed by Luz et al. [45]. The performance of different architectures of EfficientNet was assessed using an updated version of the COVIDx dataset containing 183 chest radiography images with confirmed COVID-19. The performances of these three systems were evaluated by computing four quantitative measures (e.g., accuracy, sensitivity, precision, and F1-score) for three different classes (e.g., normal, non-COVID19, and COVID-19). For an impartial comparison, these four quantitative measures were averaged, and the values are shown in Tab. 5. The proposed COVID-DeepNet system obtained better results compared with the other systems. Although the EfficientNet B3 model described in [45] achieved the same precision of 100%, the proposed COVID-DeepNet system produced better results in the other two measures (e.g., accuracy and sensitivity) by using a large dataset containing many CX-R images with confirmed COVID-19. A comparison study among three different CNN models (e.g., InceptionV3, ResNet50, and Inception-ResNetV2) was conducted by Narin et al. [23] to detect COVID-19 infected patients using CX-R images. The mean values of five different quantitative measures (e.g., accuracy, recall, specificity, precision, and F1-score) were calculated using fivefold cross-validation to assess their performance of these systems. The best performance was obtained using the pre-trained ResNet50 model with an accuracy rate of 98%, recall of 96%, and a specificity value of 100%. Although the ResNet50 model achieved the same sensitivity and precision with the proposed COVID-DeepNet system, its results for the other five measurements were inferior (Tab. 6).