Cancer is a crucial disease in women around the world which causes death. Timely and early detection of cancer can save the life of women patients. Most cancer causes deaths in women patients are categorized into breast and cervical. When compared with the death history of women patients due to cervical cancer, breast cancer is the most dangerous and life-threatening disease [1–3]. Breast cancer can be screened by two different methods as Histopathology and mammogram. The histopathology method finds the abnormality of the nucleolus of the breast cells. This method is cost expensive and the breast cancer detection accuracy is low. These limitations are overcome by screening and inspecting the cancer regions in mammogram images. Bypassing the low-intensity and low-profile laser beam through the breast cells in breast, mammogram images are obtained [4]. Breast cancers are categorized into benign and malignant based on the size of the cancer regions. The benign is low-spreading cancer and its spreading ratio is low when compared with other types. Malignant is high-spreading cancer and its spreading ratio is high when compared with the benign type. It is very important to detect both benign and malignant for further breast cancer diagnosis process. Fig. 1a shows the normal mammogram case, in which there are no suspect mass regions. Fig. 1b shows the abnormal mammogram case which is known as benign. In this case, the nerves are starting to damage. Fig. 1c shows the abnormal mammogram case which is known as malignant. In this case, the nerves are starting to damage very fast when compared with benign. Breast cancer diagnosis includes correctly identifying each stage and classifying detected breast tumors and abnormalities into proper categories. To assist radiologists and oncologists in diagnosing breast cancer in a fast and reliable manner, many Computer-Aided Diagnosis (CAD) systems have been developed over the last two decades [2]. Unfortunately, earlier CAD systems did not produce significant improvements in day-to-day breast cancer diagnosis in clinical use [3,4]. After the ‘boom’ of deep learning (DL), DL-based CAD systems and other computer vision and object recognition methods brought success to many areas of medicine, from day-to-day healthcare practices to comprehensive medical applications [5–9]. Currently, many DL-based CAD systems can be used to assist radiologists in breast cancer screening, monitoring, and diagnosis.

Many researchers used machine learning methods to detect breast cancer in mammogram images over the past two decades. The classification rate was low using the machine learning method and required a large number of dataset images for training the classifier. These conventional methods for breast cancer detection and segmentation consumed more computational time for classifying mammogram images into different classes. Also, these conventional methods segmented the cancer pixels in mammogram images more accurately in high-resolution and high-standard mammogram images. Therefore, these conventional methods are not supported by all healthcare centers. These drawbacks are overcome by developing deep learning methods to classify and segment the cancer regions in mammogram images. At present, many deep learning methods are developed to detect and classify breast cancer into different class patterns. The conventional deep learning methods required more mammogram images in the training category, which also consumed more time. In the view of physicians or radiologists, cancer mammogram image detection time is critical in the case of large population countries with low economical costs. This radiologist’s point of drawbacks will be resolved in this paper by proposing an ANFIS classifier that consumed less computational time and this classifier classifies the mammogram images with low-resolution mode. In this paper, Gabor transform-based deep learning architecture is proposed to detect and segment the cancer regions in mammogram images.

The motivation of this research work is to develop and assist the physician to identify the exact internal and external boundaries of the cancer regions in mammogram images. This will expand the life of the women patient if it is detected on time.

Therefore, the proposed breast cancer detection method plays a vital role in detecting the cancer regions in mammogram images. This paper develops a breast cancer detection approach using an ANFIS classifier. This paper is sectioned into five modules. Module 2 elaborates on the proposed breast cancer detection approach, module 3 proposes an ANFIS-based breast cancer detection approach, the experimental metrics are discussed in module 4 and the module concludes this paper.

Ueda et al. (2022) [10] applied deep learning CNN algorithm for detecting the cancer regions in mammogram images. The kernel rate of the internal Convolutional layer of the developed CNN architecture was designed using the fuzzy-based algorithm. This method was cross-validated by various validation algorithms and obtained a 95% of classification rate. Hassan et al. (2022) [11] developed Computer Aided Detection (CAD) approach for detecting cancer regions in mammogram images. The authors analyzed numerous conventional breast cancer detection methods with their experimental results. Saad Awadh Alanazi et al. (2021) constructed a deep-learning structure for detecting malignant mammogram images from non-malignant mammogram images. The data transformation was performed by the internal layers and the loss function was significantly reduced using the optimizer module in Convolutional Neural Networks (CNN). The authors obtained 95% of classification accuracy for the mammogram images available in Kaggle dataset. Desai et al. (2021) [12] employed two different classifiers for differentiating normal mammogram images from cancer-affected mammogram images in this work. The authors integrated the Multi-layer Perceptron Neural Network (MPNN) classification method with the constructed CNN structure for classifying the mammogram images into a different set of classes. The authors obtained 96.2% of classification accuracy for the mammogram images. Muhammad Tariq Sadiq et al. (2021) [13] applied Empirical Decomposition Mode (EDM) for decomposing the Electroencephalogram (EEG) signals along with different classification algorithms. Abdelhafiz et al. (2020) [14] segmented mass regions in mammogram images and these segmented mass regions were classified by the CNN method. Qian Zhang et al. (2020) [15] applied a maximum response filter bank on the mammogram images and further Rotation-Invariant Features (RIF) were computed from the filtered mammogram images. These RIF features were fused to compute the mixed fused feature image and this image was classified by the CNN classification method. The authors applied and tested the developed method on several mammogram datasets and obtained approximately 94% of the mammogram classification rate. Pacilè et al. (2020) [16] used an artificial intelligence technique to detect breast cancer-affected mammogram images. The authors determined the non-linear features from the source mammogram images to locate the distribution of tumor pixels in breast regions in mammogram images. Sasaki et al. (2020) [17] also applied artificial intelligence methods to determine the tumor regions in mammogram images. Sadhukhan et al. (2020) [18] applied different types of machine-learning algorithms for identifying cancer-affected mammogram images.

Chan et al. (2020) [19] used Artificial Intelligence (AI) methods for the detection and segmentation of abnormal mass regions in mammogram images. The abnormal mass regions were segmented by threshold segmentation approach in this work. The simulation results of this method were significantly compared with other approaches. The authors obtained 95.18% Se, 95.74% Sp, and 95.38% SA for benign case mammogram images and also obtained 94.23% Se, 94.28% Sp, and 95.37% SA for malignant case mammogram images on DDSM dataset images. Zeebaree et al. (2019) [20] utilized distance transform for segmenting the Region of Interest (RoI) in mammogram images. The segmented RoI pixels in mammogram images were classified by Neural Network (NN) method to differentiate the healthy and cancer mammogram images. The authors obtained 94.27% Se, 94.92% Sp, and 94.28% SA for benign case mammogram images and also obtained 94.18% Se, 94.67% Sp, and 95.17% SA for malignant case mammogram images on DDSM dataset images. Vijayarajeswari et al. (2019) [21] proposed a breast cancer segmentation approach using a machine learning classification method. This work used Hough Transform (HT) for performing the pixel resolution for the classification process. The transformed coordinates from the HT approach were classified the using Support Vector Machine (SVM) method. The authors obtained 94.76% Se, 94.98% Sp, and 95.75% SA for benign case mammogram images and also obtained 94.74% Se, 94.38% Sp, and 94.95% SA for malignant case mammogram images on DDSM dataset images. Ribli et al. (2018) [22] used a deep-learning model for identifying the lesions in mammogram images. The authors obtained an 89.7% lesion detection rate using their proposed methods. Sadoughi et al. (2018) [23] reviewed the applications of machine learning algorithms for detecting and classifying abnormal mammogram images from normal mammogram images. The authors compared the experimental results of these conventional methods concerning performance metrics.

From the extensive study of the conventional breast cancer detection methods, the following limitations of the conventional studies are observed as stated below.

Most of the existing methods used existing machine and deep learning algorithms for mammogram classifications.

These drawbacks are resolved by the proposed cancer region detection methods stated in this paper.

The above-mentioned research gaps are resolved by the objectives of this paper are stated as follows.

To develop fully computer-assisted methods for breast cancer detection.

The novelties of the work are stated below.

Novel statistical Ridgelet features are computed for the classification of mammogram images.

The mammogram images are classified into normal, benign, and malignant types using the ANFIS approach in this paper. This mammogram classification process consists of a noise filtering module, spatial-frequency transformation module, feature computation module, and classification module. The Gaussian Filtering Algorithm (GFA) is used as the noise filtering method and the Ridgelet transform is used as the spatial-frequency transformation module. The statistical feature metrics are computed from the transformed coefficients and these values are classified by the ANFIS technique in this paper.

Fig. 2a shows the mammogram images training workflow and Fig. 2b shows the mammogram images testing workflow.

The nerves are started from the chest region of the body and end at the nipple of the breast region. Therefore, the intensity level of all pixels in a mammogram image is different from others which creates the complexity in breast cancer detection process. To improve the smoothness of the pixels in a mammogram image, GFA is applied to the source mammogram image, which performs both noise filtering and enhancement of pixels in a mammogram image. It is a 2D-smoothing filter that uses a Convolution function for smoothing the pixels in a mammogram image. A bell-shaped Gaussian kernel is used in the Gaussian filter in this work. The Gaussian kernel of the Gaussian filter with the kernel size of 5 * 5 is given in the following equation.

(1) G(x,y)=12πσ2e−(x2+y2)2σ2where x and y are the pixel coordinates and the standard deviation is represented by σ.

The source mammogram image is convolved with the Gaussian kernel of the Gaussian filter which produces the smoothed mammogram image. Fig. 3a is the source mammogram image and Fig. 3b is the Gaussian-filtered image.

The MATLAB 2020 version has been used as the simulation tool for the detection of normal mammogram images from abnormal mammogram images [15]. In this paper, two publicly available license-free mammogram datasets have been used and the databases are the Mammographic Image Analysis Society (MIAS) [24] and Digital Database for Screening Mammography DDSM [25]. The MIAS dataset was constructed by a UK research group and the mammogram images were collected through the National Breast Screening Programme (NBSP). This dataset contains 322 mammogram images and they are split into 122 healthy mammogram images, 78 benign mammogram images, and 122 malignant mammogram images. All the mammogram images in this dataset are having an image size of 1024 * 1024 pixels in width and height respectively.

The DDSM dataset was constructed by the mammographic image analysis research community at Massachusetts General Hospital. This dataset contains the study of 2500 women and all the images in this dataset have been collected from different age groups. The mammogram images in this dataset are split into 1500 healthy mammogram images, 650 benign mammogram images, and 350 malignant mammogram images. All the mammogram images in this dataset are having an image size of 227 * 227 pixels in width and height respectively.

Table 1 is the MDI analysis on the MIAS dataset, which obtained 98.3% MDI for healthy cases, 96.1% MDI for benign cases, and 97.5% MDI for malignant cases. Therefore, the average MDI for MIAS dataset mammogram images is about 97.3%.

Table 2 is the MDI analysis on the DDSM dataset, which obtained 97.2% MDI for healthy cases, 98.3% MDI for benign cases, and 97.4% MDI for malignant cases. Therefore, the average MDI for DDSM dataset mammogram images is about 97.6%.

In this paper, the k-fold validation method is used to validate the experimental results which are stated in this work. The 4-fold validation method with 4 sets of iterations is used in this work to test the mammogram images. In the case of the MIAS dataset, the first fold obtains 97.3% of MDI, the second fold obtains 97.2% of MDI, the third fold obtains 97.3% of MDI and the fourth fold obtains 97.2% of MDI. Therefore, the average MDI is about 97.3% which is equal to the obtained experimental results stated in Table 1. In the case of the MDI dataset, the first fold obtains 97.6% of MDI, the second fold obtains 97.5% of MDI, the third fold obtains 97.6% of MDI and the fourth fold obtains 97.5% of MDI. Therefore, the average MDI is about 97.6% which is equal to the obtained experimental results stated in Table 2.

In this work, a Gaussian filter is used to detect and remove the noise contents in the source mammogram images. Based on the Gaussian filter implementation, the performance parameters Peak Signal to Noise Ratio (PSNR) is computed concerning different noise contents. The Gaussian filter obtains 56.78% of PSNR for 10% of the noise contents in the source mammogram image and the Gaussian filter obtains 51.92% of PSNR for 10% of noise the contents in the source mammogram image, which is more than enough for noise detection and removal process.

The proposed work also investigated the following metrics.

(11) Sensitivity(Se)=TPTP+FN

(12) Sensitivity(Se)=TNTN+FP

(13) Classification Accuracy (CA)=TP+TNTP+TN+FP+FN

The pixels in the category of cancer and non-cancer correctly are known as TP and TN, and the pixels in the category of cancer and non-cancer incorrectly are known as FP and FN.

Table 3 is the quantitative analysis of malignant mammogram images in the MIAS dataset. This paper obtained 97.17% Se, 98.29% Sp and 98.43% SA for the case of benign images and also obtained 98.19% Se, 99.04% Sp and 98.67% SA for the case of malignant images.

Fig. 8a shows the graphical output for the benign case and Fig. 8b shows the graphical output for the malignant case for MIAS dataset images.

Table 4 is the quantitative analysis of malignant mammogram images in the DDSM dataset. This paper obtained 98.19% Se, 98.44% Sp and 98.65% SA for the case of benign images and also obtained 98.83% Se, 98.65% Sp and 98.53% SA for the case of malignant images.

Fig. 9a shows the graphical output for the benign case and Fig. 9b shows the graphical output for the malignant case for DDSM dataset images.

Table 5 is the quantitative analysis of the proposed breast cancer detection approach on both MIAS and DDSM dataset mammogram images. It is observed that the proposed breast cancer detection approach stated in this work produces significant quantitative results in both public dataset mammogram images.

The proposed PHSA segmentation method plays an important role in the tumor pixels segmentation process on the classified abnormal mammogram images. Hence, this is important to analyze the performance of the tumor pixel segmentation using the proposed PHSA method with the conventional morphological segmentation method. Table 6 shows the performance analysis of the proposed PHSA segmentation method with the conventional morphological segmentation method on the MIAS dataset.

Table 7 shows the performance analysis of the proposed PHSA segmentation method with the conventional morphological segmentation method on the DDSM dataset.

Table 8 is the quantitative comparisons of breast cancer detection approaches on the MIAS dataset with other existing breast cancer detection approaches Saad Awadh Alanazi et al. (2021), Desai et al. (2021) and Abdelhafiz et al. (2020).

Fig. 10a shows the graphical comparisons of breast cancer detection approaches on the MIAS dataset for benign images and Fig. 10b Graphical comparisons of breast cancer detection approaches on the MIAS dataset for malignant images.

Table 9 is the quantitative comparisons of breast cancer detection approaches on the DDSM dataset with other existing breast cancer detection approaches Chan et al. (2020), Zeebaree et al. (2019) and Vijayarajeswari et al. (2019).

Fig. 11a shows the graphical comparisons of breast cancer detection approaches on the MIAS dataset for benign images and Fig. 11b Graphical comparisons of breast cancer detection approaches on the MIAS dataset for malignant images.

This paper proposes an effective breast cancer detection method using Ridgelet transform and ANFIS classifier. This work is experimentally investigated on MIAS and DDSM dataset mammogram images. The proposed work obtained 98.3% MDI for healthy cases, 96.1% MDI for the benign case, and 97.5% MDI for the malignant case. Therefore, the average MDI for MIAS dataset mammogram images is about 97.3%. The proposed work obtained 97.2% MDI for healthy cases, 98.3% MDI for the benign case, and 97.4% MDI for the malignant case. Therefore, the average MDI for DDSM dataset mammogram images is about 97.6%. This paper obtained 97.17% Se, 98.29% Sp and 98.43% SA for the case of benign images and also obtained 98.19% Se, 99.04% Sp and 98.67% SA for the case of malignant images in the MIAS dataset. This paper obtained 98.19% Se, 98.44% Sp and 98.65% SA for the case of benign images and also obtained 98.83% Se, 98.65% Sp and 98.53% SA for the case of malignant images in the DDSM dataset. The main limitation of this work is that it detects the tumor regions in mammogram images and it does not deal with any diagnosis or severity analysis on the segmented tumor regions. In the future, the methodologies stated in this work are also used to detect the severity levels of the segmented tumor regions in mammogram images.