Classical segmentation methods in breast cancer detection usually rely on well-established image processing techniques to extract regions of interest (ROI) from mammograms [33]. These methods typically focus on the separation of suspicious masses or micro-calcifications from normal breast tissue [17]. Several classical approaches, such as threshold-based, region-based, and clustering-based segmentation, can be shown in Table 1.

One of the earliest and most widely used traditional segmentation techniques is Otsu’s method, which determines an optimal threshold by maximizing the variance between the foreground and background intensities. In mammogram segmentation, Otsu serves as the core framework for separating masses from surrounding tissue by identifying intensity cutoffs that best distinguish suspicious regions. Mamindla and Ramadevi [39] extended this principle through a multilevel version of Otsu, where instead of a single binary threshold, multiple thresholds were selected to partition the image into several intensity classes. This refinement allowed the algorithm to capture subtle differences in tissue density, making the segmentation more effective, although image noise still reduced accuracy. Similarly, Santhos et al. [43] reinforced the role of Otsu by combining it with an entropy-based criterion to guide threshold selection in cases of diverse intensity distributions. Here, Otsu’s variance-based separation remained the foundation, while the additional measure helped ensure that thresholds preserved more diagnostic information from the image histogram. Across these studies, Otsu’s method consistently demonstrated its value as a simple, efficient, and interpretable baseline for mammographic image segmentation, even as its direct application was limited by sensitivity to noise and contrast variability.

Region and clustering-based techniques have also been widely explored in mammogram segmentation. In 2023, Ref. [35] employed the Ant System-based Contour Clustering (ASCC) algorithm to segment the MIAS mammogram images. This method leveraged the combination of contour-based detection and clustering principles, where marker and walker ants collaboratively identified and refined image contours, enhancing segmentation accuracy by focusing on relevant pixel groups and minimizing processing time. Another recent approach was introduced by [45], who developed an adaptive thresholding with Region Growing Model and applied it to the DDSM dataset (flowchart in Fig. 3). Their workflow began with preprocessing to normalize intensity variations, followed by adaptive thresholding to produce candidate regions. These regions were then refined through an iterative region-growing step, where neighboring pixels were evaluated for similarity to the seed region. This pipeline improved the false discovery rate (FDR) from 24.4% to 19.8%. However, the authors noted that the complexity of mass segmentation remained a challenge, especially in cases involving highly irregular tumor shapes, indicating that while adaptive models enhance accuracy, they do not fully overcome variability in tumor morphology.

The most recent advancement in traditional segmentation techniques is presented by [36], who developed a method using Gradient-based watershed segmentation applied to a dataset of 760 contrast-enhanced spectral mammography (CESM) images. Their pipeline first enhanced the CESM images using preprocessing techniques to improve contrast, after which the gradient magnitude of the image was computed to highlight intensity transitions. Based on these gradients, the watershed algorithm was applied, but with an improved marker-based strategy to avoid over-segmentation. Specifically, morphological operations were used to generate reliable markers, guiding the watershed process to separate tumor boundaries more effectively. Finally, the segmented tumor regions were passed to a DualNet classifier to validate the segmentation and improve diagnostic accuracy. This integrated workflow reduced false positives and preserved boundary precision, resulting in promising performance for breast cancer segmentation (accuracy: 0.938, sensitivity: 0.94, and specificity: 0.96).

The comparative findings in Table 1 and the recent extensions of classical methods [35,36,38,41,43,47] reveal a set of consistent trade-offs. Thresholding-based approaches such as Otsu’s method remain attractive due to their simplicity and low computational burden, but they are inherently sensitive to noise and intensity variations, which limits their generalizability in heterogeneous clinical datasets. Metaheuristic extensions with multilevel thresholding improve stability but introduce additional complexity and parameter tuning, reducing their clinical practicality. Region-based and clustering methods, such as ASCC and adaptive thresholding with region growing, address local intensity variations more effectively, yet they often require carefully chosen seeds and are prone to false positives when tumor boundaries are irregular. Meanwhile, watershed segmentation, especially with marker control, can delineate complex tumor shapes more precisely, but it still depends heavily on preprocessing quality and is vulnerable to over-segmentation. Collectively, these classical methods highlight the trade-off between simplicity and robustness, simpler methods offer transparency and efficiency but struggle with variability, while more adaptive variants achieve higher accuracy at the cost of complexity and reduce ease of integration into clinical workflows. This explains why classical approaches, despite producing competitive performance on controlled datasets, are rarely adopted in practice without hybridization or post-processing support.

Machine learning (ML)-based segmentation techniques have gained prominence in breast cancer detection due to their ability to learn complex patterns and improve segmentation accuracy [35]; some references and their overviews can be seen in Table 2.

Fuzzy C-means (FCM) has been a widely used method in ML-based segmentation for mammograms; an example of the scheme can be seen in Fig. 4. The process begins with preprocessing steps including median filtering and CLAHE, which reduce noise and enhance local contrast. A bounding box is then applied to isolate the region of interest. The refined image is subsequently segmented using FCM clustering, where pixels are assigned membership values to multiple classes rather than being strictly classified. The final segmented image highlights the suspected tumor region, enabling further diagnostic analysis. Krishnakumar and Kousalya [48] applied FCM to both the DDSM and MIAS datasets. In their model, FCM was advanced with the Distorted Contour (DC) method to improve handling inappropriate borders and intensity variations during segmentation. It starts with FCM clustering to segment mammogram images by assigning membership values to each pixel based on intensity. This is followed by iterative updates of cluster centers until convergence. The addition of the Distorted Contour (DC) method refines the segmentation by correcting irregular borders and compensating for intensity variations, resulting in a more accurate and reliable delineation of tumor boundaries. To improve segmentation efficiency, Ref. [50] proposed a Modified Adaptively Regularized Kernel-based Fuzzy C-Means method tested on the INBreast database. This approach first maps pixel intensities into a higher-dimensional kernel space, enabling better separation of complex tumor boundaries. Adaptive regularization is then applied to stabilize the clustering and reduce sensitivity to noise. In the second stage, statistical texture features are extracted from the segmented regions and classified with a Support Vector Machine (SVM), which further enhances diagnostic reliability. While the method achieved a high ROC AUC of 0.961, its reliance on kernel transformations and additional classification steps increased computational cost, sometimes producing block-like segmentations that required post-processing refinement.

In the other hand, Ref. [14] applied a modified K-means clustering algorithm to the MIAS and CBIS-DDSM datasets. In this study, K-means worked by partitioning the image pixels into clusters based on similarities in intensity, texture, and shape. This iterative process involved recalculating cluster centers and reassigning pixel memberships to enhance segmentation accuracy. Their method achieved more than 90% of accuracy and sensitivity for each dataset. Despite these successes, the variability in performance across different lesion types raised concerns about the method’s robustness and reliability when dealing with a variety of tumor shapes and sizes. In another study, Rezaee et al. [57] implemented a K-means clustering algorithm as a primary step in segmenting mammogram images before classification. After removing redundant parts and isolating the pectoral region, the remaining breast area was clustered into regions using K-means based on pixel intensity and spatial properties. The authors experimented with cluster counts ranging from 3 to 6, identifying that 4–5 clusters yielded optimal segmentation for delineating mass regions. The K-means algorithm minimized intra-cluster distance using Euclidean distance, iteratively updating cluster centers to better separate regions of interest from surrounding tissues. This segmentation step helped isolate suspected masses, which were then processed further using texture descriptors (e.g., GLCM), Pseudo-Zernike moments, and wavelet features for classification. While effective, the method showed varying accuracy depending on mass type and shape.

Overall, ML-based methods such as FCM and K-means have advanced segmentation by improving flexibility in handling noisy or ambiguous regions and by offering computational simplicity, respectively. FCM provides adaptability through partial membership assignments, which helps preserve boundary details in complex tissues, but its high computational burden, depends on parameter tuning [49], and sensitive to initialization limit scalability for large or heterogeneous datasets [53]. K-means, in contrast, remains efficient and easy to implement, yet its reliance on a predefined number of clusters and limited ability to manage irregular tumor morphologies reduce its robustness across varied lesion types [14]. The trade-off between flexibility and efficiency is therefore evident, which is FCM achieves finer boundary delineation at the cost of computational expense, while K-means offers speed and simplicity but sacrifices accuracy in complex cases.

Deep Learning (DL) has revolutionized segmentation tasks in breast cancer detection by leveraging neural networks to automatically learn hierarchical representations from mammogram images [59]. DL-based segmentation methods, like convolutional neural networks (CNNs) with variations of U-Net architectures, are widely utilized due to their ability to effectively capture spatial and contextual information, making them well-suited for various segmentation tasks [60], other methods and their performance are presented in Table 3.

Li et al. [66] developed an attention-dense U-Net for automatic breast mass segmentation using images from the DDSM dataset. The model integrates dense connections within the encoder–decoder framework to enhance feature propagation and capture fine-grained details, while attention gates filter irrelevant information from skip connections, allowing the network to focus on tumor regions more effectively. This combination improves boundary precision and robustness against noise compared to the original U-Net. Fig. 5 illustrates the workflow, where data are augmented and passed through the encoder–decoder network to predict segmentation masks. While the Attention Dense U-Net achieved superior performance over baseline U-Net and state-of-the-art models, the authors acknowledged a risk of overfitting due to the limited dataset size, which could restrict its ability to generalize to broader clinical applications. Moreover, Zeiser et al. [60] applied U-Net for mass segmentation using 2500 cases which the model was enhanced by applying data augmentation techniques, including horizontal mirroring and zooming, to address the limited size of the DDSM dataset and reduce overfitting. The model’s architecture featured down-sampling through convolutional layers with ReLU activation and max pooling to extract features, followed by up-sampling using deconvolution layers to reconstruct the segmented image. The final layer used a 1 × 1 convolution to map outputs to the desired classes, enabling precise mass delineation.

Another DL-based method is YOLO, which is commonly used for object detection and has recently been adapted for breast cancer segmentation. Su et al. [62] introduced the YOLO-LOGO segmentation model, which combines the fast ROI detection of YOLO with a Local-Global (LOGO) transformer for precise mass delineation. YOLO first localizes suspicious regions, and the LOGO module then applies dual attention mechanisms: local attention to capture fine tumor boundaries and global attention to preserve contextual breast tissue information. Tested on the CBIS-DDSM and INbreast datasets, the model achieved a high True Positive Rate (95.7%), demonstrating its robustness in complex mammogram backgrounds. However, dual-stage architecture substantially increased computational requirements, limiting scalability for deployment in resource-constrained clinical environments. Similarly, Coskun et al. [61] applied the YOLOv5 model for mass detection using the INbreast dataset. To enhance performance, they integrated a Swin Transformer into the YOLOv5 framework, where YOLOv5 performed fast bounding-box detection of candidate masses while the Swin Transformer enriched feature representations through its window-based self-attention mechanism. This model improved segmentation performance compared to earlier YOLO versions by capturing both local and global contextual information, particularly for irregularly shaped or small tumors. Nevertheless, the added complexity significantly increased GPU memory demands and computational cost, posing challenges for scaling larger datasets.

We can conclude that DL-based segmentation methods have substantially advanced breast cancer analysis by combining hierarchical feature extraction with attention mechanisms. U-Net variants, such as the Attention Dense U-Net [66], and data-augmented U-Net [60], demonstrate strong capabilities in capturing fine structural details and improving boundary definition, yet they remain highly dependent on dataset size and risk overfitting, and may also suffer from prolonged execution times on low-spec systems, which poses practical limitations for clinical deployment. The trade-off here lies between architectural complexity and practical usability: dense connections and attention modules increase accuracy and robustness, but they also heighten computational cost and require large, diverse datasets to generalize effectively. On the other hand, YOLO-based approaches [62], Coskun et al. [61] provide real-time detection, can achieve high sensitivity even in challenging cases with irregular or small tumors. However, these models demand considerable GPU memory and computational resources, making large-scale clinical deployment difficult. Thus, while deep learning approaches currently represent the state of the art in mammogram segmentation, their clinical translation will depend on striking a balance between accuracy, computational efficiency, and robustness across heterogeneous datasets because of overfitting [18,72].

Hybrid/ensemble segmentation methods have emerged as a powerful approach to breast cancer detection, combining the strengths of multiple segmentation techniques to overcome the limitations of individual methods. These hybrid/ensemble models often integrate traditional methods with machine learning (ML) or deep learning (DL) techniques, resulting in enhanced segmentation accuracy and robustness in detecting breast masses and abnormalities, the preview of methods can be seen in Table 4.

Ghuge and Saravanan [26] introduced the SRMADNet, a hybrid method combining the strengths of Swin ResUnet3+ for segmentation and Adaptive Multi-scale Attention-based DenseNet with Extreme Learning Machine (AMAD-ELM) for classification. This hybrid model combines the Swin Transformer with the Unet3+ framework, leveraging the transformer’s capacity for global contextual awareness and Unet3+’s detailed feature mapping through skip connections and multi-scale feature integration. The Swin Transformer acts as an encoder, capturing complex image representations, while Unet3+ refines these features through its advanced decoder structure, ensuring high-resolution segmentation. This combination enhances boundary delineation and segmentation precision, which is crucial for effectively identifying abnormal regions in mammograms. The model was tested on a public dataset and demonstrated a performance improvement of 13.7% over CNN and 14.2% over DenseNet. Furthermore, Jha et al. [25] proposed an ensemble learning-based hybrid segmentation approach utilizing a combination of Fuzzy C-Means (FCM) and Convolutional Neural Networks (CNNs). The FCM is used for initial segmentation, which clusters the image data into regions of interest based on pixel intensities, effectively handling the variability in image intensity. The CNN model then refines the segmentation by learning from these clustered outputs, enhancing the accuracy and precision of the segmented regions. This combined method allows for improved delineation of tumor boundaries and better performance in handling complex mammographic features. The model achieved an accuracy of 98.59% (2-scale segmentation), which further improved to 99% (5-scale segmentation).

One of the most recent innovations in hybrid segmentation incorporates gaze tracking data to enhance model supervision and mimic radiologist attention. Xie et al. [27] proposed a Grouped Fusion Network that integrates eye-tracking information with deep learning-based segmentation to improve performance, particularly in detecting challenging regions such as small masses and microcalcifications. The architecture consists of two segmentation streams using Attention U-Net variants, each processing different spatial scales of the mammogram. A gaze attention map, derived from radiologist eye-tracking data collected during screening, is used to guide the secondary segmentation stream toward previously under-attended regions. This multi-stream fusion mechanism ensures more comprehensive detection by simulating expert diagnostic behavior, the method development can be seen in Fig. 6.

Overall, hybrid segmentation methods represent an important bridge between traditional, machine learning, and deep learning approaches by integrating complementary strengths within a single pipeline. Architectures such as ET-SegNet [24] and SRMADNet [26] highlight how edge-aware modules and transformer-based encoders can enhance boundary precision and contextual understanding, delivering more accurate delineation of tumors compared to single-paradigm models, though they do so at the cost of increased architectural complexity and higher training demands. Ensemble approaches, like the combination of FCM and CNN [25], demonstrate the effectiveness of cascading classical clustering with deep refinement, producing very high accuracy and robustness across heterogeneous data, even if they remain sensitive to intensity variability and require careful parameter tuning. Optimization-based hybrids such as K-Means++ with Cuckoo Search [76] further emphasize stability and adaptability, successfully addressing intensity inhomogeneity and complex tissue structures, yet their reliance on iterative metaheuristics raises concerns about scalability when applied to large clinical datasets. Meanwhile, novel paradigms such as gaze-driven fusion networks [27] provide a unique advantage by explicitly transferring radiologists’ visual attention into the model, improving sensitivity to subtle lesions such as micro-calcifications and reducing false negatives in challenging cases. This strategy shows strong potential for bridging human diagnostic expertise with automated systems. However, the need for eye-tracking data during training requires specialized equipment and expert involvement, which may limit reproducibility, while the multi-stream architecture increases computational overhead. Taking together, these methods illustrate a clear trade-off: hybrids achieve superior segmentation and improved clinical relevance by combining multiple paradigms, but their added complexity, higher computational cost, and reliance on enriched training data still present hurdles for routine deployment.

In this subsection, we highlight several cutting-edge techniques drawn from the four segmentation approaches: Attention/Transformer U-Net, detector-led cascades (YOLO→U-Net), FCM→CNN hybrid refinement, and gaze-guided attention. These techniques were selected for their complementary innovations and clinical relevance under typical data and compute constraints. 1.

Attention/Transformer U-Net

U-Net variants enhanced with channel/spatial attention or lightweight transformer blocks strengthen the flow between global context and fine local detail. In mammography, this often tightens boundaries around small or low-contrast masses when pixel-level annotations and sufficient compute are available. For example, Attention U-Net [26,66] has been reported to improve delineation, at the cost of additional parameters and training time. 2.

Detector-led segmentation (e.g., YOLO-based pipelines)

In detector-led segmentation, a detector first proposes suspicious regions on the full mammogram, and then a dedicated segmenter (or a detection-to-mask head) refines the lesion outline within each crop. This two-stage pattern reduces full-image compute and can lower false negatives for sparse or small lesions, which is attractive in screening settings with many negatives [61,62]. But their performance depends on annotated datasets and high computational resources. 3.

FCM-based hybrid refinement (e.g., FCM + Otsu; FCM + CNN)

A fuzzy clustering step supplies a coarse mask or saliency prior that is then refined, either by classical post-processing or by a CNN. The appeal is combining FCM’s robustness on ambiguous tissue with learned features for sharper edges. For instance, hybrids such as Otsu/entropy with FCM [73], and FCM + CNN ensembles [25] are especially useful on weak label dataset, but they add steps and hyper-parameters to tune. 4.

Gaze -guided attention

Radiologists’ eye-tracking traces can be injected as soft priors to steer network attention toward clinically meaningful regions, improving sensitivity to subtle findings without heavy pixel-wise supervision. As noted in our corpus (e.g., gaze-driven fusion [27]), gains depend on the availability and integration quality of gaze data; when present, this cue can complement both DL and hybrid pipelines.

A concise summary of the Section 3.1.5 spotlights is provided in Table 5. For each technique we list the core innovation, practical applicability, and key trade-offs.