The dataset in this article contains dermoscopic images collected from publicly available databases HAM10000 dataset [22]. Given the clinical need of distinguishing malignant from benign lesions, a binary classification approach was applied, labelling lesions as either malignant (melanoma) or benign (non-melanoma). A grid of illustrative sample images from the dataset is shown in Fig. 1 which illustrates the heterogeneity in lesion properties, through various size, shape, color and texture.

A significant issue in automatic skin cancer detection is the large class imbalance in dermatological datasets, where cases of malignancy have a strong underrepresentation. This imbalance tends to bias model predictions to the majority class and manifests in low sensitivity and low accuracy in detecting the minority class, which is the clinically important melanoma. To address this problem, we used synthetic data augmentation based on several variations of generative adversarial networks (GANs, with particular emphasis on the minority melanoma class. This augmentation approach is designed to increase the model’s generalizability by letting it see a wider variety of lesion appearances as well as overcoming the lack of data.

In particular, this study used a variation of Deep Convolutional GAN (DCGAN), a standard DCGAN [23], and StyleGAN3 [24]. Further synthetic examples were generated based on these models by training them on melanoma class-only samples. Fig. 2 shows synthetic melanoma images generated by each GAN variant: (a) Standard DCGAN, (b) Modified DCGAN, and (c) StyleGAN3. Table 2 presents the various parameters that were utilized for generating synthetic samples with the GANs. Each GAN Model was evaluated in terms of their capacity to produce high-quality, clinically realistic dermoscopic images, and numerical image quality and diversity measures. Before augmentation, the number of class 1 (cancerous skin lesion) and class 2 (normal skin) observations was 1954 and 8061, respectively, showing a considerable class imbalance. By GAN augmentation, the number of cancerous skin images became 6954, and that of normal skin images became 8061. This process led to a more equitable data set and permitted the model to expose itself to a larger and wider variety of manifestations of melanoma.

Modified DCGAN: Tailored for 512 × 512 image resolution, this variant replaces conventional transposed convolution layers with upsampling followed by convolutional blocks, effectively mitigating checkerboard artifacts. The generator begins with fully connected linear layers that project the latent vector (Z) into a high-dimensional feature space, facilitating smoother transitions in the latent representation. Although this architecture improves on the standard DCGAN, it still falls short of StyleGAN3 in generating the fine-grained dermoscopic details essential for clinical realism.

Overall, the GAN-augmented samples were integrated with the original dataset to form a balanced training set, which was subsequently normalized and resized to a fixed resolution of 512 × 512 prior to input into the hybrid model. This preprocessing pipeline not only addresses class imbalance but also enhances the visual diversity of training samples, which is essential for robust melanoma detection.

To quantitatively assess the quality and diversity of synthetic melanoma images, we utilized three standard metrics: Fréchet Inception Distance (FID), Kernel Inception Distance (KID), and Inception Score (IS). These metrics compare the distribution of synthetic images to real images and are widely used to evaluate GAN performance. The GAN objective is defined as: (1)minGmaxDV(D,G)=Ex∼pdata[log⁡D(x)]+Ez∼pz[log⁡(1−D(G(z)))]where D is the discriminator, G is the generator, and z is the noise vector.

Fréchet Inception Distance (FID): FID measures the similarity between the feature distributions of real and generated images using activations from a pre-trained Inception-v3 network. It assumes that these features follow a multivariate Gaussian distribution. (2)FID=‖μr−μg‖2+Tr(Σr+Σg−2(ΣrΣg)12)where μr,Σr and μg,Σg are the means and covariances of real and generated image features, respectively.

Lower FID values indicate better quality and closer resemblance to real images.

Higher IS values indicate that the model generates diverse images that are confidently classifiable.

Lower FID and KID values correspond to higher similarity to real images, whereas a higher IS indicates greater diversity and image quality. The comparative performance of these GANs is summarized in Table 3.

The visual comparison, corroborated by the quantitative metrics in Table 3, highlights StyleGAN3’s ability to produce lesions with finer details, greater realism, and greater diversity. The FID scores, in particular, show a marked improvement with StyleGAN3 (see Fig. 3).

Following a comprehensive comparative analysis that involved qualitative (visual inspection) and quantitative (FID, KID, and Inception Score) evaluations, StyleGAN3-generated images were selected as the most effective due to their superior fidelity, enhanced diversity, and morphological consistency with real dermoscopic patterns. The architectural innovations of StyleGAN3, particularly its alias-free design and ability to disentangle style representations, substantially contribute to its empirical performance. Furthermore, the integration of these high-quality synthetic images with conventional augmentation techniques (e.g., rotation, flipping, contrast enhancement, and Gaussian noise) significantly enriched the training dataset, thereby enhancing the model’s capacity to generalize, particularly to rare or atypical cases of melanoma.

To ensure consistency in image representation, all images were resized to a 224×224 resolution before being fed into the model. Normalization was applied by scaling the intensity values of the pixels in the [0,1] range, stabilizing the gradient updates, and preventing problems related to varying intensity distributions. In addition, noise reduction techniques such as median filtering were employed to suppress artifacts, including hair occlusions and uneven lighting conditions. This preprocessing pipeline ensures that the model receives high-quality input data, facilitating more reliable feature extraction.

CNNs are basically the baseline of our structure, which extract low-level and high-level spatial features from the input images [27]. The convolution operation at layer l is defined as: (5)Fijl=∑m=0P−1∑q=0Q−1Wpqml⋅Fi+p,j+q,m−1+blwhere Fijl is the output feature map, σ is the ReLU activation, W represents the kernel weights and bl is the bias term. The convolution layers extract features such as colors, borders of the lesions, and texture, that is useful characteristics to distinguish melanoma from benign lesions. These feature maps are then fed to later modules for further computation.

Transformers have emerged as the state-of-the-art in vision tasks due to their self-attention mechanism that permits them to focus on the most important areas of an image [28]. In the proposed model, we employ the Transformer architecture to capture long-range dependencies and help understand the context of skin lesions. In contrast to CNNs that have local receptive fields, Transformers consider the entire image as a whole and thus have shown particularly good performance in lesion shape and color-variance-based diagnosis applications. The multi-head attention mechanism is computed as: (6)Attention(Q,K,V)=softmax(QKTdk)Vwhere Q, K, and V are the query, key, and value matrices, and dk is the dimension of keys.

While GRUs are typically used for sequential data, we have used them for features generated by CNN and Transformer. GRUs assist in learning temporal dependencies of feature representations and are thus helpful for the model to maintain useful context from various levels of abstractions [29]. Such sequential processing contributes to the extraction of the subtle variation of lesion structures across different spatial areas, yielding an improved classification performance. The GRU updates its hidden state ht as: (7)zt=σ(Wz⋅[ht−1,xt])    (Update gate) (8)rt=σ(Wr⋅[ht−1,xt])    (Reset gate) (9)h~t=tanh⁡(W⋅[rt⊙ht−1,xt]) (10)ht=(1−zt)⊙ht−1+zt⊙h~t

A comprehensive comparison of all the discussed models in terms of accuracy, recall, precision, F1 Score, and AUC is shown in Table 4. Overall, this evaluation provides a wider view for each model’s performance in skin cancer detection than the one gained through a single-metric analysis. The metrics have their focuses: the accuracy evaluates the percentage of correct predictions that the model has made overall (Spanning from 87.79% for VGG16 to 90.61% for the proposed model), the recall evaluates the proportion of actual positive cases that got classified as such (From 81.59% for EfficientNet to 90.88% for the proposed CNN with Parallel Self-attention Transformer and GRU), the precision reflects the reliability of the positive predictions (The proposed model with the highest of 91.12%) and the F1-score is simply the average value of precision and recall (The proposed model with the highest maximum of 91.00%). The AUC measure takes the class ability to be separated from the rest at various thresholds considered (0.8781 for VGG16 and 0.9680 for the proposed model). Such a balanced and multimeric approach is highly desirable in medical diagnostics to minimize false positive and false negative errors. The proposed model is composed of CNN Parallel Self-Attention Transformer and GRU, and it gets the best scores among all the metrics. Robust generalization and reliability are indicated by its high precision and recall, making it suitable for clinical use. Moreover, the high AUC reveals its potential to differentiate malignant and benign skin lesions at different thresholds. The proposed model (CNN with SA Transformer + GRU) was compared with the baselines using the paired t-test. Results demonstrate that all the models are significant (i.e., having p-values < 0.005) for the classification accuracy, proving the statistical superiority of the proposed model.

The ROC curves for the proposed model and other baseline models are presented in Fig. 6. We find that the AUC score of the proposed CNN with Parallel (SA Transformer + GRU) is much higher, which means it has a better class discrimination performance. This is crucial for skin cancer detection, where avoiding false negatives (missing a cancer) is vital. The ROC curve of the proposed model consistently achieves superiority over those of other conventional models like VGG16, MobileNet, and ResNet + BiLSTM (Bidirectional Long Short-Term Memory Network). It has a high true positive rate even with a low false positive rate, indicating its generality with different thresholds. This generalization to the models contributes to the robustness of the model when applied in actual clinical practice, where misclassifications would lead to life-changing decisions.

The precision-recall curve is another important evaluation metric, especially in cases where the data set is imbalanced, which is common in medical diagnostics. Fig. 7 presents the precision-recall comparison graph for the proposed model without data augmentation, achieving a precision-recall score of 0.89. Fig. 8 shows the comparison for the proposed model with the GAN-based data augmentation, achieving a significant improvement with a precision-recall score of 0.96. The precision-recall curves demonstrate the trade-off between precision (the proportion of true positive predictions out of all positive predictions) and recall (the proportion of true positive predictions out of all actual positive cases). With data augmentation, the model maintains high precision while achieving significantly better recall. This improvement is crucial in minimizing false positives and false negatives, ensuring that the model provides accurate diagnoses without missing any potentially malignant skin lesions. The high precision and recall values with augmentation indicate that the proposed model is well-suited for early and reliable skin cancer detection.

To further quantify the effectiveness of GAN-based data augmentation, we compared the performance of the proposed model on the same dataset with and without augmentation. Table 5 summarizes the key metrics for both scenarios. Without data augmentation, the model achieved an accuracy of 81.88%, AUC of 0.8320, precision of 0.796, and recall of 0.819 (see Fig. 7). In contrast, with GAN-based augmentation, the model’s performance improved substantially, achieving 90.61% accuracy, 0.9680 AUC, 91.12% precision, and 90.88% recall. This demonstrates that data augmentation not only enhances the model’s ability to generalize to unseen data but also significantly boosts its sensitivity and overall diagnostic reliability. These findings are consistent with previous research, showing that data increase is a critical strategy for improving model robustness, especially in medical imaging tasks with limited or imbalanced datasets [1,3,4].

These experiments suggest that the application of GAN-based data augmentation is very useful for improving the classification performance of the proposed model with imbalanced medical data. To gain more insight into the classification of each model, normalized confusion matrices are shown in Fig. 9, where the information of true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN) is represented. This analysis provides insight into the strengths and weaknesses of each model in the accurate detection of malignant lesions. From the confusion matrix for the above models, the proposed model showed the best results with minimal false positives and false negatives. This is further evidence of its applicability to high-risk diagnostic applications, such as skin cancer diagnosis. Others, such as EfficientNet and DenseNet + XGBoost, also exhibit a good result with the tradeoff between sensitivity and specificity being relatively lower.

This research introduces a novel deep learning architecture that combines CNNs with parallel SA transformers and GRUs for enhanced feature extraction and sequence modeling in skin cancer detection. This architecture represents a significant departure from conventional pre-trained networks and is expected to demonstrate superior performance in capturing complex features and patterns associated with skin lesions. The research utilizes GAN-based augmentation to address the data imbalance problem prevalent in skin cancer datasets, specifically by generating synthetic melanoma images. This approach can improve the robustness of the model, improve its ability to generalize from limited data, and mitigate the risk of overfitting, leading to more reliable and accurate predictions. Early and accurate detection of skin cancer, particularly melanoma, is crucial for effective treatment and improved patient outcomes [31]. The proposed deep learning system aims to provide clinicians with a more objective and reliable diagnostic tool, potentially reducing the subjectivity inherent in visual evaluations of skin lesions [32]. The performance of the model was rigorously evaluated using a five-fold cross-validation. This comprehensive evaluation strategy ensures the stability and reliability of the results, providing insight into the consistency and generalizability of the model in different data splitting scenarios [14]. The evaluation further includes comparative analysis with individual model branches (CNN-only, CNN+GRU, etc.), demonstrating the advantage of our hybrid configuration. Visual explanation techniques like Grad-CAM and metric-based comparisons validate the model’s focus and decision reliability. The observed performance gains confirm the effectiveness of combining spatial, sequential, and global features for medical image classification.

This study introduces a novel hybrid architecture, which is the combination of CNN, parallel SA Transformer, and GRU, to extract complex spatial and sequential patterns in skin lesions [33–35]. CNN extracts hierarchical visual features, Transformer emphasizes the salient regions of an image, and GRU captures the sequential latent dependencies. This parallel processing results in a richer understanding of the lesion characteristics over simple or standalone CNN-based models. The use of GAN-based augmentation further improves the model to deal with class imbalance, yields a variety of synthetic melanoma images that enhance generalization and reduce overfitting [35]. In comparison with typical ML(Machine Learning) and pre-trained CNN, like VGG16 and AlexNet, which have fixed feature extractors, and lower results, the proposed model has better results in key evaluation parameters such as accuracy, precision, recall, f1 score, and AUC [35]. The proposed model achieves 90.61% accuracy, while VGG16, AlexNet, and standard CNN accuracy were 87.79%, 89.28% and 88.95%. It also performs better than a CNN+Transformer+GRU model (89.85%). While transfer learning approaches with AlexNet, ResNet, VGG are popular [36–39], they typically need pre-trained generic features. Ensemble models also might achieve better performance, but they complicate the system without guaranteed improvement. In contrast, our proposed model learns task-specialized features with its distinctive architecture and also exploits GAN-based augmentation. While the findings are encouraging, more testing is required in multiple datasets and clinical scenarios to determine the generalizability and robustness of the results [16,34,35].

To contextualize the performance of our proposed model, we compared its results with some recent state-of-the-art (SOTA) deep learning architectures for skin cancer detection, as reported in the literature. Table 6 summarizes the key metrics of our model alongside leading transformer-based and hybrid models such as Swin Transformer and ConvNeXt.

As shown, our model achieves highly competitive results, particularly in F1-score and recall, which are crucial for medical diagnostics. While ConvNeXt achieves slightly higher accuracy, our approach demonstrates a stronger balance of sensitivity and precision, supported by robust generalization due to GAN-based augmentation. Notably, our model also incorporates explainability through Grad-CAM (Gradient-weighted Class Activation Mapping) and addresses class imbalance, features not always present in SOTA benchmarks. Recent studies such as the enhanced Faster R-CNN optimized by artificial gorilla troops algorithm [42] and xCViT [43] have further advanced the field by introducing novel optimization and fusion strategies. These works highlight the ongoing evolution of deep learning methods for skin cancer detection and underscore the importance of benchmarking hybrid and explainable models. Future work will focus on direct benchmarking against these and other emerging SOTA approaches, as well as integrating multi-modal and self-supervised learning strategies.

The proposed CNN with parallel SA Transformer and GRU architecture presents a novel and effective solution for skin cancer detection by integrating spatial, global, and sequential feature extraction mechanisms [15,33,35,44]. CNN captures local visual features, while SA Transformer models global dependencies, and GRU captures sequential patterns, resulting in a rich and holistic representation of skin lesions. To mitigate data imbalance, common in medical imaging, GAN-based augmentation is used to generate synthetic samples, enhancing the diversity of the data set and improving model generalization [3,12]. This multicomponent architecture increases robustness by exposing the model to a wide range of lesion variations during training, which is essential for real-world diagnostic accuracy [33,34]. The model achieves superior performance in accuracy, precision, recall, and AUC compared to baseline models such as EfficientNet, MobileNet, VGG16, AlexNet, and DenseNet with XGBoost, as confirmed by cross-validation [12,45]. The use of dermoscopic images further improves clinical relevance by highlighting fine skin structures, supporting its utility to assist dermatologists with accurate and timely diagnoses [3,12,14]. Despite its strengths, the model has limitations. The dependency on GAN-generated data may not fully reflect the diversity of real-world lesion conditions, introducing biases induced by the data set [4,15,33,35]. This model has only been evaluated on public datasets and has not yet been validated with real-world clinical data, which may affect its generalizability in actual clinical environments. Furthermore, the computational demands of the architecture could limit its deployment in low-resource environments. Furthermore, the interpretability of the model remains limited, making its decision-making process less transparent for clinical use [35,44]. Although StyleGAN3-based synthetic data augmentation improves class balance, synthetic images may not fully capture the diversity of real-world skin lesions, potentially limiting the model’s generalization in complex clinical settings. Future extensions should focus on evaluating the model in diverse skin tones and demographics, incorporating explainable AI (XAI) techniques, and optimizing efficiency to support greater adoption.

We also used the Grad-CAM approach to visualize the most discriminative regions in the input images responsible for the model’s decisions to improve the interpretation of our model predictions. Since Grad-CAM is intended for CNN and does not directly apply to RNN, we restricted the explanation to the CNN part of the model. The examples of Grad-CAM heatmaps overlaid over sampled images from the test set are shown in Fig. 10. The highlighted areas are important parts of the image where the model learned to classify, by providing insight into how the model is making decisions. These visualizations show that the model attends to semantically sensible locations, which enhances our trust in its predictions and supports model interpretability.

To evaluate the individual contribution of each architectural component, an ablation study was performed between 4 model variants named CNN, CNN_Transformer, CNN_GRU, and the full hybrid CNN with Parallel (Self-Attention Transformer + GRU). The results with key evaluation parameters are summarized in Table 7. The result shows that the Proposed Hybrid model (CNN with Parallel (SA Trans + GRU)) provides the best performance with accuracy 90.61%, precision 91.12%, recall 90.88%, F1-score 91.00%, AUC 0.968. The two alternative models CNN_Transformer and CNN_GRU perform better than the baseline CNN, but the combined joint use of both modules gives the best performance results. This confirms that the combination of local feature extraction (CNN), global attention (Transformer), and sequential modeling (GRU) leads to more robust and reliable skin cancer classification.

TOPSIS works by identifying solutions that are simultaneously closest to the ideal solution (representing the best performance in all metrics) and farthest from the negative ideal solution (representing the worst). The steps involved are the following. 1.

Construct the Decision Matrix: A matrix X=[xij]m×n, where m is the number of models and n=4 is the number of criteria (Accuracy, Recall, Precision, AUC).

The following table presents the normalized and weighted decision matrix derived from the experimental results shown in Table 8:

Using the above weighted normalized data, we calculate the closeness coefficient Ci∗ for each model. The resulting rankings are presented in the Table 9:

The model CNN with Parallel (SA Transformer + GRU) ranks first with the highest closeness coefficient of 0.8148. This indicates that it offers the best trade-off in accuracy, recall, precision, and AUC in the evaluated setup. Therefore, the proposed model demonstrates robust overall performance for skin cancer classification within the TOPSIS framework.