The SFGA approach selects the best machine learning algorithm from a group of algorithms. This algorithm is the one that can best understand the dataset. We then use feature importance ranking strategies to rank the most important features using that algorithm. By using these strategies, we can quickly determine the most important features out of 200.

The second feature strategy uses the SHAP algorithm, which provides a measure of feature importance specifically for linear models when dealing with multicollinearity. The salient features are extracted from the training samples, providing insights into the distinctive characteristics that aid in classification.

Once the salient features are identified, the SFGA method maps these features to corresponding image processing techniques to enhance the training images. The objective is to emphasize and highlight the salient features, making them more prominent and distinguishable. By augmenting the training images in a way that enhances the discriminative features, the deep learning model can learn more effectively and improve its classification performance.

To achieve this, SFGA applies various image processing methods that align with the salient features identified by the machine learning algorithm. These methods include contrast enhancement, edge enhancement, texture enhancement, color enhancement.

By incorporating these image processing techniques based on the identified salient features, the SFGA method enhances the training images, thereby improving the deep learning model’s discriminating power. The augmented training dataset, with enhanced salient features, is then used to retrain the deep learning model, allowing it to learn more effectively and make more accurate classifications.

To help with this process, we introduce a supporting process called “Domain Knowledge Mapping” (DKM). DKM involves exploring existing literature to identify prior image processing techniques that have successfully enhanced H&E training images for classification tasks. By relying on well-established techniques, we can avoid using unverified information. We can find the DKM mapping tables in Tables A1–A5 of the appendix. An example is shown in Table A1, which is about the enhancement of tumor-immune-epithelial cells. Fig. 2 shows the corresponding enhancement results.

With DKM, a toolbox of image processing techniques is readily available for enhancing the deep learning training images after identifying the salient features of the cells involved in the classification task. To facilitate the semi-automatic procedure of SFGA, the image processing tasks should be automated by programming the mapping of the salient features and image processing techniques and their parameters as software codes or scripts. It is important to standardize the image processing techniques and their parameters being used across all training samples for a certain cell classification task.

The Salient Features Guided Augmentation (SFGA) method is formulated through a series of definitions and methods. It involves converting an original image set into a matrix of data instances and features (p × q matrix) in Definition 1. In Definition 2, a candidate model is selected from a model set based on its performance in training using the features, accuracy, error, and F1 score. Definitions 3 and 4 describe how important and salient features are chosen based on performance changes and Shapley values, respectively. Finally, in Definition 5, the selected features are used to enhance images through established image enhancement methods.

Definition 1 (Cells Matrix). Suppose that there is an original image set X, convert the image set into data set Z, the data set Z are represented as a p×q matrix, where p is the number of data instances (cells) and q is the number of features.

Definition 2 (Candidate Model). In order to select an appropriate Training/Testing ML Model, we should build up a model set M which contains Machine learning methods M={m1m2m3⋅⋅⋅mn}. Then using all the q cell features in the dataset to train the model and calculate their performance respectively. The performance can be measured by accuracy, error and F1 score. Denote the performance data set P={p1,p2,p3,⋅⋅⋅,pn}. There is a mapping relationship f between set M and set P, i.e., f:M→P. We select m where m=f−1(max{P}). The model with highest performance will be selected to be the candidate model.

Definition 3 (Important Features). The top-h important features can be selected by the following method. For each feature, where i∈the features set, compute the initial performance P0 under the candidate model and a chosen evaluation metric (e.g., accuracy or ROC). Create a copy of the original data set Z and shuffle the values of feature i randomly while keeping other features unchanged. Recompute the performance measure Pi using the shuffle dataset. Compute the permutation importance PH value. PH value is the difference between the initial performance P0 and the recompute performance Pi, can be formulated to be: (1)PHi=P0−Pi

Then define a set PH which contains all elements PHi for each i that belongs to the features set. Choose the largest h elements in PH set where H={PH1,PH2,⋅⋅⋅,PHi,⋅⋅⋅,PHq}. The elements are the top-h important features and construct set H with them.

Definition 4 (Salient Features). The top-k salient features can be selected by the following method. Let W denote the whole features set, and S denote the subset of W, i.e., S⊂W. Denote j as the feature that is not in the set S but it is in the set W, i.e., j∈WS. Then train the set S∪j and compute the corresponding prediction result gs∪{j}(XS∪{j}) the same train the set S and compute the corresponding prediction result gs(XS). To compute the effect changes on the model prediction after containing feature j when training the model, we make a difference between the two prediction results mentioned before, can be formulated to be: (2)gs∪j(XS∪j)—gs(Xs)

And the Shapley values can be computed and used as feature attributions. The formula of the Shapley values are: (3)ϕj=∑s⊆w∖{j}|s|!(|w|−|s|−1)!|w|![gs∪{j}(xs∪{j})−gs(xs)]

Define a set PK={ϕ1,ϕ2,⋅⋅⋅,ϕj,⋅⋅⋅,ϕq}. Where the elements of the set are the Shapley values ϕj. Choose the largest h elements in PK set. The elements are the top-k salient features and construct set K with them.

Definition 5 (Enhance Images Features). The features for enhancing the images can be selected by the following method. First, choose the features both are important features and salient feature. These features are contained in the set E where set E is defined :=H∩K. Then we use well-established image enhance method to processing the images, denote set F to contain these methods. Then there exists a mapping function h that can map image enhance features to image enhance method h:E→F.

SFGA method combines these definitions to guide the selection and enhancement of features for image processing and model training.

In this paper, our experimentation utilizes the CAMELYEON 2017 dataset [20]. The CAMELYEON 2017 dataset is publicly available at https://iciar2018-challenge.grand-challenge.org/ (accessed on 10 Jan 2025). Which contains 1399 H&E-stained breast cancer sentinel lymph node sections. In collaboration with medical experts, we selected 50 Whole-Slide Images (WSI) featuring various characteristics, including ENE. Each WSI has a resolution of approximately 200,000 × 100,000 pixels in 3-channel RGB format, with an uncompressed data size of 55.88 GB per level. For practical handling, we used the compressed version at 40× magnification, with file sizes averaging 2 to 4 GB.

Given that a typical region of interest (ROI) within an H&E-stained image contains at least 20,000 cells along with various other materials, it presents a considerable labeling effort. To facilitate this, we employ the Watershed cell segmentation algorithm [21], which leverages intensity and texture differences to segment individual cells accurately. The algorithm defines markers representing cell locations and fills the space between them to create segmented regions of uniform intensity, ensuring precise and non-overlapping cell segmentation.

The primary objective of our data preparation process is two-fold. Firstly, it involves creating doctor’s annotations to establish ground truth for training samples in the construction of a deep learning model. Secondly, it aims to identify salient features from cell characteristics, which are essential for enhancing deep learning training images using our novel Salient Features Guided Augmentation (SFGA) method. The important initial step in this process is the creation of annotations by a doctor with more than a decade of experience, who was engaged from the First People’s Hospital of Foshan. The tasks in our experiment encompass cell instance segmentation, mitotic counts, nuclear pleomorphism, tubule formation, and capsule skin recognition.

The SFGA approach selects the best machine learning. We conducted a series of experiments to test the effectiveness of our deep learning-based concept for enabling Cell-Level Analytics (CLA), which is a crucial part of ITA. Our objective was to verify the performance of deep learning and our proposed SFGA-enhanced model in various cell-level recognition tasks, including tumour-immune-fibroblast classification, proliferating cell recognition, mitotic cell recognition, nuclear pleomorphism classification, and capsule skin recognition.

We used ResNet 48 as the deep learning model. The design of the experimentation is shown in Fig. 3. The experimental design included two types of data augmentation: standard augmentation (rotations from 0 to 360 degrees in 5-degree increments, scaling with six zoom levels from 0 to 5, and random shifts of 20 pixels) and light augmentation (only image rotation). Six combinations were tested: Original, Original + Augmentation, Original + (Light)Augmentation, Enhanced, Enhanced + Augmentation, and Enhanced + (Light)Augmentation.

Using breast cancer metastasis at the lymph node as the source, we split the dataset into 80% training and 20% validation. After data acquisition, training image tiles were enhanced using SFGA, and salient features were identified to determine suitable image processing techniques for improving model accuracy. The model was trained using 5-fold cross-validation to avoid overfitting and generate performance metrics, followed by performance evaluation.

Our results show that deep learning-based CLA tools can support higher-level pattern-level analytics. DL-SFGA provides satisfactory inference accuracy, meeting high-performance expectations for cancer metastasis medical evaluation. Figs. 4 and 5 present radar charts comparing different DL-SFGA + augmentation combinations for cell type recognition, and butterfly charts highlighting the performance gains of DL-SFGA over the original dataset.

Fig. 6 shows the training and validation accuracy curves for tumor-immune-fibroblast cell type classification under various data augmentation settings. The ‘Original’ dataset delivers strong performance with a final training accuracy of 78.375% and an average of 92.398%. The loss converges to 0.642 at the final training stage, averaging 0.208 overall. On the validation set, the model achieves 85.67% accuracy at epoch 200, with an average of 94.26%, and loss values of 0.509 and 0.176.

Applying standard augmentation to the ‘Original’ dataset led to a slight decrease in performance, with final training accuracy dropping to 72.5% and validation accuracy to 80.12%. However, using a lighter augmentation strategy improved performance, achieving a final training accuracy of 80% and validation accuracy of 87.4%. Using an ‘Enhanced’ dataset without further augmentation resulted in substantial performance gains, with final training accuracy reaching 86.75% and validation accuracy hitting 91.6% at epoch 200. Reapplying augmentation to this ‘Enhanced’ dataset, however, caused a decline in performance, particularly in the average validation accuracy.

In conclusion, for cell detection tasks, the best performance was achieved using a high-quality enhanced dataset without additional augmentation, highlighting the importance of superior data quality. Future research could explore more advanced augmentation techniques.

This study aims to evaluate the impact of data augmentation and enhancement on model performance for identifying Ki67 positive cells in Haematoxylin and Eosin (H&E) stained samples through six different dataset configurations. The ‘Original’ dataset served as a benchmark, achieving a final training accuracy of 75.625% and an average training accuracy of 92.134%; its performance on the validation set was equally impressive, with final and average accuracies of 81.02% and 93.58%, respectively.

However, applying standard augmentation to the ‘Original’ dataset led to a decline in all metrics, likely due to overfitting. Light augmentation improved this situation. When using the ‘Enhanced’ dataset, there was a significant performance boost, with training accuracy reaching 77.375% and averaging at 95.10%, while validation accuracy soared to 96.49%. Yet, standard augmentation on the ‘Enhanced’ dataset caused a sharp drop in validation performance, indicating that excessive augmentation on already enhanced data may lead to overfitting. Light augmentation once again showed better results.

The experimental findings suggest that although data augmentation can expand and diversify training data, it must be applied cautiously to avoid issues like overfitting. Data enhancement clearly benefits model performance, underscoring the importance of investing in high-quality, enhanced datasets for complex tasks such as Ki67 positive cell detection.

In the pursuit of identifying mitotic cells in H&E stained samples, our study involved investigating six distinct data configurations and their subsequent impact on model performance.

Using the ‘Original’ dataset, the model performed well on the training set with a final training accuracy of 69.125% and an average training accuracy of 83.119%. However, validation performance showed a decline with final and average accuracies of 53.85% and 42.43%, respectively, indicating overfitting. Increasing the original dataset did not improve this situation and may have exacerbated overfitting. Applying light augmentation, however, improved the average validation accuracy to 49.56%, suggesting that it can aid generalization.

Transitioning to the ‘Enhanced’ dataset significantly boosted performance, with the average validation accuracy rising to 81.039%. Further augmenting the enhanced dataset increased the average validation accuracy to 82.14%, despite a slight drop in final training accuracy to 63.75%. This suggests that data augmentation introduces additional diversity and robustness, positively impacting model performance.

In summary, our study shows that data enhancement is crucial for improving model performance and generalization but requires careful calibration. Future research should focus on optimizing these techniques to enhance model performance and generalizability in detecting mitotic cells in H&E-stained samples.

In the process of identifying polymorphic cells in H&E-stained samples, our study provides significant insights.

For the ‘Original’ dataset, the model achieved a training accuracy of 63.75% at epoch 200, with an average of 80.27%; validation accuracy was 63.31% at epoch 200, averaging 50.82%. The notable discrepancy between training and validation performance suggests potential overfitting. For the ‘Original + Augmentation’ dataset, despite slight decreases in both training (61.75%) and validation (61.57%) accuracies at epoch 200, this decline may be attributed to the introduction of unnecessary complexity by the augmentation strategy. In contrast, the ‘Original + (Light)Augmentation’ dataset showed improved validation accuracy (61.82% at epoch 200, averaging 61.16%), suggesting that lighter augmentation can reduce noise or distortion, thereby enhancing model performance.

When applied to the ‘Enhanced’ dataset, enhancement techniques significantly boosted model performance: training accuracy reached 89.5% at epoch 200 (averaging 94.63%), and validation accuracy was 92.42% (averaging 95.16%). This indicates that appropriate enhancement methods, such as denoising or contrast adjustment, can substantially improve learning outcomes by reducing irrelevant variations and emphasizing key features. The ‘Enhanced + (Light)Augmentation’ dataset demonstrated the best results, achieving training and validation accuracies of 82.88% (averaging 89.84%) and 85.70% (averaging 92.20%), respectively, at epoch 200.

This analysis aims to evaluate the impact of various data enhancement techniques on the performance of models detecting extranodal extension (ENE) in H&E-stained lymph nodes. The results highlight the importance of dataset manipulation and its effect on model performance. Fig. 7 shows the training and validation accuracy curves for capsule skin recognition under various data augmentation settings.

Models trained on the ‘Enhanced’ dataset showed significant advantages, with training accuracy reaching 85.24%, far exceeding the ‘Original’ dataset’s 66.21%. For validation data, the ‘Enhanced’ dataset achieved a validation accuracy of 83.46%, markedly higher than the ‘Original’ dataset’s 51.72%. This indicates that the enhanced dataset not only improves training effectiveness but also enhances the model’s adaptability to new, unseen data.

However, not all augmentation methods yield positive outcomes. Excessive augmentation led to decreased training and validation accuracies in the ‘Original + Augmentation’ dataset, dropping to 40.32% and lower levels, respectively. This suggests that improper or excessive use of augmentation can harm model performance. In contrast, the ‘Enhanced + (Light)Augmentation’ approach showed promise, achieving average training and validation accuracies of 72.66% and 71.42%, respectively. However, further application to an already ‘Enhanced’ dataset provided limited improvements. In terms of loss, the ‘Enhanced’ dataset exhibited the lowest training and validation losses, with averages of 0.437 and 0.486, respectively, significantly lower than those of the ‘Original’ and ‘Augmented’ datasets. This underscores the robustness of the enhanced dataset.

Overall, these results demonstrate that appropriate data enhancement can significantly improve ENE detection performance in H&E-stained lymph nodes.