Hyperspectral Imaging (HSI) plays a pivotal role in various domains such as remote sensing [1], earth observation [2,3], urban planning [4], agriculture [5,6], forestry [7], target/object detection [8,9], mineral exploration [10], environmental monitoring [11,12], climate change [13] food processing, bakery products, bloodstain identification, and meat processing.

Hyperspectral (HS) remote sensing plays a crucial role in urban planning by providing detailed insights and tools for efficient and informed decision-making [14]. HS remote sensors capture and analyze high-resolution spectral data across numerous narrow and contiguous spectral bands, offering comprehensive information on the composition and characteristics of urban environments [15]. HS data enables precise identification and mapping of various urban land cover types, such as vegetation, impervious surfaces, and soil [16]. Additionally, HS Imagining (HSI) facilitates the detection and monitoring of changes in land use, vegetation health, and pollution levels within urban areas [17]. These capabilities enhance urban planners’ ability to assess the impact of urbanization, analyze urban metabolism, and evaluate the effectiveness of sustainability measures. By covering the entire processing chain, from data acquisition to analysis, HS remote sensing serves as a valuable tool for urban planners seeking a deeper understanding of urban environments and their dynamics.

HSI presents both challenges and opportunities for effective classification [18,19]. In recent years, Convolutional Neural Networks (CNNs) [20], Attention Graph [21,22], and Spatial-Spectral Transformers [23,24] have demonstrated remarkable success in various tasks, prompting researchers to explore their potential in HSI analysis [25]. However, achieving robust and reliable classification results requires careful consideration of data sampling techniques [26]. Random sampling for data splitting can lead to several issues. It can result in non-representative training, validation, and test sets, causing models to overfit or underfit. Different random splits produce inconsistent results, making it hard to draw meaningful conclusions [27]. Random sampling offers no control over data distribution, introducing bias in imbalanced datasets [28]. It hinders the reproducibility of experimental results and limits the exploration of data relationships. To address these challenges, disjoint sampling is a crucial yet often overlooked consideration when evaluating spatial-spectral Hyperspectral Image Classification (HSIC) models. As demonstrated by the works [1,22,29–33], traditional evaluations using overlapping training and test samples can lead to biased results and unfair assessments of model performance.

Even though several methodologies meticulously employ disjoint sets for training and testing their models, there’s a notable inconsistency in their approach when it comes to generating land-cover maps [22,30,32]. Specifically, many of these methods deviate from the disjoint sampling principle by utilizing the entire dataset for HSIC (Thematic Maps). This practice introduces a conflict between the reported accuracy and the methodology employed. To address this inconsistency, it is essential to advocate for the use of a disjoint test set exclusively for generating land-cover maps. By doing so, the evaluation process aligns more closely with the principles of unbiased model assessment. It ensures that the model is confronted with truly unseen data during the map generation phase, fostering a more accurate representation of its real-world performance.

Moreover, disjoint sampling is essential for training and evaluating deep models [34,35]. This method involves carefully selecting diverse and representative samples from various regions, land cover types, and environmental conditions to overcome biased or non-representative training data limitations [36]. It ensures the model learns robust features, enhancing classification performance and adaptability to unseen data. Additionally, disjoint sampling facilitates fair and accurate model evaluation by keeping training, validation, and testing samples separate [37]. Furthermore, disjoint sampling is crucial in training SOTA models for HSIC, notably for CNN and Spatial-Spectral Transformer-based models. It enhances generalization, ensures fair evaluation, and enables result interpretability. The use of disjoint training, validation, and test samples is imperative in HSIC for various reasons, such as:

Unbiased Evaluation: It is crucial to evaluate HSIC models using completely separate and disjoint data for training, validation, and testing in order to properly assess a model’s true ability to generalize to new unknown examples [1,29].

Preventing Data Leakage and Mitigating Overfitting: Maintaining disjoint samples for training, validation, and testing is crucial to obtaining an accurate evaluation of a model’s true generalization performance [22,37]. Employing disjoint subsets of the data at each stage of model development is pivotal in augmenting generalization performance. Through iterative training on distinct partitions, the model is compelled to infer underlying patterns shared across diverse examples, rather than being influenced by potentially misleading idiosyncrasies within a single fixed training sample [38,39]. This practice discourages the memorization of irrelevant characteristics specific to individual data samples. Instead, it fosters the capability to effectively process a wider range of presentations, including both seen and unseen examples.

Therefore, considering the above, this paper made the following contributions:

This paper presents a novel approach for disjoint train, validation, and test splits for HSIC. Ensuring the disjoint splits eliminates data leakage between subsets, which can bias performance evaluations. The proposed technique provides a practical implementation for creating disjoint train, validation, and test splits from ground truth data. This allows researchers to obtain unbiased performance evaluations and reliable comparisons between HSIC models.

Let’s consider HSI composed of B spectral bands, each with a spatial resolution of H×W pixels. The HSI data cube, denoted as X∈R(H×W×B), is initially partitioned into overlapping 3D patches [24,40]. Each patch is centered at a spatial location (α,β) and covers a spatial extent of PS=S×S (PS = Patch size) pixels across all B bands. The total number of 3D patches (N) extracted from X (i.e., X∈R(S×S×B)) is given by (H−S+1)×(W−S+1). A patch located at (α,β) is represented as Pα,β and spans spatially from α−S−12 to α+S−12 in width and β−S−12 to β+S−12 in height. The labeling of these patches is determined by the label assigned to the central pixel within each patch as described in Algorithm 1.

The 3D patches extracted from the HSI are used to generate separate training, validation, and test sets using the proposed splitting algorithm. The key algorithm, titled “Disjoint Train, Validation, and Test Split”, handles dividing the HSI data into the respective portions. It takes the Ground Truth (GT) labels and ratios for the test and validation sets (teRatio and vrRatio) as inputs. The unique values in the GT labels and their frequency counts are identified, excluding zeros (background pixel labels). An iterative process is then used to create disjoint training, validation, and test sets based on these unique values and their indices. The resulting indices are utilized to extract and organize the corresponding Hyperspectral cubes and labels for each set. This ensures the subsets are separate while maintaining the integrity of spectral classes during model training and evaluation. The algorithm outputs the training, validation, and test samples along with their matching class labels. This partitioning approach contributes to the robustness and reliability of the subsequent analysis.

Let us consider that n, m, and p represent the finite numbers of labeled training, validation, and test samples, respectively, selected from patch data to form the training, validation, and test sets as shown in Eqs. (1)–(3): (1)DTR=(xi,yi)i=1n (2)DV=(xi,yi)i=1m (3)DTE=(xi,yi)i=1pwhere n and m are the total number of training and validation samples. The remaining p samples constitute the test set. It is important to note that the intersection of the training set, validation set, and test set is an empty set (ϕ), ensuring the distinctiveness of the samples in each set as shown in Algorithm 2, Fig. 1 and Eq. (4).

(4)DTR∩DV∩DTE=ϕ

As shown in Fig. 1, Stage 1: 3D Patch Extraction: The HSI cube is initially divided into overlapping 3D patches. Each patch is centered at a spatial point and spans a PS=S×S pixel extent across all spectral bands. This process is outlined in Algorithm 1. Stage 2: Data Splitting: The extracted patches are used in Algorithm 2 to create disjoint training, validation, and test splits. This splitting is based on the geographical locations of the HSI samples, ensuring that each set covers distinct areas. Stage 3: Feature Learning and Classification: The samples from each split (training, validation, and test) are fed into various models for feature learning and optimization. The learned features are passed through a fully connected layer for classification. The softmax function is then applied to generate class probability distributions. The class probability distributions are used to generate the final ground truth maps for the disjoint validation set, disjoint test set, and the full HSI test set. Each of these stages is marked with red dotted lines in Fig. 1, facilitating a clear and systematic replication of the workflow.

The above disjoint samples (as explained in Algorithm 2) are then processed by the baseline 2D, 3D CNN, and Spatial-Spectral Transformer models [41,42]. In a 2D CNN, the input data undergoes convolution with a 2D kernel function, resulting in the computation of the dot product between the input and the kernel function. The kernel is then applied in a strided manner over the input to cover the entire spatial dimension. Subsequently, the convolved features are subjected to an activation function, which introduces non-linearity into the model, aiding in the learning of non-linear features from the data. For 2D convolution, the activation value of the jth feature map at spatial location (x,y) in the ith layer, denoted by vi,jx,y, can be expressed as shown in Eq. (5). (5)vi,jx,y=ℱ(∑τ=1dl−1∑ρ=−γγ∑σ=−δδwi,j,τσ,ρ×vi−1,τx+σ,y+ρ+bi,j)where ℱ represents the activation function, dl−1 signifies the number of feature maps at the (l−1)th layer, while the depth of the kernel wi,j pertains to the jth feature map at the ith layer. Additionally, bi,j denotes the bias parameter for the jth feature map at the ith layer, with 2γ+1 and 2σ+1 representing the width and height of the kernel, respectively. In contrast, the 3D convolutional process initially calculates the sum of the dot product between input patches and the 3D kernel function, wherein the 3D input patches undergo convolution with the 3D kernel function [40,43]. Subsequently, these feature maps are subjected to an activation function to introduce non-linearity. The Hybrid model generates feature maps of the 3D convolutional layer by applying the 3D kernel function across B spectral bands, which are extracted post-dimensionality reduction, in the input layer. For the 3D convolutional process, the activation value at spatial location (x,y,z) in the ith layer and jth feature map can be expressed as in Eq. (6): (6)vi,jx,y=ℱ(∑τ=1dl−1∑λ=−vv∑ρ=−γγ∑σ=−δδwi,j,τσ,ρ,λ×vi−1,τx+σ,y+ρ,z+λ+bi,j)where all the parameters are the same as defined in Eq. (5) except 2v+1 which is the depth of the 3D kernel along a spectral dimension.

For the Spatial-Spectral Transformer model [44–46], consider fi,j∈RN×D as the input tensor fed into the Transformer, where N signifies the number of patches, and D denotes the dimensionality of each patch post convolutional processing. This encoding process is fused with the input embeddings, enriching the model with spatial arrangement details. At the core of the Transformer lies its foundational architecture, the encoder, which comprises multiple layers housing multimodal attention mechanisms and a feed-forward network. The attention mechanism assumes a pivotal role in facilitating the model’s ability to capture intricate relationships between diverse patches. More specifically, for a given input vi,j, each layer within the Transformer encoder encompasses layer normalization, cross attention, and Multi-layer Perceptron (MLP).

Given a query matrix Q∈ℛ(nq×dk), key matrix K∈ℛ(nk×dk), value matrix V∈ℛ(nv×dv), and mask matrix M∈ℛ(nq×nk), where nq,nk,nv, and dv represent the number of queries, keys, values, the dimensionality of keys/queries, and the dimensionality of values, respectively. Lets define the weight matrices for each head i as 𝒲iQ∈ℛdmodel×dk, 𝒲iK∈ℛdmodel×dk, and 𝒲iV∈ℛdmodel×dv. Apply linear transformation as: Qi=Q×𝒲iQ, Ki=K×𝒲iK, and Vi=V×𝒲iV. Later compute the attention scores for each head and apply the Softmax to compute the weighted sum of the values as shown in Eq. (7). (7)AttenW=Softmax(Qi×KiTdk+M)×V

Then calculate the concatenate the outputs from all heads as ℋ=Concat(ℋ1,ℋ2,⋮,ℋh) and apply a final linear transformation as ℋattl. Then given an input x∈ℛdmodel, layer normalization is computed as shown in Eq. (8). (8)Norm(x)=x−μν2+ηγ+λwhere μ is the mean and ν2 is the variance of x, η is a small constant for numerical stability and γ and λ are learnable parameters. Later the information is processed through MLP that consists of two linear transformations with a ReLU activation function as shown in (9). (9)MLP(x)=ReLu(x𝒲1+b1)𝒲2+b2where 𝒲1∈ℛdmodel×dff, b1∈ℛdff, 𝒲2∈ℛdff×dmodel, and b2∈ℛdmodel. The output of the multi-head self-attention Hattl is added to the input and normalized as presented in Eq. (10). (10)ℋ^attl=Norm(ℋattl⊕ℋ(l−1))

Eq. (10) adds the attention output ℋattl to the previous layer’s output ℋatt(l−1) and normalizes the results. (11)ℋMLPl=MLP(ℋ^attl)

Eq. (11) computes the output of the attention mechanism in the l-th layer through an MLP. The MLP consists of two linear layers with a ReLU activation function applied between them. The output of the MLP is added to the normalized attention output and normalized again as: (12)ℋl=Norm(ℋMLPl⊕ℋ^attl)

Eq. (12) adds the output ℋMLPl to the normalized attention output and then normalizes the result once more. Afterward, ℋl is flattened into (ℋl,(b×h×w,1)), where b, h, and w denote the batch size, height, and width, respectively. Finally, a Softmax function is utilized to produce the GT maps.

Average, overall, and Kappa accuracy may not always be appropriate measures, especially when there are significant differences in the number of samples in each class within a dataset. To clarify this point, consider the following scenario. Suppose we have a dataset with 90 healthy (positive) individuals and 10 not-healthy (negative) individuals. If a conventional model correctly predicts 90% of individuals as healthy, it might still predict the not-healthy individuals as healthy. What would be the best accuracy in this scenario?

In this setting, the model identifies 10 individuals as “False Negative”, 0 as “True Positive”, 0 as “False Positive”, and 90 as “True Negative”. Thus, the average accuracy would be 90%, i.e., 90+0100=0.9. However, the model is highly biased since it predicts all the not-healthy individuals as healthy. In such scenarios, overall and average accuracies can be misleading or misinterpreted, indicating that these measures alone are not sufficient for evaluating a machine learning model. Therefore, it is important to consider additional statistical measures to validate the model beyond simple accuracy metrics.

Several statistical tests can be used to validate the results. For this work, we consider Recall (True Positive Rate or Sensitivity), Precision (Positive Predictive Value, PPV), and F1 score (a harmonic mean of precision and recall). In an ideal scenario, PPV should be 1, which occurs when the numerator and denominator are equal, i.e., when True Positive (TP) equals TP + False Positive (FP), making FP equal to 0. As FP increases, PPV decreases, leading to an inappropriate model. A similar trend can be observed for Recall, where False Negative (FN) replaces FP. Recall and PPV can be computed as follows: (13)Recall=TPTP+FN (14)Precision=TPTP+FP

For a classification model to be effective, both Precision (Positive Predictive Value, PPV) and Recall need to be high, which means that both False Positives (FP) and False Negatives (FN) must be low. In addition to Recall and PPV, the F1 score should also be computed, as it combines both Recall and PPV to provide a single metric that offers statistical significance and deeper insight into the classifier’s generalization performance. The F1 score can be calculated as follows: (15)F1 score=2×Precision×RecallPrecision+Recall

A model is considered effective if it achieves high values for PPV, Recall, and the F1 score. These metrics provide a more comprehensive evaluation of the model’s performance compared to using accuracy alone. The detailed statistical results are presented in Table 11.

This paper introduced a novel technique for generating disjoint train, validation, and test splits in Hyperspectral Image Classification (HSIC). By efficiently partitioning the ground truth data, the proposed technique ensured unbiased performance evaluations and facilitated reliable comparisons between classification models. It proved to be a valuable tool for creating disjoint splits, guaranteeing that the subsets were representative of the entire dataset and that the classification results were not skewed by data leakage. While the technique demonstrated significant advantages, limitations were acknowledged, and opportunities for further improvement were identified. Future research could investigate alternative data-splitting strategies that incorporate additional factors, such as class imbalance or spatial coherence, to further enhance the representativeness and generalizability of the subsets. Addressing these aspects could lead to the development of even more robust and effective data-splitting techniques for HSIC.