Feature selection methods can be broadly categorized into two major streams: mutual information (MI)-based filter approaches and regression-based approaches.

The first stream is represented by classical methods such as mRMR (minimum redundancy maximum relevance) [9], which aim to maximize the relevance between features and labels while minimizing redundancy among selected features. These approaches rely on information-theoretic criteria, particularly mutual information, to measure pairwise dependencies between variables. They are straightforward and model-agnostic, and they can capture nonlinear relationships that linear methods may overlook. Given the full feature set F and the subset of features S that have already been selected, the next feature fj is chosen by maximizing the trade-off between its relevance to the target and its redundancy with the selected features, as follows: (1)arg⁡maxfj∈F−SMI(fj,y)−1|S|∑fi∈SMI(fi,fj)where MI(⋅,⋅) denotes the mutual information. However, they typically employ greedy search strategies, selecting features sequentially based on local criteria. As a result, they may fail to consider global interactions among features and often lead to suboptimal feature subsets.

The second stream is exemplified by regression-based methods such as efficient and robust feature selection via joint l2,1-norms minimization [10], which formulate feature selection as an optimization problem. The objective function follows: (2)minW||XW−Y||2,1+γ||W||2,1.where X is data matrix, ϒ is label matrix. These approaches aim to minimize reconstruction error, while imposing structural regularization to induce sparsity across features. Such methods benefit from global optimization and can naturally incorporate feature-label dependencies into a unified objective. However, because they fundamentally rely on a linear regression model, they are often limited in capturing complex or nonlinear relationships that are essential in many real-world datasets.

Given a dataset X∈Rn×d consisting of n patterns and d features, along with a multi-label set ϒ consisting of c labels, the basic objective function of a regression-based feature selection method can be formulated as: (3)minW||XW−Y||F2s.t. W≥0where W is a weight matrix that is subject to a non-negativity constraint. The non-negativity constraint ensures that feature selection is represented by positive values, while unselected features are represented by zeros. The larger the absolute value in W, the more reliable the corresponding feature is considered. After solving this optimization problem, features corresponding to rows with larger norms in W are selected.

The objective function is designed to find features that minimize the difference between the data projected by W and the multi-label targets. To construct this objective function, the Frobenius norm is used. The function is convex with respect to W, allowing the optimization problem to be solved efficiently. The top k features are selected based on the magnitude of each row in the optimized W.

Although Eq. (3) offers an efficient convex formulation, its modeling capacity is limited because it only captures linear relationships between features and labels. In multi-label learning, however, three important aspects are often overlooked in such regression-based approaches: 1.

Feature redundancy: selecting highly correlated features reduces the generalization power [9].

To address these limitations, we incorporate mutual information into the objective function. Mutual information captures the degree of statistical dependency between random variables and can reflect both linear and non-linear relationships [9,19,24,27,35]. The mutual information between two arbitrary random variables A and B is defined as follows: (4)MI(A,B)=∑a∈A∑b∈Bp(a,b)log⁡[p(a,b)p(a)p(b)]where P(a,b) represents the joint probability distribution function of A and B, while p(a) and p(b) represent their marginal probability distribution functions. In the case of continuous random variables, this summation is replaced by an integral: (5)MI(A,B)=∫A∫Blog⁡[p(a,b)p(a)p(b)]da db.

When the i-th feature vector and the j-th label vector are denoted by xi and yj, respectively, the relevance and redundancy using mutual information can be expressed as MI(xi,yj) and MI(xi,xj), respectively. In a multi-label environment, the relationships between labels are additionally considered as [15].

By calculating these associations, three matrices Q∈Rd×d, R∈Rc×c and S∈Rd×c can be constructed, where each element in these matrices is defined as follows: (6)Qij=MI(xi,xj)(feature redundancy)Rij=MI(yi,yj)(label dependency)Sij=MI(xi,yj)(feature-label relevance)where Qij refers to the (i,j)-th element of Q. The elements of these matrices represent the associations between features and labels. When the data is binarized for calculating mutual information, the mutual information is always greater than or equal to 0, and less than or equal to 1 when using a base-2 logarithm. Therefore, every elements in the matrices Q, R and S lies in within the range [0, 1]. The matrix Q represents feature redundancy, which should be minimized in order to eliminate redundant or highly correlated features. The matrix R captures the relationships between labels; since features associated with strongly correlated labels are desirable, the negative of this term is included in the objective to encourage their selection under a minimization framework. Similarly, S quantifies the relevance between features and individual labels. By incorporating mutual information into the regression formulation, the model is able to capture not only simple linear relationships but also complex non-linear dependencies. To promote the selection of highly relevant features, the negative of S is also incorporated into the objective function.

By combining these values with the regression Eq. (3), an objective function can be designed as follows: (7)minW||XW−Υ||F2+αTr(WTQW)−βTr(RWTW)−γTr(STW)s.t. W≥0where α,β, and γ are weights for redundancy, label correlation, and relevance, respectively, and are all positive. The hyperparameters α,β,γ control the influence of each regularization term. In Tr(WTQW), the i-th value represents the sum of the redundancies of the i-th feature with other features. Similarly, Rw(i)Tw(i) represents the sum of the associations between the i-th feature and the labels, and STw(i) represents the sum of the relevance of the i-th feature across all labels. The interpretation of each term is as follows:

Regression loss (||XW−Y||F2): encourages accurate label reconstruction via a linear regression model.

Optimizing the proposed objective function thus corresponds to selecting features that not only minimize redundancy and maximize relevance and label association, but also contribute to accurately reconstructing the label space through a linear regression model. In this formulation, the regression-based loss captures the predictive structure in a supervised setting, while the mutual information-based terms guide the selection toward features that exhibit strong statistical dependency with the labels and minimal overlap with other features. This hybrid design enables the model to balance predictive accuracy with information-theoretic feature quality. In the next subsection, we present an efficient optimization framework to solve the proposed objective function under the non-negativity constraint.

The convexity of the proposed objective function is primarily determined by the terms Tr(WTQW) and Tr(RWTW). When the matrices Q and −R are positive definite, the objective function is convex, which facilitates efficient optimization. However, since Q and −R are not guaranteed to be positive definite, we enforce convexity by adding the absolute value of the minimum eigenvalue of each matrix to its diagonal elements. This modification ensures that both Q and −R become positive definite. Importantly, because Q and R originally have zero diagonal entries, adding a uniform scalar to the diagonal does not change the mutual-information structure that drives feature relevance. This correction preserves the feature selection behavior while ensuring positive definiteness for convex optimization.

To ensure the positive definiteness of the matrices Q and R, we adjust each matrix by adding the absolute value of its minimum eigenvalue to its diagonal entries. This yields modified matrices Q^ and R^, which are guaranteed to be positive definite. The transformation is defined as follows: (8)Q^=Q+(|λmin(Q)|+ε)⋅Id, R^=R−(|λmax(R)|+ε)⋅Icwhere λmin(Q) and λmax(R) denote the smallest and largest eigenvalues of Q and R, respectively, Id, Ic are identity matrices of appropriate dimensions matching Q and R, and ε is a small positive value to guarantee strict positive definiteness. This adjustment preserves the relative structure of the feature space while ensuring the convexity of the objective function. The optimization objective can be formulated as follows: (9)minW||XW−Y||F2+αTr(WTQ^W)−βTr(R^WTW)−γTr(STW)s.t. W≥0

To solve the modified convex objective function with a non-negativity constraint, we employ the projected gradient descent (PGD) algorithm. This algorithm performs optimization based on the gradient with respect to W, and any negative entries in W are projected to zero to enforce the non-negativity constraint. The gradient of the objective function with respect to W is given by: (10)∇W=2XT(XW−Y)+2αQ^W−2βWR^−γS.

Algorithm 1 presents the complete procedure of the proposed method. In Algorithm 1, the step size η is provided as a user-defined input. In practice, a theoretically sound upper bound can be derived from the Lipschitz constant of the gradient. For the proposed objective, the gradient is given by ∇f(W)=2X⊤(XW−Y)+2αQW−2βWR+γS, and the Lipschitz constant can be estimated as L=2‖X⊤X‖2+2α‖Q‖2+2β‖R‖2. Thus, a stable choice is η=1/L, which guarantees monotonic descent and convergence under the projection constraint. Since the linear term γTr(S⊤W) does not affect L, this bound remains valid even for large γ. In addition, a backtracking line search with Armijo condition is used to further ensure stability in early iterations.

The overall computational complexity of the proposed algorithm consists of three main components: (1) the computation of the mutual information-based matrices Q, R, and S; (2) the eigenvalue correction required to ensure positive definiteness of Q and R; and (3) the iterative optimization procedure via projected gradient descent (PGD).

Q∈Rd×d encodes the pairwise mutual information between features, requiring O(d2) time. R∈Rc×c encodes the mutual information between labels, requiring O(c2) time. S∈Rd×c encodes the mutual information between features and labels, requiring O(dc) time. Thus, the total cost for constructing these matrices is O(d2+c2+dc).

To ensure the convexity of the objective function, we modify Q and R by adding the absolute value of their minimum eigenvalue to their diagonals: Computing the minimum eigenvalue of a symmetric matrix (e.g., via power iteration or Lanczos methods) typically requires O(d2) for Q and O(c2) for R, assuming low-rank approximations or a small number of iterations. Therefore, this step adds O(d2+c2) to the complexity.

In each iteration of PGD, the gradient with respect to W∈Rd×c is computed. The complexity of computing each term is as follows: XT(XW): O(ndc), Q^W: O(d2c), WR^: O(dc2). The term γS is constant once computed: O(dc). Thus, the per-iteration cost of gradient computation is: O(ndc+dc+dc2). Assuming the algorithm converges in T iterations (typically a small constant or logarithmic in practice), the total optimization cost is O(ndc+d2c+dc2).

Combining all components, the overall computational complexity is O(ndc+d2c+dc2+d2+c2). The proposed algorithm is thus computationally efficient for moderate values of d and c, and scalable to larger datasets when T is reasonably small.

The proposed method involves three hyperparameters, α, β, and γ, which control the contributions of smoothness, redundancy penalization, and label-alignment terms, respectively. In practice, these parameters are tuned over a coarse logarithmic grid, which keeps the computational cost manageable due to the convexity of the objective and the low per-iteration complexity of the solver.

This section presents the experimental results to evaluate the effectiveness of the proposed method in improving classification performance. We compared classification performance using the Multi-label k-Nearest Neighbors (MLkNN) classifier [36], the Linear Support Vector Machine (LinSVM) classifier [17,20,23], and the Multi-Label Decision Tree (MLDT) classifier [18]. In MLkNN, the value of k was set to 5. MLkNN has been widely used in previous studies to benchmark the performance of multi-label feature selection methods [7,8,19]. The regularization parameter C of the LinSVM classifier was determined through validation during the training phase. For each experiment, the training and test data were randomly split in an 8:2 ratio, and the process was repeated 10 times to compute the average performance.

Seven datasets were used in the experiments. The cal500 dataset is a multi-label music annotation dataset consisting of 500 Western popular songs, each annotated with multiple tags describing acoustic, emotional, and semantic content [37]. The corel5k dataset is an image annotation dataset comprising 5000 images, each associated with one or more textual labels from a controlled vocabulary of 260 words. The emotions dataset contains 593 music tracks with 72 audio features, each labeled with one or more of six primary emotional categories [38]. The enron dataset is a text corpus derived from company emails [13]. The genbase dataset is a biological dataset for protein function classification, where each instance represents a protein and each label corresponds to a specific function. The medical dataset consists of 978 clinical free-text reports, each labeled with up to 45 disease codes [39]. The slashdot dataset is a relational graph dataset representing a social network of users from the technology news site Slashdot. It includes directional friend/foe relationships among users, making it suitable for multi-label classification and link prediction tasks in networked data [40]. Detailed characteristics of each dataset are summarized in Table 1.

For evaluation, we used three metrics: Hamming Loss (hloss), Ranking Loss (rloss), and Multi-label Accuracy (mlacc) [8]. Lower values of Hamming Loss and Ranking Loss indicate better classification performance, while higher values of Multi-label Accuracy reflect better performance. The number of selected features was determined based on a proportion n of the total number n of feature patterns in each dataset by referring to the methodology in [41]. To uniquely distinguish each sample in a dataset, at least log2⁡n independent binary features are theoretically required, where n denotes the number of samples. For example, only three features are sufficient to perfectly differentiate eight samples. However, this number is often too strict in real-world datasets, since features are typically not fully independent. Therefore, we adopt n as a practical heuristic for the number of selected features, which provides a reasonably small yet representative subset size.

To compare against the proposed method, we selected five existing approaches: AMI [42], MDMR [16], FIMF [8], QPFS [7], and MFSJMI [17]. AMI selects features based on the first-order mutual information between features and labels. MDMR introduces a new feature evaluation function that considers mutual information between features and labels as well as among features. FIMF limits the number of labels considered during evaluation to enable fast mutual information-based feature selection. QPFS reformulates the mutual information problem into a quadratic programming framework to balance feature relevance and redundancy. MFSJMI selects features by considering label distribution and evaluating the relevance using joint mutual information. For all compared methods, including the proposed one, the hyperparameters α, β, and γ were varied over the range 10−3,10−2,…,103. The best-performing result from this grid search was reported. To enable reliable mutual information estimation, continuous features are discretized using the Label-Attribute Interdependence Maximization (LAIM) method [43], which is specifically designed for multi-label learning.

Tables 2–4 summarize the classification performance of different multi-label feature selection methods evaluated with the MLkNN classifier. The best result for each dataset and metric is highlighted in bold. Table 2 presents the Hamming loss results. The proposed method achieves the lowest loss on six out of seven datasets, clearly outperforming the existing approaches. In particular, substantial improvements are observed on the emotions, enron, and medical datasets, demonstrating that the proposed formulation effectively reduces label-wise prediction errors. Although FIMF performs slightly better on corel5k, the proposed method achieves the overall best average performance across datasets. Table 3 reports the Ranking loss results. The proposed approach again shows strong superiority, achieving the lowest ranking loss on six datasets. These results indicate that the selected features enable the classifier to preserve the relative ranking of relevant and irrelevant labels more effectively than competing methods. Notably, while AMI achieves the best result on corel5k, the proposed method shows consistent improvements in more complex datasets where label dependencies are pronounced. Table 4 presents the Multi-label accuracy results. The proposed method outperforms all baselines on six datasets, achieving particularly large gains on corel5k, enron, and medical. This confirms that the proposed approach can identify highly discriminative and non-redundant features that generalize well across diverse label spaces. Although QPFS slightly exceeds the proposed method on slashdot, the overall performance trend demonstrates the robustness and adaptability of the proposed method across different data characteristics.

Tables 5–7 present the classification results obtained using the LinSVM classifier. Table 5 summarizes the Hamming loss results. The proposed method achieves the lowest loss on all datasets. These results show that the selected features effectively reduce instance-level prediction errors even when evaluated with a linear classifier. Table 6 presents the Ranking loss results. The proposed method achieves the best results on four datasets-cal500, emotions, genbase, and medical-showing its ability to preserve the label ranking order with minimal degradation. While FIMF slightly outperforms others on corel5k and MDMR performs best on enron, the proposed approach remains highly competitive across datasets. The performance gain on emotions and medical demonstrates that the proposed feature selection method can effectively model label dependencies even under a linear decision boundary. Table 7 reports the Multi-label accuracy results. The proposed method achieves the highest accuracy on six datasets, with particularly large gains on corel5k, enron, and medical. This demonstrates the strong discriminative capability of the selected features and their ability to generalize across datasets with varying label sparsity. FIMF predicted all-zero label vectors in 9 out of 10 runs on Corel5k, leading to nearly zero multi-label accuracy.

Tables 8–10 summarize the experimental results obtained using the MLDT classifier. Table 8 reports the Hamming loss results. The proposed method achieves the lowest loss on all seven datasets, showing remarkable consistency and robustness. In particular, the performance improvements on emotions, enron, genbase, and medical are significant, indicating that the proposed feature selection strategy effectively reduces label-wise prediction errors even for complex label spaces. Table 9 presents the Ranking loss results. The proposed approach achieves the best performance on five datasets while remaining highly competitive on the others. These results suggest that the proposed method allows the MLDT classifier to better preserve the relative ranking between relevant and irrelevant labels. Notably, the performance gain on corel5k and medical is substantial, confirming that the proposed MI-guided optimization enhances feature selection for both high-dimensional and label-dependent data. Table 10 shows the Multi-label accuracy results. The proposed method achieves the highest accuracy on all seven datasets, highlighting its superiority in overall predictive capability. The large gains on emotions, enron, and medical datasets illustrate that the proposed formulation captures label correlations more effectively than existing feature selection methods.

Fig. 1 illustrates the MLkNN performance variation of the proposed and comparative feature selection methods on the emotions dataset, where the number of selected features was gradually increased from 3 to 30. Three evaluation metrics-Hamming loss, Ranking loss, and Multi-label accuracy-were employed to analyze the behavior of each method under different feature subset sizes. As shown in the figures, all methods generally improve as the number of selected features increases, but the rate and stability of improvement differ substantially. For the Hamming loss, the proposed method consistently achieves the lowest values across all feature subset sizes, indicating robust generalization and effective elimination of redundant or noisy features. While MFSJMI and QPFS also exhibit a decreasing trend, their performance fluctuates more notably, suggesting less stability in feature selection. In terms of Ranking loss, the proposed method again demonstrates superior performance, maintaining lower loss values over the entire range of feature counts. This consistent improvement shows that the proposed formulation preserves the relative order between relevant and irrelevant labels more effectively than other methods, particularly when the feature space is limited. Finally, for Multi-label accuracy, the proposed method achieves the highest accuracy throughout the experiment, with a steady upward trend as the number of features increases. Even with a small number of features (e.g., fewer than 10), the proposed method already outperforms all baselines, highlighting its ability to identify highly informative and label-discriminative features early in the selection process.

To assess whether the observed MLDT performance differences among the compared feature selection methods are statistically significant across the seven datasets, we performed the Friedman–Nemenyi non-parametric statistical test at a significance level of α=0.05. The Friedman test yielded p-values of 0.0069, 0.0897, and 0.0094 for Hamming loss, Ranking loss, and Multi-label accuracy, respectively. These results indicate that, while Ranking loss exhibits only marginal significance, the differences in Hamming loss and Multi-label accuracy are statistically significant, suggesting that the compared methods yield meaningfully distinct performance under these measures. Fig. 2 presents the corresponding Critical Distance (CD) diagrams obtained from the Nemenyi post-hoc analysis. The diagrams visualize the average ranks of the six methods and the critical distance at which the rank differences become statistically significant [44]. As shown in the figure, the proposed method consistently achieves the best rank for all three evaluation metrics, demonstrating robust and stable performance. Complementary to these visual results, Table 11 reports the complete average-rank matrix for each method and metric. The proposed method attains the lowest average ranks (1.00, 1.86, and 1.0 for Hamming loss, Ranking loss, and Accuracy, respectively), confirming that its improvements are systematic rather than dataset-specific. Overall, these findings validate the statistical reliability of the proposed feature selection approach.

The computational efficiency of each feature selection method was evaluated in terms of total execution time, as summarized in Table 12. All experiments were conducted using MATLAB R2021b on a desktop equipped with an Intel Core i7-11700 CPU (2.5 GHz) and 32 GB of RAM. The maximum number of iterations for the proposed method was set to 100. The proposed method demonstrates competitive or superior computational efficiency compared to most baselines. In particular, it consistently outperforms AMI, MDMR, and MFSJMI, which incur substantial computational overhead due to repeated mutual information estimation or graph construction processes. While the proposed approach is slightly slower than FIMF-whose operations are relatively lightweight-it achieves significantly faster convergence than other information-theoretic methods such as QPFS and MFSJMI, especially on large-scale datasets like enron, genbase, and medical. These results indicate that the proposed optimization framework maintains high scalability without sacrificing selection quality, balancing effectiveness and computational cost efficiently across diverse multi-label datasets.

Figs. 3–5 illustrate the sensitivity of the proposed method to its three hyperparameters, α, β, and γ, across the cal500, emotions, and enron datasets, using multi-label accuracy with the MLkNN classifier as the evaluation metric. Each figure contains two subplots: (a) fixes α=0.1 and varies β and γ; (b) fixes β=0.1 and varies α and γ. All surfaces are presented in 3D to visualize the joint effect of the remaining two parameters on classification performance. Overall, the results demonstrate that the proposed method is highly robust to variations in hyperparameters, as long as their values are not excessively large or small. This indicates that the method does not require fine-grained hyperparameter tuning to achieve strong performance. In the Fig. 3, the accuracy remains consistently stable across all combinations of α, β, and γ, suggesting that the method is largely insensitive to hyperparameter changes on this dataset. In the Fig. 4, similar to cal500, the classification accuracy is stable across all hyperparameter settings, confirming the method’s consistent behavior regardless of parameter choice. In the Fig. 5, a minor performance decrease is observed when α=103, while the remaining parameter settings yield stable and strong results. This suggests that overly large values of α—which weights the Q term—may introduce redundancy in this dataset. Empirically, we find that effective values typically lie in the ranges α,β∈[10−3,101] and γ∈[10−2,102]. We therefore recommend performing grid search in a modest neighborhood around these intervals. For larger-scale problems or time-sensitive applications, alternative search strategies such as random search or Bayesian optimization may be employed to further reduce tuning overhead. These guidelines ensure a balance between computational cost and performance while maintaining robustness across datasets.

Fig. 6 displays the convergence behavior of the proposed method for all used datasets. The horizontal axis represents the number of iterations of the proposed algorithm, and the vertical axis shows the value of the objective function. The objective function value drops sharply within the first three iterations and appears to converge before the tenth iteration. This indicates that the proposed algorithm operates efficiently.