In this study, we trained and evaluated our model on the benchmark datasets containing a total of 11 training and 11 testing datasets from human, mouse and rat, which were also used in iRNA-m6A and im6A-TS-CNN models (Dao et al., 2020; Liu et al., 2020). Each dataset contains the same number of positive and negative samples. Each sample is a 41nt-length RNA sequence with Adenine in the center. Detailed information about these datasets can be found in the work of Dao et al. (2020).

It is vital to extract efficacious features when developing new computational model based on machine or deep learning algorithms (Zhang and Liu, 2019). In this study, we extracted three categories of features from the sequence: one-hot encoding, sequence features, and order features.

Given a DNA sequence D, its intuitive expression is

(1) D=R1R2R3R4R5R6R7⋯RL

where RI represents the i -th nucleic acid residue at position i in the DNA sequence.

Each sequence with 41 nt is represented with a 41 × 41 vector, in which (1, 0, 0, 0) stands for G, (0, 1, 0, 0) stands for C, (0, 0, 1, 0) stands for U, and (0, 0, 0, 1) stands for A.

Transforming a DNA sequence sample into a vector based on its sequence characteristic composition is a simple but universal strategy, which can capture significant biological information (Zhen et al., 2020; Zou et al., 2019). iLearn is a comprehensive and versatile Python-based toolkit including a variety of descriptors for DNA, RNA and proteins (Zhen et al., 2020). We used iLearn to calculate and extract four types of features: nucleic acid composition, binary electron-ion interaction pseudopotentials, autocorrelation and cross-covariance, pseudo nucleic acid composition, and achieved a total of 1325 features. The names and dimensions of features used in this section are listed in Table 1. As for the specific definitions of these features, please refer to (Zhen et al., 2020; Zou et al., 2019).

KNFP integrates the information from K-mer as well as one-hot encoding, and can compensate for insufficient short-range or local sequence order information effectively (Yang et al., 2020). It has been used to identify protein-RNA binding sites and protein-circRNA interaction sites (Yang et al., 2020). K-mer can map any DNA sequence to a vector with 4k dimensions as follows:

(2) R=[φ1φ2⋯φu⋯φ4k]T

where φu(u=1,2,⋯,4k) is the frequency of the u-th k-mer along the sequence (k = 1,2,3 in this work).

On one hand, R can be transformed to a diagonal matrix R_D, if multiplied by the identity matrix. On the other hand, for a DNA sequence D of length L, the number of k-mer is L − k + 1. Each k-mer can be encoded as a one-hot vector with dimension of 4k . The product of the (L−k+1)∗4k matrix (M) and R_D is KNFP.

For example, given a sequence S = ‘ACGACGAA’, 1-mer is encoded as one-hot vectors: G = (1, 0, 0, 0), C = (0, 1, 0, 0), U = (0, 0, 1, 0), and A = (0, 0, 0, 1). Then according to the position information, S can be transformed to a matrix as follows:

M=(00010100100000010100100000010001)

For S, the frequency vector of 1-mer (G, C, U, A) is R = (0.25, 0.25, 0, 0.5). Then through multiplying R by an identity matrix, it is converted to a diagonal matrix as follows:

RD=(0.2500000.250000000000.5)

The KNFP =M×RD is given as

(00010100100000010100100000010001)∗(0.2500000.250000000000.5)=(0000.500.25000.250000000.500.25000.250000000.50000.5)

Experiments have shown that excessive feature information can interfere with the performance of classifiers. Therefore, feature selection methods should be applied to find the most informative features for training classifiers and thus reduce the dimensionality of the feature vector. F-score has been extensively applied in bioinformatics because of its effectiveness in balancing accuracy and stability (Bui et al., 2016; Li et al., 2018).

The F-score of the j-th feature is defined as

(3) F−score(j)=(x¯j(+)−x¯j)2+(x¯j(−)−x¯j)21m+−1∑k=1m+(x¯k,j(+)−x¯j(+))2+1m−−1∑k=1m−(x¯k,j(−)−x¯j(−))2,

where x¯j,x¯j(+) and x¯j(−) denote average values of the j-th feature in the combined positive and negative training datasets, the positive datasets, and the negative datasets, respectively. m+ is the total number of positive samples; m− is the total number of negative samples; x¯k,j(+) represents the j-th feature of the k-th positive sample and x¯k,j(−) represents the j-th feature of the k-th negative sample. The higher the F-score is, the more useful the corresponding feature is for the classification.

The model architecture consisted of a multi-channel CNN, a capsule network and a BiGRU network. Each of these three networks has, respectively, been shown to be effective in object detection (Li et al., 2018), protein post-translational modification site prediction (Wang et al., 2019) and social bots detection (Wu et al., 2020). The structure of the multi-input hybrid neural network is shown in Fig. 2.

The input of the multi-input hybrid neural network is the one-hot encoding, sequence features and KNFP, respectively. For each type of features, we applied 32 convolution filters and performed batch normalization to readjust the data distribution. The input of a batch in the neural network is X=[x1,x2,⋯,xn] , where xi represents a sample and n is the batch size.

The mean value of elements in each batch is obtained by:

(4) μB=1n∑i=1nxi

Then, the variance of a batch is calculated by

(5) σB2=1n∑i=1n(xi−μB)2

This allows us to normalize each element:

(6) xi′=xi−μBσB2+ϵ

The final output of the network is given by

(7) yi=γi⋅xi′+βi

where ϵ is a small positive number used to prevent the denominator from being 0.

Finally, we merge three outputs using the Swish activation function σ , which is defined as follows:

(8) σ(x)=1/(1+exp⁡(−x))

Because of the limited size of our datasets, we add a 1 × 1 multi-channel CNN to enhance representational capabilities of the model. The 1 × 1 convolution is used to maintain the size of the feature map and integrate the information by linearly weighting the input feature map of each channel. With additional layers of such 1 × 1 convolution, the final extracted features would become more concise.

The capsule network (CapsNet) was proposed by Sabour et al. (2017) and applied in stance detection (Zhao and Yang, 2020), image recognition (Qian et al., 2020) and automated classification (Mobiny et al., 2019). Since the capsule network collects location information, it can learn a good representation from a small amount of data. We use the capsule network and focus on the hierarchical relationship of local features. The output of the multi-channel CNN is adopted as the input vector of the capsule network. We make an affine transformation of the input vector as follows:

(9) u^j|i=Wijui,

where Wij is the weight matrix that needs to be trained and uI is the input vector of the capsule neural network.

Next, the weighted sum is applied to all the prediction vectors as follows:

(10) sj=∑iciju^j|i,

where cij is the coupling coefficient in the dynamic routing process.

Finally, we obtain output vectors through a non-linear activation function as follows:

(11) vj=‖sj‖21+‖sj‖2sj‖sj‖.

The third segment of this model is the BiGRU network, which helps to extract deep-level features of sequences. The current hidden layer state of the BiGRU is determined by three factors: the current input xt , the output of the forward hidden layer state at time-step (t–1) ht−1→ and the output of the reverse hidden layer state ht−1← . BiGRU can be regarded as two GRUs, so the state of hidden layer ht at time-step t can be obtained by the weighted sum of the forward hidden layer state ht−1→ and reverse hidden layer state ht−1← :

(12) ht→=GRU(xt,ht−1→)

(13) ht→= GRU(xt,ht−1←)

(14) ht=wt⋅ht→+vt⋅h←t+bt,

where GRU () represents a nonlinear transformation of the input word vector; wt and vt are the weighed matrices; and bt is the bias term.

Considering the number of datasets and the precision of the model, three feature maps were obtained from batch normalization and 1D convolution with 32 filters (kernel size = 3). The multi-channel CNN contained three 1 × 1 convolution layers and took the Swish as activation function. Considering time cost, we only employed one capsule layer with 14 num_capsule (dim_capsule = 41, routings = 3). The BiGRU had 32 hidden units followed by a fully connected layer and used the ReLU activation function. We also used dropout with a keep probability of 0.3 to prevent the model from over fitting. For stochastic gradient descent, we selected the Adam optimization algorithm. The entire program was written in Python 3.6.

To evaluate the performance of our prediction model, we used four measurements including accuracy (Acc), sensitivity (Sn), specificity (Sp), and Matthew’s correlation coefficient (MCC) on 5-fold cross-validation and independent dataset tests. The formulas are provided as follows:

(15) {Sp=TNTN+FPSn=TPFN+TPAcc=TP+TNTP+TN+FN+TPMCC=TP×TN−FP×FN(TP+FN)(FP+TN)(TP+FP)(FN+TN)

where TP, TN, FP, and FN represent the number of true positives, true negatives, false positives, and false negatives, respectively.

If too many features are extracted, the generalizability of the model will be weakened. Thus it is important to determine the appropriate step size for feature selection. As the dimension of the input matrix was 41 × N, we chose the step size as 41 and evaluated the performance of our model with feature matrices of different dimensions (41 × N, N∈(5,6,7,⋯,24,25) ) on 5-fold cross-validation tests successively. To reduce the deviation caused by the fluctuation of the neural network, we ran each test three times for the 11 datasets from the brain, liver, kidney, heart, and testis of Homo sapiens, Musmusculus, and Rattusnorvegicus. The optimal feature subset was finalized according to the average accuracy. The detailed feature selection results are shown in Fig. 5. As should be noticed, dimensions of optimal features were various for different datasets; their corresponding dimensions are listed in Table 2. These results indicate that it is necessary to establish a specialized model for each tissue type in each species.

Generally, there are two ways to select parameters, i.e., empirical choice and Bayesian optimization. With Homo sapiens as an example, we tried both methods to find the most suitable parameters. The initial parameters were set based on a previous work to compare the prediction results roughly. Then, if the prediction model was under fitting, we attempted to add more convolution kernels and neurons; otherwise if the prediction model was over fitting, we attempted to reduce the number of convolution kernels and neurons. In addition, batch normalization, dropout, and regularization were introduced to avoid over fitting during the optimizing process. Alternatively, Bayesian optimization (Snoek et al., 2012) was used to tune the key parameters including batch size, dropout rate, filter1, filter2, pool_size and etc. Finally, the optimal parameters wereselected according to the AUC value. The Baysian optimization of parameters in models for Homo sapiens and corresponding AUC values are provided in Table S4 and Fig. 6A. We compared the best results obtained by the two methods and concluded the result given by empirical adjustment parameters showed more advantages (Fig. 6B). Thus, for Musmusculus and Rattusnorvegicus, we used the empirical method to determine the parameters in neural network.

To verify the effectiveness of hybrid neural network, we compared its prediction performance with several traditional classification algorithms including logistic regression, decision tree, support vector machine (SVM), random forest (RF), gradient boosting decision tree (GBDT), extreme gradient boosting (XGBoost) and light gradient boosting machine (LightGBM) for Homo sapiens. The average accuracies of 5-fold cross-validation tests obtained by the seven algorithms are listed in Table 3. On three datasets of Homo sapiens, our model achieves the mean accuracy of 74.29%, 1.28% higher than the second best algorithm, LightGBM, which demonstrates that hybrid neural network is capable of improving the recognition performance for m6A sites of various tissues in different species.

To observe the effectiveness of extracted features intuitively, we applied t-distributed stochastic neighbor embedding (t-SNE) to visualize the feature representations. We took the brain tissue as an example and demonstrated the features after mapped through the concatenate layer and the attention layer. Each dot in Fig. 6C represents a sample, with purple dots representing m6A sites and yellow dots representing non-m6A sites. The overlapping of the two sample types on the left side of Fig. 6C indicates that it is difficult to distinguish m6A sites from non-m6A sites. However, the features were relatively separated, as shown on the right side of Fig. 6C, after selected and processed by the deep hierarchical network, which was, multi-channel CNN, capsule network and BiGRU network with the self-attention mechanism. The t-SNE plots indicated that the hybrid deep hierarchical networks could learn sequence motif information from selected features. But there are still some overlaps between the two types, indicating that our model is not completely specific for all m6A sites. This fact is consistent with the need to establish specialized models for each tissue type.