In this paper, we propose a hybrid method, namely recurrent neural networks with simulated annealing (RNN-SA) for vertical WS extrapolation. The RNN uses the Elman model where the hidden unit output is connected back to itself using an adjustable recurrent weight (w)in addition to feeding the output layer with weights (V), as shown in Fig. 3. Although the architecture is simple, this model is capable of performing a variety of prediction tasks [14,15]. The data is pre-processed in two stages. First, the data from the input (U) and recurrent connections are provided to the hidden units as inputs. Next, the output from the hidden units is sent to both the output layer and through the recurrent connections back to the hidden units. Mathematically, the output of the kth hidden unit hk(n) given the nth input vector is given by:

(1)hk(n)=σ(Ukx(n)+Wkhk(n−1)+bk)where b is the bias vector. The input x represents the actual and estimated WS from the lower heights. The output layer provides the WS value at one step higher. The output y(n) of the nth sample is given by:

(2)y(n)=σ(Vh(n)+by)

This study uses the tangent-hyperbolic activation function defined by:

(3)σ(z)=ez−e−zez+e−z

In this study, the numerical experiment is carried out by dividing the data into three parts, i.e., 70%, 10%, and 20% for training, validation, and testing, respectively. The RNN-SA is trained using the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm [16]. Since the model uses a medium number of units and layers, the BFGS algorithm is the most suitable for unconstrained optimization [17]. The BFGS algorithm is utilized to obtain the RNN model with minimum mean squared error (MSE):

(4)E=MSE=∑n=1N(yn−y^n)2N

It can be noticed that the MSE is a continuously differentiable function that represents the difference between the predicted outputs (y^) and the actual values (y). The BFGS algorithm uses an iterative update to obtain the weights that minimize the cost function e(w)=E from Eq. (4). The weight update is specified by:

(5)wk+1=wk+μkdkwhere μk and dk denote the step size and direction at iteration k. The direction update is given by:

(6){dk=−Bk−1∇e(wk)Bk+1=Bk−BkskskTBkSkTBksk+ykykTykTsk,B0=Isk=wk+1−wk,yk=∇e(wk+1)−∇e(wk)where Bk is an approximation of the Hessian matrix that represent the second derivative of the cost function. The results of the validation data showed that the RNN method with L=20 hidden units and number of iterations, K=20, exhibited the best performance. The model is further optimized using simulated annealing (SA), which is an optimization method that accepts not only better solution candidates but also includes not so good ones with a certain likelihood to enable the optimization to explore a new area for new candidates [18]. In the sense of the RNN, the SA is utilized further to optimize the trained RNN with respect to a non-differentiable cost function, namely, the mean absolute error (MAE):

(7)MAE=1N∑n=1N|yn−y^n|×100%

Given the cost function above, the solution candidate acceptance is given by:

(8)p=11+eΔEmax(T)where ΔE is the difference between new and old cost functions and T is the temperature governed by the following relation:

(9)T=T0×0.95kwhere the initial temperature T0 is set to 100 and k is the iteration number. The proposed RNN-SA utilizes two-stage optimizations. First, the BFGS method trains the RNN to minimize the MSE. The first cost function uses a smooth quadratic function that allows gradient-based optimization to obtain the best solution. Second, the SA optimizes the trained RNN weights to achieve the minimum MAE. Since the MAE uses a non-smooth absolute function, the SA offers an optimal solution for a complex recurrent model. The proposed two steps optimization approach to improve the performance metrics with different methods namely, the MSE with continuous derivative and the non-smooth MAE. The traditional RNN is typically trained using gradient based method with differentiable cost function. Whereas, the proposed RNN, in addition to the gradient based method, is further optimized using SA to enhance the performance. Both RNNs use the same architecture with recurrent units as shown in Fig. 4. In the implementation, the maximum number of iterations of the SA method is K=10.

In this study, the multilayer perceptron (MLP) is used as a benchmark for the comparison of the performance of RNN-SA. MLP has a much simpler structure than RNN [19], as shown in Fig. 4. In contrast to RNN, MLP uses a feed-forward structure that sequentially processes the data from the input layer to the output layer. Mathematically, the output of the kth hidden unit hk(n) given the nth input vector is obtained by:

(10)hk(n)=τ(Whx(n)+bk)where Vk is the input weight matrix. Given the hidden unit values, the output of the MLP is defined by:(11)y(n)=Woh(n)+b

This study utilizes gradient descent with momentum and adaptive learning rate (μ) backpropagation to train the MLP. This method is able to train any model with differentiable activation and cost function, which is suitable in this case where the activation and cost function uses differentiable tangent hyperbolic and MSE, respectively. Each weight is adjusted according to gradient descent defined by:(12)wk=wk−1+∂wkwhere the weight update (∂wn) is calculated by:

(13)∂wk=m∂wk−1+mμ(∂ek)∂wkwhere m=0.9 and μ=0.01 denote the momentum coefficient and learning rate, respectively [20]. The cost function is evaluated at each epoch. If the cost function decreases toward the cost target, then the learning rate is increased by the factor Δμ(+)=1.05. If the cost function improves by more than the factor ΔMSEmax=1.04, the weight update that increased the cost function is canceled, and the learning rate is updated by the factor Δμ(−)=0.7. The experiments use H=20 hidden units. Training halts when the maximum number of iterations (K=1200) is exceeded.

The last benchmarking method used in this paper is the support vector machine (SVM) for regression [21]. Theoretically, SVM is represented by a mapping function {(xi,yi)}i=1N, with input vectors xi∈RM and output targets yi∈R. The SVM attempts to create the relationship between xi and expected results f(xi) as follows:

(14)y=f(xi)=(ω⋅xi)+bwhere ω∈RM and b are the weight vectors and threshold, respectively. From this model, the target of SVM is to obtain the weights that minimize the following error function:

(15)E(ω)=1N∑i=1N|yi−f(xi)|ϵ+λ|ω|2where λ is a constant and the robust error function |z|ϵ is defined as follows:

(16)|z|ϵ={0,if|z|<ϵ|z|,otherwise.

The SVM model is trained using sequential minimal optimization (SMO), where the number of inputs determines the dimension of ω. Since the model has four input elements (10, 20, 30, and 40 m), the model for estimating WS at 50 m uses M=4.

Recently, state of the art methods like deep learning have been used for many tasks including wind speed prediction. For instance, convolutional neural networks (CNN) is used for WS prediction in a wind farm in Hebei Province, China with high accuracy [22]. In addition to that, long short-term memory (LSTM) network is used for WS prediction on a wind farm in Inner Mongolia [23]. These two methods are also used as benchmarking methods in this paper.

This study employs four error measures based on the difference between the measured values (y), the estimated outputs (y^), and the average of the measured values (y¯), including MSE, mean bias error (MBE), mean absolute percent error (MAPE), and the coefficient of determination (R2). These error measures are calculated using the following equations:

(17)MBE=∑n=1N(y^n−yn)N

(18)MAPE=1N∑n=1N|yn−y^nyn|×100\% 

(19)R2=(1−∑n=1N(yn−y¯)2∑n=1N(yn−y^n)2)×100\% 

Due to lack of resources in some countries, WS is typically measured up to a height of 40 m and its extrapolation to higher levels require sophisticated methods. Therefore, the WS at the higher heights is obtained using models, given the WS measured at lower heights. As shown in Fig. 1, the WS increases with height but in a random fashion [24,25]. This paper uses WS data acquired in Dhahran, Saudi Arabia during 20 June 2015 to 29 February 2016 using a LiDAR wind measurement device at 10, 20, 30, …, 180 m heights. The WS is scanned every three second and averages at each 10 min are saved and used. The data is divided into training, validation, and testing at 70%, 10%, and 20%, respectively. The LiDAR based measured WS is used as the reference for the actual WS.

Table 1 shows the performance of the RNN-SA under validation data for WS estimation at 50–180 m heights based on measured WS at 10–40 m heights. The model with the best performance is further used for testing. Table 2 summarizes the testing data experimental results of the RNN-SA, standard RNN, MLP, SVM, CNN, and LSTM techniques in terms of MSE. It is obvious that MSE values for RNN-SA increases progressively from 0.027 to 2.089 while that for MLP from 0.049 to 2.367, corresponding to 50 to 180 m heights. Similarly, for the SVM method, the MSE values varied from 0.032 at 50 to 2.632 at 180 m. In all the numerical experiments, that RNN-SA method resulted in lowest values of MSE at all heights. The developed method is able to achieve better MSE than the standard RNN for all heights. It also can be noticed that the RNN-SA outperformed other methods based on the obtained MSE values. Table 3 reports the MBE of all methods and heights. MBE may not quantify the performance or accuracy of the estimation methods; however, it indicates whether the method is over or under predictive [26]. Positive MBE represents over prediction where the estimated values are larger than the actual one. Whereas, negative MBE indicates under-prediction where the estimated values are less than the actual. The experimental results show that all methods have negative values for all the heights. This indicate that the predicted values are less than the measured WS or the model underestimated the WS at all heights.

Fig. 5 compares the MAPE values for all the methods and show almost the same pattern as in case of MSE (Table 1). The MAPE value of RNN-SA, standard RNN, MLP, and SVM increased from 2.2% to 11.2%, 2.3% to 11.3%, 2.9% to 11.7%, and 2.4% to 12.2% for 50 to 180 m of heights, respectively. The proposed method achieves the least MAPE over all methods for all the heights. RNN methods proved to be the next best estimator of WS at all heights while MLP the third best option. Fig. 6 visualizes the R2 values for all heights and methods. The figure shows that the estimated values deviate more and more from the actual values as height increases. It may be accounted to the fact that actual WS values at 10–40 m only are being used for the estimation at higher heights. So, the estimated WS near the 40 m height has higher accuracy, and as the height increases, the correlation between the estimated and the actual WS decreases. The RNN-SA outperforms the other methods and achieved the highest R2 values for all heights.

The scatter plots between the estimated RNN-SA with actual WS values at 60, 100, 140, and180 m are shown in Figs. 7a–7d, respectively. It is observed that the scatter plots start to diverge as height increases. This is confirmed by the temporal variation of actual and extrapolated WS values at 60, 100, 140, and 180 m heights, shown in Figs. 8a–8d, respectively. It is noticed that the estimated WS values follow the temporal trend of actual values at all heights, as depicted in Fig. 8. The figure also shows the coefficient of determination values for all methods. However, the trends of estimated WS at 180 (Fig. 8d) m are away from the actual values compared to the trends at 60 m (Fig. 8a), 100 m (Fig. 8b), and 140 m (Fig. 8c). The maximum number of estimated WS values for training contribute to these deviations.

The estimated WS profiles along with the actual ones up to 180 m, are compared in Fig. 9. Each line in Figs. 9a–9e represents an individual WS value at a particular time sample. It is evident from the figure that the estimated WSs at lower heights are very close to the actual values. In contrast, the estimated WS higher heights tend to deviate from the actual. Fig. 9f represents the average of WS for all time samples. The average estimated WS for all methods confirm the observation in Table 2, where the average of estimated WS is less than the actual one.

Finally, the training duration statistics is shown in Fig. 10. It can be noticed that the proposed training method takes a bit more extra training time as indicated by higher training duration. Maximum training duration of the proposed and traditional RNN approaches are 47.77 and 36.94 s, respectively. Despite the relative high difference of the maximum duration (10.8 s), the average training duration of the proposed and traditional RNN methods are 32.39 and 31.62 s, respectively. In other words, the additional step consumes a relatively small-time duration since the average training time difference is only 0.77 s. The extra training duration is compensated by the performance improvement, as shown by the data in Tables 2 and 3.