To integrate efficiently and accurately the element matrices in a general finite element mesh of 3-D hexahedral elements (including distorted elements), the power of Deep Learning was harnessed in two applications of feedforward MultiLayer Neural networks (MLN,³⁵35

MLN is also called MultiLayer Perceptron (MLP); see Footnote 32.

To train³⁷37

See Section 6 on “Network training, optimization methods”.

While Application 1.1 used one fully-connected (Section 4.6.1) feedforward neural network (Section 4), Application 1.2 relied on two neural networks: The first neural network was a classifier that took the element shape (18 normalized nodal coordinates) as input and estimated whether or not the numerical integration (quadrature) could be improved by adjusting the quadrature weights for the given element (one output), i.e., the network classifier only produced two outcomes, yes or no. If an error reduction was possible, a second neural network performed regression to predict the corrected quadrature weights (eight outputs for 2×2×2 quadrature) from the input element shape (usually distorted).

To train the classifier network, 10,000 element shapes were selected from the prepared dataset of 20,000 hexahedrals, which were divided into a training set and a validation set (Section 6.1) of 5000 elements each.³⁸38

For the definition of training set and test set, see Section 6.1. Briefly, the training set is used to optimize the network parameters, while the test set is used to see how good the network with these optimized parameters can predict the targets of never-seen-before inputs.

To train the second regression network, 10,000 element shapes were selected for which quadrature could be improved by adjusting the quadrature weights [38].

Again, the training set and the test set comprised 5000 elements each. The parameters of the neural networks (weights, biases, Figure 8, Section 4.4) were optimized (trained) using a gradient descent method (Section 6) that minimizes a loss function (Section 5.1), whose gradients with respect to the parameters are computed using backpropagation (Section 5).

The best results were obtained from a classifier with four hidden layers (Figure 7, Section 4.3, Remark 4.2) with 30 neurons (Figure 8, Figure 36, Section 4.4.3) each and a regression network that had a depth of five hidden layers, where each layer was 50 neurons wide, Figure 7. The results were obtained using the logistic sigmoid function (Figure 30) as activation function (Section 4.4.2) due to existing software, even though the rectified linear function (Figure 24) were more efficient, but yielded comparable accuracy on a few test cases.³⁹39

Information provided by author A. Oishi of [38] through a private communication to the authors on 2018 Nov 16.

To quantify the effectiveness of the approach in [38], an error-reduction ratio was introduced, i.e., the quotient of the quadrature error with quadrature weights predicted by the neural network and the error obtained with the standard quadrature weights of Gauss-Legendre quadrature with 2×2×2 integration points; see Eq. (402) in Section 10.3 with q=2 and “opt” stands for “optimized” (or “predicted”). When the error-reduction ratio is less than 1, the integration using the predicted quadrature weights is more accurate than that using the standard quadrature weights. To compute the two quadrature errors mentioned above (one for the predicted quadrature weights and one for the standard quadrature weights, both for the same 2×2×2 integration points), the reference values considered as most accurate were obtained using 30×30×30 integration points with the standard quadrature quadrature weights; see Eq. (401) in Section 10.2 with qmax=30.

For most element shapes of both the training set (a) and the test set (b), each of which comprised 5000 elements, the blue bars in Figure 10 indicate an error ratio below one, i.e., the quadrature weight correction effectively improved the accuracy of numerical quadrature.

Readers familiar with Deep Learning and neural networks can go directly to Section 10, where the details of the formulations in [38] are presented. Other sections are also of interest such as classic and state-of-the-art optimization methods in Section 6, attention and transformer unit in Section 7, historical perspective in Section 13, limitations and danger of AI in Section 14.

Readers not familiar with Deep Learning and neural networks will find below a list of the concepts that will be explained in subsequent sections. To facilitate the reading, we also provide the section number (and the link to jump to) for each concept.

Deep-learning concepts to explain and explore:

This list is continued further below in Section 2.3.2. Details of the formulation in [38] are discussed in Section 10.

One way that deep learning can be used in solid mechanics is to model complex, nonlinear constitutive behavior of materials. In single physics, balance of linear momentum and strain-displacement relation are considered as definitions or “universal principles”, leaving the constitutive law, or stress-strain relation, to a large number of models that have limitations, no matter how advanced [91]. Deep learning can help model complex constitutive behaviors in ways that traditional phenomenological models could not; see Figure 105.

Deep recurrent neural networks (RNNs) (Section 7.1) was used as a scale-bridging method to efficiently simulate multiscale problems in hydromechanics, specifically plasticity in porous media with dual porosity and dual permeability [25].⁴⁰40

Porosity is the ratio of void volume over total volume. Permeability is a scaling factor, which when multiplied by the negative of the pressure gradient, and divided by the fluid dynamic viscosity, gives the fluid velocity in Darcy’s law, Eq. (409). The expression “dual-porosity dual-permeability poromechanics problem” used in [25], p. 340, could confuse first-time readers—especially those who are familiar with traditional reservoir simulation, e.g., in [92]—since dual porosity (also called “double porosity” in [89]) and dual permeability are two different models of naturally-fractured porous media; these two models for radionuclide transport around nuclear waste repository were studied in [93]. Further added to the confusion is that the dual-porosity model is more precisely called dual-porosity-single permeability model, whereas the dual-permeability model is called dual-porosity dual-permeability model [94], which has a different meaning than the one used in [25].

The dual-porosity single-permeability (DPSP) model was first introduced for use in oil-reservoir simulation [89], Figure 11, where the fracture system was the main flow path for the fluid (e.g., two phase oil-water mixture, one-phase oil-solvent mixture). Fluid exchange is permitted between the rock matrix and the fracture system, but not between the matrix blocks. In the DPSP model, the fracture system and the rock matrix, each has its own porosity, with values not differing from each other by a large factor. On the contrary, the permeability of the fracture system is much larger than that in the rock matrix, and thus the system is considered as having only a single permeability. When the permeability of the fracture system and that of the rock matrix do not differ by a large factor, then both permeabilities are included in the more general dual-porosity dual-permeability (DPDP) model [94].

Since 60% of the world’s oil reserve and 40% of the world’s gas reserve are held in carbonate rocks, there has been a clear interest in developing an understanding of the mechanical behavior of carbonate rocks such as limestones, having from lowest porosity (Solenhofen at 3%) to high porosity (e.g., Majella at 30%). Chalk (Lixhe) is a carbonate rock with highest porosity at 42.8%. Carbonate rock reservoirs are also considered to store carbon dioxide and nuclear waste [95] [93].

In oil-reservoir simulations in which the primary interest is the flow of oil, water, and solvent, the porosity (and pore size) within each domain (rock matrix or fracture system) is treated as constant and homogeneous [94] [96].⁴¹41

See, e.g., [94], p. 295, Chap. 9 on “Advanced Topics: Fluid Flow in Fractured Reservoirs and Compositional Simulation”.

Moreover, pores have different sizes, and can be classified into different pore sub-systems. For the Majella limestone in Figure 12 with total porosity at 30%, its pore space can be partitioned into two subsystems (and thus dual porosity), the macropores with macroporosity at 11.4% and the micropores with microporosity at 19.6%. Thus the meaning of dual-porosity as used in [25] is different from that in oil-reservoir simulation. Also characteristic of porous rocks such as the Majella limestone is the non-linear stress-strain relation observed in experiments, Figure 13, due the changing size, and collapse, of the pores.

Likewise, the meaning of “dual permeability” is different in [25] in the sense that “one does not seek to obtain a single effective permeability for the entire pore space”. Even though it was not explicitly spelled out,⁴²42

At least at the beginning of Section 2 in [25].

In the problem investigated in [25], the presence of localized discontinuities demands three scales—microscale (μ), mesoscale (cm), macroscale (km)—to be considered in the modeling, see Figure 14. Classical approaches to consistently integrate microstructural properties into macroscopic constitutive laws relied on hierarchical simulation models and homogenization methods (e.g., discrete element method (DEM)–FEM coupling, FEM2). If more than two scales were to be considered, the computational complexity would become prohibitively, if not intractably, large.

Instead of coupling multiple simulation models online, two (adjacent) scales were linked by a neural network that was trained offline using data generated by simulations on the smaller scale [25]. The trained network subsequently served as a surrogate model in online simulations on the larger scale. With three scales being considered, two recurrent neural networks (RNNs) with Long Short-Term Memory (LSTM) units were employed consecutively:

(1) Mesoscale RNN with LSTM units: On the microscopic scale, a representative volume element (RVE) was an assembly of discrete-element particles, subjected to large variety of representative loading paths to generate training data for the supervised learning of the mesoscale RNN with LSTM units, a neural network that was referred to as “Mesoscale data-driven constitutive model” [25] (Figure 14). Homogenizing the results of DEM-flow model provided constitutive equations for the traction-separation law and the evolution of anisotropic permeabilities in damaged regions.

Path-dependence is a common characteristic feature of the constitutive models that are often realized as neural networks; see, e.g., [23]. For this reason, it was decided to employ RNN with LSTM units, which could mimick internal variables and corresponding evolution equations that were intrinsic to path-dependent material behavior [25]. These authors chose to use a neural network that had a depth of two hidden layers with 80 LSTM units per layer, and that had proved to be a good compromise of performance and training efforts. After each hidden layer, a dropout layer with a dropout rate 0.2 were introduced to reduce overfitting on noisy data, but yielded minor effects, as reported in [25]. The output layer was a fully-connected layer with a logistic sigmoid as activation function.

An important observation is that including micro-structural data—the porosity ϕ, the coordination number CN (number of contact points, Figure 16), the fabric tensor (defined based on the normals at the contact points, Eq. (404) in Section 11; Figure 16 provides a visualization)—as network inputs significantly improved the prediction capability of the neural network. Such improvement is not surprising since soil fabric—described by scalars (porosity, coordination number, particle size) and vectors (fabric tensors, particle orientation, branch vectors)—exerts great influence on soil behavior [99]. Coordination number⁴⁴44

The coordination number (Wikipedia version 20:43, 28 July 2020) is a concept originated from chemistry, signifying the number of bonds from the surrounding atoms to a central atom. In Figure 16 (a), the uranium borohydride U(BH₄)₄ complex (Wikipedia version 08:38, 12 March 2019) has 12 hydrogen atoms bonded to the central uranium atom.

Figure 17 illustrates the importance of incorporating microstructure data, particularly the fabric tensor, in network training to improve prediction accuracy.

Deep-learning concepts to explain and explore: (continued from above in Section 2.3.1)

Details of the formulation in [25] are discussed in Section 11.

The accurate simulation of turbulence in fluid flows ranks among the most demanding tasks in computational mechanics. Owing to both the spatial and the temporal resolution, transient analysis of turbulence by means of high-fidelity methods such as Large Eddy Simulation (LES) or direct numerical simulation (DNS) involves millions of unknowns even for simple domains.

To simulate complex geometries over larger time periods, reduced-order models (ROMs) that can capture the key features of turbulent flows within a low-dimensional approximation space need to be resorted to. Proper Orthogonal Decomposition (POD) is a common data-driven approach to construct an orthogonal basis {ϕ1(x),ϕ2(x),…ϕ∞(x)} from high-resolution data obtained from high-fidelity models or measurements, in which x is a point in a 3-D fluid domain ℬ; see Section 12.1. A flow dynamic quantity u(x,t), such as a component of the flow velocity field, can be projected on the POD basis by separation of variables as (Figure 18)

(3)u(x,t)=∑i=1i=∞ϕi(x)αi(t)≈∑i=1i=kϕi(x)αi(t), with k<∞,

where k is a finite number, which could be large, and αi(t) a time-dependent coefficient for ϕi(x). The computation would be more efficient if a much smaller subset with, say, m≪k, POD basis functions,

(4)u(x,t)≈∑j=1j=mϕij(x)αij(t), with m≪k and ij∈{1,…,k},

where {ij,j=1,…,m} is a subset of indices in the set {1,…,k}, and such that the approximation in Eq. (4) is with minimum error compared to the approximation in Eq. (3)2 using the much larger set of k POD basis functions.

In a Galerkin-Project (GP) approach to reduced-order model, a small subset of dominant modes form a basis onto which high-dimensional differential equations are projected to obtain a set of lower-dimensional differential equations for cost-efficient computational analysis.

Instead of using GP, RNNs (Recurrent Neural Networks) were used in [26] to predict the evolution of fluid flows, specifically the coefficients of the dominant POD modes, rather than solving differential equations. For this purpose, their LSTM-ROM (Long Short-Term Memory - Reduced Order Model) approach combined concepts of ROM based on POD with deep-learning neural networks using either the original LSTM units, Figure 117 (left) [24], or the bidirectional LSTM (BiLSTM), Figure 117 (right) [104], the internal states of which were well-suited for the modeling of dynamical systems.

To obtain training/testing data, which were crucial to train/test neural networks, the data from transient 3-D Direct Navier-Stokes (DNS) simulations of two physical problems, as provided by the Johns Hopkins turbulence database [105] were used [26]: (1) The Forced Isotropic Turbulence (ISO) and (2) The Magnetohydrodynamic Turbulence (MHD).

To generate training data for LSTM/BiLSTM networks, the 3-D turbulent fluid flow domain of each physical problem was decomposed into five equidistant 2-D planes (slices), with one additional equidistant 2-D plane served to generate testing data (Section 12, Figure 116, Remark 12.1). For the same subregion in each of those 2-D planes, POD was applied on the k snapshots of the velocity field (k=5,023 for ISO, k=1,024 for MHD, Section 12.1), and out of k POD modes {ϕi(x), i=1,…,k}, the five (m=5≪k) most dominant POD modes {ϕi(x), i=1,…,m} representative of the flow dynamics (Figure 18) were retained to form a reduced-order basis onto which the velocity field was projected. The coefficient αi(t) of the POD mode ϕi(x) represented the evolution of the participation of that mode in the velocity field, and was decomposed into thousands of small samples using a moving window. The first half of each sample was used as input signal to an LSTM network, whereas the second half of the sample was used as output signal for supervised training of the network. Two different methods were proposed [26]:

For both methods, variants with the original LSTM units or the BiLSTM units were implemented. Each of the employed RNN had a single hidden layer.

Demonstrative results for the prediction capabilities of both the original LSTM and the BiLSTM networks are illustrated in Figure 20. Contrary to the authors’ expectation, networks with the original LSTM units performed better than those using BiLSTM units in both physical problems of isotropic turbulence (ISO) (Figure 20a) and magnetohydrodyanmics (MHD) (Figure 20b) [26].

Details of the formulation in [26] are discussed in Section 12.

The top-down approach starts by giving up-front the mathematical big picture of what a neural network is, with the big-picture (high level) graphical representation, then gradually goes down to the detailed specifics of a processing unit (often referred to as an artificial neuron) and its low-level graphical representation. A definite advantage of this top-down approach is that readers new to the field immediately have the big picture in mind, before going down to the nitty-gritty details, and thus tend not to get lost. An excellent reference for the top-down approach is [78], and there are not many such references.

Specifically, for a multilayer feedforward network, by top-down, we mean starting from a general description in Eq. (18) and going down to the detailed construct of a neuron through a weighted sum with bias in Eq. (26) and then a nonlinear activation function in Eq. (35).

In terms of block diagrams, we begin our top-down descent from the big picture of the overall multilayer neural network with L layers in Figure 23, through Figure 34 for a typical layer (ℓ) and Figure 35 for the lower-level details of layer (ℓ), then down to the most basic level, a neuron in Figure 36 as one row in layer (ℓ) in Figure 35, the equivalent figure of Figure 8, which in turn was the starting point in [38].

The bottom-up approach typically starts with a biological neuron (see Figure 131 in Section 13.1 below), then introduces an artificial neuron that looks similar to the biological neuron (compare Figure 8 to Figure 131), with multiple inputs and a single output, which becomes an input to each of a multitude of other artificial neurons; see, e.g., [23] [21] [38] [20].⁴⁹49

Figure2 in[20] is essentially the same as Figure 8.

Unfamiliar readers when looking at the graphical representation of an artificial neural network (see, e.g., Figure 7) could be misled in thinking in terms of electrical (or fluid-flow) networks, in which Kirchhoff’s law applies at the junction where the output is split into different directions to go toward other artificial neurons. The big picture is not clear at the outset, and could be confusing to readers new to the field, who would take some time to understand; see also Footnote 5. By contrast, Figure 23 clearly shows a multilevel function composition, assuming that first-time learners are familiar with this basic mathematical concept.

A function can be graphically represented as in Figure 22.

The multiple levels of compositions in Eq. (18) can then be represented by

(23)x=y(0)︸Input→f(1)y(1)→f(2)⋯y(ℓ−1)→f(ℓ)y(ℓ)⋯→f(L−1)y(L−1)→f(L)︸Network as multilevel composition of functionsy(L)=y˜︸Output

revealing the structure of the feedforward network as a multilevel composition of functions (or chain-based network) in which the output y(ℓ−1) of the previous layer (ℓ−1) serves as the input for the current layer (ℓ), to be processed by the function f(ℓ) to produce the output y(ℓ). the input x=y(0) for the input layer (1) is the input for the entire network. The output y(L)=y˜ of the (last) layer (L) is the predicted output for the entire network.

First, an affine transformation on the inputs (see Eq. (26)) is carried out, in which the coefficients of the inputs are called the weights, and the constants are called the biases. The output y(ℓ−1) of layer (ℓ−1) is the input to layer (ℓ)

(24)y(ℓ−1)=[y1(ℓ−1),⋯,ym(ℓ−1)(ℓ−1)]T∈Rm(ℓ−1)×1.

The column matrix z(ℓ)

(25)z(ℓ)=[z1(ℓ),⋯,zm(ℓ)(ℓ)]T∈Rm(ℓ)×1

is a linear combination of the inputs in y(ℓ−1) plus the biases (i.e., an affine transformation)⁶²62

See Eq. (497) for the continuous temporal summation, counterpart of the discrete spatial summation in Eq. (26).

(26)z(ℓ)=W(ℓ)y(ℓ−1)+b(ℓ) such that zi(ℓ)=wi(ℓ)y(ℓ−1)+bi(ℓ), for i=1,…,m(ℓ),

where the m(ℓ)×m(ℓ−1) matrix W contains the weights⁶³63

It should be noted that the use of both W and WT in [78] in equations equivalent to Eq. (26) is confusing. For example, on p. 205, in Section 6.5.4 on backpropagation for fully-connected feedforward network, Algorithm 6.3, an equation that uses W in the same manner as Eq. (26) is a(k)=b(k)+W(k)h(k), whereas on p. 191, in Section 6.4, Architecture Design, Eq. (6.40) uses WT and reads as h(1)=G(1)(W(1)Tx+b(1)), which is similar to Eq. (6.36) on p. 187. On the other hand, both W and WT appear on the same p. 190 in the expressions cos(Wx+b) and h=g(WTx+b). Here, we stick to a single definition of W(ℓ) as defined in Eq. (27) and used in Eq. (26).

(27)wi(ℓ)=[wi1(ℓ),⋯,wim(ℓ−1)(ℓ)]∈Rm(ℓ−1)×1,W(ℓ)=[w1(ℓ);⋯;wm(ℓ)(ℓ)]=[w1(ℓ)⋮wm(ℓ)(ℓ)]∈Rm(ℓ)×m(ℓ−1),

and the m(ℓ)×1 column matrix b(ℓ) the biases:⁶⁴64

Eq. (26) is a linear (additive) combination of inputs with possibly non-zero biases. An additive combination of inputs with zero bias, and a “multiplicative” combination of inputs of the form zi(ℓ)=∏k=1k=m(ℓ)wik(ℓ)yk(ℓ) with zero bias, were mentioned in [12]. In [112], the author went even further to propose the general case in which y(ℓ)=ℱ(ℓ)(y(k), with k<ℓ), where ℱi(ℓ) is any differentiable function. But it is not clear whether any of these more complex functions of the inputs were used in practice, as we have not seen any such use, e.g., in [21] [78], and many other articles, including review articles such as [13] [20]. On the other hand, the additive combination has a clear theoretical foundation as the linear-order approximation to the Volterra series Eq. (496); see Eq. (497) and also [19].

(28)z(ℓ)=[z1(ℓ),⋯,zm(ℓ)(ℓ)]T∈Rm(ℓ)×1,b(ℓ)=[b1(ℓ),⋯,bm(ℓ)(ℓ)]T∈Rm(ℓ)×1.

Both the weights and the biases are collectively known as the network parameters, defined in the following matrices for layer (ℓ):

(29)θi(ℓ)=[θij(ℓ)]=[θi1(ℓ),…,θim(ℓ−1)(ℓ) | θi(m(ℓ−1)+1)(ℓ)]=[wi1(ℓ),…,wim(ℓ−1)(ℓ) | bi(ℓ)]=[wi(ℓ) | bi(ℓ)]∈R1×[m(ℓ−1)+1]

(30)Θ(ℓ)=[θij(ℓ)]=[θ1(ℓ)⋮θm(ℓ)(ℓ)]=[W(ℓ) | b(ℓ)]∈Rm(ℓ)×[m(ℓ−1)+1]

For simplicity and convenience, the set of all parameters in the network is denoted by θ, and the set of all parameters in layer (ℓ) by θ(ℓ):⁶⁵65

For the convenience in further reading, wherever possible, we use the same notation as in [78], p.xix.

(31)θ={θ(1),⋯,θ(ℓ),⋯,θ(L)}={Θ(1),⋯,Θ(ℓ),⋯,Θ(L)}, such that θ(ℓ)≡Θ(ℓ)

Note that the set θ in Eq. (31) is not a matrix, but a set of matrices, since the number of rows m(ℓ) for a layer (ℓ) may vary for different values of ℓ, even though in practice, the widths of the layers in a fully connected feed-forward network may generally be chosen to be the same.

Similar to the definition of the parameter matrix θ(ℓ) in Eq. (30), which includes the biases b(ℓ), it is convenient for use later in elucidating the backpropagation method in Section 5 (and Section 5.2 in particular) to expand the matrix y(ℓ−1) in Eq. (26) into the matrix y¯(ℓ−1) (with an overbar) as follows:

(32)z(ℓ)=W(ℓ)y(ℓ−1)+b(ℓ)≡[W(ℓ) | b(ℓ)][y(ℓ−1)1]=:θ(ℓ) y¯(ℓ−1),

with

(33)θ(ℓ):=[W(ℓ) | b(ℓ)]∈Rm(ℓ)×[m(ℓ−1)+1], and y¯(ℓ−1):=[y(ℓ−1)1]∈R[m(ℓ−1)+1]×1.

The total number of parameters of a fully-connected feedforward network is then

(34)PT:=∑ℓ=1ℓ=Lm(ℓ)×[m(ℓ−1)+1].

But why using a linear (additive) combination (or superposition) of inputs with weights, plus biases, as expressed in Eq. (26) ? See Section 13.2.

An activation function a:R→R, which is a nonlinear real-valued function, is used to decide when the information in its argument is relevant for a neuron to activate. In other words, an activation function filters out information deemed insignificant, and is applied element-wise to the matrix z(ℓ) in Eq. (26), obtained as a linear combination of the inputs plus the biases:

(35)y(ℓ)=a(z(ℓ)) such that yi(ℓ)=a(zi(ℓ))

Without the activation function, the neural network is simply a linear regression, and cannot learn and perform complex tasks, such as image classification, language translation, guiding a driver-less car, etc. See Figure 32 for the block diagram of a one-layer network.

An example is a linear one-layer network, without activation function, being unable to represent the seemingly simple XOR (exclusive-or) function, which brought down the first wave of AI (cybernetics), and that is described in Section 4.5.

Rectified linear units (ReLU). Nowadays, for the choice of activation function a(⋅), Most modern large deep-learning networks use the default,⁶⁶66

“In modern neural networks, The default recommendation is to use the rectified linear unit, or ReLU,” [78], p. 168.

(36)a(z)=z+=[z]+=max(0,z)={0 for z≤0z for 0<z

and depicted in Figure 24, for which the processing unit is called the rectified linear unit (ReLU),⁶⁸68

A similar relation can be applied to define the Leaky ReLU in Eq. (40).

To transform an alternative current into a direct current, the first step is to rectify the alternative current by eliminating its negative parts, and thus The meaning of the adjective “rectified” in rectified linear unit (ReLU). Figure 25 shows the current-voltage relation for an ideal diode, for a resistance, which is in series with the diode, and for the resulting ReLU function that rectifies an alternative current as input into the halfwave rectifier circuit in Figure 26, resulting in a halfwave current as output.

Mathematically, a periodic function remains periodic after passing through a (nonlinear) rectifier (active function):

(37)z(x+T)=z(x)⟹y(x+T)=a(z(x+T))=a(z(x))=y(x)

where T in Eq. (37) is the period of the input current z.

Biological neurons encode and transmit information over long distance by generating (firing) electrical pulses called action potentials or spikes with a wide range of frequencies [19], p. 1; see Figure 27. “To reliably encode a wide range of signals, neurons need to achieve a broad range of firing frequencies and to move smoothly between low and high firing rates” [114]. From the neuroscientific standpoint, the rectified linear function could be motivated as an idealization of the “Type I” relation between the firing rate (F) of a biological neuron and the input current (I), called the FI curve. Figure 27 describes three types of FI curves, with Type I in the middle subfigure, where there is a continuous increase in the firing rate with increase in input current.

The Shockley equation for a current I going through a diode 𝒟, in terms of the voltage VD across the diode, is given in mathematical form as:

(38)I=p[eqVD−1]⟹VD=1qlog⁡(Ip+1).

With the voltage across the resistance being VR=RI, the voltage across the diode and the resistance in series is then

(39)−V=VD+VR=VD=1qlog⁡(Ip+1)+RI,

which is plotted in Figure 29. The rectified linear function could be seen from Figure 29 as a very rough approximation of the current-voltage relation in a halfwave rectifier circuit in Figure 26, in which a diode and a resistance are in series. In the Shockley model, the diode is leaky in the sense that there is a small amount of current flow when the polarity is reversed, unlike the case of an ideal diode or ReLU (Figure 24), and is better modeled by the Leaky ReLU activation function, in which there is a small positive (instead of just flat zero) slope for negative z:

(40)a(z)=max(0.01z,z)={0.01z for z≤0z for 0<z

Prior to the introduction of ReLU, which had been long widely used in neuroscience as activation function prior to 2011,⁷¹71

See, e.g., [19], p. 14, where ReLU was called the “half-wave rectification operation”, the meaning of which is explained above in Figure 26. The logistic sigmoid function (Figure 30) was also used in neuroscience since the 1950s.

The hard non-linearity of ReLU is localized at zero, but otherwise ReLU is a very simple function—identity map for positive argument, zero for negative argument—making it highly efficient for computation.

Also, due to errors in numerical computation, it is rare to hit exactly zero, where there is a hard non-linearity in ReLU:

Thus, in addition to the ability to train deep networks, another advantage of using ReLU is the high efficiency in computing both the layer outputs and the gradients for use in optimizing the parameters (weights and biases) to lower cost or loss, i.e., training; see Section 6 on Training, and in particular Section 6.3 on Stochastic Gradient Descent.

The activation function ReLU approximates closer to how biological neurons work than other activation functions (e.g., logistic sigmoid, tanh, etc.), as it was established through experiments some sixty years ago, and have been used in neuroscience long (at least ten years) before being adopted in deep learning in 2011. Its use in deep learning is a clear influence from neuroscience; see Section 13.3 on the history of activation functions, and Section 13.3.2 on the history of the rectified linear function.

Deep-learning networks using ReLU mimic biological neural networks in the brain through a trade-off between two competing properties [113]:

The block diagram for a one-layer network is given in Figure 32, with more details in terms of the number of inputs and of outputs given in Figure 33.

For a multilayer neural network with L layers, with input-output relation shown in Figure 34, the detailed components are given in Figure 35, which generalizes Figure 33 to layer (ℓ).

And finally, we now complete our top-down descent from the big picture of the overall multilayer neural network with L layers in Figure 23, through Figure 34 for a typical layer (ℓ) and Figure 35 for the lower-level details of layer (ℓ), then down to the most basic level, a neuron in Figure 36 as one row in layer (ℓ) in Figure 35.

Consider the following one-layer network,⁷⁶76

See [78], p. 167.

(43)y˜=f(x,θ)=f(1)(x)=w1(1)x+b1(1)=wx+b=w1x1+w2x2+b,

with the following matrices

(44)w1(1)=w=[w1,w2]∈R1×2,x=[x1,x2]T∈R2×1,b1(1)=b∈R1×1

(45)θ=[θ1,θ2,θ3]T=[w1,w2,b]T

since it is written in [78], p. 14:

First-time learners, who have not seen the definition of Rosenblatt’s (1958) perceptron [119], could confuse Eq. (43) as the perceptron—which was not a linear model, but more importantly the Rosenblatt perceptron was a network with many neurons⁷⁷77

See Section 13.2 on the history of the linear combination (weighted sum) of inputs with biases.

The MSE cost function in Eq. (42) becomes

(46)J(θ)=14∑i=14[b2+(1−w1−b)2+(1−w2−b)2+(w1+w2+b)2]2

Setting the gradient of the cost function in Eq. (46) to zero and solving the resulting equations, we obtain the weights and the bias:

(47)∇θJ(θ)=[∂Jθ1,∂Jθ2,∂Jθ3]T=[∂J∂w1,∂J∂w2,∂J∂b]T

(48)∂J∂w1=0⟹2w1+w2+2b=1∂J∂w2=0⟹w1+2w2+2b=1∂J∂b=0⟹w1+w2+2b=1}⟹w1=w2=0,b=12,

from which the predicted output y˜ in Eq. (43) is a constant for any points in the dataset (or design matrix) X=[x1,…,x4]:

(49)y˜i=f(xi,θ)=12, for i=1,…,4

and thus this one-layer network cannot represent the XOR function. Eqs. (48) are called the “normal” equations.⁷⁸78

In least-square linear regression, the normal equations are often presented in matrix form, starting from the errors (or residuals) at the data points, gathered in the matrix e=y−Xθ. To minimize the squared of the errors represented by ∥e∥2, consider a perturbation θϵ=θ+ϵγ and eϵ=y−Xθϵ, then set the directional derivative of ∥e∥2 to zero, i.e., ddϵ∥eϵ∥2|ϵ=0=XT(d−Xθ)=0, which is the “normal equation” in matrix form, since the error matrix e is required to be “normal” (orthogonal) to the span of X. For the above XOR function with four data points, the relevant matrices are (using the Matlab / Octave notation) e=[e1,e2,e3,e4]T, y=[0,1,1,0]T, and X=[[0,0,1];[1,0,1];[0,1,1];[1,1,1]], which also lead to Eq. (48). See, e.g., [122] [123] and [78], p. 106.

The four points in Table 2 are not linearly separable, i.e., there is no straight line that separates these four points such that the value of the XOR function is zero for two points on one side of the line, and one for the two points on the other side of the line. One layer could not represent the XOR function, as shown above. Rosenblatt (1958) [119] wrote:

So we now add a second layer, and thus more parameters in the hope to be able to represent the XOR function, as shown in Figure 38.⁷⁹79

Our presentation is more detailed and more general than in [78], pp. 167-171, where there was no intuitive explanation of how the numbers were obtained, and where only the activation function ReLU was used.

Layer (1): six parameters (4 weights, 2 biases), plus a (nonlinear) activation function. The purpose is to change coordinates to move the four input points of the XOR function into three points, such that the two points with XOR value equal 1 are coalesced into a single point, and such that these three points are aligned on a straight line. Since these three points remain not linearly separable, the activation function then moves these three points out of alignment, and thus linearly separable.

(50)zi(1)=wi(1)xi+bi(1)=w(1)xi+b(1), for i=1,…,4

(51)wi(1)=w(1)=[1111],bi(1)=b(1)=[0−1], for i=1,…,4

(52)Z(1)=[z1(1),…,z4(1)]∈R2×4,X(1)=[x1(1),…,x4(1)]=[01010011]∈R2×4

(53)B(1)=[b1(1),…,b4(1)]=[0000−1−1−1−1]∈R2×4

(54)Z(1)=w(1)X(1)+B(1)∈R2×4.

To map the two points x2=[1,0]T and x3=[0,1]T, at which the XOR value is 1, into a single point, the two rows of w(1) are selected to be identically [1,1] as shown in Eq. (51). The first term in Eq. (54) yields three points aligned along the bisector in the first quadrant (i.e., the line z2=z1 in the z-plane), with all positive coordinates, Figure 39:

(55)w(1)X(1)=[01120112]∈R2×4.

For activation functions such as ReLu or Heaviside⁸⁰80

In general, the Heaviside function is not used as activation function since its gradient is zero, and thus would not work for gradient descent. But for this XOR problem without using gradient descent, the Heaviside function offers a workable solution as the rectified linear function.

(56)Z(1)=[0112−1001]∈R2×4,

and thus

(57)Y(1)=[y1(1),…,y4(1)]=a(Z(1))=[01120001]=X(2)=[x1(2),…,x4(2)]∈R2×4,