Consider a physical domain Ω with a boundary ∂Ω, as shown in Fig. 2. A body force per unit area b acts on Ω, and its boundary ∂Ω is decomposed into disjoint regions Γ_u and Γ_t. The prescribed displacement u¯ is imposed on Γ_u; the external traction t¯ is employed on Γ_t. Additionally, a discontinuity cuts the body into two subdomains Ω⁺ and Ω⁻, with the respective discontinuity surfaces Γc+ and Γc− (i.e., Γc = Γc+ ∪ Γc−). Suppose the constitutive law σ(u)=D:ε(u) governs the body Ω, while the cohesive zone model characterizes the discontinuity in this physical domain. Then, the differential equation for this problem can be written as:

(1){∇⋅σ=0(inΩ∖Γc)σ⋅n=t¯(on Γt)u=u¯(on Γu)t+=σ⋅m+=−t(on Γc+)t−=σ⋅m−=t(on Γc−)

In the above equations, D represents the isotropic elasticity tensor; n denotes the outward normal vector of the stress boundary Γ_u; u signifies the displacement vector; t−=−t+=t are the tractions imposed on the discontinuity surface Γc; m is the outward normal vector of Γc.

Using the virtual work principle, the equilibrium equation Eq. (1) is further rewritten as a weak form,

(2)∫Ω∖Γcδε:σdΩ+∫Γc+δΛu⋅tdΓ=∫Ωδu⋅bdΩ+∫Γtδu⋅t¯dΓ

wherein the sign ‘δ‘ denotes a virtual value of a variable that follows, and Λu=u+−u− denotes a displacement jump vector across the material surface.

To ensure the numerical tool’s high fidelity and robustness, this study introduces the nonlinear A-FEM (i.e., N-AFEM). N-AFEM integrates with a constitutive model of geomaterials, specifically a tension-compression-shear coupled cohesive zone model, to simulate localized failure and fracture propagation in geotechnical structures. This section provides a concise overview of the fundamental framework of N-AFEM used to construct displacement fields involving strong discontinuities.

For simplicity, this study employs the typical four-node, quadrilateral plane element based on the N-AFEM to illustrate the scheme of the two-dimensional displacement field. Fig. 3a provides the crack propagation within the physical domain, resulting in two basic types of crack configurations in the discretized meshes: two quadrilateral sub-domains for element ‘A’ (Fig. 3b) and one triangular along with one pentagonal for element ‘B’ (Fig. 3c). At the discontinuity surface, four internal nodes (i.e., nodes 1_c, 2_c, 1c′, 2c′) are enriched with regular nodal DoFs (degree of freedom). A cohesive element based on nodes 1_c, 2_c, 1c′, 2c′ links the two subdomains. Consequently, the cracked element is now augmented into three transitional sub-elements: two regular sub-elements for the solid domain and one cohesive element for the discontinuous crack. The incremental equilibrium equation for both cracked configurations can be unified using Eq. (2).

For the regular element Ω+, (3a)[K11+K12+K21+K22+]{Δuext+Δuint+}={Δfext+Δfint+}

For the regular element Ω−, (3b)[K11−K12−K21−K22−]{Δuext−Δuint−}={Δfext−Δfint−}

For the cohesive element, (3c)[K11cohK12cohK21cohK22coh]{Δuint−Δuint+}={Δfint−cohΔfint+coh}

where Kij± denotes the tangent matrix for regular elements, displaying forms entirely identical to those found in conventional FEM; Kijcoh denotes the tangent matrix of the cohesive element, with its components contingent upon the form of the CZM outlined in Section 2.3; Δuext± and Δuint± refer to incremental displacements of external and internal nodes. Additionally, Δfωe (where ω = ext and int) signifies the incremental nodal force regarding regular and cohesive elements.

A condensation scheme that fully condenses the additional DoFs at the elemental level is employed for the N-AFEM. The condensation procedures of the N-AFEM, as per Eqs. (3a)–(3c), are outlined herein. Initially, achieving elemental equilibrium along the shared edges of 1_c2_c and 1c′2c′ for both the regular and cohesive elements is necessary, denoted by:

(4)Δfint++Δfint+coh≅0Δfint−+Δfint−coh≅0

Next, we substitute Eqs. (3a)–(3c), into Eq. (4), enabling the expression of the internal nodal displacements in terms of the external nodes, denoted as:

(5){Δuint−=(Φ1)−1K12coh(Φ2)−1K21+Δuext+−(Φ1)−1Φ3Δuext−Δuint+=(Φ2)−1K21coh(Φ1)−1K21−Δuext−−(Φ2)−1K21+Δuext+

where Φ1=K22−+K11coh, Φ2=K22++K22coh−K21coh(Φ1)−1K12coh, Φ3=K12coh(Φ2)−1K21coh(Φ1)−1K21−+K21−.

Substituting Eq. (5) back into Eqs. (3a) and (3b), and subsequently eliminating Δfint+, Δfint−, Δfint−coh and Δfint+coh, we can obtain the condensed incremental equilibrium equation devoid of internal nodal forces or displacements,

(6)[K11−−K12−(Φ1)−1Φ3K12−(Φ1)−1K12coh(Φ2)−1K21+K12+(Φ2)−1K21coh(Φ1)−1K21−K11+−K12+(Φ2)−1K21+]{Δuext−Δuext+}={Δfext−Δfext+}

In this section, a user-defined cohesive element is introduced to characterize the intense deformation of previously formed discontinuous surfaces, accounting for the coupling between cohesive and frictional behaviors in geomaterials. Considering the uniformity of the cohesive element’s format and implementation procedures [23], this section primarily focuses on elucidating the inherent structure of the coupled cohesive zone model (CZM). This coupled CZM encompasses two modes aligned with the loading conditions of the localized band: the tension-shear mode and the compression-shear mode.

Regarding the tension-shear mode, an exponential CZM referring to the work of [24] is established herein, i.e.,

(7a)tn=∂ϕ∂δ∂δ∂δn=exp⁡(1)σtδnδcexp⁡(−δδc) (7b)ts=∂ϕ∂δ∂δ∂δs=β2exp⁡(1)σtδsδcexp⁡(−δδc)

where t_n and t_s are normal and tangential tractions on the slip surface; δ_n and δ_s are normal opening and tangential slip on the crack surface; δ_c is the critical displacement corresponding to the peak point; σ_t signifies the tensile strength of materials; The term exp(x) signifies e^x, while φ signifies the free energy facilitating coupling between cohesive surface deformation and decohesion. This is expressed based on the formulation established by [25], i.e.,

(8)ϕ=exp⁡(1)σtδc[1−(1+δ/δc)exp⁡(−δ/δc)]

wherein δ represents the effective opening displacement, defined by [26] as δ=δn2+(βδs)2 (where β is a coefficient weighting the tangential contributions).

To facilitate the comprehension of the tension-shear CZM, we present a 3D schematic diagram. In Fig. 4a, the normal traction evolves with the normal opening displacement and tangential separation displacement. Fig. 4b illustrates a similar evolution process for the tangential law. These outcomes validate the coupled behaviors of the tension-shear CZM as derived from Eqs. (7a) and (7b).

Concerning the compression-shear mode, a compression-shear-coupled law, derived from the Mohr-Coulomb strength theory, is established for the tangential direction of the discontinuous surface, (9a)tsc=βexp⁡(1)τpδsδcexp⁡(−β|δs|δc)(|δs|≤δsr,for strength softening stage) (9b)tsc=rcτp(|δs|>δsr,for residual strength stage)

where |⋅| represents the absolute value function; δ_sr serves as the threshold value distinguishing the softening and residual stages of the CZM; τ_p denotes the peak strength of the discontinuous surface, defined as

(10)τp=βσt+μs|tn|

r_c represents a reduction coefficient applied to the interface peak strength to align it with its residual strength, while μ_s denotes the frictional coefficient along the discontinuous surface (Here μ_s = tanϕ_p, ϕ_p is the peak frictional angle on the crack surface).

Furthermore, the normal expression is formulated based on a penalty stiffness method aimed at preventing the self-penetration of the cracked surfaces, i.e.,

(11)tn=αexp⁡(1)σtδcδn

wherein α is a factor adjusting the value of the penalty stiffness.

Similarly, Fig. 5 presents a 3D schematic diagram illustrating the traction-separation law under a compression-shear state. In the t_n-δ_n-δ_s coordinate system, the normal law is depicted as a plane, with the normal traction increasing linearly as the negative (or compressive) displacement increases. In contrast, the tangential law exhibits a nonlinear relation before reaching its threshold value δ_sr, resembling those given in Fig. 4. However, the peak traction increases with the rise of compressive displacement due to the incorporation of Mohr-Coulomb strength theory. Furthermore, the tangential law transitions to a linear relation once the crack surface enters the residual strength state.

In this study, the N-AFEM utilized an ABAQUS user-defined element (UEL). Furthermore, a zero-thickness 2D linear cohesive element, which is formulated from the coupled CZM (as illustrated in Section 2.3), is developed as a UEL subroutine within the N-AFEM, enabling effective incorporation of tension-compression-shear coupling for localized bands in geomaterials into the numerical method.

Concerning crack initiation and growth, this study employs a hyperbolic law in the t_n-t_s space as the crack initiation criterion [17,27], represented by Eq. (12). (12)f(tn,ts)=ts2−(c−tn⋅μs)2+(c−σt⋅μs)2

Here t_n and t_s on the crack plane denote the normal and shear stresses, respectively, of elements on the specific plane (i.e., crack surface) before cracking. The model parameters c (=βσ_t), μ_s, and σ_t, are assumed to be consistent with those of the continuum before elements start cracking. Eq. (12) serves a role similar to the Mohr-Coulomb yield surface in classical plasticity. In simpler terms, when the averaged stress of the Gauss integration points within an element reaches a certain state f(tn,ts)=0, the element initiates cracking in a prescribed direction expressed as Eq. (13).

(13)θ=45∘+φ′/2

where θ represents the angle between the crack surface and the direction of the minimum principal stress; φ′denotes the angle determined by the tangent direction of the hyperbolic curve based on Eq. (12) at the crack initiation point.

In this section, we validate the coupled CZM’s accuracy under compression-shear conditions using the direct shear test. Additionally, we analyze the correlation between the CZM parameters and the strength parameters of geomaterials to demonstrate its applicability in geotechnical structures. Tensile behaviors were previously scrutinized and validated in our prior work based on the clay beam test [20]; therefore, we omit its repetition here.

To emulate the direct shear test conducted by [28], a model is established in this subsection. As illustrated in Fig. 6, the model measures 60 mm in length and 30.4 mm in height, delineated by a phantom dotted line into two segments: the upper and lower boxes. The upper box undergoes a constant normal pressure P (20, 53, 100, and 200 kPa) on its top surface and prescribed displacement u_x on its side surfaces, while the lower box faces fixed and horizontal constraints on its bottom and side surfaces, respectively. Consequently, under these conditions, the upper and lower boxes slide along the phantom dotted line (see Fig. 6), inducing soil rupture.

The structured mesh, employing a four-node quadrilateral linear element, was used for both the upper and lower boxes, while zero-thickness cohesive elements derived from the coupled CZM were positioned along the phantom dotted line. Material parameters for this model were determined through the iterative fitting of numerical predictions to experimental data: Young’s modulus E = 1.2 + 0.6P (MPa), Poisson’s ratio ν = 0.3, β = 1.0, peak internal friction angle ϕ_p = 38.66° (μ_s = tanϕ_p = 0.176), residual internal friction angle ϕ_r = 32.62°(r_c = tan(ϕ_r)/tan(ϕ_p) = 0.8), δ_sr = 0.566 mm, δ_c = 1.2 mm. Given the sandy soil nature of this model, the tensile strength σ_t was set to zero (i.e., cohesion c = βσ_t = 0). To prevent self-penetration of crack surfaces, a normal contact stiffness of k_n = 1.13 × 10⁹ kN/m³ was assigned to the CZM.

Fig. 7 illustrates the comparison between numerically predicted curves of average shear stress (τ_xy) vs. tangential displacement (u_x) and experimental results, where τ_xy is an average from the tangential stress of all cohesive elements along the shear slip surface. Additionally, Fig. 7 displays the predicted results based on the CZM without compression-shear coupling. It can be seen that the curves obtained from the coupled CZM closely resemble the experimental outcomes. In contrast, the four curves derived from the uncoupled CZM consistently align with the abscissa of Fig. 7, suggesting this model does not factor in the contribution of normal pressure to the interfacial shear strength of materials.

Fig. 8 elaborates on the stress evolution within the direct shear model, portraying shear stress images corresponding to five loading stages identified in Fig. 7 (i.e., stages a to e). The direct shear model exhibits progressive failure, showcasing evident strain localization. Initially (Figs. 8a and 8b), both ends of the potential slip surface reach the peak material strength, indicated by shear cracks in this region. Subsequently, due to the continuous increase in tangential displacement (u_x), shear cracks further propagate from the ends towards the center section of the slip surface (see Figs. 8c and 8d). By stage e (Fig. 8e), the direct shear model has undergone propagation by the shear crack, entering a residual state with reduced shear stress observed in this image. Both the response curves and staged stress images provide preliminary evidence supporting the coupled CZM’s applicability in describing geomaterial cracking under a compressive-shear state.

This section employs the N-AFEM to model a two-dimensional soil embankment based on studies by [16] and [29] (as depicted in Fig. 9). This aims to validate the proposed method’s reliability and uncover the localized deformation and failure mechanisms within the clayey soil slope. Considering the embankment’s symmetry, half of it is considered in the calculation model, featuring a rigid rectangular foundation subject to a vertical displacement u_y simulating an external load (F denotes the reaction force from u_y). The model’s geometric dimensions and boundary conditions are outlined in Fig. 9, where multiple fixed constraints are applied to the model’s bottom, and horizontal forced constraints are placed on the right edge. The model assumes a plane strain condition, and its material parameters align with the provided references: E = 10 MPa, ν = 0.4, ϕ_p = 10° (μ_s = 0.176), ϕ_r = 1.718° (r_c = 0.17), σ_t = 32 kPa, c = βσ_t = 32 kPa, δ_sr = 0.00335 m and δ_c = 0.05 m. A four-node, quadrilateral plane element is used in this model, and the total number of elements is 4900 (= 70 × 70), as shown in Fig. 9.

In this section, we validate the effectiveness of the proposed numerical framework by comparing our predicted results with those from existing numerical models. Fig. 10 presents a comparison of rupture band trajectories obtained from the A-FEM model (this study), the AES (assumed enhanced strain) model with no opening-sliding coupling [29], and the AES model with opening-sliding coupling [16]. The three models exhibit similar rupture characteristics, encompassing overall trajectories, rupture initiation points, and rupture outcropping points. Notably, the A-FEM model demonstrates more consistent rupture band trajectories with those from the work of [16]. This consistency can be attributed to introducing an opening-sliding cohesive model into the AES model, as we also integrate a similar tension-compression-shear model CZM into the A-FEM algorithm. Additionally, we compare a quantitative parameter, the maximum reaction force at the loading point (see Fig. 9), among the three models: F_max equals 817.64 kPa [29], 805.88 kPa (this study), and 716.43 kPa [16]. Considering both rupture characteristics and peak reaction force, it is evident that the proposed numerical framework performs well in simulating the failures of soil slope models and can be employed for subsequent simulations.

This sub-section delves into analyzing the failure mode of a clay soil slope. Figs. 11a–11f illustrate the evolution of rupture bands within the slope via maximum principal strain contours at different loading stages. Initially (Fig. 11a), deformation concentrates beneath the foundation surface, evidenced by a series of strain bubbles. Subsequently, as vertical displacement (u_y) continues, a rupture band initiates at the bottom right of the rigid foundation, resulting in a nearly straight-line slip (Fig. 11b). Based on the zoomed picture at the right side of Fig. 11b, it is evident that the maximum principal stress at this stage consistently points downward to the left. This observation implies that soils neighboring the slip line are predominantly in a tension-shear state. Furthermore, when combined with the crack growth direction (refer to Eq. (13)), it provides additional insights into the rationale behind the development of a nearly straight-line slip during the initial crack stage. However, with further u_y increase, the main rupture band’s trajectory shifts from a straight line to an arc, propagating from the inclined edge of the model (Figs. 11c and 11d). We attribute the rupture band trajectory shift to the rotation of the maximum principal stress directions. This is evident from the fact that the maximum principal stress in this area exhibits varying directions (refer to the zoomed picture in Fig. 11d). Besides, the change in principal stress direction at this stage also signifies soils near the arc-shaped sliding surface experience compression-shear failure. Stages e and f (Figs. 11e and 11f) show the sliding mass moving along the cut-through rupture band, displaying evident relative displacement with the sliding bed. The maximum principal stress near the slip line has undergone a transition from a downward-left direction to an upper-right direction. This transition results in a further “cocking-up” of the rupture band at the end (refer to the zoomed picture in Fig. 11f). In summary, the slope exhibits straight-lined and arc-shaped rupture trajectories under tension-shear and compression-shear states, respectively.

Fig. 12 illustrates the displacement evolution in this model, represented by the symbol ‘U’ denoting the resultant displacement. Based on the results, the slope deformation is categorized into three processes: I, II, and III. Process I indicates no rupture on the slope, with displacement primarily under the rigid foundation (Fig. 12a). In process II, tension-shear failure initiates on the right side of the foundation, resulting in a notable discontinuity of displacement on the rupture surface. The resultant displacement focuses mainly on the inverted triangle region to the left of the slip surface and beneath the foundation (Figs. 12b–12d). This unique displacement distribution potentially contributes to the rupture band’s propagation under tension-shear loading. Process III displays a layered displacement distribution gradually increasing from shallow to deep (Figs. 12e and 12f), with the largest displacement observed in the sliding surface, indicating the rigid motion of the slide body. These findings offer new insights into the distinct deformation and progressive failure pattern showcased by the slope experiencing a combination of tension-shear and compression-shear states.

This section aims to investigate the impact of soil tensile strength on a clay soil slope by varying the parameter σ_t within the range of 15, 25, 32, 42, and 50 kPa. Fig. 13 displays the propagation trajectories of rupture bands depicted in principal strain contours. The observations indicate an upward movement of the rupture band in the slope corresponding to an increase in the tensile strength of the CZM, accompanied by a proportional reduction in its overall scale. Moreover, these strain contours illustrate that heightened tensile strength correlates with a weakening of compression-shear deformation, as evidenced by a shorter compression-shear slip line. This observation aligns with findings associated with slopes exhibiting tension cracks.

In Fig. 14a, the reaction force is plotted against vertical displacement curves for the slope model at various tensile strengths, and the peak values of these curves are presented in Fig. 14b. The response curves demonstrate a consistent trend: the reaction force gradually increases before reaching its peak with imposed displacement u_y, followed by a dramatic decrease after the peak due to complete rupture band propagation. However, both the peak reaction force and the corresponding displacement increase with higher tensile strength (see Fig. 14b). These findings suggest that (1) the peak point in the response curves can help distinguish the slope’s state to some extent, and (2) slopes with higher tensile strength tend to exhibit improved bearing capacity and stability. Regarding the latter phenomenon, we attribute it to the implementation of the coupled CZM, wherein the fracture energy of materials is proportionate to their tensile strength. As the tensile strength increases, the energy dissipation required to form a unit surface area of the fully separated crack inevitably rises. This suggests that more external energy (or work) is necessary for the slope. From a macroscopic perspective, this reflects an enhancement in the capacity and stability of slopes.

This section presents a comparative analysis of the weighting factor (β) within the coupled CZM to elucidate the influence of tension-shear and compression-shear cohesive laws on soil slope deformation and failure patterns. Fig. 15 illustrates a comparison of rupture band propagation paths in slopes with varying weighting factors: β values of 0.6, 0.8, 1.0, 1.2, and 1.4. Notably, an increase in β from 0.6 to 1.4 corresponds to the downward movement of the sliding surface, an elongation of the total propagation path, and a reduction in the relative distance between the shear head point and the slope bottom, decreasing from 5.03 to 1.26 m. Moreover, an escalated weighting factor results in a transition of the initial propagation section of the slip surface from an approximately straight line to a smoother arc, which is attributed to the slope failure mode. For models with smaller weighting factors, the initial slip surface propagation is primarily characterized as pure tension or tension-shear failure. Hence, the propagation path appears approximately linear. Conversely, larger weighting factors emphasize the role of tangential CZM, inducing distinct shear failure accompanied by an arc-shaped slip surface. Fig. 16 further depicts the relationship between the peak reaction force and the weighting factor. As the weighting factor increases, the slope model exhibits a higher peak reaction force. This finding suggests that the extension of the compression-shear segment due to increased weighting factors contributes to greater slope stability.

The highway subgrade is a soil body structure comprising a natural earth foundation and an artificial fill embankment. These components typically exhibit varying stiffness throughout their depth due to construction techniques and geostatic stress. Addressing this variability, a simplified embankment, derived from the previously analyzed soil slope, was extensively studied here by incorporating various Young’s moduli for the soil layers. Fig. 17 illustrates the embankment models, including homogeneous soil (model A), stratified soil (model B), soft interlayer soil (model C), and hard interlayer soil (model D), each identified by their corresponding Young’s modulus in the schematic diagram.

Fig. 18 presents a comparison of the deformation and failure patterns among four slope models using principal strain contours at the initial and failure stages. At the initial stage, model A, featuring homogeneous soil, displays a smoother strain bubble (Fig. 18a). Conversely, models B, C, and D exhibit strain discontinuities within strain bubbles due to abrupt Young’s modulus interfaces. The strain bubble in model B diminishes as Young’s modulus increases with depth (Fig. 18b). Contrastingly, model C exhibits strain expansion within the soft interlayer owing to a sharp decrease in Young’s modulus (Fig. 18c). In model D, the strain bubble gradually disappears along the depth, particularly at the upper surface of the hard interlayer, suggesting a mitigating effect on soil deformation. These findings suggest that deformation primarily occurs in layers with lower modulus, while hard interlayers limit deformation, potentially reducing geohazard risks in slopes.

Additionally, strain contours at the failure stage illustrate notable differences in the failure patterns of the four slope models. Models B and D exhibit shorter rupture bands, shifting the outcrop location upward compared to model A (Figs. 18e, 18f and 18h). Model C follows a similar rupture band path as model A, but within the hard interlayer, its geometric profile diverges notably from models A, B, and D. For model C, as the rupture band propagates from the initial soil layer to the soft interlayer, its growth trajectory changes from a downward convex arc to an upward convex arc (Fig. 18g). However, as the rupture band moves from the soft interlayer to the next soil layer, the trajectory adjusts from an upward convex arc to a downward convex shape. A similar shift occurs in model D when the rupture band propagates between the hard and soft interlayers. These characteristics suggest that deformation parameters, such as modulus, significantly influence slope stability by altering the failure pattern. Moreover, the rupture band generally transitions from a downward convex to an upward convex arc while moving between the hard and soft interlayers, exhibiting the opposite adjustment when moving from the soft interlayer to the hard interlayer.