In the Flexible Job Shop Scheduling Problem (FJSP), the production process involves N jobs{J₁, J₂, ···, J_N}, each consisting of P operations{O₁, O₂, ···, O_P}. Each operation can be processed on any one of multiple available machines{M₁, M₂, ···, M_A} [25]. In FJSP, all jobs are multi-operation and multi-job: the shop floor contains multiple jobs, and each job comprises multiple operations [26]. The machines are selective, meaning each operation can be processed on multiple machines, and different machines may have varying processing times. Therefore, we must consider not only the sequence of operations but also the selection of appropriate machines for each operation [27]. At any given time, as long as the primary resources permit, different operations can be processed on different machines simultaneously.

For the FJSP, some basic assumptions should be satisfied: (1)

All jobs are available for processing at time zero.

As shown in Table 1, the variable list provides the definitions of each variable used in the formulas.

In the actual production process, a finished product is usually assembled from multiple parts. Therefore, in order to ensure the readiness of sets of parts in the assembly process, it is necessary to consider the balance of multiple product completion times in the production scheduling process. To address this issue, this study introduces a dual optimization objective that aims to more fully reflect the complexity and diversity of the actual production environment. The objective of minimizing the maximum completion time alone may lead to serious deviations in the completion time of certain operations, which in turn delays the subsequent assembly. Although making the material kit as a constraint can alleviate this problem, it is difficult to strictly satisfy this constraint in complex manufacturing environments, and it will limit the flexibility of the scheduling strategy and reduce the diversity of the solution space and the availability of optimal solutions. Therefore, in this study, the optimization objectives are set to minimize the production cycle time and reduce the completion time variance of different kits of products as shown in Eqs. (1) and (2). The multi-objective optimization approach helps to achieve the trade-off between different scheduling objectives and better cope with the uncertainty in the actual production. (1)F1=Cmax (2)F2=Var(Gg) (3)AVGg=1|Gg|∑p∈GgCp (4)Var(Gg)=1|Gg|∑p∈Gg(Cp−AVGg)2

The constraints are expressed as follows:

s.t. (5)tpkl>0,Spk≥0,∀p,k,l (6)Cp<Cmax,∀p (7)Ml∈M,∀l (8)∑h∈Mpkxpkh=1,∀p,k (9)Epk−Spk≥tpkl,∀p,k,h (10)(Epk−tpkh−Epk−1)xpkh≥0,∀p,k,h (11)(Eab−tabh−Epk)xpkhxabh(ypkab+1)+(Epk−tpkh−Eab)xpkhxabh(1−ypkab)≥0,∀p,k,a,b,h

Eq. (5) indicates that the processing time of all operations is greater than 0, and processing can start at time zero on the machines; Eq. (6) ensures that the completion time of the last operation of each job does not exceed the makespan; Eq. (7) states that all machines are included in the machine set. Eq. (8) specifies that each operation of each job can be assigned to only one machine for processing; Eq. (9) defines the completion time of an operation minus its start time as greater than or equal to its processing time; Eq. (10) requires that operations assigned to the same machine must follow a sequential order; Eq. (11) ensures that each machine can process only one operation at any given time.

Reinforcement learning (RL) enables an agent to learn and achieve goals by interacting with the environment [28]. The agent selects actions based on the current state and adjusts its strategy through environmental rewards, aiming to maximize cumulative rewards. RL problems are typically modeled using Markov Decision Processes (MDPs), defined by the tuple <S, A, P, r, γ>, where S is the state set, A is the action set, P is the transition function, r is the reward function, and γ is the discount factor [29].

In deep RL, Soft Actor-Critic (SAC) is an off-policy algorithm combining maximum entropy learning with the Actor-Critic framework. SAC optimizes an entropy-regularized objective, maintaining policy randomness to enhance robustness and generalization. SAC has been successfully applied in various domains: Some researchers used SAC to maximize energy management systems; Some researchers designed a discrete decision-making strategy based on Discrete Soft Actor-Critic with Sample Filtering (DSAC-SF) for highway driving efficiency and safety; Some researchers proposed a multi-agent actor-critic approach with a heuristic attention mechanism for multi-agent pathfinding; and Some researchers applied SAC to intelligent passively mode-locked fiber laser (PMLFL) systems. However, its application to flexible job shop scheduling problems (FJSP) remains unexplored. This study extends SAC to discrete action spaces and proposes a novel experience replay method tailored for FJSP. Optimizing the sampling mechanism enhances SAC’s exploration capability in complex scheduling environments, offering a new solution for FJSP.

For off-policy reinforcement learning, the experience replay buffer stores past interactions between the agent and the environment, enabling the agent to learn from historical data rather than relying solely on new data generated by the current policy [30]. However, traditional experience replay typically employs a single replay buffer, where experiences are stored sequentially without distinguishing their importance. This often results in the overwriting of critical experiences [31]. To address these limitations, we propose the use of AMBER (Adaptive Multi-Buffer Experience Replay) as a replacement for the traditional experience replay buffer.

The AMBER (Adaptive Multi-Buffer Experience Replay) algorithm improves off-policy reinforcement learning by introducing three buffers: the Standard Experience Buffer (SEB), the High Utilization Experience Buffer (HUEB), and the High Productivity Experience Buffer (HPEB). SEB stores all experiences for diversity, while HUEB and HPEB store critical experiences where equipment utilization exceeds threshold ϵ and productivity exceeds threshold η, respectively. At the end of each episode, AMBER classifies experiences into the corresponding buffers based on utilization and productivity, with all experiences stored in SEB. During training, AMBER samples from SEB, HUEB, and HPEB using dynamically adjusted ratios λ and β, prioritizing HUEB and HPEB to accelerate convergence and enhance learning efficiency.

The parameters ϵ and η can be dynamically adjusted based on information entropy thresholds.

The empirical distribution entropy of equipment utilization and productivity is initially computed, as expressed in Eqs. (12) and (13). (12)H(U)=−∑p(u)log⁡p(u) (13)H(P)=−∑p(p)log⁡p(p)here, U denotes the set of equipment utilization rates, p(u) represents the probability distribution of utilization u, and H(U) is the entropy of equipment utilization. Similarly, P denotes the set of productivity values, p(p) is the probability distribution of productivity p, and H(P) is the corresponding entropy.

ε, η update formula, as shown in Eqs. (14) and (15): (14)εt+1=εt+b⋅H(U) (15)ηt+1=ηt+b⋅H(U)here, ϵ_t and η_t represent the current thresholds for equipment utilization and productivity, respectively, while ϵ_t+1 and η_t+1 denote the updated thresholds for equipment utilization and productivity. The term b is a tuning coefficient that controls the influence of entropy on threshold adaptation, typically constrained within the range (0, 1).

As shown in Eqs. (16) and (17), the ratios λ and β are adaptively adjusted based on rewards, but it is necessary to ensure that λ and β do not exceed 1. Here, ΔR represents the change in rewards between the current episode and the previous episode, and k is a parameter that controls the adjustment speed. (16)λnew=λold×(1+k⋅ΔR|R|) (17)βnew=βold×(1+k⋅ΔR|R|)

In this study, the SAC-AMBER algorithm is designed for the Flexible Job Shop Scheduling Problem (FJSP). As shown in Fig. 1, the SAC agent interacts with the environment, receiving experiences {S_t, R_t, a_t, S_t+1}, which are passed to the AMBER buffer. AMBER selects experiences based on predefined ratios and forwards them to the SAC agent network.

The pseudo-code for the SAC-AMBER algorithm is designed as follows (Algorithm 1):

In deep reinforcement learning, the state space, action space, reward function, and environment are critical components. The environment primarily involves the modeling of the FJSP problem, including information on the states of jobs and processing units. The following sections focus on the design of the SAC-AMBER algorithm for FJSP, specifically addressing the state space, action space, and reward function. 1.

Design of the State Space

The state is crucial in deep reinforcement learning, as it forms the foundation for the agent’s decision-making and directly influences learning efficiency and generalization capabilities. In the Flexible Job Shop Scheduling Problem (FJSP), the state space is designed at three levels S = [S_a, S_b, S_c]:

Here, S_a = [S_a,1, S_a,2, S_a,3, S_a,4] represents the state related to job features. S_a,1 denotes the machine numbers assigned to all job operations up to the current training step t. Machine numbers are indexed starting from 1, and if an operation is displayed as 0, it indicates that the operation has not yet been assigned a machine for processing. S_a,2 represents the processing times of all job operations up to the current training step t. If no machine has been selected for processing, the value is displayed as 0. S_a,3 indicates the start times of job operations. If a negative value is displayed (since the start time of the first operation could be 0, requiring differentiation), it means the operation has not yet started processing on a machine. S_a,4 represents the completion times of job operations. If a value is displayed as 0, it indicates that the operation has not yet started processing.

S_b = [S_b,1, S_b,2, S_b,3, S_b,4] represents the state related to machine features. S_b,1 indicates the status of each machine, where a value of 1 denotes that the machine has started processing, and 0 otherwise. S_b,2 represents the job operations assigned to each machine. S_b,3 denotes the total processing time each machine has completed so far. S_b,4 represents the utilization rate of each machine up to the current time step t, as shown in Eq. (18), where T_act is the actual processing time of each machine, and T_avl is the available time of each machine up to the current time step t. (18)MUr=TactTavl

S_c = [S_c,1, S_c,2, S_c,3] represents the global state. S_c,1 indicates the number of remaining operations for each job at the current time step t. If the value is 0, it means the job has been fully processed. S_c,2 represents the total utilization rate of all machines at the current time step t, as shown in Eq. (19). S_c,3 represents the total delay time of all operations on the machines at the current time step, as shown in Eq. (20). (19)MtotalUr=∑l=1m(Tact×MUr)∑Tavl (20)Ttotalla=∑l=1m(Tavl−Tact) 2.

Design of the Action Space

In the flexible job shop scheduling problem, the design of the action space is crucial as it directly impacts the decision-making flexibility and scheduling efficiency of the agent. This paper adopts a two-step action design at each time step t, as illustrated in Fig. 2. First, the processing machine is selected for the current operation of each workpiece, determining the machine number and processing time. Then, the workpiece to be processed is selected, and the workpiece number, operation count, and machines to be allocated are identified. Through this design, the agent can effectively balance exploration and exploitation in complex environments, optimizing scheduling objectives and adapting to dynamic changes.

In the job selection action space, we have chosen four common rules for selecting jobs: (1)

MTT: Select the job minimizing total processing time.

In the machine selection action space, we use four rules: (1)

SPTCO: Select the machine with the shortest processing time for the current operation.

Instead of directly outputting the corresponding workpiece operations and machine codes, the intelligences in this study output heuristic scheduling rules, which are used to decode the corresponding workpieces and machines based on the current state information, and the combination of the workpiece selection rules and machine selections designed above are combined into a hybrid rule as the action space of the intelligent body, as shown in Fig. 3 (Lines of one color in the figure connect a rule).

3.

Design of the Rewards

Since our model aims to minimize the makespan (maximum completion time) and ensure kit production, the reward function is designed as a weighted sum of the makespan and kit production variance. In this study, the makespan and the variance of completion times are normalized separately. Since the normalized values are unbounded, the Tanh function is applied to constrain the normalized data. Finally, the weights are automatically determined, and the objectives are combined into a composite goal. First, the two objective values are normalized as follows:

For each objective f_i, where f₁ represents the makespan (maximum completion time), and f₂ represents the variance of job completion times, the makespan and the variance of completion times are normalized separately at the t times observation as shown in Eq. (21). (21)fi(t)=fi(t)−μi(t)σi(t)+ωhere, μ_i^(t) is the online mean of objective f_i at time step t, and σ_i^(t) is the online standard deviation of objective f_i at time step t. ω is a small positive constant used to prevent division by zero.

The exponentially weighted moving average is used to update the mean and standard deviation, where χ ∈ (0, 1) is the smoothing parameter that controls the influence of new data on the statistics, as shown in Eqs. (22) and (23). (22)μi(t)=(1−χ)μi(t−1)+χfi(t) (23)σi(t)2=(1−χ)σi(t−1)2+χ(fi(t)−μi(t))2

To prevent the normalized values from becoming excessively large, the Tanh function is applied to constrain the normalized values, ensuring that the processed values fall within the range [−1, 1], as shown in Eq. (24). (24)fi(t)=tanh⁡(γ⋅fi(t))

Therefore, the reward function is defined as shown in Eq. (25). (25)r(t)=ψ⋅f1(t)+κ⋅f2(t)

In this objective function, f₁^(t) represents the makespan (maximum completion time), and f₂^(t) represents the variance of the completion times of critical kit products. The weights ψ and κ balance the trade-off between scheduling time and variance. To enhance the adaptability of the model, these weights are dynamically adjusted based on gradients during training. During network updates, not only are the network parameters updated, but the objective weights are also optimized.

This study utilizes three test sets of varying scales—Hurink [32], Dauzere [33], and Brandimarte [34]—to validate the feasibility and generalizability of the algorithm. Initially, the algorithm is trained on the Hurink dataset to evaluate its performance across different task scales. Subsequently, training is continued on the Brandimarte dataset to assess its robustness in large-scale and complex scenarios, ensuring stable training effectiveness. Furthermore, to verify the feasibility of the model’s design for completeness, data from the Hurink and Dauzere datasets are extracted and combined for training, further testing the algorithm’s applicability in completeness scenarios. Through comprehensive experiments across multiple datasets, the adaptability and performance of the algorithm are thoroughly validated.

A comparative experiment, as shown in Fig. 4, was designed to investigate the impact of key hyperparameters in the SAC-AMBER algorithm, such as the learning rate, soft update parameter, discount factor, entropy, Experience pool utilization threshold and Experience pool productivity threshold, all other parameters are automatically adjustable according to the training process. Each group of experiments was trained for 500 rounds and the average reward values obtained for different values of the parameters were compared separately. The figure shows that when the learning rate is 0.02, soft update parameter is 0.1, discount factor is 0.003, entropy is 0.003, Experience pool utilization threshold is 0.4 and Experience pool productivity threshold is 0.4, the reward values obtained are relatively stable.

The hyperparameter settings for the SAC-AMBER algorithm are shown in Table 2:

To validate the proposed method, SAC-AMBER is compared with heuristic rules (Rule 1: SPT + STM, Rule 2: EEW + LTM) and DRL algorithms. SPT denotes the shortest processing time, EEW is the earliest end of processing, STM is the machine with the shortest total runtime, and LTM is the machine with the longest total runtime. The Gap is calculated using a formula as shown in Eq. (26), where, C_max denotes the maximum completion time obtained by the algorithm, and Opt is the optimal solution or approximate optimal solution obtained by the exact method. (26)Gap=Cmax−OptOpt×100%

The program in this study was executed on a computer with a Windows 10 64-bit operating system. The programming environment was based on Python 3.8, utilizing PyTorch 2.1, and the experiments were conducted on an Intel(R) Core(TM) i5-10400 CPU @ 2.90 GHz.