In this paper, we assume that one scene in which the human is collaborating with the robot to complete an installation task of a glass curtain wall, as shown in Fig. 2. There is a two-step process. (1) the curtain wall should be moved to near the preinstallation location rapidly in the low-restricted area; (2) the curtain wall should be precisely installed into the frame in the high-restricted area. In step (1), there is a bigger space of movement, the efficiency of the movement should be concerned, and the effort from the human should be minimized. However, step (2), it is an accurate installing mission, and the curtain wall stability of the robot end-effector under multi-force action should be more concerned than the human’s effort.

Reinforcement learning is a method that can realize adaptive parameter adjustment online and establish the relationship between action and state uncertainty according to the target. In this paper, this method is used to adjust role parameters during the process of cooperation. The overall architecture of the method is shown in Fig. 4.

In Fig. 4, α1 is the role factor, which regulates the active and passive relationship between the collaborator and the robot in a cooperative task. The α1 is dynamically adjusted according to the performance of the comprehensive evaluation model. The robot is controlled by the admittance method [22,23]. The robot admittance control model is shown in Eq. (7): (7){x¨d−x¨0)+Dd(x˙d−x˙0)+Kd(xd−x0)=(1−α1)fr+fhwhere, Md,Dd,Kd are the expected mass matrix, damping matrix, and stiffness matrix, respectively. The {x¨d,x˙d,xd} are the expected acceleration, velocity, and position of the end of the robot, and the {x¨0,x˙0,x0} are the actual acceleration, velocity, and position of the robot’s end.

This paper proposes a reinforcement learning model to change roles allocation weight during the installation of glass curtain walls in human-robot cooperation, as shown in Fig. 5.

Here the roles allocation value a1 adjusted online according to the system’s current state and reward value. The current state st is the motion state of the robot and collaborator’s force, and the reward value is calculated as a designed model described below. A Q-learning model [24] generates the roles allocation algorithm since no prior strategy or sample. The algorithm of Q-learning is shown in Algorithm 1.

where, α is the learning rate, s is the current system state, s′ is the next sampling system state, a and a′ is respectively the behavior of the current moment (action) and the next sampling moment, r is the return value of the current moment, and γ is the discount factor. π is the strategy. The machine has to learn, by trial and error in the environment, a “strategy” π, according to which the action to be performed at state x is obtained: a = π (s). In Algorithm 1, the core intermediate variable is Q(s, a), which can be referred to as the Q table for short (mainly used for learning deterministic strategy π, namely updating strategy π), representing the expected cumulative reward obtained by the agent choosing action a under state s. Since the dynamic programming method is used and the model is unknown, it is more convenient to use γ discount cumulative reward in this paper. In formula (8), Q(s, a) is updated incrementally.

In order to verify the effectiveness of the method proposed in this paper, an experimental platform was designed for human-robot cooperation to complete the curtain wall installation task, as shown in Fig. 6.

In the experiment, to simulate the installation process and avoid the risk of collision in the actual experimental environment, a laser pointer was fixed to the curtain wall, and a laser point was used to indicate the location of the curtain wall. The curtain wall position indicator experimental device is shown in Fig. 7.

In the installation process of the curtain wall, the movement track can be divided into the low-restricted area and the high-restricted area. In the low-restricted area, the robot has plenty of room to move, and more attention should be paid to the speed of movement and less effort of the partner than to the movement accuracy of the curtain wall. Contrary to the low-restricted area, in the high-restricted area, the movement accuracy of the curtain wall should be paid more attention than the speed of movement and the effort of the collaborator.

Here are the steps: 1)

Firstly, the laser point representing the position of the curtain wall was located in the low-restricted area, which was the starting point. Following the robot’s movement, the location of the laser point was operated according to the interaction force from a six-dimensional force sensor.

In order to obtain the continuous state input vector X={fh,f˙h,x,x¨}, each variable was divided into five fuzzy sets and evenly distributed to the domain by a triangular membership function, so the total number of states was N=54=625. In this paper, the number of behaviors was set as m = 6, that is, A={a1,a2,a3,a4,a5,a6}={0.1,0.2,0.4,0.6,0.8,1}. In other words, the robot and the partner have been given six different roles allocation weights. Here, the minimum role of the collaborator was chosen as 0.1 instead of 0 for the following reasons. 1) it was needed that the operator makes a particular decision as moving guidance information to control the robot in the physical human-robot cooperation; 2) it was hazardous when the robot was given complete control in actual operation.

In this study, the system frequency was set to 1000 Hz, and the frequency of the reinforcement learning was set to 100 Hz. That was the k = 10 in the formula (10). The learning rate was designed to be α=0.95 and the discount factor was designed to be γ=0.9 in the Q-learning model (7). The role weight initial value of the collaborator was set as a1=α1=0.4.

The relationship between human-robot cooperation performance and the roles allocation weight was established with changes in the cooperative performance as the number of iterations increased in this experiment, as shown in Fig. 8. The experiment was repeated 30 times by operating the robot from its initial position to its destination. In this process, collisions should be avoided whenever possible by observing the position of the laser point. The comprehensive performance model consisting of robot’s Jerk and human-robot interaction force that were regarded as equally important (ω1=ω2=0.5) was used to estimate the effect of the role change.

As can be seen from Fig. 8, with the increase in the number of iterations, the comprehensive performance value composed of Jerk and the efforts of the cooperator showed a downward trend. When the number of iterations was more than 25 times, the comprehensive performance value tended to be stable. This experiment showed that it effectively improved robot flexibility and reduced human-robot force by changing the role weight based on reinforcement learning.

In order to evaluate the efficiency of task completion as learning progresses, the relationship between the number of iterations times and task completion time was established, as shown in Fig. 9.

From Fig. 9, the time for task completion decreases as the number of iterations increases. When the number of iterations was more than 25 times, the decline of the task completion time was slow and steady. It has shown that the roles allocation method based on reinforcement learning was a great way to improve task completion efficiency.

The Fig. 10 shows the mean value α1 of the last five iterations of the collaborator’s role over time t based on the reinforcement learning method. Initially, the robot was guided by increasing the partner role α1 gradually, and then the partner role α1 dropped rapidly once the robot was identified in free mode. In the low-restricted area, the robot played a significant role in the movement of the curtain wall, which was beneficial in reducing the labor intensity of the collaborator. However, in the stage of approaching and entering the high-restricted area, the collaborator’s role value was rapidly increased to ensure the smooth movement of the robot end curtain wall.

The relationship between the partner’s force and time t for the last five times was established, as shown in Fig. 11. The force change trend of the partner was similar to the collaborator’s role, which was first more prominent, then decreased, and then increased. This pattern of change is consistent with the relationship between the change of force and the role. The greater the weight of the partner’s role, the greater the force applied.

The cooperation performance of the dynamically changing roles according to the method proposed in this paper and the different fixed role weights was established, as shown in Fig. 12. The result has verified that the continuous adjustment of roles based on the reinforcement learning method was more conducive to the performance of human-robot cooperation than the fixed roles.

In Fig. 12, the fixed roles were designed with six levels, which were the α1=0.1,0.2,0.4,0.6,0.8,1. Compared to the fixed roles, the combination of Jerk and the partner’s force was even lower by using the adaptive role based on the reinforcement learning method, which showed that the method proposed in this paper was excellent results.

In the above experiment, the robot’s Jerk and human-robot interaction force were regarded as equally important (ω1=0.5,ω2=0.5) in the formula (9) of the comprehensive performance model. In order to explore the influence of changing the performance weighting coefficient, the Jerk, the partner’s force, and their overall performance were obtained through experiments based on reinforcement learning, as shown in Fig. 13. The weighting of the partner’s force was set as ω1=0.1,0.2,0.4,0.6,0.8,1.

From Fig. 13, the Jerk was set as the only goal in reinforcement learning when the ω1=0. Even though the sum of the Jerk was minimal, the cooperator’s force was considerable, which led to more effort for humans. If the minimum sum of the partner’s force was designed as the only consideration (ω1=1), the sum of the cooperator’s force was minimum, but the sum of the robot’s accelerations became maximum. Excessive acceleration could lead to vibration at the end of the robot, which would increase control difficulty and danger. The overall performance value was the lowest when the performance weighting coefficient ω1=0.5, that was to say, it was appropriate that the comprehensive performance model consisting of human-robot interaction force and robot’s Jerk were regarded as equally important to estimate the effect of the role change.