As shown in Fig. 1, multi-view 2D poses are constructed as a multi-view skeleton graph ς=(V,ε,A) to fuse skeleton structure and view correlation, where V={vpm|p=1,…,P; m=1,…,M} denotes all nodes in the graph with P views and M joints in each view. ε={eij|i,j∈V} denotes all edges in the graph, representing the correlations between neighboring nodes. All edges are divided into three parts: (1) skeleton-edge: direct physical connections between adjacent joints in each view of skeleton graph; (2) symmetry-edge: indirect connections between symmetric limb joints; (3) view-edge: connections between the same joints in different views of graphs. The skeleton-edge and symmetry-edge represent the skeleton structure information, and the view-edge represent the multi-view pose correlation information. A=(aij)n×n with n=P×M is the adjacency matrix, representing the connectivity between all nodes, where aij=1 if (i,j)∈ε and aij=0 if (i,j)∉ε.

A common graph convolutional operation [31] in Eq. (1) is applied as the baseline for processing multi-view skeleton graphs. Xl∈Rn×Dil and Xl+1∈Rn×Dol denote the node features of the input graph and output graph of the l graph convolution layer, respectively, where n denotes the node number in the graph, Dil and Dol represent the feature dimensions for each node. The graph convolutional operation involves the following steps. First, Xl is multiplied by Wl∈RDil×Dol, denoting a common convolution operation on the input graph through Dol filters with kernel size (Dil,1). Second, feature of each node is updated by associating with the correlative features of its neighborhood nodes through multiplying a normalized adjacency matrix A^=D−1/2A~D−1/2, where A~=IN+A and Dii=∑jA~ij. Finally, ReLU is adopted as the activation function σ(⋅) to increase the nonlinearity. (1)Xl+1=σ(A^XlWl)

In Eq. (1), all edges in the graph represent the same correlation weight Wl. Considering the diversity of spatial connection structures between joints and the differences between multi-view pose information, neighborhood nodes of each node are first divided into four categories in each single-view skeleton graph and then divided into three categories to distinguish the pose information from other views. The multi-view skeleton graph in Fig. 2 depicts different colors of dots to denote different categories of neighborhood nodes: 1) the center node itself (black); 2) a node (green) which is directly connected to the center node in the skeleton and has a shorter path to the root node (grey); 3) a node (red) which is directly connected to the center node and has a longer path to the root node; 4) a node (yellow) which is symmetric with the center node; 5) a node (blue) that represents the same joint as the center node in the second view; 6) a node (purple) that represents the same joint in the third view; 7) a node (brown) that represents the same joint in the fourth view. The root node points to the abdomen joint. The right elbow joint is taken as an example of the center node. Colorful solid lines represent different correlations between one center node and its neighborhood nodes. Different learnable weights are set to represent the correlations between each node and different neighborhood nodes. Accordingly, the graph convolutional operation in Eq. (1) is transformed into Eq. (2), where c is the category index, Wcl is the convolution kernel weight for the c category of neighborhood nodes. A^c represents the adjacency matrix between n nodes and their c type of neighborhood nodes, where A^=∑c=17A^c. (2)Xl+1=σ(∑cA^cXlWcl)

However, the specificity of different nodes in the graph is not considered in Eq. (2). Therefore, different learnable weights are assigned to each node when learning the correlations between all nodes and a certain category of neighborhood nodes to further learn elaborate skeleton structure information and view correlation information. Modified graph convolutional operation is written in Eq. (3), where Fc∈Rn×Dol is a learnable weight matrix for n nodes and Dol−dimensional features per node, and ⊙ denotes the dot product operation between two matrices. (3)Xl+1=σ(∑cA^cXlWcl⊙Fc)

Based on the graph convolution operation proposed above, a Residual Graph Convolution (RGC) module is designed as the basic module. In Fig. 3a, the RGC module is composed of a GCN Unit and a Per-Node Feature Fusion (FF) layer. GCN Unit in Fig. 3b implements the weighted feature fusion of neighboring nodes, where matrix multiplication operation is conducted between a learnable weight matrix FC for n nodes and constant adjacency matrix A^C to obtain the correlation weight matrix between neighboring nodes, which represents the skeleton structure information and view correlations. Per-Node FF layer in Fig. 3c performs 1×1 convolution to fuse multi-dimensional features for each node.

RGC is combined with the simplified Hourglass architecture to construct the Hourglass-GCN as our 3D pose estimation network. In Fig. 4, Hourglass-GCN processes three scales of multi-view skeleton graphs to extract local-to-global scale and shallow-to-deep level skeleton semantic information and view correlations efficiently. High-to-low and low-to-high processes of the Hourglass-GCN are symmetrical. Hourglass-GCN is composed of graph convolution, graph pooling, graph upsampling, and skip connection components which are all designed to be concise. The graph pooling layer transforms large scales of skeleton graphs into small scales of graphs to obtain local-to-global graph representation. The first graph pooling layer pools the trunk and limb nodes separately. The second pooling layer pools the five nodes obtained above in each view of skeleton graph. Skip layer performs the element-wise sum operation between graph features at the same scale to preserve low-level graph spatial information.

When training Hourglass-GCN, the loss function consists of the 3D pose loss LP and symmetry loss LS. In Eq. (4), LP is defined as the mean square error of the estimated and ground truth 3D joint positions, where jpm~ and jpm represent the estimated and ground truth 3D joint position of joint m in view P.

In Eq. (5), LS measures the difference between corresponding bone pairs in the left and right body parts to make generated 3D poses structurally reasonable, where BPb is the length of bone b in the left body in view P and r(b) is the corresponding right bone. The final loss function L defined in Eq. (6) is taken as the linear combination of LP and LS, where λP=1 and λS=0.01. (4)LP=∑p=1P∑m=1M‖jpm~−jpm‖22 (5)LS=∑p=1P∑b‖Bpb−Bpr(b)‖22 (6)L=λPLP+λSLS

Human3.6M Dataset. Human3.6M [44] is one of the largest datasets for 3D pose estimation, containing 3.6 million images captured from four different camera views. There are eleven subjects consisting of five women and six men performing fifteen typical activities in the indoor environment, such as discussion, talking on the phone, walking, and eating. S1, S5, S6, S7 and S8 are used for training and S9, S11 are used for testing. Ground truth 3D joint positions are recorded by the Vicon Motion Capture System, and the ground truth 2D joint positions of each human instance and camera parameters are also included in this dataset. The pose of each instance contains 17 specific joints. Most work adopts the mean per joint position error (MPJPE) between the estimated and ground truth 3D joint position as the performance evaluation index in millimeters (mm).

MPI-INF-3DHP. MPI-INF-3DHP [45] is a large 3D pose dataset containing more than 1.3 million frames deriving from four male and four female subjects taken from different views. Subjects perform diverse actions in indoor and outdoor scenes. Test set contains about 3000 frames from 6 subjects. Image sequences of two subjects and training sets are from indoor scenes with green screens, the other two sequences are from indoor scenes without green screens, the remaining two sequences are from outdoor scenes. The Percentage of Correct Keypoints (PCK) within 150 mm and the Area Under the Curve (AUC) are common test metrics.

In our experiments, node features of the input graph were fed into a batch normalization layer to keep data distribution consistent. Our model was implemented under the PyTorch framework, which was trained for 200 epochs using Adam optimizer with a batch size of 256. The learning rate was initially set to 0.0005 and was reduced by 0.95 times after each epoch with a decay rate reduced by 0.5 times after every five epochs. All experiments were conducted on one GeForce GTX 1080 GPU with CUDA 9.0.

Table 5 shows the quantitative comparison of our method with some existing 3D pose estimation methods on Human3.6M. MPJPEs of GCN-based monocular methods are generally smaller than common monocular methods, which is an inspiration to introduce skeleton structure information into the multi-view 3D pose estimation to improve the estimation accuracy. The average MPJPE of existing multi-view methods is 33.3 mm, confirming that fusing multiple views of pose information evidently improves the pose accuracy when compared with monocular methods. When the view number is four, our method achieves the minimum error of 29.4 mm, which is 3.9 mm smaller than the average error of multi-view methods. This progress is owing to constructing a multi-view skeleton graph and learning adaptive correlation weights between nodes to completely fuse multi-view pose information and extract elaborate skeleton features.

Table 6 shows the comparison with some existing methods on MPI-INF-3DHP. It can be observed that multi-view methods outperform monocular methods on account of richer pose information and view correlation features provided by multi-view model input. In comparison to the best performing multi-view method, our model exhibits relative improvements of 0.43% and 8.7% in terms of the PCK and AUC metrics on MPI-INF-3DHP, respectively, indicating that combining skeleton priors with multi-view pose information is effective in improving the pose estimation accuracy.

To evaluate the pose estimation performance of our model intuitively, qualitative 3D pose estimation results of some images in the Human3.6M test set are shown in Fig. 9. Three-view synchronized images are input into our model to estimate multi-view 3D poses. For simplicity, only the 3D poses corresponding to the middle-view images are displayed in the last two rows of Fig. 9. Estimated 3D skeleton graphs have high similarity with the target skeleton graphs. 3D positions of some occluded joints in the image can be estimated accurately on account of the precise skeleton structure and view correlation information. We also explored some instances with failure poses. The right two columns of Fig. 9 show failure cases from the “Sitting Down” action in Human3.6M. As can be seen, under some circumstances of severe occlusion or unusual human poses, estimated poses have low similarity with the target poses.