To address limitations in existing datasets, we constructed the Virtual Maintenance Human Keypoint Dataset (VMHKP) through a systematic three-phase process: task design, data collection, and annotation. The dataset includes 1100 RGB images from 11 technicians performing standardized maintenance procedures. These technicians have different body types (lean, medium, and heavy builds) and wear various work clothing (lightweight work attire, standard long-sleeve work uniforms, short-sleeve work uniforms, and work uniforms in different colors). This diversity design significantly enhances the representativeness of the dataset and effectively reduces potential body characteristic bias. Data collection employs a Microsoft Kinect Azure Developer Kit (DK) depth camera integrated with a 12-megapixel RGB camera, fixed at a position 2.5 meters away from the operation area to ensure complete capture of the technicians’ full-body movement range. To ensure data quality and operational smoothness, an image acquisition strategy of collecting one frame every 10 frames is adopted, with 20 most representative images carefully selected from continuous maintenance operation video sequences, covering the preparation, execution, and completion phases of operations. The annotation process uses the COCO-Annotator tool, following the COCO dataset’s 17 keypoint standard for 2D coordinate annotation. The annotation team consists of 3 researchers who received professional training in keypoint localization under the guidance of medical professionals. A dual verification mechanism is employed: after initial annotation, another annotator conducts independent review, and for keypoints with position differences exceeding 5 pixels, consensus is reached through discussion. During the annotation process, consistency checks are implemented by regularly sampling 10% of images for cross-annotator evaluation, ensuring the average error in keypoint localization is controlled within 3 pixels. After all annotations are completed, maintenance technicians with over 5 years of experience are invited to verify the standardization of action postures, ensuring that the annotated actions comply with actual maintenance operation standards. The final dataset passes integrity checks, providing high-quality standard data for model training and evaluation. capturing complete action sequences that embody professional characteristics such as wrist-elbow-shoulder coordination during tool operation and trunk-hip-knee support during component transport. These data provide a reliable foundation for establishing maintenance action evaluation standards. Based on industrial maintenance field investigation, we selected five representative maintenance tasks (Fig. 1). Each task was designed according to actual maintenance manuals to cover operational difficulty, tool usage, and body movement range across different scenarios, ensuring comprehensive coverage and a complete evaluation benchmark for algorithm research.

Figs. 2 and 3 demonstrate typical operation scenarios across various maintenance tasks.

This systematic construction process ensures data quality and standardization while providing reliable training and evaluation data for the human pose estimation method proposed in Section 3.2.

The VMHKP dataset demonstrates unique advantages in three dimensions: data distribution, pose characteristics, and action continuity. Data are evenly distributed across five maintenance tasks, with 220 images per category. Each task includes rich operational scenario variations, such as the common tool application task covering operation poses for 8 maintenance tools including wrenches, electric screwdrivers, and sockets. To quantitatively analyze maintenance pose characteristics, we defined three key angle parameters (Fig. 4): wrist angle, elbow angle, and trunk forward lean angle. The wrist angle is defined as the angle formed by the wrist-elbow-shoulder line, calculated separately for left and right wrists to describe the degree of arm bending. The elbow angle adopts a similar definition. Trunk forward lean angle is defined as the angle between the trunk midline and vertical direction, characterizing the degree of body forward lean.

Statistical analysis of these angles revealed significant differences in joint angle distributions across maintenance tasks (Table 1). Component transfer and transport tasks exhibited larger elbow angles (88.3∘± 14.2∘ and 102.3∘± 15.6∘, respectively), reflecting large-range motion characteristics. Component flip and transport tasks showed larger trunk forward lean angles (both around 42.7∘), demonstrating significant forward lean amplitudes. Tool operation tasks displayed relatively smaller wrist and elbow angles (approximately 85∘ and 75.2∘± 12.3∘, respectively), indicating that fine operations primarily rely on small-amplitude wrist and elbow adjustments

Another important VMHKP characteristic is action continuity. By analyzing keypoint displacement changes between adjacent frames, we quantified the continuity characteristics of maintenance actions (Table 2). Different tasks show distinct motion amplitude differences. Tool operation tasks have relatively smaller average inter-frame displacements (28.5±9.2 pixels), while component transport and flip tasks show notably larger displacements (45.8±14.2 and 42.3±13.5 pixels, respectively). Each task type includes complete action phases, with frame ranges between 12–35 frames, meeting the requirements for continuous action analysis, with detailed continuity analysis results shown in Table 2.

In summary, VMHKP provides a comprehensive foundation for human pose estimation research in virtual maintenance scenarios through balanced task distribution, rich pose characteristics, and complete action sequences. These characteristics enhance model adaptability to maintenance scenarios and provide an important basis for subsequent algorithm optimization.

The VMHPE method is specifically designed to enhance pose estimation performance in virtual maintenance scenarios through multi-scale feature processing and adaptive attention mechanisms. Fig. 5 illustrates the comprehensive architecture of our proposed approach.

The framework consists of four integrated modules working in sequence: the backbone network extracts initial features from input images, the Individual Representation Learning (IRL) module generates discriminative individual features, the Multi-scale Feature Correction (MFC) module enhances feature robustness across different scales, and finally the Keypoint Estimation (KE) module produces precise keypoint heatmaps. This sequential design progressively refines features from coarse to fine, enabling accurate pose estimation even under the challenging conditions typical in maintenance operations.

The MFC module performs adaptive correction on multi-scale feature maps F extracted by the backbone network under the guidance of individual representation. This module employs a Multi-scale Feature Correction Module (MFCM), enhancing feature robustness to pose variations and occlusions in both spatial and channel dimensions through adaptive scale weights and enhanced feature correction operations.

After obtaining individual feature correction results, VMHPE aggregates global features at different scales through the Multi-scale Feature Fusion Module (MFFM). This module combines cross-attention and channel embedding mechanisms, introducing multi-scale convolution to fuse local and global context information from multiple perspectives. The fused features, together with individual enhanced features, provide input for keypoint estimation.

Finally, the KE module adopts an encoder-decoder structure, comprehensively utilizing individual enhanced features and fused features to generate K keypoint heatmaps for each individual. During training, VMHPE jointly optimizes the entire framework in an end-to-end manner. The loss functions include heatmap loss, contrastive loss, and regularization terms. These are used to optimize keypoint localization accuracy, feature discrimination ability, and model generalization performance.

This section first introduces two core attention modules in the VMHPE model: Channel Attention and Spatial Attention (Fig. 6). As basic units for feature enhancement, these two modules achieve adaptive enhancement of input features through different feature interaction methods. In the subsequent multi-scale feature processing, these two modules will be used for correction and fusion of features at different scales.

The Channel Attention module aims to capture dependencies between feature channels and adaptively adjust the importance of each channel. This module takes global features Fglobal∈RB×C×H×W and instance features Finstance∈RB×480 as input. Initially, the instance features are mapped to the same channel dimension as the global features through a learnable linear transformation, as shown in Eq. (1): (1)Finstance′=Watn⋅Finstancewhere Watn represents the linear transformation matrix. The transformed features are then expanded to Finstance′′∈RB×C×1×1 and broadcast to match the spatial dimensions of the global features. Subsequently, the two features are concatenated along the channel dimension, as described in Eq. (2): (2)Fcat=Concat(Fglobal,Finstance′′)

To extract channel correlations, the module applies both global average pooling and max pooling operations to Fcat, generating two global descriptors: (3)Favg=AvgPool(Fcat)∈RB×2CFmax=MaxPool(Fcat)∈RB×2C

These descriptors are further concatenated and processed through a multi-layer perceptron (MLP), as shown in Eq. (4): (4)Fattn=σ(MLP(Concat(Favg,Fmax)))where the MLP consists of two fully connected layers with ReLU activation in between and Sigmoid activation σ at the end. The resulting attention weights are reshaped to [2,B,C,1,1] and applied to the original features in a residual manner: (5)Fout=Fglobal⋅(1+Wc[0])+Finstance′′⋅Wc[1]

The Spatial Attention module focuses on spatial dimension information of feature maps and, in addition to processing global and instance features, optionally accepts instance coordinate information coords∈RB×2. During feature interaction, the module computes element-wise multiplication between global features and transformed instance features: (6)Ffeat=Fglobal⋅Finstance,expanded′′

When instance coordinates are provided, the module computes relative position encoding according to Eqs. (7) and (8): (7)Rcoords=coords.reshape(−1,1,2)−pixel_coords.reshape(1,−1,2) (8)Rcoords=Rcoords.permute(0,2,1).reshape(B,2,H,W)/scale_factor

After concatenating the spatial attention map with relative position encoding, a series of convolution operations generate the final spatial attention weights: (9)Ws=σ(Conv1×1(ReLU(BN(Conv3×3(Concat(Fsum,Rcoords))))))

The final output is computed through a similar residual mechanism, as described in Eq. (10): (10)Fout=Fglobal⋅(1+Ws[:,:C,:,:])+pad(Finput⋅Ws[:,C:,:,:])where the pad operation ensures consistent output dimensions with the input. Through this design, the spatial attention module effectively captures spatial dependencies in the features.

The Multiscale Feature Correction Module (MFCM) is a feature enhancement component specifically designed for human pose estimation tasks in virtual maintenance scenarios (Fig. 7). Through adaptive multiscale processing and dual attention mechanisms, this module significantly enhances feature representation capabilities and adaptability to complex scenarios.

At the implementation level, MFCM first constructs a multiscale feature pyramid. The input global feature map F undergoes multiscale sampling, with sampling coefficients scales typically set to [1,2], corresponding to original scale and 1/2 scale. The sampling process is implemented using bilinear interpolation, which can be expressed as: (11)Fs=Interpolate(F,scale=s),s∈scaleswhere Interpolate represents the bilinear interpolation function, and s is the scaling coefficient. This multiscale design enables the module to capture feature information at different scales simultaneously, laying the foundation for subsequent feature correction. The core of feature correction lies in the synergistic effect of channel attention and spatial attention. The channel attention module adopts a “squeeze-and-excitation” mechanism, with its detailed implementation process. First, global average pooling is performed on the feature map: (12)z=GAP(Fs)=1H×W∑i=1H∑j=1WFs(i,j)

Then channel attention weights are computed through two fully connected layers with nonlinear activations: (13)Achannel=σ(W2δ(W1z))where W1∈RCr×C and W2∈RC×Cr are parameters of two fully connected layers, r is the reduction ratio (typically set to 16), δ is the ReLU activation function, and σ is the sigmoid activation function. This design learns inter-channel dependencies through two nonlinear transformations.

The spatial attention module focuses on the spatial information distribution of feature maps. It processes the feature map through parallel average pooling and max pooling branches: (14)Mavg=AvgPool(Fs),Mmax=MaxPool(Fs)

These pooled features are concatenated and processed by a convolution layer to generate spatial attention weights: (15)Aspatial=σ(Conv7×7([Mavg;Mmax]))where [;] represents channel-wise concatenation, and Conv7×7 is a 7×7 convolution layer used to model spatial attention mapping. The larger kernel size enables capturing broader spatial context information. The feature correction process integrates channel attention and spatial attention outputs through adaptive weights: (16)Fscorr=Fs+λc(Fs⊙Achannel)+λs(Fs⊙Aspatial)where λc and λs are learnable balance factors, initially set to 0.5, and ⊙ represents element-wise multiplication. This design allows the model to automatically adjust the relative importance of the two attention mechanisms during training. To better integrate multiscale information, MFCM introduces an adaptive scale weight mechanism. First, feature aggregation is performed on the global feature F: (17)g=Wg⋅GAP(F)

Then scale weights are generated through the softmax function: (18)w=softmax(g)where Wg is a learnable weight matrix. The final multiscale fusion feature is obtained through weighted summation: (19)Fenhanced=∑s∈scalesws⋅Upsample(Fscorr)

To maintain feature consistency, features at all scales are upsampled to the original resolution before fusion. This design ensures that the final enhanced features retain both fine-grained details from the original scale and contextual information from larger scales.

Given the specificity of virtual maintenance scenarios, we designed key hyperparameters based on theoretical and practical experience. The setting of the dimensionality reduction ratio r requires balancing feature representation capability and computational efficiency. The choice of r=16 is based on theoretical analysis of channel attention mechanisms, which can maintain sufficient feature representation space while controlling computational complexity. The initial values of attention scaling factors λc and λs are set to 0.5 based on the theoretical requirements of multi-scale feature fusion, aiming to achieve effective balance of features at different scales. The temperature parameter τ=0.07 in contrastive learning follows the theoretical guidance of contrastive learning to ensure effective discrimination between positive and negative samples.

To improve the model’s generalization capability and reduce overfitting risks, we designed multi-level data augmentation strategies: geometric transformation augmentation simulates different observation angles and operational postures through random rotation, scaling transformation, and horizontal flipping; photometric augmentation adapts to different lighting environments through color jittering, brightness and contrast adjustment; spatial perturbation includes random cropping and moderate perturbation of keypoint positions to enhance the model’s adaptability to spatial variations. The design of these augmentation strategies fully considers the characteristics and challenges of virtual maintenance scenarios.

The Multiscale Feature Fusion Module (MFFM) significantly enhances traditional feature fusion approaches through its innovative multi-level interaction and fusion strategies (Fig. 8). Unlike conventional methods, MFFM addresses the unique challenges of virtual maintenance scenarios, particularly the critical requirement for fine motion recognition. This targeted design enables more effective feature integration across different scales.

The first key component of MFFM is the cross-attention mechanism. Unlike traditional self-attention, cross-attention enables information exchange between features at different scales. The implementation begins with linear transformations of input features to generate queries, keys, and values: (20)Qi=WqFi+bq Ki=WkFi+bk Vi=WvFi+bvwhere i∈1,2 represents different scales, Wq, Wk, Wv and bq, bk, bv are learnable parameters. The cross-attention scores are then computed through scaled dot-product attention: (21)A12=softmax(Q1K2Tdk)A21=softmax(Q2K1Tdk)where dk is the feature dimension. These attention scores are used to weight and combine features across scales: (22)F1ca=LayerNorm(F1+A12V2)F2ca=LayerNorm(F2+A21V1)where LayerNorm denotes layer normalization, which stabilizes the feature distributions and accelerates training. The channel embedding module further enhances feature representation capability by processing spatial and channel information in parallel paths. First, the cross-attention outputs are concatenated and dimensionally reduced: (23)Fconcat=Concat[F1ca,F2ca] (24)Freduced=Conv1×1(Fconcat)

The reduced features are then processed through spatial and channel branches: (25)Fspatial=DWConvk(Freduced)Fchannel=Freduced⊙σ(MLP(GAP(Freduced)))where DWConvk represents depthwise separable convolution with kernel size k, and MLP denotes a multilayer perceptron. The outputs from both branches are combined and processed through a 1×1 convolution: (26)Fce=Conv1×1(Fspatial+Fchannel)

The multiscale convolution module captures context information at multiple scales through parallel convolutions with different kernel sizes: (27)Fms=∑k∈Kwk⋅Convk(Fce)where K=3,5,7 represents the set of convolution kernel sizes, and wk are learnable weights for each scale. The final fusion feature combines multiscale convolution output with channel embedding features: (28)Ffused=Fms+Fce

The training process employs an end-to-end approach with a comprehensive loss function. The keypoint detection loss uses Mean Square Error (MSE) to measure the difference between predicted and ground truth heatmaps: (29)Lkpt=1N∑i=1N∑j=1K|Hi,j−H^i,j|22where N is the number of samples, K is the number of keypoints, H and H^ are predicted and ground truth heatmaps, respectively. To enhance feature discriminability, a contrastive loss is incorporated: (30)Lcontra=−log⁡exp⁡(fi⋅fi+/τ)∑j≠iexp⁡(fi⋅fj/τ)where fi and fi+ are feature representations of the same individual from different viewpoints, and τ is the temperature parameter. The final loss function combines keypoint detection loss, contrastive loss, and L2 regularization: (31)L=Lkpt+αLcontra+β|θ|2where α and β are weight coefficients that balance the contribution of each term. The L2 regularization term |θ|2 helps prevent overfitting by constraining the magnitude of model parameters.

This multi-level, multi-angle feature processing approach enables MFCM and MFFM to effectively handle various challenges in virtual maintenance scenarios, providing robust feature support for subsequent keypoint estimation. The comprehensive design of attention mechanisms, feature fusion strategies, and training objectives ensures effective learning of pose-relevant features while maintaining computational efficiency through the use of depthwise separable convolutions and residual connections.

To verify VMHPE’s core components, we performed progressive addition of modules to the CID baseline. As shown in Table 6, adding the multi-scale feature correction module (MFC) improves AP by 0.8% while maintaining AR. Further introducing the multi-scale fusion attention mechanism (MFA) improves AP and AR by 0.9% and 1.1%, respectively. The complete model (MFC+MFA) achieves a total improvement of 2.3% in AP and 1.3% in AR compared to baseline.

Fig. 10 shows the performance change trend across components. Model improvement exhibits nonlinear characteristics, with MFA playing a crucial role in feature integration. AR increases significantly after adding fusion attention, from 94.3% to 95.4%, indicating enhanced detection capability across different scales. AP shows continuous improvement, reflecting the cumulative effect of components in enhancing localization accuracy.

As shown in Fig. 11, feature embedding dimensions significantly impact model performance. The model demonstrates good results with 8-dimensional features (AP = 93.9%, AR = 95.3%), reaches optimal performance at 16 dimensions (AP = 94.4%, AR = 95.8%), but shows performance degradation at 32 dimensions (AP = 92.9%, AR = 94.5%). This asymmetric performance curve indicates that excessive dimensions have a stronger negative impact than insufficient dimensions, likely due to overfitting in high-dimensional feature spaces. The 16-dimension configuration achieves an optimal balance between performance and computational efficiency.

As shown in Fig. 12, there are significant differences in keypoint localization accuracy among different pose estimation methods in virtual maintenance scenarios. The VMHPE model significantly improves keypoint prediction accuracy through multi-scale feature processing mechanisms, especially in key maintenance areas such as hand operations and torso support. In comparison, RTMO-S and YOLOXPose series lack precision in capturing fine movements, while DEKR performs well but still has errors in complex postures. This advantage in accuracy lays a reliable foundation for subsequent maintenance action evaluation based on keypoints.

Based on high-precision keypoint prediction results, we calculated similarity between operators’ actions and professional standards by comparing topological structure similarity between keypoint sequences. Professional personnel’s standard actions exhibited clear pattern characteristics-with stable wrist-elbow-shoulder angles during tool operation and specific trunk-hip-knee support angles during transport-forming reliable benchmarks for evaluation. Table 7 presents a quantitative comparison analysis of key angle chains between professional maintenance personnel and ordinary operators, highlighting the differences in posture precision across different action types.

Through experimental validation, we established a four-level assessment standard as shown in Table 8: ≥90% (professional level), 80%–89% (proficient level), 70%–79% (qualified level), and <70% (needs improvement), providing an objective, data-driven approach for standardizing maintenance training. Table 8 provides detailed descriptions of maintenance action evaluation standards based on skeletal topology similarity, including similarity ranges, evaluation standard descriptions, and key feature performance indicators for each level.

To ensure the scientific validity and practicality of the similarity assessment standards, we conducted rigorous statistical validation through expert evaluation experiments and field application verification. First, we invited 6 experienced maintenance trainers to provide professional ratings (1–10 scale) for 20 operators of different skill levels, and compared these ratings with the system-calculated similarity scores through comparative analysis. As shown in Fig. 13, expert ratings and system similarity demonstrate a high positive correlation (Pearson correlation coefficient r = 0.89, p < 0.001). Particularly around the 90% similarity threshold, expert rating consistency reached its highest level (Cohen’s Kappa = 0.86), providing strong support for our professional level threshold setting.

Second, we conducted detailed recording and analysis of the actual work performance of these 20 operators. As shown in Table 9, operators in different similarity intervals exhibited significant differences in work efficiency, operational accuracy, and skill levels.

The data shows that operators with similarity ≥90% demonstrated significantly higher maintenance efficiency than other groups (p < 0.01), the lowest operational error rate (2.3% ± 0.8%), and the highest consistency with supervisor evaluations (Kappa = 0.91). Through one-way analysis of variance, we verified that these differences have statistical significance (F = 16.8, p < 0.001), strongly supporting the setting of 90% as the professional level threshold.

It is noteworthy that we discovered an important phenomenon in our data analysis: the relationship between skill level and similarity exhibits distinct performance jumps at critical threshold points. Particularly at the 90% similarity point, the operational error rate significantly decreased from 5.7% to 2.3%, and similar ‘step effects’ were observed at 80% and 70% thresholds. This nonlinear relationship further confirms the rationality and practical value of our similarity threshold settings.

VMHPE’s high-precision prediction performance provides important support for virtual maintenance training systems. The system can collect trainees’ operation videos in real-time and extract skeletal keypoints through the algorithm. It then compares them with professional personnel’s standard actions to provide objective evaluation results. This evaluation method based on precise keypoint prediction achieves standardization and regularization of maintenance training, providing technical guarantee for improving training quality.