In recent years, some efficient methods for obtaining temporal information have been proposed. Among them, RGB difference and temporal shift methods are both simple and effective. RGB difference can simply obtain boundary and action information by performing a difference between RGB frames, whereas a temporal shift shifts the feature map in the time dimension, allowing features to overlap in time and extract dynamic information during further feature extraction.

The local spatiotemporal module proposed in this paper uses a convolutional neural network to extract features from RGB frames, obtaining local spatial features, and then extracts supplementary temporal information from the difference between multiple frames of RGB over a period of time to obtain the local spatiotemporal information of the current video segment. This module addresses the issues of traditional methods relying on complex data preprocessing and time-consuming processes when extracting short-term temporal information from local regions. It achieves a more efficient way to obtain temporal information and enhances model performance. The structure of the local module is shown in Fig. 2 and the entire module can be divided into two branches. For the input frame It at time t, the first branch directly inputs the raw input data into the convolution layer to extract features, obtaining the static information of the current video frame and the original spatial features of the current segment. The second branch obtains data from two frames before and after t, performs a difference operation on a total of five frames, smooths the feature in the channel dimension, passes through an average pooling layer in the planar dimension, and then adds the pooled feature to input into the convolutional network. At this time, it can obtain the supplementary temporal features of the current video segment. These features are divided into two paths: one directly inputs into the second-stage network of ResNet to extract features and the other combines with the static feature and upsamples according to the feature shape of the first branch, adding it to the first branch feature and inputting into ResNet. Finally, the two feature maps are re-scaled and added to obtain the final local spatiotemporal feature.

Theoretically, the neighboring frames of the input frame It at time t are It−2, It−1, It+1, and It+2. The difference between these 4 frames and It is taken, with Fdiff representing the differential features and FRGB representing the static features. The calculation process of the two paths can be represented by Eqs. (1) and (2): (1)Fdiff=Conv∑i=−2,−1,1,2Avg(It−It−i) (2)FRGB=Conv(It)

In the equations, Avg represents the average pooling layer mapping, and Conv represents the convolutional layer mapping. Finally, FL represents the features of the local module, which can be expressed by Eq. (3): (3)FL=ResNet[FRGB+UpSample(Fdiff)]+UpSample[ResNet(Fdiff)]

In the equation, UpSample represents the upsampling of features, and ResNet represents the residual network mapping.

After obtaining the local spatiotemporal features of video segments, it is necessary to further acquire the temporal features between segments. Both local and global temporal features are important for action recognition. Some video actions may occur in a few moments but have a fixed order. It is necessary to extract local spatiotemporal features and further interact features across the entire temporal dimension of the video. Some actions are slow and continuous actions, requiring the model to grasp the data features of each stage.

For this reason, this paper further proposes a time module based on global information, which uses feature differencing to extract action rhythm information. The input of the module is the local spatiotemporal features of each segment. For features of different segments, differential interaction can be performed through fixed time rules to extract the time features of fixed action rhythms. In the model proposed in this paper, different time intervals essentially represent different action rhythms.

Fig. 2 illustrates the overall structure of the time module based on global information. For the local feature Fn of the nth segment, a backup is first saved and directly connected to the lower-level network, which is theoretically similar to a residual network, preventing gradient and degradation problems and accelerating propagation. Meanwhile, the original frame-level features can also be retained in the current module. Secondly, the original Fn is input into a convolutional network to achieve compression in the channel dimension, smoothing the features. This is because there is a large gap between the features of different segments and direct differencing operations may introduce a significant amount of noise, disrupting the original spatiotemporal features. Smoothing the features in the channel dimension before differencing makes the differencing features more effective. The smoothed features are also backed up for later differencing calculations. Another copy is input into the convolutional layer for feature extraction and then the difference is calculated with the backup features of other segments to obtain the difference information of different segment features. Different features participate in differencing interactions under different translation strides.

In existing video action recognition models based on action rhythm features, different action rhythm features are often extracted by sampling at different data frequencies, and different input sizes require separate network channels for feature extraction, greatly increasing the model parameter count and training time cost. The approach proposed in this paper for extracting action rhythm features directly implements differential feature extraction at different intervals within existing local features through different translation strides, thereby controlling the model size while improving recognition performance.

For the extraction of action rhythm, as shown in the feature vector at the top of Fig. 2, different features of interaction under different displacement step sizes are marked on the vector. When the step size is small, action changes within a relatively short period of time can be obtained, which is suitable for extracting fast-paced action features. Similarly, when the step size is large, it captures slow-paced action changes. In this paper, the method of calculating differences is still used to obtain the temporal features between segments. This module addresses the issue of efficiently extracting video action rhythm features while reducing noise interference by smoothing the features before differencing.

In the specific implementation process, for the data feature Ft, displaced features are obtained through bidirectional translation. The diagrams below Fig. 2 illustrate displacement step sizes of 1, 2, and 3. On this basis, features beyond the boundaries are removed and the blank features are filled in to obtain three feature vectors that are displaced in the time dimension. Then, subtracting the three vectors yields differential information for different time spans. Blank features appearing after displacement are directly filled with null values.

Using the above method, bidirectional differential features can be obtained after bidirectional translation. Then, the features are divided into three paths: one path passes through pooling layers, convolutional layers, and upsampling layers before being passed to the next layer while the other passes through convolutional layers before being passed to the next layer; one path directly transmits the original features downward, and the three paths are added in the next layer. This approach can further enhance the robustness of the time module, making the smoothing operation on different segment features more effective. Subsequently, the features are fused deeply again through convolutional layers and activation layers, and the fused bidirectional features are added together to obtain the bidirectional differential features of the current video segment. Afterwards, the differential features are multiplied with the original features one by one, which is equivalent to treating the differential features as attention parameters of the original features. Attention mechanisms are often more effective at higher levels of network structure, so this paper applies them to the time module. Finally, the segment features with attention mechanisms are added to the original features to obtain the final features of the time module.

Using D to represent the differential function, the differential calculation process can be expressed by Eq. (4): (4)Dn,n−1=Conv(Fn)−Conv(Fn−1)

Using F to represent the smoothed features, and Dn,n−1 represents the local feature difference between the n−1 and n segments.

Next, using H to represent the merged features of the three paths, and H′ to represent the fused features, the calculation process of the unidirectional features is shown in Eqs. (5) and (6): (5)Hn,n−1=Dn,n−1+Conv(Dn,n−1)+Conv[UpSample(Dn,n−1)] (6)Hn,n−1′=Sigmoid[Conv(Hn,n−1)]

Using UpSample to represent upsampling, and the upsampling function is used again in the global module to unify the size of the three path features. Sigmoid is the activation function used in this layer of the temporal module. Finally, using F to represent the final features output by the temporal module, as shown in equation Eq. (7): (7)Fn′=F^n+F^n⊙12[Hn,n−1′+Hn+1,n′]

Using F^n to represent the original local features of the nth segment, and ⊙ represents element-wise multiplication. This means the original features are multiplied with the bidirectional features and then added, and finally the original features are added again to obtain the temporal information of the current stage.

This paper proposes to obtain the difference information by subtracting the previous and next 2 frames from the sampled frame, instead of subtracting each consecutive adjacent frame separately. In addition, considering that for fast actions, subtracting frames with a large time interval may introduce significant noise to the difference information, rendering it ineffective, a pooling layer is added in the channel dimension to smooth features and extract key information. Based on these three schemes for extracting differential RGB information, comparative experiments are conducted in this section to test the performance of each scheme.

As shown in Table 1, It,t∈{1,2,3,4,5} in the table represents the RGB frames at time t, where I3 is randomly sampled for spatial feature extraction, and the other 4 frames are the 2 frames before and after time t. Diffi−j represents the differential information between frame Ii and frame Ij. To demonstrate the effectiveness of differential RGB, the model performance without using differential information was first tested. The Concat function was used to directly concatenate the 2 frames before and after the sampled frame for information extraction. Experimental results show that the spatial-temporal module using differential information achieves better experimental results, reaching 85.851% in accuracy Top1.

Replacing the differential between adjacent frames with the differential from sampled frames reduces accuracy. Since greater time distance between RGB frames introduces more noise, this paper applies average pooling to each frame after obtaining the differential frames. The pooled features are stacked and further smoothed through a channel-wise average pooling layer, compressing differences between features at different time points. This method improved experimental results, achieving 85.851% accuracy on the UCF-101 dataset.

In the time module, to extract action rhythm information, the translation stride during differential interaction is also an important experimental parameter worthy of consideration. Different schemes have been detailed in the previous section, and in this section, experimental results and performance analysis are directly provided.

Table 2 shows the accuracy Top1 and Top5 achieved when performing local feature differential in the time module with different translation strides. From the experimental results, it can be seen that the model with a stride of 1-1-2 achieves a higher accuracy Top1, reaching 85.931%. Compared to the original scheme 1-1-1, it improves the accuracy by 0.487%. The model with a stride of 1-2-2 obtains a 0.027% improvement in accuracy Top5 compared to the original scheme, reaching 97.159%, thereby verifying the effectiveness of global stage differential features on the UCF-101 dataset. However, when the stride is set to 1-2-3, the recognition accuracy significantly decreases, indicating that differential information with a large time span is no longer effective and may even affect recognition performance.

Finally, based on the best model scheme and hyperparameters obtained from the experimental tests, this paper provides the optimal recognition accuracy based on the UCF-101 dataset.

Table 3 presents a comparison of the accuracy of the proposed model with other action recognition models. Among them, the TSM model was pre-trained on simpler datasets like ImageNet. Under the same simple pre-training conditions, the proposed model achieved the highest accuracy of 87%. The TDN, HoCNet, TSM, MEACI-NET, MTNet, and CANet models were further pre-trained on the large-scale Kinetics-400 dataset. Due to the much larger number of samples in this dataset compared to UCF-101, these models can learn more complex data representations, resulting in a significant improvement in final accuracy. Even under the condition of pre-training on both ImageNet and Kinetics-400, the proposed model still achieved the highest recognition accuracy of 97.6%.

Specifically, the recognition accuracy for continuous actions with a strong rhythmic pattern was improved. Examples include BlowDryHair, CleanAndJerk, HorseRiding, JugglingBalls, and Rowing, shown in Fig. 5. These five actions involve the subject performing highly repetitive movements throughout the video, maintaining a certain frequency, and exhibiting a distinct action rhythm. This demonstrates the effectiveness of extracting action rhythm features through feature differences at different scales.

In summary, the video action recognition model based on spatiotemporal features proposed in this paper can effectively improve the recognition accuracy and achieve better performance on the UCF-101 dataset. We initialized our model using the pre-trained model weights from ImageNet and Kinetics-400. ImageNet is a large-scale image dataset that contains over 14 million images, spanning 1000 categories; whereas Kinetics-400 is a large-scale video dataset that includes 400 action categories. By leveraging these pre-trained models, we ensured that our model has learned a rich and diverse feature representation, allowing it to benefit from a broader and more varied training data. This approach also enabled the accuracy of the model proposed in this paper reached the top level in the field.

In this section, we first conducted tests on multiple hyperparameters of federated learning, including the number of training rounds, the number of user samples, and the degree of dataset distribution. Based on these tests, we then validated the effectiveness of the personalized federated learning mechanism proposed in this paper for video action recognition.

After deploying the dataset and model into the FedML framework, experimental tests are first conducted on the setting of hyperparameters. In this section, the total number of users C is set to 20 as a fixed parameter and kept constant. Experimental results are tested with different numbers of users sampled per round S and the number of training epochs E for each user.

As shown in Table 4, when the user training epoch is 1, the model converges after 450 communication rounds, while when the user training epoch is 5, it converges after 145 communication rounds. Although the number of communication rounds decreases, the total number of training epochs increases from 1 × 450 to 5 × 145 = 725 epochs, significantly increasing the training cost. Furthermore, when the training epoch further increases to 10, the recognition accuracy actually decreases.

From the results, it can be observed that the training effect of the model under federated learning is not stable and does not steadily increase with the increase in local training epochs of users. This is due to the non-independent and identically distributed nature of the data, resulting in significant differences between the data distribution of each user and the overall dataset. In traditional deep learning training, each training epoch allows the model to learn complete data features, and the model is optimized with increasing training epochs. However, under federated learning conditions, an increase in training epochs can lead to the model learning too many individual characteristics of user local data, causing the model to overfit. This not only increases the training cost but also fails to achieve better performance. Increasing the aggregation frequency of the model can make the global model closer to the original optimal parameters.

After determining the user training epoch, this paper also conducted experiments based on different numbers of user samples per round. The test results are shown in Table 5. It can be observed that as the number of user samples increases, the training effect of the model also improves. This is because when the total number of users is fixed, the more users sampled per round, the more data participates in the training, and the impact of each user’s individual characteristics on the aggregation is reduced. The parameters aggregated by the central server tend to be more balanced. However, the training time cost will inevitably increase with the increase in the number of samples. From the perspective of simulating a real federated learning environment, the sampling value cannot be set too high. Therefore, in the following experiments, the number of sampled users per round was set to S = 4, meaning that 1/5 of the users (data) participate in the training each round.

The following experiments are conducted based on different data distribution scenarios, with reference to federated learning datasets from other fields to set parameters, using two grouping methods: Dirichlet data distribution and uniform grouping. The experimental results are shown in Table 6, where Dir(∗) represents the Dirichlet distribution, and the parameter α controls the degree of data dispersion. In other studies, for datasets MINIST and CIFAR-10 with 10 categories, α is often set to 0.5. Since this paper’s experiments use a video behavior recognition dataset with a large amount of data and sample sizes, it is necessary to experiment to test the best data grouping method. The uniform grouping data distribution matches the original dataset and is used to compare the impact of non-IID data grouping on training effectiveness.

From the experimental results, it can be observed that under uniform grouping, the model’s convergence speed in federated learning training is the fastest. However, when using Dirichlet distribution for grouping, as the α value decreases, the data distribution becomes more scattered, requiring more rounds for model convergence. This also affects the model’s optimization process, leading to suboptimal training effectiveness and impacting the highest recognition accuracy after convergence. The experimental results further illustrate the impact of unevenly distributed data storage on recognition performance in the federated learning environment. Considering the use of Dirichlet distribution for data grouping, each user’s allocation of data types and quantities is completely random, resulting in fewer common features among user local data. This is suitable for scenarios where public models are used for parameter training. In practical applications, however, each user’s local data often exhibits strong personalized characteristics. Similar to datasets like MINIST and CIFAR-10 with special labels, they are more conducive to personalized federated learning research. Therefore, in testing the personalized federated learning scheme, we ensure each user’s training and test sets have the same sample distribution, with the same data categories proportionally represented in both sets. Based on the experimental results of hyperparameters and dataset grouping methods, the total number of users C is set to 20, the number of users sampled in each federated learning communication round S is set to 4, and the number of local training rounds E for each user is set to 1. The dataset grouping method is Dir(1). Under the above parameter settings, experiments were conducted to verify the personalized federated learning-based optimization model for video action recognition proposed in this paper, comparing the experimental results under conventional federated learning training and personalized federated learning conditions.

Table 7 presents the highest accuracy rates Top1 and Top5 achieved by the model in this paper on local datasets of 20 users under conventional federated learning and personalized federated learning. From the average accuracy rates, the personalized federated learning approach proposed in this paper achieves better results on both indicators, with the Top1 accuracy reaching 97.792%, an improvement of 1.155%, and the Top5 accuracy reaching 99.861%, an improvement of 0.079%.

The experimental results validate the necessity of users holding private parameters, especially in the application scenario of surveillance video action recognition. Local user data inherently possesses strong individual characteristics. For example, in a home surveillance environment, static features such as background and main subjects can vary significantly between users and do not need to be included in the public aggregation on the central server. The proposed video action recognition method based on personalized federated learning and spatiotemporal features designates the first two layers of the network, which focus on extracting static features, as private layers. The parameters from the subsequent three stages, which extract action features, are used for public aggregation, thereby enhancing the model’s training performance.