These approaches typically begin with preprocessing steps, such as histogram equalization or bilateral filtering to reduce noise. Then segmentation techniques, such as Watershed, GrabCut and Active Contours, are commonly used for human segmentation which are highly sensitive to image quality. For keypoint detection, techniques such as skeletonization of silhouettes and contour-based joint estimation are applied to estimate joint positions.

Then a variety of features are extracted to capture the spatial and temporal dynamics of the interaction. For segmentation, this includes features like Scale-Invariant Feature Transform, Histogram of Oriented Gradients, Motion History Image, Bag of Words-based descriptors and Motion Boundary Histograms while for keypoints, this includes keypoint angles, distances and pairwise keypoint displacements. Then these features are optionally optimized using techniques like Linear Discriminant Analysis or t-SNE before being fed into machine learning classifiers such as Support Vector Machines, Random Forests or k-Nearest Neighbors for final classification. Examples are works done by [2,3].

Some more recent works also incorporated depth information alongside RGB data. For example, in [4], both gray scale and depth silhouette features along with point-based features were extracted before cross entropy optimization and classification using a Maximum Entropy Markov Model.

Many deep learning approaches for human interaction classification typically use fine-tuned Convolutional Neural Network (CNN) architectures to extract spatial features from individual frames followed by 3D CNNs or Recurrent Neural Networks to model temporal dynamics across sequences. Segmentation-based models such as Mask R-CNN [5], trained on diverse RGB datasets, have also gained popularity for their robustness to varying video resolutions. For example, [6] used a DenseNet encoder with a custom decoder for semantic segmentation, refining inputs by suppressing irrelevant regions before passing them into an EfficientNet BiLSTM. Multi-stream architectures improve performance by letting each stream specialize in a different modality, such as RGB frames, optical flow or keypoint coordinates. The information is then combined using fusion techniques, as explored in [7].

There also has been increasing focus on systems that explicitly model spatial and temporal relationships between interacting individuals. With the advent of advanced keypoint detection models like OpenPose [8] and HRNet [9], modeling interactions through body joint movements has become more feasible. Works such as [10,11] have leveraged Interactional Relational Networks and Graph Convolutional Networks to effectively represent both intra and inter-person relationships, offering promising results in complex interaction scenarios.

However, recent works demonstrate that intelligent fusion of heterogeneous features significantly improves model reproducibility and generalization [12]. The proposed system builds on this idea, fusing features from different modalities that are both efficient to extract and effective for accurate classification.

Accurate segmentation is essential for effective feature extraction in activity classification. To achieve this, we use Mask R-CNN [5], which performs accurate instance-level segmentation by generating separate masks for each person. This enables easy removal of irrelevant individuals. It extends Faster R-CNN [13] by adding a parallel branch for mask prediction. The overall loss function L for training Mask R-CNN is defined as a multi-task loss combining classification loss Lcls, bounding box regression Lbox, and mask prediction loss Lmask. The classification loss Lcls is computed as: (1)Lcls=−∑cyclog⁡(pc)where yc is the ground-truth label for class c (1 if the RoI belongs to class c, 0 otherwise), and pc is the predicted probability for class c. The bounding box regression loss Lbox is defined as: (2)Lbox=∑i∈{x,y,w,h}smoothL1(ti−vi)where ti represents the predicted bounding box coordinates (center coordinates (x,y) and dimensions (w,h)), vi represents the ground-truth bounding box coordinates, and smoothL1(ti−vi) is the smooth L1 loss function, which is less sensitive to outliers compared to the standard L2 loss. Finally, the mask prediction loss Lmask is defined as the average binary cross-entropy loss over the predicted mask computed for each ROI and ground-truth class k: (3)Lmask=−1m2∑1≤i,j≤m[yijlog⁡(y^ijk)+(1−yij)log⁡(1−y^ijk)]where m is the resolution of the predicted mask, yij is the ground-truth binary mask value at pixel (i,j), and y^ijk is the predicted mask value for class k at pixel (i,j).

For our experiments, we used a pre-trained Mask R-CNN. RGB frames were resized to an optimal resolution, and a confidence threshold was applied to segment each individual in the frame. Using the ROI boxes, only gray scale silhouettes of relevant individuals were extracted, effectively removing irrelevant individuals. Fig. 2 shows instance segmentation results on the UT interaction dataset (UT) [14], which we evaluated on for our experiments. In the top row, red bounding boxes indicate ROIs, while different colors distinguish individual person instances detected by Mask R-CNN. The bottom row displays corresponding grayscale silhouettes of the segmented individuals within the ROIs.

For keypoints detection and tracking of interacting individuals across frames in ROI, we use YOLO11-Pose [15] from the YOLO framework. YOLO11-Pose builds on YOLOv8’s loss structure. Detection loss includes bounding box loss Lbox (IoU or Smooth L1), classification loss Lcls (Binary Cross-Entropy), and Distribution Focal Loss Ldfl (DFL). For pose estimation, additional losses are used for keypoint localization loss Lkptloc (Smooth L1) and keypoint visibility loss Lkptvis (Binary Cross-Entropy), combined in a weighted total loss computed as: (4)Ltotal=λbox∗Lbox+λcls∗Lcls+λdfl∗Ldfl+λkptloc∗Lkptloc+λkptvis∗Lkptviswhere λbox, λcls, λdfl, λkptloc and λkptvis are task-specific weighting factors. A key innovation in YOLO11 is the introduction of new Cross Stage Partial with Spatial Attention (C2PSA) and C3k2 blocks which enhance feature extraction while maintaining efficiency. C2PSA uses Spatial Pyramid Pooling, which is fast for multi-scale aggregation with a position-sensitive attention mechanism. C3k2 unlike the C2f block used in previous YOLO versions, employs two smaller convolutions instead of one large convolution. Both blocks split the input feature map x into two branches. One branch retains the input features, and the other processes them through specialized modules before being concatenated and fused via a 1 × 1 convolution. The equations are given below: (5)C2PSA(x)=Conv1×1(Concat(a,PSABlock(b))) (6)C3k2(x)=Conv1×1(Concat(y0,y1,…,yn))

In C2PSA, a and b are the split branches of x, and PSABlock applies multi-head attention and feed-forward transformations. In C3k2, y0 and y1 are the initial split branches, and y2,…,yn are the outputs of the C3k or Bottleneck modules.

For our experiments, we used a pretrained YOLO11x-Pose, the largest YOLO11-Pose variant, due to its superior keypoints detection performance, with ByteTrack tracker for correct keypoints assignment across frames. ByteTrack assigns unique IDs to persons across frames by first matching high-confidence detections using Intersection over Union, then re-associating low-confidence detections to reduce identity switches. Again, using the ROI boxes, keypoints of irrelevant individuals were removed.

YOLO11-Pose detects 17 keypoints, namely nose, eyes, ears, shoulders, elbows, wrists, hips, knees, and ankles. In our approach, we exclude the ear and eye keypoints because they are highly correlated with the nose and often suffer from unreliable detection depending on the person’s orientation in the frame. For example, when an individual is in a lateral view, one eye or ear may not be detected, introducing inconsistencies and noise into the feature representation which ultimately results in lower classification accuracy. By retaining 13 keypoints, each representing distinct and stable body part positions, our system preserves the most informative information for effective classification. Fig. 3 visualizes keypoints detection and tracking on the UT dataset. Red bounding boxes indicate ROIs and colored markers represent individual keypoints, track ids and the bounding boxes of individuals.

The following Algorithm 1 is used in our system to filter out irrelevant individuals using the ROI boxes during segmentation and keypoint detection, so features of only relevant individuals are extracted later:

Our system captures both spatial and temporal features from gray scale silhouettes and keypoints. This ensures the extraction of features essential for accurate classification. The details are provided below:

Long Short-Term Memory Networks (LSTMs) have been widely used for activity classification, due to their ability to model temporal dependencies across frame sequences. They use gating mechanisms to control the flow of information. The input gate regulates how much new information is added to the cell state, while the forget gate controls how much of the previous cell state should be forgotten. The candidate cell state determines the potential update to cell state, and the output gate decides how much of the current cell state is exposed to the next layer. These interactions collectively update both the cell state and the hidden state at each time step.

Bidirectional LSTMs (BiLSTMs) process sequences in both forward and backward directions, showing better performance than regular LSTMs. At each time step, hidden states from both passes are concatenated, and the final representation combines the last forward and backward states, yielding rich contextual features for classification. We use BiLSTM for this very reason. This final representation is fed into a fully connected layer and softmax to generate class probabilities. For our experiments, we explored various BiLSTM hyperparameters to achieve the best possible accuracy. Table 1 shows the hyperparameters of BiLSTM that gave the best accuracies on the UT interaction (UT) [14], SBU kinect interaction (SBU) [3] and ISR-UoL 3D social activity (ISR-UoL-3D) [16] datasets used in our experiments.

Evaluation of the proposed system is performed on three benchmark datasets: UT interaction (UT) [14], SBU kinetic interaction (SBU) [3] and ISR-UoL 3D social activity (ISR-UoL-3D) [16] dataset. Details of these datasets are as follows:

We now evaluate our proposed system by comparing it with state-of-the-art systems. The results in Table 12 demonstrate that our system performs better than most existing approaches, highlighting its effectiveness. However, it underperforms than [11] on the UT and SBU datasets and [10] on the SBU dataset. Specifically, [11] used OpenPose [8] for keypoint detection, which is much more accurate but considerably more computationally demanding than YOLO11-Pose used in our system. Also, [10] relied on ground truth keypoint coordinates, giving it an inherent advantage. These observations suggest potential directions for further improving our system to achieve even stronger results.