Frames from diverse RGB datasets (offices, homes, clinical settings) undergo a standardized preprocessing pipeline for spatial and intensity consistency. Each frame I∈RH×W×3 is resized via bilinear interpolation to I′∈RH′×W′×3, ensuring spatial continuity and color fidelity. Pixel intensities are then normalized using min–max scaling using Eq. (1). (1)In(x,y,c)=I′(x,y,c)−min(I′)max(I′)−min(I′),c∈{R,G,B},which maps values to [0,1] and mitigates illumination and contrast discrepancies. The preprocessing step is illustrated in Fig. 2. Finally, Gaussian smoothing is applied to suppress sensor noise and compression artifacts using Eq. (2), where the kernel is defined as Eq. (3).(2)If(x,y)=∑i=−kk∑j=−kkG(i,j)⋅In(x+i,y+j), (3)G(i,j)=12πσ2exp⁡(−i2+j22σ2)

This preprocessing ensures uniform scale, normalized intensity distribution, and improved visual quality, providing consistent inputs for subsequent multimodal feature extraction.

Semantic instance segmentation is applied to each RGB frame using Mask R-CNN with a ResNet-50 backbone and FPN, implemented in Detectron2, to decouple human and environmental components. The network, pretrained on COCO, produces instance-specific binary masks {Mi}i=1N, class labels {yi}i=1N, and confidence scores {si}i=1N from an input image I∈RH×W×3. The overall multi-task loss integrates classification, bounding box regression, and mask prediction using Eq. (5). (4)𝒮:I↦{(Mi,yi,si)}i=1N,Mi∈{0,1}H×W,yi∈𝒞 (5)Ltotal=Lcls+Lbox+Lmask,here, the classification loss is the softmax cross-entropy by Eq. (6), the bounding-box regression loss employs Smooth L1 using Eq. (7), and the mask prediction loss applies pixel-wise binary cross-entropy while region proposals are generated by the RPN using classification and bounding-box regression losses. (6)Lcls=−∑k=1|𝒞|yklog⁡p^k, (7)Lbox=∑i∈{x,y,w,h}smoothL1(ti−t^i),smoothL1(x)={0.5x2,|x|<1,|x|−0.5,otherwise,

For human-centered analysis, the largest connected component across all predicted instance masks is selected as the primary human silhouette using Eq. (8). (8)M∗=arg⁡maxCj⊆⋃Ni=1MiArea(Cj),where {Cj} denotes the set of connected components. Remaining components correspond to scene objects and are preserved for downstream context-aware reasoning (e.g., affordance learning, pose–scene interaction). An example of this segmentation is shown in Fig. 3.

Following segmentation, the human is isolated as an RGB silhouette, serving as input for human-centric feature extraction. From this silhouette, 2D pose keypoints and a 3D point cloud are derived, capturing spatial, structural, and geometric cues essential for motion-based activity analysis.

Keypoint Extraction via MoveNet

To capture the articulation and spatial configuration of the human body, we employ MoveNet [8], a lightweight yet accurate real-time pose estimator that predicts 17 anatomical keypoints, denoted as K={ki}i=117, where each keypoint is ki=(xi,yi,ci). Where (xi,yi) are 2D coordinates and ci∈[0,1] is the confidence score. These landmarks span the head, torso, and limbs, enabling fine-grained skeletal representation. The estimation process leverages multiple loss functions to ensure spatial coherence, anatomical plausibility, and robustness under occlusion.

Keypoints are represented as 2D Gaussian heatmaps, and their spatial locations are optimized using mean squared error. Coordinate-level refinement is performed via Smooth L1 regression, while confidence-weighted losses mitigate the influence of occluded or low-confidence keypoints. These losses collectively enforce accurate, anatomically plausible, and robust keypoint estimation under varying visual conditions.

Structural Consistency Loss: Limb geometry is preserved via relative displacement as provided in Eq. (9).(9)Lstruct=∑(i,j)∈ℰ∥(ki−kj)−(k^i−k^j)∥2,where ℰ denotes anatomically connected keypoint pairs. Bone-Length Consistency Loss maintains anthropomorphic proportions.

Laplacian Smoothness Loss encourages local pose smoothness and is calculated using Eq. (10). (10)Llap=∑i=1N∥ki−1|𝒩(i)|∑j∈𝒩(i)kj∥2where 𝒩(i) is the set of adjacent joints for keypoint i. Scale-Invariant Loss normalizes errors by a reference body scale using Eq. (11). (11)Lscale=1s2∑i=1N∥ki−k^i∥2,s=∥kshoulder−khip∥,

To enhance robustness, the keypoint schema was refined to 13 landmarks, with the nose serving as the cranial proxy and the shoulder midpoint defining the torso vector. Each 2D keypoint is mapped to pseudo-3D by sampling MiDaS depth, normalizing to a reference joint, and combining with 2D coordinates to form pseudo-3D joints. We approximate (Xi,Yi,Zi) by (xi,yi,zi) with zi from MiDaS, producing a sparse 3-D point cloud of skeletal landmarks. Followed by root-relative normalization with the pelvis joint kroot to remove global translation and standardize scale using Eq. (12). (12)k~i=ki−kroots,s=maxi,j∥ki−kj∥2

The resulting normalized descriptor, fpose=[k~1,k~2,…,k~13], was subsequently used for classification and temporal modeling. The extracted key body points and corresponding silhouette are illustrated in Fig. 4.

To infer 3D structure from monocular silhouettes, we use MiDaS [5], a state-of-the-art scale-invariant depth estimator suited for silhouettes without metric references. Depth is optimized using the scale-invariant loss in Eq. (13). (13)Lsi=1n∑i(log⁡D^i−log⁡Di∗+α)2−1n2(∑i(log⁡D^i−log⁡Di∗+α))2,α=1n∑i(log⁡Di∗−log⁡D^i)

Predicted log-depth values are mapped back as D^(u,v)=exp⁡(fθ(I(u,v))), where fθ denotes the network output for pixel (u,v). The depth map is lifted into a 3D point cloud using the intrinsic matrix, with multiple maps fused by confidence weighting (Eq. (14)). (14)D^fused(u,v)=∑iwi(u,v)⋅D^i(u,v)∑iwi(u,v),wi(u,v)=exp⁡(−σi(u,v)2),where wi(u,v) encodes pixel-wise reliability. Surface normals are estimated using Eq. (15), providing local geometric consistency. (15)nuv=(Pu+1,v−Puv)×(Pu,v+1−Puv)∥(Pu+1,v−Puv)×(Pu,v+1−Puv)∥

To suppress unreliable 3D points, a depth confidence function is defined using Eq. (16). (16)C(u,v)=exp⁡(−|D^(u,v)−D¯|/σD)where D¯ and σD denote the mean and standard deviation of predicted depths, respectively. This formulation ensures that the resulting point cloud remains anthropometrically realistic, as illustrated in Fig. 5.

Environmental context is modeled by analyzing non-human scene components via semantic segmentation. AKAZE detects keypoints and MSER extracts regions, with features computed only on non-human masks to remain independent of the actor’s silhouette. Both AKAZE and MSER feature visualizations are illustrated in Fig. 6.

The second stage extracts geometric and frequency descriptors: surface normals yield histograms, directional vectors, and Zernike moments, while Gabor filtering captures structural and textural cues without explicit segmentation.

To comprehensively encode human motion and scene context, SCENET-3D constructs a composite latent embedding by jointly modelling skeletal dynamics, object-centric structure, and global environmental signatures. Let S∈Rds, O∈RdO and G∈Rdg denote the skeleton-, object-, and global-scene descriptors, respectively. Each modality is first normalized and projected into a shared latent space of dimension mmm via trainable linear maps s∼=WsS+bs, o∼=WoS+bo, g∼=WgS+bg with W(⋅)∈Rm×d(⋅). Introducing non-negative modality weights α=[αs,αo,αg]⊤, the fusion operator F:Rm×Rm×Rm→Rm is defined using Eq. (23). (23)F=σ(αss∼,αoo∼,αgg∼+∑i,j∈{s,o,g},i<jTij(i∼⊙j∼))where ⊙ denotes element-wise interaction capturing cross-modal correlations, Tij∈Rm×m are bilinear interaction matrices, and σ(·) is a pointwise non-linearity. If x=[s∼⊤,o∼⊤,g∼⊤]⊤∈R3m denotes the concatenation of projected modalities, then this fusion can also be written compactly using Eq. (24). (24)F=σ(Wfx+x⊤Kx)

With Wf∈Rm×3m linear fusion matrix K∈R3m×3m a symmetric kernel capturing all second-order cross-terms. The resulting F∈Rm constitutes the composite latent embedding unifying pose, object, and scene cues. Fusing human motion and environmental semantics, the vector is processed by a transformer encoder with multi-head self-attention. For an input sequence X∈Rn×d, queries, keys, and values are computed using Eq. (25): (25)Q=XWQ,K=XWK,V=XWV

The attention mechanism is then defined using Eq. (26). (26)Attention(Q,K,V)=softmax(QKTdk)V

Multi-head attention aggregates h parallel attention heads given in Eq. (27).(27)MHA(Q,K,V)=Concat(head1,…,headh)WOwith headi=Attention(QWQi,KWKi,VWVi). The encoder output Z is passed through a feed-forward network with residual connections and normalization.

Finally, the activity class probabilities are computed using a softmax layer using Eq. (28).(28)y^=softmax(Wc⋅zcls+bc)where zcls is the classification token embedding. This fusion–transformer pipeline enables robust recognition by jointly reasoning over pose and scene features, effectively disambiguating visually similar activities in cluttered environments.

As given in Table 1, the transformer-based model was optimally configured with 6 encoder layers, a model dimension of 768, and a feed-forward hidden size of 3072, using GeLU activation and a dropout rate of 0.1. Each encoder layer employed 12 attention heads with 64-dimensional per-head projections and attention dropout of 0.1. Layer normalization was applied in a pre-norm configuration with ε set to 1e−5. The fully connected classification head comprised two hidden layers of 512 neurons each with ReLU activation, followed by a Softmax output layer for 10 classes, resulting in a total parameter count of approximately 199k. This setup balances model complexity and performance, providing effective representation learning while controlling overfitting.

All experiments for SCENET-3D were conducted in a cloud-based environment using Google Colab kernels, which provided access to an NVIDIA Tesla T4 GPU (16 GB GDDR6, 2560 CUDA cores, Turing architecture), a dual-core Intel Xeon CPU (2.20 GHz), and 27 GB of system RAM. Storage and dataset management were handled through Google Drive integration with temporary VM scratch space for caching. The system comprising the Transformer backbone, pseudo-3D skeleton encoder, and scene-object fusion module—was implemented in Python 3.10 with TensorFlow 2.6.0 and the Keras API, accelerated using CUDA 11.0 and cuDNN 8.0 for parallelized training and inference. NumPy, Pandas, and OpenCV supported numerical processing, structured data handling, and augmentation, while Matplotlib, Seaborn, and TensorBoard were used to visualize learning dynamics.

We evaluated on three benchmarks: N-UCLA Multiview Action 3D (RGB, depth, skeletons of 10 actions across three Kinect views), ETRI-Activity3D (55+ daily activities with RGB-D and motion capture), and CAD-120 (10 complex activities with RGB-D, skeletons, object tracks, and sub-activity labels). These datasets cover diverse modalities and environments for comprehensive evaluation.

The UCLA confusion matrix (Fig. 10) shows strong diagonal dominance, with most activities (Ex1, Ex4, Ex5, Ex6, Ex9, Ex10) above 94% accuracy. On ETRI-Activity3D (Fig. 11), the model achieves average precision, recall, and F1 of 0.8058, 0.8056, and 0.8047. Distinctive activities (Ex49–Ex55) exceed 0.94, while overlapping motions (Ex2, Ex7, Ex8, Ex29) lower discriminability. Well-defined actions (Ex6, Ex28, Ex38–Ex55) show near-perfect recognition, highlighting the need for temporal modeling or data augmentation for ambiguous classes. On CAD-120 (Fig. 10), the framework achieves >90% accuracy for Cleaning, Having a meal, Microwaving, Putting, Stacking, and Taking objects.

ROC curves offer threshold-independent evaluation of TPR–FPR trade-offs. UCLA curves cluster near the top-left, showing near-perfect separability. CAD-120 also approaches the ideal, confirming robust recognition of structured activities. ETRI shows greater variability, with lower sensitivity at low FPRs due to subtle motions and complex backgrounds. Overall, UCLA performs best, CAD-120 is balanced, and ETRI needs stronger feature discrimination (Fig. 11).

Table 2 shows performance across three benchmarks. UCLA achieves the highest accuracy (95.8%) with balanced precision, recall, and F1. ETRI scores slightly lower due to variability, while CAD-120 attains competitive metrics, demonstrating strong generalization to structured activities. All experiments on UCLA, ETRI-Activity3D and CAD-120 were repeated five times with different random seeds, and Table 2 reports mean ± standard deviation for accuracy, precision, recall and F1-score. These statistics provide direct estimates of variability and thus approximate 95% confidence intervals. We further performed paired two-tailed t-tests comparing our full model with the strongest baseline for each dataset. All improvements were significant at p < 0.05, confirming that the reported gains are statistically robust rather than due to random variation.

Ablation results (Table 3) show that removing preprocessing reduces accuracy by ~5%–6%, while excluding point clouds or MoveNet keypoints causes the largest drop (20%–25%), confirming the necessity of 3D structure and skeleton cues. Object-wise scene analysis adds 8%–12%, and surface normals plus Gabor features contribute 3%–5%, refining recognition in cluttered settings.

SCENET-3D employs a three-stage pipeline scene analysis, scene-object analysis, and human-pose estimation combining handcrafted descriptors, transformer-based representations, and pseudo-3D point clouds for robust, interpretable understanding. Stage one captures holistic scene attributes (lighting, geometry, texture) via Zernike moments and Gabor filters. Stage two localizes objects using AKAZE and MSER keypoints to preserve local geometric and textural features. The transformer encoder enhances handcrafted and point cloud features by modeling cross-modal and temporal dependencies among skeleton, scene-object, and 3D spatial representations. This context-aware reasoning helps disambiguate visually similar actions. Ablation studies show that removing object, global scene, or point cloud features reduces accuracy by 8%–12%, 3%–5%, and 5%–7%, respectively, highlighting the importance of integrating local, global, temporal, and spatial cues. This hybrid approach outperforms pipelines using only deep learning, handcrafted, or point-cloud features.

Table 4 highlights the significant variation in computational demands across different vision modules. Tasks like semantic segmentation and point-cloud processing are highly resource-intensive, reflecting their complexity in analyzing large-scale scene and spatial data. In contrast, modules such as region detection, feature extraction, and pose estimation are relatively lightweight, offering efficient processing for targeted analyses. Intermediate techniques, like multi-scale Gabor filtering, balance complexity and efficiency, providing rich feature representations without excessive computational cost.

Table 5 compares classification accuracies across state-of-the-art methods. On UCLA, several models exceed 90%, with the proposed system achieving the highest (95.8%). For the more complex ETRI-Activity3D dataset, accuracies are lower overall, yet our approach leads with 89.4%. On CAD-120, results are more balanced, with the proposed method slightly surpassing prior work, demonstrating robustness across varying dataset complexities.