Image-based scene graph prediction is an extensively researched field, with notable advancements in modeling semantic relationships between objects within images. However, there has also been a recent surge in research on 3D-based scene graph prediction using point clouds. Table 1 shows a comparative analysis of the 3D scene graph prediction models. The pioneering work in 3D scene graph prediction is from the 3DSSG framework [15], which introduced the 3DSSG generation dataset leveraging the 3RScan dataset and a neural network named SGPN, which combines Graph Convolution Networks (GCNs) and PointNet [16]. The SGPN model utilizes GCNs to generate 3D graphs efficiently, but its performance is limited in environments with long-tail distributions. The SGGpoint model [19], developed by Zhang et al., employed an EdgeGCN to capture edge-based relationships within point clouds. Further improving the reasoning capabilities, Wu et al. [23] introduced the Scene Graph Fusion Network (SGFN), which sequentially generates 3D graphs from RGB-D sequences and integrates them through a Graph Fusion Attention (FAT) mechanism.

Knowledge-based research has also been conducted by scholars such as Zhang et al. [22] who proposed a knowledge-inspired network for embedding labels via meta-embedding and intervening features in scene graph prediction models. Feng et al. [31] used hierarchical symbolic knowledge to leverage external knowledge to improve the model’s classification performance for ambiguous relationships. The SGFormer [28] model employs an LLM to enhance the visual features of objects by leveraging knowledge from the semantic injection layer. Lang3DSG [29] inserted natural language information into the model by encoding objects and relationships as text using CLIP [30]. Similarly, visual-linguistic semantics-assisted training (VL-SAT) [24] overcomes the limitations of the existing point clouds by inserting natural language information through CLIP and using additional image data to provide visual information. These models require both point cloud and additional image modalities to improve performance, but their reliance on extra input data can be a limitation in real-world applications where only point cloud data may be available.

Meanwhile, SGRec3D [32] proposed a method for effectively training limited point-cloud data by using an autoencoder for pre-training.

Few-shot learning has recently garnered significant attention for training models with limited labeled data. One of the pioneering works in this domain is Matching Networks proposed by Vinyals et al. [33], which employs an attention mechanism to compare a small number of labeled examples with the query set, leveraging the concept of support and query samples to perform classification. Another notable approach is Prototypical Networks proposed by Snell et al. [34], which represents each class with a prototype, typically the mean of its support set, and classifies queries based on their proximity to these prototypes in the embedding space. Further advancements include model-agnostic meta-learning (MAML) by Finn et al. [35], which trains models to enable rapid adaptation to new tasks with few gradient steps. Sung et al. [36] improved the performance of relation networks in few-shot scenarios using a learnable deep distance metric to compute the similarities between samples. Recent work has also integrated transformer architectures, as seen in the few-Shot Transformer by Ye et al. [37], which captures long-range dependency and context. MetaOptNet by Lee et al. [38] combined optimization-based meta-learning with support set regularization to enhance the performance on standard benchmarks. Meanwhile, there is also a study that addresses the Few-Shot Class-Incremental Learning (FSCIL) problem by proposing a Filter Bank Network (FBN) [39], which augments learnable convolution filters instead of data, thereby effectively integrating new classes.

As input, we take a point cloud P∈RN×3 with N 3D points and with a set M∈{M1,M2,...,Mk} of k semantic instance masks described in 3DSSG [15]. We aim to generate a 3d semantic scene graph G={O,R}. The set of objects O={oi}i=1K represents the classification result of a point cloud P separated by a mask M. The set of relations R={rij}i,j represents the classification of predicates in a relational triple ⟨subject,predicate,object⟩ whose subject is oi and whose object is oj.

The encoder comprises a Node Encoder for objects and an Edge Encoder for relations. Node Encoder extracts the initial node ϕo∈Rk×c and the Edge Encoder extracts the initial edge ϕr∈Rk×c. As input, the node encoder takes a semantic instance Pi∈RN×3 extracted using the mask mi∈M. We apply this to PointNet [17] to extract the features the point cloud, in the process extracting a c-dimensional representation of the object’s initial features ϕn.

The edge encoder uses the same approach as the SGFN [23]. It extracts various features between the semantic instance pi of subject and the semantic instance pj of object in the relational triple ⟨subject,predicate,object⟩ and passes them through the fully connected layer. Eq. (1) calculates the edge-encoding process. μ and σ are the mean and standard deviation of 3D points in each instance, respectively, b=(bx,by,bz) is the size, ν=bxbybz is the volumn, and l=max(bx,by,bz) is the maximum side length. cat(⋅) is concatenate and MLP(⋅) is fully connect layer. The initial edge feature, ϕr,ij, is the edge encoder’s output. (1)ϕr,ij=MLP(cat(μi−μj,σi−σj,bi−bj,lnlilj,lnνiνj))

For message propagation between the initial nodes ϕn and edges ϕr, we applied GNN-based structures such as VL-SAT [24] and SGFN [23]. We enhanced the information of neighboring features through two message-passing layers. Eqs. (2) and (3) respectively show the nodes’ and edges’ message-passing processes. ℓ denotes the message parsing layer, go(⋅),gr(⋅) denote fully connected layers. FAN(⋅) is the feature-wise attention network (FAN) proposed by SGFN [23]. (2)oiℓ+1=go([oiℓ,max(FAN(oiℓ,rijℓ,ojℓ))]) (3)eijℓ+1=gr([oiℓ,rijℓ,ojℓ])

The FAN applies FAT as a multi-head approach [40]. Eq. (4) shows the FAT, which takes a query Q and a target T as inputs, where ga(⋅) is a fully connected layers and ⊙ is an elementwise multiplication. Q is passed through ga(⋅) and normalized using softmax. It then performs element-wise multiplication with T. Input features Q and T are divided into h heads, and an attention function is applied, as shown in Eq. (5). Each head is then concatenated. Finally, multi-head feature-wise attention (MFAT) was applied to define FAN, as shown in Eq. (6). g^q(⋅), g^e(⋅), g^t(⋅) is a single fully connected layer. (4)FAT(Q,T)=softmax(ga(Q))⊙T (5)MFAT(Q,T)=[FAT(qi,ti)]i=1h (6)FAN(oi,rij,oj)=MFAT([g^q(oi),g^e(eij)],g^t(oj))

Fusion prototypical loss is designed by combining traditional classification loss and prototypical loss. The traditional classification loss is effective for classifying many labels, whereas the prototypical loss is better suited for sparse labels. In Section 3.4.1, we introduce prototypical loss, and in Section 3.4.2, we describe the fused loss.

We experimented with the 3DSSG data [15]. This dataset was based on the 3RScan dataset [16], which contains 3D scene data annotated using 3d semantic scene graphs. The dataset contained 1553 indoor 3D scenes with masks per instance, 160 object classes, and 26 predicate classes as labels. We used the same training/validation split as 3DSSG [15]. The 3DSSG dataset has a extremely long-tailed distribution for both objects and predicates. Fig. 3 shows a graph of the number of data points per class for objects and Predictates in the training data, sorted in descending order. In the objects class shown in Fig. 3a, approximately half of the classes had 50 or fewer data items. The predicates in Fig. 3b also exhibits a pronouncedly long-tailed distribution. These distributions skew the model, which is why it is critical to design a model that performs equally well across all classes.

In conformance with the experimental setup detailed in 3DSSG [15], the 3D scenes were consistently placed within the same coordinate system during both the training and testing phases. To assess the accuracy of object and predicate predictions, we employed the top-k accuracy (A@k) metric [24]. To evaluate the triplets, we calculate triplet scores by multiplying the scores of the subject, predicate, and object, and subsequently determined A@k as the evaluation criterion. A triplet is deemed accurate only if all its components–subject, predicate, and object–are correctly identified. To provide a balanced assessment of performance with a long-tailed distribution, we computed the average top-k accuracy, named the average top-k accuracy (mA@k), over all predicate and object classes [24].

Our network had a batch size of eight, and used the AdamW optimizer [41,42]. We trained for 100 epochs on an NVIDIA A100 GPU. This took approximately 48 hours. The learning rate was 0.0001 and we followed a cosine annealing learning rate strategy. The Pytorch platform was used [43] with the parameters set as follows: λobj=λproto=0.1, λrel=1. We referenced the object (λobj) and predicate (λrel) learning weights used in our previous work (VL-SAT) [24]. The newly added prototype loss ℒProto is used to improve object performance, which plays a similar role to ℒobjoracle in vl-sat [24]. Therefore, we set λproto to equal the hyperparameter weights of ℒobjoracle. The point cloud was sampled using 128 points, and all embedding vectors inside the model including the prototype dimension, were set to 512 dimensions.

Table 2 shows the performance of our model compared to state-of-the-art models. The VL-SAT model is multimodal, and the rest are single models, such as ours. The base model is similarly to the SGFN [23] and non-VL-SAT [24] models. The FPN was trained by applying a prototypical loss in to the base model. Overall, the base model appeared to perform similarly to the SGFN model. Object performs somewhat worse than the SGFN model, and predicate performs somewhat worse, but slightly better regarding average top-k accuracy. This suggests that the base model performs slightly worse than SGFN but is more robust to long-tailed distributions. Triplet performed poorly overall compared to SGFN. Comparing the base model to our FPN, we observed a substantial performance increase across objects, predicates, and triplets. Object exhibited performance improvement of A@1 1.37, A@5 1.5, and A@10 1.84. Predicate performed similarly for A@K, but shows a substantial improvement in average top-k accuracy with mA@1 3.66, mA@3 4.67, and mA@5 4.3. The triplet also shows an overall performance improvement, especially in the average top-k accuracy, with mA@50 of 11.38 and mA@100 of 11.98. This indicates that embedding the node features resulting from graph reasoning into the prototype space improves the overall performance of the model. The improvement in average top-k accuracy across the different parts also shows that our FPN can train well on data with long-tailed distributions. This trend is similar for VL-SAT multimodal model, showing that embedding in a prototype space allows a single model to extract as much information as a multimodal models.

In this section, the experimental reports are divided into three categories based on the number of objects and predicates per class: Head, Body, and Tail, respectively. In addition, we applied the proposed fusion-loss method to the SGFN [23] and VL-SAT [24] models. For experimental fairness, only 3D models were used for the VL-SAT multimodal model.

Table 3 shows that for objects, applying fusion loss improves the overall performance. Except for the mA@1 metric of the body part in SGFN and the mA@5 metric of the head part in VL-SAT, the performance improved. In particular, the model with FPN in the tail part shows a substantial performance improvement, with mA@1 of 0.46, mA@5 of +4.78, and mA@10 of +2.16 for SGFN, and mA@1 of +1.86, mA@5 of +6.79, and mA@10 of +16.89 for VL-SAT. These substantial performance improvements in the tail part demonstrate that FPN can capture the features of sparsely labeled objects in a general scene graph prediction model.

Objects show consistent performance gains per backbone, while predicates show average performance gains but no consistency in performance gains per backbone. This suggests that the model does not mitigate backbone-specific architectural disparities, which is probably due to the model primarily focusing on object prototypes.

In this section, we compared the base model, ground truth, and our proposed FPN. The input point cloud, the ground-truth graph, the base model’s prediction graph, and the proposed FPN’s prediction graph are depicted in Fig. 4. Each label denotes an object, and each line represents its relationship with other objects. The arrowheads signify the directions of the linkages. A red arrow indicates an incorrect prediction of no relationship. The base model incorrectly predicted three nodes, while FPN incorrectly predicted four nodes. In some cases, our method misclassified sparse classes like “toilet” as many classes like “chair”. In general, FCN performed better on the edges. The base model predicted numerous “no relationship” overall, and our methods occasionally incorrectly predicted some relationships.