In the DRE task, a dialogue d is defined as a tuple (S,U,A), comprising a set of speakers S={sk}k=1K, a set of utterances U={un}n=1N and a set of entity pairs A={(ei,ej)g}g=1G, where un={sk,xn,1,xn,2,...,xn,M}, xn,m refers to the m-th token in the n-th sentence spoken by the k-th speaker sk, and ei and ej refer to the subject and object of the g-th entity pair, respectively. Given an argument pair (ei,ej) within the dialogue d, the goal of DRE is to predict the relations ri,j∈R between ei and ej, where R refers to the relation set of the dataset. Additionally, the datasets often include several useful words from the sentences as triggers to assist in relation prediction for specific argument pairs.

To effectively filter out conflicting features and further fuse the consistent features of the arguments and triggers, we propose the MF2F model. This model comprises five modules that form a cohesive pipeline, as illustrated in Fig. 2.

The first module is the dialogue encoding module, which utilizes a pre-trained language model with the mask-based multihead self-attention mechanism to obtain three scales of dialogue token embeddings. The second module is the argument encoding module, where an attention mechanism aggregates different mentions to form the argument embedding. Using the three scales of dialogue token embeddings, we derive three corresponding argument embeddings. The third module is the trigger prediction module, which includes a discriminator followed by an average pooling layer to identify trigger tokens and generate their original prototype embeddings for a given argument pair. The fourth module is the feature filtering module, which constructs a filter unit to eliminate conflicting features between the argument and the trigger. The filtering unit primarily utilizes average pooling to preserve consistent features between the original representation and the filtering template while discarding inconsistent features, which are likely the culprits of conflict. Specifically, for a trigger and its corresponding two arguments, when one of the three requires feature filtering, the representations of the other two are aggregated into a template via a fully connected layer. Finally, in the prediction and training module, the filtered trigger and argument representations are combined and fed into a relation classifier for RE.

We then incorporate a relation classification loss alongside the trigger prediction loss to optimize the model, transforming our MF2F into a multitask model and further enhancing its performance in the DRE task. Specifically, before training, we utilize prompt tuning with an LLM to automate the annotation of triggers for samples lacking manual annotations, thereby addressing the issue of insufficient annotated triggers in the dataset.

In this section, we obtain three types of dialogue encodings: global, local, and speaker-based. The first two encodings capture dependencies between tokens at the dialogue and utterance levels, respectively, while the third focuses on token dependencies based on whether the utterances are spoken by the same person.

First, we concatenate the utterances and special tokens into a long sequence X: (1)X=[[CLS],x1,1,...,x1,M1,...,xN,1,...,xN,MN,[SEP]]where [CLS] and [SEP] serve as special tokens indicating the classification and the end of the dialogue, respectively. This dialogue X is then fed into a pre-trained model [38,39], such as RoBERTa, to obtain the token embeddings H: (2)H=RoBERTa(X)=[h[CLS]′,hx1,1′,...,hx1,M1′,...,hxN,1′,...,hxN,MN′,h[SEP]′]

Next, we add the sinusoid position information Hpos and argument type information Htype to H to generate the global token embeddings Hg: (3)Hg=H+Hpos+Htype=[h[CLS],hx1,1,...,hx1,M1,...,hxN,1,...,hxN,MN,h[SEP]]

To capture local interactions between a token and others within the same utterance or neighboring utterances, we design a local mask within the multihead self-attention mechanism [40]: (4)Hq=[head1q;…,headiq;…,headNaq] (5)headiq=softmax(HgWiQ⋅(HgWiK)Tdk+Mq)Hgwhere WQ∈Rd×dq, and WK∈Rd×dk are trainable parameters, and Mq represents the local mask. There are two types of local masks: one is the self-local mask Msl, that is defined as: (6)Msl[i,j]={0,if the ith and the jth token belongs to the same utterance−∞,otherwise

The other is the neighbor-local mask Mnl, that is defined as: (7)Mnl[i,j]={0,if the ith and the jth token belongs to the adjacent utterance−∞,otherwise

We calculate self-local token embeddings Hsl and neighbor-local token embeddings Hnl using Eqs. (4) and (5) with respective local masks. These embeddings are then fused together to generate the local token embeddings Hl using a gate mechanism: (8)Hl=Gl∘Hsl+(1−Gl)Hnl (9)Gl=sigmoid(FC(Il))where Il=[Hsl;Hnl;Hsl−Hnl;Hsl∘Hnl]; and FC() represents a fully connected layer.

Because the speaker information is particularly relevant in dialogue encoding, we employ two speaker-based masks: the self-speaker mask Mss [41] and the other-speaker mask Mos as defined in Eqs. (4) and (5), to obtain embedding Hss and Hos, respectively. The two speaker masks are defined as: (10)Mss[i,j]={0,if the ith and the jth token is spoken by the same speaker−∞,otherwise (11)Mos[i,j]=−∞−Mss[i,j]

Finally, self-speaker token embeddings Hss and other-speaker token embeddings Hos are blended into the speaker-based token embeddings Hs using a gate mechanism similar to that in Eqs. (8) and (9).

After obtaining different scales of dialogue encodings, we use them to calculate three types of argument encodings: global argument encoding, local argument encoding, and speaker-based argument encoding. While these encodings follow the same calculation process, they differ in the type of dialogue encoding they utilize.

To capture the various dependencies between an argument and the other mentions, we apply the mutual attention mechanism to obtain the global encoding heig, the local encoding heil and the speaker-based encoding heis for the argument ei. Let heiq represent a specific embedding of the argument ei, where q∈{g,l,s}. It is defined as follows: (12)heiq=∑m=1Niαj,i,m⋅hi,mqwhere Ni represents the number of mentions for the argument ei, and hi,mq represents the embedding of the m-th mention of ei. This is calculated as follows: (13)hi,mq=1bm−cm+1∑z=bmcmhzqwhere hzq represents the token embedding corresponding to the j-th mention of ei; and z is the index of the token in the dialogue encoding, ranging from the start bm to the end cm. αj,i,m represents the attention weight between hi,mq and the embedding of the other argument ej in a given argument pair. It is defined as: (14)αj,i,m=exp(h′ejq⋅hi,mq)∑n=1Niexp(h′ejq⋅hi,mq) (15)h′ejq=1Nj∑m=1Njhj,mqwhere hj,mq represents the embedding of the m-th mention of ej.

Based on previous studies [16,42], most triggers are notional words, including nouns, verbs, and adjectives. We collect noun, verb, and adjective tokens as candidate triggers. A discriminator is then constructed to calculate the probability P of each candidate c being a trigger token for a given entity pair ei and ej. The probability P is calculated as follows: (16)p(c|ei,ej)=sigmoid(FC(hc′)) (17)hc′=[hcg;heig;hejg]where heig and hejg represent the global embeddings of ei and ej, respectively, and hcg represents the global embedding of c. We select the top-Kt candidates with the highest probabilities as the true triggers, which are then used to obtain their original prototype trigger embedding hei,ejt using an average pooling layer as: (18)hei,ejt=1Kt∑k=1Kthkgwhere hpg represents the global embedding of the k-th final predicted trigger token.

Before combining the embeddings of the trigger with those of the arguments for relation classification, it is essential to filter out features that may mutually influence each other. On the one hand, because multiple relations often share the same triggers, the features of a trigger carry semantic information from various relations. On the other hand, an argument may belong to more than one relation, meaning its features also encompass semantic information from diverse relations. This overlap can lead to incorrect classification outcomes if the full features of the trigger and argument pair are used. To address this issue, we propose a filtering unit. Its core technique is average pooling, which retains consistent features from the original vectors while filtering out inconsistent ones. Given a trigger t and two arguments ei and ej, the feature filtering process for t is defined as follows: (19)ζei,ejt=12(hei,ejt+hei,ejg) (20)hei,ejg=FC([heig;hejg])where hei,ejt represents the prototype embedding of t; heig and hejg represent the global embedding of ei and ej, respectively; and hei,ejg represents the filtering template generated by heig and hejg. Similarly, the feature filtering process for ei and ej is defined as follows: (21)ζeiq=12(heiq+hej,tq) (22)hej,tq=FC([hejg;ζei,ejt])where hejq represents one of the local embedding and speaker-based embedding for the argument ej, i.e., q∈{l,s}; and hej,tq represents the filtering template created by heiq and ζei,ejt.

We combine the results of relation existence prediction and relation classification to make the final prediction.

For relation existence, we use a binary classifier to evaluate the relation between the two given arguments ei and ej: (23)flagexist={Trueif Pexist≥0.5Falseotherwise (24)Pexist=Binary Classifier(X)where X represents the dialogue tokens sequence.

For relation classification, we concatenate the filtered embeddings and input them into a multiclass classifier to predict relations for the argument pair: (25)r^class={i∈{1,…,r,…,|R|},pi≥0.5} (26)P=[p1,…,pr,…,p|R|]=sigmoid(FC(Hcon)) (27)Hcon=[ζeil;ζejl;ζeis;ζejs;ζei,ejt]where r^class represents the predicted relations for relation classification; pr denotes the probability of the argument pair belonging to the r-th relation; ζei,ejt is the filtered embedding of trigger t; and ζeq represents one of the two embeddings of argument e (i.e., q∈{l,s}).

The final RE result is obtained as follows: (28)r^={r^classif flagexist=True and r^class≠ϕunanswerableotherwisewhere “unanswerable” represents that the argument pair has no relation.

We then adopt both relation classification loss and trigger prediction loss to optimize the model, transforming our MF2F into a multitask model, which enhances performance in the DRE task. The final loss of our model is defined as follows: (29)loss=α⋅lossrel+(1−α)⋅losstriwhere α is a hyperparameter that regulates the balance between the two losses; and lossrel and losstri represent the relation classification loss and the trigger prediction loss, respectively, which is defined as: (30)lossrel=−1|A|∑(ei,ej)∈A∑r∈R[yei,ej,rlog⁡pei,ej,r+(1−yei,ej,r)(1−log⁡pei,ej,r)] (31)losstri=−1|A|∑(ei,ej)∈A1|Tei,ejcan|∑t^∈Tei,ejprewt^(yei,ejt^log⁡pei,ejt^+(1−yei,ejt^)(1−log⁡pei,ejt^))where Tei,ejpre denotes the set of predicted trigger tokens for the given arguments ei and ej; yei,ejt^ represents the true label for the token t^; pei,ejt^=p(t^|ei,ej) calculated by Eq. (15); and wt^ represents a weight that encourages the model to identify tokens with similar semantics to the trigger tokens as supplements. It is defined as follows: (32)wt^={1−scoret^,if scoret^>scoreth1otherwise (33)scoret^=maxt∈Tei,ejtrucos⁡(t^,t)where Tei,ejtru represents the set of true trigger tokens for two given arguments ei and ej; and scoreth represents a threshold that determines whether the semantics between t^ and t is close enough with SBert [43].

Before training, we leverage the strong text comprehension abilities of an LLM to automatically annotate triggers for samples lacking manual annotations. We design prompts based on CoT and ICL to facilitate this task. In constructing the CoT, we break down the trigger annotation process into three subtasks. First, we prompt the LLM to explain why a given argument pair has such a particular relation, encouraging the model to explore the semantics relevant to the task. Second, we prompt the LLM to predict the trigger and assess its accuracy. Third, we prompt the LLM to provide an explanation for each predicted trigger. The prompt also includes a contextual example to help the LLM learn the thought patterns necessary for trigger identification and to master the correct output format. For quality control in predicting triggers [44], we employ the Local Outlier Factor (LOF) [45] to evaluate the quality of the explanations. Outliers are considered poor explanations, and the corresponding triggers are discarded to ensure the acquisition of high-quality triggers.

The dataset used in our experiments is the DialogRE dataset, which comprises dialogues extracted from the American TV series Friends. It includes two versions: DialogREv1 and DialogREv2. DialogRE contains 1788 dialogues, 37 relations, and 8119 relational triplets, the majority of which describe the relations between characters in the dialogues. Notably, annotated triggers are provided for a subset of these triplets. For our experiments, we utilize the standard three partitions of the data: training, development, and testing, as structured in the DialogRE dataset.

To conduct a comprehensive performance evaluation, we compare our model against 11 baselines and SOTA methods, which are categorized into three groups, including Sequence-based models: BERTs [16], RoBERTas [17], CoIn [18] and SimpleRE [19]; Graph-based models: HGAT [21], GDPNet [22], AMR [37] and TUCORE-GCN [17]; and Trigger-enhanced models: TREND [24], GRASP [25], TLAG [29] and KEPT [26]. We use the standard Precision (P), Recall (R) and micro-F1 (F1) to evaluate the model performance: (34)P=TPTP+FP (35)R=TPTP+FN (36)F1=2∗P∗RP+Rwhere TP, FP and FN stand for true positive, false positive, and false negative, respectively. In the rest of the paper, F1 refers to micro-F1 unless otherwise specified.

Our model utilizes AdamW as the optimizer with a Cosine Annealing scheduler, with a weight decay of 1e-3. The pre-trained model RoBERTa-large serves as our encoder, with a learning rate of 5e-6. We trained the model using a batch size of 2 for 30 epochs, with a learning rate of 1e-4 for parameters other than those of the pre-trained model. The multitask learning loss weights are assigned as 0.90 for RE and 0.10 for trigger prediction. We insert special tokens to represent speaker indices between the utterances, forming an input sequence. This sequence is then divided into sub-word tokens. If the length of the tokenized sequence exceeds 512, we split it into two overlapping sub-sequences. Experiments were conducted on a server equipped with two NVIDIA TITAN RTX GPUs, while the environment for LLM-based trigger annotation experiments was based on an Intel Core i7-12700K processor.

As shown in Table 1, our MF2F achieves the best performance among all models, reaching an F1 score of 76.3% on DialogREv1 and 77.0% on DialogREv2. It surpasses the best baseline, GRASP, by 1.2% and 1.5% on both versions of the DialogRE dataset, respectively. This clearly demonstrates the effectiveness of our approach.

Trigger-enhanced models outperform the other two categories. Compared to sequence-based and graph-based models, trigger-enhanced models achieve an average improvement of at least 2.3% in F1 score on the test dataset. This confirms the efficacy of enhancing the DRE model with triggers through multitask learning and feature filtration.

Moreover, our model outperforms all other trigger-enhanced models. In addition to multitask learning and feature filtration, it incorporates feature filtering and LLM-based automatic trigger annotation. These mechanisms further enhance the model’s performance in the DRE task.

To showcase the efficacy of the feature filtering mechanism, we established two ablation scenarios. The first scenario, labeled MF2Frdm_tpl, uses filtering templates from randomly initialized vector, replacing each of the three filter templates with random templates individually and calculating the average results. The second scenario, denoted as MF2Fwot_tpl, utilizes the original embeddings of the arguments and the trigger prototype directly for relation classification, bypassing feature filtering.

As indicated in Table 2, we observe an average drop of 3.2% in the F1 score across the dataset when the feature filtering step is removed, underscoring the effectiveness of this mechanism. Interestingly, even when using a randomly initialized vector to create the filtering template, the model still achieves over 0.8% improvement compared to the scenario without feature filtering, although this does not match the performance of templates derived from the argument or trigger. This suggests that the model can learn a relatively effective filtering template from scratch through optimization, further confirming the utility of the feature filtering mechanism from a different perspective.

We also developed two additional ablation scenarios to assess the effectiveness of our LLM-based automatic trigger annotation strategy. In the first scenario, MF2Fwot_aua, triggers automatically annotated by the LLM are excluded from the training phase. In the second scenario, MF2Fful_aua, all triggers used in the training phase are obtained through automatic annotation by the LLM.

The results presented in Table 2 show an average performance reduction of 1.9% when triggers automatically annotated by the LLM are excluded. Furthermore, the model’s performance drops by 1.4% on average when the triggers are entirely annotated by the LLM; however, it still outperforms the case without any automatically annotated triggers. These findings suggest that LLM-based automatic trigger annotation based is a valid approach for improving model performance in the DRE task.

In trigger-based dialogue relation extraction, the loss function weight is a critical hyperparameter that balances the losses of the two tasks during training. It ensures that while relation extraction remains the primary focus, the trigger prediction task is also optimized. To investigate its impact, we conducted experiments with α values of 0.80, 0.85, 0.90, 0.95, and 1.00 on DialogREv2, while keeping all other parameters constant.

As shown in Fig. 3, experimental results indicate that an excessive or insufficient focus on the relation extraction task negatively impacts performance. When training exclusively focuses on the relation extraction task, the F1 drops to its lowest value of 75.0%.

To evaluate the applicability of the two innovative technologies in our model, we integrated the feature filtering and LLM-based trigger annotation into the recent trigger-enhanced DRE model, TREND. The results are showcased in Table 3. By incorporating the feature filtering into TREND, its performance increases by 4.7%. When applying the LLM-based trigger annotation, TREND’s performance improves by 1.7%. Notably, when both technologies are implemented simultaneously, the performance is enhanced by 5.2%. These findings suggest that both feature filtering and LLM-based trigger annotation can be utilized individually or together in other DRE models to effectively improve their performance.

To demonstrate the working principle of the large model data augmentation method and the feature filtering mechanism in MF2F, we conducted four case studies on the DialogREv2 test dataset. In the Case1, we focus on investigating whether LLMs can extract appropriate trigger words to enhance the training of our MF2F. In the Case2, we focused on scenarios where a single trigger corresponds to two relations, using MF2Fwot_tpl (without filtering templates) and MF2F (with filtering templates) to perform relation prediction on the test set, and visualized the representation space of the relation using principal component analysis (PCA). In the Case3, we focused on scenarios where a single entity corresponds to two relations, using MF2Fwot_tpl and MF2F to perform relation prediction and also visualized the representation space of the relation. In the Case4, to further investigate the capability of MF2F under complex conditions, we divided DialogREv2 based on the number of arguments, and compared the results of MF2F and GRASP under these conditions.

As shown in Table 4, we selected three examples that originally did not have annotation triggers to illustrate the impact of these annotations extracted by LLMs. In Dialogue1, Speaker2 fired Tim, with the LLM extracting the trigger “fire”, directly indicating the boss-employee relation. In Dialogue2, Speaker1 visited Vail, and the LLM’s extraction of “go skiing” suggests that Speaker1 skied during the visit. Dialogue3 follows a similar pattern. These examples highlight how LLM annotations compensate for the lack of human annotations, effectively training the model and thereby enhancing its performance.

In the Case2 and Case3, as shown in Fig. 4, we observed that without filtering templates, F1 scores dropped by 19.8% and 9.3%, underscoring the effectiveness of the filtering mechanism in both scenarios.

Figs. 5 and 6 illustrate that for pairs of triples corresponding to the same trigger or the same entity, the red dots represent relation representation for one triple, while the blue dots represent those for the other, easily confusable triple. The filtering mechanism successfully separated previously overlapping relation representations, resulting in clearer boundaries and enabling effective classification of triples with different relations under similar conditions.

The results indicate that while MF2F performs slightly worse than GRASP by 0.9% on data with a small number of arguments, it significantly outperforms GRASP on data with a larger number of arguments, as shown in Fig. 7. For data with a small number of arguments, MF2F still has room for improvement, as the limited number of conflicts restricts the model’s ability to fully demonstrate its advantages. In contrast, in dialogues with a larger number of arguments, where conflicts occur more frequently, the model effectively handles these challenges. This demonstrates the efficiency of our feature filtering method in mitigating feature conflict issues.