Entity extraction stands as the initial and pivotal step in the knowledge extraction, involving the automatic identification of named entities from the original corpus. This foundational process relies on the automatic detection and categorization of named entities within a given corpus. Fig. 9 illustrates various categories to which these named entities can be ascribed, such as Person, Country, City, and more. This process enables the identification of important entities, laying the foundation for subsequent knowledge extraction and analysis tasks.

Generally, there are three typical methods in this field, and they are the rule-based, the traditional machine learning-based, and deep learning-based extraction methods. Fig. 10 displays the classification results obtained from these methods, showcasing the effectiveness and performance of each strategy in identifying and categorizing named entities. Here, rule extraction is an early pattern implemented by manually designing rules primarily toward proper nouns within a specific domain’s text. It is based on painstakingly handcrafted patterns, necessitating a lot of human efforts, culminating in limited extraction capacity and constrained scalability. Meanwhile, the related extraction methods based on machine learning models have witnessed remarkable progress. These methods integrate machine learning algorithms into entity extraction to achieve automatic or semi-automatic entity identification. Finally, with the development of artificial neural networks, some deep learning-based methods have been proposed to attain heightened proficiency in entity extraction tasks with reduced human intervention. This progression signifies a remarkable leap forward in the field.

For the traditional machine learning-based entity extraction, the inception involved the utilization of supervised machine learning algorithms. For example, the decision trees and the conditional random field (CRF) model [14] were employed to realize entity recognition for Telugu-English code-mixed social media data. In addition, Sykes et al. [15] conducted a comprehensive comparison between rule-based and machine learning-based extraction methods, effectively emphasizing the latter’s superior adaptability and versatility.

Furthermore, with the progressive evolution of web technologies, the combination of open-linked data with machine learning algorithms frequently yields more compelling results. Within this framework, the basic idea revolves around employing machine learning to extract entities with similar contextual features from the web page, subsequently achieving entity classification or clustering. For example, Whitelaw et al. [16] proposed an iterative approach for expanding the entity corpus in a network environment. This approach depended on the construction of feature models grounded in known entities, enabling the processing of massive datasets. With this mechanism, they effectively modeled new entities to achieve continuous iterative expansion of the entity. Furthermore, employing unsupervised learning algorithms, Jain and Pennacchiotti [17] successfully extracted newly emerging named entities from the server logs of search engines. Then, this method found practical application in search engine technology, allowing for automatic information completion based on user-input keywords. Additionally, while constructing KG, an application of intelligent corpus annotation for entity extraction was presented [18].

Deep learning-based methods offer the advantage of automated text feature selection for entity extraction, which reduces incompleteness and manual work. They have shown promising results in this area. One such method employs convolutional neural networks (CNN) to automatically learn features from the input text data. For example, a deep learning-based method [19–21] was proposed for entity extraction that utilized a CNN to learn contextual features from the input text data. Those models were trained on a large dataset of annotated text datasets and exhibited satisfactory performance on several benchmark datasets. Similarly, Cho et al. [22] presented a deep learning-based strategy for entity recognition in biomedical texts, combining a CNN with a long short-term memory (LSTM) network. Their model acquired complex features from the input text data, attaining remarkable accuracy on a challenging biomedical entity recognition task. Overall, deep learning-based methods have the potential to advance entity extraction, thereby constructing more precise and comprehensive KG. Hence, this enhancement leads to more powerful applications in many fields, such as natural language processing (NLP) and information retrieval. Notably, Table 3 showcases the top 10 entity extraction models from 2019 to 2022, ranked by F1-score on various open-source datasets (CoNLL 2003 [23], ACE 2005 [24], and Ontonotes 2005 [25]) This information is sourced from the Papers with Code website (https://paperswithcode.com).

For the CoNLL 2003 dataset, several remarkable models have garnered high F1-score, highlighting their prowess in entity extraction. The Automated Concatenation of Embeddings (ACE) combined with document-context [26] stands out at an impressive 94.60%. Close on its heels, the Co-regularized Language Understanding with Knowledge-based Embeddings (LUKE) [27] achieves a commendable 94.22%, while the Autoregressive Structured Prediction (ASP) fused with Text-to-Text Transfer Transformer (T5)-3B [28] achieves a noteworthy 94.10% (the “3B” represents 3 billion parameters). These models have demonstrated outstanding performance in extracting entities from the CoNLL 2003 dataset.

In the ACE 2005 dataset, the Princeton University Relation Extraction system (PURE) [35] emerges triumphant with an F1-score of 90.90%. Not far behind, PromptNER [36] has a second position with 88.26%, followed by Parallel Instance Query Network (PIQN) [37] with 87.42%. These models have shown strong performance in entity extraction from the ACE 2005 dataset.

For the Ontonotes 2005 dataset, the Bidirectional Encoder Representations from Transformers (BERT)-Machine Reading Comprehension (MRC)+dice coefficient (DSC) [44] attains the pinnacle with the highest F1-score of 92.07%. Packed Levitated (PL)-Marker [30] follows closely behind with 91.90%, closely trailed by Baseline+Boundary Smoothing (BS) [45] achieving 91.74%. These models have demonstrated their efficacy in entity extraction from the Ontonotes 2005 dataset.

These state-of-the-art models serve as prime examples of the evolutions made in entity extraction techniques, leveraging various approaches including deep learning, Prompt, and co-regularization methods. With remarkable F1-score on their respective datasets, these models indicate their effectiveness in extracting entities from text.

Relation extraction is an important subtask of knowledge extraction. In the past, this task entailed manual rule construction, followed by pattern-matching techniques to extract corresponding relation instances from text. However, the advent of machine learning has revolutionized relation extraction methods. It leverages lexical and syntactic attributes for model training, effectively transmuting relation extraction challenges into classification or clustering problems. According to the extent of human involvement and dependence on labeled corpus, machine learning-based relation extraction approaches can be divided into the supervised learning-based relation extraction, the semi-supervised learning-based relation extraction, and the unsupervised learning-based relation extraction.

(1) Relation Extraction with Supervised Learning Method

Supervised learning-based relation extraction is an automatic mode on the basis of meticulously labeled training data. Through ongoing learning from these training samples, the classification and predictions are performed on datasets. Within this paradigm, binary relation extraction is treated as a classification problem. As shown in the following definition, the triple (ei,rk,ej) indicates that there is a semantic relationship rk between the head entity ei and the tail entity ej, and function f(⋅) represents the relationship classifier employed in the context:

f(ei,rk,ej)={1if the triple(ei,rk,ej)exists,0otherwise.

Supervised learning employs two primary categories of relationship classification methods: feature vector-based and kernel function-based. In the first category, features are extracted from the training samples and represented as sequence feature vectors, using the results of part-of-speech tagging and syntax parsing. Here, the prominent methods are support vector machine (SVM) [49] and maximum entropy (ME) [50]. For example, a classifier system employing SVM integrated lexical features from polarity lexicons and lists of offensive/profane words to identify and classify offensive language in social media [49]. The second category of methods effectively avoids the challenges of dimensionality caused by nonlinear transformations. Recent years have witnessed widespread utilization of kernel function-based approaches in many fields [51]. In relation extraction using kernel functions, such as convolution kernel [52], tree kernel [53], subsequence kernel [54] and some improved kernel methods [55] play a pivotal role. The key of these methods involves projecting the implicit feature vector of a sentence into the feature space using the kernel function and calculating the inner product between these projections, so as to assess the similarity of the relationship between entities. For example, an innovative tree kernel, termed feature-enriched tree kernel (FTK) was proposed [53], while achieving a 5.4% enhancement in F-measure over the traditional convolution tree kernel.

Supervised learning-based relation extraction methods yield excellent experimental results, but their effectiveness heavily depends on the classification features provided by part-of-speech tagging and syntactic parsing. To address this problem, the use of supervised relation extraction has witnessed a surge of interest driven by the deep learning model [56]. A recurrent neural network (RNN)-based relation extraction model was proposed by Socher et al. [57]. This approach involved vectorizing each node of the syntactic tree through syntactic analysis. Guided by the syntactic structure, it iterated continuously from the lowest word vector of the tree. Finally, the vector representation of the sentence was attained and employed as the foundation for relationship classification. This method effectively utilized syntactic structure information, but overlooked the position information of words. Then, convolutional neural network (CNN) took center stage. Here, the word vector was treated as an initialization parameter, engaging in convolution training with dynamic optimization during the learning process, culminating in classification [58]. To complement the local dependencies captured by piecewise CNNs, a self-attention mechanism was proposed to capture rich contextual dependencies [59]. The experiments were performed on the NYT dataset and the experimental results demonstrated that the model provided a new benchmark in the area under curve (AUC) metric. Expanding on the CNN-based methodology presented [57], a refined iteration emerged [60]. This advancement involved inputting both word vectors and word positional vectors, with the sentence representation being obtained through the learning of convolutional, pooling, and nonlinear layers. This method fully considered the entity location information and other related lexical features, which achieved good relation extraction results. On the standard SemEval-2018 Task 7 dataset, the CNN method achieved superior performance when compared to alternative relation extraction methods [61]. Furthermore, except for deep learning models, deep reinforcement learning has also been used in the relation extraction [62]. This approach casts the relation extraction as a two-step decision-making game, employing the Q-Learning algorithm with value function approximation to learn control policy. The experiments were conducted on the ACE 2005 corpus, and they showed that the deep reinforcement learning model achieved a state-of-the-art performance in relation extraction tasks.

(2) Relation Extraction with Semi-supervised Learning Method

Semi-supervised learning-based relation extraction aims to realize the binary relation classification with limited training samples, thus circumventing the constraints imposed by manual annotation of extensive training data. It mainly adopts the bootstrapping method and some other methods.

The idea of bootstrapping is to artificially construct a small set of initial relation instances as a seed. This seed set serves as the foundation for model training, and through iterative expansion, it gradually augments to encompass a more extensive collection of relation instances, and finally completes the relation extraction task. Here, the entity alignment technique was improved to reduce the data noise [63]. However, this method operates under the assumption that a single entity pair corresponds to just one relationship. To address this limitation, a multi-instance multi-label (MIML) method was proposed to model the relationship extraction. This methodology describes the situation where an entity pair may have multiple relationships [64]. Moreover, the integration of a Bayesian network with MIML was explored for relation extraction [65], further expanding and capabilities of the approach.

Although the bootstrapping method is intuitive and effective, it may introduce a large number of noisy instances during seed expansion, leading to semantic drift. To address it, a deep co-learning was proposed [66], and it was a semi-supervised end-to-end deep learning method for evaluating the credibility of Arabic Blogs. A coupled semi-supervised learning method was used to establish constraints between different categories of extraction templates [67]. This strategic implementation effectively curbed the generation of false templates, thereby bolstering the precision of relation extraction. Meanwhile, a method was proposed by combining matrix-vector recursive neural network (MV-RNN) with bootstrapping [68]. Here, through the tree structure in MV-RNN, the semantic information of the entire sentence could be extracted as relation classifier features, and it greatly improved the accuracy of the results avoiding the problem of requiring a large amount of corpus in MV-RNN. More recently, the integration of transfer learning, a popular machine learning strategy, into the semi-supervised learning, yielded a novel framework [69]. Applied within the context of low-resource entity and relation extraction in the scientific domain, this framework demonstrated satisfactory performance, underscoring its potential and versatility.

(3) Relation Extraction with Unsupervised Learning Method

Unsupervised learning-based relation extraction assumes that pairs of entities with the same semantic relation have similar contexts, and it transforms the relation extraction task into a clustering problem. Hence, it does not require manual corpus annotation, but the accuracy rate is relatively low.

Generally, it is implemented using various clustering algorithms. For example, the large pre-trained language model was used for adaptive clustering on contextualized relational features to improve computational performance in relation classification [70]. After taking the entity set in the Wikipedia entry as the object, and using the dependency features and shallow grammar templates, all semantic relationship instances corresponding to entities were extracted in a large-scale corpus by pattern clustering [71]. Meanwhile, the templates were extracted and aggregated from search engine summaries, and they were clustered to discover implicit semantic relationships represented by entity pairs [72]. To further elevate the efficacy of relational templates’ clustering, a co-clustering algorithm was used, leveraging the dual nature of the dual of relational instances and relational templates. Moreover, the integration of a logistic regression model played a pivotal role in filtering representative extraction templates from the clustering results of relational templates [73].

For the above three basic types of machine learning, Table 4 offers a comprehensive comparison and analysis of the algorithms used in relation extraction. The table highlights the distinctions between classical algorithms, extraction ideas, human intervention levels, and extraction performance across different methods. Specifically, supervised and semi-supervised methods extract relationships by classifying learning sample data labels, while unsupervised-based methods cluster data and group related entities together to achieve relationship extraction. A noteworthy observation is that the performance of the extraction model improves with an increased corpus size and manual intervention. Taking into account the trade-off between performance and computational efforts, the semi-supervised learning methods emerge as favorable choices for relation extraction. It not only ensures better extraction performance but also avoids the limitation of requiring an extensive amount of manually annotated corpus. Moreover, Table 5 provides a comprehensive overview of the performance exhibited by various excellent relation extraction models on two famous datasets (NYT [74] and WebNLG [75]) in recent years. The F1-score achieved by these models indicate their effectiveness in relation extraction.

Focusing on the NYT dataset, UniRel [76] achieved the highest F1-score of 93.7%, followed by Relation Extraction By End-to-end Language generation (REBEL) [77] with 93.4%, and Djacency lIst oRiented rElational faCT (DIRECT) [78] at 92.5%. These models have exhibited robust competence in extracting relations from the NYT dataset.

For the WebNLG dataset, UniRel [78] maintains its lead, achieving the highest F1-score of 94.7%, followed by Partition Filter Network (PFN) [79] with 93.6%, and Set Prediction Networks (SPN) [80] with 93.4%. These models have exhibited remarkable performance in relation extraction from the WebNLG dataset. Furthermore, other models such as Translating Decoding Schema for Joint Extraction of Entities and Relations (TDEER) [81] and Representation Iterative Fusion based on Heterogeneous Graph Neural Network for Joint Entity and Relation Extraction (RIFRE) [82] also achieved comparable performance on both datasets, further highlighting the consistent excellence of these methodologies.

These excellent models utilize various techniques, including RNNs, partition-based methods, and joint learning frameworks. The impressive F1-score attained by these models underscores their ability to effectively extract relations from text across different datasets. This analysis showcases the advancements in relation extraction models and their potential applications in numerous NLP tasks.

Due to the diversity of information expression, entity ambiguity remains a frequent and formidable hurdle in natural language understanding. For example, “apple”, can signify either a fruit or the renowned technology company. However, when contextual information such as “Steve Jobs” is provided, it becomes evident that in the given text, the entity “apple” refers to the company, as illustrated in Fig. 11. The presence of contextual cues helps disambiguate the intended meaning of entities, aiding in enhancing the precision and comprehension of text.

Entity linking is an effective disambiguation method. By linking entity mentions to the corresponding entities in the knowledge bases, this method conducts precise entity annotation within documents. This enables computers to attain a more profound grasp of the semantic information of the text, effectively addressing the problems of synonym and polysemy. Commonly, entity linking leverages well-established knowledge bases such as Wikipedia, DBPedia, and Freebase. Specifically, open knowledge bases like Baidu Encyclopedia and Interactive Encyclopedia also find prominent applications in Chinese entity linking. Generally speaking, entity linking includes two subtasks: entity recognition and entity disambiguation. By skillfully addressing both aspects, this technique contributes to a more comprehensive and nuanced understanding of textual content.

(1) Entity Recognition

Entity recognition aims to identify fragments of text, and it may link to specific entries in the knowledge bases, including a specific word or phrase. Typical entity types include place names, person names, institution names, times, dates, percentages, and amounts. With the development of Internet information, novel entity categories have surfaced in recent years, including movie titles and product names, reflecting the expanding landscape of entity recognition [89,90].

Currently, entity recognition technology is mainly based on statistical machine learning methods, treating the task as a sequence labeling problem. There are three main solutions: hidden Markov model (HMM), maximum entropy Markov model (MEMM), and CRF model. An overview of these models, their key evaluation criteria, and comparisons are delineated in Table 6. Over recent years, numerous studies have utilized these three models. For instance, a novel generative model was proposed [91], linking it to HMM while proving its generically identifiable nature without any observed training labels. However, HMM exhibits the tag bias problem by assuming that the current tag solely depends on the previous one. Additionally, its use of local normalization to compute the probability of observation series makes calculating the function complex, leading to reduced efficiency. MEMM addresses the tag bias problem but introduces the tag inconsistency issue. Furthermore, it often relies on manually designed features, which are pivotal for the model’s performance and generalization ability, making feature selection a challenging task. CRF model is different from HMM and MEMM. By employing global modeling, CRF simultaneously considers the observation and labeling of the entire sequence when calculating the probability. This enables the avoidance of label bias and local normalization problems found in HMM and MEMM. Moreover, the CRF model resolves the label inconsistency problem present in MEMM by modeling dependencies between observations and label sequences concurrently. Additionally, CRF utilizes complex and rich feature representations that capture dependencies between observed and labeled sequences, thereby enhancing model performance and generalization. Unlike MEMM, CRF model is less dependent on manually selected features and can learn feature weights that suit the task, thereby streamlining feature engineering. By overcoming some of the main shortcomings of HMM and MEMM models, CRF achieves superior performance in sequence labeling and has been widely used in the entity recognition field [92].

(2) Entity Disambiguation

In the context of a specific entity mentioned in the text, entity disambiguation is mainly used to analyze the semantic information and select the corresponding entity from the candidate entities. This process depends on the candidate entity and the contextual information. Usually, given the ambiguity in natural language, there are numerous candidate entities vying for consideration. Methods of entity disambiguation are mainly based on supervised learning and unsupervised learning.

Most works adopt supervised learning methods for disambiguation and use training data to automatically design ranking models. The requisite training data for entity linking comprises an ordered list of all candidate entities associated with the target mentioned in a given context. In this list, the first entity is usually the one that the mention refers to in this context. By linearly combining features like entity popularity, semantic similarity, and connection between entities, a maximum margin-based data were employed to train feature weights, while achieving entity disambiguation through a ranking model [93]. Moreover, two machine learning sorting-based methods using listwise and pairwise were presented to implement entity disambiguation, outperforming traditional disambiguation methods [94]. The listwise method is from the LiNet algorithm, which uses the ordered list as a training instance to obtain a sorting model. The essence of the pairwise method is to transform the sorting problem into a classification problem. It combines the items in the ordered list into pairs and constructs training instances according to the relative positional relationship between the items to develop a sorting perception. Furthermore, recent developments have seen the combination of supervised learning with graph theory to address the entity disambiguation task [95], showcasing the innovative fusion of established techniques to enhance disambiguation accuracy.

On the other side, unsupervised learning algorithms have been used in entity disambiguation. For example, a clustering-based personal name disambiguation system was proposed to extract personal attributes and social relations between entities from text, subsequently mapping them onto an undirected weighted graph [96]. Clustering algorithms were then used to cluster these graphs, each cluster contained all web pages that directed a person. Significant models have been shown in Table 7 in the field of entity disambiguation, where datasets are ACE2004 [97] and AIDA-CoNLL [98]. These models focus on disambiguating entities, which is crucial for accurately identifying the intended meaning of ambiguous terms.

For the ACE 2004 dataset, Entity Disambiguation by Reasoning over a Knowledge Base (KBED) [99] stands out with the highest F1-score of 93.4%, followed by LUKE[confidence-order] [100] with 91.9%, and Representation and Fine-grained typing for Entity Disambiguation (ReFinED) [101] at 91.6%. These models have shown strong performance in entity disambiguation on the ACE 2004 dataset.

Regarding the AIDA-CoNLL dataset, LUKE[confidence-order] [100] maintains its prominence with the highest F1-score of 95.0%, closely followed by Dynamic Context Augmentation (DCA)-Supervised Learning (SL)+Triples [104] and DeepType [105] with F1-score of 94.9%. These models have demonstrated excellent performance in disambiguating entities in the AIDA-CoNLL dataset. It is worth mentioning that LUKE[confidence-order] [100] is the only model present in both datasets, indicating its robustness and effectiveness across different evaluation scenarios.

The top-performing models employ diverse techniques such as knowledge-based methods, confidence ordering, and deep learning approaches. Their high F1-score emphasize their proficiency in accurately disambiguating entities and determining their intended meanings in different contexts.