The research on abstractive summarization mainly depends on the seq2seq model proposed by Cho et al. [1], which solves the length inequality of input and output in generative tasks. The seq2seq model contains two parts: encoder and decoder. The encoder encodes the input into a context vector C, and the decoder decodes the output by C. The Seq2seq model was originally used for Neural Machine Translation (NMT), and firstly proposed by Rush et al. [9] based on attention mechanism [10] for abstractive summarization, and it proved to have good performance.

Pre-trained language model has become an important technology in NLP field in recent years. The main idea is that the model’s parameters are no longer randomly initialized, but trained in advance by some tasks (such as Language Model) and large-scale text corpus. Then they are fine-tuned on the small dataset of specific tasks, and it makes it easy to tarin a model. The early pre-trained Language Model is Embeddings from Language Model (ELMo) [11], which can complete the feature extraction by bidirectional LSTM and fine-tune the downstream tasks. A Generative Pre-Training Language Model (GPT) can achieve very good performance by replacing LSTM with Transformer [12] in the text generation task. Based on GPT, Devlin et al. [3] considered using bidirectional Transformer and higher quality large-scale dataset for pre-training and obtained a better pre-trained language model BERT.

Liu et al. [13] proposed BERTSum for extractive summarization, a simple variant of BERT, and the model outperformed baseline on the CNN/DailyMail dataset. Later, Liu et al. [13] joined the decoder structure based on BERTSum to complete abstractive summarization and conducted experiments on the previous dataset. The experimental results showed that their model was superior to the previous model in both extraction summarization and abstractive summarization. The goal of UNILM proposed by Li et al. [4] is to adapt BERT to generative tasks, which is the same as that of Masked Sequence to Sequence Pre-training Model (MASS) proposed by Song et al. [14]. But UNILM is more succinct, sticking to BERT’s idea and only using encoders to complete various NLP tasks. The UNILM is trained based on three objectives: Unidirectional LM (left-to-right and right-to-left), Bidirectional LM, and seq2seqLM. Seq2seqLM can complete abstractive summarization. It defines the source text as the first sentence and the corresponding summary as the second sentence. The first sentence is encoded by Bidirectional LM, and the second sentence is encoded by Unidirectional LM (left-to-right).

Ma et al. [15] proposed a method to improve semantic relevance in seq2seq model. By calculating the cosine similarity between the semantic vector of the source text and the summary, we can get the measure of semantic relevance between them. The larger the cosine value is, the more relevant they are, and the negative value of the cosine similarity is added to the loss function to maximize the semantic relevance between them. At the same time, Ma et al. [16] also proposed an autoencoder as an assistant supervisor method to improve the text representation. By minimizing the L2 distance between the summary encoder vector and the source text encoder vector, we can supervise the semantic representation of the source text and improve the semantic representation of the source text.

In 2017, Sabour et al. [7] proposed a new neural network structure called Capsule Network. The input and output of Capsule Network are all in the form of vectors, and the results of image classification experiments showed that Capsule Network has a strong ability of feature aggregation. Zhao et al. [17] proposed a model based on Capsule Network to do text classification. As a result, the model performed better than the baseline system in the experiment.

Based on the methods mentioned above, we complete abstractive summarization by adopting the idea of seq2seqLM, and added the semantic supervision method into the model. We conducted relevant experiments on the Chinese dataset LCSTS [8], and analyzed the experimental results.

Our model structure is shown in Fig. 1, and it is composed of four parts. Embedding Layer is responsible for transforming the input token into a vector representation. Transformer Layer is responsible for encoding the token vector representation according to the context information. Output Layer is used to parse the encoded result of Transformer Layer. And the last part is the Semantic Supervision module proposed by us, which is responsible for supervising the semantic encoding of Transformer Layer.

Embedding Layer

BERT’s embedding layer contains Token Embedding, Segment Embedding and Position Embedding. Token Embedding is the vector representation of tokens, which is obtained by looking up the embedding matrix with token Id. Segment Embedding is used to express whether the current token comes from the first segment or the second segment. Position Embedding is the position vector of the current token. Fig. 1 shows Embedding Layer of BERT. The input representation follows that of BERT. We added a special token ([CLS]) at the beginning of input, and added a special token ([SEP]) at the end of every segment. T={T1,T2,…,Tn} represents the token sequence of the source text, and S={S1,S2,…,Sn} represents the token sequence of the summary. We got the input X={[CLS],T1,T2,…,Tn,[SEP],S1,…,Sm,[SEP]} of the model by splicing T, S and special token. By summing corresponding Token Embedding, Position Embedding and Segment Embedding, we can get a vector representation of each input token.

Transformer Layer

Transformer Layer consists of N Transformer Blocks which share the same structure but have different parameters to be trained. Transformer was originally proposed by Vaswani et al. [12], but only the Encoder part of Transformer is used in BERT. The reason why BERT can perform well in many NLP tasks is that it depends on a large amount of unsupervised data and the excellent semantic encoding capability of Transformer.

The input of seq2seqLM is the same as that of BERT, but the main difference is that seq2seqLM changes the mask matrix of multi-head attention in Transformer. As shown on the left of Fig. 2, the source text’s tokens can attend to each other from both directions (left-to-right and right-to-left), while every token of the summary can only attend to its left context (including itself) and all tokens in the source text. The mask matrix is designed as follows [4]:

(1) Mij={0,allow to attend-∞,prevent from attending

The element of the mask matrix is 0, which means the ith token can attend to the jth token. In contrast, the element is -∞, which means the ith token can’t attend to the jth token. On the right of Fig. 2, we showed the self-attention mask matrix M in Eq. (1), which is designed for the text summarization. The left part of M is set 0 so that all tokens can attend to the source text token. Our goal is to predict the summary, and attention from the source text to the summary is unnecessary, we set the upper right elements -∞. On the bottom right side, we set its lower triangular matrix elements 0, and other elements -∞, which prevents the current tokens of the summary from paying attention to the tokens after it.

The output of Embedding Layer is defined as T0={X1,X2,…,Xn}, where X_i represents the vector representation of the ith token and n represents the length of the input sequence. We abbreviated the output of the lth Transformer block as: Tl=Transformerl(Tl-1). In each Transformer Block, by aggregating multiple self-attention heads, we can get the output of the current multi-head attention. For the lth Transformer block, the output A_l of the multi-head attention is computed as follows:

(2) Al=Concat(head1,head2,…,headh)Wo

where headi=softmax(QKTdk+M)V

and Q=Tl-1WiQ,K=Tl-1WiK,V=Tl-1WiV

Tl-1∈Rn×dh is the output of the (l −1)th Transformer Block, where n is the length of the input sequence and d_h is the embedding size. WiQ,WiK,WiV ∈Rdh×dk and Wo∈Rdh×dh are the linearly projected matrices where d_k = d_h/h and d_h is the number of parallel attention heads. M∈Rn×n is the mask matrix in Eq. (1).

Output Layer

We took the output of the last Transformer Block as the input of Output Layer. Output Layer consists of three parts: two full connection layers and one Layer Normalization.

The first full connection layer is used to add nonlinear operations to BERT’s output, and we use GELU as the activation function, which is widely used in BERT. In Eq. (3), T^N is the output of the last Transformer Block, W₁ is the matrix to be trained, b₁ is the value of bias, and O₁ is the output of the first full connection layer.

(3) O1=GELU(W1TN+b1)

Different from Batch Normalization [18], Layer Normalization [19] does not depend on batch size and the length of the input sequence. Adding Layer Normalization can avoid gradient disappearance. In Eq. (4), LN(*) is Layer Normalization and O₂ is the output of LN(*).

(4) O2=LN(O1)

The second full connection layer is used to parse the output, which contains n×I (n is the length of output and I is the size of vocabulary) units, and we use softmax as the activation function. The softmax function is commonly used in multi-classification, and it map the output of multiple neurons to the interval (0, 1). Predicting a word is equivalent to a multi-classification task. In Eq. (5), W₃ is the matrix to be trained, b₃ is the value of bias, and O₃ is the final output of our model.

(5) O3=softmax(W3O2+b3)

For lack of fine-grained semantic representation in BERT, it can’t produce high-quality summaries when it was applied to text summarization. And there are semantic deviations between the original input and the encoded result passed multi-layer encoder. We hope to improve these problems by adding semantic supervision based on Capsule Network. The implementation of semantic supervision is shown on the right side of Fig. 1. At the training stage, we took the result of Token Embedding as the input of Capsule Network and got the semantic representation V_i of the input. At the same time, we did the same operation for the output of the last Transformer Block to get the semantic representation V_o of the output. We implemented the semantic supervision by minimizing the distance d(Vi,Vo) between the semantic representation V_i and V_o. d(Vi,Vo) is calculated as Eq. (6).

(6) d(Vi,Vo)=∥Vi-Vo∥2

Ma et al. [15] directly took the input and output results of the model as semantic representations, which had low generalization capability. So we added a Capsule Network [7] which is capable of high-level feature clustering so as to extract semantic features. The Capsule Network uses vectors as input and output, and vector has a good representational capability, such as using vectors to represent words in word2vec. Of course, our experiment also showed that Capsule Network performed better than LSTM [2] and GRU [20]. We define a set of input vectors u={u1,u2,…,un}, and the output of Capsule Network is v={v1,v2,…,vn}. The output of Capsule Network is calculated as follows:

(7) uj|i=Wijui

(8) bij=uj|ivj

It can be seen from Eq. (8) that the calculation of b_ij requires v_j, but v_j is the final output, so it is impossible to calculate b_ij directly. b_ij is usually given an initial value and computed iteratively. Based on this idea, Sabour et al. [7] proposed a Dynamic Routing algorithm in their paper.

We took the output of Embedding layer X={X1,X2,…,Xn} as the input u={u1,u2,…,un} of Capsule Network and got the output v={v1,v2,…,vn} where X∈Rn×dh (n is the length of the input sequence and d_h is the embedding size). Each vector v_i in v represents a property, and the length of the vector represents the probability that the property exists. We calculated the norm of each vector in v to form a new vector as shown in Eq. (12), and V_i is the fine-grained semantic representation of the input X. Similarly, we regarded the output TN∈Rn×dh of BERT as the input u′={u1′,u2′,…,un′}, and got the output v′={v1′,v2′,…,vn′} by Capsule Network. By calculating the norm of each vector in v′, we got a new vector as shown in Eq. (13), and V_o is the fine-grained semantic representation of the BERT’s output.

(12) Vi={|v1|,|v2|,…,|vn|}

(13) Vo={|v1′|,|v2′|,…,|vn′|}

We found that the longer the input sequence is, the larger the semantic deviations are. So we use different intensity semantic supervision for different lengths of the input. We controlled the intensity of supervision by the parameter λ in Eq. (14) where l_s is the length of the input sequence. The longer the input sequence is, the larger the supervision intensity is, and the shorter the input sequence is, the lower the supervision intensity is.

(14) λ=ls1+ls

The loss function of Semantic Supervision can be written as follows:

(15) Ls=λd(Vi,Vo)

There are two loss functions in our model that need to be optimized. The first one is the categorical cross-entropy loss in Eq. (16), where N is the number of all samples, y∈Rn is the true label of the input sample, ŷ∈Rn is the corresponding prediction label, D is the sample set, n is the length of summary and m is the vocabulary size. The other one is the semantic supervision loss defined in Eq. (15). Our objective is to minimize the loss function in Eq. (17).

(16) Lc=-1N∑ y∈D∑ i=1n∑ j=1myijlogŷij

(17) L=Lc+Ls

During training, we used Adam optimizer [21] with the setting: learning rate α=1×10-5, two momentum parameters β1=0.9, β2=0.999 and ε=1×10-8.

We conducted experiments on LCSTS dataset [8] to evaluate the proposed method. LCSTS is a large-scale Chinese short text summarization dataset collected from Sina Weibo, which is a famous social media website in China. As shown in Tab. 1, it consists of more than 2.4 million pairs (source text and summary) and is split into three parts. PART I includes 2,400,591 pairs, PART II includes 10,666 pairs, and PART III includes 1,106 pairs. Besides, the pairs of PART II and PART III also have manual scores (according to the relevance between the source text and summary) ranging from 1 to 5. Following the previous work [8], we only chose pairs with scores no less than 3 and used PART I as the training set, PART II as the validation set, and PART III as the test set.

We used the ROUGE scores [22] to evaluate our summarization model which has been widely used for text summarization. They can measure the quality of the summary by computing the overlap between the generated summary and the reference summary. Following the previous work [8], we used ROUGE-1 (1-gram), ROUGE-2 (bigrams), and ROUGE-L (longest common subsequence) scores as the evaluation metric of the experimental results.

We used the Chinese glossary of BERT-base, which contains 21,128 characters, but the number we counted all the characters in PART I of LCSTS is 10,728. To reduce the computation, we only used the characters of the intersection between them, including 7,655 characters. In our model, we used the default embedding size 768 of BERT-base, the number of heads h = 12, and the number of Transformer blocks N = 12. For Capsule network, we set the number of output capsules to 50 and the output dimension to 16, and the number of routes to 3. We set the batch size to 16, and we used Dropout [23] in our model. Our model was trained on a single NVIDIA 2080Ti GPU. Following the previous work [24], we implemented the Beam Search and set the beam size to 3.

We have compared the proposed model with the following model’s ROUGE score, and we would briefly introduce them next.

RNN and RNN-context [8] are two seq2seq baseline models. The former uses GRU as encoder and decoder. Based on that, the latter adds attention mechanism.

CopyNet [25] is the attention-based seq2seq model with the copy mechanism. The copy mechanism allows some tokens of the generated summary to be copied from the source content and it can effectively improve the problem of abstractive summarization with repeated words.

DRGD [26] is a seq2seq-based model with a deep recurrent generative decoder. The model combines the decoder with a variational autoencoder and uses a recurrent latent random model to learn latent structure information implied in the target summaries.

WEAN [27] is a novel model based on the encoder-decoder framework and its full name is Word Embedding Attention Network. The model generates the words by querying distributed word representations, hoping to capture the meaning of the corresponding words.

Seq2Seq+superAE [16] is a seq2seq-based model with an assistant supervisor. The assistant supervisor uses the representation of the summary to supervise that of the source content. And the model uses the autoencoder as an assistant supervisor. Besides, to determine the strength of supervision more dynamically, Adversarial Learning is introduced in the model.