In this paper, we choose controlled long text generation based on the prompt method, i.e., generating open-domain long text based on the input prompt text, and the hidden space of the latent variable model C-VAE is utilized to obtain latent representations from textual relations. Fig. 1 illustrates the model diagram of the variational autoencoder VAE, whose architecture consists of an encoder (inference network) and a decoder (generation network), where the encoder maps the input data x to the probability distribution qφ(z|x,y) of the latent variable z, and then samples from this distribution are obtained, and the latent variable z is obtained through the computation of the mean, variance, and parameters. The decoder generates the reconstructed probability distribution pθ(x′∣y,z) of the data x′, using z as input. The traditional VAE decoder pθ(x|y,z) is usually in autoregressive form; in this paper, we use a pre-trained GPT-2-based model as the decoder.

The goal of C-VAE training is to maximize the data log-likelihood Ex,y∼D[log⁡pθ(x)]. However, since posterior inference is difficult to achieve, C-VAE approximates the posterior distribution through variational inference methods to maximize the lower bound on the marginal probability of the observed data (ELBO): (1)Ex,y∼𝒟logpθ(x|y)≥Ex,y∼𝒟[Ez∼qϕ(z|x,y)logpθ(x|y,z)]−Ex,y∼𝒟[KL(qϕ(z|x,y)∥p(z|y))]

Here, Ex,y∼𝒟[KL(qϕ(z|x,y)∥p(z|y))] is the KL dispersion between the latent space distribution qϕ(z|x) and the prior distribution p(z) learned by the encoder, which is used to ensure the continuity and compactness of the latent space. x is the input text, and y is the condition.

Graph Convolutional Encoder is a model for learning graph data representation by combining a node’s neighbor information and features to generate a low-dimensional representation of the node. The focus of this paper is on the global relations of the text. Fig. 2 illustrates a graph convolutional encoder with a two-layer convolutional network. Each word in the input text is treated as a node in the graph, the node features are vector representations of the input text, and the connecting lines between the nodes are the values of the elements in the resulting adjacency matrix A. After the text passes through the graph convolutional network, the text features are updated to obtain a new vector representation of the text.

The GCN formula is as follows: (2)Hl+1=σ(D~12A~D~12HlWl)

Here A~ is the new adjacency matrix obtained by adding the unit matrix to the adjacency matrix A, D is the degree matrix of the adjacency matrix A, W is the weight parameter matrix of the Lth layer, σ is a nonlinear activation function, e.g., ReLU(Rectified Linear Unit). For a two-layer convolutional network, the forward propagation formula is: (3)z=f(X,A)=softmax(A~ReLU(A~XW0)W1)

For the loss function, the traditional cross-entropy function is chosen.

The rest of the paper is organized as follows. Section 3 details the overall architectural design of the model proposed in this paper. Section 4 is the experimental part and comparative analysis of the experimental results. Section 5 summarizes the work of this paper and provides an outlook on the next steps.

After the input text data is encoded, the resulting text vector can accurately represent the nodes in the graph, i.e., the feature matrix X. Thus, only the adjacency matrix A of the text nodes needs to be constructed. Afterward, the feature matrix X and the adjacency matrix A are inputted into the GCN encoder with a two-layer convolutional network to obtain new node representations containing comprehensive relationships among the texts.

Commonly used methods for structuring text graphs include dependent syntactic analysis trees, sentence component analysis, AMR (Abstract Meaning Representation) graphs, information extraction graphs, knowledge graphs, etc. Still, all of these methods have some limitations. For example, sentence component analysis may have difficulty in describing sentence components in specific languages and text types [41]; the construction of AMR graphs relies on the subjective interpretation of manual annotators, which may lead to inconsistency and subjectivity problems [42]; information extraction graphs usually focus on the extraction of entities and relations, which may ignore important information in the context, etc. [43].

A filter exists in a convolutional neural network, which extracts information from an image through a constantly moving fixed-size filter window. Inspired by the filter window, this paper proposes to design a “sliding window” to traverse the input text. In this window, the adjacency matrix A is constructed by the features of the text, and the word embedding of the text is used as the feature matrix X of the text, thus completing the graph structuring of the text.

However, the sliding window approach also introduces a new problem: the relationships between texts are far or near. When using a sliding window to obtain relationships, if the window size is too small, the complete relationship information cannot be obtained; if the window size is too large, unnecessary information, i.e., noise, is introduced. This affects the construction of the final graph.

In response to the new problems triggered, a dynamic sliding window can be designed to solve the problem. The so-called dynamic sliding is based on the strength of the textual relationship, which adjusts the window size in real time to capture the textual relationship under different relationship strengths. The specific process is as follows: set the initial window size k (e.g., the initial size is set to 3), and then traverse the entire text sequence, the three words within the first window, the average cosine similarity value can be obtained after the cosine similarity is calculated using its vector representation. Set a threshold T when the average cosine similarity is greater than the threshold T, increase the window size, and vice versa, reduce the window (minimum of 2) until traversing the entire text sequence, and ultimately get the adjacency matrix A, in which each element of the element matrix in the adjacency matrix A represents the strength of the relationship between word i and word j, the larger the value indicates that the more similar the two words are to each other, and the smaller the value indicates that the more the similarity is lower. The specific process is shown in Fig. 4.

Traditional Transformer only relies on Positional Encoding and Self-Attention to model inter-word relationships, which makes it easy to ignore long-distance dependencies. In this paper, the method adopts GCN to model the text as a Graph Representation. It dynamically connects the related words through the sliding window to enhance the global dependency modeling. In the dynamic sliding window, the strengths and weaknesses of the relationships between the texts are further recorded, and the final graph structure representation is improved according to the strengths and weaknesses of the relationships so that the resulting textual relationships have better strengths and weaknesses compared to the textual relationship representations obtained from traditional coding. The following demonstrates how the dynamic sliding window captures strong and weak textual relations.

Now consider the set of word vectors for two windows W1 and W2, W2>W1 and containing W2 contains W1, and the average cosine similarity between them is: (4)AνgCosineSimilarity(W1,W2)=1min(|W1|,|W2|)∑a∈W1∑b∈W2Cosine Similarity(a,b)

Here, W1, W2 are the window sizes, respectively. Define a threshold T when AvgCosineSimilarity(W1,W2)>T, it represents a higher similarity between the two windows, which means that the textual contexts within the two windows are similar, i.e., better textual relations can be obtained when W1 is increased to the size of W2. Similarly, when AvgCosineSimilarity (W1,W2)<T, which represents a lower similarity between the two windows, suggests that there is a large gap between the textual contexts within the two windows, at which point the window can be narrowed down to focus on a tighter context. This proves that this paper’s dynamic sliding window approach can better capture the semantic relationships between texts.

When data is encoded, traditional Transformer models use only the last hidden state from the encoder to generate the latent space. After the final self-attention layer, a sequence of vectors is obtained with the number of vectors as the number of input tokens. However, the vector sequence obtained after such encoding may not be sufficient to summarize continuous data and retain long-term knowledge, mainly due to the limitations of the self-attention mechanism, positional encoding, etc. This paper proposes a vector sequence merging method to merge vector sequences into a single vector through attention scores for subsequent potential representation generation work. A single vector contains both global and local input text information, and the single vector can better process and retain feature information than vector sequences. Also, due to the use of Attention Score Weighted Merging, each feature is weighted and merged during vector sequence merging, thus further emphasizing important features and weakening minor features. The following is a validation of the methodology of this paper in terms of both weights and expectations.

First, from the point of view of weights, suppose there is an input sequence X={x1,x2,x3…xn}, where x denotes the xth element in the sequence. Using the self-attention mechanism, it is possible to compute a vector Ai of attention weights at each position and then apply these weights to the original vector sequence. This results in a weighted vector and Z, where the weighted sum for each position i is computed as follows: (5)Zi=∑j=1nAi,j⋅xj

Here, Ai,j is the attentional weight from position i to position j. To combine this sequence into a single vector, consider weighting and summing all the positions, i.e., final_output=∑i=1nzi rearrange this equation and introduce attentional weights: (6)final_output=∑i=1n(∑j=1nAi,j⋅xj)

Obtained by swapping the summation order: (7)final_output=∑j=1n(∑i=1nAi,j)⋅xj

The portion in parentheses is the sum of all the attentional weights for position j, which can be thought of as the composite weight for position j. The weight of position j is the sum of all the attentional weights for position j. Let Cj=∑i=1nAi,j The above equation can be simplified as: (8)final_output=∑j=1nCj⋅xj

This equation shows that the weighted composite Cj obtained through the self-attention mechanism and the weighted sum of the original vector sequence X form the final representation. These synthesized weights can better capture the relationships between different positions in the sequence, thus providing richer semantic information.

Moving on to the expectation perspective, suppose again that there is an input sequence X, denoted {x1,x2,…,xT}, of length T. Each xt is a vector. Now compute the attentional weight αt, where the attentional weight αt is denoted: (9)αt=exp⁡(et)∑k=1Texp⁡(ek)

Here, et is some rating or score associated with xt can be computed using any appropriate mechanism (e.g., dot product, scaled dot product, etc.). The attention-weighted average vector is then computed: (10)AttentionAvg=∑t=1Tαt∗xt

First, the attention weights are of the nature: ∑t=1Tαt=1, this is because the weights are standardized probability distributions. Next, the expected value of the weighted average vector of attention is computed: (11)E[AttentionAvg]=E[∑t=1Tαt⋅xt]=∑t=1TE[αt⋅xt]=∑t=1TE[αt]⋅E[xt](linear property)=∑t=1TE[αt]⋅xt(Definition of expectation)=∑t=1T(exp⁡(et)∑k=1Texp⁡(ek))⋅xt

By normalizing αt, the weighted average vector of expected attention will be more concentrated on the parts with higher scores et, i.e., the model will pay more attention to the tasks’ relevant parts.

After obtaining the potential representation Z of the potential space, Z is usually added to the subsequent decoding and generation process by either concatenating Z with the input vectors of the decoder or multiplication operations, etc. or by fusing Z with each layer of the decoder according to the attention weights. In this paper, we not only use the method of concatenating Z with the input vectors but also add Z to the K and V matrices of the attention computation to obtain new K' and V' matrices and compute the attention. In this way, Z is fully integrated into the decoder generation process for better control of text generation.

For the traditional embedding approach, the drawback is the introduction of noise in the self-attention computation. The proof is as follows: ei,ej is defined as the marker embedding of the two inputs, and αi,j is the attention weight of the ith and jth markers. According to the self-attention formula. (12)Attention(Q,K,V)=softmax(QTKd)V

The expression αi,j can be obtained as: (13)αi,j=(Wqei)T(Wkej)=eiT(Wq)TWkej

Wq and Wk are the parameter matrices used to map the input marker embedding into the query (Q) and key (K) spaces. Denoting the right-hand side of this equation as (ei,ej), Z is embedded directly into the tags according to the embedding, i.e.: (14)αi,j=[Wq(ei+z)]T[Wk(ej+z)]=(ei,ej)+(ei,z)+(z,ej)+(z,z)

It can be noted that the resulting equation introduces a redundant term (z,z) introduces additional noise to the attention mechanism.

To solve the problems caused by the embedding method, this paper introduces potential attention computation while using the embedding method; specifically, the potential representations obtained previously are divided into L vectors [Z1,Z2…ZL], and the potential representations are merged into the original self-attention computation by adding the potential representations to the matrices K and V in each layer of the attention computation.

Integrate Z into K and V to get a new K^′ and V^′ (15)K′=(ZKK)∈R(1+l)×dV′=(ZVV)∈R(1+l)×d

The attention score α can be denoted as α=(Q∗K′), this is the first time Z is fused with the feature vector, after which the attention score is multiplied by the new value matrix in the subsequent self-attention computation. Attention=(α∗V′), this is the second time Z is fused with the feature vector, and in this computation, the value matrix V' already contains information about the latent variable Z. When the embedding method is used again, the effect of noise is resolved. (16)[(ei,ej)+(ei,z)+(z,ej)+(z,z)]∗(V′)=[(ei,ej)+(ei,z)+(z,ej)+(z,z)]∗(ZVV)=(ei,ej)∗(ZVV)+(ei,z)∗(ZVV)+(z,ej)∗(ZVV)+(z,z)∗(ZVV)

It can be seen that in the results of the above equation, the noise (z,z) present in the original embedding method is finally multiplied by a matrix that contains information about the incorporated latent variable Z. Therefore, this item is equivalent to the weighting of the latent variable Z and is no longer noisy.

The datasets used in this paper are Arxiv [44], Yelp [45], WritingPrompts [46], and WikiPlots [47]. In this paper, the Arxiv dataset is used to generate controlled text, and the Arxiv dataset is divided into a training dataset, testing dataset, and validation dataset in the ratio of 80:10:10. Arxiv is an online dataset that extracts abstracts from the ‘Arxiv’ articles. Arxiv is an online dataset for extracting abstracts from ‘arxiv’ articles. Arxiv mainly searches for topic queries in arxiv and then collects article content-matching Abstracts. This paper selects article types with keywords ‘artificial intelligence,’ ‘computer vision,’ and ‘text generation.’ Meanwhile, this paper experiments with the GSPT-CVAE model on WritingPrompts and WikiPlots datasets to show the model’s performance under different data domains. On the other hand, the Yelp dataset is mainly used for pre-experimental analysis and textual relationship validation visualization experiments.

The comparison methods chosen for this paper are as follows:

FIST: This method is a technique for fine-tuning natural language processing models by introducing special tokens to guide the model to focus on specific tasks or samples during the fine-tuning process [48]. These unique tokens can mark important locations or sample attributes in the input sequence, thus making the model better adapted to the target task.

This paper evaluates the following automated metrics for the target text:

Perplexity (PPL): complexity PPL is used to evaluate language models and is often considered a proxy for the generation quality. All GPT -2-based models use a byte-level tokenization scheme (PPL-BPE), based on which the models’ word-level complexity (PPL-Word) is also computed in this paper to compare them with previous models.

This paper uses the smallest publicly available version of GPT-2 as the model’s decoder for computational efficiency. Specifically, it has L = 12 layers, H = 12 attention heads per layer, and a model dimensionality of d = 768 units, totaling 117 million parameters. In addition, To address the posterior collapse problem, we implement a cyclic annealing schedule by modifying the coefficient β before KL divergence in (2). Specifically, we have kept β close to zero in the first half of the cyclic schedule, linearly annealed β to 1 in the next one-fourth of the cyclic schedule and kept β=1 in the remaining one-fourth of the cyclic schedule. Finally, In evaluation, we generate stories using the top-k top-p random sampling scheme with k = 100 and p = 0.95. Temperature smoothing technique is also applied with T = 0.95.

When structuring a text graph, the text similarity within the window needs to be compared with a threshold to dynamically adjust the window size to better capture the relationships between the texts. Therefore, the selection of the size of the threshold is critical. By choosing different thresholds for several experiments, the results are shown in Fig. 5. As can be seen from the figure, when the threshold is too small, almost all pairs of nodes in the graph are connected, resulting in the graph becoming too dense with nodes connected by a large number of irrelevant edges. These edges represent texts unrelated to the context, which can interfere with subsequent feature propagation and learning. At the same time, when the threshold is too large, the connectivity of the adjacency matrix decreases, and too low a connectivity may lead to isolated nodes or small isolated subgraphs, which prevents the GCN from efficiently propagating the features and, thus, from capturing the relationships between texts.