Chinese Long-Text Classification Strategy Based on Fusion Features

1. Introduction

In the process of modernization, the management of a large amount of Chinese text data on the network has become a major issue. More efficient text classification management for different types of text can improve the speed of online text retrieval, lay the foundation for further network data analysis, and provide assistance in technical fields such as text reading, automatic text recognition, computer chat sentence recognition, etc. [1-3]. Although traditional machine learning text classification methods can achieve certain text classification results, the limitations of their feature extraction performance lead to higher classification performance only in regular text and short text. When they face long-text data with more text features, the classification accuracy and efficiency of classification operations will be significantly reduced [4-6]. Therefore, there is a need for an efficient text classification method that can provide the most basic information classification processing capabilities for computer deep semantic recognition technology, and lay the foundation for computer text semantic analysis technology and deep data analysis technology.

2. Related Works

The research in the field of text classification is gradually intensifying and different approaches are being applied in this field. Yao et al. [7] developed a model for the text classification problem, which is based on a single-production corpus to build a text graph. It shows the text graph network model can effectively improve the speed of text division. Akhter et al. [8] designed a machine learning long text classification method for multi-format, multi-purpose text datasets. The results showed that the division accuracy of this program was 91.8%, 95.4%, and 93.3% on large, medium, and small datasets, respec¬tively, which had superior performance advantage compared with similar models. Liu and Guo [9] employed the LSTM pattern to form a network that can extract higher-level feature representation from word vectors. The model was able to extract higher-level feature representations from word vectors and apply them to contextual representations. Finally, a softmax classifier was applied to process the contextual data. The results showed that the pattern captured local features of phrases more accurately and had higher accuracy in overall classification. Huan et al. [10] combined multi-scale neural networks with bi-directional networks to form a new text classification model. Hybrid attention was applied to extract deep semantics and deep semantics were combined with shallow semantics to form a more com¬prehensive and accurate classification model. The results showed that the program could significantly improve the classification of text. The authors of [11] designed a hybrid recurrent attention network, which combined a long-short term neural network with a network to catch contextual data information in both directions and judged the long-term dependency of long text to capture words with higher importance weight from the text. The results of the study showed that the method was highly practical in different dataset types.

On the other hand, as a more mature algorithmic tool, the applications of long short-term memory (LSTM) are gradually diversifying. Yu et al. [12] explored the cellular learning capability of LSTM and distinguished them into two main parts: LSTM-dominated networks and integrated LSTM. The future directions of both algorithms were analyzed. Bukhari et al. [13] proposed a financial market forecasting model combining fractional order derivatives and LSTM neural networks. Results showed that the program was effective in financial data calculation. Alhussein et al. [14] combined networks with LSTM to form a comprehensive deep learning model, which could make a more comprehensive and accurate predictions of single-household short-term household electricity compliance. Ding et al. [15] proposed a prediction model combining LSTM and attention mechanism for the flood prediction problem, which applied variable control methods to the selection of hyperparameters and focused on the interpretation of attention weights. The results of the study showed that in most cases, the model was better than those of the same type of models and had some validity.

This study uses LSTM as the basic model, combines it with attention mechanism, and uses bi-directional encoder representations from transformers (BERT) model for text processing to form a higher performance long-text classification model, which has better application effects.

3. Design of Chinese Long-Text Classification Pattern

3.1 Long-Text Vector Processing Strategy

The proposed long-text classification model is broken into two portions. The first portion is a BERT training model for long-text word vectors and sentence vectors, and the second portion is a bi-directional long short-term memory (Bi-LSTM) model. The overall pattern tectonic diagram is denoted in Fig. 1.

The model structure includes five parts: word embedding, sentence vector integration, convolutional neural network (CNN), Bi-LSTM, and attention layer. The model first segments the long text, and then uses the BERT model to obtain sentence vectors and word vectors. Subsequently, the CNN layer captures local text features, combines sentence and feature vectors, and forms global text information through Bi-LSTM. The attention mechanism improves classification accuracy. In the model preprocessing step, images, chaotic characters, etc., are discarded, and only the text corpus is retained. After removing the stop words, the text length threshold is determined. According to the threshold, text is processed. If it is too long, it is deleted, and if it is too short, it is supplemented. Long text is processed using segmentation to meet BERT input requirements and avoid information loss. The model mainly includes input layer, convolution layer, pooling layer, and fusion layer.

The input layer is a matrix constructed based on word vectors in the form of [TeX:] $$n \cdot d$$, where n is the longitude representation of the local version and d is the dimensional representation of the word vectors. The single row of the input layer corresponds to the d-dimensional word vector corresponding to a single word, and the convolutional layer word is extracted using the multiple sets of convolutional kernels of different sizes to achieve the extraction.

In Eq. (1), W means the convolution kernel with the shape features [TeX:] $$h \cdot d$$. h is the height representation of kernel specification, d is the dimensional representation of word vector, and [TeX:] $$X_{i: i+h-1}^j$$ is the vocabulary vector matrix from the i-th word to i+1 word of the j group of data in local text. [TeX:] $$f(\cdot)$$ activation function in the nonlinear state is represented by b, which is the quantized representation of the bias term. The convolution count relies on the vocabulary vector of the local text, and the result of the convolution operation can be obtained as Eq. (2):

In Eq. (2), n-h+1 denotes the dimensionality specification of the result vector generated by the convolution kernel. The results obtained by the convolutional level operation need to be reduced by the pooling layer, and this dimensionality reduction process can help the model avoid the problem of overfitting. Then the stabilization of the patterns is improved in the process of part characteristic gain, and the study mainly adopts the maximum pooling method for the collection of the most text characteristic. The formula of maximum pooling operation is shown in Eq. (3):

This study used different specifications of convolutional kernels in the feature extraction process, which are mainly divided into three different size specifications of small and large. The feature vectors formed by fusion after pooling of different size convolutional kernels are shown in Eq. (4):

The sentence vector formed by the BERT pattern is combined with the feature vector obtained by the CNN model, therefore final sentence vector of local text is formed as shown in Eq. (5):

In Equation (5), [TeX:] $$l_j$$ denotes the BERT model forming sentence vectors, and [TeX:] $$z^j$$ denotes the feature vectors obtained by the CNN model.

3.2 Bi-LSTM Model Design

The Bi-LSTM pattern used in the study mainly consists of a cell structure and three groups of gated cells, which are input-gated cells, forget-gated cells, and output-gated cells. The specific structures are shown in Fig. 2.

In Fig. 2, the Bi-LSTM model takes the part characteristic of text data [TeX:] $$s_t$$ as the input vector based on the temporal order, [TeX:] $$c_{t-1}$$ means recall unit of a moment ago of the moment, and [TeX:] $$h_{t-1}$$ represents the result of a moment ago of the moment. Since the LSTM model used in the study is a bi-directional acquisition of the up and down two-way features of the text, the model forward hidden state output at the current moment is shown in Eq. (6):

The model backward hidden state output at the current moment is shown in Equation (7).

Since the model cannot focus on the data of all nodes, the study introduces an attention mechanism to assign weights or a weight to characteristics of the export data of different nodes for fusion. The specific structure is as Fig. 3.

In Fig. 3, Query represents the decoder, while key and value are both encoders, and the decoder content is finally formed by the encoder to obtain the attention focus. In the process of attention calculation, the correlation degree of the encoder key and value needs to be calculated first to get the weight coefficient corresponding to the encoder key. The specific calculation process of the weight coefficient is shown in Eq. (8):

The [TeX:] $$f(\quad)$$ in Eq. (8) denotes the scoring function of the attention mechanism. The final long-text classification probability is expressed as:

In Eq. (9), [TeX:] $$W_c$$ denotes the weight matrix between the final hidden hierarchy of the model and the output classification, and [TeX:] $$b_c$$ denotes the bias.

4. Analysis of the Application Effect of Long-Text Classification Model

4.1 Analysis of the Impact of Word Vectors and Local Text on Performance

When analyzing the application effect of the long-text classification model, the research will analyze it from the local performance and the overall classification application effect. The training effects of models with different local-text lengths under the progression of training times are shown in Fig. 4.

From Fig. 4, although the overall performance convergence speed was not as fast as the advantages in maritime data collection environments, it still had the best performance among all specifications. It can be seen that L200 is the optimal local text length for this model.

4.2 Analysis of the Application Effect under Different Datasets

The study compared the model designed in the study with the recurrent neural network (RNN)-text model, CNN-LSTM model, and hierarchical attention networks (HAN) model, respectively. The comparative analysis of classification is in Fig. 5.

From Fig. 5, it can be seen that in the comparison of classification accuracy, in the maritime dataset environment, the classification accuracy of the model designed in this study was 92.93%. In the Chinese dataset of Fudan University, the classification accuracy of the model designed in this study was 93.77%, indicating that the classification accuracy of the designed model was the highest. In the comparison of calculation time, the model studied and designed in the maritime dataset had a calculation time of 85.42 seconds, while in the Chinese dataset of Fudan University, the running time of the model studied and designed was 117.51 seconds. It can be seen that the model studied and designed in the maritime dataset environment had the shortest calculation time.

5. Conclusion

To solve the problem of insufficient classification accuracy caused by the difficulty in extracting semantic features in long-text classification, this study improved the LSTM model and introduced an attention mechanism, incorporating the BERT model into the text vector processing classification model for long-text data. The accuracy of this model was 92.93% and 93.77%, respectively, which was the highest among the same type of models. The calculation time was 85.42 seconds and 117.51 seconds, respectively, which was the shortest among the same type of models. The research designed model had the fastest convergence speed among the same type, which meant that it achieved the best classification effect the fastest. It overcame the problem of classification failure caused by complex text features in long-text classification environments, and also provided the possibility for specific classification paths for long texts. Although the models designed in the study had significant advantages, research mainly focused on long-text classification. Therefore, designing comprehensive and robust long- and short-text classification models is the main research direction in the future.

Conflict of Interest

The authors declare that they have no competing interests.

Funding

The research was supported by: Research on image super-resolution reconstruction based on depth learning (No. JJKH20221281KJ).