Sage Journals: Discover world-class research

Abstract

In the era of rapid internet expansion and technological progress, discerning real from fake news poses a growing challenge, exposing users to potential misinformation. The existing literature primarily focuses on analyzing individual features in fake news, overlooking multimodal feature fusion recognition. Compared to single-modal approaches, multimodal fusion allows for a more comprehensive and enriched capture of information from different data modalities (such as text and images), thereby improving the performance and effectiveness of the model. This study proposes a model using multimodal fusion to identify fake news, aiming to curb misinformation. The framework integrates textual and visual information, using early fusion, joint fusion and late fusion strategies to combine them. The proposed framework processes textual and visual information through data cleaning and feature extraction before classification. Fake news classification is accomplished through a model, achieving accuracy of 85% and 90% in the Gossipcop and Fakeddit datasets, with F1-scores of 90% and 88%, showcasing its performance. The study presents outcomes across different training periods, demonstrating the effectiveness of multimodal fusion in combining text and image recognition for combating fake news. This research contributes significantly to addressing the critical issue of misinformation, emphasizing a comprehensive approach for detection accuracy enhancement.

Keywords

Fake news detection multimodal fusion natural language processing text classification deep learning

Introduction

In the era of advanced technology today, various types of information are rapidly transmitted over the internet, and the speed of dissemination is correspondingly swift. There has been a significant transformation in the dissemination methods, from traditional means such as newspapers, magazines, posters and television to the current internet-based communication. The process has evolved from one-way message delivery to users being able to engage in interactions on platforms, further spreading information through sharing and commenting. Social media platforms such as Twitter, Facebook and Line are among the most used platforms for news commentary.^1–3 However, these platforms also witness a significant amount of fake news, which quickly spreads widely on social networks due to their low cost and high speed of dissemination.⁴ Upon accessing information on the internet, users cannot accurately determine whether an article is real or fake news in the first instance. The content of an article is based on the information received by the author and written from their perspective, with various motivations behind the writing. These motivations can include malicious intent, political biases, the deliberate dissemination of incorrect information, and the fabrication of events. Such misleading information is collectively called “fake news,”⁵ which can mislead users, leading to incorrect information and erroneous judgments. Incorrect information can become a threat as it spreads rapidly,⁶ resulting in significant losses, especially on social media platforms. Examples include false claims about COVID-19 and fraud during the 2016 US presidential election,⁷ both propagated extensively through social media, causing many users to make erroneous judgements.⁸

The textual and visual information of fake news are closely related. Scholars have pointed out that articles with accompanying images not only attract readers’ attention but also enhance the persuasiveness of the content. Consequently, unlike in the past when fake news articles were primarily text-based, nowadays news articles and any content intended to capture public attention utilize highly provocative images to attract the masses, including fake news. Artificial intelligence (AI) technologies have become increasingly mature in recent years, therefore many researchers have focused on the issue of fake news detection employing various machine learning and deep learning techniques.⁹ The accuracy of these methods has achieved impressive results, with some studies employing multiple levels and categories to identify fake news, using different machine learning and deep learning classifiers, resulting in accuracy rates of over 90%. Other scholars have used the characteristics of text, social media, and users to obtain features and identify instances of online bullying through different machine learning classification methods.¹⁰ Numerous researchers have successfully employed AI methods in their studies of fake news detection. Therefore, this research will adopt an AI-based approach to detecting fake news and related studies. However, previous research on fake news detection has relied chiefly on single-text analysis or natural language processing techniques, with less emphasis on multimodal analysis and the combined analysis of textual and visual information. Neglecting either of these modalities may limit the comprehensive exploration of online fake news issues. Hence, this study employs a multimodal fusion approach, using different fusion techniques to merge textual and visual information features, aiming to improve the quality of fake news prediction. When using both text and images, data imbalance issues may arise, leading to an inability to accurately convey information. Therefore, we incorporated additional preprocessing methods in the experiments to enhance the final performance. The contributions of this research are as follows.

This study proposes a model using multimodal fusion to identify fake news, aiming to curb misinformation. The framework integrates textual and visual information, undergoing data cleaning before feature extraction.

The model combines different modalities using a multimodal fusion approach, showing an advantage over single-modality methods like the Bidirectional Encoder Representations from Transformers (BERT). Using the Gossipcop and Fakeddit datasets, accuracy for title text features was only 63% and 83%, respectively, while for image features, it was 72% and 65%, indicating the need for a multimodal approach that achieves accuracy rates of 85% and 90%.

The fusion of textual and image data significantly enhances accuracy and F1-score. Early fusion tests improved accuracy and F1-score by up to 15% and 17%, respectively. Late fusion tests further improved accuracy by an additional 3% and F1-score by 4%, resulting in a final accuracy and F1-score of 90%.

The proposed model does not only achieve results comparable to deep learning architectures more quickly but also offers flexibility. The best F1-scores for the Gossipcop Dataset and the Fakeddit Dataset were 90% and 88%, respectively, after implementing joint fusion and late fusion together, demonstrating its potential for practical application in fake news detection.

Multimodal fusion

Multimodal fusion is the combination of multiple forms of information, such as text, speech, images, sound and waves, which can be considered different modalities.¹¹ Each modality has its way of representation and exhibits different patterns, but there may be mutual influence and complementarity among different modalities. The characteristics of multimodal data are redundancy and complementarity, and by effectively extracting feature information from each modality, richer information can be obtained.^12,13 Multimodal fusion methods can be classified into three types: early fusion, joint fusion and late fusion.¹² Different fusion methods can affect the recognition performance of the model, and more details can be found in the ‘Multimodal fusion strategies’ section.

2.1 Multimodal fusion strategies

Feature fusion methods can be classified into three types:

Early fusion

After extracting features from different modalities, the features are fused and inputted into the same model for training. This process is called rare fusion. In Figure 1(a), early fusion Type I refers to fusing the original features, while Type II refers to fusing features extracted through neural network methods.

Figure 1.

Multimodal fusion Methods¹².

Joint fusion or intermediate fusion

Joint fusion is a method that transforms data of different modalities into high-dimensional representations and performs fusion at the intermediate layers of the model. Its main advantage lies in the flexibility to choose the fusion location. Compared to early fusion, the critical difference in joint fusion is that the Loss value is returned from the prediction model to the feature extraction model in the neural network during the training process, creating better feature representations for each training iteration. In Figure 1(b), joint fusion is divided into two types: Type I indicates that both features require feature extraction. By contrast, Type II indicates that one of the features already contains the required information in the dataset and does not need to be extracted again.

Late fusion

Late fusion refers to the process of combining the results predicted by multiple models for different modalities to make the final decision. Therefore, late fusion is also known as decision-level fusion, as shown in Figure 1(c).

Fake news detection

Fake news spreads rapidly through platforms and social media, making dissemination faster, cheaper and more accessible. With the development of the internet, anyone can use the web or social networks to propagate their own messages or harmful content, leading to the rapid spread of fake news. Therefore, identifying fake news has attracted significant attention from researchers and has become an important research topic. Numerous studies have focused on using artificial intelligence methods for text detection to identify fake news. S. R. Sahoo et al. have conducted fake news research using the social media platform Facebook,¹⁴ wherein they proposed a fake news detection method using machine learning and deep learning classifiers in the Chrome environment, yielding promising results with an accuracy rate as high as 99.4%. Other researchers have proposed deep learning methods for fake news detection, such as a hybrid convolutional neural network–recurrent neural network (CNN-RNN) approach¹⁵ or enhanced deep learning long short-term memory (LSTM) techniques.¹⁶

However, combining textual and visual information can provide various cues for fake news detection.¹⁷ Therefore, fake news detection based on multimodal information has become a recent research focus.^18–22 In the past few years, some studies have started investigating fake news's visual features.^23,24 Since it is challenging to find images that match the fabricated stories when creating fake news, fake images inconsistent with the textual content are often used.²⁵ As online articles and news reports often include a significant amount of text and images, such as screenshots of news-related reports, message screenshots and any other visuals, extracting relevant features between images and text is crucial. Deep learning has achieved remarkable success in text classification, and in recent years, language pre-training models such as Embeddings from Language Model (ELMo), Generative Pre-trained Transformer (GPT) and BERT²⁶ have brought breakthroughs in text processing. Hua, J. et al. proposed the BERT-based back-Translation Text and Entire-image multimodal model with Contrastive learning (TTEC)²⁷ to address the challenge of detecting fake news, which suffers from low efficacy due to limited multimodal information and small data size. The experimental results demonstrate that TTEC achieves a macro F1-score of 0.805. Jing, J. et al. propose a progressive multimodal rumor detection model,²⁸ which consists mainly of a BERT-based text feature extractor, a transformer-based visual feature extractor and a progressive multimodal feature fusion process. The method achieved 83.3% accuracy on the Twitter dataset. Despite significant progress in multimodal fake news detection, several challenges remain. First, although some existing models attempt to improve performance by leveraging multimodal representations, they often utilize only the final layer of pre-trained models, such as BERT,²⁶ to realize text representations, neglecting the contextual information in multimodal representations. Second, despite efforts to combine representations from different modalities, the aggregation of visual and textual features is still inadequate. It requires utilizing the complementarity of information and the similarity between different modalities. Third, most current research investigates the performance of fake news detection on a single data type or specific domain datasets. However, research on the generalization ability of multimodal methods to domain-independent and domain-specific datasets is insufficient. Therefore, multimodal approaches can address the above questions and have become a popular topic in fake news detection in recent years.

Methodology

Research framework

This study aims to distinguish between fake news and real news using different fusion methods, as shown in Figure 2. The flowchart describes from top to bottom a fake news detection method that enhances accuracy through multimodal fusion technologies. It starts with analyzing the data sources, for example Gossipcop and Fakeddit, with a set of tools to scrub the data, which can be transformed into a format for the machine learning models.

Figure 2.

Research framework.

Within the feature extraction phase, content text features, headline text features and image features are processed by the BERT and ResNet50 models, respectively. The BERT model is used to process textual data. Its bidirectional processing feature makes it particularly strong in text comprehension, enabling it to capture the subtle semantic differences in the context of news articles. The ResNet50 model is used for image data analysis and effectively identifies and processes visual information related to news through its deep residual network structure. This structure allows it to engage in deeper learning without causing training difficulties, which is very useful for extracting complex features from images. After feature processing, the flowchart delineates three fusion strategies: early fusion, joint fusion and late fusion, aimed at combining text and image features. Finally, these combined features are input into several machine learning models, including Support Vector Machine (SVM), Logistic Regression (LR), XGBoost (Extreme Gradient Boosting) and CatBoost, for the final fake news identification. In this process, both textual and pictorial contents within news data are considered to achieve greater accuracy in the veracity of news information. Furthermore, different machine learning models are employed to train and test the variability among different variables, ensuring the correctness of the information. This sequence of steps constitutes a framework for fake news detection that integratively utilizes text and image data. The following is a more detailed description within the research framework. The specifications of the system used in this study are presented in Table 1.

Table 1.

Development environment specifications.

	Version
Hardware facilities and environment	CPU：Intel(R) Core(TM) i9-10900 K CPU @ 3.70GHz
	Memory：64 GB
	GPU：NVIDIA GeForce RTX 3090 *2
	CUDA version：11.1
Software environment	Python: 3.8.5
	Scikit-learn 0.24.0
	Simple Transformers 0.61.4
	PyTorch 1.7.0
	Genism 3.8.3
	Transformers 4.5.1

Fake news-related datasets

Fakeddit dataset

This study uses the Fakeddit dataset²⁹ as the training dataset, which includes textual and image data of fake news across multiple categories, forming a novel multimodal dataset. The original paper categorized the samples in this dataset into binary, 3-class and 6-class classifications, with corresponding highest accuracies of 0.8909, 0.8890, and 0.8588, respectively. In this research, we experiment and compare using the binary labels from this dataset. In total, the dataset comprises 562,707 samples of both real and fake news data, with 39% (220,140 samples) representing fake news and 61% (342,568 samples) representing real news.

Gossipcop dataset

The Gossipcop dataset is provided by the Gossipcop website. The website contains records of various fake news articles and is considered a credible source. The dataset consists of a total of 21,666 samples. However, since the data collection for this dataset was conducted in 2019, some websites mentioned may have been taken down or are no longer accessible. Therefore, the pre-processed dataset used in this study contains fewer samples, as explained in the ‘Data pre-processing’ section. Of the available samples, 5196 are labeled as fake news and 16,470 are labeled as actual news, totaling 21,666 samples.

Gossipcop is a reputable dataset focused on celebrity news, offering verified true and false news, essential for studying fake news in a highly prevalent domain. Fakeddit, a novel multimodal dataset, includes extensive text and image data across multiple fake news categories with rich annotations, making it ideal for exploring multimodal fusion techniques. These datasets provide a broad coverage, diverse data types and credible information, essential for validating the effectiveness and generalizability to the model of this study.

Data pre-processing

As mentioned previously, the fake news datasets used in this study came from the Fakeddit and Gossipcop websites, and consisted of raw, unprocessed textual data. Tables 2 and 3 provide simple statistics of the data from the two sources. Although the datasets seem large, the amount of available data is significantly reduced after data cleaning. This is primarily due to many URLs failing to reconnect and losing various data instances.

Table 2.

Fake news datasets.

Website	Fake news		Real news		Total
	Quantity	Percentage	Quantity	Percentage	Total
Gossipcop Number of Original Data	5196	24%	16,470	76%	21,666
Fakeddit Number of Original Data	220,140	39%	342,568	61%	562,708
Gossipcop Number of Cleaned Data	4331	24%	13,557	76%	17,888
Fakeddit Number of Cleaned Data	37,283	40%	56,896	60%	94,179

Table 3.

Train and test data.

Website	Train data total		Test data total
Website	Fake news	Real news	Fake news	Real news
Number of Gossipcop Data	A total of 12,466 records		A total of 5422 records
Number of Gossipcop Data	1620 records	10,846 records	2,711 records	2,711 records
Number of Fakeddit Data	A total of 79,265 records		A total of 14,914 records
Number of Fakeddit Data	29,826 records	49,439 records	7,457 records	7,457 records

Data cleaning

After obtaining the dataset, this study first performs English tokenization on the data, filtering out punctuation marks, stickers, and hyperlinks and removing unnecessary noise. The data are then processed with stop word removal.²⁴ Lastly, an error-correction spell check is conducted to correct any misspelled English terms and replace them with the correct words, ensuring accurate interpretation and judgment when extracting data features.

Feature extraction

Word2vec and Doc2vec

This study utilizes two methods for text feature extraction: Word2vec³⁰ (word embeddings, word vectors) and Doc2vec³¹ (sentence embeddings, paragraph vectors, Paragraph2vec). Word2Vec is used to transform words into vector space, capturing the semantic relationships between words within the context. Doc2vec represents features of sentences in a similar way that vectors represent individual words, mapping each sentence to a vector space where semantically similar sentences are positioned close to each other. By using Doc2Vec, the study can analyze the textual content at a document level, incorporating a broader context than Word2Vec's word-level analysis. This method allows the system to capture the essence of entire articles, aiding in the identification of fake news by analyzing how topics are discussed or presented in the document. The well-known natural language processing (NLP) library, Gensim,³² obtains 50 Word2vec features for each text content and 50 Doc2vec features for each text content.

BERT feature vectors

In this study, BERT Base feature vectors with a dimension of 768 are obtained. BERT has two versions: the Base version used in this research, with a hidden state size of 768, and the Large version, which has a hidden state size of 1024. The number 768 represents the hidden state size, indicating that each token is transformed into a 768-dimensional vector. The result of outputs from all layers forms a three-dimensional tensor where the dimensions are the number of tokens, the hidden state size (768), and the number of layers (12).

Image feature vectors

How computers discern images is different from how humans perceive them. While humans can directly recognize the shape and color of an image, computers interpret images as combinations of numerical values. In other words, an image can be seen as an organized collection of numbers. In this study, the image model employs ResNet50 to extract the image features. ResNet50 is a widely used model known for its moderate size and excellent performance in recent years. The research extracts the 1000-dimensional vector obtained from the last convolutional layer of ResNet50 as the image features.

Fusion methods

This study combines two types of features, text and image, to enhance the accuracy and diversity of the model. There are three fusion methods: early, joint, and late. These three methods are introduced as follows.

Early fusion

This study employs the early fusion method using the Concat and Add³¹ techniques, as shown in Figure 3.

(1) Concat

Figure 3.

Early fusion.

In this study, the early fusion method uses the concat and add techniques, as shown in Figure 3. The following Equation (1) represents the concatenation of individual output channels:

Z_{c o n c a t} = \sum_{i = 1}^{c} X_{i} * K_{i} + \sum_{i = 1}^{c} X_{i} * K_{i + c}

(1)Notation explanation: X = the first feature value, K = Kernel, c = number of features.

(2) Add

The add method involves adding the features together, keeping the number of channels unchanged. In this study, the dimension of the image or text features remains fixed, even when the amount of information increases. The add operation is performed by adding the corresponding features and then proceeding with the subsequent convolutional operation. Add can be considered a particular form of concat, where the computation is simplified compared to concat. If the number of channels is the same and followed by convolution, add is equivalent to concat followed by sharing the same convolutional kernel for corresponding channels. The following Equation (2) represents the addition of individual output channels:

Z_{a d d} = \sum_{i = 1}^{c} (X_{i} + Y_{i}) * K_{i} = \sum_{i = 1}^{c} X_{i} * K_{i} + \sum_{i = 1}^{c} Y_{i} * K_{i}

(2)Notation explanation: X = the first feature value, Y = the second feature value, K = Kernel, c = number of features.

Joint fusion

Joint fusion refers to transforming different modal data into high-dimensional feature representations and fusing them at the intermediate layer of the model. In this study, we employ text and image modal features. First, both modal data are transformed into high-dimensional vectors. Then, element-wise multiplication is performed, followed by a sum pooling operation. This yields the result of the joint fusion method, as illustrated in Figure 4.

Figure 4.

Joint fusion.

Late fusion

Late fusion involves feature fusion using averaged fusion³³ and ensemble learning³⁴ methods, as shown in Figure 5.

(1) Averaged fusion

Figure 5.

Late fusion.

The results obtained from all classifiers are averaged to obtain the final prediction result. For example, if Classifier 1 predicts 60% as fake news and 40% as real news, and Classifier 2 predicts 35% as fake news and 65% as real news, taking the average of these results yields 47.5% as fake news and 52.5% as real news. Therefore, the result for this data instance is considered as real news.

(2) Ensemble learning

Ensemble learning aims to generate a more robust model by integrating multiple models. It mainly consists of three methods: Bagging,³⁵ Boosting³⁶ and Stacking.³⁷ In this study, we employ the XGBoost³⁸ and CatBoost³⁹ methods for the final training. XGBoost and CatBoost are based on the gradient boosting–tree⁴⁰ framework in Boosting. Each training data item is assigned different weights in Boosting based on its difficulty. For example, if a wrong judgment is made in the first training, its weight will be increased in the following training to make it easier for the following estimator to extract. This process is repeated to train a stronger estimator and improve accuracy. CatBoost is a relatively new open-source machine learning algorithm in the Boosting method. Therefore, in addition to XGBoost, CatBoost is also used for classification prediction in this study.

Classification models

As mentioned previously, this study employs four different classification models to accurately distinguish between authentic and fabricated content to achieve the fake news detection process: Logistic Regression⁴¹ (LR), Support Vector Machine⁴² (SVM), XGBoost³⁸ and CatBoost.³⁹ LR analyzes the relationship between dependent and independent variables. In this study, LR is used to determine whether the content of the text and image dataset is fake news, making it suitable for binary classification. SVM is effective in solving classification and regression problems with high-dimensional features. It performs well even when the number of features exceeds the number of samples, and it exhibits high accuracy and strong capability regardless of the data size. XGBoost leverages Boosting techniques to combine multiple decision trees into a more robust predictive model, which can be used for classification and regression tasks, including predicting continuous numerical values. This approach enhances the accuracy and reliability of identifying fake news in text and image datasets. One of CatBoost's advantages is that it reduces the need for extensive hyper-parameter tuning, thereby lowering the risk of overfitting and making the model more generalizable. It is particularly well-suited for handling categorical and numerical features, which aligns perfectly with the requirements of this study.

Evaluation metrics

To accurately evaluate the performance of the fake news detection model, this study employs four key evaluation metrics: Accuracy, Precision score, Recall score and F1-score. The calculation formulas for these metrics are presented, respectively, as Equations (3), (4), (5) and (6). Recall, also known as sensitivity, is directly used in the F1-score calculation. Higher sensitivity improves the F1-score if precision remains constant. F1-score provides a balance between sensitivity (recall) and precision, offering a comprehensive view of a model's accuracy in both identifying positives and avoiding false positives.

Accurary = \frac{TP + TN}{TP + TN + FP + FN} \times 100 %

(3)

Precision = \frac{TP}{TP + FP} \times 100 %

(4)

Recall score = \frac{TP}{TP + FN} \times 100 %

(5)

F 1 - score = \frac{2 \times Precision \times Recall}{Precision + Recall} \times 100 %

(6)These metrics are derived from the confusion matrix, which consists of four elements: True Positives (TP), True Negatives (TN), False Positives (FP) and False Negatives (FN). The confusion matrix is commonly used to evaluate the performance of classification models and serves as a reference for evaluation metrics, especially in supervised learning. The predicted result matches the actual result, indicating a correct prediction. In the context of fake news classification, TP refers to predicting a news article as fake, and the actual result confirms it as fake. TN refers to predicting a news article as real, and the actual result confirms it as real. Successful prediction of real news falls under the category of TN. FP refers to predicting a news article as fake, but the actual result confirms it as real. Incorrectly predicting a real news article as fake falls under the category of FP. FN refers to predicting a news article as real, but the actual result confirms it as fake. Incorrectly predicting a fake news article as real falls under the category of FN.

Experiments

As mentioned previously, this study employs four different classification models, LR, SVM, XGBoost and CatBoost, to accurately distinguish between authentic and fabricated content, aiming to achieve effective fake news detection. Each of these models is chosen for its unique strengths and methodologies, making them suitable for various aspects of data classification involved in the detection process. To optimize the performance and accuracy of these models, specific parameters and settings were utilized. The image shape is set to (64, 64, 3), which accommodates the three color channels in the images. For text data, the shape is set to (100). The embedding dimension is configured at 128 to effectively capture the semantic information in the text data. The models are designed with 64 hidden units, a dropout rate of 0.3 to prevent over-fitting, and a learning rate of 0.001 to ensure stable training. The training process runs for 10 epochs with a batch size of 32. In terms of the network layers, the architecture is detailed as follows: the image processing component comprises 50 layers, the headline processing part consists of 13 layers, the body text processing also includes 13 layers, the fusion and classification part incorporates 3 layers. These configurations collectively contribute to the robustness and efficiency of the fake news detection models used in this study.

Through the research framework presented in Figure 2, this section encompasses four experimental scenarios and their results. Experiment #1 aims to study the impact of single-modal data and data imbalance on the accuracy of fake news detection algorithms. Its purpose is twofold: to determine how the disproportion between fake news and real news in the dataset affects the reliability of prediction results and to evaluate the effectiveness of single-modal data balancing techniques in achieving balanced representation and more accurate classification outcomes. The primary focus of Experiment #2 is to examine the effectiveness of different fusion methods, specifically Concat fusion and Add fusion, in the context of early fusion for fake news detection. It aims to explore how varying the dimensionality of image features and combining them with text features affects the model's performance in terms of accuracy, F1-score, and recall, particularly across different sizes of datasets. The research investigates the optimal combination of text and image data to improve the detection of fake news.

Moreover, Experiment #3 primarily investigates the effectiveness of the joint fusion method in the classification and prediction of fake news. It aims to determine how the combination of features transformed into high-dimensional vectors at the intermediate layer of the model impacts the overall performance, as measured by accuracy and F1-score, in different datasets (Gossipcop and Fakeddit). The study seeks to understand the improvements in fake news detection accuracy and precision when employing this joint fusion strategy. Experiment #4 primarily aims to explore the effectiveness of late fusion methods, specifically the averaged fusion method and ensemble learning method, in the classification and prediction of fake news. It focuses on determining how these late fusion techniques, which involve averaging results from separate classifiers for image and text information, impact the accuracy, recall rate and F1-score of fake news detection in different datasets. The goal is to assess whether these late fusion methods provide an improvement in identifying fake versus real news.

Experiment #1: data balancing comparison using a single-modal approach

The final purpose of Experiment #1 is to separately incorporate all textual and image single-feature information into past classification models such as LR, SVM, XGBoost, and CatBoost for comparison. First, the data undergo basic feature processing separately, with text features extracted using BERT and image features extracted using ResNet50 to obtain 1000-dimensional vectors. We integrate and compare the results obtained from different features to provide a basis for comparison and judgment in subsequent feature fusion experiments. Previous research has extensively used single features to identify fake news, with most studies focusing on text recognition. Studies using single-image recognition for fake news detection are relatively scarce. Therefore, this paper aims to observe whether conclusions drawn from text and images affect the research results, further subdividing the text into title and content features.

However, the data used in this study exhibit a significant imbalance in the proportion of fake news to real news. Table 4 presents the quantities of imbalanced and balanced datasets, and the original data are split into training and testing sets. In the training dataset, there is a notable disparity between the proportions of fake news and real news, especially in the Gossipcop Dataset, where the ratio of fake news to real news is 3:17. The data imbalance can potentially affect accuracy. There is a need to employ data balancing techniques to achieve reliable accuracy and obtain the most accurate predictive results. After data balancing, the number of fake and real news in the Gossipcop Dataset is determined to be 10,846, resulting in a 1:1 ratio. Similarly, in the Fakeddit Dataset, the number of fake news is adjusted to match the number of real news, resulting in 29,826 instances for both classes, also with a 1:1 ratio. The ratio of the training set to the test set is 8:2.

Table 4.

Data balancing—imbalanced versus balanced dataset quantities.

	Gossipcop Dataset			Fakeddit Dataset
Dataset	Total		Real News: Fake News	Total		Real News: Fake News
Dataset	Fake News	Real News	Real News: Fake News	Fake News	Real News	Real News: Fake News
Imbalanced Train Original Data	12,466		3:17	79,265		2:3
Imbalanced Train Original Data	1620	10,846	3:17	29,826	49,439	2:3
BalancedTrain Data	21,692		1:1	59,652		1:1
BalancedTrain Data	10,846	10,846	1:1	29,826	29,826	1:1
Test Data	5422		1:1	14,914		1:1
Test Data	2711	2711	1:1	7457	7457	1:1

The results and improvements in model recognition performance after data balancing can be observed in Tables 5 and 6. From Tables 5 and 6, it can be observed that in the case of using imbalanced data, when using pure textual BERT features, the F1-score reaches a maximum of 0.82 in the Fakeddit Dataset, surpassing the value of 0.73 in the Gossipcop Dataset. Furthermore, when solely utilizing image features (ResNet50), as presented in Table 6, the highest F1-score in the Gossipcop Dataset is 0.77, outperforming the 0.72 in the Fakeddit Dataset. Consequently, it can be inferred that in imbalanced scenarios, pure textual recognition outperforms pure image recognition, especially in the case of the Fakeddit Dataset.

Table 5.

Comparison between balanced and imbalanced data—title text feature.

Dataset	Gossipcop Dataset				Fakeddit Dataset
Evaluation metrics
Classifiers	Accuracy	Precision	Recall score	F1-score	Accuracy	Precision	Recall score	F1-score
LR(balance)	0.72	0.8	0.79	0.79	0.84	0.87	0.81	0.84
LR(imbalance)	0.61	0.56	0.98	0.71	0.83	0.85	0.79	0.82
SVM(balance)	0.7	0.69	0.82	0.75	0.83	0.86	0.79	0.82
SVM(imbalance)	0.6	0.56	0.99	0.71	0.83	0.86	0.79	0.82
XGBoost(balance)	0.69	0.65	0.86	0.74	0.84	0.76	0.8	0.78
XGBoost(imbalance)	0.62	0.57	0.97	0.72	0.82	0.86	0.77	0.81
CatBoost(balance)	0.72	0.7	0.85	0.77	0.84	0.8	0.82	0.8
CatBoost(imbalance)	0.63	0.57	0.98	0.73	0.83	0.86	0.78	0.82

Table 6.

Comparing balanced and imbalanced data—image feature.

Dataset	Gossipcop Dataset				Fakeddit Dataset
Evaluation metrics
Classifiers	Accuracy	Precision	Recall score	F1-score	Accuracy	Precision	Recall score	F1-score
LR(balance)	0.6	0.71	0.85	0.77	0.64	0.92	0.82	0.87
LR(imbalance)	0.5	0.5	1	0.67	0.63	0.88	0.95	0.71
SVM(balance)	0.59	0.74	0.88	0.8	0.68	0.89	0.83	0.86
SVM(imbalance)	0.5	0.5	1	0.67	0.65	0.87	0.94	0.72
XGBoost(balance)	0.75	0.79	0.75	0.77	0.65	0.92	0.79	0.85
XGBoost(imbalance)	0.71	0.63	0.97	0.77	0.63	0.89	0.95	0.72
CatBoost(balance)	0.76	0.8	0.76	0.78	0.64	0.88	0.89	0.88
CatBoost(imbalance)	0.72	0.63	0.97	0.76	0.63	0.89	0.94	0.72

However, in balanced scenarios, the improvement in the F1-score is relatively modest when using pure textual BERT features. By contrast, with pure image features (ResNet50) in Table 6, the F1-score reaches a remarkable 0.88, a 16% improvement, demonstrating that in balanced conditions, pure image recognition surpasses pure textual recognition, with the most significant impact seen in the Fakeddit Dataset.

After balancing, the Gossipcop Dataset shows an improvement of 11% in accuracy and 24% in precision but a decrease of 19% in recall, resulting in a decrease in overall accuracy. The overall F1-score has improved by a maximum of 8%. In the Fakeddit Dataset, the balanced dataset shows improvements of up to 2% in accuracy, 2% in precision, 4% in recall and 2% in F1-score. This indicates an increase in the proportion of correctly predicted fake news, with improvements observed in all metrics. Compared to the Gossipcop Dataset, the Fakeddit Dataset had a smaller imbalance in the original data between true and fake news, resulting in a maximum difference of 4% after balancing the data.

This study presents the results in terms of accuracy and F1-score. Overall, the results show improvements in all single-feature recognition methods but decreased recall, indicating an increase in false predictions for true news. It is observed that the overall improvement in the Gossipcop Dataset is larger than in the Fakeddit Dataset, which is due to the larger disparity in the true and false news proportions in the Gossipcop Dataset compared to the Fakeddit Dataset. Therefore, balancing the data can significantly enhance the overall performance, and the more imbalanced the data, the poorer the results. When faced with real-world scenarios with highly imbalanced data, data balancing technology can be used to improve the reliability and accuracy of the fake news detection systems. Based on these comparative results, in subsequent experimental research, we will use balanced data for model training and conduct further comparisons and analysis of the outcomes.

Experiment #2: early fusion for fusion classification and prediction with each information

This study used two methods to perform the experiment under early fusion. The first method is the Concat fusion method, and the second is the Add fusion method. The following two sections provide introductions for each method, with comparisons using the Concat and Add fusion methods.

Comparison using Concat fusion method

The text and image data are combined and concatenated when using the Concat fusion method. In this study, some modifications were made when concatenating the image features. The image features consist of a total of 1000 vector features, and they were divided into three groups: Fake News All Text Features + the first 300 vectors, Fake News All Text Features + the first 500 vectors, and Fake News All Text Features + all 1000 vectors, where Fake News All Text Features include both title text features and body text features. The results of these fusion methods were compared and presented under four classifiers, as in Table 7. The following provides a detailed explanation of the experimental results of the fusion approach.

Table 7.

Performance of concatenation of title text features, body text features, and three different image vector features (300 vector features, 500 vector features, 1000 vector features) in four classifiers.

Dataset	Gossipcop Dataset					Fakeddit Dataset
	Evaluation metrics
	Classifiers	Accuracy	Precision	Recall score	F1-score	Accuracy	Precision	Recall score	F1-score
Fake News All Text Features + Image First 300 Vector Features	LR	0.61	0.58	0.99	0.73	0.84	0.84	0.86	0.85
	SVM	0.6	0.56	0.99	0.72	0.83	0.84	0.83	0.83
	XGBoot	0.69	0.64	0.97	0.77	0.82	0.83	0.81	0.82
	CatBoost	0.7	0.63	0.98	0.77	0.85	0.84	0.75	0.79
Fake News All Text Features + Image First 500 Vector Features	LR	0.63	0.57	0.99	0.72	0.85	0.87	0.78	0.82
	SVM	0.62	0.58	0.98	0.73	0.84	0.86	0.81	0.84
	XGBoot	0.71	0.63	0.98	0.77	0.82	0.83	0.76	0.79
	CatBoost	0.71	0.62	0.99	0.76	0.82	0.82	0.76	0.79
Fake News All Text Features + Image First 1000 Vector Features	LR	0.77	0.74	0.83	0.78	0.83	0.87	0.79	0.83
	SVM	0.76	0.74	0.88	0.8	0.86	0.84	0.81	0.82
	XGBoot	0.7	0.64	0.95	0.76	0.82	0.89	0.75	0.81
	CatBoost	0.74	0.67	0.93	0.78	0.85	0.88	0.76	0.82

Using different dimensional image features in combination with all text features, the results in terms of F1-score indicate that in Gossipcop Dataset, the combination of text features and the first 1000 vector features of the image outperform the combination of text features and the first 500 vector features, which in turn outperforms the combination of text features and the first 300 vector features. The respective F1-scores are 0.8, 0.77, and 0.77. The difference between the 1000-dimensional image vectors and the 500-dimensional and 300-dimensional image vectors is 0.03, indicating that reducing the dimensionality of the image features is not suitable when the dataset is small. In the Fakeddit Dataset, the F1-score shows that using text features combined with the image's first 300 vector features outperforms the combinations with the first 500 and 1000 vector features, respectively. The F1-scores obtained are 0.85, 0.84, and 0.83. The difference between the 300-dimensional image vectors and the 500-dimensional and 1000-dimensional image vectors is 0.01 and 0.02, respectively. This suggests that when the dataset is large, the dimensionality of the image features can be reduced from 1000 to 300 to decrease model complexity.

Comparison using Add fusion method

In this experiment, the Add fusion method maintained the same number of channels. The title and text information were added together, and the image was fused using 1000 vector features. The results are presented in Table 8, and further explanations are provided as follows.

Table 8.

Performance of Add fusion method on four classifiers.

Dataset	Gossipcop Dataset				Fakeddit Dataset
Evaluation metrics
Classifiers	Accuracy	Precision	Recall score	F1-score	Accuracy	Precision	Recall score	F1-score
LR	0.84	0.72	0.85	0.87	0.85	0.83	0.84	0.83
SVM	0.82	0.79	0.82	0.8	0.83	0.89	0.78	0.83
XGBoot	0.72	0.61	0.95	0.74	0.83	0.81	0.88	0.84
CatBoost	0.83	0.72	0.93	0.81	0.87	0.83	0.85	0.83

In this study, the results are presented and expressed in terms of accuracy and F1-score values. In the Add fusion method, for the Gossipcop Dataset, the best accuracy achieved was 0.84, and the best recall was 0.95, indicating that 95% of the true fake news cases were correctly identified as fake news. The best F1-score value was 0.87, representing a balanced result of precision and recall. Conversely, for the Fakeddit Dataset, the best accuracy achieved in the Add fusion method was 0.87, showing a 1% improvement compared to using a single feature. The best recall was 0.88, slightly lower than the Gossipcop Dataset performance. The best F1-score was 0.84, indicating that the Add fusion method improved accuracy and F1-score compared to individual models.

By combining textual and image features, the model can gain a more comprehensive understanding of the news content, thereby enhancing the accuracy and reliability of fake news detection. This multimodal fusion approach overcomes the limitations of single-modal information capture by providing richer data features for precise classification. However, the Concat fusion method involves handling a large number of high-dimensional features, which may increase the computational cost and training time of the model. Additionally, an excessive number of features can lead to model overfitting, especially when the training data are insufficient. Although the Add fusion method is computationally simpler, it may not fully utilize the complete information from each modality. Therefore, this experiment suggests that when dealing with smaller datasets, such as the Gossipcop Dataset, the Add fusion method can achieve the best F1-score with simpler computations. For larger datasets, such as the Fakeddit Dataset, the Concat fusion method can be employed, which, despite being more computationally complex, allows for fuller utilization of the complete information from each modality.

Experiment #3: joint fusion for fusion classification and prediction with each information

In this study, each feature was transformed into high-dimensional vectors and fusion at the intermediate layer of the model, completing the joint fusion method. That is, text and image modal features are transformed into high-dimensional vectors and predictions are made using LR, SVM, XGBoost, and CatBoost classifiers. Table 9 presents the results obtained from the two datasets individually and the comparative results. In this joint fusion, the best results can be seen with an F1-score of 0.9 for the Gossipcop Dataset and an accuracy of 0.89 for the Fakeddit Dataset.

Table 9.

Performance of the joint fusion method across four classifiers.

Dataset	Gossipcop Dataset				Fakeddit Dataset
Evaluation metrics
Classifiers	Accuracy	Precision	Recall score	F1-score	Accuracy	Precision	Recall score	F1-score
LR	0.83	0.79	0.97	0.87	0.87	0.88	0.87	0.87
SVM	0.84	0.8	0.98	0.88	0.84	0.87	0.86	0.86
XGBoot	0.85	0.85	0.97	0.9	0.89	0.84	0.93	0.88
CatBoost	0.85	0.86	0.95	0.9	0.89	0.86	0.91	0.88

We can see that using the joint fusion method has achieved noticeable improvements in the experiment. For the Gossipcop Dataset, the accuracy increased from 0.84 to 0.85, and the F1-score increased from 0.87 to 0.9. For the Fakeddit Dataset, the accuracy increased from 0.87 to 0.89, and the F1-score increased from 0.85 to 0.88. Compared to Experiment #2, we can observe that in the Gossipcop Dataset, the F1-scores using the Joint Fusion technique are superior to those of the Early Fusion's Concat and Add methods, with F1-scores of 0.9, 0.8 and 0.87, respectively. Similarly, in the Fakeddit Dataset, the F1-scores using the Joint Fusion technique also outperform the Early Fusion's Concat and Add methods, with F1-scores of 0.88, 0.5 and 0.84, respectively. By using Joint Fusion, which integrates features from different modalities at the intermediate layers, the results are clearly superior to those of the single-modal methods in Experiment #1.

Experiment #4: late fusion for fusion classification and prediction with each information

This study conducted the late fusion using the averaged fusion method and the ensemble learning method. The following two sections explain the two methods.

Averaged fusion method

The averaged fusion method obtains results separately for image and text information using individual classifiers. The study uses the weights from the last layer of the model to make a judgment. The results are then averaged, and the predicted label is determined based on which category has a higher proportion of averaged scores—either fake or real news. Table 10 presents the detailed numerical results and explanations for the averaged fusion method in late fusion.

Table 10.

Performance of averaged fusion method on four classifiers.

Dataset	Gossipcop Dataset				Fakeddit Dataset
Evaluation metrics
Classifiers	Accuracy	Precision	Recall score	F1-score	Accuracy	Precision	Recall score	F1-score
LR	0.81	0.8	0.83	0.81	0.84	0.83	0.84	0.83
SVM	0.79	0.76	0.81	0.78	0.86	0.82	0.9	0.86
XGBoost	0.84	0.68	0.97	0.8	0.89	0.83	0.86	0.84
CatBoost	0.85	0.66	0.99	0.79	0.9	0.83	0.85	0.84

Using the averaged fusion method in the Gossipcop Dataset, the best accuracy achieved was 0.85, with a recall rate of 0.99. This means that only 1% of the fake news data were incorrectly predicted, which is a relatively high proportion. The best F1-score obtained was 0.81. In the Fakeddit Dataset, the accuracy reached its highest value of 0.9, the best of all the methods in Experiments #1–4. When considering accuracy alone, this method is suitable for the Fakeddit Dataset. The results also show that multimodal fusion strategies significantly outperform single-modality methods in Tables 5 and 6 in terms of accuracy and F1-scores.

Comparison using ensemble learning method

In late fusion, the ensemble learning method integrated predictions using single-feature recognition. This study employed two methods, namely, XGBoost and CatBoost. Table 11 presents the results and comparisons for all experiments.

Table 11.

Ensemble learning performance in four classifiers.

Code	Dataset	Gossipcop Dataset				Fakeddit Dataset
	Evaluation metrics
	Classifiers	Accuracy	Precision	Recall score	F1-score	Accuracy	Precision	Recall score	F1-score
A	XGBoost(Only Title)	0.69	0.65	0.86	0.74	0.84	0.76	0.8	0.78
A	CatBoost(Only Title)	0.72	0.7	0.85	0.77	0.84	0.8	0.82	0.8
B	XGBoost(Only ResNet50)	0.75	0.79	0.75	0.77	0.65	0.92	0.79	0.85
B	CatBoost(Only ResNet50)	0.76	0.8	0.76	0.78	0.64	0.88	0.89	0.88
C	XGBoost(concat, BERT + img300)	0.69	0.64	0.97	0.77	0.82	0.83	0.81	0.82
C	CatBoost(concat, BERT + img300)	0.7	0.63	0.98	0.77	0.85	0.84	0.75	0.79
D	XGBoost(concat, BERT + img500)	0.71	0.63	0.98	0.77	0.82	0.83	0.76	0.79
D	CatBoost(concat, BERT + img500)	0.71	0.62	0.99	0.76	0.82	0.82	0.76	0.79
E	XGBoost(concat, BERT + img1000)	0.7	0.64	0.95	0.76	0.82	0.89	0.75	0.81
E	CatBoost(concat, BERT + img1000)	0.74	0.67	0.93	0.78	0.85	0.88	0.76	0.82
F	XGBoost(add)	0.72	0.61	0.95	0.74	0.83	0.81	0.88	0.84
F	CatBoost(add)	0.83	0.72	0.93	0.81	0.87	0.83	0.85	0.83
G	XGBoost(Joint Fusion)	0.85	0.85	0.97	0.9	0.89	0.84	0.93	0.88
G	CatBoost(Joint Fusion)	0.85	0.86	0.95	0.9	0.89	0.86	0.91	0.88
H	XGBoost(Averaged Fusion)	0.84	0.68	0.97	0.8	0.89	0.83	0.86	0.84
H	CatBoost(Averaged Fusion)	0.85	0.66	0.99	0.79	0.9	0.83	0.85	0.84

This experiment was conducted with balanced datasets and involved comparisons. First, when using a single BERT feature with only the title text (Code A), the highest accuracy achieved is 0.84. When using a single image feature (Code B) and extracting data with ResNet50, the accuracy of the Gossipcop Dataset information can reach 0.76. However, compared to the accuracy of text recognition alone, the accuracy is relatively lower, with a recall of 0.97. In early fusion, concatenating the data using the Concat method (Codes C, D, E), the highest accuracy can reach 0.85, with the best F1-score of 0.82. However, when using the Add method (Code F) in early fusion, the highest accuracy can reach 0.87, with the best F1-score of 0.83. Using the joint fusion method (Code G), the accuracy can reach 0.89, and the F1-score can reach 0.88. Finally, in late fusion, using averaged fusion (Code H) and the ensemble learning method in this experiment, the accuracy can reach 0.9, but the best F1-score is only 0.84. Based on the pure use of accuracy to explain the results, accuracy is improved from Codes A to H. It can be concluded that using multimodal approaches is an effective method to improve accuracy. Furthermore, when comparing XGBoost and CatBoost, CatBoost demonstrates higher accuracy.

In the aforementioned experiments, the classification training time for each model varied. The recognition time for a single text feature was the least, at 56 min and 34 s. The recognition time for a single image feature was 1 h and 9 min. For Early Fusion using the Add method, the training time was 1 h and 25 min. Joint Fusion required a minimum of 1 h and 19 min, and Late Fusion using the Averaged Fusion method took at least 1 h and 17 min. Using the Gossipcop and Fakeddit datasets, when employing a single recognition method, the F1-score for title text features was 79% and 84%, respectively, while for image features, it was 80% and 88%. However, in multimodal experiments, the F1-score reached 90% and 88%. That is, the multimodal experiments significantly improved the F1-score, achieving up to 90%, highlighting the effectiveness of multimodal fusion in enhancing fake news detection accuracy.

Conclusions

This study proposes an effective model for detecting fake news by combining different modalities using a multimodal fusion approach. This model can handle natural language tasks compared to the deep learning text recognition method BERT. Using a single feature is challenging in the context of fake news detection. The Gossipcop and Fakeddit datasets from the FakeNewsNet dataset were used to compare and validate the proposed model. When using a single recognition method, the accuracy for title text features was 63% and 83%, while for image features, it was 72% and 65%, respectively. The results show that text features’ accuracy is higher than image features. This could be because fake news content often utilizes diverse images, and the image features extracted by ResNet50 may not capture the relevant features related to fake news. This suggests that the accuracy of image features in a single recognition model is lower, which is a reasonable phenomenon.

Although the performance of single-feature recognition is unsatisfactory, it still plays a specific role in fake news detection. Subsequent experimental results demonstrate that a multimodal fusion approach can significantly improve accuracy. This study compares feature fusion models and finds that feature fusion recognition methods can effectively enhance model accuracy. In the early fusion tests, the accuracy and F1-score can be improved by up to 15% and 17%, respectively, compared to single features. In the late fusion tests, an additional 3% improvement in accuracy and 4% in F1-score can be achieved, resulting in a final accuracy and F1-score of 90%. It is also observed that integrating both fusion methods leads to higher results. After implementing joint fusion and late fusion together, the best F1-score for Gossipcop Dataset and Fakeddit Dataset were 90% and 88%, respectively. In conclusion, the experimental process and results confirm that multimodal fusion enhances the results in fake news detection, and the overlapping use of two fusion methods achieves the optimal F1-score.

The fusion of textual and image data forms the dataset for fake news identification in this paper. The proposed model and method ultimately contribute to improving judgment in practical applications of fake news identification. Recognizing fake news is a challenging process; however, this research observes that using different feature extraction and fusion methods can enhance accuracy and F1-score in the identification process. Therefore, in future practical applications for discerning fake news issues, the framework proposed in this paper can be utilized. It not only achieves results comparable to deep learning architectures more quickly but also provides flexibility in combining different features to achieve ideal confirmation and diagnostic analysis.

As the presentation of news information becomes increasingly diversified, future research should explore the integration of fake news in the form of images and audio. In training models that combine visual and auditory news content, the complexity of the image content, the accuracy of the correspondence between audio and visual content, and the labeling techniques will all impact the overall training results of the model. In other words, the quality and quantity of the video data used for training will significantly affect the effectiveness of fake news detection models. The challenge of developing multimodal fake news detection models that can handle text, images, audio or videos while balancing generalizability and computational efficiency remains a limitation of the current technology, yet it presents an area for further development.

Footnotes

Author contributions

Conceptualization: S.-Y.L. and Y.-H.C.; methodology: S.-Y.L., Y.-C.C., and Y.-H.C.; software: Y.-H.C. and S.-H.L.; validation: Y.-C.C. and S.-H.L.; formal analysis: Y.-H.C.; investigation: Y.-H.C.; resources: S.-Y.L.; data curation: S.-Y.L. and Y.-H.C.; writing—original draft preparation: S.-Y.L., Y.-C.C., and Y.-H.C.; writing—review and editing: S.-Y.L., Y.-C.C., and K.-M.C.; visualization: Y.-C.C. and K.-M.C.; supervision: S.-Y.L. and Y.-C.C.; project administration: S.-Y.L., Y.-C.C., and K.-M.C.; funding acquisition: S.-Y.L. All authors have read and agreed to the published version of the manuscript.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this paper.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research is supported by the National Science and Technology Council, Taiwan, under Grant numbers of 109-2410-H-197-002-MY3, 112-2410-H-A49-087-MY2, and 112-2221-E-216-005.

ORCID iD

Yen-Chiu Chen

References

Lazer

Baum

Benkler

, et al. The science of fake news. Science 2018; 359: 1094–1096.

Ali

Latif

Ghauri

, et al. Linguistic features and bi-LSTM for identification of fake news. Electronics 2023; 12: 2942.

Jang

Park

Seo

. Fake news analysis modeling using quote retweet. Electronics 2019; 8: 1377.

Song

Ning

Zhang

, et al. Knowledge augmented transformer for adversarial multidomain multiclassification multimodal fake news detection. Neurocomputing 2021; 462: 88–100.

Pierri

Ceri

. False news on social media: a data-driven survey. ACM Sigmod Record 2019; 48: 18–27.

Bakshy

Messing

Adamic

. Exposure to ideologically diverse news and opinion on Facebook. Science 2015; 348: 1130–1132.

Barthel

Mitchell

Holcomb

Many Americans believe fake news is sowing confusion. Pew Research Center. https://www.pewresearch.org/journalism/2016/12/15/fake-news-methodology/ (accessed 30 June 2024).

Pennycook

Rand

. The psychology of fake news. Trends Cogn Sci 2021; 25: 388–402.

Subramanian

Sathiskumar

Deepalakshmi

, et al. A survey on hate speech detection and sentiment analysis using machine learning and deep learning models. Alexandria Eng J 2023; 80: 110–121.

10.

Balakrishnan

Khan

Arabnia

. Improving cyberbullying detection using twitter users’ psychological features and machine learning. Comput Secur 2020; 90: 101710.

11.

Zhu

Togo

Ogawa

, et al. Multimodal natural language explanation generation for visual question answering based on multiple reference data. Electronics 2023; 12: 2183.

12.

Huang

Pareek

Seyyedi

, et al. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Digit Med 2020; 3: 136.

13.

Baltrušaitis

Ahuja

Morency

. Multimodal machine learning: a survey and taxonomy. IEEE Trans Pattern Anal Mach Intell 2018; 41: 423–443.

14.

Sahoo

Gupta

. Multiple features based approach for automatic fake news detection on social networks using deep learning. Appl Soft Comput 2021; 100: 106983.

15.

Nasir

Khan

Varlamis

. Fake news detection: a hybrid CNN-RNN based deep learning approach. Int J Inform Manage Data Insigh 2021; 1: 100007.

16.

Chauhan

Palivela

. Optimization and improvement of fake news detection using deep learning approaches for societal benefit. Int J Inform Manage Data Insigh 2021; 1: 100051.

17.

Wang

Han

. Detecting fake news on Chinese social media based on hybrid feature fusion method. Expert Syst Appl 2022; 208: 118111.

18.

Jin

Cao

Guo

, et al. Multimodal fusion with recurrent neural networks for rumor detection on microblogs. The 25th ACM international conference on Multimedia. 2017:795–816.

19.

Khattar

Goud

Gupta

, et al. Mvae: multimodal variational autoencoder for fake news detection. The world wide web conference; 2019:2915–2921.

20.

Wang

Jin

, et al. Eann: Event adversarial neural networks for multi-modal fake news detection. The 24th ACM SIGKDD international conference on knowledge discovery & data mining; 2018:849–857.

21.

Wang

Qian

, et al. Fake news detection via knowledge-driven multimodal graph convolutional networks. 2020 international conference on multimedia retrieval; 2020:540–547.

22.

Zhang

Fang

Qian

, et al. Multi-modal knowledge-aware event memory network for social media rumor detection. The 27th ACM International Conference on Multimedia. ; 2019:1942–1951.

23.

Mangal

Sharma

. Fake news detection with integration of embedded text cues and image features. 2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions)(ICRITO 2020). IEEE; 2020:68–72.

24.

Cao

Yang

, et al. Exploiting multi-domain visual information for fake news detection. 2019 IEEE International Conference on Data Mining (ICDM 2019). IEEE; 2019:518–527.

25.

Peng

, et al. A survey on text classification: from traditional to deep learning. ACM Trans Intell Syst Technol (TIST) 2022; 13: 1–41.

26.

Devlin

Chang

Lee

, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:181004805. Published online 2018.

27.

Hua

Cui

, et al. Multimodal fake news detection through data augmentation-based contrastive learning. Appl Soft Comput 2023; 136: 110125.

28.

Jing

Sun

, et al. Multimodal fake news detection via progressive fusion networks. Inf Process Manag 2023; 60: 103120.

29.

Nakamura

Levy

Wang

. r/fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection. arXiv preprint arXiv:191103854 . 2019. doi: 10.48550/arXiv.1911.03854.

30.

Church

. Word2Vec. Nat Lang Eng 2017; 23: 155–162.

31.

Kim

Seo

Cho

, et al. Multi-co-training for document classification using various document representations: tF–IDF, LDA, and Doc2Vec. Inf Sci 2019; 477: 15–29.

32.

Řeh\uuřek

Sojka

. Gensim—statistical semantics in python. Retrieved from genism org. Published online 2011.

33.

Akilan

Safaei

, et al. A late fusion approach for harnessing multi-CNN model high-level features. 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC). 2017:566–571.

34.

Sagi

Rokach

. Ensemble learning: a survey. Wiley Interdiscipl Re: Data Min Knowl Dis 2018; 8: e1249.

35.

Breiman

. Bagging predictors. Mach Learn 1996; 24: 123–140.

36.

Schapire

. A brief introduction to boosting. Ijcai. Citeseer 1999; 99: 1401–1406.

37.

Džeroski

Ženko

. Is combining classifiers with stacking better than selecting the best one? Mach Learn 2004; 54: 255–273.

38.

Chen

. Xgboost: extreme gradient boosting. R Package Version 0.4-2. 2015; 1: 1–4.

39.

Dorogush

Ershov

Gulin

. CatBoost: gradient boosting with categorical features support. arXiv preprint arXiv:181011363 . 2018.

40.

Friedman

. Stochastic gradient boosting. Comput Stat Data Anal 2002; 38: 367–378.

41.

Menard

. Applied Logistic Regression Analysis. United States: SAGE Publications, 2002.

42.

Noble

. What is a support vector machine? Nat Biotechnol 2006; 24: 1565–1567.

Text–image multimodal fusion model for enhanced fake news detection

Abstract

Keywords

Introduction

Multimodal fusion

2.1 Multimodal fusion strategies

Early fusion

Joint fusion or intermediate fusion

Late fusion

Fake news detection

Methodology

Research framework

Fake news-related datasets

Fakeddit dataset

Gossipcop dataset

Data pre-processing

Data cleaning

Feature extraction

Word2vec and Doc2vec

BERT feature vectors

Image feature vectors

Fusion methods

Early fusion

Joint fusion

Late fusion

Classification models

Evaluation metrics

Experiments

Experiment #1: data balancing comparison using a single-modal approach

Experiment #2: early fusion for fusion classification and prediction with each information

Comparison using Concat fusion method

Comparison using Add fusion method

Experiment #3: joint fusion for fusion classification and prediction with each information

Experiment #4: late fusion for fusion classification and prediction with each information

Averaged fusion method

Comparison using ensemble learning method

Conclusions

Footnotes

Author contributions

Declaration of conflicting interests

Funding

ORCID iD

References