Performance Analysis and Comparative Evaluation of Machine Learning Models for Fake News Detection | by Bilal Ali Shah | Jun, 2023

Bilal Ali Shah

Ubaid Ur Rehman

Introduction:

The idea of pretend information, as outlined by the Reuters Institute for the Examine of Journalism, entails “false data knowingly circulated with particular strategic intent–both political or business” [1]. In our digital period, this downside has intensified, because it has grow to be more and more straightforward to propagate falsehoods and evade accountability, typically leveraging the defend of anonymity. In a notable incident, Pakistan’s Protection Minister Khawaja Asif issued a nuclear menace in direction of Israel, mistakenly swayed by a fabricated information story about Israel’s alleged provocation [2]. If even high-ranking officers chargeable for nuclear arsenals wrestle to differentiate reality from falsehood, one can solely think about the challenges confronted by unusual web people. Due to this fact, this speedy unfold of pretend information by way of social media platforms and on-line information retailers poses vital challenges in sustaining the authenticity and accuracy of reports sources and highlights the need of growing efficient pretend information detectors.

Analysis Aims:

The primary goal of this analysis challenge is to guage the efficiency of varied machine studying fashions in detecting pretend information. Particularly, we goal to:

• Evaluate the accuracy, precision, recall, and F1-score of various machine studying fashions.
• Analyze the impression of characteristic extraction strategies on mannequin efficiency.
• Examine the effectiveness of various classifiers in distinguishing between pretend and actual information articles.

Datasets:

We utilized two curated datasets from Kaggle.

1. Fake and Real News Dataset
2. Fake News Classification

These are curated datasets of reports articles collected from respected information sources and faux information web sites. The datasets every encompass a title column and a textual content column. Pretend Information Classification additionally has the binary column “label” to point whether or not an article is pretend information. The Pretend and Actual Information Dataset, alternatively, consists of two separate datasets comprised solely of pretend and actual information respectively.

Preprocessing:

To organize the dataset for evaluation, we carried out a number of preprocessing steps. Step one was to merge the datasets. Firstly, we created 0,1 label columns for the separate Pretend and Actual datasets, and concatenated them collectively.

df2_fake['label'] = 1
df2_true['label'] = 0
df_merged = pd.concat((df2_fake, df2_true))

We then merged the resultant dataset with the Pretend Information Classification dataset. We employed easy information cleansing equivalent to dropping ineffective rows, eradicating duplicates, and eradicating rows with null values.

dfconcat = pd.concat((df, df_merged[['title','text','label']]))
dfconcat.drop('Unnamed: 0', inplace=True, axis=1)
dfconcat.dropna(inplace=True)
dfconcat.drop_duplicates(subset='textual content', inplace=True)
dfconcat.reset_index(drop=True, inplace=True)

The ultimate step was changing all textual content to lowercase, eradicating “cease phrases” (phrases that don’t carry any that means and are thus ineffective), particular characters, digits, and hyperlinks. For this, we made the next “textCleaning” operate:

def textCleaning(column, sample):
column = column.str.decrease()
column = column.str.change(sample, ' ', regex=True)
column = column.str.change(r'http [S]*', ' ' ,regex=True)  # eradicating hyperlinks
column = column.str.change(r'd+', ' ', regex=True)  # eradicating digits
column = column.str.change(r'n', ' ', regex=True)  # eradicating new line symbols
column = column.str.change(r'[^ ws]', ' ', regex=True)  # eradicating punctuation and symbols
return column

“sample” is a regex sample we made by combining all of the “cease phrases” into one regex sample that may very well be used to take away the cease phrases:

cease = pd.read_table('stop_words.txt', header=None)[0]
sample = r'b('+ '|'.be part of(cease)+r')b'  # becoming a member of the checklist of cease phrases to make a regex sample

Subsequent, we concatenated the title textual content to the article textual content column, which was used for characteristic extraction.

dfconcat['text'] = dfconcat['title'] + ' ' + dfconcat['text']
dfconcat.drop('title', inplace =True, axis =1)
dfconcat['text'] = textCleaning(dfconcat['text'], sample)

Exploratory Knowledge Evaluation:

Earlier than performing characteristic extraction and coaching our mannequin, it’s important to first have a better have a look at the intricacies inside our information. That’s exactly what EDA entails! The very very first thing one ought to verify when tackling a Classification downside is class imbalance. We do that by utlizing the Seaborn library:

We are able to observe that our classifier, “label”, is comparatively balanced, with round 43% “Pretend Information” cases towards virtually 57% “Actual Information” cases. This was an essential step since any vital class imbalance left untreated would have prompted bother with our Classification fashions going ahead.

Shifting on, we search for attention-grabbing patterns within the information. Beginning off with textual content size evaluation. Firstly, we create a brand new column in our dataset representing every article’s respective phrase depend (excluding phrases that now we have already eliminated):

dfconcat['length'] = dfconcat['text'].apply(lambda x: len (x.cut up ()))

We as soon as once more make use of Seaborn to visualise the distribution of article size by label:

These distributions fairly clearly present that Pretend Information articles are usually shorter than Actual Information.

Subsequent, we use the “wordcloud” library to visualise probably the most prevalent phrases all through the dataset on the whole and in every class. Beginning with the overall phrase cloud, we concatenate all its “textual content” cases into one lengthy string with a single area between all adjoining rows. This string is then handed into the WordCloud() to generate the overall phrase cloud:

For the separate phrase cloud, we separate the dataset into “pretend” and “actual” datasets. It provides the next phrase clouds:

That is an attention-grabbing visualization since we are able to examine the labelled phrase clouds with the total one, and see which phrases are disproportionately prevalent in a given class. For instance, whereas “donald trump” is extremely prevalent in our dataset, the phrase appears virtually equally standard in each classes (barely leaning towards Pretend Information). Then again, “hillary clinton” is a comparatively small characteristic within the Full and Actual Information phrase clouds, however is sort of as giant(frequent) within the Pretend Information phrase cloud. This means that the phrase “hillary clinson” is disproportionately prevalent in pretend information articles.

One other such characteristic is “u” (presumably the casual type of “you”). This phrase is virtually invisible within the Full and Actual Information phrase clouds, but it is without doubt one of the most prevalent phrases within the Pretend Information phrase cloud. Utilizing this truth mixed with our earlier commentary that pretend information articles usually have shorter phrase counts, we might assume that pretend information tends to be shared extra by way of casual means than formal articles.

Function Extraction:

To symbolize the textual content material of the information articles, we employed two characteristic extraction strategies:

1. Bag-of-Phrases (BoW)
2. Time period Frequency-Inverse Doc Frequency (TF-IDF)

The Bag-of-Phrases approach was carried out from scratch. First, we made a “vocabulary checklist” that included all of the distinctive phrases in our articles. Subsequent, we counted the variety of occasions that phrase occurred in every article, thus giving us an enormous 62200 x 234266 matrix. We needed to cut up this implementation in two elements because it was fairly reminiscence intensive, and retailer the info in a scipy compressed sparse row (csr) matrix as numpy arrays had been too storage intensive. Right here’s the code:

# Concatenate all article textual content right into a single string
all_text = ' '.be part of(dfconcat['text'])
# tokenize the textual content into particular person phrases
tokens = all_text.cut up()
# create a set to retailer distinctive tokens
vocab_set = set(tokens)
# convert the set to a listing
vocab_list = checklist(vocab_set)
import scipy.sparse as sp
# dividing into two as a result of the reminiscence the entire thing takes is large
n = int(dfconcat['text'].form[0]/2)
matrix1 = np.empty((n,len(vocab_list)),dtype=np.uint16)
# Populate Bag of Phrases
# for every phrase append the depend of the matrix at related place
i = 0
for article in dfconcat['text'][:n]:
j = 0
# print (i)
for phrase in vocab_list:
depend = article.depend(phrase)
matrix1[i,j] = depend
j += 1
i += 1
matrix1 = sp.csr_matrix(matrix1)

Equally, we populated the second half of the bag-of-words utilizing the remaining article texts, after which merged them collectively in a single sparse row matrix.

The options extracted utilizing this method had been then used within the varied machine studying fashions we examined. Time period Frequency-Inverse Doc Frequency (TF-IDF) options had been extracting utilizing sci-kit study’s TfidfVectorizer().

X_tfidf = TfidfVectorizer().fit_transform(dfconcat['text'])

We ran the next machine studying fashions for pretend information detection:

• Naïve Bayes
• Logistic Regression
• Help Vector Machine (SVM)
• Random Forest
• XGBoost
• Convolutional Neural Networks (CNN)

All of those algorithms had been fine-tuned utilizing completely different parameters till we obtained to the very best end result utilizing trial-and-error.

Naïve Bayes

Naïve Bayes is a probabilistic mannequin that makes use of the Bayes Theorem to make predictions for an end result variable Y given a set of options X:

the place: P(X|Y ) is the probability,

P(Y ) is the prior chance,

and P(X) is a normalizing fixed.

The Naïve Bayes mannequin makes the “naïve” assumption that every one the enter options are impartial of one another, given the output class. This permits us to simplify the probability time period as follows:

As a substitute of taking the product, one can even take the sum of the logarithm of the likelihoods.

Utilizing these likelihoods, we discover the posterior chances and select the category because the prediction that maximizes the posterior chance.

class NaiveBayes:
def __init__(self):
self.courses = [0,1]
self.class_priors = None
self.feature_likelihoods = None
def match(self, X_train, y_train):  
# discover class priors
fake_prior = np.imply(np.the place(y_train==1,1,0))
real_prior = np.imply(np.the place(y_train==0,1,0))
self.class_priors = [real_prior,fake_prior]
# discover likelihoods
self.feature_likelihoods = np.zeros((2, X_train.form[1]))
for i, label in enumerate(self.courses):
X_i = X_train[y_train == label]
total_count = X_i.sum()
# making use of add-1 smoothing (laplace smoothing)
self.feature_likelihoods[i] = (X_i.sum(axis=0) + 1) / (total_count + 1 * X_train.form[1])
# print(self.feature_likelihoods[i])
def predict(self, X_test):
y_pred = np.zeros(X_test.form[0])
for i in vary(X_test.form[0]):
posteriors = []
for j in vary(2):
likelihoods = self.feature_likelihoods[j]
# want the nonzero indices since we're utilizing sparse matrices
nonzero_indices = X_test[i].indices
likelihoods_nonzero = likelihoods[nonzero_indices]
log_likelihoods = np.log(likelihoods_nonzero)
# print(log_likelihoods)
log_priors = np.log(self.class_priors[j])
posterior = np.sum(log_likelihoods) + log_priors
posteriors.append(posterior)
# now, to the prediction:
if posteriors[0] > posteriors[1]:
y_pred[i] = 0
else:
y_pred[i] = 1
return y_pred
nb = NaiveBayes()
X_train, X_test, y_train, y_test = train_test_split(sparse_X, np.array(dfconcat['label']),test_size=0.2, random_state=42)
nb.match(X_train,y_train)
y_pred = nb.predict(X_test)
Consider(y_test,y_pred)  # 'Consider' is customized operate

Furthermore, we additionally used sci-kit study’s implementation of Multinomial Naïve Bayes and in contrast the performances. That gave us accuracies of 90% and 89%, respectively.

nb = MultinomialNB()
X = CountVectorizer().fit_transform(dfconcat['text'])
X_train, X_test, y_train, y_test = train_test_split(X, np.array(dfconcat['label']),test_size=0.2, random_state=42)
nb.match(X_train, y_train)
y_pred = nb.predict(X_test)
Consider(y_test, y_pred)

Logistic Regression

Logistic Regression makes use of the utmost probability methodology to seek out the chance of an enter 𝓍ᵢ belonging to class 1.

the place β₀ is the intercept/bias time period and β is the weights vector for 𝓍ᵢ. It classifies 𝓍ᵢ as 1 if P(𝓍ᵢ) ≥ 0.5 and 0 in any other case. Utilizing Logistic Regression, we had been in a position to get an accuracy of 95%.

logreg = LogisticRegression(C=100,max_iter=1500)
X = CountVectorizer().fit_transform(dfconcat['text'])
X_train, X_test, y_train, y_test = train_test_split(X, np.array(dfconcat['label']),test_size=0.2)
logreg.match(X_train,y_train)
y_pred = logreg.predict(X_test)
Consider(y_test,y_pred)

Help Vector Machine

Help Vector Machine (SVM) finds the utmost margin hyperplane in Rd that greatest separates the 2 (or extra for multi-class classification issues) courses. It achieves this by reworking the characteristic vectors into larger dimensions utilizing a kernel. We examined it on a number of kernels, and the radial foundation operate (rbf) kernel gave us the very best accuracy. RBF transforms the options into an infinite dimensional illustration of these options. Utilizing SVM with the rbf kernel, we had been in a position to get an accuracy of 96%.

X = CountVectorizer().fit_transform(dfconcat['text'])
X_train, X_test, y_train, y_test = train_test_split(X, np.array(dfconcat['label']), test_size=0.2, random_state=42)
svm_model = SVC(kernel='rbf', C=10)
svm_model.match(X_train, y_train)
y_pred = svm_model.predict(X_test)
Consider(y_test, y_pred)

Random Forest

Random Forest is a tree-based ensemble methodology that trains a number of fashions utilizing random splits of the coaching information on impartial resolution bushes, utilizing random subsets of the options at every node, and aggregates their outcomes to supply a remaining prediction. It combines the ability of a number of resolution bushes to make correct classifications. Utilizing Random Forest, we managed to attain an accuracy of 92%.

rf = RandomForestClassifier()
X = CountVectorizer().fit_transform(dfconcat['text'])
X_train, X_test, y_train, y_test = train_test_split(X,np.array(dfconcat['label']),test_size=0.2,random_state=42)
rf.match(X_train, y_train)
y_pred = rf.predict(X_test)
Consider(y_test,y_pred)

XGBoost

XGBoost (Excessive Gradient Boosting) makes use of an ensemble of weak learners, usually resolution bushes, in a boosting framework. Boosting is a sequential course of the place every weak learner corrects the errors of earlier learners. The algorithm minimizes a loss operate by iteratively becoming weak fashions to the residuals or gradients of the loss operate. It combines resolution bushes with gradient descent and makes predictions utilizing weighted voting of every learner. Utilizing XGBoost, we obtained an accuracy of 94%.

X = CountVectorizer().fit_transform(dfconcat['text'])
X_train, X_test, y_train, y_test = train_test_split(X,np.array(dfconcat['label']),test_size=0.2, random_state=42)
dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)
params = {
'goal': 'binary:logistic',
'eval_metric': 'error'
}
mannequin = xgb.practice(params, dtrain)
y_pred = mannequin.predict(dtest)
y_pred = np.the place(y_pred>=0.5,1,0)
Consider(y_test, y_pred)

Convolutional Neural Community

Convolutional Neural Community (CNN) is a deep studying algorithm primarily used for analyzing visible information, equivalent to movies and pictures. Nevertheless, it can be used for textual content information utilizing a one-dimensional CNN (since in textual content now we have one-dimensional information within the type of sequences, whereas conventional CNNs deal with 2-D information of pictures with peak and width). Utilizing CNN, we achieved the very best accuracy at 97%.

from tensorflow.keras.preprocessing.textual content import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.fashions import Sequential
from tensorflow.keras.layers import Embedding, Conv1D, GlobalMaxPooling1D, Dense, Dropout
tokenizer = Tokenizer()
tokenizer.fit_on_texts(dfconcat['text'])
sequences = tokenizer.texts_to_sequences(dfconcat['text'])
sequences = pad_sequences(sequences, maxlen=1000)
X_train, X_test, y_train, y_test = train_test_split(sequences, np.array(dfconcat['label']), test_size=0.2, random_state=42)
embedding_dim = 100
vocab_size = len(tokenizer.word_index) + 1
cnn = Sequential()
cnn.add(Embedding(vocab_size, embedding_dim, input_length=max_sequence_length))
cnn.add(Conv1D(128, 5, activation='relu'))
cnn.add(GlobalMaxPooling1D())
cnn.add(Dense(64, activation='relu'))
cnn.add(Dropout(0.2))
cnn.add(Dense(1, activation='sigmoid'))
cnn.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

We used the next metrics to guage our classification efficiency:

• Accuracy
• Precision
• Recall
• F1-score

We current the efficiency metrics of the completely different machine studying fashions in Desk 1.

Primarily based on the outcomes, the CNN mannequin achieved the very best accuracy, precision, recall, and F1-score, indicating its superior efficiency in detecting pretend information. SVM additionally demonstrated aggressive outcomes, whereas Naïve Bayes was the worst out of those all (however nonetheless first rate at 0.89 F1-score).

TF-IDF Outcomes

All the outcomes reported thus far had been outcomes we obtained utilizing Bag-of-Phrases illustration of the textual content information (apart from Convolutional Neural Community’s outcomes, since that mannequin sequences as an alternative). We additionally ran machine studying fashions on the TF-IDF options. Right here’s the code for the characteristic extraction:

X = TfidfVectorizer().fit_transform(dfconcat['text'])
X_train, X_test, y_train, y_test = train_test_split ( X, dfconcat['label'], test_size =0.2)

The outcomes are reported in Desk 2.

Apparently, Logistic Regression noticed a median of two% enchancment in all of the metrics, making it match SVM efficiency, which barely improved. Random Forest additionally noticed slight enchancment, whereas Naïve Bayes & XGBoost carried out worse. Naïve Bayes noticed a 4% enchancment in precision however recall fell by 16%! This demonstrates how completely different characteristic extraction strategies will be extra suited to completely different machine studying fashions, and to get to the optimum efficiency stage we have to check out varied mixtures of options and fashions utilizing trial-and-error.

On this analysis challenge of pretend information classification, we investigated the effectiveness of varied machine studying fashions. By means of experimentation on two curated datasets, we in contrast the efficiency of various fashions utilizing two completely different characteristic extraction strategies and evaluated their strengths and limitations. The CNN mannequin outperformed different fashions, adopted by the SVM mannequin and the Logistic Regression mannequin. These findings spotlight the potential of synthetic intelligence, notably DL strategies, in pretend information detection. A limitation of this examine is that we weren’t ready to make use of n-gram options as a result of time constraints, and deep studying fashions just like the Multi-Layer Perceptron (MLP) & Lengthy Brief-Time period Reminiscence (LSTM) classifier ought to have additionally been part of the used fashions as they’re additionally standard selections for textual content classification duties. Regardless of that, our outcomes contribute to the event of strong strategies for combating the proliferation of pretend information and sustaining the authenticity of reports sources.