Spam or Ham
Supervised means in essence to be watched over, like a parent watching their child play so that they avoid getting hurt. Supervised Machine Learning means training a model on already labeled data points so that the model is able to generalize and extrapolate predictions for new data points.
Spam vs Ham
Let’s start with the classic issue of identifying spam vs not spam emails, a binary classification problem. Lets examine a few techniques on how we can do this effectively.
Naive Bayes Classifier
In Probability Theory the probability function denoted as reads what is the probability of a random variable given that we have observed . In this case where or for Spam or Ham. Here is the definition of Bayes Theorem:
Notice we have switched the language of the problem now from what is the probability of the following email is spam given these features to what is the probability that you get these features considering the email is spam?. That is the power of Baye’s theorem.
But why is it called naive? Well since it makes our lives easier of course. We represent as a vector that captures the frequency of each unique word present in the email. When we assume that every single frequency for each unique word is independent (which is obviously not the case) we get that
So we can rewrite our initial equation under this assumption to get
Before we confuse ourselves with all of this mathematical notation, its always good to translate each of these probability calculates into plain english, starting off with
-
- Why? well what is another way of framing if this email is spam or not? It means given an email, is it more likely to be spam or not spam? So literally we are calculating
We can estimate the probability of each of these values using a training corpus. From our training corpus we estimate the values as follows
Data Pipeline
Let’s build a miniature data processing pipeline for our naive bayes classification model. We’ll take a corpus which for me is just a csv that looks something like this:
v1,v2,,,
ham,"Go until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...",,,
ham,Ok lar... Joking wif u oni...,,,
spam,Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's,,,There is around labeled spam vs ham emails. Lets divide the corpus into two datasets train and test. Then we can compute the frequencies for each word per label (Spam or Ham)
spam_count ham_count spam_total ham_total spam_emails ham_emails
word
to 552 1229 14300 54567 598 3859
a 304 822 14300 54567 598 3859
call 271 179 14300 54567 598 3859
your 214 337 14300 54567 598 3859
you 205 1347 14300 54567 598 3859
the 165 893 14300 54567 598 3859
for 163 394 14300 54567 598 3859
or 148 183 14300 54567 598 3859
free 146 40 14300 54567 598 3859
is 131 578 14300 54567 598 3859With this table we can effectively answer and for example
With this table we can calculate the probability of a spam or ham. Take for example this email plucked straight from my spam box
[SUBJECT LINE] You Are Our 3rd Winner !! Confirm & Claim 🎁 Lowe's Gorilla Carts
[BODY]
Congratulations!
You are a Winner of a
Gorilla Carts
You have been selected as one of the lucky few for a
unique opportunity to receive a
Gorilla Carts
Get it NowWhen we tokenize the body of this email, we represent as a vector for each of the counts in the email. The word frequences look like this
spam_count ham_count in_email p_given_spam p_given_ham
word
congratulations! 2 0 1 0.000140 0.000000
you 205 1347 2 0.014336 0.024685
are 61 313 1 0.004266 0.005736
a 304 822 4 0.021259 0.015064
winner 10 0 1 0.000699 0.000000
of 74 411 2 0.005175 0.007532
gorilla 0 0 2 0.000000 0.000000
carts 0 0 2 0.000000 0.000000
have 110 349 1 0.007692 0.006396
been 32 67 1 0.002238 0.001228
selected 22 2 1 0.001538 0.000037
as 27 119 1 0.001888 0.002181
one 6 107 1 0.000420 0.001961
the 165 893 1 0.011538 0.016365
lucky 8 4 1 0.000559 0.000073
few 0 24 1 0.000000 0.000440
for 163 394 1 0.011399 0.007220
unique 1 1 1 0.000070 0.000018
opportunity 0 3 1 0.000000 0.000055
to 552 1229 1 0.038601 0.022523
receive 26 3 1 0.001818 0.000055
get 62 238 1 0.004336 0.004362
it 20 342 1 0.001399 0.006268
now 75 119 1 0.005245 0.002181So the feature variables will reflect the in_email column. We will also be using a multinomial classifier which means that we will be taking its occurrence in the email into account.
Remember . I won’t do this for all unique words in the spam email but for the first one the word “congratulations!” would be for each
Of course there is an issue, in a product chain if any of the values are zero, the entire product is zero. Which we see for words like congratulations! and opportunity. So we will use a technique called Laplace Smoothing which says that for we have
Applying Laplace Smoothing with we get
spam_count ham_count in_email p_given_spam p_given_ham
word
congratulations! 2 0 1 0.000115 0.000015
you 205 1347 2 0.007892 0.020310
are 61 313 1 0.002375 0.004731
a 304 822 4 0.011684 0.012400
winner 10 0 1 0.000421 0.000015
of 74 411 2 0.002873 0.006208
gorilla 0 0 2 0.000038 0.000015
carts 0 0 2 0.000038 0.000015
have 110 349 1 0.004252 0.005273
been 32 67 1 0.001264 0.001025
selected 22 2 1 0.000881 0.000045
as 27 119 1 0.001073 0.001808
one 6 107 1 0.000268 0.001627
the 165 893 1 0.006359 0.013470
lucky 8 4 1 0.000345 0.000075
few 0 24 1 0.000038 0.000377
for 163 394 1 0.006283 0.005951
unique 1 1 1 0.000077 0.000030
opportunity 0 3 1 0.000038 0.000060
to 552 1229 1 0.021185 0.018532
receive 26 3 1 0.001034 0.000060
get 62 238 1 0.002414 0.003601
it 20 342 1 0.000805 0.005168
now 75 119 1 0.002912 0.001808This simple technique, although naive works incredibly well in practice!