Diving into Text Classification

Let's take a look at a practical use for the machine learning theory described in the preceding section. If you have spent any time reading news articles online, you'll probably have noticed that many sites take advantage of so-called "clickbait" – the practice of publishing headlines that deliberately withhold crucial information and imply that something exceptional happened to make readers click on an otherwise fairly boring article.

For example, "17 Insanely Awesome Starbucks You Need To See" is an example of a real headline that we can call "clickbait." It uses several tricks to try to make readers click through to the full article, even though the article itself is not very interesting: it uses an exact number (17), invoking curiosity to find out what all 17 are; it uses exaggeration ("insanely"), although there is nothing actually "insane" about the Starbucks in question; and it claims that you "need" to see them, although you can probably do just fine without.

On the other hand, publications with stronger commitments to ethical journalism would publish a headline such as "Ralph Nader enters US presidential race as independent" (another real headline). This headline, in direct contrast to the other one, is not clickbait. It is stating a simple fact; it is giving as much relevant information as possible upfront, and it is not trying to mislead the reader in any way.

To a computer, these headlines are difficult to tell apart. They both use standard English words, they are both similar in length, and there are not any specific rules that let you say with certainty "this is how to identify that a headline can be classified as clickbait."

This is a great example to use for machine learning – we, as humans, can tell which headlines are clickbait and which are not, but it is difficult to express this distinction as specific rules. Therefore, it makes sense to show a machine thousands of labeled examples – telling it 'these are clickbait, and these are not' – and see whether the machine can infer the rules on its own.

There are some important fundamentals and terminologies you need to be familiar with to fully follow along. For convenience, we'll summarize these here, but if you are not familiar with vectorization, training data, labels, classification, and evaluation, please note that these are complicated topics and that you may need to spend some time reading more about these in third-party resources.

Let's start by taking a look at TF-IDF vectorization.

Looking at TF-IDF Vectorization

Humans are used to reading and writing text, but computers prefer working with numerical data. For machines to be able to meaningfully process natural language text, we need to first convert this text into a meaningful binary format. There are many different ways of doing this, but a simple one is TF-IDF or Term Frequency, Inverse Document Frequency.

The fundamental idea of TF-IDF is that the more often a word appears in a text, the more important that word is. So, in an article about "electric cars," it is likely that the words "electric" and "car" will appear often, indicating that these words should be given more attention in any analysis that we do. Unfortunately, there are many common words, such as "the," and even though these words appear frequently, they are not important. To compensate for this, we don't only look at term frequency, but also inverse document frequency. A word that often appears in a single article but does not appear in many different articles is more important than a term that appears often in all articles. The exact weighting equation is not too important, and our Python library, sklearn, will take care of it for us, but out of interest, instead of using simple frequency counts as in the previous example, we will use the following equation:

word_freq(w, d) x log (N/doc_freq(w))

Here:

• word_freq(w,d) means the count of word w in document d.
• N means the total number of documents in our collection.
• doc_freq(w) means the number of documents that the word w appears in.

The point of vectorization is to transform the text into vectors, or an array of numbers that can be processed by a machine.

Term frequency, the first part of TF-IDF, relates to how often specific words are used. We'll start by looking at a manual example of vectorization using only term frequency, and then see how we can use a standard Python library for the full version of TF-IDF.

Counter-intuitively, we can ignore the order that the words in a given text are presented in and look only at their frequency. For example, we have two very short sentences in two documents, shown as follows:

1. "a cat and a dog"
2. "a cat and a fish"

We could first create a mapping table, assigning a single number to each word across all of our documents. This would look as follows:

'a' = 0

'cat' = 1

'and' = 2

'dog' = 3 (We skip the second "a" in the first document, as we already assigned it to a number.)

'fish'= 4 (We skip all the words before fish in the second document as they are all already assigned a number.)

The numbers map to what can be used as indices in an array. We will create a single array, containing only numbers to represent our document. The zeroth element of the array will indicate how many times the word "a" appears, as this was assigned the index "0." Once we have this mapping, we can represent both documents entirely numerically with arrays, as shown in the following figure:

Figure 1.5: Vectorized example – values and indices

The 2 at the zeroth index of the first array indicates that the word a appears twice in our first document, and the next three ones indicate that the words cat, and, and dog appear once each. Fish doesn't appear at all in the first document, so the 4th index of the array is a 0. The second array looks very similar, but there is a 0 at the 3rd index to indicate that the dog doesn't appear and a 1 at the fourth index to indicate that fish appears once.

Note that the ordering is lost. The documents a cat dog and and a cat and a dog look the same now, but surprisingly this is hardly ever a problem in text processing.

We have seen how to convert text into a vectorized form for computers to read, which is an important first step. Before we get to use this in a practical example, we will define some basic terminology in machine learning classification tasks.