Classification Metrics — Confusion Matrix Explained
I am starting a series of blog posts aiming to cover the basics of various data science and machine learning concepts. I’m mainly doing this to understand better these concepts myself. I hope that during this process, I can help others understand them too. Okay, let’s do it!
In the field of machine learning, a confusion matrix (also known as an error matrix) is a table that allows us to visualize the performance of an algorithm. It is used for classification tasks only.
The name comes from the fact that it makes it easy to see if an algorithm is confusing two or more classes (i.e. doesn’t make correct predictions)
Binary (2 classes) Classifier Example
Let’s start with the simplest example. Imagine that we trained a machine learning model to detect if there is a dog in a photo or not. This is a binary classification task meaning that there are only two classes (“dog” or “not a dog” in the photo). The labels used for the training process are 1 if there is a dog in the photo and 0 otherwise. In the binary classification task, we often call these classes Positive and Negative. When we pass a new photo to our model it predicts whether there is a dog in our photo.
Now, imagine that we want to classify 10 new photos. We could use our classifier to do the categorization of the photos. Each photo receives a prediction containing the label (0 or 1) which represents the two classes (dog or not a dog). Therefore, for each photo, we’ll have the predicted class and the actual class. Given this information, we could generate a confusion matrix for these 10 photos. Later, I’ll give you a link to an awesome example for plotting confusion matrices. For now, let’s say that the following confusion matrix is returned after we have passed the predicted classes and actual classes:
Each column of the matrix represents the instances in the actual class, while each row represents the instances of the predicted class (or vice versa). We trained a model to detect between two classes, so we end up having only 4 cells that represent different information:
- Тhe cell on row one, column one, contains the number of True Negatives (in our case, this cell contains the number of correctly predicted photos that don’t contain a dog). The model truly predicts that there isn’t a dog in these 3 photos.
- Тhe cell on row one, column two, contains the number of False Positives (in our case, this cell contains the number of predicted as dogs photos, but actually the photos don’t contain a dog). The model falsely predicts that there is a dog in these 2 photos.
- Тhe cell on row two, column one, contains the number of False Negatives (in our case, this cell contains the number of predicted as not dogs, but actually the photos contain a dog). The model falsely predicts that there isn’t a dog in this 1 photo.
- Тhe cell on row two, column two, contains the number of True Positives (in our case, this cell contains the number of correctly predicted photos that contain a dog). The model truly predicts that there is a dog in these 4 photos.
We could easily see that our model predict 7 of 10 photos correctly and misclassified (confused) 3 photos. From the observations above we could decide if our classifier is good enough or not and proceed with additional analysis on the misclassified photos.
Multiclass Classifier Example
This was a very simple example. In some cases, we need to train a model that can predict between more than two classes. Let’s imagine that we want to train a model that predicts if a photo contains a dog, cat, or rabbit. In this case, the number of classes will be 3. Now imagine that we’re passing 27 photos to be classified (predicted) and we get the following confusion matrix:
Again, each column of the matrix represents the instances in the actual class, while each row represents the instances of the predicted class. However, this time we have 9 cells because we have 3 classes.
Please note that by True Cats I mean photos that were classified as cats and are actual cats. Also, by False Cats, I mean photos that we classified as cats but actually are not cats. The True/False word tells us if the predictions are correct and the Cats/Dogs/Rabbits words tell us the actual class.
Some insights that could be extracted from this confusion matrix are:
- The model predicted (classified) correctly only 15 of all 27 photos. There are 3 correctly predicted photos that contain a cat, 4 correctly predicted photos that contain a dog, and 8 correctly predicted photos that contain a rabbit.
- From all misclassified photos, we have: 2 photos predicted as a dog but actually containing a cat, 2 photos predicted as a cat but actually containing a dog, 1 photo predicted as a rabbit but actually containing a dog, 3 photos predicted as a cat but actually containing a rabbit, and 5 photos predicted as a dog but actually containing a rabbit.
- Also, we could see that we don’t have any wrongly predicted as rabbit photos that actually contain a cat.
Now you know how to read a confusion matrix and what it represents. Here is a great example of how you can easily generate beautiful confusion matrices using the sklearn package.
Summary
Confusion Matrices could be used to analyze the performance of a classifier and to give us insights into which direction we should work to improve our classifier. However, we just looked at two confusion matrices generated from classifiers trained with a low number of classes (2 and 3). Imagine that we have to train a classifier with 100 classes. In this case, maybe we’ll need some metrics that aggregates the information provided by the confusion matrix. Stay tuned for the next articles in which I’ll show you how we could use accuracy, precision, and recall metrics which definitions are just formulas that use the values from the matrix.
Alright, that’s all folks! I hope that you enjoyed the blog post. Please let me know if you have any feedback for me. Here is an image summarizing the confusion matrix definition and the examples.
If you want to be notified when I post a new blog post you can subscribe to my newsletter. Here is my LinkedIn profile in case you want to connect with me. I’ll be happy to be connected with you.
