Once you have trained a binary classification model, it produces granular probability scores for each of the entities.
In this article, we’ll go over the meaning and logic behind these scores for binary classification models that try to predict if an entity will belong to a certain class in the future.
For example, a churn model tries to predict if a specific customer will get the “churned” class. Each customer will get a probability score of them churning.
Note that the model always predicts what will happen. For example, a customer with a 3% churn probability has a 3% probability of churning.
Probability scores in binary classification models
In binary classification, the model is trained to predict the probability of an entity being classified into one of two classes as defined in the Target. The probability score produced by the model for an entity indicates the likelihood of it being positive (“1” class).
For example, if an entity is assigned an 80% probability score in a churn model, it means there is an 80% chance that the entity will churn.
Probability range
Generally, the probability score can be between 0 and 100%, but in some cases, the highest probability score might be lower than 100%, and the lowest probability score will be higher than 0%.
In cases where the target is unbalanced, the highest probability score might be lower than 100%, even if the model is accurate. This is because the model is biased towards the majority class, and its goal is to maximize overall accuracy. As a result, it might predict a low probability for the minority class, even if it is correct.
Therefore, when evaluating a model, it’s important to analyze the different evaluation metrics (precision and recall), which take into account the data's class balance.
How do probability scores classify between “0” and “1”?
Since binary classification models produce probability scores for the entities, there is no clear classification decision.
That’s where the threshold comes in - to make a prediction, we need to choose a threshold value to separate the predictions of the positive and negative classes. If the probability score is above the threshold, the entity is classified as positive. Otherwise, it is classified as negative.
Choosing the threshold depends on the business objective and includes a trade-off between precision and recall. In Pecan’s dashboards, the threshold is configurable and can be set according to your business understanding of the problem and the planned application of the model’s results.