In a binary classification model, results are classified into two mutually exclusive populations that can be labeled (e.g. “converted” vs. “did not convert”). In Pecan, there will be a prediction of either 1 or 0 in the “Label” column for each entity. There can be no other classifications like “null”, “2” or “$75”.

(That being said: behind the scenes, each binary model does indeed produce a score between 0 and 1. However, your model’s threshold setting determines the final prediction – whether each entity is assigned a prediction of 0 or 1.)

Below is a list of metrics that are useful for evaluating the results of a binary model in Pecan:

Base Rate

Precision Rate

Detection Rate

Venn diagram

Accuracy Rate

AUC (Area Under the Curve)

AUPRC (Area Under the Precision-Recall Curve)

LogLoss (Logarithmic Loss)

The base rate describes the percentage of the population that currently exhibits the target behavior (or meets certain criteria). In other words, it indicates the probability that members of the population will carry out the behavior we want to predict in the absence of other information.

In Pecan’s dashboard, the baseline your model is compared with uses the base rate of the test set.

The precision rate indicates how precise a model was in predicting instances of the target behavior. In other words, it calculates the percentage of positive predictions that turned out to be correct:

This formula answers the question: “When you predict a positive outcome, how often are you correct?

For example, if a model predicts that 1,000 people will become High-Value Customers, but only 800 of them actually do so, then the model’s precision rate is 80%.

This metric is important in cases where taking action has a significant cost or risk, and we want to make sure we are acting only on the right population. Picking a conservative threshold will improve precision, typically at the expense of detection (see below).

The detection rate indicates a model’s ability to detect instances of target behavior. In other words, it calculates the percentage of all target behavior that was correctly predicted:

Detection rate = Detected correctly / (Detected correctly + Ignored incorrectly)

This formula answers the question: “How much of a particular outcome did you manage to predict?

For example, if 1,000 people became High-Value Customers, but the model predicted only 400 of them to do so, then the model’s detection rate is 40%.

This metric is important in cases where not acting has a significant cost or risk, and you want to miss as few positive instances as possible. Picking a liberal threshold will improve this metric, typically at the expense of precision (see above).

The trade-off between precision and detection

As explained above and as illustrated in the Venn diagram below, adjusting a model’s threshold affects the balance between precision and detection.

A higher precision rate means that your model is increasingly correct when predicting instances of the target behavior. However, this comes at the expense of the detection rate, since fewer instances of the target behavior will be detected overall.

Conversely, a higher detection rate means that your model will detect more instances of the target behavior overall. But this comes at the expense of the precision rate since there will be more incorrect detections of the target behavior.

Pecan uses a Venn diagram to compare your model’s predictions against actual outcomes. It communicates both the precision and detection of your model, illustrating the percentage of correct and incorrect predictions in either direction.

It appears in the dashboard for a binary model, and appears as follows:

Here’s how to interpret the colors of the Venn Diagram:

So, for example, if this Venn diagram was for a churn model, “detected correctly” (the purple area) would represent instances where a customer was correctly predicted to churn.

The exact numbers for each result appear on the right side of the diagram.

The numbers behind your Venn Diagram give rise to the precision and detection metrics of your model. As illustrated below, as your model’s sensitivity threshold varies, so too does the ratio of correct and incorrect predictions.

For a model to be considered good (i.e. have relatively high precision and detection), the majority of your population should fall within the purple and gray circles.

However, model performance will vary depending on where you set its sensitivity threshold. This cutoff point should be established based on your unique business goals (see Understanding threshold logic).

Accuracy rate indicates how frequently your model is correct, but not how frequently it is incorrect. It does not consider false-positive and false-negative predictions, even though they can have a strong impact on the usefulness of a model:

Accuracy rate = (Detected correctly + Ignored correctly) / Total number of predictions

This formula answers the question: “How frequently are your predictions correct?”

Unfortunately, the goal of simply maximizing correct predictions is generally misguided. Accuracy is not always a useful measure of evaluation, especially in imbalanced datasets where instances of target behavior are rare (so a model that predicts “no” for everyone is almost always right).

For example, say you have a model designed to detect fraudulent activity (which comprises only 0.1% of all customer activity). Now let’s say the model predicts that that all activity is non-fraudulent – this means your model would be correct 99.9% of the time (a 99.9% accuracy rate). But this would still be a poor model because you can’t understand how good the model is at actually predicting fraudulent activity).

If, however, it was a more balanced dataset in terms of target behavior (i.e. a higher base rate), such as 45% of customer activity being fraudulent, the metric would be more sensitive to making mistakes, and thus be more useful,

Generally, it's difficult to use accuracy rate to guide actionable business goals and measure model impact, so it is not a useful metric for most use-cases.

Area Under the Curve demonstrates the diagnostic ability of your model as its threshold is varied. It allows for the statement: “When the rate of false-positive predictions is X, the rate of true-positive predictions is Y.”

This helps you understand how well a model separates positive and negative instances of target behavior. AUC score is attained by calculating the area under an ROC curve. It doesn’t depend on the threshold you choose (neither do AUPRC and LogLoss).

If the total area under the curve amounts to a fraction of 1.0, this represents a perfect predictive score.

Meanwhile, a score of 0.5 means the model ranks no better than random.

In Pecan, an AUC between about 0.65 and 0.95 is generally a healthy range. Above 0.95, check for leakage or overfitting before trusting the model; below 0.65, the model separates the classes poorly (see What should your AUC be?).

In Pecan, AUC can be used as an optimization metric for binary models.

For a deeper dive into this metric, see Understanding Area Under the Curve (AUC).

AUPRC measures how well your model's highest scores pick out the entities that really do the behavior. Like AUC, it doesn't depend on the threshold you choose. Unlike AUC, it focuses on the top of the list, which is usually where you act.

How it works

Pecan sorts your entities from the highest score to the lowest and walks down the list. Each time it reaches an entity that really did the behavior, it notes the precision so far: how many of the entities up to that point did the behavior. AUPRC is the average of those precision values. It ranges from 0 to 1, and higher is better.

Example

Five customers, sorted by their churn score:

AUPRC = average of 100% and 67% = 0.83. If the model had put both churners at the top, AUPRC would be 1.0.

Reading the score

Always read AUPRC next to your base rate. A model that ranks entities at random scores about the base rate, not 0.5. In the example above, 2 of 5 customers churned, so a random model would score about 0.4.

This is why a low-looking AUPRC can be a strong result. If only 2% of your entities do the behavior, an AUPRC of 0.15 is about 7 times better than random. Judge the score by how far it is above your base rate, not by how close it is to 1.

AUPRC or AUC?

Both measure how well the model ranks entities. AUC looks at the whole list, so when the behavior is rare, it can stay high even when many of your top-scored entities are wrong: most of the list is easy to rank. AUPRC shows that cost, because every wrong entity near the top lowers the precision at that point.

Use AUPRC when the behavior is rare and you'll act on the top of the list, for example calling the customers most likely to churn. AUC is fine when the behavior isn't rare, or when the whole list matters as much as the top.

In Pecan

AUPRC is the default optimization metric for binary models. Pecan trains several candidate models and deploys the one with the highest AUPRC unless you choose a different metric. See Optimization metrics for binary models.

In the dashboard, you'll find AUPRC in:

AUPRC is also the score the Draft Model Optimizer uses to decide whether a new version of a binary model is better.

LogLoss is an advanced metric that’s commonly used for predictive models. It helps differentiate between “good” and “poor” models by indicating how close their predictions are to the actual values (to 0 or 1 in the case of binary classification).

A model with perfect predictive ability would have a LogLoss of 0. And the more predictions diverge from the actual values, the higher the score.

What’s unique about LogLoss is that it penalizes a model more for making bigger mistakes than for making smaller mistakes. This is useful for binary models since it tends to push predictions towards higher and lower values that are more clearly distinguishable (as opposed to a flatter prediction curve where many entities receive a similar value).

This is a lot more helpful when trying to identify whom to perform a business treatment on. It’s also why, unlike the accuracy rate, this metric is more robust for evaluating unbalanced datasets.

This is how LogLoss is calculated:

Since LogLoss penalizes a model more for bigger errors than for smaller errors, using it to optimize a model can help you be more certain of individual predictions. Like AUC and AUPRC, LogLoss doesn’t depend on the threshold you choose: it’s calculated from the scores themselves.