For an overview of optimization metrics in Pecan, including why and when to use them, see Introduction to selecting an optimization metric.
This article explains how to select one for a binary model, which metrics are available, and when to use each one.
How to select an optimization metric for binary models
When you select an optimization metric, Pecan trains several candidate models and deploys the one that scores best on that metric. If you don't select one, Pecan uses the default, AUPRC.
Open your predictive flow and go to the Train & Evaluate tab. The Model configuration panel is on the left.
Under Advanced settings, expand Optimization metric.
Under Model type, select Binary. The list of metrics appears once a model type is selected. If you leave Model type empty, Pecan detects the model type from your label and uses the default metric.
Under Optimization metric, select a metric. AUPRC is selected by default. Use the table below to choose.
If you select precision or recall, set Entities to optimize model for. Read more
Click Train model.
Which metric to choose
In the list | Metric | Choose it when |
AUPRC | AUPRC (default) | You'll act on the entities with the highest scores, and acting on the wrong ones has a cost. The best starting point when the behavior you predict is rare. |
AUC | You need the whole population ranked well, and the behavior you predict isn't rare. | |
precision | Acting on the wrong entity is costly, and you'll act on a set number or share of entities. | |
recall | Missing an entity that does the behavior is costly. | |
accuracy | Both outcomes are about equally common, and both kinds of error cost about the same. | |
logloss | You use the predicted probability itself, not only the ranking. |
Which optimization metrics to use, and when
AUPRC (Area Under the Precision-Recall Curve)
AUPRC is the default optimization metric for binary models in Pecan.
Pecan sorts your entities from the highest score to the lowest. At every point down that list, it checks two things: how many of the entities so far really did the behavior (precision), and how many of all the entities that did it you've already found (detection). AUPRC sums up that trade-off in one score between 0 and 1. Higher is better.
Selecting AUPRC favors a model whose top-scored entities are mostly right. That's what matters when you act on the top of the list, for example calling the customers most likely to churn.
Read the score against your base rate. A model that guesses at random scores about the base rate. So an AUPRC of 0.15 can be a strong result when only 2% of entities do the behavior: that's about 7 times better than random.
AUC (Area Under the Curve)
AUC scores how well the model ranks entities that do the behavior above entities that don't, across the whole population and regardless of the model's threshold.
Selecting AUC favors a model that ranks the whole list well. It's a good choice when the behavior you predict isn't rare, or when the whole list matters as much as the top of it.
When the behavior is rare, AUC can look high even when many of the top-scored entities are wrong, because most of the list is easy to rank. AUPRC shows that cost more clearly, which is why it's the default.
Precision Rate
Selecting this metric (precision in the list) favors a model that maximizes True Positives and minimizes False Positives. In other words, it favors making correct positive predictions, even if it means missing some instances of the behavior.
In general, choose Precision Rate when the cost of False Positives is high (taking action has a significant cost or risk), and you want to act only on the right entities.
When you choose Precision Rate, you also set the share or number of entities to optimize for. Note: this metric may not be right if the behavior you predict is rare.
Detection Rate
Selecting this metric (recall in the list) favors a model that maximizes True Positives and minimizes False Negatives. In other words, it favors finding all instances of the behavior, even if it means more False Positives.
In general, choose Detection Rate when the cost of False Negatives is high (not taking action has a significant cost or risk), and you want to miss as few as possible.
When you choose Detection Rate, you also set the share or number of entities to optimize for.
Accuracy Rate
Selecting this metric (accuracy in the list) favors a model that makes the most correct predictions overall, positive or negative, even if it means more False Positives or False Negatives.
In general, Accuracy Rate is appropriate when both outcomes are about equally common and both kinds of error cost about the same. Avoid it when the behavior is rare.
Here's why. Say you're predicting high-value players, and they're only 2% of the population (a very low base rate). A model that predicts no one is high-value is correct 98% of the time, so its Accuracy Rate is 98%, even though it finds no high-value players at all. AUPRC or Detection Rate would be more appropriate.
LogLoss (Logarithmic Loss)
Selecting this metric (logloss in the list) favors a model whose predicted probabilities are closest to what actually happens, not only one that ranks well. LogLoss penalizes confident wrong predictions most. The lower the score, the better.
In general, LogLoss is useful when you act on the probability value itself.
For example, a business that wants to call every customer with a probability above 50% of upselling may choose LogLoss. A business that simply wants to call the 500 customers most likely to upsell, whatever their probabilities, may choose AUPRC instead.
How to choose which entities to optimize for
When you choose precision or recall, you also set Entities to optimize model for: the group whose predictions you care most about. Pecan calculates the metric for the entities with the highest scores only.
Choose Top percent to optimize for a share of entities, for example the top 10%.
Choose Top absolute number to optimize for a set number of entities, for example the top 1,000.
If the number you enter is larger than the population, Pecan calculates the metric for the whole population.
Example: precision for the top 1,000
Say you're planning a costly ad campaign for the 1,000 players likely to generate the most revenue. Targeting a player who won't generate revenue (a False Positive) wastes budget, so you choose Precision Rate.
You can only target 1,000 players, so you want those 1,000 to be right. Select precision as your optimization metric, then Top absolute number with 1,000.
Example: detection for the top 20%
Now say you're offering a small discount to the 20% of users most likely to churn. Missing a user who churns (a False Negative) costs far more than the discount, so you choose Detection Rate.
You want your top 20% to catch as many of the users who'll churn as possible. Select recall as your optimization metric, then Top percent with 20.
Note: precision and rare behavior
If the behavior you predict is rare, avoid optimizing for precision on a group larger than your base rate.
Here's why. Say 5% of your entities churn, and you optimize precision for the top 10%. Even a perfect model can fill only half of that group with entities that churn, so the best possible precision is 50%. Precision stays low however good the model is, and it doesn't separate good models from weak ones.
If you plan to act on a group larger than your base rate, optimize for Detection Rate (recall) instead. It reflects the model's quality better, and it can get close to 100%.
If you have questions about choosing an optimization metric for your model, reach out to a Pecan expert or your customer success manager.


