In a regression model, predicted and actual results exist on a continuum. Instead of being classified into mutually exclusive populations (e.g. with a label of “0” or “1”), results are represented as numbers.

Whereas a binary model separates a population into two classes (one of which is of interest and will receive a business treatment), in a regression model, each member of the population is of interest. What may differ between them is the degree of interest.

Therefore, for regression models, you need metrics that take into account each member of the population. Below is a list of metrics that are useful for evaluating the results of a regression model in Pecan:

Mean Absolute Percentage Error (MAPE)

Weighted MAPE (WMAPE)

Weighted MPE (WMPE)

Weighted MAPE by Weighted MPE

Explained Variance (R2)

Normalized Gini

Root Mean Squared Error (RMSE)

Root Mean Squared Logarithmic Error (RMSLE)

Root Mean Squared Percentage Error (RMSPE)

Mean Absolute Percentage Error, a common metric for forecasting error, reflects the prediction accuracy of a regression model. It calculates the absolute percentage error for each prediction, and then takes the average. It is calculated as follows:

For example: a $100 prediction vs. a $110 actual outcome produces a percentage error of 10% (it is “absolute” because it doesn’t consider the direction of the error). In general, a percentage approaching zero indicates a better model. But what is practically considered “a good percentage” will depend on your business benchmarks.

The downside of using Mean Absolute Percentage Error is that not all errors are equal, and if you have an unbalanced distribution of results, the MAPE may be high even if the errors are not significant overall.

Let’s say you have a user who is predicted to spend 2$ but spends only $1 – this gives an error of 100%. Then you have a user who is predicted to spend $1,000 but actually spends 1,100$ – an error of only 10%. Even though the MAPE will score better for the second example, it’s clearly not a fair representation of error value. This issue is resolved with Weighted MAPE.

In Pecan, MAPE can be used as an optimization metric for regression models.

Like MAPE, Weighted Mean Absolute Percentage Error expresses prediction accuracy as a measure of error. The difference is that it weighs each prediction error relative to the results of the entire dataset (instead of calculating each error independently.) In other words: the greater the percentage error of a prediction, the greater its effect on the score.

For predictive models in Pecan, Weighted MAPE is typically considered a more reliable metric than the non-weighted variations of absolute percentage error. It is calculated as follows:

For example: a $1 error would have a greater impact on the score if it occurred for a $10 purchase than for a $100 purchase. (Compare this to MAPE, where a $1 error would have the same impact whether the purchase value was $10 or $100.)

This approach “levels the playing field”, making it possible to compare model performance for disparate datasets. For example: you can evaluate a model’s ability to predict the outcomes of entirely different marketing campaigns or channels. (However, in some cases, it may be necessary to split your dataset into multiple cohorts based on business logic – like by campaign – find the Weighted MAPE for each of them, and then use their average to score the model.)

In Pecan, Weighted MAPE is displayed in the dashboard as Mean absolute error. It can be used as an optimization metric on its own, and it's also a component of WMAPE by WMPE, which is the default optimization metric for regression models.

Like Weighted MAPE, Weighted Mean Percentage Error calculates the mean prediction error of the population and gives greater weight to errors of greater value.

The difference is that it does not use absolute values. Instead, it factors in whether each error is a positive or negative value – thereby expressing direction. A value above zero represents an over-prediction, and a value below zero would represent an under-prediction.

Weighted MPE is calculated as follows:

For example: if a model predicts total revenue of $100,000 from all users, and the actual revenue turned out to be $102,000 the Weighted MPE would be -0.02 (an under-predicting model). But if the actual revenue turned out to be $980,000, the score would be +0.02 (an over-predicting model).

Since Weighted MPE calculates the mean percentage error of all summed predicted values, positive and negative values will cancel each other out. This means the metric will usually be closer to zero than Weighted MAPE, whose absolute values do not cancel each other out.

This makes for an interesting insight: if your Weighted MPE and Weighted MAPE are very similar, it means that most of your errors are biased in one direction. This indicates indicating that your model is either drastically over-predicting or under-predicting.

In Pecan, Weighted MPE is displayed in the dashboard for regression models as Error bias, the dashboard's default main metric. It can be used as an optimization metric on its own, and it's also a component of WMAPE by WMPE, which is the default optimization metric for regression models.

Weighted MAPE and Weighted MPE are distinct metrics listed separately in the dashboard for regression models in Pecan. However, Pecan calculates “Weighted MAPE by Weighted MPE” as an effective and reliable metric for evaluating the performance of regression models—it is the default optimization metric for these models. To determine this metric, Pecan follows a systematic approach:

This method ensures that the selected model not only has minimal bias (as indicated by a low WMPE) but also the lowest error magnitude (as indicated by WMAPE) within an acceptable range of bias. By prioritizing models with the smallest WMPE and optimizing for the lowest WMAPE within those, Pecan provides a balanced and reliable evaluation of model performance.

Unlike the above metrics which measure error, R2 detects the level of correlation between a model’s predictions and the actual results. It indicates the percentage of variance that’s explained by the model (and not by the data itself). The higher the score, the better a predictive model can explain variance based on the features that have been fed into the model.

R2 is calculated by summing the square of the distance between predicted values and actual values, and then dividing that value by the square of the distance between the actual values and the mean. More simply, it calculates the variance explained by the model and divides that by the total variance. This is shown below:

A score close to 1 (or 100% in percentage terms) means your model has achieved a high correlation between predictions and actual results. A score closer to 0 indicates poor predictive performance due to a poor model and/or a large amount of variance in your dataset.

As an example, a score of 0.71 means that 71% of the variability of predictions is accounted for by the model, and the remaining 29% is accounted for by variance in the original data.

Using R2, you can ensure that most of the variability in predictions is attributed to your model – and not variability in your data. This provides a measure of how well results can be replicated by the model, using a given benchmark.

Graphical representation

R2 measures the goodness of fit by calculating how far each dot is from the fitted regression line (the basis of the model’s predictions). This distance is known as the “residual”. In the below illustration, actual outcomes are plotted along a linear regression line.

The more scattered the plots, the more variance is accounted for by the values themselves, and the worse the predictive ability of the model (giving a lower R2 score). To put it another way: the further plots are from the regression line (the greater the residuals), the more they punish the model (the greater the impact on the R2 score).

This metric is particularly useful for models that are designed to establish a trend (e.g. attempting to rank customers between 0 and 1). Each prediction may be off by a certain amount, but you would want to ensure that higher predictions correspond with higher actual results. If ranking is all that matters for your use case, see Normalized Gini below.

In Pecan, R2 can be used as an optimization metric for regression models. The dashboard also uses it in a health check that compares how consistently the model performs on the training and test sets.

Most regression metrics measure how close each prediction is to the actual value. Normalized Gini asks a different question: did the model put your entities in the right order? Is the model ranking the entities correctly?

A simple way to tell them apart: if your question is "Who are my top spenders?" (or, “Are top ranked entities my top spenders?”), Normalized Gini is the metric to look at. If your question is "How much will they spend?", use Weighted MAPE.

Pecan sorts your entities from the highest prediction to the lowest. It then walks down that list and tracks how quickly the actual total value adds up. A model that ranks well puts the entities with the highest actual values at the top, so most of the total is reached early in the list.

Pecan compares this to the best possible ranking, which is sorting by the actual values themselves. Normalized Gini is the ratio between the two:

Because only the order matters, the size of the predictions doesn't affect the score. A model that predicts 10 times every customer's actual spend can still score a perfect 1.0, as long as the customers are in the right order.

Five customers, and two models predicting their spend over the next 12 months:

Model X predicts every amount 10 times too low, so its Weighted MAPE is poor. But it puts all five customers in exactly the right order, so its Normalized Gini is perfect. If you only need to pick your top two customers for a VIP offer, Model X picks the right ones.

Model Y predicts amounts much more accurately, so it's the better choice for revenue planning. But it ranks customer B above customer A, so its Normalized Gini is lower.

As with the other metrics, what counts as a good score depends on your use case and business benchmarks.

Normalized Gini scores the order (ranking) of your entire list, not only the top of it. A model that gets the top 10% right but mixes up everyone else will still lose points. In practice, a model that ranks well overall is also the one that puts the right customers at the top.

If all actual values in your test set are the same, there's nothing to rank, and the score is left blank.

Normalized Gini fits use cases where you act on the order of predictions rather than on the amounts. For example, an LTV model used to choose the top 10% of customers for VIP treatment, retention offers or lookalike audiences.

Stick with Weighted MAPE (or the default, WMAPE by WMPE) when the predicted amount itself is used downstream. For example, ROAS calculations, budget planning or revenue forecasting.

R2 also rewards a model whose higher predictions go with higher actual values, but it still penalizes predictions that are far from the actual amounts. Normalized Gini ignores the amounts completely.

Normalized Gini can be used as an optimization metric for regression models. To select it, go to Advanced settings > Optimization metric, set Model type to Regression, and choose NORMALIZED_GINI. It's available for Production grade training only, not Fast training.

When a model is trained with Normalized Gini as its optimization metric, Pecan trains several candidate models and deploys the one with the highest Normalized Gini. You can see each candidate's score in the Normalized Gini column of the trained models table, under Model Technical Details.

The main metrics at the top of the dashboard stay the same: Error bias and Mean absolute error. Since your model was chosen for how well it ranks, its Mean absolute error may be higher than that of a model optimized for accuracy. That's expected. Judge it by its Normalized Gini.

Root Mean Squared Error calculates the average distance between a model’s predicted values and the actual values in the dataset (this distance is known as the “residual” or prediction error). In doing so, it tells you how concentrated the actual data is around the line of best fit (the model’s regression line).

RMSE is the standard deviation of the residuals of a dataset. As shown below, it’s calculated by squaring the distance between the actual values (“y”) and the predicted values (“ŷ”), calculating the mean, and then taking its square root. The lower the score, the better the model is able to “fit” the dataset – indicating better predictive performance.

This metric is similar to Explained Variance (R2), but it gives more weight to larger prediction errors – and penalizes outliers – because it squares the residual (the prediction error). As a result, it also produces predictions that are closer to the mean than to the median.

Therefore, you would use this metric if you wish to penalize large errors and have predictions that are more on the “conservative” side.

In Pecan, RMSE can be used as an optimization metric for regression models. It isn't shown in the dashboard.

Like RMSE, this metric measures how spread out the residuals are in a predictive model (where a “residual” is the distance between an actual result and the model’s regression line). This tells you how concentrated the actual data is around the line of best fit. The lower the score, the better the model.

The difference is that the Root Mean Squared Logarithmic Error doesn’t immediately square the residual before calculating the mean value. Instead, it does a log transformation on the residual, calculates the mean, and then takes the square root. It is calculated as follows:

Whereas RMSE emphasizes the impact of outliers, RMSLE reduces the impact of them. This is appropriate when the magnitude of outliers is not meaningful – that is when the magnitude doesn't significantly affect the usefulness of the model. Accordingly, it’s appropriate for business use-cases where over-predicting is more acceptable than under-predicting.

In Pecan, RMSLE can be used as an optimization metric for regression models. It isn't shown in the dashboard.

Like RMSE, this metric measures the distance between a model’s predicted values and the actual values in the dataset (a.k.a. the “residual” or prediction error) – thus indicating how concentrated the actual data is around the line of best fit. The lower the score, the better the predictive performance of the model.

The difference is that the Root Mean Squared Percentage Error “scales down” the magnitude of the final value. Once the residual percentage error is calculated and squared, it’s divided by the square of the observed values. Then it’s divided by the number of samples before the square root is taken. It is calculated as follows:

The approach reduces the impact of prediction error on your model. It is appropriate for business use-cases where the magnitude of either over-predicting or under-predicting doesn’t significantly affect the usefulness of the model.

In Pecan, RMSPE can be used as an optimization metric for regression models. It isn't shown in the dashboard.