Skip to content

evaluation

mae(y_true, y_pred)

Return mean absolute error (MAE).

Parameters:

Name Type Description Default
y_true DataFrame

Ground truth (correct) target values.

required
y_pred DataFrame

Predicted values.

required

Returns:

Name Type Description
scores DataFrame

Score per series.

mape(y_true, y_pred)

Return mean absolute percentage error (MAPE).

Parameters:

Name Type Description Default
y_true DataFrame

Ground truth (correct) target values.

required
y_pred DataFrame

Predicted values.

required

Returns:

Name Type Description
scores DataFrame

Score per series.

mase(y_true, y_pred, y_train, sp=1)

Return mean absolute scaled error (MASE).

Parameters:

Name Type Description Default
y_true DataFrame

Ground truth (correct) target values.

required
y_pred DataFrame

Predicted values.

required
y_train DataFrame

Observed training values.

required

Returns:

Name Type Description
scores DataFrame

Score per series.

mse(y_true, y_pred)

Return mean squared error (MSE).

Parameters:

Name Type Description Default
y_true DataFrame

Ground truth (correct) target values.

required
y_pred DataFrame

Predicted values.

required

Returns:

Name Type Description
scores DataFrame

Score per series.

overforecast(y_true, y_pred)

Return total overforecast.

Overforecast (positive forecast bias) is the difference between actual and predicted for predicted values greater than actual.

Parameters:

Name Type Description Default
y_true DataFrame

Ground truth (correct) target values.

required
y_pred DataFrame

Predicted values.

required

Returns:

Name Type Description
scores DataFrame

Score per series.

rank_fva(y_true, y_pred, y_pred_bench=None, scoring=None, descending=False)

Sorts point forecasts in y_pred across entities / time-series by score.

Parameters:

Name Type Description Default
y_true DataFrame

Panel DataFrame of observed values.

required
y_pred DataFrame

Panel DataFrame of point forecasts.

required
y_pred_bench DataFrame

Panel DataFrame of benchmark forecast values.

None
scoring Optional[metric]

If None, defaults to SMAPE.

None
descending bool

Sort in descending order. Defaults to False.

False

Returns:

Name Type Description
ranks DataFrame

Cross-sectional DataFrame with two columns: entity name and score.

rank_point_forecasts(y_true, y_pred, sort_by='smape', descending=False)

Sorts point forecasts in y_pred across entities / time-series by score.

Parameters:

Name Type Description Default
y_true DataFrame

Panel DataFrame of observed values.

required
y_pred DataFrame

Panel DataFrame of point forecasts.

required
sort_by str

Method to sort forecasts by:

  • mean
  • median
  • std
  • cv (coefficient of variation / volatility)
  • mae (mean absolute error)
  • mape (mean absolute percentage error)
  • mase (mean absolute scaled error)
  • mse (mean squared error)
  • rmse (root mean squared error)
  • rmsse (root mean scaled squared error)
  • smape (symmetric mean absolute percentage error)
  • smape_original
  • overforecast
  • underforecast
'smape'
descending bool

Sort in descending order. Defaults to False.

False

Returns:

Name Type Description
ranks DataFrame

Cross-sectional DataFrame with two columns: entity name and score.

rank_residuals(y_resids, sort_by='abs_bias', descending=False)

Sorts point forecasts in y_pred across entities / time-series by score.

Parameters:

Name Type Description Default
y_resids DataFrame

Panel DataFrame of residuals by splits.

required
sort_by str

Method to sort residuals by: bias, abs_bias (absolute bias), normality (via skew and kurtosis tests), or autocorr (Ljung-box test for lag = 1). Defaults to abs_bias.

'abs_bias'
descending bool

Sort in descending order. Defaults to False.

False

Returns:

Name Type Description
ranks DataFrame

Cross-sectional DataFrame with two columns: entity name and score.

rmse(y_true, y_pred)

Return root mean squared error (RMSE).

Parameters:

Name Type Description Default
y_true DataFrame

Ground truth (correct) target values.

required
y_pred DataFrame

Predicted values.

required

Returns:

Name Type Description
scores DataFrame

Score per series.

rmsse(y_true, y_pred, y_train, sp=1)

Return root mean squared scaled error (RMSSE).

Parameters:

Name Type Description Default
y_true DataFrame

Ground truth (correct) target values.

required
y_pred DataFrame

Predicted values.

required
y_train DataFrame

Observed training values.

required

Returns:

Name Type Description
scores DataFrame

Score per series.

smape(y_true, y_pred)

Return symmetric mean absolute percentage error (sMAPE).

Use third version of SMAPE formula from https://en.wikipedia.org/wiki/Symmetric_mean_absolute_percentage_error to deal with zero division error

Parameters:

Name Type Description Default
y_true DataFrame

Ground truth (correct) target values.

required
y_pred DataFrame

Predicted values.

required

Returns:

Name Type Description
scores DataFrame

Score per series.

underforecast(y_true, y_pred)

Return total underforecast.

Underforecast (negative forecast bias) is the difference between actual and predicted for predicted values less than actual.

Parameters:

Name Type Description Default
y_true DataFrame

Ground truth (correct) target values.

required
y_pred DataFrame

Predicted values.

required

Returns:

Name Type Description
scores DataFrame

Score per series.