Model Backtesting & Validation

Ensuring the accuracy and reliability of ECL models

What is Backtesting?

Backtesting is the process of comparing a model's predictions (ECL, PD, LGD) against actual observed outcomes (Default Rates, Realized Losses) over a historical period. It is a critical component of Model Validation required by regulators and auditors to ensure that the IFRS 9 models are performing as expected.

IFRS 9 requires models to be "unbiased and probability-weighted." Regular backtesting helps identify if a model is:

  • Over-estimating risk: Leading to excessive provisions and reduced capital efficiency.
  • Under-estimating risk: Leading to insufficient coverage and potential surprise losses.

PD Calibration: The Binomial Test

The most common test for Probability of Default (PD) calibration is the Binomial Test. It assesses whether the observed default rate in a portfolio is statistically consistent with the predicted PD.

How it Works

We assume that defaults follow a Binomial distribution. We calculate a Z-score to measure how many standard deviations the observed default rate is from the expected PD.

Z = (Observed Rate - Expected PD) / √[ (Expected PD * (1 - Expected PD)) / N ]

Where N is the number of loans in the portfolio.

The Traffic Light Approach

Results are typically categorized using a "Traffic Light" system based on the p-value derived from the Z-score:

Green Zone

No significant difference. The model is calibrated correctly. (p-value ≥ 0.05)

Amber Zone

Possible deviation. Monitor closely; may require investigation. (0.01 ≤ p-value < 0.05)

Red Zone

Significant deviation. The model is likely failing. Recalibration required. (p-value < 0.01)

Loss Coverage Analysis

While PD calibration focuses on the frequency of defaults, Loss Coverage Analysis focuses on the financial impact. It compares the total Expected Credit Loss (ECL) provisioned against the total Realized Losses (write-offs).

Key Metrics
  • Total Predicted ECL: The sum of ECL for all loans at the start of the period.
  • Total Realized Loss: The actual amount lost (Exposure * Actual LGD) for defaulted loans.
  • Coverage Ratio: Predicted ECL / Realized Loss.
Interpretation:
  • Ratio > 1.0: Conservative (Surplus). The model provisioned more than was lost.
  • Ratio < 1.0: Under-provisioned (Shortfall). The model underestimated the losses.

Ideally, the ratio should be close to 1.0, or slightly above 1.0 to include a margin of conservatism. A ratio significantly below 1.0 indicates that the LGD or EAD parameters might be underestimated.

Population Stability Index (PSI)

PSI measures the shift in the distribution of a population over time (e.g., comparing the portfolio at model development vs. current portfolio). It helps determine if the model is still relevant for the current population.

PSI Value Interpretation Action
< 0.10 No significant shift No action required
0.10 - 0.25 Moderate shift Monitor and investigate
> 0.25 Significant shift Model may need redevelopment

Try It Yourself

Use our interactive Backtesting Tool to simulate a portfolio, generate synthetic defaults, and run these statistical tests in real-time.

Launch Backtesting Tool
Reporting Next: Backtesting Tool