8. Scenario plausibility

A conditional forecast always succeeds. You impose a path, RISE finds shocks that deliver it, and a chart comes back. There is no path so absurd that no shock sequence produces it, so success carries no information.

What the chart does not say is whether those shocks are mild judgment or a sequence the model would never generate. Two runs can publish the same path while demanding wildly different things of the model.

rise.scenario.plausibility puts a number on the difference.

sp = simplan(m, [0, 12], 1);
sp = free_shocks(sp, 'all', 1);
sp = append(sp, 'Y', 1, 0.6);

db = simulate(m, 'simul_historical_data', sp, 'simul_periods', 12);

rep = rise.scenario.plausibility(m, db);

Scenario plausibility
----------------------------------------------------------
shock slots scored          3 (3 shocks x 1 periods)
largest move                EPS_S at period 1, -0.42 sigma
sum of squares              0.200   (tail probability 0.9776)
divergence from the model   0.1001
q                           0.6270   a scenario the model is comfortable with
----------------------------------------------------------

8.1. How to read it

q is the headline. It lives in [0.5, 1] and is the bias of a coin that would be as far from a fair one as the scenario is from the model. Near 0.5 the model would hardly notice; near 1 it would essentially never produce this on its own.

largest move names the shock doing most of the work, in units of its own standard deviation, and the period it happens in. This is the Leeper–Zha reading: is the intervention modest?

sum of squares and its tail probability treat the shocks jointly. Useful, but it is the divergence — scaled by the number of shock slots — that discriminates between scenarios of different length.

Note

q is not a test. It has no size and no critical value, and the verdicts the report prints are a reading aid. What q is good at is comparison: this scenario against that one, this attribution rule against another, on the same model.

8.2. What it is measuring

The model believes its structural shocks are standard normal over the nh slots the scenario touches. A scenario implies a different distribution, with mean u and variance S. The distance between the two is the Kullback–Leibler divergence

D = 0.5 * ( trace(S) + u'u - nh - log(det(S)) )

calibrated onto [0.5, 1] by

q = ( 1 + sqrt(1 - exp(-2*D/nh)) ) / 2

The calibration is against a binomial rather than a Bernoulli, which is what keeps q from drifting to one merely because the scenario is long.

A hard conditional forecast returns one shock path rather than a distribution, so S defaults to the identity: the scenario shifts the shocks’ mean and leaves their spread alone, and D is half the sum of squared standardized shocks. Where a forecast carries shock uncertainty, pass its covariance through Covariance.

8.3. Who is made to do the work

The most useful thing the metric exposes is a choice that used to be invisible. Impose the same target twice — once letting every shock share it, once insisting a single shock is responsible:

rule                sum of squares       q    worst move
all shocks share            0.2001  0.6270    EPS_S -0.42 sd
EPS_D alone                 2.5208  0.8770    EPS_D +1.59 sd

The published path is identical. Attributing the move to demand alone costs twelve times the model violence, and moves the verdict from “comfortable” to “being leaned on”. That is a modelling assumption with a price, and now the price is printed.

8.4. Options

Periods      which forecast periods to score (default: every period
             in which some shock is non-zero)
Shocks       which shocks to score (default: all of the model's)
Covariance   the scenario's shock covariance, nh by nh
Print        show the report (default true)

8.5. What it refuses

noShocksToScore        every shock is zero: the model was asked for
                       nothing, so there is no distance to measure
unknownShock           a name that is not an exogenous variable
noShocksInDatabank     the databank carries none of the model's shocks
periodOutOfRange       a period the forecast does not reach
badCovariance          the covariance is not nh by nh

All carry the prefix RISE:scenario:. A scenario that asks for nothing raises an error rather than returning a flattering q = 0.5.

8.6. References

  • Antolín-Díaz, Petrella and Rubio-Ramírez (2021), “Structural scenario analysis with SVARs”, Journal of Monetary Economics 117, 798–815 — the divergence and its calibration.

  • Leeper and Zha (2003), “Modest policy interventions”, Journal of Monetary Economics 50, 1673–1700 — the per-shock reading.

  • McCulloch (1989), “Local model influence”, JASA 84, 473–478 — calibrating a divergence against a coin.

8.7. See also