4. Filtering

This chapter covers the linear and nonlinear filtering machinery that filter, estimate, forecast, and the historical decomposition route into.

4.1. Linear filtration

Filtering with constant parameters can be done efficiently with the algorithms of De Jong et al. (2003), Koopman and Durbin (2000, 2003), and Durbin and Koopman (2012). These procedures cannot be used directly in a regime-switching context.

The main references for the filtration of Markov-switching state-space models are Kim (1999) and Kim and Nelson (2001), which combine the Kalman and Hamilton filters. Similar algorithms appear in the engineering literature (Bar-Shalom et al., 2001, 2004).

RISE uses an extension of the Durbin-Koopman (2012) algorithm except for the initialisation process, which in the non-stationary case has not been extended to regime-switching models. For non-stationarity, RISE applies an arbitrary variance for the initialisation.

For efficiency, RISE deviates from Kim-Nelson (2001) when collapsing the regimes. Kim-Nelson collapse the updates; RISE collapses the forecasts. The two procedures yield numerically similar results but the RISE procedure is faster.

4.1.1. Constraints and announced shocks

The filter applies the model as the simulation does. Each period’s state is \(y_t = g(x_{t-1}, \eta_t)\), the one-step map from the previous state and the period’s drawn shocks; with occasionally-binding constraints, \(g\) includes the constraint engine, which solves the enforcing shocks (the fudge shocks of the kinks, or the shocks a simul_constraints row names) so that the constraints hold.

  • Announced shocks. A shock with a positive solve_shock_horizon is announced up to that many periods ahead. The announcements are part of a period’s information and need not come true: the simulation draws every page a shock’s horizon covers afresh each period, and the filter integrates over all of them. The enforcing shocks are solved, never drawn, and carry no variance.

  • Constraints. Where the map is linear (no enforcement on the way) the update is the Kalman update. Where the enforcement bends it, the filter updates the inputs \((x_{t-1}, \eta_t)\) instead, with the map linearized where the previous iterate landed (an iterated extended Kalman update); where the data contradict the enforcement at the prediction (a rate predicted at its floor and observed above it), the step is taken with the enforcement released. The updated state is \(g\) at the updated inputs, so the filtered series respect the constraints. With one shock and its observable measured without error the update agrees with inverting the model for that shock. A branch whose enforcers barely move the constraints they hold (in a reduced model whose basis leaves out their responses, see Model reduction) amplifies the slopes beyond what double precision resolves; it is not used, and the update keeps the Kalman update and its projection. The likelihood is then not the constrained model’s: the filter warns (rise:filter:illConditionedConstrainedUpdate, once per call) and marks the periods in its output (field ill_conditioned_updates, at every filtering level). An observed variable held at its bound (a rate observed at its floor) says nothing about the shadow variable behind it, and there too the Kalman update stands. Apart from these two cases, an update through the constrained step that fails outright – its innovation covariance not positive definite, the constrained step or its slopes failing, the iteration not settling – stops the filter with retcode 314 (CONSTRAINED_UPDATE_FAILED): there is then no likelihood of the constrained model, and an estimation treats the draw as inadmissible.

  • Regimes. A regime-switching filter predicts the next period once for each regime it may be in, and weights the predictions by the transition probabilities at the current state. Each constrained prediction is taken in the regime of its branch, as the branch’s covariance and update are; the anticipated pages of the constraint engine stay in that regime. Under a switch rule (simul_regime a function) the rule selects the regime instead.

  • simul_disable_constraints = true filters the model as if it had no constraints.

  • Where the update falls back. Where the constrained update takes its designed fallback (an observed variable held at its bound) and the projection cannot put the state back on the constraints with the observed variables fixed, the filtered state is the Kalman update and need not satisfy the occasionally binding constraints; the likelihood of those periods is the Kalman update’s.

4.1.2. Smoothing under regime switching

The regime-switching filters imm, imm_o, gpbn (any history length), immn and crskfc share one smoother. It smooths each pair of linked regime histories (a history this period and one it can lead to next period) against that pair’s own prediction, and weighs the pairs with Kim’s posterior weights on the chain of histories. As a result, any identity that the model states in every regime holds in the smoothed path, the probability-weighted average included: a level and its growth rate, for example, stay consistent even when both are unobserved. The likelihood and the filtered series are not affected. crskfc uses this smoother by default (hkkm_smoother = false keeps its classical one); kim_nelson keeps its own.

With gpbn of order 1 (gpb1), which predicts every regime from one collapsed state, the smoother pairs each regime of a period with each regime of the next through the pair’s own update of the next period, and weighs the pairs by their likelihood of the next observation. The smoothed path keeps the model’s identities. On a volatility-switching test model, gpb1’s smoothed unobservables are 2-3% less accurate than with the smoother it replaces (which broke the identities), and its smoothed regime probabilities are more accurate. Higher orders (gpb2, …) need no such pairing and smooth more accurately.

Under occasionally binding constraints the smoothed series reproduce the observed data, but in binding periods they need not respect the constraints.

4.1.3. Historical decomposition

Warning

Exact only for constant-parameter linear models.

4.1.4. Observables decomposition

Important

Available only for constant-parameter models.

4.2. Conditional forecasting via filtering

Conditional forecasting can be implemented using filtering algorithms. The smoother treats the conditioning values as observations on the forecast horizon and fills in everything else by least squares – no simulation plan is required, which makes this the most convenient route when its assumptions hold.

The assumptions are restrictive: it works only for linear models, and the conditioning variables must be on the observed side (declared in @observables). For anything else – hard conditions on unobserved states, nonlinear solutions, regime switching, anticipated shocks, soft conditions enforced through shocks rather than measurement noise – use a simulation plan through Forecasting and simulation.

4.2.1. Hard conditions

Extend the observation database to cover the forecast horizon. Numeric entries pin the observed variable at that date; NaN entries are free. Then call filter:

db = struct();
db.PAI = [pai_history; pai_future_conditions];   % conditioned
db.R   = [r_history;   nan(H, 1)];               % free
db.DY  = [dy_history;  nan(H, 1)];               % free

m = set(m, data = db);
[filt, ll, ~, rc, m] = filter(m);

The smoothed series in filt are the conditional forecast for every endogenous variable and shock. The implied shock path is the least-squares one that makes the conditioning observations consistent with the model.

4.2.2. Soft conditions

Relate the conditioning observation X0 to a model variable X as \(X0_t = X_t + \sigma \varepsilon_t\). The scalar \(\sigma\) controls the softness of the condition:

  • \(\sigma \to \infty\) – the condition is irrelevant.

  • \(\sigma \to 0\) – hard conditions.

  • \(\sigma\) strictly positive and finite (small enough) – soft conditions.

4.2.3. When to choose this route

  • Model is linear (constant-parameter or regime-switching).

  • Conditions are on observed variables.

  • You want a least-squares decomposition of the implied shock path.

  • You do not need to control anticipation: the smoother treats the conditioning path as fully observed up to its date, which is closest to “unanticipated until realised” in spirit.

4.2.4. When the route is the wrong tool

  • Higher-order or strongly nonlinear solutions – the least-squares interpretation breaks down.

  • Conditions on unobserved states (e.g. natural rate, output gap) – declare them as @observables first or switch to a simulation plan.

  • You want to choose which shocks the solver may use to enforce a condition (the simplan identification rule).

  • You want to mix anticipated and unanticipated shocks in the same scenario.

4.3. Known regime paths and heteroskedastic shocks

When the regimes of a Markov-switching model are known over the sample – a policy regime with known dates, or a volatile episode such as a pandemic – the model can be filtered on that path instead of with regime probabilities. The filter myKnownRegimesFilter is the exact Kalman filter of the time-varying system

\[x_t = c(s_t) + T(s_t)\, x_{t-1} + R(s_t)\, \varepsilon_t, \qquad y_t = Z x_t + v_t, \quad \operatorname{cov}(v_t) = H(s_t),\]

where \(s_t\) is the given regime of period \(t\) and \(T(s), R(s), c(s)\) are the model’s first-order decision rules in regime \(s\).

4.3.1. The call

The path is passed with the filter, through kf_user_algo; the same option goes to estimate:

s = ones(1, n);  s(161:164) = 2;           % s(t): the regime of period t
[f, LogLik] = filter(m, kf_user_algo = {@myKnownRegimesFilter, s});
m_est = estimate(m, estim_priors = priors, ...
                 kf_user_algo = {@myKnownRegimesFilter, s});

  • s has one entry per period of the estimation sample. s(t) is the regime of period \(t\): the regime whose solution produces period \(t\) from period \(t-1\) and the shocks of period \(t\) (so the shocks of period \(t\) have the standard deviations of regime s(t)).

  • With several Markov chains, the entries are the composite regimes, the rows of m.markov_chains.regimes.

  • The initial state and covariance are those of the regime of the first period, s(1): its steady state and, when its solution is stationary, its unconditional covariance. kf_user_init = {a0, P0} imposes them; kf_init_variance applies as for the other filters.

  • The output has the usual fields; the regime probabilities are the 0/1 indicators of s. Missing observations, measurement errors and kf_presample are handled as in the other filters. The smoothed shocks are the unit-variance innovations of the model’s equations.

4.3.2. Data simulated on a path

In simulate, the first point of a simulation (with simul_burn = 0) is the initial condition and carries the first entry of simul_regime; simulated period \(t\) carries entry \(t+1\). To simulate \(n\) periods with the regimes s and filter them with s:

db = simulate(m, simul_regime = [s(1), s], simul_periods = n+1, simul_burn = 0);
% keep the simulated periods 1 to n: points 2 to n+1 of each series

4.3.3. Heteroskedastic shocks

The heteroskedastic filter lets the shocks of given periods have a larger variance, leaving the decision rules and the steady state unchanged. In RISE a standard deviation is a coefficient of a unit-variance shock in the model equations; writing it as a switching parameter of a chain that moves nothing else gives that filter:

@exogenous eg eu
@parameters phi bet kappa rho_g rho_u sig_u vol_tp_1_2 vol_tp_2_1
@parameters(vol,2) sig_g

@model
  y  = phi*y{+1} + g;
  pi = bet*pi{+1} + kappa*y + u;
  g  = rho_g*g{-1} + sig_g*eg;
  u  = rho_u*u{-1} + sig_u*eu;

@parameterization
  sig_g(vol,1) = 1;   sig_g(vol,2) = 8;
  vol_tp_1_2 = 0.02;  vol_tp_2_1 = 0.5;   % needed to solve the model

then filter or estimate with the path of the vol regime as above. sig_g_vol_2 is an ordinary parameter: fixed, or estimated with a prior. To estimate a scale instead, write the shock as scale_g*sig_g*eg with @parameters(vol,2) scale_g. One regime is needed per distinct set of scales: quarters whose shocks have different scales need a regime each. The tutorial Estimation/heteroskedastic_shocks estimates such a model with and without its volatile episode.

4.3.4. What the agents know

The decision rules are those of agents who know the regime process and forecast future regimes with the transition probabilities. The known path is the econometrician’s information. The two views coincide when the decision rules do not depend on the expected regimes:

  • At first order, when only standard deviations switch, certainty equivalence makes \(T(s)\) and the steady state the same in every regime and independent of the transition probabilities; only \(R(s)\) scales. Filtering on the path is then exactly the heteroskedastic filter, and the transition probabilities play no role.

  • At first order, when the volatility chain also moves other parameters (a policy coefficient on the same chain, for instance), \(T(s)\) depends on the regime and on the transition probabilities: the filter conditions on agents who anticipate regime changes.

  • At second order and above, the risk terms depend on the regime and on its transition probabilities: a Markov-switching volatility is then a volatility the agents anticipate.

4.3.5. Limits

The filter refuses, with an error identifier:

  • occasionally-binding constraints (rise:knownRegimes:constraints): it is a Kalman filter. With simul_disable_constraints = true the model is filtered as if it had none;

  • anticipated (news) shocks, solve_shock_horizon > 0 (rise:knownRegimes:anticipatedShocks);

  • solve_order > 1 (rise:knownRegimes:order);

  • deterministic exogenous variables (rise:knownRegimes:exogenous);

  • a path of the wrong length or with regimes out of range (rise:knownRegimes:path).

The alternative when another chain must be filtered with probabilities alongside the known one is a transition probability driven by an observed dummy (see @transition_functions): with vol_tp_1_2 = 1/(1+exp(-(g*D-c))) and vol_tp_2_1 = 1/(1+exp(-(c-g*D))), \(D_t = 1\) when period \(t+1\) is volatile and \(g = 2c\) large but finite (for instance \(c = 40\)), the default filter reproduces the known-path likelihood. Its probabilities are never exactly 0 or 1.

4.4. Nonlinear filtering

When a DSGE model is solved with a higher-order perturbation – or otherwise has a nonlinear state-space representation – the regime-switching Kalman/Hamilton filter no longer applies. RISE then uses a sigma-point filter for regime-switching nonlinear state-space models, following Binning and Maih (2015).

4.4.1. When the nonlinear filter is used

filter – and, through it, estimate, forecast with conditioning, the historical decomposition, … – picks the algorithm automatically from the solution:

  • solve_order == 1 – the linear regime-switching Kalman/Hamilton filter (a fast constant-parameter variant when there is a single regime, no occasionally-binding constraint, and no real-time information).

  • solve_order >= 2 – a nonlinear sigma-point filter (the default is the switching divided-difference filter).

Nonlinear filtering with real-time information is not supported (RISE raises an error).

4.4.2. Choosing the filter

Three sigma-point filters are available; the divided-difference filter is the default, the other two are selected by passing them as the filtering algorithm via kf_user_algo (they all share the standard filter signature, so they plug in directly):

% default: switching divided-difference filter
[fkst, LogLik] = filter(m);

% switching unscented Kalman filter
[fkst, LogLik] = filter(m, kf_user_algo = 'switching_unscented_kalman_filter');

% switching cubature Kalman filter
[fkst, LogLik] = filter(m, kf_user_algo = 'switching_cubature_kalman_filter');

The same option can be passed to estimate – it changes which filter computes the likelihood that is maximized.

4.5. Options

The filtering options apply equally to the linear and nonlinear cases (see also help dsge_model/filter):

Option

Effect

kf_filtering_level

0 likelihood only; 1 + filtered (one-step-ahead) series; 2 + updated series; 3 + smoothed series. Default 3.

kf_ergodic

Initialise regime probabilities at the ergodic distribution. Default true.

kf_init_variance

Initial state covariance (scalar = isotropic; matrix = explicit). Use 1 for non-stationary models that lack a finite unconditional variance.

kf_user_init

User-supplied initial state and covariance, for fine control of the initialisation.

kf_presample

Number of initial observations to discard before evaluating the likelihood. Default 0.

kf_householder_chol

Return the Cholesky factor when taking the Householder transformation; primarily used by the divided-difference filter. Default false.

kf_user_algo

The filtering algorithm. '' (default) chooses it from the solution, as described above; a name such as 'imm' (the linear regime-switching filter) or 'switching_unscented_kalman_filter', a function handle or a cell {name, extra-args...} imposes one (see User-defined filter).

kf_riccati_tol

Steady-state switch of the linear filter: once the Kalman gain changes by less than this between two periods, after the last missing observation, the gain and the covariances are frozen (constant-parameter models without constraints, on every route). 0 (default) never switches, so the likelihood is the exact one. Dynare’s riccati_tol is 1e-6: set kf_riccati_tol = 1e-6 to reproduce Dynare’s likelihoods.

simul_disable_constraints

true filters a model with occasionally-binding constraints as if it had none (on a constant-parameter model, the plain Kalman filter). Default false: the filter honors the constraints in its predictions and in its updated series (see Occasionally-binding constraints).

simul_optimize

Solver and limits of the constraint engine, which the filter uses in every period where a constraint binds. '' (default) chooses automatically: a linear solve, repeated from where it lands when a constraint changes branch (the kinks of max and min are piecewise linear), then Newton iterations, then SQP. {'', 'MaxIter', 5000} raises the iteration cap and keeps the automatic choice; a named solver ('linear', 'kkt', 'jacobian', 'fmincon') is used alone.

4.6. User-defined filter

A filter of your own can be supplied through kf_user_algo – a function (or function name, or cell {name, extra-args...}) with signature:

[LogLik, Incr, retcode, Filters] = myFilter(syst, y, U, z, options)

If it needs information beyond what RISE passes, prefix the name with * so RISE first calls it as myFilter(modelObject, struct) to let it fetch what it needs. See Extending RISE.

4.7. The available sigma-point filters

4.7.1. Switching divided-difference filter

The default RISE nonlinear filter. Uses divided-difference approximations to propagate the state moments through the nonlinear transition and observation maps. Robust on regime-switching models where the unscented transform’s sigma points can fall outside the admissible parameter region.

4.7.2. Switching unscented Kalman filter

Uses the unscented transform to propagate sigma points through the nonlinear maps. Faster than the divided-difference filter on well-behaved problems; can be less robust on awkward switching specifications.

4.7.3. Switching cubature Kalman filter

Uses a spherical-radial cubature rule to propagate the state moments. A modern alternative to the unscented Kalman filter with similar accuracy and somewhat better numerical behavior at high state dimension.