Forecasting - Revision history

Rb at 17:55, 5 November 2013

2013-11-05T17:55:22Z

Rb at 15:40, 5 November 2013

2013-11-05T15:40:32Z

Rb at 14:47, 5 November 2013

2013-11-05T14:47:17Z

Show changes

Rb: /* Literature */

2013-11-05T14:27:12Z

‎Literature

Show changes

Rb at 14:17, 5 November 2013

2013-11-05T14:17:03Z

Rb at 12:05, 5 November 2013

2013-11-05T12:05:23Z

Rb at 12:03, 5 November 2013

2013-11-05T12:03:11Z

Rb: /* Comparing MSE */

2013-11-05T11:01:40Z

‎Comparing MSE

Rb at 11:01, 5 November 2013

2013-11-05T11:01:00Z

Rb: /* Literature */

2013-11-05T10:58:26Z

‎Literature

@@ Line 135: / Line 135: @@
 The most common summary measures of forecast performance are the following
-<math>\textbf{Forecast Bias}: bias_i = \sum_{\tau} u_{i,\tau}\\
+<math>\textbf{Mean Forecast Bias}: bias_i = \frac{1}{P} \sum_{\tau} u_{i,\tau}\\
-\textbf{Mean Square Error}: MSE_i = \sum_{\tau} u_{i,\tau}^2\\
+\textbf{Mean Square Error}: MSE_i = \frac{1}{P}  \sum_{\tau} u_{i,\tau}^2\\
-\textbf{Mean Absolute Error}: MAE_i = \sum_{\tau} |u_{i,\tau}|\\</math>
+\textbf{Mean Absolute Error}: MAE_i = \frac{1}{P}  \sum_{\tau} |u_{i,\tau}|\\</math>
-Here the summations are over all forecast periods <math>\tau</math>. When these are used we would argue that that smaller measures (in absolute terms for the bias measure) are to be preferred. What these measures cannot tell us is whether the differences in these measures, between different models, are statistically significant.
+Here the summations are over all <math>P</math> forecast periods <math>\tau</math>. When these are used we would argue that that smaller measures (in absolute terms for the bias measure) are to be preferred. What these measures cannot tell us is whether the differences in these measures, between different models, are statistically significant.
 == Comparing Different Forecasts ==
-We can compare two different types of approaches to comparing two or more forecast models. Clark and McCracken ([[[
+We can compare two different types of approaches to comparing two or more forecast models. Clark and McCracken ([http://research.stlouisfed.org/wp/2011/2011-025.pdf 2011]) describe the traditional approach as the ''population level'' approach and distinguish this approach from the ''finite sample inference''. The key issue that the different approaches address is the fact that model parameters are estimated on the basis of finite samples. The second approach, ''finite sample inference'', accepts these parameters estimated as they are, and asks whether, given these estimated parameters, different models deliver significantly different inference. The former, the ''population level'' approach, however, uses the same sample evidence, produced with parameters estimated on the basis of finite samples, but tests hypotheses based on true model parameters . This implies that the resulting test statistics will have to take account of the variation in the test statistic introduced through the parameter estimation process.
 In what follows we will point out which tests belongs to which category.
@@ Line 192: / Line 190: @@
 <math>u_{A,\tau} = \alpha (u_{A,\tau}-u_{B,\tau}) + \epsilon_{\tau}</math>
-in which the null hypothesis of Model A encompassing Model B is represented by <math>H_0: \alpha = 0</math>. The <math>ENC-t</math> test statistic is then essentially equivalent to the t-test on <math>\alpha</math>. It is again important that the standard error to <math>\hat{\alpha}</math> is estimated using robust methods, such as Newey-West standard errors.
+in which the null hypothesis of Model A encompassing Model B is represented by <math>H_0: \alpha = 0</math> (<math>H_A: \alpha > )</math>, one-sided!). The <math>ENC-t</math> test statistic is then essentially equivalent to the t-test on <math>\alpha</math>. It is again important that the standard error to <math>\hat{\alpha}</math> is estimated using robust methods, such as Newey-West standard errors.
 So far so good, but as it turns out, the asymptotic distribution of this test statistic will, as for <math>MSE-t</math>, depend on the forecasting scheme (recursive, rolling fixed) and the relation between models A and B. Therefore, a bootstrap methodology is again called for.

@@ Line 171: / Line 171: @@
 As it turns out, the test statistic
-<math>DM_{AB} = \frac{\bar{d}_{AB}}{\hat{\sigma}_{\bar{d}_{AB}}}</math>
+<math>MSE-t_{AB} = \frac{\bar{d}_{AB}}{\hat{\sigma}_{\bar{d}_{AB}}}</math>
-is exactly the same. To be precise, in this context one should condition <math>\bar{d}_{AB}(\hat{\theta_A},\hat{\theta_B})</math> on the estimated model parameters <math>\theta_A</math> and <math>\theta_B</math>. And the hypothesis tested is <math>H_0:MSE_A(\theta_A)=MSE_B(\theta_B)</math>, stating that the models have equal predictive ability under the true (but unknown) model parameters.
+is exactly the same<ref>The test stat has a different label to indicate that the underlying assumptions are different to those for the <math>DM</math>-test. The name <math>MSE-t</math> is that used in the Clark and McCracken (2011) paper.
 As it turns out, this difference to the DM setup, although it may sound minor or technical, wreaks havoc when it comes to deriving the asymptotic distribution of the test statistic. Under the simple assumption made in DM the test statistic was asymptotically standard normally distributed. Now the test’s distribution under the null hypothesis of equal predictive ability depends on a number of features, such as whether the models are nested or not and which forecasting scheme has been used.
-As it turns out the distributions are such that in general one has to employ bootstrapping techniques to establish correct critical values.
+As it turns out the distributions are such that in general one has to employ bootstrapping techniques to establish correct critical values (see Clark and McCracken, 2011, for details).
 = Literature =

@@ Line 165: / Line 165: @@
 where <math>\hat{\alpha}</math> will be equal to |d<sub>AB</sub> and the estimated regression standard error will be an estimate for <math>\hat{\sigma}_{\bar{d}_{AB}}</math>. If you want to allow for heteroskedasticity and/or autocorrelation you should employ an OLS routine that calculates Newey-West standard errors (for example the one discussed [[ExampleCodeOLShac|here]]. It is then obvious that the t-statistic for the constant term is equivalent to <math>DM_{AB}</math>.
-=== Comparing MSE ===
+=== Comparing MSE when using estimated models ===
-The following tests belong to the category of ''population level'' tests. Here we compare forecasts from two models, A and B, and decide whether the null hypothesis <math>H_0:MSE_A=MSE_B</math> for true, but unknown population parameters can be rejected. The
+The following tests belong to the category of ''population level'' tests. Here we compare forecasts from two models, A and B, and decide whether the null hypothesis <math>H_0:MSE_A=MSE_B</math> for true, but unknown population parameters can be rejected. The difference to the DM test discussed in the previous sub-section is that we now allow the forecasts to come from estimated models and hence we need to allow for parameter uncertainty to come into the equation.
 = Literature =
@@ Line 175: / Line 175: @@
 Clark, T.E. and McCracken, M.W. (2011) Advances in Forecast Evaluation, Working Paper [[[
 http://research.stlouisfed.org/wp/2011/2011-025.pdf|
-http://research.stlouisfed.org/wp/2011/2011-025.pdf]]|2011-025B]
+http://research.stlouisfed.org/wp/2011/2011-025.pdf]] 2011-025B]
 A significant part of this paper buids on a previous survey paper:
@@ Line 181: / Line 181: @@
 West, K.W. (2006) Forecast Evaluation, in: Handbook of Economic Forecasting, Volume 1, edited by G. Elliott, C. W.J. Granger, A. G. Timmermann
-A review of the uses and abuses of the Diebold-Mariano Test has recently been provided by the man himself, [ http://www.ssc.upenn.edu/~fdiebold/papers/paper113/Diebold_DM%20Test.pdf|Francis Diebold].
+A review of the uses and abuses of the Diebold-Mariano Test has recently been provided by the man himself, [ http://www.ssc.upenn.edu/~fdiebold/papers/paper113/Diebold_DM%20Test.pdf Francis Diebold].
 =Footnotes=
   <references />

@@ Line 155: / Line 155: @@
 The hypothesis the DM test is designed to test is <math>H_0: MSE_A = MSE_B</math>, where we refer to two Forecast series, A and B. At the core of the test statistic id what is called the ''loss differential'', <math>d_{AB,\tau}=L(u_{A,\tau})-L(u_{B,\tau})</math>. If we used a quadratic loss function this would be <math>d_{AB\tau}=u_{A,\tau}^2-u_{B,\tau}^2</math>. The test statistic is then
-<math>DM_{AB} = frac{\bar{d}_{AB}}{\hat{\sigma}_{\bar{d}_{AB}}}</math>
+<math>DM_{AB} = \frac{\bar{d}_{AB}}{\hat{\sigma}_{\bar{d}_{AB}}}</math>
-where <math>\bar{d}_{AB}=frac{1}{P}\sum_{\tau} d_{AB,\tau}</math>, assuming that we have <math>P</math> forecast periods and <math>\hat{\sigma}_{\bar{d}_{AB}}</math> is a consistent estimate of <math>\bar{d}_{AB}</math>s standard deviation. If the series of <math>d_{AB,\tau}</math> is covariance stationary, then, under the null hypothesis of equal predictive ability it is easy to show that asymptotically <math>DM_{AB}</math> is standard normally distributed.
+where <math>\bar{d}_{AB}=\frac{1}{P}\sum_{\tau} d_{AB,\tau}</math>, assuming that we have <math>P</math> forecast periods and <math>\hat{\sigma}_{\bar{d}_{AB}}</math> is a consistent estimate of <math>\bar{d}_{AB}</math>s standard deviation. If the series of <math>d_{AB,\tau}</math> is covariance stationary, then, under the null hypothesis of equal predictive ability it is easy to show that asymptotically <math>DM_{AB}</math> is standard normally distributed.
 The only complication here is that <math>\hat{\sigma}_{\bar{d}_{AB}}</math> needs to be estimated consistently, for instance allowing for autocorrelation and heteroskedasticity. The simplest way of achieving this is to use a regression framework. You merely need to estimate an OLS regression
@@ Line 163: / Line 163: @@
 <math>d_{AB,\tau} = \alpha + \epsilon_{\tau}</math>
-where <math>\hat{\alpha}</math> will be equal to |d<sub>AB</sub> and the estimated regression standard error will be an estimate for <math>\hat{\sigma}_{\bar{d}_{AB}}</math>. If you want to allow for heteroskedasticity and/or autocorrelation you should employ an OLS routine that calculates Newey-West standard errors (for example the one discussed [[ExampleCodeOLShac#OLShac.m here]]. It is then obvious that the t-statistic for the constant term is equivalent to <math>DM_{AB}</math>.
+where <math>\hat{\alpha}</math> will be equal to |d<sub>AB</sub> and the estimated regression standard error will be an estimate for <math>\hat{\sigma}_{\bar{d}_{AB}}</math>. If you want to allow for heteroskedasticity and/or autocorrelation you should employ an OLS routine that calculates Newey-West standard errors (for example the one discussed [[ExampleCodeOLShac|here]]. It is then obvious that the t-statistic for the constant term is equivalent to <math>DM_{AB}</math>.
 === Comparing MSE ===

@@ Line 97: / Line 97: @@
      count = count+1;        % increase forecast counter by 1
 end</source>
-The difference to the recursive scheme is that the size of.
+The difference to the recursive scheme is that the size of the estimation window remains constant. If you compare the two code snippets you can see that the difference in the code is rather minimal.
 = Model Setup =
@@ Line 121: / Line 121: @@
 \textbf{Model C}: y_{t} = \alpha_0 + \alpha_1 * z_t + u_{Ct}</math>
-These models are in general different, unless the following parameter restrictions are valid: <math>\beta_1=0</math> in Model A and <math>\alpha_1=0</math> in Model B. If both these restrictions hold, the two models will deliver identical results. Such models are called overlapping. Comparing models of this sort is equally complicated.
+These models are in general different, unless the following parameter restrictions are valid, <math>\beta_1=0</math> in Model A and <math>\alpha_1=0</math> in Model B. If both these restrictions hold, the two models will deliver identical results. Such models are called overlapping. Comparing models of this sort is equally complicated.
 = Forecast Evaluation =
@@ Line 139: / Line 139: @@
 \textbf{Mean Absolute Error}: MAE_i = \sum_{\tau} |u_{i,\tau}|\\</math>
-When these are used we would argue that that smaller measures (in absolute terms for the bias measure) are to be preferred. What these measures cannot tell us is whether the differences in these measures, between different models, are statistically significant.
+Here the summations are over all forecast periods <math>\tau</math>. When these are used we would argue that that smaller measures (in absolute terms for the bias measure) are to be preferred. What these measures cannot tell us is whether the differences in these measures, between different models, are statistically significant.
 == Comparing Different Forecasts ==
-We can compare two different types of approaches to comparing two or more forecast models. Clark and McCracken ([http://research.stlouisfed.org/wp/2011/2011-025.pdf| 2011]) describe the traditional approach as the ''population level'' approach and distinguishes this approach from the ''finite sample inference''. The key issue that the different approaches address is the fact that model parameters are estimated on the basis of finite samples. The second approach, ''finite sample inference'', accepts these parameters estimated as they are, and asks whether, given these estimated parameters, different models deliver significantly different inference. The former, the ''population level'' approach, however, uses the same sample evidence, produced with parameters estimated on the basis of finite samples, but tests hypotheses based on true model parameters . This implies that the resulting test statistics will have to take account of the variation in the test statistic introduced through the parameter estimation process.
+We can compare two different types of approaches to comparing two or more forecast models. Clark and McCracken ([[[
 In what follows we will point out which tests belongs to which category.
 === Comparing MSE ===
-The following tests belong to the category of ''population level'' tests. Here we compare forecasts from two models, A and B, and decide whether the null hypothesis <math>H_0:MSE_A=MSE_B</math> for true, but unknown population parameters can be rejected.
+The following tests belong to the category of ''population level'' tests. Here we compare forecasts from two models, A and B, and decide whether the null hypothesis <math>H_0:MSE_A=MSE_B</math> for true, but unknown population parameters can be rejected. The
 = Literature =
@@ Line 157: / Line 173: @@
 An excellent overview of the issues at hand is given in
-Clark, T.E. and McCracken, M.W. (2011) Advances in Forecast Evaluation, Working Paper 2011-025B [http://research.stlouisfed.org/wp/2011/2011-025.pdf|2011-025B]
+Clark, T.E. and McCracken, M.W. (2011) Advances in Forecast Evaluation, Working Paper [[[
 A significant part of this paper buids on a previous survey paper:
@@ Line 163: / Line 181: @@
 West, K.W. (2006) Forecast Evaluation, in: Handbook of Economic Forecasting, Volume 1, edited by G. Elliott, C. W.J. Granger, A. G. Timmermann
+A review of the uses and abuses of the Diebold-Mariano Test has recently been provided by the man himself, [ http://www.ssc.upenn.edu/~fdiebold/papers/paper113/Diebold_DM%20Test.pdf|Francis Diebold].
 =Footnotes=
   <references />

← Older revision		Revision as of 11:01, 5 November 2013
Line 150:		Line 150:

	The following tests belong to the category of ''population level'' tests. Here we compare forecasts from two models, A and B, and decide whether the null hypothesis <math>H_0:MSE_A=MSE_B</math> for true, but unknown population parameters can be rejected.		The following tests belong to the category of ''population level'' tests. Here we compare forecasts from two models, A and B, and decide whether the null hypothesis <math>H_0:MSE_A=MSE_B</math> for true, but unknown population parameters can be rejected.
		+
		+	This Section is not complete yet.

	= Literature =		= Literature =