MaxLik - Revision history

Rb: /* MATLAB implementation */

2015-10-14T20:46:30Z

‎MATLAB implementation

Rb: /* Literature */

2013-11-14T10:05:57Z

‎Literature

Rb: /* Estimating standard errors of parameter estimates */

2013-11-13T18:48:48Z

‎Estimating standard errors of parameter estimates

Admin: /* Estimating standard errors of parameter estimates */

2013-07-25T12:59:51Z

‎Estimating standard errors of parameter estimates

Admin at 12:56, 26 October 2012

2012-10-26T12:56:41Z

Admin at 12:52, 26 October 2012

2012-10-26T12:52:45Z

Admin at 09:48, 26 October 2012

2012-10-26T09:48:37Z

Admin at 08:54, 26 October 2012

2012-10-26T08:54:16Z

Admin: /* MATLAB implementation */

2012-10-19T17:15:03Z

‎MATLAB implementation

Admin at 17:12, 19 October 2012

2012-10-19T17:12:48Z

@@ Line 100: / Line 100: @@
 == MATLAB implementation ==
-In order to calculate the standard errors for the ML parameter estimates <math>\widehat{\mathbf{\theta}}</math> we therefore require estimates of <math>H(\widehat{\mathbf{\theta}})</math> and <math>J(\widehat{\mathbf{\theta}})</math>. In fact the optimisation routine <source enclose="none">fminunc</source> can return an estimate of the Hessian, <math>H(\widehat{\mathbf{\theta}})</math>. You can consult <source enclose="none">doc fminunc</source> for details on how to have the function return this estimate. Here, however, we prefer a slightly different approach. We will use dedicated functions to calculate <math>H(\widehat{\mathbf{\theta}})</math> (<source enclose="none">HessMp</source>) and <math>J(\widehat{\mathbf{\theta}})</math> (<source enclose="none">gradp</source>). The only input required for these functions are a handle to your objective function and then all the inputs required by that function (the first of which should be a parameter vector). For the parameter vector we will of course use our estimated parameters. The functions will then return estimates for <math>H(\widehat{\mathbf{\theta}})</math> and <math>J(\widehat{\mathbf{\theta}})</math>.
+In order to calculate the standard errors for the ML parameter estimates <math>\widehat{\mathbf{\theta}}</math> we therefore require estimates of <math>H(\widehat{\mathbf{\theta}})</math> and <math>J(\widehat{\mathbf{\theta}})</math>. In fact the optimisation routine <source enclose="none">fminunc</source> can return an estimate of the Hessian, <math>H(\widehat{\mathbf{\theta}})</math>. You can consult <source enclose="none">doc fminunc</source> for details on how to have the function return this estimate. Here, however, we prefer a slightly different approach. We will use dedicated functions to calculate <math>H(\widehat{\mathbf{\theta}})</math> (<source enclose="none">HessMp</source>) and <math>J(\widehat{\mathbf{\theta}})</math> (<source enclose="none">gradp</source>). You can download these functions from [[MaxLikCode]]. The only input required for these functions are a handle to your objective function and then all the inputs required by that function (the first of which should be a parameter vector). For the parameter vector we will of course use our estimated parameters. The functions will then return estimates for <math>H(\widehat{\mathbf{\theta}})</math> and <math>J(\widehat{\mathbf{\theta}})</math>.
 <source>H = HessMp(@nll_lin,thetaopt,datamat,1); % this returns the negative of the Hessian

@@ Line 272: / Line 272: @@
 Martin V., Hurn S. and Harris D. (2012) ''Econometric Modelling with Time Series: Specification, Estimation and Testing (Themes in Modern Econometrics)'', Chapter 3 gives an excellent introduction into nonlinear optimisation strategies. Chapters 1 to 4.
 =Footnotes=
   <references />

@@ Line 92: / Line 92: @@
 <math>V(\widehat{\mathbf{\theta}})=J^{-1}(\widehat{\mathbf{\theta}})=\left(\sum_{t=1}^T \left( \mathbf{g}_t \mathbf{g}_t' \right)\right)^{-1}</math>
-If the estimated model is correctly specified then the OPG and the Hessian approach to calculation <math>V(\widehat{\mathbf{\theta}})</math> will be asymptotically equivalent. If that is not the case then one is in the territory of textitquasi maximum likelihood estimation. Without going into any details we merely note that parameters of the conditional mean function (here <math>\mathbf{\beta}</math>) can still be estimated consistently. Further, the parameter estimate’s standard errors are estimated with the so-called ''sandwich estimator'':
+If the estimated model is correctly specified then the OPG and the Hessian approach to calculation <math>V(\widehat{\mathbf{\theta}})</math> will be asymptotically equivalent. If that is not the case then one is in the territory of quasi maximum likelihood estimation. Without going into any details we merely note that parameters of the conditional mean function (here <math>\mathbf{\beta}</math>) can still be estimated consistently. Further, the parameter estimate’s standard errors are estimated with the so-called ''sandwich estimator'':
 <math>V(\widehat{\mathbf{\theta}})=\left( H(\widehat{\mathbf{\theta}}) J^{-1}(\widehat{\mathbf{\theta}})^{-1} H(\widehat{\mathbf{\theta}}) \right)^{-1}</math>

@@ Line 77: / Line 77: @@
 <math>(\widehat{\mathbf{\theta}}-\mathbf{\theta}_0) \sim N(0,I^{-1}(\mathbf{\theta}_0)).</math>
-where <math>\mathbf{\theta}_0</math> is the unknown population parameter and <math>I^(-1)(\mathbf{\theta}_0)</math> is the information matrix evaluated at the unknown population parameter. Here we do not wish to elaborate on the statistical details of this result<ref>E.g. how different assumptions on the dependence and independence properties require different Laws of Large Numbers and Central Limit Theorems to establish this result
+where <math>\mathbf{\theta}_0</math> is the unknown population parameter and <math>I^{-1}(\mathbf{\theta}_0)</math> is the information matrix evaluated at the unknown population parameter. Here we do not wish to elaborate on the statistical details of this result<ref>E.g. how different assumptions on the dependence and independence properties require different Laws of Large Numbers and Central Limit Theorems to establish this result
 </ref> but rather concentrate on how this result is used to obtain standard errors for estimated parameters.

@@ Line 158: / Line 158: @@
 This will deliver the following type of output
-<source>LR test =   4.09; p-value = 0.1295</source>
+<source>LR test =   1.66; p-value = 0.4369</source>
 in this case indicating that the null hypothesis (restrictions are valid) is not rejected as the p-value is larger than even 0.1.
@@ Line 224: / Line 224: @@
 This will deliver the following type of output
-<source>Wald test =   4.17; p-value = 0.1241 </source>
+<source>Wald test =   1.67; p-value = 0.4339</source>
 where in this particular example the null hypothesis is not rejected as the p-value exceeds and of the usual significance levels. This result is very similar to that presented in the LR test above, as it should be when applied to the same data and hypothesis.
@@ Line 257: / Line 257: @@
 fprintf('LM test = %6.2f; p-value = %6.4f \n', LM, LM_p);</source>
 Two more notes on this code. When the gradient is called you can see that the final input is a &quot;1&quot;. This ensures that we calculate the gradient of the entire (negative) log-likelihood function, as opposed to the gradient at every observation as it was required for the OPG calculation. The variance covariance matrix of the restricted parameter vector has to be estimated here again. In this case this is done with the inverse of the negative Hessian, but any of the above approaches could have been used.
 = Literature =

@@ Line 1: / Line 1: @@
 = Introduction =
@@ Line 239: / Line 235: @@
 <math>LM = G(\mathbf{\theta}_r)' V(\widehat{\mathbf{\theta}}_r) G(\mathbf{\theta}_r)</math>
 = Literature =

@@ Line 1: / Line 1: @@
 = Introduction =
@@ Line 61: / Line 65: @@
 = Calling <source enclose="none">fminunc</source> =
-Calling the optimisation function is exactly as explained in the [[NonlinOptImp|Section]]. I assume that you imported your data and saved them in a vector <source enclose="none">y</source> and a matrix <source enclose="none">x</source> where the latter has three columns, the first being a vector of ones to represent the constant, the second a vector with values for <math>x_1</math> and the third a vector with values for <math>x_2</math>.
+Calling the optimisation function is exactly as explained in the Nonlinear Optimisation [[NonlinOptImp|Section]]. I assume that you imported your data and saved them in a vector <source enclose="none">y</source> and a matrix <source enclose="none">x</source> where the latter has three columns, the first being a vector of ones to represent the constant, the second a vector with values for <math>x_1</math> and the third a vector with values for <math>x_2</math>.
 <source>datamat = [y x];                        % define data matrix for use in nll_lin
@@ Line 137: / Line 141: @@
 === MATLAB implementation ===
-Assume that you estimate a linear model where the unrestricted model estimates a <math>(5 \times 1)</math> dimensional <math>\widehat{\theta}_u</math>. Further you want to test the null hypothesis that the second and third elements (<source enclose="none">b2</source> and <source enclose="none">b3</source> in the code below) of <math>\mathbf{\theta}</math> are equal to 0. The code below assumes that the matrix <source enclose="none">datamat</source> is defined as in the above example but with more explanatory variables <source enclose="none">datamat = [y x]</source>, and that you have already estimated the unrestricted parameter estimate <math>\widehat{\mathbf{\theta}}_u</math>, saved in <source enclose="none">thetaopt</source>.
+Assume that you estimate a linear model where the unrestricted model estimates a <math>(6 \times 1)</math> dimensional <math>\widehat{\theta}_u</math>. Further you want to test the null hypothesis that the second and third elements (<source enclose="none">b(2)</source> and <source enclose="none">b(3)</source> in the code below) of <math>\mathbf{\theta}</math> are equal to 0. The code below assumes that the matrix <source enclose="none">datamat</source> is defined as in the above example but with more explanatory variables <source enclose="none">datamat = [y x]</source>, and that you have already estimated the unrestricted parameter estimate <math>\widehat{\mathbf{\theta}}_u</math>, saved in <source enclose="none">thetaopt</source>.
 <source>% Testing multiple restriction
@@ Line 158: / Line 162: @@
 This will deliver the following type of output
-<source>LR test = 145.28; p-value = 0.0000 </source>
+<source>LR test =   4.09; p-value = 0.1295</source>
-in this case indicating that the null hypothesis (restrictions are valid) is clearly rejected as the p-value is virtually close to 0.
+in this case indicating that the null hypothesis (restrictions are valid) is not rejected as the p-value is larger than even 0.1.
 == Wald Test ==
@@ Line 199: / Line 203: @@
 <math>W = (\mathbf{R} \widehat{\mathbf{\theta}}_u - \mathbf{b})' (\mathbf{R} V(\widehat{\mathbf{\theta}}_u) \mathbf{R}')^{-1} (\mathbf{R} \widehat{\mathbf{\theta}}_u - \mathbf{b})</math>
-<math>V(\widehat{\mathbf{\theta}}_u))</math> is the variance-covariance matrix of the estimated parameter vector and can be estimated in any of the ways discussed earlier. The test statistic is distributed as <math>\chi^2</math> distributed with <math>r</math> degrees of freedom. If you have nonlinear restrictions to be tested then the basic structure of the test statistic remains the same but that <math>(r \times 1)</math> term <math>(\mathbf{R} \widehat{\mathbf{\theta}}_u - \mathbf{b})</math> will be replaced by <math>C(\widehat{\mathbf{\theta}}_u)</math> and <math>\mathbf{R}</math> will be replaced by <math>\partial C(\widehat{\mathbf{\theta}}_u)/partial \widehat{\mathbf{\theta}}_u</math>:
+<math>V(\widehat{\mathbf{\theta}}_u))</math> is the variance-covariance matrix of the estimated parameter vector and can be estimated in any of the ways discussed earlier. The test statistic is distributed as <math>\chi^2</math> distributed with <math>r</math> degrees of freedom. If you have nonlinear restrictions to be tested then the basic structure of the test statistic remains the same but that <math>(r \times 1)</math> term <math>(\mathbf{R} \widehat{\mathbf{\theta}}_u - \mathbf{b})</math> will be replaced by <math>C(\widehat{\mathbf{\theta}}_u)</math> and <math>\mathbf{R}</math> will be replaced by <math>\partial C(\widehat{\mathbf{\theta}}_u)/\partial \widehat{\mathbf{\theta}}_u</math>:
 <math>W = C(\widehat{\mathbf{\theta}}_u)' \left( \left(\frac{\partial C(\widehat{\mathbf{\theta}}_u)}{\partial \widehat{\mathbf{\theta}}_u}\right) V(\widehat{\mathbf{\theta}}_u) \left( \frac{\partial C(\widehat{\mathbf{\theta}}_u)}{\partial \widehat{\mathbf{\theta}}_u} \right)'\right)^{-1} C(\widehat{\mathbf{\theta}}_u)</math>
 The differentiable function <math>C</math> is defined such that <math>C(\widehat{\mathbf{\theta}}_u)=0</math> and encompasses the nonlinear restrictions.
 == Lagrange Multiplier (LM) Test ==

@@ Line 118: / Line 118: @@
 If you do this for the above linear model you should of course find that the ML standard errors are fairly similar to the OLS standard errors. The advantage is that the ML procedure is available when OLS is not (you merely need to change the objective function to reflect, say, a nonlinear model). The Hessian and Gradient functions are downloadable from here: [[media:HessMp.m|HessMp.m]] and [[media:gradp.m|gradp.m]]. These functions are from [http://www.hec.unil.ch/matlabcodes/econometrics.html : Eric Jondeau and Michael Rockinger webpage with MATLAB codes].
-If you were to perform hypotheses tests on individual parameters you could do so with a standard t-test strategy. You should just be aware that here we only know the distribution of a t-test asymptotically, and that is a standard normal distribution.
+If you were to perform hypotheses tests on individual parameters you could do so with a standard t-test strategy. You should just be aware that here we only know the distribution of a t-test asymptotically, and that this distribution is a standard normal distribution.
 = Testing in a ML framework =

@@ Line 147: / Line 147: @@
 theta0_r = [mean(y); 0; 0; std(y)];       % this sets the initial parameter vector
-[theta_r] = fminsearch(@nll_lin,theta0_r,options,datamat(:,[1 2 5 6]),1);
+[theta_r] = fminunc(@nll_lin,theta0_r,options,datamat(:,[1 2 5 6]),1);
 L_u = -nll_lin(thetaopt,datamat,1);             % calculates unrestricted logLikelihood

@@ Line 137: / Line 137: @@
 === MATLAB implementation ===
-Assume that you estimate a linear model where the unrestricted model estimates a <math>(5 \times 1)</math> dimensional <math>\widehat{\theta}_u</math>. Further you want to test the null hypothesis that the second and third elements (<source enclose="none">b2</source> and <source enclose="none">b3</source> in the code below) of <math>\mathbf{\theta}</math> are equal to 0. The code below assumes that you have already estimated the unrestricted parameter estimate <math>\widehat{\mathbf{\theta}}_u</math>, saved in <source enclose="none">thetaopt</source>.
+Assume that you estimate a linear model where the unrestricted model estimates a <math>(5 \times 1)</math> dimensional <math>\widehat{\theta}_u</math>. Further you want to test the null hypothesis that the second and third elements (<source enclose="none">b2</source> and <source enclose="none">b3</source> in the code below) of <math>\mathbf{\theta}</math> are equal to 0. The code below assumes that the matrix <source enclose="none">datamat</source> is defined as in the above example but with more explanatory variables <source enclose="none">datamat = [y x]</source>, and that you have already estimated the unrestricted parameter estimate <math>\widehat{\mathbf{\theta}}_u</math>, saved in <source enclose="none">thetaopt</source>.
 <source>% Testing multiple restriction
@@ Line 144: / Line 144: @@
 % LR test
 % estimate the restricted model
 theta0_r = [mean(y); 0; 0; std(y)];       % this sets the initial parameter vector
-        % do not hand in x1 and x2
 [theta_r] = fminsearch(@nll_lin,theta0_r,options,datamat(:,[1 2 5 6]),1);
@@ Line 155: / Line 156: @@
 fprintf('LR test = %6.2f; p-value = %6.4f \n', LR, LR_p);</source>
 == Wald Test ==