Generalized Least Squares

(20 points) Consider the linear model \[ Y = \mathbf {X}\beta + \epsilon , \qquad \epsilon \sim \mathcal {N}_{n}(0, \sigma ^{2}I_{n}), \] where \(\mathbf {X}\) is an \(n \times p\) design matrix with \(n > p\) and \(\operatorname {rank}(\mathbf {X}) = p\). Let \[ H = \mathbf {X}(\mathbf {X}^{\top }\mathbf {X})^{-1}\mathbf {X}^{\top } \] denote the hat matrix and \(\hat {Y} = HY\) the ordinary least squares (OLS) fitted values.

1. Show that \(H\) is a projection matrix by verifying that it is symmetric and idempotent.
2. Let \(e = Y - \hat {Y}\) be the residual vector. Show that \(\hat {Y} \perp e\), i.e. \(\hat {Y}\) is orthogonal to \(e\).

3. (Ridge regression) For \(\lambda > 0\), ridge regression fitted values are \[ \hat {Y}^{\mathrm {RR}} = S_{\lambda }Y, \qquad S_{\lambda } = \mathbf {X}(\mathbf {X}^{\top }\mathbf {X} + \lambda I_{p})^{-1}\mathbf {X}^{\top }. \]

   Let \(\mathbf {X} = UDV^{\top }\) denote the singular value decomposition of \(\mathbf {X}\), where \(U \in \mathbb {R}^{n \times p}\) and \(V \in \mathbb {R}^{p \times p}\) have orthonormal columns, and \[ D = \operatorname {diag}(d_{1}, \ldots , d_{p}), \qquad d_{j} > 0. \]

   Recall that the ridge regression fitting operator can be written as \[ S_{\lambda } = \sum _{j=1}^{p} \frac {d_{j}^{2}}{d_{j}^{2} + \lambda } u_{j}u_{j}^{\top }, \] where \(u_{j}\) denotes the \(j\)th column of \(U\).

   1. Is \(S_{\lambda }\) a projection matrix? Briefly justify your answer.
   2. Describe how the eigenvalues of \(S_{\lambda }\) depend on \(\lambda \), and what happens as \(\lambda \to 0\) and as \(\lambda \to \infty \).
   3. In term of bias and/or variance, explain the advantage of ridge regression in comparison to OLS.

(20 points) We observe the following six data points: \[ (x_{i}, y_{i}) \in \{(0, 3), (1, 2), (2, 4), (3, 5), (4, 7), (5, 8)\}, \qquad i = 1, \ldots , 6. \]

We fit a quadratic spline with knots at \(x = 2\) and \(x = 4\) using the truncated power basis \[ 1,\qquad x,\qquad x^{2},\qquad (x - 2)^{2}_{+},\qquad (x - 4)^{2}_{+}, \] where \((x - a)^{2}_{+} = (x - a)^{2}\) if \(x > a\) and \(0\) otherwise.

1. Express the data in the linear model form \[ Y = \mathbf {X}\beta + \epsilon , \] clearly specifying the design matrix \(\mathbf {X}\) corresponding to the above basis functions.
2. How many degrees of freedom does this spline model have? Justify your answer both in terms of the number of parameters and the number of constraints for this model.
3. Explain how you would compute the least squares estimator \(\hat {\beta }\) for this model. You do not need to evaluate the expression.
4. Let \[ f(x) = \beta _{0} + \beta _{1}x + \beta _{2}x^{2} + \beta _{3}(x - 2)^{2}_{+} + \beta _{4}(x - 4)^{2}_{+} \] denote a quadratic spline with knots at \(x = 2\) and \(x = 4\). Show that \(f(x)\) and \(f'(x)\) are both continuous at \(x = 2\) and \(x = 4\).

A scale has two pans. The measurement given by the scale is the difference between the weights on pan 1 and pan 2 plus a random error. Thus, if a weight \(\mu _1\) is put on pan 1 and a weight \(\mu _2\) is put on pan 2, then the measurement is \(Y=\mu _1-\mu _2+\epsilon \). Suppose that \(\mathbb {E}[\epsilon ]=0\), \(\operatorname {Var}(\epsilon )=\sigma ^2\), and in repeated use of the scale observations \(Y_i\) are independent.

Suppose two objects, #1 and #2, have weights \(\beta _1\) and \(\beta _2\). Measurements are then taken as follows:

• Object #1 is put on pan 1 and nothing on pan 2
• Object #2 is put on pan 2 and nothing on pan 1
• Object #1 is put on pan 1 and object #2 is put on pan 2
• Objects #1 and #2 are both put on pan 1 and nothing on pan 2

(a) Let \(\mathbf {Y}=(Y_1,Y_2,Y_3,Y_4)^\prime \) be the vector of observations. Formulate this as a linear model.

(b) Find the least squares estimators of \(\beta _1\) and \(\beta _2\).

(c) Find the variances of the least squares estimators as well as the covariance between them.

Derive, using only first principles, the least-squares estimators of the slope and the intercept of a linear regression problem. What can you say about the optimality properties of those estimators?

Consider the simple linear regression model \(Y_i=\beta _0+\beta _1x_i+\epsilon _i\) (\(i=1,2,\ldots ,n\)). Assume independent \(N(0,\sigma ^2)\) errors.

(a) Find the least squares estimators \(\hat \beta _0\) of \(\hat \beta _1\) of \(\beta _0\) and \(\beta _1\).

(b) Calculate the covariance matrix of \((\hat \beta _0,\hat \beta _1)\).

Let \(y_i=\alpha x_i+\epsilon _i\), where \(\epsilon _i\), \(1\leq i\leq n\), are independent, identically distributed normal \(N(0,\sigma ^2)\) random variables.

(a) Find the maximum likelihood estimators for \(\alpha \) and \(\sigma ^2\).

(b) Find the distributions of the maximum likelihood estimators.

(c) Compute the least-squares estimator for \(\alpha \).

Consider the linear model: \(Y_i=\beta +\epsilon _i\), \(1\le i\le n\). Here, \(\epsilon _i\)’s are i.i.d. \(N(0,\sigma ^2)\), and \(\sigma \) and \(\beta \) are unknown.

(a) Find the least squares estimator of \(\beta \);

(b) Find the UMVU estimator for \(\beta \) (prove the optimality);

(c) Find the UMP test for \(H_0:\beta =\beta _0\) versus \(H_1:\beta =\beta _1\);

(d) Discuss–without proofs–how to find the UMP test for \(H_0:\beta >\beta _0\) vs. \(H_1:\beta \le \beta _0\) from tests of the form in part (c) above.

Let \(Y_i=\alpha ^{i/n}+\epsilon _i\), \(1\le i\le n\). We assume that \(\epsilon _1,\epsilon _2,\ldots ,\epsilon _n\) are independent, identically distributed random variables with \(\E \epsilon _i=0\), \(0<\sigma ^2=\operatorname {Var}\epsilon _i<\infty \) and \(\E \epsilon _i^4<\infty \).

(a) Find the least-squares estimator for \(\alpha \).

(b) Show that the estimator is asymptotically normal.

(c) Find an estimator for \(\sigma ^2\).