Ridge Regression

(20 points) Consider the linear model \[ Y = \mathbf {X}\beta + \epsilon , \qquad \epsilon \sim \mathcal {N}_{n}(0, \sigma ^{2}I_{n}), \] where \(\mathbf {X}\) is an \(n \times p\) design matrix with \(n > p\) and \(\operatorname {rank}(\mathbf {X}) = p\). Let \[ H = \mathbf {X}(\mathbf {X}^{\top }\mathbf {X})^{-1}\mathbf {X}^{\top } \] denote the hat matrix and \(\hat {Y} = HY\) the ordinary least squares (OLS) fitted values.

1. Show that \(H\) is a projection matrix by verifying that it is symmetric and idempotent.
2. Let \(e = Y - \hat {Y}\) be the residual vector. Show that \(\hat {Y} \perp e\), i.e. \(\hat {Y}\) is orthogonal to \(e\).

3. (Ridge regression) For \(\lambda > 0\), ridge regression fitted values are \[ \hat {Y}^{\mathrm {RR}} = S_{\lambda }Y, \qquad S_{\lambda } = \mathbf {X}(\mathbf {X}^{\top }\mathbf {X} + \lambda I_{p})^{-1}\mathbf {X}^{\top }. \]

   Let \(\mathbf {X} = UDV^{\top }\) denote the singular value decomposition of \(\mathbf {X}\), where \(U \in \mathbb {R}^{n \times p}\) and \(V \in \mathbb {R}^{p \times p}\) have orthonormal columns, and \[ D = \operatorname {diag}(d_{1}, \ldots , d_{p}), \qquad d_{j} > 0. \]

   Recall that the ridge regression fitting operator can be written as \[ S_{\lambda } = \sum _{j=1}^{p} \frac {d_{j}^{2}}{d_{j}^{2} + \lambda } u_{j}u_{j}^{\top }, \] where \(u_{j}\) denotes the \(j\)th column of \(U\).

   1. Is \(S_{\lambda }\) a projection matrix? Briefly justify your answer.
   2. Describe how the eigenvalues of \(S_{\lambda }\) depend on \(\lambda \), and what happens as \(\lambda \to 0\) and as \(\lambda \to \infty \).
   3. In term of bias and/or variance, explain the advantage of ridge regression in comparison to OLS.