Math 6010 • Linear Models

(20 points) Consider the linear model
\[
Y = \mathbf{X}\beta + \epsilon, \qquad \epsilon \sim \mathcal{N}_{n}(0, \sigma^{2}I_{n}),
\]
where \(\mathbf{X}\) is an \(n \times p\) design matrix with \(n > p\) and \(\operatorname{rank}(\mathbf{X}) = p\). Let
\[
H = \mathbf{X}(\mathbf{X}^{\top}\mathbf{X})^{-1}\mathbf{X}^{\top}
\]
denote the hat matrix and \(\hat{Y} = HY\) the ordinary least squares (OLS) fitted values.
\begin{enumerate}
\item Show that \(H\) is a projection matrix by verifying that it is symmetric and idempotent.
\item Let \(e = Y - \hat{Y}\) be the residual vector. Show that \(\hat{Y} \perp e\), i.e. \(\hat{Y}\) is orthogonal to \(e\).
\item (Ridge regression) For \(\lambda > 0\), ridge regression fitted values are
\[
\hat{Y}^{\mathrm{RR}} = S_{\lambda}Y, \qquad S_{\lambda} = \mathbf{X}(\mathbf{X}^{\top}\mathbf{X} + \lambda I_{p})^{-1}\mathbf{X}^{\top}.
\]

Let \(\mathbf{X} = UDV^{\top}\) denote the singular value decomposition of \(\mathbf{X}\), where \(U \in \mathbb{R}^{n \times p}\) and \(V \in \mathbb{R}^{p \times p}\)
have orthonormal columns, and
\[
D = \operatorname{diag}(d_{1}, \ldots, d_{p}), \qquad d_{j} > 0.
\]

Recall that the ridge regression fitting operator can be written as
\[
S_{\lambda} = \sum_{j=1}^{p} \frac{d_{j}^{2}}{d_{j}^{2} + \lambda} u_{j}u_{j}^{\top},
\]
where \(u_{j}\) denotes the \(j\)th column of \(U\).
\begin{enumerate}
\item Is \(S_{\lambda}\) a projection matrix? Briefly justify your answer.
\item Describe how the eigenvalues of \(S_{\lambda}\) depend on \(\lambda\), and what happens as \(\lambda \to 0\) and as
\(\lambda \to \infty\).
\item In term of bias and/or variance, explain the advantage of ridge regression in comparison to OLS.
\end{enumerate}
\end{enumerate}