Orthogonal Projections

(20 points) Let \[ v = (1, -1, 0)^{\top }, \qquad \Omega = \operatorname {Span}\{v\} \subseteq \mathbb {R}^{3}. \]

1. Write the projection matrix \(P_{\Omega }\) onto \(\Omega \).
2. Verify that \(P_{\Omega }\) is a projection matrix by checking that it is symmetric and idempotent. (To avoid a long calculation, you can write out these calculations in terms of \(v\).)
3. Determine the eigenvalues of \(P_{\Omega }\) and their multiplicities. What is \(\operatorname {rank}(P_{\Omega })\)?
4. Find an orthogonal basis \(\{b_{1}, b_{2}\}\) for \(\Omega ^{\perp }\) and verify that \(P_{\Omega }b_{i} = 0\) for \(i = 1, 2\).

(20 points) Consider the linear model \[ Y = \mathbf {X}\beta + \epsilon , \qquad \epsilon \sim \mathcal {N}_{n}(0, \sigma ^{2}I_{n}), \] where \(\mathbf {X}\) is an \(n \times p\) design matrix with \(n > p\) and \(\operatorname {rank}(\mathbf {X}) = p\). Let \[ H = \mathbf {X}(\mathbf {X}^{\top }\mathbf {X})^{-1}\mathbf {X}^{\top } \] denote the hat matrix and \(\hat {Y} = HY\) the ordinary least squares (OLS) fitted values.

1. Show that \(H\) is a projection matrix by verifying that it is symmetric and idempotent.
2. Let \(e = Y - \hat {Y}\) be the residual vector. Show that \(\hat {Y} \perp e\), i.e. \(\hat {Y}\) is orthogonal to \(e\).

3. (Ridge regression) For \(\lambda > 0\), ridge regression fitted values are \[ \hat {Y}^{\mathrm {RR}} = S_{\lambda }Y, \qquad S_{\lambda } = \mathbf {X}(\mathbf {X}^{\top }\mathbf {X} + \lambda I_{p})^{-1}\mathbf {X}^{\top }. \]

   Let \(\mathbf {X} = UDV^{\top }\) denote the singular value decomposition of \(\mathbf {X}\), where \(U \in \mathbb {R}^{n \times p}\) and \(V \in \mathbb {R}^{p \times p}\) have orthonormal columns, and \[ D = \operatorname {diag}(d_{1}, \ldots , d_{p}), \qquad d_{j} > 0. \]

   Recall that the ridge regression fitting operator can be written as \[ S_{\lambda } = \sum _{j=1}^{p} \frac {d_{j}^{2}}{d_{j}^{2} + \lambda } u_{j}u_{j}^{\top }, \] where \(u_{j}\) denotes the \(j\)th column of \(U\).

   1. Is \(S_{\lambda }\) a projection matrix? Briefly justify your answer.
   2. Describe how the eigenvalues of \(S_{\lambda }\) depend on \(\lambda \), and what happens as \(\lambda \to 0\) and as \(\lambda \to \infty \).
   3. In term of bias and/or variance, explain the advantage of ridge regression in comparison to OLS.

Let \((\Omega ,\mathcal {F},P)\) be a probability space, let \(\mathcal {G}\) be a sub-\(\sigma \)-field of \(\mathcal {F}\), and let \(M\) be the closed subspace of \(L^2(\Omega ,\mathcal {F},P)\) consisting of the \(\mathcal {G}\)-measurable functions in \(L^2\). Show that \(T(X)=E[X\mid \mathcal {G}]\) coincides with the orthogonal projection of \(L^2(\Omega ,\mathcal {F},P)\) onto \(M\).

Let \((\Omega ,\mathcal F,P)\) be a probability space, let \(\mathcal G\) be a sub-\(\sigma \)-field of \(\mathcal F\), and let \(M\) be the closed subspace of \(L^2(\Omega ,\mathcal F,P)\) consisting of the \(\mathcal G\)-measurable functions in \(L^2\). Show that \(T(X)=\mathbb {E}[X\mid \mathcal G]\) coincides with the orthogonal projection of \(L^2(\Omega ,\mathcal F,P)\) onto \(M\).