Linear Models Qualifying Exam (MATH 6010)

(20 points) Let \(Z = (Z_{1}, Z_{2}, Z_{3})^{\top } \sim \mathcal {N}(0, I_{3})\) and define the random vector \[ X = \begin {pmatrix} X_{1} \\ X_{2} \end {pmatrix} = \begin {pmatrix} Z_{1} + Z_{2} \\ Z_{2} + Z_{3} \end {pmatrix}. \]

1. Compute the mean vector \(\mathbb {E}[X]\) and the covariance matrix \(\Sigma = \operatorname {Cov}[X]\).
2. Does \(X\) have a probability density function on \(\mathbb {R}^{2}\)? Briefly justify your answer.
3. Does the joint moment generating function satisfy \[ M_{X}(t_{1}, t_{2}) = M_{X}(t_{1}, 0)M_{X}(0, t_{2})? \] Explain your reasoning.
4. Let \(Q = \|X\|^{2}\). Identify a symmetric matrix \(A\) such that \(Q = Z^{\top }AZ\) and compute \(\mathbb {E}[Q]\).

(20 points) Let \[ v = (1, -1, 0)^{\top }, \qquad \Omega = \operatorname {Span}\{v\} \subseteq \mathbb {R}^{3}. \]

1. Write the projection matrix \(P_{\Omega }\) onto \(\Omega \).
2. Verify that \(P_{\Omega }\) is a projection matrix by checking that it is symmetric and idempotent. (To avoid a long calculation, you can write out these calculations in terms of \(v\).)
3. Determine the eigenvalues of \(P_{\Omega }\) and their multiplicities. What is \(\operatorname {rank}(P_{\Omega })\)?
4. Find an orthogonal basis \(\{b_{1}, b_{2}\}\) for \(\Omega ^{\perp }\) and verify that \(P_{\Omega }b_{i} = 0\) for \(i = 1, 2\).

(20 points) Researchers compare three classroom settings and measure a standardized reading score. The observed scores are: \[ \begin {aligned} \text {Group 1: }&18, 21, 20 &&(n_{1} = 3),\\ \text {Group 2: }&23, 22, 24, 21 &&(n_{2} = 4),\\ \text {Group 3: }&26, 28, 27, 29, 25 &&(n_{3} = 5). \end {aligned} \]

Suppose the reading score for the \(i\)th individual in group \(j\) is modeled as \[ Y_{ij} = \mu _{j} + \epsilon _{ij}, \qquad \epsilon _{ij} \sim \mathcal {N}(0, \sigma ^{2}), \] with all errors independent.

Equivalently, this can be written in matrix form as \[ Y = \mathbf {X}\beta + \epsilon . \]

1. Write the model in matrix form by explicitly constructing the design matrix \(\mathbf {X}\), the parameter vector \(\beta \), and the response vector \(Y\).
2. Express the null hypothesis “all three group means are equal” in the form \[ H_{0}: A\beta = c, \] for explicit matrices \(A\) and \(c\).
3. You are told that the residual sum of squares under the full model is \[ RSS = 38.4 \] and under the reduced model (i.e. under \(H_{0}\)) it is \[ RSS_{H_{0}} = 62.1. \] Compute the \(F\)-statistic for testing \(H_{0}\). State the degrees of freedom and give the conclusion at level \(\alpha = 0.05\).
4. Could a \(t\)-test be used instead of an \(F\)-test here? Briefly explain.

(20 points) Researchers study how a child’s reading score \(Y\) depends on home reading time \(x_{1}\) (hours/week) and classroom support level \(x_{2}\) (a small integer score). It is believed that extra home reading time is more effective when classroom support is higher, so the mean is modeled with an interaction between \(x_{1}\) and \(x_{2}\). To make the intercept interpretable, home reading time is centered at \(1\) hour/week.

Consider the linear model \[ Y_{i} = \beta _{0} + \beta _{1}(x_{1,i} - 1) + \beta _{2}(x_{1,i} \times x_{2,i}) + \epsilon _{i}, \qquad i = 1, \ldots , 5, \] where \(\epsilon _{i} \sim \mathcal {N}(0, \sigma ^{2})\) are independent. The observed data consist of: \[ (x_{1}, x_{2}, Y) = (0, 1, 4), (1, 0, 3), (2, 1, 7), (1, 2, 6), (3, 1, 9). \]

A least squares fit yields \(\hat {\beta } = (\hat {\beta }_{0}, \hat {\beta }_{1}, \hat {\beta }_{2})^{\top } = (2.10, 0.90, 1.20)^{\top }\) and \(RSS = 1.50\).

1. Write down the design matrix \(\mathbf {X}\) for this regression model and the parameter vector \(\beta = (\beta _{0}, \beta _{1}, \beta _{2})^{\top }\). Identify the estimated variance \(S^{2}\).
2. We wish to test the joint null hypothesis that there is no effect of the predictors beyond the baseline, \[ H_{0}: \beta _{1} = \beta _{2} = 0. \] Express this hypothesis in matrix form \(A\beta = 0\) for an appropriate matrix \(A\), and state \(q = \operatorname {rank}(A)\).
3. Write down the \(F\)-statistic for testing \(H_{0}\) and state the distribution of \(F\) under \(H_{0}\). (You do not need to evaluate the formula.)
4. Using Scheffé’s method, construct simultaneous 95% confidence intervals for \(\beta _{0}, \beta _{1}, \beta _{2}\). You may express these intervals in terms of \((\mathbf {X}^{\top }\mathbf {X})^{-1}\), but plug in all other quantities.

(20 points) Consider the linear model \[ Y = \mathbf {X}\beta + \epsilon , \qquad \epsilon \sim \mathcal {N}_{n}(0, \sigma ^{2}I_{n}), \] where \(\mathbf {X}\) is an \(n \times p\) design matrix with \(n > p\) and \(\operatorname {rank}(\mathbf {X}) = p\). Let \[ H = \mathbf {X}(\mathbf {X}^{\top }\mathbf {X})^{-1}\mathbf {X}^{\top } \] denote the hat matrix and \(\hat {Y} = HY\) the ordinary least squares (OLS) fitted values.

1. Show that \(H\) is a projection matrix by verifying that it is symmetric and idempotent.
2. Let \(e = Y - \hat {Y}\) be the residual vector. Show that \(\hat {Y} \perp e\), i.e. \(\hat {Y}\) is orthogonal to \(e\).

3. (Ridge regression) For \(\lambda > 0\), ridge regression fitted values are \[ \hat {Y}^{\mathrm {RR}} = S_{\lambda }Y, \qquad S_{\lambda } = \mathbf {X}(\mathbf {X}^{\top }\mathbf {X} + \lambda I_{p})^{-1}\mathbf {X}^{\top }. \]

   Let \(\mathbf {X} = UDV^{\top }\) denote the singular value decomposition of \(\mathbf {X}\), where \(U \in \mathbb {R}^{n \times p}\) and \(V \in \mathbb {R}^{p \times p}\) have orthonormal columns, and \[ D = \operatorname {diag}(d_{1}, \ldots , d_{p}), \qquad d_{j} > 0. \]

   Recall that the ridge regression fitting operator can be written as \[ S_{\lambda } = \sum _{j=1}^{p} \frac {d_{j}^{2}}{d_{j}^{2} + \lambda } u_{j}u_{j}^{\top }, \] where \(u_{j}\) denotes the \(j\)th column of \(U\).

   1. Is \(S_{\lambda }\) a projection matrix? Briefly justify your answer.
   2. Describe how the eigenvalues of \(S_{\lambda }\) depend on \(\lambda \), and what happens as \(\lambda \to 0\) and as \(\lambda \to \infty \).
   3. In term of bias and/or variance, explain the advantage of ridge regression in comparison to OLS.

(20 points) We observe the following six data points: \[ (x_{i}, y_{i}) \in \{(0, 3), (1, 2), (2, 4), (3, 5), (4, 7), (5, 8)\}, \qquad i = 1, \ldots , 6. \]

We fit a quadratic spline with knots at \(x = 2\) and \(x = 4\) using the truncated power basis \[ 1,\qquad x,\qquad x^{2},\qquad (x - 2)^{2}_{+},\qquad (x - 4)^{2}_{+}, \] where \((x - a)^{2}_{+} = (x - a)^{2}\) if \(x > a\) and \(0\) otherwise.

1. Express the data in the linear model form \[ Y = \mathbf {X}\beta + \epsilon , \] clearly specifying the design matrix \(\mathbf {X}\) corresponding to the above basis functions.
2. How many degrees of freedom does this spline model have? Justify your answer both in terms of the number of parameters and the number of constraints for this model.
3. Explain how you would compute the least squares estimator \(\hat {\beta }\) for this model. You do not need to evaluate the expression.
4. Let \[ f(x) = \beta _{0} + \beta _{1}x + \beta _{2}x^{2} + \beta _{3}(x - 2)^{2}_{+} + \beta _{4}(x - 4)^{2}_{+} \] denote a quadratic spline with knots at \(x = 2\) and \(x = 4\). Show that \(f(x)\) and \(f'(x)\) are both continuous at \(x = 2\) and \(x = 4\).