Linear Models

(20 points) We observe the following six data points: \[ (x_{i}, y_{i}) \in \{(0, 3), (1, 2), (2, 4), (3, 5), (4, 7), (5, 8)\}, \qquad i = 1, \ldots , 6. \]

We fit a quadratic spline with knots at \(x = 2\) and \(x = 4\) using the truncated power basis \[ 1,\qquad x,\qquad x^{2},\qquad (x - 2)^{2}_{+},\qquad (x - 4)^{2}_{+}, \] where \((x - a)^{2}_{+} = (x - a)^{2}\) if \(x > a\) and \(0\) otherwise.

1. Express the data in the linear model form \[ Y = \mathbf {X}\beta + \epsilon , \] clearly specifying the design matrix \(\mathbf {X}\) corresponding to the above basis functions.
2. How many degrees of freedom does this spline model have? Justify your answer both in terms of the number of parameters and the number of constraints for this model.
3. Explain how you would compute the least squares estimator \(\hat {\beta }\) for this model. You do not need to evaluate the expression.
4. Let \[ f(x) = \beta _{0} + \beta _{1}x + \beta _{2}x^{2} + \beta _{3}(x - 2)^{2}_{+} + \beta _{4}(x - 4)^{2}_{+} \] denote a quadratic spline with knots at \(x = 2\) and \(x = 4\). Show that \(f(x)\) and \(f'(x)\) are both continuous at \(x = 2\) and \(x = 4\).

Consider the linear model

where the \(\epsilon _{i}\) are independent normals with \(\mathbb {E}\epsilon _{i} = 0\) and \(\mathbb {E}\epsilon _{i}^{2} = i\sigma ^{2}\). Here \(\beta \in \mathbb {R}^{d}\) and \(\sigma ^{2} > 0\) are unknown parameters, while the \(\mathbf {x}_{i} \in \mathbb {R}^{d}\) are all known and deterministic. Compute the maximum likelihood estimator of \((\beta , \sigma ^{2} )\).

Consider the linear model

where the \(\epsilon _{i}\) are independent normal random variables with \(\mathbb {E}\epsilon _{i} = 0\) and \(\mathbb {E}\epsilon _{i}^{2} = \sigma ^{2}\). Assume \(\beta \) and \(\sigma ^{2} > 0\) are deterministic but unknown, while the \(x_{i}\) are deterministic and known. Assume also that \(\sum _i x_i \ne 0\). Find the likelihood ratio test for \(\beta = 0\) versus

Consider the density function

where \(C\) is the constant \[ C = \int _{-\infty }^{\infty } e^{-t^{4}}\,dt. \]

Consider the linear model

where the \(\epsilon _{i}\) are iid with pdf \(f(t)\). Assume that \(\beta \in \mathbb {R}^{d}\) and \(\sigma ^{2} > 0\) are deterministic but unknown, while \(\mathbf {x}_{i} \in \mathbb {R}^{d}\) are deterministic and known.

1. Write down the likelihood function of the observations as a function of \(\beta \) (do not try to find the maximum likelihood estimator).
2. Assume temporarily that the \(\epsilon _{i}\) are iid normal with mean zero and variance one, and write down the likelihood function of the observations as a function of \(\beta \) in that case.
3. If you were to solve for the maximum likelihood estimators in the first two parts would they be the same? (Note you are not being asked to actually solve for the estimators).

A scale has two pans. The measurement given by the scale is the difference between the weights on pan 1 and pan 2 plus a random error. Thus, if a weight \(\mu _1\) is put on pan 1 and a weight \(\mu _2\) is put on pan 2, then the measurement is \(Y=\mu _1-\mu _2+\epsilon \). Suppose that \(\mathbb {E}[\epsilon ]=0\), \(\operatorname {Var}(\epsilon )=\sigma ^2\), and in repeated use of the scale observations \(Y_i\) are independent.

Suppose two objects, #1 and #2, have weights \(\beta _1\) and \(\beta _2\). Measurements are then taken as follows:

• Object #1 is put on pan 1 and nothing on pan 2
• Object #2 is put on pan 2 and nothing on pan 1
• Object #1 is put on pan 1 and object #2 is put on pan 2
• Objects #1 and #2 are both put on pan 1 and nothing on pan 2

(a) Let \(\mathbf {Y}=(Y_1,Y_2,Y_3,Y_4)^\prime \) be the vector of observations. Formulate this as a linear model.

(b) Find the least squares estimators of \(\beta _1\) and \(\beta _2\).

(c) Find the variances of the least squares estimators as well as the covariance between them.

Consider the simple linear regression model \(Y_i=\beta _0+\beta _1x_i+\epsilon _i\) (\(i=1,2,\ldots ,n\)). Assume independent \(N(0,\sigma ^2)\) errors.

(a) Find the least squares estimators \(\hat \beta _0\) of \(\hat \beta _1\) of \(\beta _0\) and \(\beta _1\).

(b) Calculate the covariance matrix of \((\hat \beta _0,\hat \beta _1)\).

Consider the linear model: \(Y_i=\beta +\epsilon _i\), \(1\le i\le n\). Here, \(\epsilon _i\)’s are i.i.d. \(N(0,\sigma ^2)\), and \(\sigma \) and \(\beta \) are unknown.

(a) Find the least squares estimator of \(\beta \);

(b) Find the UMVU estimator for \(\beta \) (prove the optimality);

(c) Find the UMP test for \(H_0:\beta =\beta _0\) versus \(H_1:\beta =\beta _1\);

(d) Discuss–without proofs–how to find the UMP test for \(H_0:\beta >\beta _0\) vs. \(H_1:\beta \le \beta _0\) from tests of the form in part (c) above.

Consider the linear model: \[ Y_{ij}=\beta _i+\epsilon _{ij},\qquad 1\le i\le 2,\qquad 1\le j\le J. \] Suppose \(\epsilon _{ij}\)’s are independent and for some (known) \(a_1\) and \(a_2\), \(\epsilon _{ij}\sim N(0,a_i^2\sigma ^2)\).

(a) Find the U.M.V.U.E.’s of \(\beta _1\) and \(\beta _2\).

(b) Suppose you know that for some unknown \(\beta \), \(\beta _1=\beta _2\). Find the U.M.V.U.E. of \(\beta \).