Analysis of Variance

(20 points) Researchers compare three classroom settings and measure a standardized reading score. The observed scores are: \[ \begin {aligned} \text {Group 1: }&18, 21, 20 &&(n_{1} = 3),\\ \text {Group 2: }&23, 22, 24, 21 &&(n_{2} = 4),\\ \text {Group 3: }&26, 28, 27, 29, 25 &&(n_{3} = 5). \end {aligned} \]

Suppose the reading score for the \(i\)th individual in group \(j\) is modeled as \[ Y_{ij} = \mu _{j} + \epsilon _{ij}, \qquad \epsilon _{ij} \sim \mathcal {N}(0, \sigma ^{2}), \] with all errors independent.

Equivalently, this can be written in matrix form as \[ Y = \mathbf {X}\beta + \epsilon . \]

1. Write the model in matrix form by explicitly constructing the design matrix \(\mathbf {X}\), the parameter vector \(\beta \), and the response vector \(Y\).
2. Express the null hypothesis “all three group means are equal” in the form \[ H_{0}: A\beta = c, \] for explicit matrices \(A\) and \(c\).
3. You are told that the residual sum of squares under the full model is \[ RSS = 38.4 \] and under the reduced model (i.e. under \(H_{0}\)) it is \[ RSS_{H_{0}} = 62.1. \] Compute the \(F\)-statistic for testing \(H_{0}\). State the degrees of freedom and give the conclusion at level \(\alpha = 0.05\).
4. Could a \(t\)-test be used instead of an \(F\)-test here? Briefly explain.

Consider the density function

where \(C\) is the constant \[ C = \int _{-\infty }^{\infty } e^{-t^{4}}\,dt. \]

Consider the linear model

where the \(\epsilon _{i}\) are iid with pdf \(f(t)\). Assume that \(\beta \in \mathbb {R}^{d}\) and \(\sigma ^{2} > 0\) are deterministic but unknown, while \(\mathbf {x}_{i} \in \mathbb {R}^{d}\) are deterministic and known.

1. Write down the likelihood function of the observations as a function of \(\beta \) (do not try to find the maximum likelihood estimator).
2. Assume temporarily that the \(\epsilon _{i}\) are iid normal with mean zero and variance one, and write down the likelihood function of the observations as a function of \(\beta \) in that case.
3. If you were to solve for the maximum likelihood estimators in the first two parts would they be the same? (Note you are not being asked to actually solve for the estimators).

Let \(A_1,A_2,\ldots \), be a sequence of events (not necessarily independent) such that \[ \sum _{n=1}^{\infty }\mathbb {P}(A_n)=\infty \qquad \text {and}\qquad \mathbb {P}(A_n\cap A_m)\leq \mathbb {P}(A_n)\mathbb {P}(A_m) \quad \text {for } m\ne n. \] Prove that \(\mathbb {P}(A_n\text { i.o.})=1\) or provide a counterexample. Hint: Consider the mean and variance of \[ \sum _{i=1}^{n}\mathbf {1}_{A_i}. \]

Let \(X\) and \(Y\) be any two random variables. Suppose \(\mathbb {E}[X^2]<\infty \). The conditional variance of \(X\) given \(Y\) is defined to be \[ \operatorname {Var}(X\mid Y)=\mathbb {E}\bigl [(X-\mathbb {E}[X\mid Y])^2\mid Y\bigr ]. \] Prove the conditional variance formula: \[ \operatorname {Var}(X)=\mathbb {E}[\operatorname {Var}(X\mid Y)]+\operatorname {Var}(\mathbb {E}[X\mid Y]). \]

(Pólya’s Urn) An urn initially contains \(r\) red and \(b\) blue balls. A ball is chosen uniformly at random (i.e. with probability \(1/(r+b)\) each). If it comes up red (resp. blue), then it is returned and another red (resp. blue) ball is added to the urn. The process is repeated indefinitely. Let \(R_n\) be the number of red balls in the urn after \(n\) draws.

1. A vector \((X_1,X_2,\ldots ,X_n)\) of random variables is said to be exchangeable if \[ (X_1,\ldots ,X_n)\sim (X_{\pi (1)},\ldots ,X_{\pi (n)}) \qquad \text {for all permutations }\pi , \] where \(\sim \) denotes equality in distribution. If \(X_i\) is the indicator that a red ball is drawn from the urn at time \(i\), prove that \((X_1,\ldots ,X_n)\) is exchangeable.
2. Find the mean and variance of \(S_n=X_1+\cdots +X_n\), the total number of red balls added to the urn up to time \(n\).

Let \((X_n)\) be i.i.d. random variables with mean \(\mu \).

1. Prove that \[ \lim _{n\to \infty } \frac {1}{\log n}\sum _{i=1}^{n}\frac {X_i}{i} = \mu \qquad \text {almost surely.} \]
2. If the random variables also have a finite variance \(\sigma ^2<\infty \), then find \(a_n\) such that \[ a_n\left (\sum _{i=1}^{n}\frac {X_i}{i}-\mu \log n\right ) \] has a weak limit. Identify this weak limit. (You need to prove the weak convergence, once you have identified \(a_n\) and the limiting distribution.)

Let \(X_1,X_2,\ldots ,X_n\) be iid random variables with density \[ h(t;\theta )=\theta t^{\theta -1}I\{0\leq t\leq 1\}, \qquad \theta >0. \] Find the maximum likelihood estimator of \(\theta \) and compute the asymptotic variance of the maximum likelihood estimator.

Let \(X_1,X_2,\ldots ,X_n\) be iid with common pdf \[ f(x;\theta )=\frac {\theta }{(1+x)^{\theta +1}}, \qquad 0<x<\infty ,\ \theta >0. \] What is a lower bound for the variance of an unbiased estimator of \(\theta \)?

Let \(X_1,X_2,\ldots ,X_n\) be iid with common pdf \[ f(x;\theta )=\frac {1}{2\theta }e^{-|x|/\theta }, \qquad -\infty <x<\infty , \] with \(\theta \) unknown but positive. Compute the Cramer-Rao lower bound for the variance of an unbiased estimator of \(\theta \), and extend your calculation to finding the UMVUE by finding a linear combination of \[ \sum _{i=1}^n \frac {\partial }{\partial \theta }\log f(X_i;\theta ) \] that is unbiased.

Let \(X_1,X_2,\ldots ,X_{25}\) denote a random sample of size 25 from a normal distribution with mean \(\theta \) and variance 100. Find a uniformly most powerful critical region of size \(\alpha =0.10\) for testing \(H_0:\theta =75\) against \(H_1:\theta >75\).

(a) Find the distribution of \(\int _0^1 B_t\,dt\), where \((B_t)\) is Brownian motion.

(b) Given an i.i.d. sequence \(X_1,X_2,\ldots \) with mean 0 and variance 1, what does Donsker’s invariance principle tell us about this sequence in connection with the distribution of \(\int _0^1 B_t\,dt\)?

Let \(X_1,\ldots ,X_{2n+1}\) be independent lognormal\((\mu ,\sigma ^2)\). That is, \(X_1=e^{Y_1},\ldots ,X_{2n+1}=e^{Y_{2n+1}}\) with \(Y_1,\ldots ,Y_{2n+1}\) being independent \(N(\mu ,\sigma ^2)\). Use a theorem to show that the sample median \(X_{n+1:2n+1}\) is asymptotically normal and find its asymptotic mean and asymptotic variance.

Let \(X_1,X_2,\ldots \) be i.i.d. with mean \(\mu \) and finite variance. Show that \[ U_n:=\binom {n}{2}^{-1}\sum _{1\leq i<j\leq n}X_iX_j \] converges in probability to \(\mu ^2\) as \(n\to \infty \).

Let \(X\) be a random variable with values in an interval \(I\), and suppose that \(f\) and \(g\) are nondecreasing functions on \(I\) such that \(f(X)\) and \(g(X)\) have finite variance. Show that \(\operatorname {Cov}(f(X),g(X))\geq 0\).

Hint: If \(X_1\) and \(X_2\) are i.i.d. as \(X\), then \((f(X_1)-f(X_2))(g(X_1)-g(X_2))\geq 0\).

Suppose \(X_1\) and \(X_2\) are independent random variables that satisfy the following three properties: (i) \(Y_1:=(X_1+X_2)/\sqrt {2}\) is standard normal; (ii) \(Y_2:=(X_1-X_2)/\sqrt {2}\) is standard normal; and (iii) \(Y_1\) and \(Y_2\) are independent. Prove that \(X_1\) and \(X_2\) are normally distributed. Can you compute their respective means and variances?

Let \(X_1,\ldots ,X_n\) be independent \(\operatorname {LogNormal}(\mu ,\sigma ^2)\) (i.e., \(X_1=e^{Y_1},\ldots ,X_n=e^{Y_n}\) with \(Y_1,\ldots ,Y_n\) being independent \(N(\mu ,\sigma ^2)\)). Argue that the sample median is asymptotically normal and find the asymptotic mean and variance.

Let \(X_1,\ldots ,X_n\) be a random sample from \(N(\theta ,\theta )\), where \(\theta >0\). In particular, the population mean and the population variance are equal but unknown.

(a) Find the method-of-moments estimator of \(\theta \) based on the second sample moment.

(b) Find the maximum likelihood estimator of \(\theta \).

Let \(X_1,X_2,\ldots \) be i.i.d. with mean \(\mu \) and variance \(\sigma ^2\), and put \(S_0:=0\) and \(S_n:=X_1+\cdots +X_n\) for each \(n\geq 1\). Let \(N\) be a nonnegative-integer-valued random variable independent of \(X_1,X_2,\ldots \). Find the mean and variance of \(S_N\) by conditioning on \(N\).

Let \(X_i\) be a random sample of size \(n\) with pdf \(f(x;\theta )=\theta x^{\theta -1}\) (\(0<x<1\), \(\theta >0\).)

4a) (5 points) Find a complete and sufficient statistic.

4b) (5 points) Find a uniformly minimum variance unbiased estimator (UMVUE) of \(1/\theta \).

Suppose \(X_1,X_2,\ldots \) have common mean \(\mu \) and variances \(\sigma _1^2,\sigma _2^2,\ldots \). Prove that if \(\sup _n\sigma _n^2<\infty \) and \(\lim _{|i-j|\to \infty }\mathbb {E}[X_iX_j]=0\), then \((X_1+\cdots +X_n)/n\) converges to \(\mu \) in probability.

Let \(X_1,X_2,\ldots ,X_n\) and \(Y_1,Y_2,\ldots ,Y_m\) be independent, identically distributed random variables with a common continuous distribution function. Compute the expected value and the variance of \[ Z=\sum _{1\leq i\leq n}\sum _{1\leq j\leq n} I\{X_i\leq Y_j\}. \]

where \(I\{A\}\) denotes the indicator function of the event \(A\).

Let \(X_1,X_2,X_3,\ldots \) be i.i.d. with finite mean and variance, and put \(S_n=X_1+X_2+\cdots +X_n\). Show that \(S_n-n\mathbb {E}[X_1]\) and \((S_n-n\mathbb {E}[X_1])^2-n\operatorname {Var}(X_1)\) are martingales.

Let \(X_1,X_2,\ldots \) be i.i.d. mean 0, variance \(\sigma ^2\in (0,\infty )\).

(a) Use the central limit theorem as well as Kolmogorov’s 0-1 law to conclude that \(\limsup _{n\to \infty }S_n/\sqrt n=\infty \) a.s.

(b) Show that \(S_n/\sqrt n\) does not converge in probability.