Estimator Distributions

(20 points) We observe the following six data points: \[ (x_{i}, y_{i}) \in \{(0, 3), (1, 2), (2, 4), (3, 5), (4, 7), (5, 8)\}, \qquad i = 1, \ldots , 6. \]

We fit a quadratic spline with knots at \(x = 2\) and \(x = 4\) using the truncated power basis \[ 1,\qquad x,\qquad x^{2},\qquad (x - 2)^{2}_{+},\qquad (x - 4)^{2}_{+}, \] where \((x - a)^{2}_{+} = (x - a)^{2}\) if \(x > a\) and \(0\) otherwise.

1. Express the data in the linear model form \[ Y = \mathbf {X}\beta + \epsilon , \] clearly specifying the design matrix \(\mathbf {X}\) corresponding to the above basis functions.
2. How many degrees of freedom does this spline model have? Justify your answer both in terms of the number of parameters and the number of constraints for this model.
3. Explain how you would compute the least squares estimator \(\hat {\beta }\) for this model. You do not need to evaluate the expression.
4. Let \[ f(x) = \beta _{0} + \beta _{1}x + \beta _{2}x^{2} + \beta _{3}(x - 2)^{2}_{+} + \beta _{4}(x - 4)^{2}_{+} \] denote a quadratic spline with knots at \(x = 2\) and \(x = 4\). Show that \(f(x)\) and \(f'(x)\) are both continuous at \(x = 2\) and \(x = 4\).

Let \(\epsilon _{1}, \ldots , \epsilon _{n}\) and \(\eta _{1}, \ldots , \eta _{m}\) be independent normal random variables with \(\mathbb {E}\epsilon _{i} = \mathbb {E}\eta _{j} = 0\) and \(\mathbb {E}\epsilon _{i}^{2} = 1, \mathbb {E}\eta _{j}^{2} = 4\). We have the models

where the \(x_{i}, y_{i}\) are given and non-random.

1. Find the maximum likelihood estimator for \((\beta _{1}, \beta _{2}, \beta _{3} )\).
2. Compute the distribution of the estimator.

Consider the linear model

where the \(\epsilon _{i}\) are independent normals with \(\mathbb {E}\epsilon _{i} = 0\) and \(\mathbb {E}\epsilon _{i}^{2} = i\sigma ^{2}\). Here \(\beta \in \mathbb {R}^{d}\) and \(\sigma ^{2} > 0\) are unknown parameters, while the \(\mathbf {x}_{i} \in \mathbb {R}^{d}\) are all known and deterministic. Compute the maximum likelihood estimator of \((\beta , \sigma ^{2} )\).

Let \((X_{1}, X_{2})\) be independent normal random variables with \(\mathbb {E}X_{1} = \mathbb {E}X_{2} = 0\) and \(\mathbb {E}X_{1}^{2} =\) \(1, \mathbb {E}X_{2}^{2} = 4\). Define \(Y_{1} = X_{1} + X_{2}\) and \(Y_{2} = X_{1} + 2X_{2}\) and let \(\mathbf {Y} = (Y_{1}, Y_{2})^{\top }\).

1. Compute the covariance matrix of \(\mathbf {Y}\).
2. Find a matrix \(\mathbf {D}\) such that \(\mathbf {Y}^{\top }\mathbf {D}\mathbf {Y}\) has a chi-squared distribution.

Consider the density function

where \(C\) is the constant \[ C = \int _{-\infty }^{\infty } e^{-t^{4}}\,dt. \]

Consider the linear model

where the \(\epsilon _{i}\) are iid with pdf \(f(t)\). Assume that \(\beta \in \mathbb {R}^{d}\) and \(\sigma ^{2} > 0\) are deterministic but unknown, while \(\mathbf {x}_{i} \in \mathbb {R}^{d}\) are deterministic and known.

1. Write down the likelihood function of the observations as a function of \(\beta \) (do not try to find the maximum likelihood estimator).
2. Assume temporarily that the \(\epsilon _{i}\) are iid normal with mean zero and variance one, and write down the likelihood function of the observations as a function of \(\beta \) in that case.
3. If you were to solve for the maximum likelihood estimators in the first two parts would they be the same? (Note you are not being asked to actually solve for the estimators).

Let \(U_1,U_2,\ldots ,U_n\) be independent and identically distributed random variables, uniform on \([0,1]\). Let \(U_{1,n}\leq U_{2,n}\leq \cdots \leq U_{n,n}\) be the order statistics. Show that \(nU_{3,n}\) converges in distribution and compute the limit.

Let \(X_1,X_2,\ldots ,X_n\) be independent and identically distributed exponential\((\lambda )\) random variables, meaning that their pdf is \[ f(t)= \begin {cases} 0, & \text {if } -\infty <t<0,\\ \dfrac {1}{\lambda }e^{-t/\lambda }, & \text {if } 0\leq t<\infty . \end {cases} \] Find the uniformly minimum variance unbiased estimator for \(\tau =\lambda ^5\).

Let \(X_1,X_2,\ldots ,X_n\) be independent and identically distributed random variables with density function \[ f(t)= \begin {cases} 0, & \text {if } -\infty <t<\theta ,\\ \dfrac {2\theta }{(\theta +t)^2}, & \text {if } \theta \leq t<\infty , \end {cases} \] for \(\theta >0\) a parameter. Find the maximum likelihood estimator for \(\theta \).

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with density function \[ f(t,\theta )= \begin {cases} 0, & \text {if } t<0,\\ \dfrac {1}{\theta }e^{-t/\theta }, & \text {if } t\geq 0. \end {cases} \] Let \(\alpha >0\) and \[ \tau =\frac {1}{\theta ^\alpha }. \]

(a) How big \(n\) should be (as a function of \(\alpha \)) so that \(\tau \) has uniformly minimum variance unbiased estimator?

(b) Find the uniformly minimum variance unbiased estimator for \(\tau \) (you need to justify that your estimator has the required property).

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with density function \[ f(t,\theta )= \begin {cases} 0, & \text {if } t\notin [-\theta ,\theta ],\\ \dfrac {5t^4}{2\theta ^5}, & \text {if } -\theta \leq t\leq \theta . \end {cases} \] Find the maximum likelihood estimator for \(\theta \) and compute its bias.

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with density function \[ f(t,\theta )= \begin {cases} 0, & \text {if } t\notin [-\theta ,\theta ],\\ \dfrac {10t^9}{2\theta ^{10}}, & \text {if } -\theta \leq t\leq \theta . \end {cases} \] Find a moment estimator for \(\theta \).

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with density function \[ f(t\theta )=\frac {1}{2}e^{-|t-\theta |}, \qquad -\infty <t<\infty . \] The parameter \(\theta \) could be any real number. Find all maximum likelihood estimators for \(\theta \).

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables that have the uniform distribution on \([0,1]\). Let \(X_{1,n}\leq X_{2,n}\leq \cdots \leq X_{n,n}\) be the order statistics. Let \(k\geq 1\) be a fixed integer. Show that \[ Y_n=nX_{k,n} \] converges in distribution and compute the limiting distribution.

Let \(\{X_{i,n}:i,n\geq 1\}\) be i.i.d. random variables with mass function \(\{f(x):x=0,1,2,\ldots \}\). Fix \(z\in \{1,2,\ldots \}\). Define random variables \((Z_n)\) as follows: \[ Z_0=z \qquad \text {and}\qquad Z_n=\sum _{i=1}^{Z_{n-1}}X_{i,n} \qquad \text {for } n\geq 1. \] I.e. \(Z_n\) is a branching process with offspring distribution \(f\) and initial population \(z\). For \(s\in [0,1]\) let \(g(s)=\sum _{x=0}^{\infty }f(x)s^x\). Suppose that \(s_0\in (0,1)\) solves \(g(s)=s\). Show that then \(s_0^{Z_n}\) is a martingale in the filtration \(\mathcal {F}_n=\sigma (X_{i,m}:i\geq 1,m\leq n)\). Use this to conclude that \[ \mathbb {P}(Z_n=0\text { for some }n\geq 0)=s_0^z. \]

Let \(X_n:\Omega \to \mathbb {R}\) be a sequence of integrable random variables with the same distribution. Prove that \(X_n/n\to 0\) almost surely. Hint: Borel-Cantelli.

(Pólya’s Urn) An urn initially contains \(r\) red and \(b\) blue balls. A ball is chosen uniformly at random (i.e. with probability \(1/(r+b)\) each). If If it comes up red (resp. blue), then it is returned and another red (resp. blue) ball is added to the urn. The process is repeated indefinitely. Let \(R_n\) be the number of red balls in the urn after \(n\) draws.

a) Prove that the fraction of red balls \(M_n=R_n/(r+b+n)\) has an almost sure limit \(M_\infty \) as \(n\to \infty \).

b) Suppose now \(r=b=1\). Prove that \(\mathbb {P}(R_n=k)=1/(n+1)\) for \(1\leq k\leq n+1\). Hint: Write down the probability of choosing \(k\) red and \(n-k\) blue balls in some fixed order.

c) Still assuming \(r=b=1\), what is the distribution of \(M_\infty \)? Hint: use part b) to compute \(\mathbb {P}(M_\infty \leq x)\).

Let \(g:\mathbb {R}\to \mathbb {R}\) be bounded and continuous and fix \(\lambda >0\). Use properties of the Poisson distribution to show that \[ \sum _{k=0}^{\infty }g(k/n)\frac {(n\lambda )^k}{k!}e^{-n\lambda } \to g(\lambda ) \qquad \text {as } n\to \infty . \]

Let \(X_1,X_2\) be iid with density \[ f(x;\theta )= \begin {cases} 3x^2/\theta ^3, & 0<x<\theta ,\\ 0, & \text {otherwise.} \end {cases} \] Find a complete, sufficient statistic for \(\theta \in [0,\infty )\) based on \(X_1\) and \(X_2\).

Let \(X_1,X_2,\ldots ,X_n\) be iid random variables with a Beta\((\theta ,\theta )\) distribution, i.e. their density is \[ f(x;\theta )=\frac {\Gamma (2\theta )}{\Gamma (\theta )^2} x^{\theta -1}(1-x)^{\theta -1}, \qquad 0<x<1. \] Show that a best critical region for testing \(H_0:\theta =1\) versus \(H_A:\theta =2\) is \[ \left \{(x_1,x_2,\ldots ,x_n): c\leq \prod _{i=1}^n x_i\right \}. \]

Let \(X_1,X_2,X_3\) be iid and distributed according to \(f(x)=2x\), \(0<x<1\). Let \(X_{(1)}<X_{(2)}<X_{(3)}\) be the order statistics. Compute the probability that the smallest of the \(X_i\) exceeds the median of the distribution.

Consider the density function \[ p(x;\alpha ,\lambda )=(2\pi x^3)^{-3/2} \exp \left \{(\alpha \lambda )^{1/2}-\frac {1}{2}\log \lambda -\frac {1}{2}\alpha x-\frac {\lambda }{2}x^{-1}\right \}, \qquad 0<x<\infty . \] Given \(X_1,X_2,\ldots ,X_n\) iid from this density find a sufficient statistic (that involves all of the \(X_i\)) for the parameters \((\alpha /2,\lambda /2)\).

Suppose that \(X_1,\ldots ,X_n\) form a random sample from a distribution for which the pdf \(f(x|\theta _1,\theta _2)\) is as follows: \[ f(x|\theta _1,\theta _2)= \begin {cases} \dfrac {1}{\theta _2-\theta _1}, & \theta _1<x<\theta _2,\\ 0, & \text {otherwise.} \end {cases} \] The values of \(\theta _1\) and \(\theta _2\) are unknown, other than \(-\infty <\theta _1<\theta _2<\infty \). Find their MLEs.

Let \(X_1,X_2,\ldots ,X_n\) be iid with a Poisson\((\theta )\) distribution, so that their common probability mass function is \[ \mathbb {P}(X_i=k)=e^{-\theta }\frac {\theta ^k}{k!}, \qquad k=0,1,2,\ldots ,\ \theta >0. \] Find the UMVUE of \(e^{-\theta }\).

Let \(X_1,X_2,\ldots ,X_{25}\) denote a random sample of size 25 from a \(N(\theta ,100)\) distribution. Find a uniformly most powerful critical region of size \(\alpha =.10\) for testing \(H_0:\theta =75\) against \(H_A:\theta >75\).

Let \(\{X_i:i\geq 1\}\) be i.i.d. random variables distributed uniformly on \([0,1]\). Prove that the distribution of \[ \frac {4\sum _{i=1}^{n} iX_i-n^2}{n^{3/2}} \] converges weakly and identify the limiting distribution.

Let \(X_1,X_2\) be iid with density \[ f(x;\theta )= \begin {cases} 3x^2/\theta ^3, & 0<x<\theta ,\\ 0, & \text {otherwise.} \end {cases} \] Find a complete, sufficient statistic for \(\theta \in [0,\infty )\) based on \(X_1\) and \(X_2\).

Let \(X_1,X_2,\ldots ,X_n\) be iid random variables with a Beta\((\theta ,\theta )\) distribution, i.e. their density is \[ f(x;\theta )=\frac {\Gamma (2\theta )}{\Gamma (\theta )^2} x^{\theta -1}(1-x)^{\theta -1}, \qquad 0<x<1. \] Show that a best critical region for testing \(H_0:\theta =1\) versus \(H_A:\theta =2\) is \[ \left \{(x_1,x_2,\ldots ,x_n): c\leq \prod _{i=1}^n x_i\right \}. \]

Let \(X_1,X_2,X_3\) be iid and distributed according to \(f(x)=2x\), \(0<x<1\). Let \(X_{(1)}<X_{(2)}<X_{(3)}\) be the order statistics. Compute the probability that the smallest of the \(X_i\) exceeds the median of the distribution.

Consider the density function \[ p(x;\alpha ,\lambda )=(2\pi x^3)^{-3/2} \exp \left \{(\alpha \lambda )^{1/2}-\frac {1}{2}\log \lambda -\frac {1}{2}\alpha x-\frac {\lambda }{2}x^{-1}\right \}, \qquad 0<x<\infty . \] Given \(X_1,X_2,\ldots ,X_n\) iid from this density find a sufficient statistic (that involves all of the \(X_i\)) for the parameters \((\alpha /2,\lambda /2)\).

Suppose that \(X_1,\ldots ,X_n\) form a random sample from a distribution for which the pdf \(f(x|\theta _1,\theta _2)\) is as follows: \[ f(x|\theta _1,\theta _2)= \begin {cases} \dfrac {1}{\theta _2-\theta _1}, & \theta _1<x<\theta _2,\\ 0, & \text {otherwise.} \end {cases} \] The values of \(\theta _1\) and \(\theta _2\) are unknown, other than \(-\infty <\theta _1<\theta _2<\infty \). Find their MLEs.

Let \(X_1,X_2,\ldots ,X_n\) be iid with a Poisson\((\theta )\) distribution, so that their common probability mass function is \[ \mathbb {P}(X_i=k)=e^{-\theta }\frac {\theta ^k}{k!}, \qquad k=0,1,2,\ldots ,\ \theta >0. \] Find the UMVUE of \(e^{-\theta }\).

Let \(X_1,X_2,\ldots ,X_{25}\) denote a random sample of size 25 from a \(N(\theta ,100)\) distribution. Find a uniformly most powerful critical region of size \(\alpha =.10\) for testing \(H_0:\theta =75\) against \(H_A:\theta >75\).

Let \(X_1,X_2,\ldots ,X_n\) be iid random variables with probability mass function \[ f(t;\theta )=\theta (1-\theta )^{t-1}, \qquad t=1,2,3,\ldots \] Find the uniformly minimum variance unbiased estimator of \(\theta \).

Let \(X_1,X_2,\ldots ,X_n\) be iid random variables with density \[ h(t;\theta )=\theta t^{\theta -1}I\{0\leq t\leq 1\}, \qquad \theta >0. \] Find the maximum likelihood estimator of \(\theta \) and compute the asymptotic variance of the maximum likelihood estimator.

Let \(X_1,X_2,\ldots ,X_n\) be iid with density \[ h(t;\theta )=\frac {1}{2}I\{\theta -1\leq t\leq \theta +1\}. \] Find all maximum likelihood estimators of \(\theta \). Show that if \(\hat \theta _n\) is a maximum likelihood estimator then \(\mathbb {E}[\hat \theta _n]\to \theta \) as \(n\to \infty \).

Let \(X_1,X_2,\ldots ,X_n\) be iid with density \[ f(x)=2x/\theta ^2, \qquad 0<x<\theta . \] Let \(Y=X_{(n)}=X_{n:n}\) be the largest order statistic of the \(X_i\). Show that \(Y/\theta \) is a pivotal quantity for \(\theta \) and use it to construct a 99% confidence interval for \(\theta \).

Let \(X_1,X_2,\ldots ,X_n\) be iid with a Poisson\((\theta )\) distribution, so that their common probability mass function is \[ \mathbb {P}(X_i=k)=e^{-\theta }\frac {\theta ^k}{k!}, \qquad k=0,1,2,\ldots ,\ \theta >0. \] Find a sufficient statistic for \(\theta \).

Assume that the lifetimes of 100 lightbulbs are iid random variables with an Exponential(5) distribution, meaning that their common pdf is \[ f(x;\theta )=\frac {1}{5}e^{-x/5}, \qquad x>0. \] Use normal approximation to estimate the probability that in total the 100 lightbulbs last longer than 550 hours. You can express your answer as an integral.

Let \(X_1,X_2,\ldots ,X_n\) be iid with common pdf \[ f(x;\theta )=\frac {\theta }{(1+x)^{\theta +1}}, \qquad 0<x<\infty ,\ \theta >0. \] What is a lower bound for the variance of an unbiased estimator of \(\theta \)?

Let \(X_1,\ldots ,X_m\) be \(m\geq 2\) i.i.d. standard normal random variables, and define the random vector \[ \mathbf {X}:= \begin {bmatrix} X_1\\ \vdots \\ X_m \end {bmatrix}. \] Prove that the distribution of \(\mathbf {X}\) is the same as the distribution of \(A\mathbf {X}\) for every \(m\times m\) orthogonal matrix \(A\) that is non random.

Fix \(0<\lambda <\rho \) and let \(X\), \(Y\), and \(Z\) be three independent Gamma distributed random variables with common scale parameter 1 and shape parameters \(\lambda \), \(\rho -\lambda \), and \(\rho \), respectively. (A Gamma random variable with scale parameter 1 and shape parameter \(\lambda \) has pdf \(\frac {1}{\Gamma (\lambda )}x^{\lambda -1}e^{-x}\mathbf {1}_{\{x>0\}}\).) Let \[ U=\frac {ZX}{X+Y},\qquad V=\frac {ZY}{X+Y},\qquad \text {and}\qquad W=X+Y. \] Prove that random vector \((U,V,W)\) has the same distribution as \((X,Y,Z)\). (It may be useful to note that \(Z=U+V\).)

Consider an array \(\{X_{i,j}\}\) of random variables such that, for every \(n\geq 1\), the collection \(X_{1,n},\ldots ,X_{n,n}\) is i.i.d. with common distribution \[ \mathbb {P}\{X_{1,n}=1\} = \mathbb {P}\{X_{1,n}=-1\} = \frac {1}{2}\left (1-\frac {1}{n^2}\right ); \] and \(\mathbb {P}\{X_{1,n}=n^2\}=n^{-2}\). Let \(S_n:=X_{1,n}+\cdots +X_{n,n}\). Prove that \(S_n/\sqrt {n}\Rightarrow \mathrm {N}(0,\sigma ^2)\) for some \(\sigma \in (0,\infty )\), although \(\mathbb {E}(S_n/\sqrt {n})\to \infty \) as \(n\to \infty \).

Let \(X_1,X_2,\ldots ,X_n\) be iid with common pdf \(f(x)=2x\), \(0<x<1\) and zero elsewhere. Find the limiting distribution of \(n(1-X_{(n)})\), where \(X_{(n)}=\max \{X_1,\ldots ,X_n\}\).

Let \(X_1\sim N(0,4\theta )\), \(X_2\sim N(0,9\theta )\), \(X_3\sim N(0,4\theta ^2)\), \(X_4\sim N(0,12\theta ^2)\), and assume that all \(X\)s are independent of each other. Using all four \(X\) variables and the \(t\) distribution, compute a \(100(1-\alpha )\%\) confidence interval for \(\theta \).

Let \(X_1,X_2,\ldots ,X_n\) be iid with common pdf \[ f(x;\theta )=\frac {1}{2\theta }e^{-|x|/\theta }, \qquad -\infty <x<\infty , \] with \(\theta \) unknown but positive. Compute the Cramer-Rao lower bound for the variance of an unbiased estimator of \(\theta \), and extend your calculation to finding the UMVUE by finding a linear combination of \[ \sum _{i=1}^n \frac {\partial }{\partial \theta }\log f(X_i;\theta ) \] that is unbiased.

If a random sample of size \(n\) is taken from a distribution having pdf \(f(x;\theta )=2x/\theta ^2\), \(0<x\leq \theta \) and zero elsewhere, find:

(a) the MLE \(\hat \theta \) for \(\theta \),

(b) the constant \(c\) so that \(\mathbb {E}[c\hat \theta ]=\theta \),

(c) the MLE for the median of the distribution.

If \(X_1,X_2,\ldots ,X_n\) is a random sample from a distribution having pdf of the form \(f(x;\theta )=\theta x^{\theta -1}\), \(0<x<1\) and zero elsewhere, show that a best critical region for testing \(H_0:\theta =1\) against \(H_1:\theta =2\) is \[ C=\left \{(x_1,x_2,\ldots ,x_n): c\leq \prod _{i=1}^n x_i\right \}. \]

Let \(X_1,X_2,\ldots ,X_{25}\) denote a random sample of size 25 from a normal distribution with mean \(\theta \) and variance 100. Find a uniformly most powerful critical region of size \(\alpha =0.10\) for testing \(H_0:\theta =75\) against \(H_1:\theta >75\).

Let \(Y_1,Y_2,\ldots ,Y_n\) be independent random variables, each with the \(N(\beta x_i,\gamma ^2x_i^2)\) distribution, where the numbers \(x_1,x_2,\ldots ,x_n\) are known, not all are equal, and none of them is zero. Suppose \(\beta \) and \(\gamma \) are unknown. Find the maximum likelihood estimators of \(\beta \) and \(\gamma ^2\).

Let \(X_1,X_2,\ldots ,X_n\) be a random sample from a Gamma\((\alpha ,\theta )\) distribution, with \(\alpha \) known and \(\theta \) unknown. Find the Fisher information of this distribution and show that the MLE of \(\theta \) is an efficient estimator of \(\theta \).

Let \(X_1,\ldots ,X_n\) be iid exponential(1) random variables. Find the joint pdf of the pair \((X_{(1)},X_{(3)})\), where \(X_{(i)}\) denotes the \(i\)th order statistic, i.e. \[ X_{(1)}\leq X_{(2)}\leq \cdots \leq X_{(n)}. \]

Assume that \(X_1,\ldots ,X_n\) are a random sample with common pdf \(f(x;\theta )=(\theta +1)x^{-\theta -2}\) for \(x>1\), and zero otherwise. Is the method of moments estimator for \(\theta \) the same as the maximum likelihood estimator for \(\theta \)?

Let \(X_1,\ldots ,X_n\) be iid with the Gamma\((\theta ,\kappa )\) distribution, meaning that their common pdf is \[ f(x;\theta ,\kappa )=\frac {1}{\theta ^\kappa \Gamma (\kappa )} x^{\kappa -1}e^{-x/\theta }, \qquad x>0, \] with \(\kappa \) known but \(\theta \) unknown. Derive a \(100(1-\alpha )\%\) equal-tailed confidence interval for \(\theta \) based on the sufficient statistic.

Suppose that \(X_1,\ldots ,X_n\) are iid from a \(\BETA (\theta _1,\theta _2)\) distribution, meaning that the pdf is \[ f(x;\theta _1,\theta _2)= \frac {\Gamma (\theta _1+\theta _2)}{\Gamma (\theta _1)\Gamma (\theta _2)} x^{\theta _1-1}(1-x)^{\theta _2-1}, \qquad 0<x<1. \] Find jointly sufficient statistics for \(\theta _1\) and \(\theta _2\), and use them to find a UMVUEs for \(\theta _1\) and \(\theta _2\).

Let \(X_1,X_2,\ldots ,X_n\) be iid from a distribution with CDF \(F(x)=1-1/x\) for \(1\leq x<\infty \), and zero otherwise.

(a) Derive the CDF of the smallest order statistic \(X_{(1)}\).

(b) Find the limiting distribution of \(X_{(1)}\).

(c) Find the limiting distribution of \(X_{(1)}^n\).

Let \(X_1,\ldots ,X_n\) be iid Uniform\((0,\theta )\) with \(\theta >0\), and \(X_{(n)}\) be the largest order statistic.

(a) Find the probability that the random interval \((X_{(n)},2X_{(n)})\) contains \(\theta \).

(b) Find the constant \(c\) such that \((X_{(n)},cX_{(n)})\) is a \(100(1-\alpha )\%\) confidence interval for \(\theta \).

Let \(X_1,X_2,\ldots \) be i.i.d. with common distribution \(\mathbb {P}(X_1=1)=\mathbb {P}(X_1=-1)=1/2\), and put \(S_0:=0\) and \(S_n:=X_1+\cdots +X_n\) for each \(n\geq 1\). Consider the stopping time \(N:=\min \{n\geq 1:S_n=1\}\).

(a) Let \(g(u):=\mathbb {E}[u^{X_1}]\) for all \(u>1\), and show that the optional stopping theorem applies to the martingale \(M_n:=u^{S_n}g(u)^{-n}\), where \(u>1\), and the stopping time \(N\).

(b) Use part (a) to show that \[ \mathbb {E}[v^N]=\frac {1-\sqrt {1-v^2}}{v}, \qquad 0<v<1. \]

(c) Use part (b) to show that \[ \mathbb {P}(N=2m+1)= \frac {1}{m+1}\binom {2m}{m}\frac {1}{2^{2m+1}}, \qquad m\geq 0. \]

We assume that the number of people entering a store follows a Poisson distribution with parameter \(\lambda \). Every customer spends money in the store, independently of each other according to a uniform distribution on \([10,100]\). We observe \(X\), the total amount of money spent in the store on a given day.

(a) Estimate \(\lambda \) using the method of moments.

(b) Let assume that we have the stores income \(X_1,X_2,\ldots ,X_n\) for \(n\) days and \(X_1,X_2,\ldots ,X_n\) are independent and distributed as \(X\). What would be the moment estimator for \(\lambda \) based on \(X_1,X_2,\ldots ,X_n\)?

Let \(X_1,X_2,\ldots ,X_n\) be independent and identically distributed random variables with density function \[ f(t)= \begin {cases} 0, & \text {if } t<0,\\ \dfrac {1}{\lambda }\exp (-t/\lambda ), & \text {if } t\geq 0. \end {cases} \] Find the uniformly minimum variance unbiased estimator for \(\theta =1/\lambda \).

Let \(U_1,U_2,\ldots ,U_n\) be independent and identically distributed random variables, uniform on \([0,1]\). Let \(U_{1,n}\leq U_{2,n}\leq \cdots \leq U_{n,n}\) denote the order statistics.

(a) Compute the limit distribution of \(Z=n(U_{3,n}-U_{2,n})\).

(b) Show that \(Z_1=n(U_{3,n}-U_{2,n})\) and \(Z_2=n(U_{4,n}-U_{3,n})\) are asymptotically independent.

Let \(X_1,X_2,\ldots ,X_n\) be a random sample from the Poisson distribution with parameter \(\lambda >0\). Use the Rao–Blackwell theorem and/or the Lehmann–Scheffé theorem to find a UMVUE of \(\mathbb {P}\{X_1=0\}=e^{-\lambda }\).

Find the maximum likelihood estimators of \(\theta \) and \(\alpha \), based on a random sample \(X_1,X_2,\ldots ,X_n\) from the distribution with density \[ f(x;\theta ,\alpha )=\frac {\alpha }{\theta ^\alpha }x^{\alpha -1}, \qquad 0\leq x\leq \theta . \] Here \(\theta \) and \(\alpha \) are positive.

1000 individuals were classified according to sex and to whether or not they were color-blind, with the following results: male normal 442; female normal 514; male color-blind 38; and female color-blind 6. According to a genetic model, the four relative frequencies should be of the form \(p/2\); \(p^2/2+p(1-p)\); \((1-p)/2\); and \((1-p)^2/2\), respectively. Suppose we want to test whether the data fit the model.

(a) Find the maximum likelihood estimator \(\hat p\) of \(p\).

(b) Describe a size \(\alpha =0.10\) test of the null hypothesis that the data fit the model, assuming that \(\hat p=0.913\). Specify the critical region as precisely as possible, perhaps in terms of a quantile of a certain \(\chi ^2\) distribution.

A scale has two pans. The measurement given by the scale is the difference between the weights on pan 1 and pan 2 plus a random error. Thus, if a weight \(\mu _1\) is put on pan 1 and a weight \(\mu _2\) is put on pan 2, then the measurement is \(Y=\mu _1-\mu _2+\epsilon \). Suppose that \(\mathbb {E}[\epsilon ]=0\), \(\operatorname {Var}(\epsilon )=\sigma ^2\), and in repeated use of the scale observations \(Y_i\) are independent.

Suppose two objects, #1 and #2, have weights \(\beta _1\) and \(\beta _2\). Measurements are then taken as follows:

• Object #1 is put on pan 1 and nothing on pan 2
• Object #2 is put on pan 2 and nothing on pan 1
• Object #1 is put on pan 1 and object #2 is put on pan 2
• Objects #1 and #2 are both put on pan 1 and nothing on pan 2

(a) Let \(\mathbf {Y}=(Y_1,Y_2,Y_3,Y_4)^\prime \) be the vector of observations. Formulate this as a linear model.

(b) Find the least squares estimators of \(\beta _1\) and \(\beta _2\).

(c) Find the variances of the least squares estimators as well as the covariance between them.

Let \(X_1,X_2,\ldots ,X_n\) be a random sample from a \(N(\theta ,\sigma ^2)\) distribution (\(\sigma ^2\) known), and let the prior distribution of \(\theta \) be \(N(\mu _0,\sigma _0^2)\), where \(\mu _0\) and \(\sigma _0^2\) are known.

(a) Find the posterior distribution of \(\theta \).

(b) Find the Bayes estimator under squared error loss.

Let \(X\) and \(Y\) be independent random variables. Show that if the distribution of \(X\) is absolutely continuous with respect to the Lebesgue measure, then the distribution of \(X+Y\) is also absolutely continuous with respect to the Lebesgue measure.

Let \(X_1,\ldots ,X_n\) be independent identically distributed random variables with \(\mathbb {E} X_1=\mu \) and \(0<\operatorname {Var}(X_1)=\sigma ^2<\infty \). Let \[ Y_n=\sum _{1\leq i<j\leq n}X_iX_j. \] Find numerical sequences \(a_n\) and \(b_n\) such that \((Y_n-a_n)/b_n\) has a non-degenerate limit distribution.

Let \(Z_n\) be a Galton–Watson branching process with offspring distribution \(\{p_k,\ k=0,1,2,\ldots \}\) and \(Z_0=x\) (with \(x\) a positive integer), and let \[ f(\theta )=\sum p_k\theta ^k \] be the associated pgf. Suppose that \(\rho \in (0,1)\) satisfies \(f(\rho )=\rho \). Show that \(\rho ^{Z_n}\) is a martingale, and use this to conclude that \(\mathbb {P}(Z_n=0\text { for some }n\geq 0)=\rho ^x\).

By considering the Poisson distribution, show that \[ e^{-n}\left (1+n+\frac {n^2}{2!}+\cdots +\frac {n^n}{n!}\right )\to \frac {1}{2} \] as \(n\to \infty \).

Let \(X_1,\ldots ,X_n\) be independent identically distributed random variables with \(\mathbb {E} X_1=\mu \) and \(0<\operatorname {Var}(X_1)=\sigma ^2<\infty \). Let \[ Y_n=\sum _{1\leq i<j\leq n}X_iX_j. \] Find numerical sequences \(a_n\) and \(b_n\) such that \((Y_n-a_n)/b_n\) has a non-degenerate limit distribution.

Let \(Z_n\) be a Galton–Watson branching process with offspring distribution \(\{p_k,\ k=0,1,2,\ldots \}\) and \(Z_0=x\) (with \(x\) a positive integer), and let \[ f(\theta )=\sum p_k\theta ^k \] be the associated pgf. Suppose that \(\rho \in (0,1)\) satisfies \(f(\rho )=\rho \). Show that \(\rho ^{Z_n}\) is a martingale, and use this to conclude that \(\mathbb {P}(Z_n=0\text { for some }n\geq 0)=\rho ^x\).

By considering the Poisson distribution, show that \[ e^{-n}\left (1+n+\frac {n^2}{2!}+\cdots +\frac {n^n}{n!}\right )\to \frac {1}{2} \] as \(n\to \infty \).

Let \(X_1,X_2,\ldots ,X_n\) be independent, identically distributed random variables with distribution function \[ F(t)= \begin {cases} 0, & \text {if } -\infty <t<0,\\ t^3, & \text {if } 0\leq t\leq 1,\\ 1, & \text {if } t>0. \end {cases} \] Show that \[ Y_n=n^{1/3}X_{1,n} \] converges in distribution, where \(X_{1,n}=\min \{X_1,X_2,\ldots ,X_n\}\).

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with density function \[ h(t,\theta )= \begin {cases} 0, & \text {if } t\notin [-\theta ,\theta ],\\ \dfrac {5}{2}\theta ^{-5}t^4, & \text {if } -\theta \leq t\leq \theta . \end {cases} \] Find a moment estimator for \(\theta \).

Let \(X_1,\ldots ,X_n\) be independent, identically distributed random variables with density function \[ h(t,\theta )= \begin {cases} 0, & \text {if } -\infty <t<0,\\ \theta (t+1)^{-\theta -1}, & \text {if } 0\leq t<\infty , \end {cases} \] \(\theta >0\).

(a) Find the maximum likelihood estimator for \(\theta \).

(b) Is the estimator unbiased?

(c) Find the asymptotic variance of the maximum likelihood estimator for \(\theta \).

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with density function \[ h(t,\theta )= \begin {cases} 0, & \text {if } t\notin [0,\theta ],\\ 2\theta ^{-2}t, & \text {if } 0\leq t\leq \theta . \end {cases} \] Find the uniformly minimum variance unbiased estimator for \(\theta \). Explain your answer. You need to prove directly that the sufficient statistic is also complete in this case.

Let \(X_1\) and \(X_2\) be independent random variables. The density function function of \(X_1\) is \[ f(t)= \begin {cases} 0, & \text {if } t\notin [0,1],\\ \dfrac {e^t}{e-1}, & \text {if } 0\leq t\leq 1. \end {cases} \] The distribution of \(X_2\) is \[ \mathbb {P}\{X_2=1\}=p \quad \text {and}\quad \mathbb {P}\{X_2=-1\}=q,\qquad p+q=1. \] Compute the moment generating function of \(X_1X_2\).

Let \(X_1,\ldots ,X_n\) be an i.i.d. sample from a \(\operatorname {Uniform}(0,\theta )\) distribution, where \(\theta >0\) is unknown. Find a 95% confidence interval for \(\theta \).

Let \(X_1,\ldots ,X_n\) denote an independent sample from an exponential distribution with [unknown] mean \(\theta >0\). What does the Neyman–Pearson lemma say about \(H_0:\theta =1\) versus \(H_a:\theta =2\)? Explain carefully, and identify explicitly the rejection region.

Derive, using only first principles, the least-squares estimators of the slope and the intercept of a linear regression problem. What can you say about the optimality properties of those estimators?

The following data are times (in hours) between failures of air conditioning equipment in a particular airplane: \[ 74\quad 57\quad 48\quad 29\quad 502\quad 12\quad 70\quad 21\quad 29\quad 386\quad 59\quad 27\quad 153\quad 26\quad 326. \] Assume that the data are observed values of an i.i.d. random sample from an exponential distribution, \(X_i\sim \operatorname {Exponential}(\theta )\). Test \(H_0:\theta =125\) versus \(H_a:\theta \ne 125\). (A chi-square table is provided.)

If \(X\) is either (a) Poisson\((\lambda )\) or (b) gamma\((\lambda ,1)\) (density proportional to \(x^{\lambda -1}e^{-x}\), \(x>0\)), show that \((X-\mathbb {E}[X])/\sqrt {\operatorname {Var}(X)}\) converges in distribution to \(N(0,1)\) as \(\lambda \to \infty \) (\(\lambda \) need not be an integer).

Let \(X\) and \(Y\) be independent, standard-normal random variables. Find the distribution of \(X/Y\)?

Let \(X_1,\ldots ,X_n\) be a random sample from \(N(\theta ,\theta )\), where \(\theta >0\). In particular, the population mean and the population variance are equal but unknown.

(a) Find the method-of-moments estimator of \(\theta \) based on the second sample moment.

(b) Find the maximum likelihood estimator of \(\theta \).

Let \(X_1,\ldots ,X_n\) be a random sample from a \(N(\theta ,1)\) distribution, and let the prior distribution of \(\theta \) be \(N(\mu _0,1)\), where \(\mu _0\) is known.

(a) Find the posterior distribution of \(\theta \).

(b) Find the Bayes estimator of \(\theta \) under squared error loss, and show that it is a weighted average of the prior mean \(\mu _0\) and the sample mean \(\bar X\).

Let \(X_1,\ldots ,X_n\) be a random sample from \(\operatorname {Poisson}(\theta )\).

(a) Find a complete, sufficient statistic.

(b) Noting that \(\Prob _\theta (X_1=1)=\theta e^{-\theta }\), use the Rao–Blackwell and Lehmann–Scheffé theorems to find a UMVUE of \(\tau (\theta ):=\theta e^{-\theta }\).

Assume a sample of size \(n\) from \(\operatorname {Uniform}[\eta -\theta ,\eta +\theta ]\), which is a location-scale family. Here \(\eta \) is real and \(\theta >0\).

(a) Find a \(100(1-\alpha )\%\) confidence interval for \(\theta \).

(b) Find a \(100(1-\alpha )\%\) confidence interval for \(\eta \).

It is not necessary to find the explicit distribution of the pivotal quantities; just denote the needed quantiles by \(x_p\) and \(y_p\) for an appropriate \(p\).

Consider the simple linear regression model \(Y_i=\beta _0+\beta _1x_i+\epsilon _i\) (\(i=1,2,\ldots ,n\)). Assume independent \(N(0,\sigma ^2)\) errors.

(a) Find the least squares estimators \(\hat \beta _0\) of \(\hat \beta _1\) of \(\beta _0\) and \(\beta _1\).

(b) Calculate the covariance matrix of \((\hat \beta _0,\hat \beta _1)\).

Let \(U_1,U_2,\ldots ,U_n\) be independent random variables uniform on \([0,1]\) and let \(U_{1,n}\leq U_{2,n}\leq \cdots \leq U_{n,n}\) be the corresponding order statistics.

a. Compute the asymptotic distribution of \(nU_{2,n}\). (5 points)

b. Show that \(nU_{1,n}\) and \(n(1-U_{n,n})\) are asymptotically independent. (5 points)

Let \(y_i=\alpha x_i+\epsilon _i\), \(1\leq i\leq n\), where \(\epsilon _1,\epsilon _2,\ldots ,\epsilon _n\) are independent identically distributed normal \(N(0,\sigma ^2)\) random variables.

a. Determine the maximum likelihood estimators for \(\alpha \) and \(\sigma ^2\). (5 points)

b. Compute the distribution of the maximum likelihood estimator for \(\alpha \). (5 points)

c. Compute the distribution of the maximum likelihood estimator for \(\sigma ^2\). (5 points)

Let \(X\) and \(Y\) be two independent random variables with distribution functions \(F\) and \(G\) and density functions \(f\) and \(g\).

a. Determine the distribution function of \(XY\). (5 points)

b. Does \(XY\) always have a density function? (5 points)

c. Assuming that \(XY\) has a density function, compute it. (5 points)

Let \(X_1,X_2,\ldots ,X_n\) be independent, identically distributed random variables with density function \[ f(t;\theta )= \begin {cases} 0, & \text {if } t<\theta ,\\ e^{-(t-\theta )}, & \text {if } t>\theta . \end {cases} \]

a. Find a moment estimator for \(\theta \). (5 points)

b. Find the maximum likelihood estimator for \(\theta \). (5 points)

c. Determine the asymptotic efficiency between the two estimators. (5 points)

The number of customers entering a store on a given day is a Poisson random variable with parameter \(\theta \). The money spent by a customer is uniformly distributed on \([0,\eta ]\). The number of customers and the money spent in the store are independent. We observe \(Z_1,Z_2,\ldots ,Z_n\), the total spending (the money collected by the store) on \(n\) days. We can assume that \(Z_1,Z_2,\ldots ,Z_n\) are independent.

a. Provide estimators for \(\theta \) and \(\eta \) using the method of moments. (10 points)

b. Provide estimators for \(\theta \) and \(\eta \) using the likelihood method. (10 points)

Let \(X_1,\ldots ,X_n\) be independent identically distributed random variables with distribution function \(F\). We assume that \(F\) is strictly increasing and continuous on its support. Let \(x_{1/2}\) denote the median.

a. Find a \(1-\alpha \) confidence interval for \(x_{1/2}\). (5 points)

b. Provide a large sample approximation for the confidence interval. (5 points)

Let \(X\) and \(Y\) be independent exponential random variables with parameters \(\lambda \) and \(\mu \). (For example, \(\mathbb {P}(X>x)=e^{-\lambda x}\) for \(x>0\).)

(a) Show that \(Z:=\min (X,Y)\) is independent of the event \(\{X<Y\}\).

(b) Find \(\mathbb {P}(X=Z)\).

(c) Find the distribution of \(U:=(X-Y)^+=\max (X-Y,0)\).

(d) Find the distribution of \(V:=\max (X,Y)-\min (X,Y)\).

Let \(X_i\) be a random sample of size \(n\) with pdf \(f(x;\theta )=\frac {1}{\theta }e^{-(x-\theta )/\theta }\) (\(x>\theta \), \(\theta >0\)).

2a) (5 points) What is the MME of \(\theta \)? Is it a reasonable estimator?

2b) (5 points) What is the MLE of \(\theta \)?

Let \(X_i\) be a random sample of size \(n\) with pdf \(f(x;\theta )=\theta x^{\theta -1}\) (\(0<x<1\), \(\theta >0\).)

4a) (5 points) Find a complete and sufficient statistic.

4b) (5 points) Find a uniformly minimum variance unbiased estimator (UMVUE) of \(1/\theta \).

Let \(X_i\) be a random sample of size \(n\) with pdf \(f(x;\theta )=\frac {2x}{\theta ^2}\) (\(0<x<\theta \)).

8a) (5 points) Use the MME \(\tilde \theta \) to find an unbiased estimator of \(\theta \).

8b) (10 points) Use the MLE \(\hat \theta \) to find another unbiased estimator of \(\theta \).

8c) (5 points) Using the factorization criterion, find one sufficient statistic for \(\theta \). Which of the two unbiased estimators you have found has a lower variance?

8d) (5 points) Calculate \(\operatorname {Var}(\tilde \theta )\) and \(\operatorname {Var}(\hat \theta )\). Does this confirm your answer?

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with distribution function \(F\). We use \[ \hat f_n(t)=\frac {1}{nh}\sum _{1\leq i\leq n} K\left (\frac {t-X_i}{h}\right ) \] to estimate \(f=F'\).

We assume that the following conditions are satisfied: \[ \int _{-1}^{1}K(u)\,du=1, \qquad K(t)=0,\quad \text {if } t\notin [-1,1]. \] \(F\) is three times differentiable and the derivatives are uniformly bounded on the real line \[ h=h(n)\to 0. \]

(i) Compute the bias of \(\hat f_n(t)\) for any \(t\) and any \(n\). (7 points)

(ii) Prove that \(\hat f_n(t)\) is asymptotically unbiased for any \(t\) as \(n\to \infty \). (7 points)

(iii) Compute the mean square error of \(\hat f_n(t)\) (7 points)

Let \(X\) be a random variable with distribution function \(F\). Prove that \(X\) is symmetric around 0 (i.e. \(F(x)=1-F(-x)\) for all \(x\)) if and only if the characteristic function of \(X\) is real. (7 points)

Let Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with distribution function \[ F(t)=\left (\frac {t}{\theta }\right )^2 \quad \text {if } 0\leq t\leq \theta . \]

(i) Find the maximum likelihood estimator for \(\theta \). (7 points)

(ii) Compute the limit distribution of the maximum likelihood estimator. (7 points)

Let Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with \(\E X_1=\mu \), \(\operatorname {Var} X_1=\sigma ^2\). We assume that \(\E X_4<\infty \). The random variable \[ \hat \sigma _n^2=\frac {1}{2n}\sum _{2\leq i\leq n}(X_i-X_{i-1})^2 \] is suggested to estimate \(\sigma ^2\).

(i) Establish some basic properties of \(\hat \sigma _n^2\). (7 points)

(ii) Is \(\hat \sigma _n^2\) a better estimator than the sample variance? Justify your answer. (7 points)

Let \((X_1,\ldots ,X_m)\) be a multinomial random vector with parameters \(N\) and \(p_1,p_2,\ldots ,p_m\). Note that \(N=X_1+\cdots +X_m\) and \(p_1+p_2+\cdots +p_m=1\). Find the maximum likelihood estimators for \(p_1,p_2,\ldots ,p_m\). (7 points)

Let \(X_1, X_2, \ldots \) be an i.i.d. sample from the uniform distribution on \([\theta , 1+\theta ]\), where \(\theta > 0\) is unknown. Is there an unique maximum likelihood estimator of \(\theta \) that is based on \(\{X_i\}_{i=1}^{\infty }\)? Prove or disprove.

Suppose \(X_1, \ldots , X_n\) is an i.i.d. sample from the exponential density with mean \(\theta \), where \(\theta > 0\) is unknown. Find an estimator of \(\theta \) based on \(X_1, \ldots , X_n\). Is your estimator complete? Is it sufficient? Justify your reasoning.

Let \(X\) have the binomial distribution with parameters \(n=10\) and \(p=\theta \) unknown. Suppose we impose a uniform-\((0,1)\) prior on \(\theta \). Compute the posterior mean of \(\theta \). Is it consistent?

Let \(X_1, X_2, \ldots \) denote an i.i.d. sample with common distribution function \(F\). Define \[ F_n(x) = n^{-1}\sum _{j=1}^{n} I\{X_j \leq x\} \] for all \(x \in (-\infty , \infty )\); i.e., \(F_n\) is the empirical distribution function based on the \(X_i\)’s. Prove that for all \(x \in (-\infty , \infty )\) fixed, \[ \operatorname {Var}\bigl (F_n(x)-F(x)\bigr ) \leq \frac {1}{4n}. \] Use this to conclude that \(F_n(x)\) is a consistent estimator for \(F(x)\).

From here on let \(X_1, X_2, \ldots \) denote an i.i.d. sample from an exponential density with mean \(1/\theta \).

Is the MLE a consistent estimator of \(\theta \)? Prove or disprove.

Compute the distributions of: \[ \begin {array}{ll} \text {(a)} & \max _{1\leq i\leq n} X_i,\\ \text {(b)} & \min _{1\leq i\leq n} X_i,\\ \text {(c)} & X_1+\cdots +X_n. \end {array} \]

Let \(\{X_i\}_{i=1}^{\infty }\) be i.i.d. with the “standard Cauchy distribution.” That is, the density function of \(X_1\) is \[ f(x)=\frac {1}{\pi (1+x^2)} \qquad -\infty <x<\infty . \] Prove that \(\max _{1\le i\le n}X_i/n\) converges weakly. Identify the limit.

Let \(X_1,\ldots ,X_n\) be independent, identically distributed random variables with a continuous distribution function \(F\). Let \(x_{1/4}\) be defined by \(F(x_{1/4})=1/4\).

(a) Find a \(1-\alpha \) lower confidence bound for \(x_{1/4}\).

(b) Provide a large sample approximation.

Let \(X_1,X_2,\ldots ,X_n\) be independent, identically distributed random variables with distribution function \[ F(t)=\frac {1}{1+e^{-t}},\qquad -\infty <t<\infty . \] Show that \[ Y_n=X_{n,n}-\log n \] converges in distribution, where \(X_{n,n}=\max \{X_1,X_2,\ldots ,X_n\}\).

Let \(X_1,\ldots ,X_n\) be independent, identically distributed random variables with probability mass function \[ f(t,\theta )=\theta (1-\theta )^{t-1}I\{t=1,2,\ldots \}\qquad (0<\theta <1). \] Find the uniformly minimum variance unbiased estimator for \(\theta \).

Let \(\epsilon _i\), \(1\leq i\leq n\), be independent identically distributed normal \(N(0,\sigma ^2)\) random variables and consider the model \[ y_i=i\alpha +\gamma i^2+\epsilon _i,\qquad 1\leq i\leq n. \] We observe \(y_i\), \(1\leq i\leq n\).

(a) Find the maximum likelihood estimators for \(\alpha \), \(\gamma \), and \(\sigma ^2\).

(b) Compute the joint distribution of the maximum likelihood estimators for \((\alpha ,\gamma )\).

Suppose \(X_1,X_2,\ldots \) are independent with distribution, \[ \mathbb {P}\{X_n=n^\alpha \}=\mathbb {P}\{X_n=-n^{-\alpha }\}=\frac {1}{2}, \qquad n=1,2,\ldots . \] Prove that \(\sum _{i=1}^{n}X_n/n\to 0\) a.s. if \(0<\alpha <\frac {1}{2}\).

Suppose \(X_{\alpha ,\beta }\) has a Gamma distribution with parameters \((\alpha ,\beta )\). That is, the density function of \(X_{\alpha ,\beta }\) is \[ f(x)=\frac {\beta ^\alpha }{\Gamma (\alpha )}x^{\alpha -1}e^{-\beta x}, \qquad x>0, \] where \(\Gamma (\alpha )=\int _0^\infty x^{\alpha -1}e^{-x}\,dx\) for all \(\alpha >0\). Prove that \((\beta X_{\alpha ,\beta }-\alpha )/\sqrt {\alpha }\Rightarrow N(0,1)\).

Let \(X_1,X_2,\ldots ,X_n\) be independent, identically distributed random variables with density function \[ f(t,\theta )= \begin {cases} 0, & \text {if } t\notin [0,\theta ],\\ \dfrac {1}{\theta }, & \text {if } t\in [0,\theta ]. \end {cases} \] We use \(\hat {\theta }=cX_{1,n}\) and \(\hat {\eta }=\alpha \bar {X}\) to estimate \(\theta \), where \[ X_{1,n}=\min _{1\leq i\leq n}X_i\quad \text {and}\quad \bar {X}=\frac {1}{n}\sum _{1\leq i\leq n}X_i. \] (a) Find \(c\) and \(\alpha \) such that the estimators are unbiased.

(b) Compute the relative efficiency.

Let \(X_1,X_2,\ldots ,X_n\) be independent, identically distributed Poisson \((\theta )\) random variables. Find the uniformly minimum unbiased estimator for \(\eta =\theta ^2 e^{-\theta }\). (You must explain your answer.)

Let \(X_1,X_2,\ldots ,X_n\) and \(Y_1,Y_2,\ldots ,Y_m\) be independent, identically distributed random variables with a common continuous distribution function. Compute the expected value and the variance of \[ Z=\sum _{1\leq i\leq n}\sum _{1\leq j\leq n} I\{X_i\leq Y_j\}. \]

where \(I\{A\}\) denotes the indicator function of the event \(A\).

Let \(X_1,X_2,\ldots ,X_n\) be independent, identically distributed random variables with density function \[ f(t,\theta )= \begin {cases} 0, & \text {if } t\leq \theta ,\\ e^{-(t-\theta )}, & \text {if } t>\theta . \end {cases} \] (a) Find two \(1-\alpha \) equal-tail confidence intervals for \(\theta \). One should be based on the smallest order statistic and the other should be based on the sample mean.

(b) Compare the expected lengths of the confidence intervals.

Let \(\{X_n\}_{n=1}^{\infty }\) be i.i.d. random variables with distribution, \(\mathbb {P}\{X_1=2\}=\mathbb {P}\{X_1=-2\}=1/2\). Define \(S_n=\sum _{i=1}^{n}X_i\) and \(T=\inf \{n\ge 1:S_n\in \{-2,6\}\}\). Prove that \(T<\infty \) a.s., and then compute \(\mathbb {P}\{S_T=-2\}\).

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed Poisson random variables with common probability mass function \[ p(k,\theta )=\frac {\theta ^k}{k!}e^{-\theta },\qquad k=0,1,\ldots \] Let \[ \tau =\theta e^{-\theta }. \] Find the uniformly minimum variance unbiased estimator for \(\tau \).

Let \(X_1,X_2,\ldots ,X_n\) be independent, identically distributed uniform \((0,\theta )\) random variables with common density function \[ f(t)= \begin {cases} 0, & \text {if } t\notin [0,\theta ],\\ 1/\theta , & \text {if } t\in [0,\theta ]. \end {cases} \] (a) Find the maximum likelihood estimator for \(\theta \).

(b) Find a \(1-\alpha \) equal-tail confidence interval for \(\theta \).

\(X\) and \(Y\) are two independent identically distributed normal \(N(0,\sigma ^2)\) random variables. Let \[ Z=X^2+Y^2 \] and \[ W=\frac {X}{(X^2+Y^2)^{1/2}}. \] (a) Show that \(Z\) and \(W\) are independent.

(b) Compute the marginal distributions of \(Z\) and \(W\).

Let \(y_i=\alpha x_i+\epsilon _i\), where \(\epsilon _i\), \(1\leq i\leq n\), are independent, identically distributed normal \(N(0,\sigma ^2)\) random variables.

(a) Find the maximum likelihood estimators for \(\alpha \) and \(\sigma ^2\).

(b) Find the distributions of the maximum likelihood estimators.

(c) Compute the least-squares estimator for \(\alpha \).

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed Poisson random variables with common probability mass function \[ p(k,\theta )=\frac {\theta ^k}{k!}e^{-\theta },\qquad k=0,1,\ldots \] Using the Cramer-Rao inequality show that the sample mean is a uniformly minimum variance unbiased estimator for \(\theta \).

Let \(X\) be such that \(\mathbb {E}[X^+]=\infty \) and \(\mathbb {E}[X^-]<\infty \), and let \(X_1,X_2,X_3,\ldots \) be i.i.d. with common distribution that of \(X\). With \(S_n=X_1+X_2+\cdots +X_n\), show that \(\lim _{n\to \infty }S_n/n=\infty \) a.s.

Let \(X_1,X_2,X_3,\ldots \) be a sequence of random variables, each of which has all moments finite, and suppose that \(\lim _{n\to \infty }\mathbb {E}[(X_n)^k]=k!\) for each \(k\ge 1\). Show that \(X_n\) converges in distribution.

Let \(\mathbb {P}\{X=1\}=\mathbb {P}\{X=-1\}=1/2\), let \(X_1,X_2,X_3,\ldots \) be i.i.d. with common distribution that of \(X\), and put \(S_n=X_1+X_2+\cdots +X_n\). Define \(T=\min \{n\ge 0:S_n=-A\text { or }S_n=B\}\), where \(A\) and \(B\) are positive integers. Using the martingales of problem 5, find \(\mathbb {P}\{S_T=B\}\) and \(\mathbb {E}[T]\).

Consider the following model: \[ y_i=f(x_i)+\epsilon _i,\qquad 1\leq i\leq n, \] where \[ f(t)=a_0+a_1t+\cdots +a_pt^p. \] The parameters \(a_0,a_1,\ldots ,a_p\) are unknown. The errors \(\epsilon _1,\epsilon _2,\ldots ,\epsilon _n\) are independent normal \(N(0,\sigma ^2)\) random variables. It is assumed that \(x_i\in [0,1]\) and known. We observe \((y_i,x_i)\), \(1\leq i\leq n\).

1. Estimate the parameters \(a_0,a_1,\ldots ,a_p\) and \(\sigma ^2\). (15 points)
2. Get the distribution of the estimator for \(\sigma ^2\). (5 points)

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with distribution function \(F\). We assume that \(f=F'\) has a bounded second derivative. \(K(t)\) is a non-negative function satisfying \[ K(t)=K(-t)\quad \text {(symmetric),} \] \[ \int _{-\infty }^{\infty }K(t)\,dt=1 \quad \text {and}\quad K(t)=0\text { if }t\notin [-1,1]. \] The sequence \(h=h(n)\) is positive and goes to 0. We estimate \(f\) with \[ \hat {f}_n=\frac {1}{nh}\sum _{1\leq i\leq n}K\left (\frac {t-X_i}{h}\right ). \]

1. Compute \(B_n\), the bias of \(\hat {f}_n\). (10 points)
2. Compute the limit of \(B_n/h^2(n)\) as \(n\to \infty \). (10 points)

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables uniform on \([\theta ,\theta +1]\).

1. Find a sufficient statistic for \(\theta \). (5 points)
2. Is your statistic complete? (5 points)
3. Find the maximum likelihood estimator for \(\theta \). (10 points)

Let \(X_1,X_2,X_3\) be i.i.d., each with an exponential distribution with mean 1. Find the joint distribution of \[ Y_1=\frac {X_1}{X_1+X_2},\qquad Y_2=\frac {X_1+X_2}{X_1+X_2+X_3},\qquad Y_3=X_1+X_2+X_3. \] (7 points) In particular, are they independent? (3 points)

Let \(X_1,X_2,\ldots \) be independent random variables with distribution functions \(F_1,F_2,F_3,\ldots \). Let \(Y=\sup _{1\le i<\infty }X_i\).

(a) Show that \(\mathbb {P}\{Y<\infty \}\) is 0 or 1 depending on whether \(\sum _{1\le i<\infty }(1-F_i(x))\) converges for some \(x\). (5 points)

(b) Show that if \(\mathbb {P}\{Y<\infty \}=1\), then \(\prod _{1\le i<\infty }F_i(x)\) converges for all \(x\) and it is the distribution function of \(Y\).

Let \(X_1,X_2,X_3\) be i.i.d., each with an exponential distribution with mean 1. Find the joint distribution of \[ Y_1=\frac {X_1}{X_1+X_2},\qquad Y_2=\frac {X_1+X_2}{X_1+X_2+X_3},\qquad Y_3=X_1+X_2+X_3. \] (7 points) In particular, are they independent? (3 points)

Let \(X_1,X_2,\ldots ,X_n\) denote a random sample from a Poisson distribution with parameter \(\theta >0\).

(a) Show that \((-1)^{X_1}\) is unbiased for \(e^{-2\theta }\), and use the Lehmann–Scheffé theorem to deduce a UMVUE of \(e^{-2\theta }\) based on the sample. (Hint: The conditional distribution of \(X_1\) given \(X_1+\cdots +X_n\) has a simple form.) (5 points)

(b) Argue that the UMVUE in part (a) is a consistent estimator of \(e^{-2\theta }\). (5 points)

Suppose \(X_1,\ldots ,X_n\) is an i.i.d. sample from a normal population with \[ \mathbb {E} X_1=\operatorname {Var}(X_1)=\mu >0. \]

(a) Compute the maximum likelihood estimator \(\hat {\mu }\) of \(\mu \);

(b) Is \(\hat {\mu }\) consistent?

(c) Is \(\hat {\mu }\) asymptotically normal?

Suppose \(X_1,\ldots ,X_n\) is an i.i.d. sample from a normal distribution with mean \(\mu \) and variance 1.

(a) Find the UMVU estimator for \(\mu \). (Prove the optimality criterion.)

(b) Put a \(N(\theta ,\tau ^2)\) prior on \(\mu \) and find the minimax estimator of \(\mu \).

Consider the linear model: \(Y_i=\beta +\epsilon _i\), \(1\le i\le n\). Here, \(\epsilon _i\)’s are i.i.d. \(N(0,\sigma ^2)\), and \(\sigma \) and \(\beta \) are unknown.

(a) Find the least squares estimator of \(\beta \);

(b) Find the UMVU estimator for \(\beta \) (prove the optimality);

(c) Find the UMP test for \(H_0:\beta =\beta _0\) versus \(H_1:\beta =\beta _1\);

(d) Discuss–without proofs–how to find the UMP test for \(H_0:\beta >\beta _0\) vs. \(H_1:\beta \le \beta _0\) from tests of the form in part (c) above.

Let \(X_1,\ldots ,X_n\) be an i.i.d. sample which is uniformly chosen from the interval \((\theta _1,\theta _2)\).

(a) Prove that there are no one-dimensional (i.e., not vector-valued) sufficient statistics for \(h(\theta _1,\theta _2)\) where \(h\) is a one-to-one measurable function.

(b) Is there a one-dimensional (i.e., not vector-valued) sufficient statistics for \(\mu =(\theta _1+\theta _2)/2\)?

Let \(X_1,\ldots ,X_n\) be an i.i.d. sample from a uniform \((0,\theta )\) distribution, where \(\theta >0\).

(a) Find the maximum likelihood estimator \(\hat {\theta }\) for \(\theta \);

(b) Construct a \(100(1-\alpha )\%\) confidence interval for \(\theta \) based on \(\hat {\theta }\).

Suppose \(X\sim \operatorname {POISS}(\theta )\). Put a \(\operatorname {GAMMA}(\alpha ,\beta )\) prior on \(\theta \) and suppose we have the following loss function: \[ \ell (\theta ,a)=\frac {(a-\theta )^2}{\theta }. \] Find the Bayes’ estimator of \(\theta \).

Construct a \((1-\alpha )\) two-sided confidence interval for the correlation coefficient of a bivariate normal distribution. (HINT. This is an exponential family.)

Consider the linear model: \[ Y_{ij}=\beta _i+\epsilon _{ij},\qquad 1\le i\le 2,\qquad 1\le j\le J. \] Suppose \(\epsilon _{ij}\)’s are independent and for some (known) \(a_1\) and \(a_2\), \(\epsilon _{ij}\sim N(0,a_i^2\sigma ^2)\).

(a) Find the U.M.V.U.E.’s of \(\beta _1\) and \(\beta _2\).

(b) Suppose you know that for some unknown \(\beta \), \(\beta _1=\beta _2\). Find the U.M.V.U.E. of \(\beta \).

Suppose \(X\sim \operatorname {Poisson}(\theta )\). Put a \(\operatorname {Gamma}(\alpha ,\beta )\) prior on \(\theta \) and suppose we have the following loss function: \(\ell (\theta ,a)=(a-\theta )^2/\theta \). Find the Bayes’ estimator of \(\theta \).

Construct a \((1-\alpha )\) two-sided confidence interval for the correlation coefficient of a bivariate normal distribution. (HINT. This is an exponential family.)

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with density function \(f\). We assume that \(f'\) is bounded. Let \(K\) be a function satisfying \(\int _{-\infty }^{\infty }K(u)\,du=1\), \(K'(u)\) is bounded and \(K(u)=0\), if \(|u|\geq a\) where \(a\) is a constant. The density \(f\) is estimated by \[ \hat f_n(t)=\frac {1}{nh}\sum _{1\leq i\leq n}K\left (\frac {t-X_i}{h}\right ). \]

Show that \(\hat f_n(t)\) is an almost surely uniformly consistent estimator for \(f\) on \([\alpha ,\beta ]\), \(-\infty <\alpha <\beta <\infty \).

Let \(U_1,U_2,\ldots ,U_n\) be independent identically distributed random variables, uniform on \([0,1]\). Let \(U_{1,n}\le U_{2,n}\le \cdots \le U_{n,n}\) denote the order statistics. Let \(X_1,X_2,\ldots ,X_{n+1}\) be independent identically distributed exponential random variables with \(\mathbb {E} X_1=1\). Define \(S(i)=X_1+\cdots +X_i\).

(a) Prove that the random vectors \[ \{U_{1,n},\ldots ,U_{n,n}\} \quad \text {and}\quad \left \{\frac {S(1)}{S(n+1)},\ldots ,\frac {S(n)}{S(n+1)}\right \} \] have the same distribution.

(b) Compute the asymptotic distribution of \(n(U_{i+3,n}-U_{i,n})\), as \(n\to \infty \) when \(i\) is fixed.

Let \(Y_i=\alpha ^{i/n}+\epsilon _i\), \(1\le i\le n\). We assume that \(\epsilon _1,\epsilon _2,\ldots ,\epsilon _n\) are independent, identically distributed random variables with \(\E \epsilon _i=0\), \(0<\sigma ^2=\operatorname {Var}\epsilon _i<\infty \) and \(\E \epsilon _i^4<\infty \).

(a) Find the least-squares estimator for \(\alpha \).

(b) Show that the estimator is asymptotically normal.

(c) Find an estimator for \(\sigma ^2\).

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with density function \(f\). We assume that \(f'\) is bounded. Let \(K\) be a function satisfying \(\int _{-\infty }^{\infty }K(u)du=1\), \(K'(u)\) is bounded and \(K(u)=0\), if \(|u|\ge a\) where \(a\) is a constant. The density \(f\) is estimated by \[ \hat f_n(t)=\frac {1}{nh}\sum _{1\le i\le n} K\left (\frac {t-X_i}{h}\right ). \] Show that \(\hat f_n(t)\) is an almost surely uniformly consistent estimator for \(f\) on \([\alpha ,\beta ]\), \(-\infty <\alpha <\beta <\infty \).

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with \(\mathbb {P}\{X_i=1\}=p\), \(\mathbb {P}\{X_i=0\}=1-p\).

(a) Compute the maximum likelihood estimator of \(\sigma ^2=p(1-p)\).

(b) Compute the bias, the variance and the mean-square error of the estimator.

(c) Is the estimator asymptotically efficient?

(d) Find the uniformly minimum variance unbiased estimator for \(\sigma ^2\).