Maximum Likelihood Estimation

Let \(\epsilon _{1}, \ldots , \epsilon _{n}\) and \(\eta _{1}, \ldots , \eta _{m}\) be independent normal random variables with \(\mathbb {E}\epsilon _{i} = \mathbb {E}\eta _{j} = 0\) and \(\mathbb {E}\epsilon _{i}^{2} = 1, \mathbb {E}\eta _{j}^{2} = 4\). We have the models

where the \(x_{i}, y_{i}\) are given and non-random.

1. Find the maximum likelihood estimator for \((\beta _{1}, \beta _{2}, \beta _{3} )\).
2. Compute the distribution of the estimator.

Consider the linear model

where the \(\epsilon _{i}\) are independent normals with \(\mathbb {E}\epsilon _{i} = 0\) and \(\mathbb {E}\epsilon _{i}^{2} = i\sigma ^{2}\). Here \(\beta \in \mathbb {R}^{d}\) and \(\sigma ^{2} > 0\) are unknown parameters, while the \(\mathbf {x}_{i} \in \mathbb {R}^{d}\) are all known and deterministic. Compute the maximum likelihood estimator of \((\beta , \sigma ^{2} )\).

Consider the linear model

where the \(\epsilon _{i}\) are independent normal random variables with \(\mathbb {E}\epsilon _{i} = 0\) and \(\mathbb {E}\epsilon _{i}^{2} = \sigma ^{2}\). Assume \(\beta \) and \(\sigma ^{2} > 0\) are deterministic but unknown, while the \(x_{i}\) are deterministic and known. Assume also that \(\sum _i x_i \ne 0\). Find the likelihood ratio test for \(\beta = 0\) versus

Consider the density function

where \(C\) is the constant \[ C = \int _{-\infty }^{\infty } e^{-t^{4}}\,dt. \]

Consider the linear model

where the \(\epsilon _{i}\) are iid with pdf \(f(t)\). Assume that \(\beta \in \mathbb {R}^{d}\) and \(\sigma ^{2} > 0\) are deterministic but unknown, while \(\mathbf {x}_{i} \in \mathbb {R}^{d}\) are deterministic and known.

1. Write down the likelihood function of the observations as a function of \(\beta \) (do not try to find the maximum likelihood estimator).
2. Assume temporarily that the \(\epsilon _{i}\) are iid normal with mean zero and variance one, and write down the likelihood function of the observations as a function of \(\beta \) in that case.
3. If you were to solve for the maximum likelihood estimators in the first two parts would they be the same? (Note you are not being asked to actually solve for the estimators).

Let \(X_1,X_2,\ldots ,X_n\) be independent and identically distributed random variables with density function \[ f(t)= \begin {cases} 0, & \text {if } -\infty <t<\theta ,\\ \dfrac {2\theta }{(\theta +t)^2}, & \text {if } \theta \leq t<\infty , \end {cases} \] for \(\theta >0\) a parameter. Find the maximum likelihood estimator for \(\theta \).

Let \(X_1,\ldots ,X_n\) be independent and identically distributed random variables. The density function of \(X_i\) is \[ g(t;\theta _i)=\theta _i t^{-\theta _i-1}1\{t\geq 1\}. \] Here \(\theta _i>0\) is an unknown parameter. We wish to test \[ H_0:\theta _1=\theta _2=\cdots =\theta _n \] against the alternative that \(H_0\) is not true. Find a test using the generalized likelihood ratio.

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with density function \[ f(t,\theta )= \begin {cases} 0, & \text {if } t\notin [-\theta ,\theta ],\\ \dfrac {5t^4}{2\theta ^5}, & \text {if } -\theta \leq t\leq \theta . \end {cases} \] Find the maximum likelihood estimator for \(\theta \) and compute its bias.

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with density function \[ f(t\theta )=\frac {1}{2}e^{-|t-\theta |}, \qquad -\infty <t<\infty . \] The parameter \(\theta \) could be any real number. Find all maximum likelihood estimators for \(\theta \).

Let \(X_1,X_2,\ldots ,X_n\) be iid with the same pdf as in question 5. Find the form of the rejection region coming from the generalized likelihood ratio test for \(H_0:\theta =\theta _0\) versus \(H_a:\theta \ne \theta _0\).

Let \(X_1,\ldots ,X_n\sim N(\mu _1,1)\) and \(Y_1,\ldots ,Y_m\sim N(\mu _2,1)\), where all random variables are independent. Derive the likelihood ratio test for \[ H_0:\mu _1^2=\mu _2^2 \quad \text {vs.}H_a:\mu _1^2\ne \mu _2^2. \] What is the distribution of the test statistic?

Let \(X_1,\ldots ,X_n\) be iid random variables. The density function of \(X_i\) is \[ g(t;\theta _i)=\theta _i t^{-\theta _i-1}1\{t\geq 1\}. \] Here \(\theta _i>0\) is an unknown parameter. We wish to test \[ H_0:\theta _1=\theta _2=\cdots =\theta _n \] against the alternative that \(H_0\) is not true. Find a test using the generalized likelihood ratio.

A particular gene occurs as one of two alleles (\(A\) and \(a\)), with allele \(A\) having frequency \(\theta \) in the population. In other words, a random copy of the gene is \(A\) with probability \(\theta \) and \(a\) with probability \(1-\theta \). A diploid genotype is just two genes put together (in any particular order) so the probability of each diploid genotype is \(\theta ^2\) for \(AA\), \(2\theta (1-\theta )\) for \(Aa\), and \((1-\theta )^2\) for \(aa\). Suppose we test a random sample of people and find that \(k_1\) of them are \(AA\), \(k_2\) of them are \(Aa\), and \(k_3\) of them are \(aa\). Find the MLE of \(\theta \).

A food processing company packages honey in glass jars. Each jar is supposed to contain 10 fluid ounces of honey. Previous experience suggests that the volume of a randomly selected jar of honey is normally distributed with a known variance of 2 ounces. At a significance level of \(\alpha =.05\) derive the likelihood ratio test for testing the null hypothesis \(H_0:\mu =10\) versus \(H_A:\mu \ne 10\).

A particular gene occurs as one of two alleles (\(A\) and \(a\)), with allele \(A\) having frequency \(\theta \) in the population. In other words, a random copy of the gene is \(A\) with probability \(\theta \) and \(a\) with probability \(1-\theta \). A diploid genotype is just two genes put together (in any particular order) so the probability of each diploid genotype is \(\theta ^2\) for \(AA\), \(2\theta (1-\theta )\) for \(Aa\), and \((1-\theta )^2\) for \(aa\). Suppose we test a random sample of people and find that \(k_1\) of them are \(AA\), \(k_2\) of them are \(Aa\), and \(k_3\) of them are \(aa\). Find the MLE of \(\theta \).

A food processing company packages honey in glass jars. Each jar is supposed to contain 10 fluid ounces of honey. Previous experience suggests that the volume of a randomly selected jar of honey is normally distributed with a known variance of 2 ounces. At a significance level of \(\alpha =.05\) derive the likelihood ratio test for testing the null hypothesis \(H_0:\mu =10\) versus \(H_A:\mu \ne 10\).

Let \(X_1,X_2,\ldots ,X_n\) be iid random variables with density function \[ h(t;\eta )=e^{-(t-\eta )}I\{t\geq \eta \}, \] where \(I\) indicates the indicator function. We want to test \(H_0:\eta =\eta _0\) against \(H_A:\eta \ne \eta _0\). Find a test using the likelihood method and compute the power function.

Let \(X_1,X_2,\ldots ,X_n\) be iid random variables with density \[ h(t;\theta )=\theta t^{\theta -1}I\{0\leq t\leq 1\}, \qquad \theta >0. \] Find the maximum likelihood estimator of \(\theta \) and compute the asymptotic variance of the maximum likelihood estimator.

Let \(X_1,X_2,\ldots ,X_n\) be iid with density \[ h(t;\theta )=\frac {1}{2}I\{\theta -1\leq t\leq \theta +1\}. \] Find all maximum likelihood estimators of \(\theta \). Show that if \(\hat \theta _n\) is a maximum likelihood estimator then \(\mathbb {E}[\hat \theta _n]\to \theta \) as \(n\to \infty \).

Let \(X_1,X_2,\ldots ,X_n\) be iid \(N(\mu _1,\sigma _1^2)\) and \(Y_1,Y_2,\ldots ,Y_m\) be iid \(N(\mu _2,\sigma _2^2)\). The \(X\) and the \(Y\) are also independent of each other. Assume that \(\sigma _1^2,\sigma _2^2\) are known while \(\mu _1,\mu _2\) are unknown. We wish to test \(H_0:\mu _1=\mu _2\) against \(H_A:\mu _1\ne \mu _2\). Find a test using the likelihood method. Compute the distribution of the likelihood ratio test.

Let \(p_n\) and \(q_n\) denote two probability mass functions on \(\mathbb {Z}^n\), for every integer \(n\geq 1\). That is, for all \(n\geq 1\):

1. \(p_n(\mathbf {x})\geq 0\) and \(q_n(\mathbf {x})\geq 0\) for all \(\mathbf {x}\in \mathbb {Z}^n\); and
2. \(\sum _{\mathbf {x}\in \mathbb {Z}^n}p_n(\mathbf {x}) =\sum _{\mathbf {x}\in \mathbb {Z}^n}q_n(\mathbf {x})=1\).

Now suppose \(\{X_i\}_{i=1}^{\infty }\) is a collection of random variables such that \((X_1,\ldots ,X_n)\) has mass function \(p_n\) for every \(n\geq 1\). That is, \[ \mathbb {P}\{X_1=x_1,\ldots ,X_n=x_n\} = p_n(x_1,\ldots ,x_n) \qquad \text {for all }(x_1,\ldots ,x_n)\in \mathbb {Z}^n. \]

1. Prove that \(p_n(X_1,\ldots ,X_n)>0\) a.s. for all \(n\geq 1\).
2. Prove that \(\{Z_n\}_{n=1}^{\infty }\) is a martingale, where, for all \(n\geq 1\), \[ Z_n:= \frac {q_n(X_1,\ldots ,X_n)}{p_n(X_1,\ldots ,X_n)}, \] almost surely on the \(\mathbb {P}\)-measure-one event \(\{p_n(X_1,\ldots ,X_n)>0\}\), and \(Z_n:=0\) otherwise.
3. Prove that \(\lim _{n\to \infty }Z_n\) exists a.s. and is finite a.s. (This is a starting point for likelihood-ratio testing in classical statistics.)

If a random sample of size \(n\) is taken from a distribution having pdf \(f(x;\theta )=2x/\theta ^2\), \(0<x\leq \theta \) and zero elsewhere, find:

(a) the MLE \(\hat \theta \) for \(\theta \),

(b) the constant \(c\) so that \(\mathbb {E}[c\hat \theta ]=\theta \),

(c) the MLE for the median of the distribution.

Let \(Y_1,Y_2,\ldots ,Y_n\) be independent random variables, each with the \(N(\beta x_i,\gamma ^2x_i^2)\) distribution, where the numbers \(x_1,x_2,\ldots ,x_n\) are known, not all are equal, and none of them is zero. Suppose \(\beta \) and \(\gamma \) are unknown. Find the maximum likelihood estimators of \(\beta \) and \(\gamma ^2\).

Let \(X_1,X_2,\ldots ,X_n\) be a random sample from a Gamma\((\alpha ,\theta )\) distribution, with \(\alpha \) known and \(\theta \) unknown. Find the Fisher information of this distribution and show that the MLE of \(\theta \) is an efficient estimator of \(\theta \).

Suppose that an engineer wishes to compare the number of complaints per week filed by union stewards for two different shifts at a manufacturing plant. It is known that the number of complaints per week on the \(i\)th shift has a Poisson distribution with mean \(\theta _i\) for \(i=1,2\). One hundred independent observations on the number of complaints gave means \(\bar x=20\) for shift 1 and \(\bar y=22\) for shift 2. Using this data test \(H_0:\theta _1=\theta _2\) versus \(H_1:\theta _1\ne \theta _2\) by the likelihood ratio method, with \(\alpha =0.01\). (Hint: First find the likelihood ratio test statistic for \(\lambda \) and then use the distribution of \(-2\log \lambda \).)

Assume that \(X_1,\ldots ,X_n\) are a random sample with common pdf \(f(x;\theta )=(\theta +1)x^{-\theta -2}\) for \(x>1\), and zero otherwise. Is the method of moments estimator for \(\theta \) the same as the maximum likelihood estimator for \(\theta \)?

Let \(Y_1<Y_2<\cdots <Y_5\) be the order statistics of a random sample of size \(n=5\) from a distribution with pdf \(f(x;\theta )=e^{-|x-\theta |}/2\), \(-\infty <x<\infty \), with \(\theta \) unknown. Find the likelihood ratio test \(\lambda \) for testing \(H_0:\theta =\theta _0\) versus \(H_1:\theta \ne \theta _0\).

Let \(X_1,\ldots ,X_n\) be a random sample from the \(N(\theta ,1)\) distribution, \(\theta \) unknown. We wish to test \(H_0:\theta =\theta _0\) versus \(H_A:\theta <\theta _0\). Find a test using the generalized likelihood ratio \(\lambda \).

Let \(X_1,X_2,\ldots ,X_n\) be independent, random variables with density functions \[ h(t;\theta _i)= \begin {cases} \dfrac {1}{\theta _i}e^{-t/\theta _i}, & \text {if } 0\leq t<\infty ,\\ 0, & \text {if } -\infty <t<0. \end {cases} \] We wish to test \(H_0:\theta _1=\theta _2=\cdots =\theta _n\) against the alternative that \(H_0\) is not true.

(a) Compute \(\lambda \), the generalized likelihood ratio and explain the application of the generalized likelihood test.

(b) Provide a large sample approximation for \(-2\log \lambda \) under \(H_0\).

Let \(X_1,X_2,\ldots ,X_n\) be independent, identically distributed random variables with density function \[ h(t;\sigma )=\frac {1}{(2\pi \sigma ^2)^{1/2}} e^{-(t-1)^2/(2\sigma ^2)}. \] We wish to test \(H_0:\sigma =\sigma _0\) against \(H_A:\sigma \ne \sigma _0\).

(a) Find a test, using the generalized likelihood ratio \(\lambda \).

(b) Provide a large sample approximation for \(-2\log \lambda \) under \(H_0\).

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed normal \(N(\mu _1,\sigma ^2)\) random variables. Let \(Y_1,Y_2,\ldots ,Y_m\) be independent identically distributed normal \(N(\mu _2,\sigma ^2)\) random variables. The two samples are independent. The parameters \(\mu _1,\mu _2,\sigma ^2\) are unknown. We wish to test \(H_0:\mu _1=\mu _2\). Show that the two sample t-test and the likelihood ratio test are equivalent.

Find the maximum likelihood estimators of \(\theta \) and \(\alpha \), based on a random sample \(X_1,X_2,\ldots ,X_n\) from the distribution with density \[ f(x;\theta ,\alpha )=\frac {\alpha }{\theta ^\alpha }x^{\alpha -1}, \qquad 0\leq x\leq \theta . \] Here \(\theta \) and \(\alpha \) are positive.

1000 individuals were classified according to sex and to whether or not they were color-blind, with the following results: male normal 442; female normal 514; male color-blind 38; and female color-blind 6. According to a genetic model, the four relative frequencies should be of the form \(p/2\); \(p^2/2+p(1-p)\); \((1-p)/2\); and \((1-p)^2/2\), respectively. Suppose we want to test whether the data fit the model.

(a) Find the maximum likelihood estimator \(\hat p\) of \(p\).

(b) Describe a size \(\alpha =0.10\) test of the null hypothesis that the data fit the model, assuming that \(\hat p=0.913\). Specify the critical region as precisely as possible, perhaps in terms of a quantile of a certain \(\chi ^2\) distribution.

Let \(X_1,\ldots ,X_n\) be a random sample from \(\operatorname {Uniform}[0,\theta ]\), where \(\theta \) is a positive unknown. Derive the generalized likelihood ratio (GLR) test of \(H_0:\theta =\theta _0\) vs. \(H_a:\theta \ne \theta _0\).

Let \(X_1,X_2,\ldots \) be i.i.d. integer-valued random variables with probability mass function \(f(a):=\Prob \{X_1=a\}\) for all \(a\in \mathbb {Z}\), and suppose \(f(a)>0\) for all \(a\in \mathbb {Z}\). Now let \(g\) be another probability mass function on \(\mathbb {Z}\) and define the likelihood ratio, \[ \Lambda _n:=\prod _{j=1}^{n}\frac {g(X_j)}{f(X_j)}. \]

(a) Consider the log-likelihood \(\log \Lambda _n\), where “log” denotes the natural logarithm. Prove that \[ \lim _{n\to \infty }n^{-1}\log \Lambda _n =\mathbb {D}(g\,\|\,f) \qquad \text {a.s.,} \] where \[ \mathbb {D}(g\,\|\,f):= \sum _{a=-\infty }^{\infty } f(a)\log \left [\frac {g(a)}{f(a)}\right ], \] provided that \[ \mathbb {E}(g\,\|\,f):= \sum _{a=-\infty }^{\infty } \left |\log \left [\frac {g(a)}{f(a)}\right ]\right |f(a)<\infty . \]

(b) Verify that \(\log x\leq x-1\) for all \(x>0\), and that the inequality is strict except when \(x=1\). Use this to prove the following [Gibbs’ inequality]: \(\mathbb {D}(g\,\|\,f)<0\).

(c) Conclude from the preceding that if \(\mathbb {E}(g\,\|\,f)<\infty \), then \(\Lambda _n\to 0\) a.s. very rapidly as \(n\to \infty \).

This fact plays an important role in statistics.

Let \(X_1,\ldots ,X_n\) be independent, identically distributed random variables with density function \[ h(t,\theta )= \begin {cases} 0, & \text {if } -\infty <t<0,\\ \theta (t+1)^{-\theta -1}, & \text {if } 0\leq t<\infty , \end {cases} \] \(\theta >0\).

(a) Find the maximum likelihood estimator for \(\theta \).

(b) Is the estimator unbiased?

(c) Find the asymptotic variance of the maximum likelihood estimator for \(\theta \).

Let \(X_1,\ldots ,X_n\) be a random sample from \(N(\theta ,\theta )\), where \(\theta >0\). In particular, the population mean and the population variance are equal but unknown.

(a) Find the method-of-moments estimator of \(\theta \) based on the second sample moment.

(b) Find the maximum likelihood estimator of \(\theta \).

Let \(X_1,\ldots ,X_n\) be a random sample from \(N(\mu ,\sigma _0^2)\), where \(\sigma _0^2\) is known. Find the generalized likelihood ratio test of size \(\alpha \) of \(H_0:\mu \geq \mu _0\) vs. \(H_a:\mu <\mu _0\).

Let \(y_i=\alpha x_i+\epsilon _i\), \(1\leq i\leq n\), where \(\epsilon _1,\epsilon _2,\ldots ,\epsilon _n\) are independent identically distributed normal \(N(0,\sigma ^2)\) random variables.

a. Determine the maximum likelihood estimators for \(\alpha \) and \(\sigma ^2\). (5 points)

b. Compute the distribution of the maximum likelihood estimator for \(\alpha \). (5 points)

c. Compute the distribution of the maximum likelihood estimator for \(\sigma ^2\). (5 points)

Let \(X_1,X_2,\ldots ,X_n\) be independent, identically distributed random variables with density function \[ f(t;\theta )= \begin {cases} 0, & \text {if } t<\theta ,\\ e^{-(t-\theta )}, & \text {if } t>\theta . \end {cases} \]

a. Find a moment estimator for \(\theta \). (5 points)

b. Find the maximum likelihood estimator for \(\theta \). (5 points)

c. Determine the asymptotic efficiency between the two estimators. (5 points)

The number of customers entering a store on a given day is a Poisson random variable with parameter \(\theta \). The money spent by a customer is uniformly distributed on \([0,\eta ]\). The number of customers and the money spent in the store are independent. We observe \(Z_1,Z_2,\ldots ,Z_n\), the total spending (the money collected by the store) on \(n\) days. We can assume that \(Z_1,Z_2,\ldots ,Z_n\) are independent.

a. Provide estimators for \(\theta \) and \(\eta \) using the method of moments. (10 points)

b. Provide estimators for \(\theta \) and \(\eta \) using the likelihood method. (10 points)

Let \(X_1,\ldots ,X_n\) be independent identically distributed Poisson random variables with parameter \(\theta _1\). Let \(Y_1,\ldots ,Y_m\) be independent identically distributed Poisson random variables with parameter \(\theta _2\). We assume that the two samples are independent. We wish to test \(H_0:\theta _1=\theta _2\) against the alternative that \(H_0\) is not true.

a. Derive the likelihood ratio test. (5 points)

b. Provide a large sample approximation for the rejection region. (2 points)

Let \(X_1,\ldots ,X_N\) be independent random variables. The distibution of \(X_i\) is binomial \((n_i,p_i)\). We wish to test \(H_0:p_1=p_2=\cdots =p_N\) against the alternative that \(H_0\) is not true.

a. Derive the likelihood ratio test. (5 points)

b. Provide a large sample approximation for the rejection region. (2 points)

Let \(X_i\) be a random sample of size \(n\) with pdf \(f(x;\eta )=e^{-(x-\eta )}\) (\(x>\eta \), \(\eta >0\)).

1a) (5 points) Find the MLE \(\hat \eta \) and compute its pdf. Do you recognize it?

1b) (5 points) Is \(\hat \eta \) unbiased, asymptotically unbiased, MSE consistent, simple consistent?

Let \(X_i\) be a random sample of size \(n\) with pdf \(f(x;\theta )=\frac {1}{\theta }e^{-(x-\theta )/\theta }\) (\(x>\theta \), \(\theta >0\)).

2a) (5 points) What is the MME of \(\theta \)? Is it a reasonable estimator?

2b) (5 points) What is the MLE of \(\theta \)?

(10 points) Let \(X_i\sim \operatorname {POI}(\mu )\) be a random sample of size \(n\). Find the MLE of \(e^{-\mu }\). Is it unbiased, asymptotically unbiased, MSE consistent, simple consistent? [Hint: The MGF of \(\operatorname {POI}(\mu )\) is \(e^{\mu (e^t-1)}\).]

Consider the pdf \(f(x;\theta )=\theta x^{-(\theta +1)}\) (\(x\geq 1\)). Let \(X_i\) be a random sample of size \(n\) with pdf \(f(x;\theta _1)\). Let \(Y_i\) be a random sample of size \(m\) with pdf \(f(x;\theta _2)\). The two sets of random variables are independent. We wish to test \[ H_0:\theta _1=\theta _2 \quad \text {against}\quad H_a:\theta _1\ne \theta _2. \]

6a) (5 points) If one considers just the \(X\)-data, find the MLE \(\hat \theta _1\).

6b) (5 points) Derive a formula for the critical region obtained using the generalized likelihood ratio method.

Let \(X_i\sim \operatorname {EXP}(\theta )\) be a random sample of size \(n\).

7a) (5 points) Find the MGF of \(\frac {2n\bar X}{\theta }\) and identify its distribution.

7b) (10 points) Derive the generalized likelihood ratio test of \(H_0:\theta =\theta _0\) against \(H_a:\theta \ne \theta _0\).

7c) (5 points) The following data are times (in hours) between failures of air conditioning equipment in a particular airplane: 74, 57, 48, 29, 502, 12, 70, 21, 29, 386, 59, 27, 153, 26, 326. Test \(H_0:\theta =125\) against \(H_a:\theta \ne 125\).

Let \(X_i\) be a random sample of size \(n\) with pdf \(f(x;\theta )=\frac {2x}{\theta ^2}\) (\(0<x<\theta \)).

8a) (5 points) Use the MME \(\tilde \theta \) to find an unbiased estimator of \(\theta \).

8b) (10 points) Use the MLE \(\hat \theta \) to find another unbiased estimator of \(\theta \).

8c) (5 points) Using the factorization criterion, find one sufficient statistic for \(\theta \). Which of the two unbiased estimators you have found has a lower variance?

8d) (5 points) Calculate \(\operatorname {Var}(\tilde \theta )\) and \(\operatorname {Var}(\hat \theta )\). Does this confirm your answer?

Let \(X_1,X_2,\ldots ,X_n\) and \(Y_1,Y_2,\ldots ,Y_m\) be two independent random samples. We assume that \(X_1,X_2,\ldots ,X_n\) are independent identically distributed exponential \((\lambda )\) random variables and \(Y_1,Y_2,\ldots ,Y_m\) are independent identically distributed exponential \((\mu )\) random variables. We want to test \(H_0:\lambda =\mu \) against the alternative \(H_A:\lambda \ne \mu \).

(i) Find \(T_{m,n}\) the likelihood ratio test. (7 points)

(ii) Prove that \(-2\log T_{m,n}\) converges in distribution under the null hypothesis when \(\min (m,n)\to \infty \). (7 points)

(iii) Show that the likelihood ratio test is asymptotically consistent and prove that the power goes to 1 as \(\min (m,n)\to \infty \). (7 points)

Let \(X_1,X_2,\ldots ,X_n\) and \(Y_1,Y_2,\ldots ,Y_m\) be two independent random samples. We assume that \(X_1,X_2,\ldots ,X_n\) are independent identically distributed normal \((\mu _1,\sigma ^2)\) random variables and \(Y_1,Y_2,\ldots ,Y_m\) are independent identically distributed normal \((\mu _2,\sigma ^2)\) random variables (\(\sigma ^2\) is unknown). We want to test \(H_0:\mu _1=\mu _2\) against the alternative \(H_A:\mu _1\ne \mu _2\).

Show that the likelihood ratio test and the two sample t-test are equivalent. (7 points)

Let Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with distribution function \[ F(t)=\left (\frac {t}{\theta }\right )^2 \quad \text {if } 0\leq t\leq \theta . \]

(i) Find the maximum likelihood estimator for \(\theta \). (7 points)

(ii) Compute the limit distribution of the maximum likelihood estimator. (7 points)

Let \((X_1,\ldots ,X_m)\) be a multinomial random vector with parameters \(N\) and \(p_1,p_2,\ldots ,p_m\). Note that \(N=X_1+\cdots +X_m\) and \(p_1+p_2+\cdots +p_m=1\). Find the maximum likelihood estimators for \(p_1,p_2,\ldots ,p_m\). (7 points)

Let \(X_1, X_2, \ldots \) be an i.i.d. sample from the uniform distribution on \([\theta , 1+\theta ]\), where \(\theta > 0\) is unknown. Is there an unique maximum likelihood estimator of \(\theta \) that is based on \(\{X_i\}_{i=1}^{\infty }\)? Prove or disprove.

Is the MLE a consistent estimator of \(\theta \)? Prove or disprove.

Develop carefully the likelihood ratio test for \(H_0:\theta =1\) versus \(H_1:\theta =2\). Discuss the various optimality properties—if any—of your test.

Let \(X_1,\ldots ,X_n\) be independent, identically distributed random variables with density function \[ f(t,\theta _1)= \begin {cases} 0, & \text {if } t\notin [1,\infty ),\\ \theta _1 t^{-\theta _1-1}, & \text {if } 1\leq t<\infty . \end {cases} \] Let \(Y_1,\ldots ,Y_m\) be independent, identically distributed random variables with density function \[ f(t,\theta _2)= \begin {cases} 0, & \text {if } t\notin [1,\infty ),\\ \theta _2 t^{-\theta _2-1}, & \text {if } 1\leq t<\infty . \end {cases} \] We wish to test \(H_0:\theta _1=\theta _2\) against the alternative that \(\theta _1\neq \theta _2\). The two samples are independent.

(a) Find a test using the likelihood ratio.

(b) Provide a large sample approximation for the likelihood ratio test.

Let \(X_1,\ldots ,X_n\) be independent random variables with density functions \[ f(t,\theta _i)= \begin {cases} 0, & \text {if } t\notin [0,\infty ),\\ \dfrac {1}{\theta _i}e^{-t/\theta _i}, & \text {if } 0\leq t<\infty . \end {cases} \] We wish to test \(H_0:\theta _1=\theta _2=\cdots =\theta _n\) against the alternative that \(H_0\) is not true.

(a) Find a test using the likelihood ratio.

(b) Provide a large sample approximation for the likelihood ratio test.

Let \(\epsilon _i\), \(1\leq i\leq n\), be independent identically distributed normal \(N(0,\sigma ^2)\) random variables and consider the model \[ y_i=i\alpha +\gamma i^2+\epsilon _i,\qquad 1\leq i\leq n. \] We observe \(y_i\), \(1\leq i\leq n\).

(a) Find the maximum likelihood estimators for \(\alpha \), \(\gamma \), and \(\sigma ^2\).

(b) Compute the joint distribution of the maximum likelihood estimators for \((\alpha ,\gamma )\).

Let \(X_1,X_2,\ldots ,X_n\) be independent, identically distributed normal \(N(\mu ,\sigma ^2)\) random variables. We wish to test \(H_0:\mu =\mu _0\) against \(H_A:\mu \neq \mu _0\) (\(\sigma \) is unknown). Show that the t-test and the likelihood tests are equivalent.

Let \(X_1,X_2,\ldots ,X_n\) be independent, identically distributed uniform \((0,\theta )\) random variables with common density function \[ f(t)= \begin {cases} 0, & \text {if } t\notin [0,\theta ],\\ 1/\theta , & \text {if } t\in [0,\theta ]. \end {cases} \] (a) Find the maximum likelihood estimator for \(\theta \).

(b) Find a \(1-\alpha \) equal-tail confidence interval for \(\theta \).

Let \(X_1,X_2,\ldots ,X_n\) be independent exponential \((\theta _i)\) random variables, i.e. the density of \(X_i\) is \[ f(t,\theta _i)= \begin {cases} 0, & \text {if } -\infty <t<0,\\ \dfrac {1}{\theta _i}e^{-t/\theta _i}, & \text {if } 0\leq t<\infty . \end {cases} \] We wish to test \(H_0:\theta _1=\theta _2=\cdots =\theta _n\) against the alternative that \(H_0\) is not true.

(a) Find the likelihood ratio test.

(b) Show that the likelihood ratio test is distribution free under \(H_0\).

(c) Show that the likelihood ratio test is equivalent with the \(\chi ^2\) test under \(H_0\).

(d) Approximate the critical values of the likelihood ratio test under \(H_0\).

Let \(y_i=\alpha x_i+\epsilon _i\), where \(\epsilon _i\), \(1\leq i\leq n\), are independent, identically distributed normal \(N(0,\sigma ^2)\) random variables.

(a) Find the maximum likelihood estimators for \(\alpha \) and \(\sigma ^2\).

(b) Find the distributions of the maximum likelihood estimators.

(c) Compute the least-squares estimator for \(\alpha \).

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed exponential \((\lambda )\) random variables. Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed exponential \((\mu )\) random variables. The two samples are independent. We wish to test \(H_0:\theta =\mu \) against the alternative that \(H_0\) is not true.

(a) Find the likelihood ratio test.

(b) Find the asymptotic distribution of the likelihood ratio under \(H_0\).

(c) Find an equal-tail confidence interval for \(\tau =\lambda /\mu \).

Let \(X_1,X_2,\ldots ,X_n\) be independent random variables. We assume that \(X_i\) has a Poisson\((\lambda _i)\) distribution. We wish to test \[ H_0:\lambda _1=\lambda _2=\cdots =\lambda _n \] against the alternative \[ H_A:H_0\text { is not true.} \]

1. Show that the likelihood ratio test and the \(\chi ^2\) tests are equivalent under \(H_0\). (15 points)
2. Explain why and how the normal as well as the \(\chi ^2\) distributions can be used to approximate the critical values. (5 points)

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables uniform on \([\theta ,\theta +1]\).

1. Find a sufficient statistic for \(\theta \). (5 points)
2. Is your statistic complete? (5 points)
3. Find the maximum likelihood estimator for \(\theta \). (10 points)

Let \(X_1,X_2,\ldots ,X_n\) and \(Y_1,Y_2,\ldots ,Y_m\) be independent random samples of independent observations. The distribution of \(X_i\) is normal \(N(\mu _1,\sigma _1^2)\) and the distribution of \(Y_i\) is normal \(N(\mu _2,\sigma _2^2)\). The values of \(\sigma _1\) and \(\sigma _2\) are known. We wish to test \(H_0:\mu _1=\mu _2\) against \(H_A:\mu _1\neq \mu _2\). Show that the likelihood ratio test and the t-test are equivalent under \(H_0\). (20 points)

Let \(X_1,\ldots ,X_n\) be a random sample from the \(N(\mu ,\sigma ^2)\) distribution, with both \(\mu \) and \(\sigma ^2\) unknown.

(a) Derive the likelihood ratio test for \(H:\sigma ^2=\sigma _0^2\) against all alternatives. Here \(\sigma _0^2\) is a known positive constant. (5 points)

(b) Do the same for \(H:\sigma ^2\ge \sigma _0^2\). (5 points)

Let \(X_1,X_2,\ldots ,X_n\) be independent Poisson random variables with parameters \(\theta _1,\theta _2,\ldots ,\theta _n\). We wish to test \(H_o:\theta _1=\theta _2=\cdots =\theta _n\) against the alternative that \(H_o\) is not true.

(a) Find the likelihood ratio test. (5 points)

(b) Show that the likelihood ratio is asymptotically normal (after centralizing and normalizing) under \(H_o\). (5 points)

Suppose \(X_1,\ldots ,X_n\) is an i.i.d. sample from a normal population with \[ \mathbb {E} X_1=\operatorname {Var}(X_1)=\mu >0. \]

(a) Compute the maximum likelihood estimator \(\hat {\mu }\) of \(\mu \);

(b) Is \(\hat {\mu }\) consistent?

(c) Is \(\hat {\mu }\) asymptotically normal?

Let \(X_1,\ldots ,X_n\) be an i.i.d. sample from a uniform \((0,\theta )\) distribution, where \(\theta >0\).

(a) Find the maximum likelihood estimator \(\hat {\theta }\) for \(\theta \);

(b) Construct a \(100(1-\alpha )\%\) confidence interval for \(\theta \) based on \(\hat {\theta }\).

Suppose \(X_1,\cdots ,X_m\) are independent with \(X_j\sim \operatorname {Binomial}(n_j,p_j)\).

(a) Find the UMVUE’s of \(p_1,\cdots ,p_m\).

(b) Suppose you know that \(p_1=p_2=\cdots =p_m\). Let \(p\) denote this common (but unknown) value. Find the UMVUE of \(p\).

(c) Find the likelihood ratio statistic, \(\lambda \), for \(H_0:p_1=\cdots =p_m\) versus \(H_1:p_i\ne p_j\), for some \(i\) and \(j\).

(d) It can be shown that \(2\log \lambda \) is approximately the same as the usual \(\chi ^2\) statistics. Using this fact, find an approximate test for \(H_0\) vs \(H_1\) above.

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables, uniformly distributed on \([0,\theta ]\), \(\theta >\theta _0\). We want to test \(H_0:\theta =\theta _0\) against \(H_A:\theta >\theta _0\).

(a) Find the uniformly most powerful test. (You must prove your claim.)

(b) Show that the uniformly most powerful test and the likelihood ratio test are equivalent,

(c) Compute the power function of the most powerful test.

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with \(\mathbb {P}\{X_i=1\}=p\), \(\mathbb {P}\{X_i=0\}=1-p\).

(a) Compute the maximum likelihood estimator of \(\sigma ^2=p(1-p)\).

(b) Compute the bias, the variance and the mean-square error of the estimator.

(c) Is the estimator asymptotically efficient?

(d) Find the uniformly minimum variance unbiased estimator for \(\sigma ^2\).