Likelihood Ratio Tests

Consider the linear model

where the \(\epsilon _{i}\) are independent normal random variables with \(\mathbb {E}\epsilon _{i} = 0\) and \(\mathbb {E}\epsilon _{i}^{2} = \sigma ^{2}\). Assume \(\beta \) and \(\sigma ^{2} > 0\) are deterministic but unknown, while the \(x_{i}\) are deterministic and known. Assume also that \(\sum _i x_i \ne 0\). Find the likelihood ratio test for \(\beta = 0\) versus

Let \(X_1,\ldots ,X_n\) be independent and identically distributed random variables. The density function of \(X_i\) is \[ g(t;\theta _i)=\theta _i t^{-\theta _i-1}1\{t\geq 1\}. \] Here \(\theta _i>0\) is an unknown parameter. We wish to test \[ H_0:\theta _1=\theta _2=\cdots =\theta _n \] against the alternative that \(H_0\) is not true. Find a test using the generalized likelihood ratio.

Let \(X_1,X_2,\ldots ,X_n\) be iid with the same pdf as in question 5. Find the form of the rejection region coming from the generalized likelihood ratio test for \(H_0:\theta =\theta _0\) versus \(H_a:\theta \ne \theta _0\).

Let \(X_1,\ldots ,X_n\sim N(\mu _1,1)\) and \(Y_1,\ldots ,Y_m\sim N(\mu _2,1)\), where all random variables are independent. Derive the likelihood ratio test for \[ H_0:\mu _1^2=\mu _2^2 \quad \text {vs.}H_a:\mu _1^2\ne \mu _2^2. \] What is the distribution of the test statistic?

Let \(X_1,\ldots ,X_n\) be iid random variables. The density function of \(X_i\) is \[ g(t;\theta _i)=\theta _i t^{-\theta _i-1}1\{t\geq 1\}. \] Here \(\theta _i>0\) is an unknown parameter. We wish to test \[ H_0:\theta _1=\theta _2=\cdots =\theta _n \] against the alternative that \(H_0\) is not true. Find a test using the generalized likelihood ratio.

A food processing company packages honey in glass jars. Each jar is supposed to contain 10 fluid ounces of honey. Previous experience suggests that the volume of a randomly selected jar of honey is normally distributed with a known variance of 2 ounces. At a significance level of \(\alpha =.05\) derive the likelihood ratio test for testing the null hypothesis \(H_0:\mu =10\) versus \(H_A:\mu \ne 10\).

A food processing company packages honey in glass jars. Each jar is supposed to contain 10 fluid ounces of honey. Previous experience suggests that the volume of a randomly selected jar of honey is normally distributed with a known variance of 2 ounces. At a significance level of \(\alpha =.05\) derive the likelihood ratio test for testing the null hypothesis \(H_0:\mu =10\) versus \(H_A:\mu \ne 10\).

Let \(X_1,X_2,\ldots ,X_n\) be iid random variables with density function \[ h(t;\eta )=e^{-(t-\eta )}I\{t\geq \eta \}, \] where \(I\) indicates the indicator function. We want to test \(H_0:\eta =\eta _0\) against \(H_A:\eta \ne \eta _0\). Find a test using the likelihood method and compute the power function.

Let \(X_1,X_2,\ldots ,X_n\) be iid with density \[ h(t;\theta )=\frac {1}{2}I\{\theta -1\leq t\leq \theta +1\}. \] Find all maximum likelihood estimators of \(\theta \). Show that if \(\hat \theta _n\) is a maximum likelihood estimator then \(\mathbb {E}[\hat \theta _n]\to \theta \) as \(n\to \infty \).

Let \(X_1,X_2,\ldots ,X_n\) be iid \(N(\mu _1,\sigma _1^2)\) and \(Y_1,Y_2,\ldots ,Y_m\) be iid \(N(\mu _2,\sigma _2^2)\). The \(X\) and the \(Y\) are also independent of each other. Assume that \(\sigma _1^2,\sigma _2^2\) are known while \(\mu _1,\mu _2\) are unknown. We wish to test \(H_0:\mu _1=\mu _2\) against \(H_A:\mu _1\ne \mu _2\). Find a test using the likelihood method. Compute the distribution of the likelihood ratio test.

Let \(p_n\) and \(q_n\) denote two probability mass functions on \(\mathbb {Z}^n\), for every integer \(n\geq 1\). That is, for all \(n\geq 1\):

1. \(p_n(\mathbf {x})\geq 0\) and \(q_n(\mathbf {x})\geq 0\) for all \(\mathbf {x}\in \mathbb {Z}^n\); and
2. \(\sum _{\mathbf {x}\in \mathbb {Z}^n}p_n(\mathbf {x}) =\sum _{\mathbf {x}\in \mathbb {Z}^n}q_n(\mathbf {x})=1\).

Now suppose \(\{X_i\}_{i=1}^{\infty }\) is a collection of random variables such that \((X_1,\ldots ,X_n)\) has mass function \(p_n\) for every \(n\geq 1\). That is, \[ \mathbb {P}\{X_1=x_1,\ldots ,X_n=x_n\} = p_n(x_1,\ldots ,x_n) \qquad \text {for all }(x_1,\ldots ,x_n)\in \mathbb {Z}^n. \]

1. Prove that \(p_n(X_1,\ldots ,X_n)>0\) a.s. for all \(n\geq 1\).
2. Prove that \(\{Z_n\}_{n=1}^{\infty }\) is a martingale, where, for all \(n\geq 1\), \[ Z_n:= \frac {q_n(X_1,\ldots ,X_n)}{p_n(X_1,\ldots ,X_n)}, \] almost surely on the \(\mathbb {P}\)-measure-one event \(\{p_n(X_1,\ldots ,X_n)>0\}\), and \(Z_n:=0\) otherwise.
3. Prove that \(\lim _{n\to \infty }Z_n\) exists a.s. and is finite a.s. (This is a starting point for likelihood-ratio testing in classical statistics.)

Suppose that an engineer wishes to compare the number of complaints per week filed by union stewards for two different shifts at a manufacturing plant. It is known that the number of complaints per week on the \(i\)th shift has a Poisson distribution with mean \(\theta _i\) for \(i=1,2\). One hundred independent observations on the number of complaints gave means \(\bar x=20\) for shift 1 and \(\bar y=22\) for shift 2. Using this data test \(H_0:\theta _1=\theta _2\) versus \(H_1:\theta _1\ne \theta _2\) by the likelihood ratio method, with \(\alpha =0.01\). (Hint: First find the likelihood ratio test statistic for \(\lambda \) and then use the distribution of \(-2\log \lambda \).)

Assume that \(X_1,\ldots ,X_n\) are a random sample with common pdf \(f(x;\theta )=(\theta +1)x^{-\theta -2}\) for \(x>1\), and zero otherwise. Is the method of moments estimator for \(\theta \) the same as the maximum likelihood estimator for \(\theta \)?

Let \(Y_1<Y_2<\cdots <Y_5\) be the order statistics of a random sample of size \(n=5\) from a distribution with pdf \(f(x;\theta )=e^{-|x-\theta |}/2\), \(-\infty <x<\infty \), with \(\theta \) unknown. Find the likelihood ratio test \(\lambda \) for testing \(H_0:\theta =\theta _0\) versus \(H_1:\theta \ne \theta _0\).

Let \(X_1,\ldots ,X_n\) be a random sample from the \(N(\theta ,1)\) distribution, \(\theta \) unknown. We wish to test \(H_0:\theta =\theta _0\) versus \(H_A:\theta <\theta _0\). Find a test using the generalized likelihood ratio \(\lambda \).

Let \(X_1,X_2,\ldots ,X_n\) be independent, random variables with density functions \[ h(t;\theta _i)= \begin {cases} \dfrac {1}{\theta _i}e^{-t/\theta _i}, & \text {if } 0\leq t<\infty ,\\ 0, & \text {if } -\infty <t<0. \end {cases} \] We wish to test \(H_0:\theta _1=\theta _2=\cdots =\theta _n\) against the alternative that \(H_0\) is not true.

(a) Compute \(\lambda \), the generalized likelihood ratio and explain the application of the generalized likelihood test.

(b) Provide a large sample approximation for \(-2\log \lambda \) under \(H_0\).

Let \(X_1,X_2,\ldots ,X_n\) be independent, identically distributed random variables with density function \[ h(t;\sigma )=\frac {1}{(2\pi \sigma ^2)^{1/2}} e^{-(t-1)^2/(2\sigma ^2)}. \] We wish to test \(H_0:\sigma =\sigma _0\) against \(H_A:\sigma \ne \sigma _0\).

(a) Find a test, using the generalized likelihood ratio \(\lambda \).

(b) Provide a large sample approximation for \(-2\log \lambda \) under \(H_0\).

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed normal \(N(\mu _1,\sigma ^2)\) random variables. Let \(Y_1,Y_2,\ldots ,Y_m\) be independent identically distributed normal \(N(\mu _2,\sigma ^2)\) random variables. The two samples are independent. The parameters \(\mu _1,\mu _2,\sigma ^2\) are unknown. We wish to test \(H_0:\mu _1=\mu _2\). Show that the two sample t-test and the likelihood ratio test are equivalent.

Find the maximum likelihood estimators of \(\theta \) and \(\alpha \), based on a random sample \(X_1,X_2,\ldots ,X_n\) from the distribution with density \[ f(x;\theta ,\alpha )=\frac {\alpha }{\theta ^\alpha }x^{\alpha -1}, \qquad 0\leq x\leq \theta . \] Here \(\theta \) and \(\alpha \) are positive.

Let \(X_1,\ldots ,X_n\) be a random sample from \(\operatorname {Uniform}[0,\theta ]\), where \(\theta \) is a positive unknown. Derive the generalized likelihood ratio (GLR) test of \(H_0:\theta =\theta _0\) vs. \(H_a:\theta \ne \theta _0\).

Let \(X_1,X_2,\ldots \) be i.i.d. integer-valued random variables with probability mass function \(f(a):=\Prob \{X_1=a\}\) for all \(a\in \mathbb {Z}\), and suppose \(f(a)>0\) for all \(a\in \mathbb {Z}\). Now let \(g\) be another probability mass function on \(\mathbb {Z}\) and define the likelihood ratio, \[ \Lambda _n:=\prod _{j=1}^{n}\frac {g(X_j)}{f(X_j)}. \]

(a) Consider the log-likelihood \(\log \Lambda _n\), where “log” denotes the natural logarithm. Prove that \[ \lim _{n\to \infty }n^{-1}\log \Lambda _n =\mathbb {D}(g\,\|\,f) \qquad \text {a.s.,} \] where \[ \mathbb {D}(g\,\|\,f):= \sum _{a=-\infty }^{\infty } f(a)\log \left [\frac {g(a)}{f(a)}\right ], \] provided that \[ \mathbb {E}(g\,\|\,f):= \sum _{a=-\infty }^{\infty } \left |\log \left [\frac {g(a)}{f(a)}\right ]\right |f(a)<\infty . \]

(b) Verify that \(\log x\leq x-1\) for all \(x>0\), and that the inequality is strict except when \(x=1\). Use this to prove the following [Gibbs’ inequality]: \(\mathbb {D}(g\,\|\,f)<0\).

(c) Conclude from the preceding that if \(\mathbb {E}(g\,\|\,f)<\infty \), then \(\Lambda _n\to 0\) a.s. very rapidly as \(n\to \infty \).

This fact plays an important role in statistics.

Let \(X_1,\ldots ,X_n\) be a random sample from \(N(\mu ,\sigma _0^2)\), where \(\sigma _0^2\) is known. Find the generalized likelihood ratio test of size \(\alpha \) of \(H_0:\mu \geq \mu _0\) vs. \(H_a:\mu <\mu _0\).

Let \(X_1,\ldots ,X_n\) be independent identically distributed Poisson random variables with parameter \(\theta _1\). Let \(Y_1,\ldots ,Y_m\) be independent identically distributed Poisson random variables with parameter \(\theta _2\). We assume that the two samples are independent. We wish to test \(H_0:\theta _1=\theta _2\) against the alternative that \(H_0\) is not true.

a. Derive the likelihood ratio test. (5 points)

b. Provide a large sample approximation for the rejection region. (2 points)

Let \(X_1,\ldots ,X_N\) be independent random variables. The distibution of \(X_i\) is binomial \((n_i,p_i)\). We wish to test \(H_0:p_1=p_2=\cdots =p_N\) against the alternative that \(H_0\) is not true.

a. Derive the likelihood ratio test. (5 points)

b. Provide a large sample approximation for the rejection region. (2 points)

Consider the pdf \(f(x;\theta )=\theta x^{-(\theta +1)}\) (\(x\geq 1\)). Let \(X_i\) be a random sample of size \(n\) with pdf \(f(x;\theta _1)\). Let \(Y_i\) be a random sample of size \(m\) with pdf \(f(x;\theta _2)\). The two sets of random variables are independent. We wish to test \[ H_0:\theta _1=\theta _2 \quad \text {against}\quad H_a:\theta _1\ne \theta _2. \]

6a) (5 points) If one considers just the \(X\)-data, find the MLE \(\hat \theta _1\).

6b) (5 points) Derive a formula for the critical region obtained using the generalized likelihood ratio method.

Let \(X_i\sim \operatorname {EXP}(\theta )\) be a random sample of size \(n\).

7a) (5 points) Find the MGF of \(\frac {2n\bar X}{\theta }\) and identify its distribution.

7b) (10 points) Derive the generalized likelihood ratio test of \(H_0:\theta =\theta _0\) against \(H_a:\theta \ne \theta _0\).

7c) (5 points) The following data are times (in hours) between failures of air conditioning equipment in a particular airplane: 74, 57, 48, 29, 502, 12, 70, 21, 29, 386, 59, 27, 153, 26, 326. Test \(H_0:\theta =125\) against \(H_a:\theta \ne 125\).

Let \(X_1,X_2,\ldots ,X_n\) and \(Y_1,Y_2,\ldots ,Y_m\) be two independent random samples. We assume that \(X_1,X_2,\ldots ,X_n\) are independent identically distributed exponential \((\lambda )\) random variables and \(Y_1,Y_2,\ldots ,Y_m\) are independent identically distributed exponential \((\mu )\) random variables. We want to test \(H_0:\lambda =\mu \) against the alternative \(H_A:\lambda \ne \mu \).

(i) Find \(T_{m,n}\) the likelihood ratio test. (7 points)

(ii) Prove that \(-2\log T_{m,n}\) converges in distribution under the null hypothesis when \(\min (m,n)\to \infty \). (7 points)

(iii) Show that the likelihood ratio test is asymptotically consistent and prove that the power goes to 1 as \(\min (m,n)\to \infty \). (7 points)

Let \(X_1,X_2,\ldots ,X_n\) and \(Y_1,Y_2,\ldots ,Y_m\) be two independent random samples. We assume that \(X_1,X_2,\ldots ,X_n\) are independent identically distributed normal \((\mu _1,\sigma ^2)\) random variables and \(Y_1,Y_2,\ldots ,Y_m\) are independent identically distributed normal \((\mu _2,\sigma ^2)\) random variables (\(\sigma ^2\) is unknown). We want to test \(H_0:\mu _1=\mu _2\) against the alternative \(H_A:\mu _1\ne \mu _2\).

Show that the likelihood ratio test and the two sample t-test are equivalent. (7 points)

Let \((X_1,\ldots ,X_m)\) be a multinomial random vector with parameters \(N\) and \(p_1,p_2,\ldots ,p_m\). Note that \(N=X_1+\cdots +X_m\) and \(p_1+p_2+\cdots +p_m=1\). Find the maximum likelihood estimators for \(p_1,p_2,\ldots ,p_m\). (7 points)

Let \(X_1, X_2, \ldots \) be an i.i.d. sample from the uniform distribution on \([\theta , 1+\theta ]\), where \(\theta > 0\) is unknown. Is there an unique maximum likelihood estimator of \(\theta \) that is based on \(\{X_i\}_{i=1}^{\infty }\)? Prove or disprove.

Develop carefully the likelihood ratio test for \(H_0:\theta =1\) versus \(H_1:\theta =2\). Discuss the various optimality properties—if any—of your test.

Let \(X_1,\ldots ,X_n\) be independent, identically distributed random variables with density function \[ f(t,\theta _1)= \begin {cases} 0, & \text {if } t\notin [1,\infty ),\\ \theta _1 t^{-\theta _1-1}, & \text {if } 1\leq t<\infty . \end {cases} \] Let \(Y_1,\ldots ,Y_m\) be independent, identically distributed random variables with density function \[ f(t,\theta _2)= \begin {cases} 0, & \text {if } t\notin [1,\infty ),\\ \theta _2 t^{-\theta _2-1}, & \text {if } 1\leq t<\infty . \end {cases} \] We wish to test \(H_0:\theta _1=\theta _2\) against the alternative that \(\theta _1\neq \theta _2\). The two samples are independent.

(a) Find a test using the likelihood ratio.

(b) Provide a large sample approximation for the likelihood ratio test.

Let \(X_1,\ldots ,X_n\) be independent random variables with density functions \[ f(t,\theta _i)= \begin {cases} 0, & \text {if } t\notin [0,\infty ),\\ \dfrac {1}{\theta _i}e^{-t/\theta _i}, & \text {if } 0\leq t<\infty . \end {cases} \] We wish to test \(H_0:\theta _1=\theta _2=\cdots =\theta _n\) against the alternative that \(H_0\) is not true.

(a) Find a test using the likelihood ratio.

(b) Provide a large sample approximation for the likelihood ratio test.

Let \(X_1,X_2,\ldots ,X_n\) be independent, identically distributed normal \(N(\mu ,\sigma ^2)\) random variables. We wish to test \(H_0:\mu =\mu _0\) against \(H_A:\mu \neq \mu _0\) (\(\sigma \) is unknown). Show that the t-test and the likelihood tests are equivalent.

Let \(X_1,X_2,\ldots ,X_n\) be independent exponential \((\theta _i)\) random variables, i.e. the density of \(X_i\) is \[ f(t,\theta _i)= \begin {cases} 0, & \text {if } -\infty <t<0,\\ \dfrac {1}{\theta _i}e^{-t/\theta _i}, & \text {if } 0\leq t<\infty . \end {cases} \] We wish to test \(H_0:\theta _1=\theta _2=\cdots =\theta _n\) against the alternative that \(H_0\) is not true.

(a) Find the likelihood ratio test.

(b) Show that the likelihood ratio test is distribution free under \(H_0\).

(c) Show that the likelihood ratio test is equivalent with the \(\chi ^2\) test under \(H_0\).

(d) Approximate the critical values of the likelihood ratio test under \(H_0\).

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed exponential \((\lambda )\) random variables. Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed exponential \((\mu )\) random variables. The two samples are independent. We wish to test \(H_0:\theta =\mu \) against the alternative that \(H_0\) is not true.

(a) Find the likelihood ratio test.

(b) Find the asymptotic distribution of the likelihood ratio under \(H_0\).

(c) Find an equal-tail confidence interval for \(\tau =\lambda /\mu \).

Let \(X_1,X_2,\ldots ,X_n\) be independent random variables. We assume that \(X_i\) has a Poisson\((\lambda _i)\) distribution. We wish to test \[ H_0:\lambda _1=\lambda _2=\cdots =\lambda _n \] against the alternative \[ H_A:H_0\text { is not true.} \]

1. Show that the likelihood ratio test and the \(\chi ^2\) tests are equivalent under \(H_0\). (15 points)
2. Explain why and how the normal as well as the \(\chi ^2\) distributions can be used to approximate the critical values. (5 points)

Let \(X_1,X_2,\ldots ,X_n\) and \(Y_1,Y_2,\ldots ,Y_m\) be independent random samples of independent observations. The distribution of \(X_i\) is normal \(N(\mu _1,\sigma _1^2)\) and the distribution of \(Y_i\) is normal \(N(\mu _2,\sigma _2^2)\). The values of \(\sigma _1\) and \(\sigma _2\) are known. We wish to test \(H_0:\mu _1=\mu _2\) against \(H_A:\mu _1\neq \mu _2\). Show that the likelihood ratio test and the t-test are equivalent under \(H_0\). (20 points)

Let \(X_1,\ldots ,X_n\) be a random sample from the \(N(\mu ,\sigma ^2)\) distribution, with both \(\mu \) and \(\sigma ^2\) unknown.

(a) Derive the likelihood ratio test for \(H:\sigma ^2=\sigma _0^2\) against all alternatives. Here \(\sigma _0^2\) is a known positive constant. (5 points)

(b) Do the same for \(H:\sigma ^2\ge \sigma _0^2\). (5 points)

Let \(X_1,X_2,\ldots ,X_n\) be independent Poisson random variables with parameters \(\theta _1,\theta _2,\ldots ,\theta _n\). We wish to test \(H_o:\theta _1=\theta _2=\cdots =\theta _n\) against the alternative that \(H_o\) is not true.

(a) Find the likelihood ratio test. (5 points)

(b) Show that the likelihood ratio is asymptotically normal (after centralizing and normalizing) under \(H_o\). (5 points)

Suppose \(X_1,\cdots ,X_m\) are independent with \(X_j\sim \operatorname {Binomial}(n_j,p_j)\).

(a) Find the UMVUE’s of \(p_1,\cdots ,p_m\).

(b) Suppose you know that \(p_1=p_2=\cdots =p_m\). Let \(p\) denote this common (but unknown) value. Find the UMVUE of \(p\).

(c) Find the likelihood ratio statistic, \(\lambda \), for \(H_0:p_1=\cdots =p_m\) versus \(H_1:p_i\ne p_j\), for some \(i\) and \(j\).

(d) It can be shown that \(2\log \lambda \) is approximately the same as the usual \(\chi ^2\) statistics. Using this fact, find an approximate test for \(H_0\) vs \(H_1\) above.

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables, uniformly distributed on \([0,\theta ]\), \(\theta >\theta _0\). We want to test \(H_0:\theta =\theta _0\) against \(H_A:\theta >\theta _0\).

(a) Find the uniformly most powerful test. (You must prove your claim.)

(b) Show that the uniformly most powerful test and the likelihood ratio test are equivalent,

(c) Compute the power function of the most powerful test.