Central Limit Theorems

Let \((a_{n})_{n \ge 1}\) be a nondecreasing sequence of positive numbers such that \[ \lim _{n \to \infty }\frac {a_{n}^{2}}{\sum _{k=1}^{n}a_{k}^{2}} = 0. \]

Let \((X_{n})_{n \ge 1}\) be a sequence of i.i.d. random variables with mean 0 and variance 1. Prove that \[ \frac {\sum _{k=1}^{n} a_{k}X_{k}}{\sqrt {\sum _{k=1}^{n}a_{k}^{2}}} \]

converges weakly to the standard normal distribution.

Without using the Central Limit Theorem or the Law of the Iterated Logarithm, prove the Weak Law of Large Numbers when \(X_{i} \in L^{1}\), i.e. show that when \(X_{1}, X_{2}, \ldots \) is a sequence of i.i.d. variables with \(\mathbb {E}[X_{1}] = \mu \) and \(X_{1} \in L^{1}\), then

Hint: first, prove the claim when \(X_{1} \in L^{2}\). Then for \(a > 0\), consider

Give an example of a sequence of independent random variables \(\{Z_{k}\}_{k \in \mathbb {N}}\) satisfying \(\mathbb {E}[Z_{k}] = 0\), \(\operatorname {Var}(Z_{k}) = 1\), for each \(k\), and such that \((Z_{1} + \cdots + Z_{n})/\sqrt {n}\) does not converge in distribution to a standard normal. Justify any claims you make.

Let \(\{X_{n}\}_{n \in \mathbb {N}}\) be independent random variables such that for each \(n\), \(X_{n}\) takes the values \(+1\) and \(-1\) with probability \((1 - 2^{-n})/2\) each and the value \(2^{k}\) with probability \(2^{-k}\), for integers \(k > n\).

1. Show that \(\mathbb {E}[|X_{n}|] = \infty \) for all \(n \in \mathbb {N}\).
2. Show that \((X_{1} + \cdots + X_{n})/\sqrt {n}\) converges in distribution to a standard normal. (Hint: Check that we can write \(X_{n} = (1 - B_{n})Y_{n} + 2^{n}B_{n}Z_{n}\), where \(B_{n}\) is \(\operatorname {Bernoulli}(2^{-n})\), \(Y_{n}\) takes values \(\pm 1\) equally likely, \(Z_{n}\) takes the value \(2^{k}\) with probability \(2^{-k}\), for \(k \in \mathbb {N}\), and \(\{B_{n}, Z_{n}, Y_{n} : n \ge 1\}\) are mutually independent. Now, \(X_{n}\) is a small perturbation of \((1 - B_{n})Y_{n}\). Work out the details.)

Use your limit theorems to prove that the following limit exists and to compute its value: \[ \lim _{n\to \infty }\int _{-\sqrt {n}}^{\sqrt {n}} \left (1-\frac {x^2}{2n}\right )^n\,dx. \] You need to fully justify the use of whatever limit theorem you choose.

Let \(X\) and \(Y\) be independent \(N(0,1)\) random variables, and let \(Z=X+Y\).

1. Show that \(\mathbb {E}[Z\mid X>0,Y>0]=2\sqrt {2/\pi }\).
2. Find the distribution and the density of \(Z\) given that \(X>0\) and \(Y>0\). (You can use the CDF of a standard normal, \(\Phi (x)=\mathbb {P}(X\leq x)\), in your expression.)

Let \(X_n\) be a sequence of independent random variables with \(\mathbb {P}(X_n=1)=1/n=1-\mathbb {P}(X_n=0)\). Prove that \[ \left (\sum _{i=1}^{n}X_i-\log n\right )/\sqrt {\log n} \] converges weakly to a standard normal.

Let \(X\) and \(Y\) be two standard normal random variables (defined on a common probability space and having positive variances).

1. Prove that they are independent if and only if they are jointly normal (i.e. \((X,Y)\) is a two-dimensional normal random variable) and uncorrelated.
2. Give an example of two standard normal random variables that are uncorrelated but not jointly normal.

Let \(X_n\) be i.i.d. random variables with mean \(\mu \) and variance \(\sigma ^2<\infty \). Prove (without using the law of the iterated logarithm) that we have almost surely \[ \overline {\lim }_{n\to \infty }\frac {S_n-n\mu }{\sigma \sqrt {n}}=\infty \qquad \text {and}\qquad \underline {\lim }_{n\to \infty }\frac {S_n-n\mu }{\sigma \sqrt {n}}=-\infty . \]

Let \(X_1,\ldots ,X_m\) be \(m\geq 2\) i.i.d. standard normal random variables, and define the random vector \[ \mathbf {X}:= \begin {bmatrix} X_1\\ \vdots \\ X_m \end {bmatrix}. \] Prove that the distribution of \(\mathbf {X}\) is the same as the distribution of \(A\mathbf {X}\) for every \(m\times m\) orthogonal matrix \(A\) that is non random.

Consider an array \(\{X_{i,j}\}\) of random variables such that, for every \(n\geq 1\), the collection \(X_{1,n},\ldots ,X_{n,n}\) is i.i.d. with common distribution \[ \mathbb {P}\{X_{1,n}=1\} = \mathbb {P}\{X_{1,n}=-1\} = \frac {1}{2}\left (1-\frac {1}{n^2}\right ); \] and \(\mathbb {P}\{X_{1,n}=n^2\}=n^{-2}\). Let \(S_n:=X_{1,n}+\cdots +X_{n,n}\). Prove that \(S_n/\sqrt {n}\Rightarrow \mathrm {N}(0,\sigma ^2)\) for some \(\sigma \in (0,\infty )\), although \(\mathbb {E}(S_n/\sqrt {n})\to \infty \) as \(n\to \infty \).

If \(X_1,X_2,\ldots \) are independent standard normal random variables, then find non-random sequences \(a_n,b_n\to \infty \) such that \(a_n\min _{1\leq i\leq n}X_i+b_n\) converges weakly. Identify the limiting distribution.

Let \(X_1,X_2,\ldots ,X_n\) be iid exponential(1) random variables and \[ \bar X_n=\frac {1}{n}\sum _{i=1}^n X_i \] be the sample mean. Find two sequences of numbers \(a_n\) and \(b_n\) such that \[ \frac {\log \bar X_n-a_n}{b_n} \] converges in distribution to a standard normal random variable.

Let \(X_1,X_2,\ldots \) be independent and identically distributed random variables uniform on \([0,1]\) and let \(\alpha >0\). Show that there are numerical sequences \(a_n\) and \(b_n\) such that \[ Y_n=\frac {\sum _{k=1}^{n}k^{\alpha }X_k-a_n}{b_n} \] converges in distribution to a standard normal random variable.

Let \(X\) and \(Y\) be independent standard normal random variables. Compute the density function of \(Z=X/Y\).

Let \(X_1,X_2,\ldots \) be independent and identically distributed random variables uniform on \([0,1]\). Show that for all \(\alpha >-1\) there are numerical sequences \(a_n\) and \(b_n\) such that \[ Y_n= \frac {\sum _{1\leq k\leq n} k^{\alpha }X_k-a_n}{b_n} \] converges in distribution to a standard normal random variable.

Compute \[ \lim _{n\to \infty } e^{-n} \sum _{i=0}^{n+\sqrt {n}}\frac {n^i}{i!}. \]

Let \(Z=\operatorname {N}(0,1)\) denote a standard normal random variable. Suppose \(f:\mathbb {R}\to \mathbb {R}\) is continuously differentiable and there exists \(c>1\) such that \[ |f(x)|+|f'(x)|\leq c\exp (c|x|) \qquad \text {for all }x\in \mathbb {R}. \] Derive the following [Stein’s differential equation]: \[ \mathbb {E}[f'(Z)]=\mathbb {E}[Zf(Z)]. \]

Let \(X,X_1,X_2,\ldots \) be independent random variables, all distributed according to the standard normal distribution.

(a) Prove that there exist finite constants \(c_1,c_2,x_0>0\), such that \[ \frac {c_1}{x}e^{-x^2/2}\leq \mathbb {P}\{X>x\}\leq \frac {c_2}{x}e^{-x^2/2} \qquad \text {for all }x\geq x_0. \]

(b) Use the inequalities of part (a) in order to prove that if \(X_1,X_2,\ldots \) are independent standard normal random variables then \[ \limsup _{n\to \infty }\frac {X_n}{\sqrt {2\ln n}} = -\liminf _{n\to \infty }\frac {X_n}{\sqrt {2\ln n}} =1 \qquad \text {a.s.} \]

Suppose that, for every integer \(n\geq 1\), \(X(n)\) has a Gamma distribution with parameters \(n+1\) and 1; that is, the probability density function of \(X(n)\) is \[ f_n(x):=\frac {x^ne^{-x}}{n!}\mathbf {1}_{(0,\infty )}(x). \]

(a) Compute \(\mathbb {E}(X(n))\) and \(\operatorname {Var}(X(n))\) for every integer \(n\geq 1\).

(b) Prove that, as \(n\to \infty \), \[ \frac {X(n)-n}{\sqrt {n}}\Rightarrow \mathrm {N}(0,1). \]

(a) Let \(\Phi \) and \(\phi \) be the standard normal distribution and density functions, respectively. Show that \[ \lim _{x\to \infty }\frac {1-\Phi (x)}{(1/x)\phi (x)}=1. \]

(b) Let \(X_1,X_2,\ldots \) be independent identically distributed standard normal random variables. Show that \[ \limsup _{n\to \infty }\frac {1}{\sqrt {\log n}}|X_n|=c \quad \text {almost surely} \] and compute the value of \(c\).

Let \(X_1,X_2,\ldots \) be independent identically distributed standard normal random variables. Show that \[ \limsup _{n\to \infty }\frac {1}{\sqrt {\log n}}|X_n|=c \quad \text {almost surely} \] and compute the value of \(c\).

Suppose \(X_1\) and \(X_2\) are independent random variables that satisfy the following three properties: (i) \(Y_1:=(X_1+X_2)/\sqrt {2}\) is standard normal; (ii) \(Y_2:=(X_1-X_2)/\sqrt {2}\) is standard normal; and (iii) \(Y_1\) and \(Y_2\) are independent. Prove that \(X_1\) and \(X_2\) are normally distributed. Can you compute their respective means and variances?

Let \(\Phi \) and \(\phi \) denote the standard normal distribution and density functions.

(a) Prove that for all \(x>0\) \[ 1-\Phi (x)\leq \frac {1}{x}\phi (x). \] (5 points)

Let \(X_{1,n}<X_{2,n}\leq \cdots \leq X_{n,n}\) denote the order statistics from a sample of \(n\) independent identically distributed standard normal random variables.

(b) Show that \(X_{n,n}\) converges in probability to infinity. (5 points)

(c) Show that for all \(\epsilon >0\) \[ \lim _{n\to \infty } \mathbb {P}\left \{X_{n,n}\leq \sqrt {(2+\epsilon )\log n}\right \}=1. \] (5 points)

Let \(X\) and \(Y\) be independent standard normals and consider \((U,V)=(aX+bY,cX+dY)\). Assume that at least one of \(a\) and \(b\) is nonzero, and at least one of \(c\) and \(d\) is nonzero. (Otherwise either \(U\) or \(V\) would be 0 and the problem is not interesting.)

5a) (5 points) Show that if \(ad=bc\), then \(U\) and \(V\) cannot be independent random variables.

5b) (5 points) Assume \(ad\ne bc\) and find the joint pdf of \((U,V)\).

5c) (5 points) Find a necessary and sufficient condition for \(U\) and \(V\) to be independent.

(a) State the central limit theorem as well as Kolmogorov’s 0-1 law.

(b) For arbitrary events \(A_n\), \(n\geq 1\), show that \(\inf _{n\geq 1}\mathbb {P}(A_n)>0\) implies \(\mathbb {P}(A_n\ \text {i.o.})>0\).

(c) Let \(X_1,X_2,\ldots \) be i.i.d. mean 0, variance 1 random variables, and put \(S_n=X_1+\cdots +X_n\) for each \(n\geq 1\). Show that \(\limsup _{n\to \infty }S_n=\infty \) a.s.

Let \(X_1,X_2,\ldots \) be i.i.d. mean 0, variance 1 random variables and \(S_n=X_1+\cdots +X_n\) for each \(n\geq 1\). Show directly that the characteristic function of \(S_n/\sqrt {n}\) converges pointwise to the characteristic function of the standard normal distribution. Notice that this is a key step in the proof of the central limit theorem, so you are not allowed to use the central limit theorem in your derivation.

Let \(X\) and \(Y\) be independent, each having the standard normal distribution, and let \((R,\Theta )\) be the polar coordinates for \((X,Y)\).

(a) Show that \(X+Y\) and \(X-Y\) are independent, and that \(R^2=[(X+Y)^2+(X-Y)^2]/2\), and conclude that the conditional distribution of \(R^2\) given \(X-Y\) is the chi-squared with one degree of freedom translated by \((X-Y)^2/2\).

(b) Show that the conditional distribution of \(R^2\) given \(\Theta \) is chi-squared with two degrees of freedom.

(c) If \(X-Y=0\), the conditional distribution of \(R^2\) is chi-squared with one degree of freedom. If \(\Theta =\pi /4\) or \(\Theta =5\pi /4\), the conditional distribution of \(R^2\) is chi-squared with two degrees of freedom. But the events \(\{X-Y=0\}\) and \(\{\Theta =\pi /4\}\cup \{\Theta =5\pi /4\}\) are the same. Resolve the apparent contradiction.

Let \(X_1\) and \(X_2\) be independent standard normal random variables. Find the necessary and sufficient condition for the independence of \(Y_1=a_1X_1+a_2X_2\) and \(Y_2=b_1X_1+b_2X_2\). (7 points)

Suppose \(X\) and \(Y\) are two independent standard normal random variables.

(a) Prove that for all twice continuously differentiable functions \(f,g:\mathbb {R}\to \mathbb {R}\), \[ \operatorname {Cov}(f(X),g(X))=\int _0^1 \E \!\left [f'(X)g'\!\left (sX+(1-s^2)^{1/2}Y\right )\right ]\,ds. \]

(Hint: Check it first for \(f(x):=\exp (itx)\) and \(g(x):=\exp (itx)\).)

(b) Conclude the following “Poincaré inequality,” due to J. Nash (1958): \[ \Var (f(X))\le \E \!\left [(f'(X))^2\right ]. \]

Suppose \(\{X_n\}_{n=1}^{\infty }\) are i.i.d. standard normal random variables. Prove that \[ M_n:=(n+1)^{-1/2}\exp \left (\frac {S_n^2}{2n+2}\right ) \] defines a martingale, where \(S_n=\sum _{i=1}^{n}X_i\).

Let \(X_1,X_2,X_3,\ldots \) be i.i.d. nonnegative random variables with mean 1 and variance 1, and put \(S_n=X_1+X_2+\cdots +X_n\). Show that \(2(\sqrt {S_n}-\sqrt {n})\) converges in distribution to a standard normal as \(n\to \infty \).

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with a continuous distribution function \(F\). The ordered observations are \(X_{1,n}\leq X_{2,n}\leq \cdots \leq X_{n,n}\). Let \[ T_n=\sum _{i=1}^{n-1}\left (F(X_{i+1,n})-F(X_{i,n})\right )^2. \] Show that there are numerical sequences \(a_n\) and \(b_n\) such that \((T_n-a_n)/b_n\) converges in distribution to a standard normal random variable. (20 points)

Let \(X_1,X_2,\ldots \) be independent identically distributed positive random variables with \(\E X_1=\mu \) and \(\operatorname {Var}(X_1)=\sigma ^2\). Let \[ N(t)=\min \left \{k:\sum _{1\le i\le k}X_i>t\right \},\qquad 0<t<\infty . \]

(a) Show that \[ \lim _{t\to \infty }\frac {1}{t}N(t)=\frac {1}{\mu } \] almost surely. (5 points)

(b) Show that \((N(t)-t/\mu )/(t\sigma ^2/\mu ^3)^{1/2}\) goes in distribution to a standard normal random variable. (5 points)

Let \(X_1,X_2,\ldots ,X_n\) be independent identically distributed random variables with distribution function \(F\) and density function \(f\). We assume that the second derivative of \(f\) is bounded. We estimate \(f\) by \[ f_n(t)=\frac {F_n(t+h_n)-F_n(t-h_n)}{2h_n}, \] where \(F_n\) is the empirical distribution function and \(h_n\to 0\) as \(n\to \infty \).

(a) Show that \(\E f_n(t)\to f(t)\) as \(n\to \infty \). (5 points)

(b) Show that if \(nh_n\to \infty \), then \[ \frac {f_n(t)-\E f_n(t)}{(\operatorname {Var}(f_n(t)))^{1/2}} \] is asymptotically standard normal as \(n\to \infty \). (5 points)

Let \(X_1,\ldots ,X_n\) be a random sample from \(N(\theta ,\theta )\) (i.e., normal with mean and variance both equal to \(\theta \)), where \(\theta >0\).

(a) Give three pivotal quantities (or pivots) involving the entire sample, which respectively have a standard normal distribution, a \(t\)-distribution, and a chi-squared distribution, and indicate the numbers of degrees of freedom. (5 points)

(b) Use the normal pivotal quantity in part (a) to obtain a \(100(1-\alpha )\) percent confidence interval for \(\theta \). (5 points)

Let \(X_1,X_2,\ldots \) be i.i.d. mean 0, variance \(\sigma ^2\in (0,\infty )\).

(a) Use the central limit theorem as well as Kolmogorov’s 0-1 law to conclude that \(\limsup _{n\to \infty }S_n/\sqrt n=\infty \) a.s.

(b) Show that \(S_n/\sqrt n\) does not converge in probability.