Let \(X_1,X_2,\ldots \) be i.i.d. integer-valued random variables with probability mass function \(f(a):=\Prob \{X_1=a\}\) for all \(a\in \mathbb {Z}\), and suppose \(f(a)>0\) for all \(a\in \mathbb {Z}\). Now let \(g\) be another probability mass function on \(\mathbb {Z}\) and define the likelihood ratio, \[ \Lambda _n:=\prod _{j=1}^{n}\frac {g(X_j)}{f(X_j)}. \]
(a) Consider the log-likelihood \(\log \Lambda _n\), where “log” denotes the natural logarithm. Prove that \[ \lim _{n\to \infty }n^{-1}\log \Lambda _n =\mathbb {D}(g\,\|\,f) \qquad \text {a.s.,} \] where \[ \mathbb {D}(g\,\|\,f):= \sum _{a=-\infty }^{\infty } f(a)\log \left [\frac {g(a)}{f(a)}\right ], \] provided that \[ \mathbb {E}(g\,\|\,f):= \sum _{a=-\infty }^{\infty } \left |\log \left [\frac {g(a)}{f(a)}\right ]\right |f(a)<\infty . \]
(b) Verify that \(\log x\leq x-1\) for all \(x>0\), and that the inequality is strict except when \(x=1\). Use this to prove the following [Gibbs’ inequality]: \(\mathbb {D}(g\,\|\,f)<0\).
(c) Conclude from the preceding that if \(\mathbb {E}(g\,\|\,f)<\infty \), then \(\Lambda _n\to 0\) a.s. very rapidly as \(n\to \infty \).
This fact plays an important role in statistics.