%% This document created by Scientific Word (R) Version 2.5
%% Starting shell: article

\documentclass[12pt,thmsa]{article}%
\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{sw20lart}%
\setcounter{MaxMatrixCols}{30}%
\usepackage{graphicx}
%TCIDATA{OutputFilter=latex2.dll}
%TCIDATA{Version=5.50.0.2890}
%TCIDATA{TCIstyle=article/art4.lat,lart,article}
%TCIDATA{Created=Thu Oct 15 23:57:36 1998}
%TCIDATA{LastRevised=Monday, February 02, 2009 19:53:24}
%TCIDATA{<META NAME="GraphicsSave" CONTENT="32">}
%TCIDATA{<META NAME="SaveForMode" CONTENT="1">}
%TCIDATA{BibliographyScheme=Manual}
%BeginMSIPreambleData
\providecommand{\U}[1]{\protect\rule{.1in}{.1in}}
%EndMSIPreambleData
\setcounter{section}{1}
\setcounter{page}{4}
\setlength{\oddsidemargin}{0mm}
\setlength{\evensidemargin}{0mm}
\setlength{\topmargin}{-7mm}
\setlength{\headheight}{7mm}
\setlength{\headsep}{4.5mm}
\setlength{\textheight}{225mm}
\setlength{\textwidth}{158mm}
\begin{document}
\section{\bigskip Estimation}

\subsection{Data and questions}

\textbf{Data set 2.1\medskip\quad}\emph{Wages in the USA\newline}These data
are taken from Dyer, D. (1981), \textit{Canadian Journal of Statistics},
\textbf{9}. They comprise annual wages (in multiples of US\$100) of 30
production line workers in a large American firm.%

\[%
\begin{tabular}
[c]{ccccc}%
\multicolumn{5}{l}{\textbf{Table 2.1} \textit{US wages}}\\\hline
112 & 123 & 128 & 116 & 119\\
154 & 103 & 132 & 140 & 111\\
119 & 115 & 107 & 108 & 101\\
108 & 107 & 151 & 105 & 157\\
112 & 125 & 103 & 158 & 112\\
156 & 119 & 104 & 104 & 115\\\hline
\end{tabular}
\]
$\square$\bigskip\newline\textbf{Data set 2.2}\medskip\quad\emph{Severe
ideopathic respiratory distress syndrome}\newline The data were published by
van Vliet, P.K. and Gupta, J.M. (1973), Sodium bicarbonate in ideopathic
respiratory distress syndrome, \textit{Arch. Diseases in Childhood},
\textbf{48}, 249-255. The condition, known by its acronym SIRDS, is very
serious and can result in the death. The data comprise birth weights (kg) of
50 infants who displayed SIRDS%
\[
.%
\begin{tabular}
[c]{llllll}%
\multicolumn{6}{l}{\textbf{Table 2.2} Birth weights (kg) of infants with}\\
\multicolumn{6}{l}{severe ideopathic respiratory distress syndrome}%
\\\cline{1-5}%
1.050* & 2.500* & 1.890* & 1.760 & 2.830 & \\
1.175* & 1.030* & 1.940* & 1.930 & 1.410 & \\
1.230* & 1.100* & 2.200* & 2.015 & 1.715 & \\
1.310* & 1.185* & 2.270* & 2.090 & 1.720 & \\
1.500* & 1.225* & 2.440* & 2.600 & 2.040 & \\
1.600* & 1.262* & 2.560* & 2.700 & 2.200 & \\
1.720* & 1.295* & 2.730* & 2.950 & 2.400 & \\
1.750* & 1.300* & 1.130 & 3.160 & 2.550 & \\
1.770* & 1.550* & 1.575 & 3.400 & 2.570 & \\
2.275* & 1.820* & 1.680 & 3.640 & 3.005 & \\\cline{1-5}%
\multicolumn{5}{l}{{*} child died} & \\\cline{1-5}%
\end{tabular}
\]
There are important questions to be asked about these data. Can we identify
children at risk quickly and accurately? Is it possible to relate the risk of
death to birthweight?\newline$\square\bigskip$\newpage\newline\noindent
\textbf{Data set 2.3}\medskip\quad\emph{Silver content of Byzantine
coins}\newline A number of coins from the reign of King Manuel I, Comnemus
(1143 - 80) were discovered in Cyprus. They arise from four different coinages
at intervals throughout his reign. The question of interest is whether there
is any significant difference in their silver content with the passage of
time; there is a suspicion that it was deliberately and steadily reduced. The
data give the silver content (\%Ag) of the coins.%
\[%
\begin{tabular}
[c]{ccccc}%
\multicolumn{5}{l}{\textbf{Table 2.3}\quad Silver content of coins}%
\\\cline{1-4}%
First & Second & Third & Fourth & \\\cline{1-4}%
5.9 & 6.9 & 4.9 & 5.3 & \\
6.8 & 9.0 & 5.5 & 5.6 & \\
6.4 & 6.6 & 4.6 & 5.5 & \\
7.0 & 8.1 & 4.5 & 5.1 & \\
6.6 & 9.3 &  & 6.2 & \\
7.7 & 9.2 &  & 5.8 & \\
7.2 & 8.6 &  & 5.8 & \\
6.9 &  &  &  & \\
6.2 &  &  &  & \\\cline{1-4}%
\end{tabular}
\]
On the face of it the suspicion could be correct in that the fourth coinage
would seem to have lower silver content than, say, the first coinage, but
there is a need for firm statistical evidence if it is to be
confirmed.\newline$\square\bigskip$\newline\textbf{Data set 2.4}\medskip
\quad\emph{Radiocarbon dating}\newline Radiocarbon datings have given
historians a powerful tool for determining more precisely the periods during
which ancient civilisations flourished. The following set of data came from
the Lake Lamoka site and were contributed by S.R. Wilson from the Australian
National University to Andrews, D.F. and Herzberg, A.M. (1985), \emph{Data},
Springer-Verlag: New York.%
\[%
\begin{tabular}
[c]{lcc}%
\multicolumn{3}{l}{\textbf{Table 2.4}\quad Radiocarbon dating}\\\cline{1-2}%
Sample & Radiocarbon age & \\
number & determination & \\\cline{1-2}%
C-288 & 2419 & \\
M-26 & 2485 & \\
C-367 & 3433 & \\
M-195 & 2575 & \\
M-911 & 2521 & \\
M-912 & 2451 & \\
Y-1279 & 2550 & \\
Y-1280 & 2540 & \\\cline{1-2}%
\end{tabular}
\]
\newline$\square$

\subsection{Desirable properties of estimators}

\noindent Suppose we have $X=X_{1},X_{2},\ldots,X_{n}$ drawn from a
distribution with some parameter $\theta$. Often $X_{1},X_{2},\ldots,X_{n}$
form a \textit{sample}.\bigskip\newline\textbf{Definition 2.0}
\emph{\ Estimators}\bigskip\newline An \emph{estimator} $\widehat{\theta}_{n}$
of $\theta$ is just a function of the observed data which (we hope) forms a
useful approximation to the parameter:%
\[
\widehat{\theta}_{n}=g(X_{1},X_{2},\ldots,X_{n}).
\]
Note that $\widehat{\theta}_{n}$ can depend only on the observed data, and not
on any unknown parameters. You have already met a number of estimators in
Mods, such as. the sample mean and sample variance. The estimator is a
function of random variables, so is itself a random variable, with a
distribution, mean, and variance, etc. \bigskip\newline Here is a familiar
property which you have already met.\bigskip\newline\textbf{Definition 2.1}
\emph{Unbiasedness} \medskip\newline$\widehat{\theta}_{n}$ is said to be
\emph{unbiased} for $\theta$ if
\[
E\left(  \widehat{\theta}_{n}\right)  =\theta,\quad\forall\theta\in\Theta.
\]
$\square$\bigskip\newline

\noindent Next a new property, but one which appeals to common-sense. The idea
is that, the larger the amount of data, the closer the estimate should be to
the parameter to be estimated. This is expressed in terms of the estimator
converging in probability to the parameter value.\bigskip\newline%
\textbf{Convergence in probability (borrowed from probability course)}

\noindent A sequence of random variables $Z_{1},Z_{2},\ldots,Z_{n},...$ is
said to \textit{converge in probability} to a constant $z$ if for any
$\epsilon>0$, as $n\rightarrow\infty$%
\[
P(\left\vert Z_{n}-z\right\vert >\epsilon)\rightarrow0.
\]
\textbf{Definition 2.2 }\emph{Consistency}\medskip\newline$\widehat{\theta
}_{n}$ is said to be \emph{consistent} for $\theta$ if
\[
\widehat{\theta}_{n}\overset{P}{\rightarrow}\theta.
\]
$\square\bigskip$

\noindent\textbf{Definition 2.3} \emph{Efficiency}\medskip\ \newline%
$\widehat{\theta}_{A}$ is said to be more \emph{efficient} than $\widehat
{\theta}_{B}$ if
\[
V\left(  \widehat{\theta}_{A}\right)  <V\left(  \widehat{\theta}_{B}\right)
,\quad\forall\theta\in\Theta.
\]
$\square\bigskip$\newline Again, this appeals to common-sense.

\subsection{Revision and extension of maximum likelihood estimation}

\noindent The basic idea starts with the joint distribution of $X=X_{1}%
,X_{2},\ldots,X_{n}$ depending upon a parameter $\theta$,
\[
f\left(  \mathbf{x};\theta\right)  =f\left(  x_{1},x_{2},\ldots,x_{n}%
;\theta\right)  .
\]
For fixed $\theta$, probability statements can be made about $X$. If we have
observations, $x$, but $\theta$ is unknown, we regard information about
$\theta$ as being contained in the likelihood
\[
L(\theta;\mathbf{x})=f\left(  \mathbf{x};\theta\right)  ,
\]
where $L$ is regarded as a function of $\theta$ with $\mathbf{x}$
fixed.\bigskip\newline\noindent\textbf{Example 2.1}\medskip\ \newline Suppose
$X=X_{1},X_{2},\ldots,X_{n}$ are independent Bernoulli random variables with
parameter $\theta\in\left[  0,1\right]  $.
\[
i.e.\quad P\left(  X_{i}=1\right)  =\theta,\ P\left(  X_{i}=0\right)
=1-\theta.
\]
Observations are $x=\left(  1,0,0,1,0,1,1\right)  $ and
\[%
\begin{array}
[c]{rcl}%
L(\theta;\mathbf{x}) & = &
%TCIMACRO{\dprod \limits_{i=1}^{7}}%
%BeginExpansion
{\displaystyle\prod\limits_{i=1}^{7}}
%EndExpansion
f\left(  x_{i};\theta\right) \\
&  & \\
& = &
%TCIMACRO{\dprod \limits_{i=1}^{7}}%
%BeginExpansion
{\displaystyle\prod\limits_{i=1}^{7}}
%EndExpansion
\theta^{x_{i}}\left(  1-\theta\right)  ^{1-x_{i}}\\
&  & \\
& = & \theta^{4}\left(  1-\theta\right)  ^{3}.
\end{array}
\]
In general, for a sample size $n$,
\[
L(\theta;\mathbf{x})=\prod\limits_{i=1}^{n}\theta^{x_{i}}\left(
1-\theta\right)  ^{1-x_{i}}=\theta^{\sum x_{i}}\left(  1-\theta\right)
^{n-\sum x_{i}}.
\]
$\blacksquare$\noindent\vspace{20pt} \ 

\noindent\textbf{Maximum likelihood estimation}\medskip\ \newline The value of
$\theta$ which maximises $L(\theta)$ is called the \emph{maximum likelihood
estimate} of $\theta$.\bigskip\medskip\ \newline\textbf{Example 2.1}%
\medskip\textbf{\ (continued)\newline}In the previous example, we can find
this by differentiating $L(\theta)$, or equivalently by differentiating
$l(\theta)=\log L(\theta).$%
\[%
\begin{array}
[c]{rcl}%
l(\theta) & = & \sum_{i}x_{i}\log\theta+\left(  n-\sum_{i}x_{i}\right)
\log\left(  1-\theta\right) \\
&  & \\
\dfrac{\partial l(\theta)}{\partial\theta} & = & \left.  \sum_{i}x_{i}\right/
\theta-\left.  \left(  n-\sum_{i}x_{i}\right)  \right/  \left(  1-\theta
\right)
\end{array}
\]
Putting $\dfrac{\partial l(\theta)}{\partial\theta}=0$ we obtain
\[
\theta=\left.  \sum\nolimits_{i}x_{i}\right/  n.
\]
\medskip\ \newline This leads us to $\widehat{\theta}_{n},$ the \emph{maximum
likelihood estimator} based on a sample of size $n$%
\[
\widehat{\theta}_{n}=\left.  \sum\nolimits_{i}X_{i}\right/  n.
\]
$\blacksquare$\vspace{20pt}

\noindent\textbf{Example 2.2}\medskip\ \newline Suppose $X=X_{1},X_{2}%
,\ldots,X_{n}$ are independent normal random variables, $N\left(  \mu
,\sigma^{2}\right)  $.
\[%
\begin{array}
[c]{rcl}%
L\left(  \mu,\sigma^{2}\right)  & = &
%TCIMACRO{\dprod \limits_{i=1}^{n}}%
%BeginExpansion
{\displaystyle\prod\limits_{i=1}^{n}}
%EndExpansion
\dfrac{1}{\sqrt{2\pi\sigma^{2}}}\exp\left[  -\dfrac{\left(  x_{i}-\mu\right)
^{2}}{2\sigma^{2}}\right] \\
&  & \\
& = & \left(  2\pi\sigma^{2}\right)  ^{-\left.  n\right/  2}\exp\left[
-\dfrac{1}{2\sigma^{2}}%
%TCIMACRO{\dsum \limits_{i=1}^{n}}%
%BeginExpansion
{\displaystyle\sum\limits_{i=1}^{n}}
%EndExpansion
\left(  x_{i}-\mu\right)  ^{2}\right]
\end{array}
\]
and
\[%
\begin{array}
[c]{rcl}%
l\left(  \mu,\sigma^{2}\right)  & = & \log L\left(  \mu,\sigma^{2}\right) \\
&  & \\
& = & -\dfrac{n}{2}\log\left(  2\pi\sigma^{2}\right)  -\dfrac{1}{2\sigma^{2}}%
%TCIMACRO{\dsum \limits_{i=1}^{n}}%
%BeginExpansion
{\displaystyle\sum\limits_{i=1}^{n}}
%EndExpansion
\left(  x_{i}-\mu\right)  ^{2}%
\end{array}
\]
Differentiating,
\[%
\begin{array}
[c]{rcl}%
\dfrac{\partial l\left(  \mu,\sigma^{2}\right)  }{\partial\mu} & = & \dfrac
{1}{\sigma^{2}}%
%TCIMACRO{\dsum \limits_{i=1}^{n}}%
%BeginExpansion
{\displaystyle\sum\limits_{i=1}^{n}}
%EndExpansion
\left(  x_{i}-\mu\right)  ,\\
&  & \\
\dfrac{\partial l\left(  \mu,\sigma^{2}\right)  }{\partial\sigma^{2}} & = &
-\dfrac{n}{2\sigma^{2}}+\dfrac{1}{2\sigma^{4}}%
%TCIMACRO{\dsum \limits_{i=1}^{n}}%
%BeginExpansion
{\displaystyle\sum\limits_{i=1}^{n}}
%EndExpansion
\left(  x_{i}-\mu\right)  ^{2}%
\end{array}
\]
Equating these derivatives to zero results in
\[
\widehat{\mu}=\overline{X},\quad\widehat{\sigma}^{2}=\frac{1}{n}%
\sum\limits_{i=1}^{n}\left(  X_{i}-\widehat{\mu}\right)  ^{2}=\frac{1}{n}%
\sum\limits_{i=1}^{n}\left(  X_{i}-\overline{X}\right)  ^{2}.
\]
$\blacksquare$\newpage

\subsection{Order statistics}

To expand our armoury of useful and descriptive statistics we shall need to
know about \emph{order statistics}.\bigskip\newline\textbf{Definition
2.4}\quad\emph{Order statistics}\textbf{\medskip}\newline Suppose that $X_{1}$
, $X_{2}$, . . , $X_{n}$ is a random sample then the full set of \textit{order
statistics} is the re-ordering $X_{\left(  1\right)  }$ , $X_{\left(
2\right)  }$, . . , $X_{\left(  n\right)  }$ such that $X_{\left(  1\right)
}\leq X_{\left(  2\right)  }\leq.\;.\leq X_{\left(  n\right)  }.$ In
particular the minimum order statistic is $X_{\left(  1\right)  }%
=\mathrm{\min\,}X_{i}$ and the maximum order statistic is $X_{\left(
n\right)  }=\max\,X_{i}$. If the random sample is derived from a continuous
distribution then all the inequalities will be strict.\newline$\square
$\bigskip\newline\textbf{Example 2.3}\medskip\ \textit{US wages}\newline The
data in Table 2.1 are shown below, with the data having been ordered.%
\[%
\begin{tabular}
[c]{ccccc}%
\multicolumn{5}{l}{\textbf{Table 2.1:} \textit{US wages}}\\\hline
101 & 107 & 112 & 119 & 140\\
103 & 107 & 112 & 119 & 151\\
103 & 108 & 115 & 123 & 154\\
104 & 108 & 115 & 125 & 156\\
104 & 111 & 116 & 128 & 157\\
105 & 112 & 119 & 132 & 158\\\hline
\end{tabular}
\]
You can see that the realisation of the minimum order statistic is 101 and the
realisation of the maximum order statistic is 158.\newline$\blacksquare$
\smallskip\ \newline The distributions of $X_{(1)}=\min\left\{  X_{i}\right\}
$ and $X_{(n)}=\max\left\{  X_{i}\right\}  $ are straightforward to derive.
The c.d.f. of $X_{(1)} $ is
\begin{align*}
F_{\left(  1\right)  }(x)  & =1-P\left(  X_{(1)}>x)\right)  =1-P\left(
{X}_{1}>x,\ldots,X_{n}>x\right) \\
& =1-\left[  1-F(x)\right]  ^{n}%
\end{align*}
so that, differentiating,
\[
f_{\left(  1\right)  }(x)=F_{\left(  1\right)  }^{\prime}(x)=n\left[
1-F(x)\right]  ^{n-1}f(x).
\]
Similarly
\[
F_{\left(  n\right)  }(x)=P\left(  X_{(n)}\leq x\right)  =F(x)^{n}%
\]
giving
\[
f_{\left(  n\right)  }(x)=nF(x)^{n-1}f(x).
\]
\textbf{Example 2.4}\medskip\ \ \newline Suppose $X_{i}\sim U(0,\theta)$.
Then
\[
F(x)=\left\{
\begin{array}
[c]{cc}%
0, & x<0,\\
\dfrac{x}{\theta}, & 0\leq x<\theta,\\
1, & x\geq\theta,
\end{array}
\right.  \quad\mathrm{{and}\quad}f(x)=\left\{
\begin{array}
[c]{cc}%
\dfrac{1}{\theta}, & x\in(0,\theta),\\
0, & \mathrm{otherwise}{.}%
\end{array}
\right.
\]
\medskip\ \newline The p.d.f.'s of $X_{(1)}=\min\left\{  X_{i}\right\}  $ and
$X_{(n)}=\max\left\{  X_{i}\right\}  $ are given by
\[
f_{\left(  1\right)  }(x)=\frac{n\left(  \theta-x\right)  ^{n-1}}{\theta^{n}%
},\quad x\in(0,\theta)
\]
and
\[
f_{\left(  n\right)  }(x)=\frac{nx^{n-1}}{\theta^{n}},\quad x\in(0,\theta).
\]
$\blacksquare\medskip$\newline For a continuous random sample, each with
density function $f(x)$ and cumulative distribution function $F(x)$, the joint
density function of the order statistics is
\[
f(x_{\left(  1\right)  },x_{\left(  2\right)  },..,x_{\left(  n\right)
})=n!\prod\limits_{i=1}^{n}\;f(x_{\left(  i\right)  }),\quad x_{\left(
1\right)  }<x_{\left(  2\right)  }<\cdots<x_{\left(  n\right)  },
\]
and the marginal distribution for $X_{\left(  r\right)  }$ is
\[
f_{\left(  r\right)  }(x)=\frac{n!}{\left(  r-1\right)  !(n-r)!}%
F(x)^{r-1}(1-F(x))^{n-r}f(x).
\]
\textbf{Lemma 2.1}\quad The p.d.f. of $X_{(r)}$ is given by
\[
f_{\left(  r\right)  }(x)=\frac{n!}{(r-1)!(n-r)!}F(x)^{r-1}\left[
1-F(x)\right]  ^{n-r}f(x).
\]
$\square$\medskip\ \newline\textbf{Proof}\quad Putting $r=1$ and $r=n$ in the
expression gives
\[
f_{\left(  1\right)  }(x)=n\left[  1-F(x)\right]  ^{n-1}f(x),\quad f_{\left(
n\right)  }(x)=nF(x)^{n-1}f(x).
\]
The c.d.f. $F_{\left(  r\right)  }$ of $X_{(r)}$ is
\[
F_{\left(  r\right)  }(x)=P\left(  X_{(r)}\leq x\right)  =\sum_{j=r}%
^{n}{\binom{n}{j}}\left[  F(x)\right]  ^{j}\left[  1-F(x)\right]  ^{n-j}%
\]
\emph{i.e.} the probability that at least $r$ of the $X_{i}$'s are less than
or equal to $x$.\medskip\newline Therefore
\[
F_{\left(  r\right)  }(x)-F_{\left(  r+1\right)  }(x)={\binom{n}{r}}\left[
F(x)\right]  ^{r}\left[  1-F(x)\right]  ^{n-r}%
\]
and, by differentiation,
\begin{align*}
f_{\left(  r+1\right)  }(x)  & =f_{\left(  r\right)  }(x)-{\binom{n}{r}%
}\left[  F(x)\right]  ^{r-1}\left[  1-F(x)\right]  ^{n-r-1}\left[
r-nF(x)\right]  f(x)\\
& \\
& =\frac{n!}{r!(n-r)!}F(x)^{r-1}\left[  1-F(x)\right]  ^{n-r-1}(n-r)F(x)f(x)\\
& \\
& =\frac{n!}{r!\left(  n-(r+1)\right)  !}F(x)^{(r+1)-1}\left[  1-F(x)\right]
^{n-(r+1)}f(x),
\end{align*}
\medskip\ \newline so the proposition is proved inductively.\newline%
$\blacksquare\medskip$\newline There are many uses of the order statistics in
terms of analysing and describing data, three in particular being the
\emph{median}, the \emph{lower quartile} and the \emph{upper quartile}%
.\bigskip\newline\textbf{Definition 2.5}\quad\emph{Median}\medskip\newline The
\emph{median} of a random sample is defined by%
\[
X_{m}=X_{\left(  \frac{1}{2}\left(  n+1\right)  \right)  .}%
\]
This is the central order statistic if the size of the sample is odd and is
the average of the two central order statistics if it is even, so that
\[
\text{ if\quad}n=2k+1,\text{ then }X_{m}=X_{\left(  k+1\right)  },\text{\quad
and if\quad}n=2k,\text{ then }X_{m}=\frac{1}{2}\left(  X_{\left(  k\right)
}+X_{\left(  k+1\right)  }\right)  .
\]
$\square\medskip$\newline\textbf{Example 2.3 (revisited)}\quad\textit{US
wages}\newline For the wages data, the realisation is $x_{m}=\frac{1}%
{2}\left(  115+115\right)  .$ Thus 50\% of the data have a smaller value than
the median and 50\% of the data have a higher value.\newline$\blacksquare
\bigskip$\newline\textbf{Definition 2.6}\quad\emph{Quartiles}\medskip\newline
The \emph{lower quartile} is defined by%
\[
q_{L}=X_{\left(  \frac{1}{4}\left(  n+1\right)  \right)  },
\]
\newline and the \emph{upper quartile} is defined by%
\[
q_{U}=X_{\left(  \frac{3}{4}\left(  n+1\right)  \right)  }.
\]
$\square\medskip$\newline\textbf{Example 2.3 (revisited)}\quad\textit{US
wages}\newline The wages data comprises 30 values and $31/4=7\frac{3}{4}$, so
the realisation of the lower quartile is $q_{L}=x_{\left(  7\frac{3}%
{4}\right)  }=107+\frac{3}{4}\left(  107-107\right)  =107.$ The realisation of
the upper quartile is $q_{U}=x_{\left(  23\frac{1}{4}\right)  }=128+\frac
{1}{4}\left(  132-128\right)  =129.$\newline$\blacksquare\medskip$\newline You
will see one of the main uses of order statistics in the next sub-section,
where we look at how to produce useful plots of the data.

\subsubsection{Plotting the data}

For univariate data there are many useful plots each of which attempts to give
a graphical representation of the data. These include histograms, boxplots and
plots of empirical cumulative distribution functions (see \textit{Rice,
Chapter} 10 for examples and descriptions of these plots). For paired data,
scatter plots are a very quick way of visually checking for association
between the paired observations.\bigskip\newline\textbf{Boxplots}%
\medskip\newline Boxplots are particularly useful in giving an intuitive feel
for the way the data are distributed. They are very easy to construct, but
their main advantage is that all computer packages can produce them on-screen
instantly. Let us look at how they are constructed.%
%TCIMACRO{\FRAME{dhFU}{3.9435in}{1.9329in}{0pt}{\Qcb{\QTR{bf}{Figure 2.1}
%Boxplot construction}}{}{a5lhood_f1.eps}%
%{\special{ language "Scientific Word";  type "GRAPHIC";
%maintain-aspect-ratio TRUE;  display "USEDEF";  valid_file "F";
%width 3.9435in;  height 1.9329in;  depth 0pt;  original-width 4.1226in;
%original-height 2.0055in;  cropleft "0";  croptop "1";  cropright "1";
%cropbottom "0";  filename 'a5lhood_f1.eps';file-properties "XNPEU";}}}%
%BeginExpansion
\begin{center}
\includegraphics[
height=1.9329in,
width=3.9435in
]%
{a5lhood_f1.eps}%
\\
\textbf{Figure 2.1} Boxplot construction
\end{center}
%EndExpansion
With one exception the plot is largely self-explanatory; that exception is the
idea of adjacent values. These are calculated by first calculating the
inter-quartile range $q_{U}-q_{L}$; then the upper adjacent value is the
largest value not exceeding that distance from the upper quartile, the lower
adjacent value is the smallest value not less than that distance from the
lower quartile. Any point which lies outside these boundaries is depicted as a
point and considered as a possible outlier (i.e. a value which is non-typical
for some reason or other). Thus we have a picture which, in some sense,
represents the way the data are distributed.\bigskip\newline\textbf{Example
2.3 (revisited)}\quad\textit{US wages}\newline A boxplot of the wage data,
produced using the R package, is shown in Figure 2.2.%
%TCIMACRO{\FRAME{dhFU}{3.5371in}{3.5267in}{0pt}{\Qcb{\QTR{bf}{Figure 2.2}
%Boxplot of US wages data}}{}{a5lhood_f2.eps}%
%{\special{ language "Scientific Word";  type "GRAPHIC";
%maintain-aspect-ratio TRUE;  display "USEDEF";  valid_file "F";
%width 3.5371in;  height 3.5267in;  depth 0pt;  original-width 5.8496in;
%original-height 5.8323in;  cropleft "0";  croptop "1";  cropright "1";
%cropbottom "0";  filename 'a5lhood_f2.eps';file-properties "XNPEU";}}}%
%BeginExpansion
\begin{center}
\includegraphics[
height=3.5267in,
width=3.5371in
]%
{a5lhood_f2.eps}%
\\
\textbf{Figure 2.2} Boxplot of US wages data
\end{center}
%EndExpansion
It shows some skewness in the data and some possible outliers, but overall you
can see that most of the workers at that time earned between \$10,000 and
\$14,000 with a median wage around \$11,500 and 50\% of workers earning
between just over \$10,500 and just under \$13,000.\newline$\blacksquare
\medskip$\newline But boxplots are \emph{really} useful when it comes to
comparison.\bigskip\newline\textbf{Example 2.5}\quad\textit{Infants with
SIRDS}\medskip\newline The data set in Table 2.2 is reproduced below.\newline%
\[
.%
\begin{tabular}
[c]{llllll}%
\multicolumn{6}{l}{\textbf{Table 2.2} Birth weights (kg) of infants with}\\
\multicolumn{6}{l}{severe ideopathic respiratory distress syndrome}%
\\\cline{1-5}%
1.050* & 2.500* & 1.890* & 1.760 & 2.830 & \\
1.175* & 1.030* & 1.940* & 1.930 & 1.410 & \\
1.230* & 1.100* & 2.200* & 2.015 & 1.715 & \\
1.310* & 1.185* & 2.270* & 2.090 & 1.720 & \\
1.500* & 1.225* & 2.440* & 2.600 & 2.040 & \\
1.600* & 1.262* & 2.560* & 2.700 & 2.200 & \\
1.720* & 1.295* & 2.730* & 2.950 & 2.400 & \\
1.750* & 1.300* & 1.130 & 3.160 & 2.550 & \\
1.770* & 1.550* & 1.575 & 3.400 & 2.570 & \\
2.275* & 1.820* & 1.680 & 3.640 & 3.005 & \\\cline{1-5}%
\multicolumn{5}{l}{{*} child died} & \\\cline{1-5}%
\end{tabular}
\]
One way of comparing the birthweights of infants who died with the
birthweights of those who survived is to use a graph with the boxplots
compared against the same scale.%
%TCIMACRO{\FRAME{dhFU}{3.5362in}{3.5267in}{0pt}{\Qcb{\QTR{bf}{Figure 2.3}
%Boxplots of SIRDS data}}{}{a5lhood_f3.eps}%
%{\special{ language "Scientific Word";  type "GRAPHIC";
%maintain-aspect-ratio TRUE;  display "USEDEF";  valid_file "F";
%width 3.5362in;  height 3.5267in;  depth 0pt;  original-width 5.8496in;
%original-height 5.8323in;  cropleft "0";  croptop "1";  cropright "1";
%cropbottom "0";  filename 'a5lhood_f3.eps';file-properties "XNPEU";}}}%
%BeginExpansion
\begin{center}
\includegraphics[
height=3.5267in,
width=3.5362in
]%
{a5lhood_f3.eps}%
\\
\textbf{Figure 2.3} Boxplots of SIRDS data
\end{center}
%EndExpansion
The first thing you notice about this plot is that the median value of
birtheweight for children who died is less than the lower quartile for those
who survived. The plot makes it immediately clear that the two groups have
typically different birthweights.\newline$\blacksquare$\bigskip\newline%
\textbf{Probability Plots\medskip}\newline These are sometimes called
\emph{quantile-quantile plots}. They can be used to examine the plausibility
of a density function from which a set of data could have been drawn. If such
a known distribution is found then analysis of the data is performed more
readily, particularly if this distribution should turn out to be normal. We
need some theory before we develop the graphical method.\smallskip
\bigskip\ \newline\textbf{Definition 2.7\medskip}\newline Let $X$ be a
continuous random variable with cumulative distribution function $F_{X}(x)$,
then the \emph{probability integral transform} is
\[
Y=F_{X}(X).
\]
$\square\medskip$\newline The transformed random variable $Y$ has a
particularly useful property.\bigskip\newline\textbf{Lemma 2.2\medskip}\quad
For $F_{X}(x)$ strictly monotonic increasing on $\left(  a,b\right)  $ , $X$
taking values only in $\left(  a,b\right)  ,$ then if $Y=$ $F_{X}(X),$%
\[
Y\sim U(0,1).
\]
$Y$ so defined is uniformly distributed on $(0,1)$.\newline$\square\medskip
$\newline\textbf{Proof}\quad For $0\leq y\leq1,$%
\[
P\left(  Y\leq y\right)  =P\left(  F_{X}(X)\leq y\right)  =P\left(  X\leq
F_{X}^{-1}(y)\right)  =F_{X}\left(  F_{X}^{-1}(y)\right)  =y,
\]
giving the cumulative distribution function of the uniform distribution on
$(0,1)$.\newline$\blacksquare\medskip$\newline We shall also need a result
concerning the expected values of order statistics from a uniform
distribution.\bigskip\newline\textbf{Lemma 2.3}\quad If $Y_{\left(  1\right)
},Y_{\left(  2\right)  },\ldots,Y_{\left(  n\right)  }$ are the order
statistics of a random sample drawn from a uniform distribution $U(0,1)$,
then
\[
E\left(  Y_{\left(  k\right)  }\right)  =\frac{k}{n+1}=P\left(  Y_{i}\leq
\frac{k}{n+1}\right)  .
\]
[Proof left as an exercise]\newline$\square\medskip$\newline We conclude that,
for the uniform distribution, $E\left(  Y_{\left(  1\right)  }\right)
,E\left(  Y_{\left(  2\right)  }\right)  ,\ldots,E(Y_{\left(  n\right)  })$
splits the interval $(0,1)$ into equal bites of probability, each of length
$\dfrac{1}{n+1}$.\smallskip\ \newline Using the results of the two lemmas, it
is now possible to give the method of construction for a probability plot. The
question we wish to address is:\newline\textit{`Is the data set plausibly
drawn from a distribution with cdf }$F(x)? $'\textit{\ }\smallskip\ \newline
To construct a probability plot, you take the following steps:

\begin{enumerate}
\item[(i)] Arrange the data as $x_{\left(  1\right)  }<x_{\left(  2\right)
}<\cdots<x_{\left(  n\right)  }.$ From Lemma 2.2, if the supposition is
correct, $F(x_{\left(  1\right)  }),F(x_{\left(  2\right)  }),\ldots
,F(x_{\left(  n\right)  })$ are realisations of order statistics from
$U(0,1).$

\item[(ii)] Solve%
\[
F(z_{\left(  k\right)  })=\frac{k}{n+1}%
\]
\ for $z_{\left(  k\right)  }$: remember that $E\left(  F(X_{\left(  k\right)
}\right)  =\dfrac{k}{n+1}$.

\item[(iii)] Plot the ordered pairs $\left(  x_{\left(  k\right)  },z_{\left(
k\right)  }\right)  $.
\end{enumerate}

\noindent If the supposition is reasonable we should get an approximate
straight line, $z=x$.\medskip\newline If the distribution contains some
unknown parameters we need to adapt the method, if possible.\bigskip
\newline\textbf{Example 2.6}\medskip\newline Suppose we wish to check whether
a distribution is exponential with mean $\mu$. Then we note $F(x)=1-e^{-x/\mu
},$ so that
\[
\frac{k}{n+1}\approx1-\mathrm{e}^{-x_{\left(  k\right)  }/\mu}.
\]
Re-arranging
\[
x_{\left(  k\right)  }\approx-\mu\log\left(  1-\frac{k}{n+1}\right)  .
\]
Hence, if the distribution is plausibly exponential, plot $\left(  x_{\left(
k\right)  },\log\left(  1-\frac{k}{n+1}\right)  \right)  $ and expect to see
an approximate straight line.\smallskip\ The slope will provide an estimate of
$\mu$.\ \newline$\blacksquare$\bigskip\newline\textbf{Example 2.3
(revisited)\quad\ }\medskip\textit{US wages}\newline Many financial modellers
assume that wages have a \emph{Pareto distribution} with c.d.f.%
\[
F\left(  x\right)  =1-\left(  \frac{\alpha}{x}\right)  ^{\theta},\quad
x>\alpha,
\]
where $\alpha$ represents the minimum wage and both $\alpha$ and $\theta$ are
unknown parameters. Writing%
\[
\frac{k}{n+1}\approx1-\left(  \frac{\alpha}{x_{\left(  k\right)  }}\right)
^{\theta}%
\]
and re-arranging,%
\[
x_{\left(  k\right)  }\approx\alpha\left(  1-\frac{k}{n+1}\right)
^{-1/\theta}.
\]
Clearly we cannot produce a probability plot because we do not know the value
of $\theta$, but taking logs of both sides gives%
\[
\log x_{\left(  k\right)  }\approx\log\alpha-\frac{1}{\theta}\log\left(
1-\frac{k}{n+1}\right)  ,
\]
so plotting ordered pairs%
\[
\left(  \log x_{\left(  k\right)  },\log\left(  1-\frac{k}{n+1}\right)
\right)
\]
should produce a straight line if the distribution is plausible as a model for
the US wages data.%
%TCIMACRO{\FRAME{dhFU}{3.8303in}{3.8181in}{0pt}{\Qcb{\QTR{bf}{Figure 2.4}
%Pareto probability plot for US wages data}}{}{a5lhood_f4.eps}%
%{\special{ language "Scientific Word";  type "GRAPHIC";
%maintain-aspect-ratio TRUE;  display "USEDEF";  valid_file "F";
%width 3.8303in;  height 3.8181in;  depth 0pt;  original-width 5.8496in;
%original-height 5.8323in;  cropleft "0";  croptop "1";  cropright "1";
%cropbottom "0";  filename 'a5lhood_f4.eps';file-properties "XNPEU";}}}%
%BeginExpansion
\begin{center}
\includegraphics[
height=3.8181in,
width=3.8303in
]%
{a5lhood_f4.eps}%
\\
\textbf{Figure 2.4} Pareto probability plot for US wages data
\end{center}
%EndExpansion
As you can see, a Pareto probability model doesn't look very
convincing.\newline$\blacksquare$\bigskip\newline In practice the distribution
we wish to consider is frequently the normal distribution, which has two
unknown parameters. We do a little more adjustment.\bigskip\smallskip
\ \newline\textbf{Definition 2.8}\quad\emph{Normal scores\medskip\newline}The
numbers
\[
z_{\left(  k\right)  }=\Phi^{-1}\left(  \frac{k}{n+1}\right)  ,
\]
where $\Phi$ is the c.d.f. of $Z\sim N(0,1)$, are called the \textit{normal
scores}. They are the $\dfrac{k}{n+1}$ quantiles of the standard normal
distribution.\smallskip\newline$\square\bigskip$\ \newline We use the normal
scores to construct a \emph{normal probability plot}. Suppose that we wish to
check whether or not the data could have been drawn from a normal distribution
$N\left(  \mu,\sigma^{2}\right)  .$ As above take the ordered data $x_{\left(
1\right)  }<x_{\left(  2\right)  }<\cdots<x_{\left(  n\right)  }.$ Note that,
if $X\sim N\left(  \mu,\sigma^{2}\right)  $, then
\[
\frac{X-\mu}{\sigma}\sim N(0,1).
\]
Plot the pairs$\left(  x_{\left(  k\right)  },z_{\left(  k\right)  }\right)
.$ If the data are plausibly normal then we should have an approximate
straight line with slope $1/\sigma$ and intercept $\mu/\sigma.\medskip
$\newline In practice you will find that most statistical computer packages
will generate normal scores and produce probability plots: in fact, most
packages will do this for a variety of distributions. The plots in the
examples which follow were generated using the R package.\bigskip
\newline\textbf{Example 2.7}\quad\emph{Silver content of Byzantine
coins\medskip}\newline The data are reproduced below.%
\[%
\begin{tabular}
[c]{ccccc}%
\multicolumn{5}{l}{\textbf{Table 2.3}\quad Silver content of coins}%
\\\cline{1-4}%
First & Second & Third & Fourth & \\\cline{1-4}%
5.9 & 6.9 & 4.9 & 5.3 & \\
6.8 & 9.0 & 5.5 & 5.6 & \\
6.4 & 6.6 & 4.6 & 5.5 & \\
7.0 & 8.1 & 4.5 & 5.1 & \\
6.6 & 9.3 &  & 6.2 & \\
7.7 & 9.2 &  & 5.8 & \\
7.2 & 8.6 &  & 5.8 & \\
6.9 &  &  &  & \\
6.2 &  &  &  & \\\cline{1-4}%
\end{tabular}
\]
The graph shows a normal probability plot of the silver content of the first
coinage.%
%TCIMACRO{\FRAME{dhFU}{3.8303in}{3.8181in}{0pt}{\Qcb{\QTR{bf}{Figure
%2.5}\ Silver content of coins}}{}{a5lhood_f5.eps}%
%{\special{ language "Scientific Word";  type "GRAPHIC";
%maintain-aspect-ratio TRUE;  display "USEDEF";  valid_file "F";
%width 3.8303in;  height 3.8181in;  depth 0pt;  original-width 5.8496in;
%original-height 5.8323in;  cropleft "0";  croptop "1";  cropright "1";
%cropbottom "0";  filename 'a5lhood_f5.eps';file-properties "XNPEU";}}}%
%BeginExpansion
\begin{center}
\includegraphics[
height=3.8181in,
width=3.8303in
]%
{a5lhood_f5.eps}%
\\
\textbf{Figure 2.5}\ Silver content of coins
\end{center}
%EndExpansion
It looks as if a normal distribution is appropriate.\newline$\blacksquare
\bigskip$\newline\textbf{Example 2.8}\quad\emph{Radio-carbon dating\medskip
}\newline The data from Table 2.4 are reproduced below.%
\[%
\begin{tabular}
[c]{lcc}%
\multicolumn{3}{l}{\textbf{Table 2.4}\quad Radiocarbon dating}\\\cline{1-2}%
Sample & Radiocarbon age & \\
number & determination & \\\cline{1-2}%
C-288 & 2419 & \\
M-26 & 2485 & \\
C-367 & 3433 & \\
M-195 & 2575 & \\
M-911 & 2521 & \\
M-912 & 2451 & \\
Y-1279 & 2550 & \\
Y-1280 & 2540 & \\\cline{1-2}%
\end{tabular}
\]
Radioactive-carbon dating was undertaken on 8 samples from a single early
site. There is one rather obvious outlier which is easily seen from the plot.%
%TCIMACRO{\FRAME{dhFU}{3.8303in}{3.8181in}{0pt}{\Qcb{\QTR{bf}{Figure 2.6}
%Radiocarbon dating}}{}{a5lhood_f6.eps}{\special{ language "Scientific Word";
%type "GRAPHIC";  maintain-aspect-ratio TRUE;  display "USEDEF";
%valid_file "F";  width 3.8303in;  height 3.8181in;  depth 0pt;
%original-width 5.8496in;  original-height 5.8323in;  cropleft "0";
%croptop "1";  cropright "1";  cropbottom "0";
%filename 'a5lhood_f6.eps';file-properties "XNPEU";}}}%
%BeginExpansion
\begin{center}
\includegraphics[
height=3.8181in,
width=3.8303in
]%
{a5lhood_f6.eps}%
\\
\textbf{Figure 2.6} Radiocarbon dating
\end{center}
%EndExpansion
If possible we would check out the outlier with the laboratory and the
collector but, since this is not possible, we will drop it and consider
whether the remaining data points could have been drawn from a normal
distribution.%
%TCIMACRO{\FRAME{dhFU}{3.8303in}{3.8181in}{0pt}{\Qcb{Figure 2.7 Radiocarbon
%dating}}{}{a5lhood_f7.eps}{\special{ language "Scientific Word";
%type "GRAPHIC";  maintain-aspect-ratio TRUE;  display "USEDEF";
%valid_file "F";  width 3.8303in;  height 3.8181in;  depth 0pt;
%original-width 5.8496in;  original-height 5.8323in;  cropleft "0";
%croptop "1";  cropright "1";  cropbottom "0";
%filename 'a5lhood_f7.eps';file-properties "XNPEU";}}}%
%BeginExpansion
\begin{center}
\includegraphics[
height=3.8181in,
width=3.8303in
]%
{a5lhood_f7.eps}%
\\
Figure 2.7 Radiocarbon dating
\end{center}
%EndExpansion
As can be seen from Figure 2.7, the 7 remaining data points do give an
approximate straight line on a normal probability plot.\newline$\blacksquare$

\subsection{Comparing estimators\medskip}

Remember that we began with three desirable properties of estimators. These
were\medskip\newline%
\begin{tabular}
[c]{ll}%
\emph{Unbiasedness} & $E\left(  \widehat{\theta}_{n}\right)  =\theta
,\quad\forall\theta\in\Theta.$\\
\emph{Consistency} & $\widehat{\theta}_{n}\overset{P}{\rightarrow}\theta.$\\
\emph{Efficiency}\medskip & $\widehat{\theta}_{A}$ is more \emph{efficient}
than $\widehat{\theta}_{B}$ if $V\left(  \widehat{\theta}_{A}\right)
<V\left(  \widehat{\theta}_{B}\right)  ,\quad\forall\theta\in\Theta.$%
\end{tabular}
\medskip\newline Even if we assume unbiasedness and consistency to be
desirable, it is possible to have more than one such estimator.\vspace{20pt}
\newline\textbf{Example\medskip\ 2.9}\ \newline Consider the linear estimator
\[
\widehat{\theta}_{n}=\sum_{i=1}^{n}a_{i}X_{i}%
\]
where $E\left(  X_{i}\right)  =\theta,\ V\left(  X_{i}\right)  =\sigma^{2}$
for $1\leq i\leq n$.
\[
E\left(  \widehat{\theta}_{n}\right)  =\sum_{i=1}^{n}a_{i}E\left(
X_{i}\right)  =\theta\sum_{i=1}^{n}a_{i},
\]
so the estimator is unbiased provided
\[
\sum_{i=1}^{n}a_{i}=1.
\]
\ 

\noindent For i.i.d. random variables,
\[
V\left(  \widehat{\theta}_{n}\right)  =\sum\limits_{i=1}^{n}a_{i}^{2}%
\sigma^{2}.
\]
Now
\[%
\begin{array}
[c]{rrrcl}
&  &
%TCIMACRO{\dsum \limits_{i=1}^{n}}%
%BeginExpansion
{\displaystyle\sum\limits_{i=1}^{n}}
%EndExpansion
\left(  a_{i}-\dfrac{1}{n}\right)  ^{2} & \geq & 0\\
& \Rightarrow & \sum\limits_{i=1}^{n}a_{i}^{2}-\dfrac{2}{n}\sum\limits_{i=1}%
^{n}a_{i}+\dfrac{1}{n} & \geq & 0\\
\mathrm{and,\ if\ }\sum_{i=1}^{n}a_{i}=1, &  &  &  & \\
& \Rightarrow & \sum\limits_{i=1}^{n}a_{i}^{2} & \geq & \dfrac{1}{n}.
\end{array}
\]
Equality occurs iff $a_{i}=\dfrac{1}{n},\ 1\leq i\leq n.$\medskip\newline The
conclusion then is that, if $\widehat{\theta}_{n}$ is a linear unbiased
estimator of the form $\sum_{i=1}^{n}a_{i}X_{i}$ and if $\overline{X}%
=\dfrac{1}{n}\sum_{i=1}^{n}X_{i},$ then
\[
V\left(  \overline{X}\right)  \leq V\left(  \widehat{\theta}_{n}\right)  .
\]
\medskip\ \newline\emph{Of all linear unbiased estimators}, $\overline{X}%
$\emph{\ is most efficient}.\newline$\blacksquare$\ 

\subsubsection{Food for thought}

\begin{enumerate}
\item Is the sample mean the ``best'' estimator of the distribution
mean?\medskip\ \newline

\item If $\overline{X}$ is unbiased for $\theta$, is $g\left(  \overline
{X}\right)  $ unbiased for $g\left(  \theta\right)  $?\medskip\ \newline

\item Is there any reason to doubt the principle of seeking the unbiased
estimator with the minimum variance?\medskip\ \newline

\item Can a biased estimator be \textquotedblleft better\textquotedblright%
\ (whatever that means!) than an unbiased estimator?
\end{enumerate}

\paragraph{Is the sample mean the \textquotedblleft best\textquotedblright%
\ estimator of the distribution mean?}

\ \medskip\newline Let $X_{1},X_{2},\ldots,X_{n}$ be a random sample from a
uniform distribution $U(\theta-\frac{1}{2},\theta+\frac{1}{2}).$%
\[
E\left(  \overline{X}\right)  =\theta,\quad V\left(  \overline{X}\right)
=\frac{1}{12n}.
\]
\medskip\ \newline Now consider $\widehat{\theta}_{n}=\frac{1}{2}\left(
X_{(1)}+X_{(n)}\right)  $. Obviously $\widehat{\theta}_{n}$ is symmetrically
distributed about $\theta$ so
\[
E\left(  \widehat{\theta}_{n}\right)  =\theta.
\]
Now
\[%
\begin{array}
[c]{ccr}%
f_{X_{(1)}X_{(n)}}(u,v) & = & n(n-1)\left[  F(v)-F(u)\right]  ^{n-2}%
f(u)f(v),\\
&  & \theta-\frac{1}{2}<u<v<\theta+\frac{1}{2}.
\end{array}
\]
where
\[
F(x)=\left\{
\begin{array}
[c]{ll}%
0, & x<\theta-\frac{1}{2},\\
x+\frac{1}{2}-\theta, & \theta-\frac{1}{2}\leq x<\theta+\frac{1}{2},\\
1, & x\geq\theta+\frac{1}{2},
\end{array}
\right.
\]
and
\[
f(x)=F^{\prime}(x),
\]


\noindent so that
\[%
\begin{array}
[c]{ccl}%
f_{X_{(1)}X_{(n)}}(u,v) & = & n(n-1)\left[  v-u\right]  ^{n-2},\\
&  & \qquad\theta-\frac{1}{2}<u<v<\theta+\frac{1}{2}.
\end{array}
\]
\medskip\ \newline You have my word as a gentleman that, after a lot of boring
slog,
\[
V\left(  \widehat{\theta}_{n}\right)  =\frac{1}{2(n+1)(n+2)}%
\]
and
\[
V\left(  \widehat{\theta}_{n}\right)  <V\left(  \overline{X}\right)
\quad\mathrm{{for}\quad}n>2\mathrm{.}%
\]
\medskip\ \newline\emph{Question}: Does this mean we can sometimes do better
by ignoring some of the data values?

\ 

\paragraph{If $\overline{X}$ is unbiased for $\theta$, is $g\left(
\overline{X}\right)  $ unbiased for $g\left(  \theta\right)  $?}

\ \newline Let
\[
f(x)=\frac{1}{\theta}e^{-\left.  x\right/  \theta},\quad x\geq0.
\]
Then
\[
E\left(  \overline{X}\right)  =\frac{1}{n}\sum_{i=1}^{n}E\left(  X_{i}\right)
=\theta,\quad V\left(  \overline{X}\right)  =\frac{1}{n^{2}}\sum_{i=1}%
^{n}V\left(  X_{i}\right)  =\frac{\theta^{2}}{n}.
\]
Now consider $g(x)=x^{2}.$\medskip\
\begin{align*}
E\left[  g\left(  \overline{X}\right)  \right]   & =E\left(  \overline{X}%
^{2}\right) \\
& \\
& =V\left(  \overline{X}\right)  +E\left(  \overline{X}\right)  ^{2}\\
& \\
& =\frac{\theta^{2}}{n}+\theta^{2}=\theta^{2}\left(  1+\frac{1}{n}\right)  .
\end{align*}
\ 

\paragraph{Is there any reason to doubt the principle of seeking the unbiased
estimator with the minimum variance?}

\ \newline\emph{Finals question} 1979 $-$ IV $-$ 15, part (iv).\medskip
\ \newline$X_{1},X_{2},\ldots,X_{n}$ is a random sample from a Poisson
distribution with mean $\lambda$. Hence
\[
Y=\sum_{i=1}^{n}X_{i}\sim Poisson\left(  n\lambda\right)  .
\]
Part (iv) asks for an unbiased minimum variance estimator of $e^{-2n\lambda}
$.\medskip\ \newline Let this be $g(Y)$. Then
\[%
\begin{array}
[c]{cccrcl}
& E\left[  g(Y)\right]  & = &
%TCIMACRO{\dsum \limits_{y=0}^{\infty}}%
%BeginExpansion
{\displaystyle\sum\limits_{y=0}^{\infty}}
%EndExpansion
g(y)\dfrac{e^{-n\lambda}(n\lambda)^{y}}{y!} & = & e^{-2n\lambda}\\
\mathrm{or} &  &  &  &  & \\
&  &  &
%TCIMACRO{\dsum \limits_{y=0}^{\infty}}%
%BeginExpansion
{\displaystyle\sum\limits_{y=0}^{\infty}}
%EndExpansion
g(y)\dfrac{(n\lambda)^{y}}{y!} & = & e^{-n\lambda}.
\end{array}
\]
But
\[
e^{-x}=\sum\limits_{k=0}^{\infty}\left(  -1\right)  ^{k}\frac{x^{k}}{k!}%
\]
so that
\[
g(y)=\left(  -1\right)  ^{y},\quad y=0,1,\ldots
\]
\emph{i.e.}
\[
g(y)=\left\{
\begin{array}
[c]{rl}%
-1, & y\mathrm{\ odd,}\\
1, & y\mathrm{\ even.}%
\end{array}
\right.
\]
\textbf{THIS IS COMPLETE AND UTTER RUBBISH!}\ 

\paragraph{Can a biased estimator be \textquotedblleft
better\textquotedblright\ than an unbiased estimator?}

\ \newline Look at the distributions of the following two estimators of
$\theta$.%
%TCIMACRO{\FRAME{dhFU}{2.8245in}{2.2701in}{0pt}{\Qcb{\QTR{bf}{Figure 2.8}}}%
%{}{a5lhood_f8.eps}{\special{ language "Scientific Word";  type "GRAPHIC";
%maintain-aspect-ratio TRUE;  display "USEDEF";  valid_file "F";
%width 2.8245in;  height 2.2701in;  depth 0pt;  original-width 4.3033in;
%original-height 3.4515in;  cropleft "0";  croptop "1";  cropright "1";
%cropbottom "0";  filename 'a5lhood_f8.EPS';file-properties "XNPEU";}}}%
%BeginExpansion
\begin{center}
\includegraphics[
height=2.2701in,
width=2.8245in
]%
{a5lhood_f8.eps}%
\\
\textbf{Figure 2.8}%
\end{center}
%EndExpansion
\noindent Which do you prefer?\medskip\ \newline Even though $\widehat{\theta
}_{1}$ is biased,
\[
E\left[  \left(  \widehat{\theta}_{1}-\theta\right)  ^{2}\right]  <V\left(
\widehat{\theta}_{2}\right)  .
\]
$\widehat{\theta}_{1}$ has smaller \emph{mean squared error}.\ 

\paragraph{Are m.l.e.'s the answer?}

\ \newline The maximum likelihood principle is intuitively appealing and does
not involve worries about bias.

\begin{enumerate}
\item[(i)] Maximum likelihood estimators are asymptotically unbiased.\medskip
\ \newline

\item[(ii)] If $\widehat{\theta}$ is the m.l.e. of $\theta$, then $g\left(
\widehat{\theta}\right)  $is the m.l.e. of $g\left(  \theta\right)  $.
\end{enumerate}

\medskip\ \newline This is the \emph{invariance property} of maximum
likelihood estimators.\ 

\paragraph{Are there \emph{any} problems with m.l.e.'s?}

Consider $X_{1},X_{2},\ldots,X_{n}$ where $X_{i}\sim U(\theta-\frac{1}%
{2},\theta+\frac{1}{2}),\ 1\leq i\leq n,$ and note that $\theta-\frac{1}%
{2}\leq x_{(1)}<x_{(n)}\leq\theta+\frac{1}{2}.$\medskip\ \newline The
likelihood function is
\[
L(\theta;\mathbf{x})=\left\{
\begin{array}
[c]{ll}%
1, & x_{(n)}-\frac{1}{2}\leq\theta\leq x_{(1)}+\frac{1}{2},\\
0, & \mathrm{otherwise.}%
\end{array}
\right.
\]
\medskip\ \newline Clearly
\[
\max_{\theta}\left\{  L(\theta;\mathbf{x})\right\}  =1\quad\forall\theta
\in\left[  x_{(n)}-\frac{1}{2},x_{(1)}+\frac{1}{2}\right]
\]
\emph{i.e.} the likelihood function is maximised for \emph{every} statistic
$\widehat{\theta}\left(  X_{1},X_{2},\ldots,X_{n}\right)  $ such that
$\widehat{\theta}(\mathbf{X})\in\left[  X_{(n)}-\frac{1}{2},X_{(1)}+\frac
{1}{2}\right]  .$\medskip\ \newline\textbf{Conclusion:} An m.l.e. may not be unique.

\subsection{More about likelihood}

\subsubsection{Invariance property of m.l.e.'s}

\noindent\textbf{Lemma 2.4}\quad If $\widehat{\theta}$ is an m.l.e. of
$\theta$ and if $g$ is a function, then $g\left(  \widehat{\theta}\right)  $
is an m.l.e. of $g(\theta)$.\newline$\square$\vspace{20pt} \newline%
\textbf{Proof}\medskip\ \quad If $g$ is one-to-one, then
\[
L(\theta)=L\left(  g^{-1}\left(  g(\theta)\right)  \right)
\]
are both maximised by $\widehat{\theta}$, so
\[%
\begin{array}
[c]{rrl}
& \widehat{\theta}= & g^{-1}\left(  \widehat{g(\theta)}\right) \\
\mathrm{or} &  & \\
& g\left(  \widehat{\theta}\right)  = & \widehat{g(\theta)}.
\end{array}
\]
If $g$ is many-to-one, then $\widehat{\theta}$ which maximises $L(\theta)$
still corresponds to $g\left(  \widehat{\theta}\right)  $, so $g\left(
\widehat{\theta}\right)  $ still corresponds to the maximum of $L(\theta
)$\newline$\blacksquare$\bigskip

\noindent\textbf{Example\medskip\ 2.10\newline}Suppose $X_{1},X_{2}%
,\ldots,X_{n}$ is a random sample from a Bernoulli distribution $B(1,\theta)$.
Consider m.l.e.'s of the mean, $\theta$, and variance, $\theta(1-\theta
)$.\medskip\ \newline Note, by the way, that $\theta(1-\theta)$ is not a 1-1
function of $\theta$.\medskip\ \newline The log-likelihood is
\[%
\begin{array}
[c]{rrl}
& l(\theta)= & \sum_{i}x_{i}\log\theta+\left(  n-\sum_{i}x_{i}\right)
\log(1-\theta)\\
\mathrm{and} &  & \\
& \dfrac{dl(\theta)}{d\theta}= & \left.  \sum_{i}x_{i}\right/  \theta-\left.
\left(  n-\sum_{i}x_{i}\right)  \right/  (1-\theta)
\end{array}
\]
so it is easily shown that the m.l.e. of $\theta$ is $\widehat{\theta
}=\overline{X\mathrm{.}}$\medskip\ \newline Putting $\nu=\theta(1-\theta)$,
\[
\frac{dl(\nu)}{d\nu}=\frac{dl\left(  \nu(\theta)\right)  }{d\theta}\cdot
\frac{d\theta}{d\nu}%
\]
so it is easily seen that, since $\dfrac{d\theta}{d\nu}$ is not, in general,
equal to zero,
\[
\widehat{\nu}=\nu\left(  \widehat{\theta}\right)  =\overline{X}\left(
1-\overline{X}\right)  .
\]
$\blacksquare$

\subsubsection{Relative likelihood}

If $\sup\limits_{\theta}L(\theta)<\infty$, the \emph{relative likelihood} is
\[
RL(\theta)=\frac{L(\theta)}{\sup\limits_{\theta}L(\theta)};\quad0\leq
RL(\theta)\leq1.
\]
Relative likelihood is invariant to known 1-1 transformations of $x$, for if
$y$ is a 1-1 function of $x$,
\[
f_{Y}(y;\theta)=f_{X}\left(  x(y);\theta\right)  \left|  \frac{dx}{dy}\right|
.
\]
$\left|  \dfrac{dx}{dy}\right|  $ is independent of $\theta$, so
\[
RL_{X}(\theta)=RL_{Y}(\theta).
\]


\subsubsection{Likelihood summaries}

\noindent Realistic statistical problems often have many parameters. These
cause problems because it can be hard to visualise $L(\theta)$, and it becomes
necessary to use summaries.\medskip\ \newline\textbf{Key idea}\smallskip
\ \newline In large samples, log-likelihoods are often approximately quadratic
near the maximum.\vspace{20pt} \ 

\noindent\textbf{Example 2.11}\medskip\ \newline Suppose $X_{1},X_{2}%
,\ldots,X_{n}$ is a random sample from an exponential distribution with
parameter $\lambda$.\newline\emph{i.e.}
\[
f_{X}(x)=\lambda e^{-\lambda x},\quad x\geq0.
\]
Then
\[%
\begin{array}
[c]{rlrrl}%
l(\lambda)= & n\log\lambda-\lambda\sum_{i}x_{i}, &  & \dfrac{dl(\lambda
)}{d\lambda}= & \left.  n\right/  \lambda-\sum_{i}x_{i},\\
&  &  &  & \\
\dfrac{d^{2}l(\lambda)}{d\lambda^{2}}= & -\left.  n\right/  \lambda^{2}, &  &
\dfrac{d^{3}l(\lambda)}{d\lambda^{3}}= & \left.  2n\right/  \lambda^{3}.
\end{array}
\]
The log-likelihood has a maximum at $\widehat{\lambda}=n\left/  \sum_{i}%
x_{i}\right.  $, so
\[%
\begin{array}
[c]{rll}%
RL(\lambda)= & \left(  \dfrac{\lambda}{\widehat{\lambda}}\right)
^{n}e^{n-\lambda\sum_{i}x_{i}} & \\
&  & \\
= & \left(  \dfrac{\lambda}{\widehat{\lambda}}e^{1-\left.  \lambda\right/
\widehat{\lambda}}\right)  ^{n}, & \lambda>0.\\
&  & \\
\rightarrow & 1\quad\mathrm{{as}\quad}\lambda\rightarrow\widehat{\lambda
}\mathrm{.} &
\end{array}
\]
\ 

\noindent Now, what does the likelihood look like in the neighbourhood of
$\widehat{\lambda}$, as $n\rightarrow\infty$?
\begin{align*}
\log RL(\lambda)  & =l(\lambda)-l(\widehat{\lambda})\\
& =l(\widehat{\lambda})+l^{\prime}(\widehat{\lambda})\left(  \lambda
-\widehat{\lambda}\right)  +\frac{1}{2}l^{\prime\prime}(\widehat{\lambda
})\left(  \lambda-\widehat{\lambda}\right)  ^{2}-l(\widehat{\lambda})\\
& +O(\lambda-\widehat{\lambda})^{3}%
\end{align*}
using Taylor series$.$\medskip\ \newline Now $l^{\prime}(\widehat{\lambda})=0$
and $l^{\prime\prime}(\widehat{\lambda})=-n\left/  \widehat{\lambda}%
^{2}\right.  $, so
\[
\log RL(\lambda)\simeq-\frac{n\left(  \lambda-\widehat{\lambda}\right)  ^{2}%
}{2\widehat{\lambda}^{2}}\rightarrow-\infty\quad\mathrm{as\quad}%
n\rightarrow\infty
\]
unless $\lambda=\widehat{\lambda}.$\medskip\ \newline Thus, as $n\rightarrow
\infty$,
\[
RL(\lambda)\rightarrow\left\{
\begin{array}
[c]{ll}%
1, & \lambda=\widehat{\lambda},\\
0, & \mathrm{otherwise.}%
\end{array}
\right.
\]
\vspace{20pt} \ 

\noindent\textbf{Conclusion}\medskip\ \newline Likelihood becomes more
concentrated about the maximum as $n\rightarrow\infty$, and values far from
the maximum become less and less plausible.\newline$\blacksquare$\vspace{20pt}
\newline\textbf{In general}\medskip\ \newline We call the value $\widehat
{\theta}$ which maximises $L(\theta)$ or, equivalently, $l(\theta)=\log
L(\theta)$ the \emph{maximum likelihood estimate}, and
\[
J(\theta)=-\frac{\partial^{2}l(\theta)}{\partial\theta^{2}}%
\]
is called the \emph{observed information}.\medskip\ \newline Usually
$J(\theta)>0$ and $J(\widehat{\theta})$ measures the concentration of
$l(\theta)$ at $\widehat{\theta}$. Close to $\widehat{\theta}$, we summarise
\[
l(\theta)\simeq l(\widehat{\theta})-\frac{1}{2}\left(  \theta-\widehat{\theta
}\right)  ^{2}J(\widehat{\theta}).
\]


\subsubsection{Information}

\noindent In a model with log-likelihood $l(\theta)$, the \emph{observed
information} is
\[
J(\theta)=-\frac{\partial^{2}l(\theta)}{\partial\theta^{2}}.
\]
When observations are independent, $L(\theta)$ is a product of densities so
\[
l(\theta)=\sum_{i}\log f\left(  x_{i};\theta\right)
\]
and
\[
J(\theta)=-\sum_{i}\frac{\partial^{2}}{\partial\theta^{2}}\log f\left(
x_{i};\theta\right)  .
\]
\newline\newline\ \newline Since
\[
l(\theta)\simeq l(\widehat{\theta})-\frac{1}{2}\left(  \theta-\widehat{\theta
}\right)  ^{2}J(\widehat{\theta}),
\]
for $\theta$ near to $\widehat{\theta}$, we see that large $J(\widehat{\theta
})$ implies that $l(\theta)$ is more concentrated about $\widehat{\theta}%
$.\medskip\ \newline This means that the data are less ambiguous about
possible values of $\theta$, \emph{i.e.} we have more information about
$\theta$.\ 

\subsubsection{Expected information}

\paragraph{Univariate distributions}

\ \newline Before an experiment is conducted, we have no data so that we
cannot evaluate $J(\theta)$.\medskip\ \newline But we can find its expected
value
\[
I(\theta)=E\left(  -\frac{\partial^{2}l(\theta)}{\partial\theta^{2}}\right)  .
\]
This is called the \emph{expected information} or \emph{Fisher's
information}.\medskip\ \newline If the observations are a random sample, then
the whole sample expected information is
\[
I(\theta)=ni(\theta)
\]
where
\[
i(\theta)=E\left(  -\frac{\partial^{2}}{\partial\theta^{2}}\log f\left(
X_{i};\theta\right)  \right)  ,
\]
the single observation Fisher information.\bigskip%
%TCIMACRO{\FRAME{dhFU}{2.7838in}{3.4186in}{0pt}{\Qcb{Sir Ronald Aylmer Fisher
%(1890 - 1962)}}{}{fisher.gif}{\special{ language "Scientific Word";
%type "GRAPHIC";  maintain-aspect-ratio TRUE;  display "USEDEF";
%valid_file "F";  width 2.7838in;  height 3.4186in;  depth 0pt;
%original-width 7.875in;  original-height 9.6876in;  cropleft "0";
%croptop "1";  cropright "1";  cropbottom "0";
%filename 'Fisher.gif';file-properties "XNPEU";}}}%
%BeginExpansion
\begin{center}
\includegraphics[
natheight=9.687600in,
natwidth=7.875000in,
height=3.4186in,
width=2.7838in
]%
{Fisher.gif}%
\\
Sir Ronald Aylmer Fisher (1890 - 1962)
\end{center}
%EndExpansion


\noindent\textbf{Example 2.12}\medskip\ \newline Suppose $X_{1},X_{2}%
,\ldots,X_{n}$ is a random sample from a Poisson distribution with parameter
$\theta.$%
\[
L(\theta)=\prod\limits_{i=1}^{n}\frac{\theta^{x_{i}}e^{-\theta}}{x_{i}!},
\]
giving
\[
l(\theta)=\log L(\theta)=\sum_{i}x_{i}\log\theta-n\theta-\sum_{i}\log x_{i}!
\]
Thus
\[
J(\theta)=-\frac{\partial^{2}l(\theta)}{\partial\theta^{2}}=\left.
\sum\nolimits_{i}x_{i}\right/  \theta^{2}.
\]
To find $I(\theta)$, we need $E\left(  X_{i}\right)  =\theta$ and
\[
I(\theta)=\frac{1}{\theta^{2}}\sum\limits_{i}E\left(  X_{i}\right)  =\frac
{n}{\theta}.
\]
$\blacksquare$

\paragraph{Multivariate distributions}

\ \newline If $\mathbf{\theta}$ is a $(p\times1)$ vector of parameters, then
$\mathbf{I(\theta)}$ and $\mathbf{J(\theta)}$ are $(p\times p)$ matrices.
\[
\left\{  \mathbf{J}(\mathbf{\theta})\right\}  _{rs}=-\frac{\partial
^{2}l(\mathbf{\theta})}{\partial\theta_{r}\partial\theta_{s}}\quad
\mathrm{{and}\quad}\left\{  \mathbf{I}(\mathbf{\theta})\right\}
_{rs}=E\left(  -\frac{\partial^{2}l(\mathbf{\theta})}{\partial\theta
_{r}\partial\theta_{s}}\right)  \mathrm{.}%
\]
These matrices are obviously symmetric.\medskip\ \newline We can also write
the above as
\[
\mathbf{J(\theta)}=-\frac{\partial^{2}l(\mathbf{\theta})}{\partial
\mathbf{\theta}\partial\mathbf{\theta}^{T}}\mathrm{\quad{and}\quad
}\mathbf{I(\theta)}=E\left(  -\frac{\partial^{2}l(\mathbf{\theta})}%
{\partial\mathbf{\theta}\partial\mathbf{\theta}^{T}}\right)  \mathrm{.}%
\]


\noindent\textbf{Example 2.13}\medskip\ \newline$X_{1},X_{2},\ldots,X_{n}$ is
a random sample from a normal distribution with parameters $\mu$ and
$\sigma^{2}$. We have already seen that
\[
L\left(  \mu,\sigma^{2}\right)  =\left(  2\pi\sigma^{2}\right)  ^{-\left.
n\right/  2}\exp\left[  -\frac{1}{2\sigma^{2}}\sum\nolimits_{i}\left(
x_{i}-\mu\right)  ^{2}\right]  ,
\]
so
\[
l\left(  \mu,\sigma^{2}\right)  =-\frac{n}{2}\log2\pi-\frac{n}{2}\log
\sigma^{2}-\frac{1}{2\sigma^{2}}\sum\nolimits_{i}\left(  x_{i}-\mu\right)
^{2}.
\]
and
\[%
\begin{array}
[c]{rcl}%
\dfrac{\partial l}{\partial\mu} & = & \dfrac{1}{\sigma^{2}}\sum\nolimits_{i}%
\left(  x_{i}-\mu\right)  ,\\
&  & \\
\dfrac{\partial l}{\partial\sigma^{2}} & = & -\dfrac{n}{2\sigma^{2}}+\dfrac
{1}{2\sigma^{4}}\sum\nolimits_{i}\left(  x_{i}-\mu\right)  ^{2},\\
&  & \\
\dfrac{\partial^{2}l}{\partial\mu^{2}} & = & -\dfrac{n}{\sigma^{2}},\\
&  & \\
\dfrac{\partial^{2}l}{\partial\mu\partial\sigma^{2}} & = & -\dfrac{1}%
{\sigma^{4}}\sum\nolimits_{i}\left(  x_{i}-\mu\right)  .\\
&  & \\
\dfrac{\partial^{2}l}{\partial\left(  \sigma^{2}\right)  ^{2}} & = & \dfrac
{n}{2\sigma^{4}}-\dfrac{1}{\sigma^{6}}\sum\nolimits_{i}\left(  x_{i}%
-\mu\right)  ^{2}.
\end{array}
\]
\
\[
\mathbf{J}(\mu,\sigma^{2})=\left(
\begin{array}
[c]{cc}%
\dfrac{n}{\sigma^{2}} & \dfrac{1}{\sigma^{4}}\sum\nolimits_{i}\left(
x_{i}-\mu\right) \\
\dfrac{1}{\sigma^{4}}\sum\nolimits_{i}\left(  x_{i}-\mu\right)  & \dfrac
{1}{\sigma^{6}}\sum\nolimits_{i}\left(  x_{i}-\mu\right)  ^{2}-\dfrac
{n}{2\sigma^{4}}%
\end{array}
\right)  \mathrm{.}%
\]
To find $\mathbf{I}(\mu,\sigma^{2})$, use
\begin{align*}
E\left(  X_{i}\right)   & =\mu,\\
V\left(  X_{i}\right)   & =E\left[  \left(  X_{i}-\mu\right)  ^{2}\right]
=\sigma^{2},
\end{align*}
$\ $ so that
\[
\mathbf{I}(\mu,\sigma^{2})=E\left(  \mathbf{J}(\mu,\sigma^{2})\right)
=\left(
\begin{array}
[c]{cc}%
\dfrac{n}{\sigma^{2}} & 0\\
0 & \dfrac{n}{2\sigma^{4}}%
\end{array}
\right)  \mathrm{.}%
\]


\newpage\noindent\textbf{Example 2.14}\quad\emph{Censored exponential
data\medskip}\ \newline Lifetimes of $n$ components, safety devices, etc. are
observed for a time $c$, when $r$ have failed and $(n-r)$ are still
OK.\medskip\ \newline We have two kinds of observation:

\begin{enumerate}
\item[(i)] Exact failure times $x_{i}$ observed if $x_{i}\leq c$, so that
\[
f(x;\lambda)=\lambda e^{-\lambda x},\quad x\geq0;
\]


\item[(ii)] $x_{i}$ unobserved if $x_{i}>c$,
\[
P\left(  X>c\right)  =e^{-\lambda c}.
\]

\end{enumerate}

Data are therefore $x_{1},\ldots,x_{r},{\underbrace{c,\ldots,c}%
_{n-r\mathrm{\ times}}}$\medskip\ \newline The $(n-r)$ components, safety
devices, etc. which have not failed are said to be \emph{censored}%
.\bigskip\newline The likelihood is
\begin{align*}
L(\lambda)  & =\prod\limits_{i=1}^{r}\lambda e^{-\lambda x_{i}}\prod
\limits_{i=r+1}^{n}e^{-\lambda c}\\
& =\lambda^{r}\exp\left[  -\lambda\left(  \sum_{i=1}^{r}x_{i}+(n-r)c\right)
\right]  .
\end{align*}
\
\[%
\begin{array}
[c]{rcl}%
l(\lambda) & = & r\log\lambda-\lambda\left(  \sum_{i=1}^{r}x_{i}+(n-r)c\right)
\\
&  & \\
l^{\prime}(\lambda) & = & \left.  r\right/  \lambda-\left(  \sum_{i=1}%
^{r}x_{i}+(n-r)c\right) \\
&  & \\
l^{\prime\prime}(\lambda) & = & -r\left/  \lambda^{2}\right.  .
\end{array}
\]
Thus $J(\lambda)=r\left/  \lambda^{2}\right.  >0$ if $r>0$ so we must observe
\emph{at least one} exact failure time.
\[
I(\lambda)=E\left(  r\left/  \lambda^{2}\right.  \right)  =\frac{1}%
{\lambda^{2}}E\left(  \#X_{i}\mathrm{{\ observed~exactly}.}\right)
\]
Now $P\left(  X_{i}\mathrm{\ observed~exactly}\right)  =P\left(  X_{i}\leq
c\right)  =1-e^{-\lambda c}$, so
\[
I_{c}(\lambda)=\frac{n\left(  1-e^{-\lambda c}\right)  }{\lambda^{2}}.
\]
\medskip\ \newline No censoring if $c\rightarrow\infty$, giving
\[
I_{\infty}(\lambda)=\frac{n}{\lambda^{2}}>I_{c}(\lambda)
\]
as one might expect.\medskip\ \newline The asymptotic efficiency when there is
censoring at $c$ relative to no censoring is
\[
I_{c}(\lambda)\left/  I_{\infty}(\lambda)\right.  =1-e^{-\lambda c}.
\]
$\blacksquare\bigskip$

\noindent\textbf{Revision Example 2.15}\quad\emph{Events in a Poisson
process}\medskip\ \newline Events are observed for period $\left(  0,T\right)
$.\medskip\ \newline$n$ events occur at times $0<t_{1}<t_{2}<\ldots<t_{n}%
<T$\medskip\ \newline Two observers $A$ and $B$. $A$ records exact times, $B$
uses an automatic counter and goes to the pub (\emph{i.e.} $B$ merely records
how many events there are).\medskip\ \newline$A$ knows exact times, and times
between events are independent and exponentially distributed, so
\begin{align*}
L_{A}(\lambda)  & =\lambda e^{-\lambda t_{1}}\times\lambda e^{-\lambda\left(
t_{2}-t_{1}\right)  }\times\cdots\times\lambda e^{-\lambda\left(
t_{n}-t_{n-1}\right)  }\times e^{-\lambda\left(  T-t_{n}\right)  }\\
& =\lambda^{n}e^{-\lambda T}.
\end{align*}
$B$ merely observes the event $\left[  N=n\right]  $, where $N\sim Poi(\lambda
T)$, so
\[
L_{B}(\lambda)=\frac{\left(  \lambda T\right)  ^{n}e^{-\lambda T}}{n!}.
\]
\ 

\noindent Log-likelihoods are
\begin{align*}
l_{A}(\lambda)  & =n\log\lambda-\lambda T,\\
l_{B}(\lambda)  & =n\log\lambda+n\log T-\lambda T-\log n!
\end{align*}
and
\[
J_{A}(\lambda)=J_{B}(\lambda)=n\left/  \lambda^{2}\right.  .
\]
\medskip\ \newline$E(N)=\lambda T$, so $I_{A}(\lambda)=I_{B}(\lambda)=T\left/
\lambda\right.  $, and both observers get the same information. As usual, the
one who went to the pub did the right thing.\newline$\blacksquare$

\subsubsection{Maximum likelihood estimates}

The maximum likelihood estimate $\widehat{\theta}$ of $\theta$ maximises
$L(\theta)$ and often (but not always) satisfies the \emph{likelihood
equation}
\[
\frac{\partial l}{\partial\theta}\left(  \widehat{\theta}\right)  =0,
\]
with
\[
J(\widehat{\theta})=-\frac{\partial^{2}l}{\partial\theta^{2}}\left(
\widehat{\theta}\right)  >0
\]
for a maximum.\newline\newline\ \newline In the vector case, $\widehat
{\mathbf{\theta}}$ solves simultaneously
\[
\frac{\partial l}{\partial\theta_{r}}\left(  \widehat{\mathbf{\theta}}\right)
=0,\quad r=1,\ldots,p,
\]
with $\mathbf{J}(\widehat{\mathbf{\theta}})$ \textit{positive definite} \ This
implies%
\[
\det\mathbf{J}(\widehat{\mathbf{\theta}})>0
\]


\noindent If the likelihood equation has many solutions, we find them all and
check $L(\theta)$ for each. Usually, the equation has to be solved
numerically. One way is by Newton-Raphson.\medskip\ \newline Suppose we have a
starting value $\theta_{0}$. Then
\[
0=\frac{\partial l}{\partial\theta}\left(  \widehat{\theta}\right)
\simeq\frac{\partial l}{\partial\theta}\left(  \theta_{0}\right)
+\frac{\partial^{2}l}{\partial\theta^{2}}\left(  \theta_{0}\right)  \left(
\widehat{\theta}-\theta_{0}\right)
\]
which may be re-arranged to
\[
\widehat{\theta}=\theta_{0}+\frac{U(\theta_{0})}{J(\theta_{0})},
\]
where
\begin{align*}
U(\theta)  & =\frac{\partial l}{\partial\theta}\mathrm{\ is~the~}%
\emph{score~function}\mathrm{,}\\
J(\theta)  & =-\frac{\partial^{2}l}{\partial\theta^{2}}\mathrm{\ is~the~}%
\emph{observed~information}\mathrm{.}%
\end{align*}
\medskip\ \newline Now we iterate using $\theta_{0}$ as a starting value and
\[
\theta_{n+1}=\theta_{n}+\frac{U(\theta_{n})}{J(\theta_{n})}.
\]
\newline\noindent\textbf{Example 2.16}\quad\emph{Extreme value (Gumbel)
distribution}\medskip\ \newline This distribution is used to model such things
as annual maximum temperature. Data due to Bliss on numbers of beetles killed
by exposure to carbon disulphide are fitted by this model. The c.d.f. is
\[
F(x)=\exp\left(  -e^{-(x-\eta)}\right)  ,\quad x\in\mathbb{R},\ \eta
\in\mathbb{R},
\]
and the density is
\[
f(x)=\exp\left[  -(x-\eta)-e^{-(x-\eta)}\right]  ,\quad x\in\mathbb{R}%
,\ \eta\in\mathbb{R}.
\]
The sample log-likelihood is
\[
l(\eta)=-\sum_{i}(x_{i}-\eta)-\sum_{i}e^{-(x_{i}-\eta)},
\]
so that
\begin{align*}
U(\eta)  & =n-\sum_{i}e^{-(x_{i}-\eta)},\\
J(\eta)  & =\sum_{i}e^{-(x_{i}-\eta)}.
\end{align*}
Starting at $\eta_{0}=\overline{x}$, iterate using
\[
\eta_{n+1}=\eta_{n}+\frac{n-\sum_{i}e^{-(x_{i}-\eta_{n})}}{\sum_{i}%
e^{-(x_{i}-\eta_{n})}.}%
\]
$\blacksquare$

\paragraph{Fisher scoring}

\ \newline This simply involves replacing $J(\theta)$ with $I(\theta).$
\newline\newline

\noindent\textbf{Example 2.17}\quad\emph{Extreme value distribution\medskip
}\ \newline We need
\begin{align*}
I(\eta)  & =E\left[  J(\eta)\right]  =\sum_{i}E\left[  e^{-(X_{i}-\eta
)}\right] \\
& =n\int_{-\infty}^{\infty}e^{-(x-\eta)}\exp\left[  -(x-\eta)-e^{-(x-\eta
)}\right]  dx.
\end{align*}
Put $u=e^{-(x-\eta)}$ and the integral becomes
\[
I(\eta)=n\int_{0}^{\infty}ue^{-u}du=n,
\]
so Fisher scoring gives the iteration
\[
\eta_{n+1}=\eta_{n}+1-\frac{1}{n}\sum_{i}e^{-(x_{i}-\eta_{n})}.
\]
$\blacksquare$

\subsubsection{Sufficient statistics}

\noindent You have already seen a likelihood which cannot be summarised by a
quadratic.\medskip\ \newline\textbf{Example 2.18}
\[
f\left(  x_{i};\theta\right)  =\theta^{-1},\quad0<x_{i}<\theta,
\]
so
\[
L(\theta)=\theta^{-n},\quad0<\max\left\{  x_{i}\right\}  <\theta.
\]%
%TCIMACRO{\FRAME{dhFU}{4.0179in}{2.4042in}{0pt}{\Qcb{\QTR{bf}{Figure 2.9}}}%
%{}{a5lhood_f9.eps}{\special{ language "Scientific Word";  type "GRAPHIC";
%maintain-aspect-ratio TRUE;  display "USEDEF";  valid_file "F";
%width 4.0179in;  height 2.4042in;  depth 0pt;  original-width 4.6942in;
%original-height 2.7968in;  cropleft "0";  croptop "1";  cropright "1";
%cropbottom "0";  filename 'a5lhood_f9.EPS';file-properties "XNPEU";}}}%
%BeginExpansion
\begin{center}
\includegraphics[
height=2.4042in,
width=4.0179in
]%
{a5lhood_f9.eps}%
\\
\textbf{Figure 2.9}%
\end{center}
%EndExpansion
~\medskip

\noindent Clearly a quadratic approximation is useless here.\medskip

\noindent\textbf{Definition\medskip\ 2.9}\quad\emph{Sufficient statistic\ }%
\newline If $S=s\left(  \mathbf{X}\right)  $ is such that the conditional
density $f_{\mathbf{X}|S}\left(  \mathbf{x}|s;\theta\right)  $ is independent
of $\theta$, then $S$ is a \emph{sufficient statistic} for $\theta$%
.\medskip\newline The important question is:

\begin{quotation}
\noindent Does $s(\mathbf{x})$ reduce the dimensionality of the
problem?\medskip$\square$
\end{quotation}

\noindent As we will see, the definition is equivalent to saying that the
likelihood $L(\theta)$ only depends upon data $\mathbf{x}$ through $s\left(
\mathbf{x}\right)  $. So maximum likelihood inference also only depends on
$s\left(  \mathbf{x}\right)  .$\medskip\ \newline\newpage

\noindent\textbf{Example 2.19}\medskip\ \newline Suppose $X_{1},X_{2}\sim
B(n,\theta)$ and consider
\begin{align*}
& P\left(  X_{1}=x|X_{1}+X_{2}=r\right) \\
& \\
& =\frac{P\left(  X_{1}=x,X_{1}+X_{2}=r\right)  }{P\left(  X_{1}%
+X_{2}=r\right)  }\\
& \\
& =\frac{P\left(  X_{1}=x,X_{2}=r-x\right)  }{P\left(  X_{1}+X_{2}=r\right)
}\\
& \\
& =\frac{{\dbinom{n}{x}}\theta^{x}\left(  1-\theta\right)  ^{n-x}{\dbinom
{n}{{r-x}}}\theta^{r-x}\left(  1-\theta\right)  ^{n-r+x}}{{\dbinom{{2n}}{r}%
}\theta^{r}\left(  1-\theta\right)  ^{2n-r}}\\
& \\
& =\frac{{\dbinom{n}{x}}{\dbinom{n}{{r-x}}}}{{\dbinom{{2n}}{r}}}.
\end{align*}
This does not contain $\theta,$ so that $X_{1}+X_{2}$ is a sufficient
statistic for $\theta$.\newline$\blacksquare\bigskip$\newline\ \noindent

\noindent\textbf{Theorem\medskip\ 2.1}\quad\emph{Factorization Theorem}%
\ \newline$s(\mathbf{X})$ is a \emph{sufficient statistic} for $\theta$ if and
only if there exist functions $g$ and $h$ such that
\[
f(\mathbf{x};\theta)=g\left(  s(\mathbf{x});\theta\right)  h(\mathbf{x})
\]
for all $\mathbf{x}\in\mathbb{R}^{n},\ \theta\in\Theta$.\newline$\square
$\bigskip\newline\noindent\textbf{Proof for discrete random variables}

\begin{enumerate}
\item[(i)] Let $s(\mathbf{x})=a$ and suppose the factorization condition to be
satisfied, so that $f(\mathbf{x};\theta)=g\left(  s(\mathbf{x});\theta\right)
h(\mathbf{x})$.\medskip\ \newline Then
\[
P\left(  s(\mathbf{X})=a\right)  =\sum_{\mathbf{y}\in s^{-1}(a)}%
p(\mathbf{y})=g(a;\theta)\sum_{\mathbf{y}\in s^{-1}(a)}h(\mathbf{y}).
\]
Hence
\[
P\left(  \mathbf{X}=\mathbf{x}\mid s(\mathbf{X})=a\right)  =\frac
{h(\mathbf{x})}{\sum\limits_{\mathbf{y}\in s^{-1}(a)}h(\mathbf{y})}%
\]
and this does not depend upon $\theta$.\ 

\item[(ii)] Let $s(\mathbf{X})$ be a sufficient statistic for $\theta$. Then
\[
P(\mathbf{X}=\mathbf{x})=P\left(  \mathbf{X}=\mathbf{x}\mid s(\mathbf{X}%
)=a\right)  P\left(  s(\mathbf{X})=a\right)  .
\]
But sufficiency $\Rightarrow P\left(  \mathbf{X}=\mathbf{x}\mid s(\mathbf{X}%
)=a\right)  $ does not depend upon $\theta,$ so writing \medskip
\ \newline$P\left(  s(\mathbf{X})=a\right)  =g(a;\theta)$ and $P\left(
\mathbf{X}=\mathbf{x}\mid s(\mathbf{X})=a\right)  =h(\mathbf{x})$ gives the result.
\end{enumerate}

\noindent The proof in the continuous case requires measure theory and is
beyond the scope of this course.\newline$\blacksquare$\bigskip\newline%
\noindent\textbf{Example 2.20\medskip}\newline$X_{1},X_{2},\ldots,X_{n}\sim
U(0,\theta)$, so that
\[
L(\theta)=\theta^{-n},\quad0<x_{1},\ldots,x_{n}<\theta.
\]%
\[
L(\theta)=\theta^{-n},\quad\theta>x_{(n)}.
\]
This factorizes with $s(\mathbf{x})=x_{(n)}$ and $h\left(  \mathbf{x}\right)
=1$, so that $X_{(n)}$ is a sufficient statistic for $\theta.\blacksquare
$\bigskip\newline\noindent\textbf{Example\medskip\ 2.21\ }\newline Suppose
$X_{1},X_{2},\ldots,X_{n}$ is a random sample from a Bernoulli distribution.
Then
\[
p(\mathbf{x};\theta)=\theta^{\sum_{i}x_{i}}(1-\theta)^{n-\sum_{i}x_{i}%
}\mathrm{.}%
\]
Trivially this factorizes with $s(\mathbf{x})=\sum_{i}x_{i}$ and $h\left(
\mathbf{x}\right)  =1$\newline$\blacksquare\mathrm{\bigskip}$\newline%
\noindent\textbf{Example\medskip\ 2.22}\ \newline Suppose $X_{1},X_{2}%
,\ldots,X_{n}$ is a random sample from a $N(\mu,\sigma^{2})$ distribution,
where $(\mu,\sigma^{2})^{T}$ is a vector of unknown parameters. Then
\begin{align*}
f(\mathbf{x};\mu,\sigma^{2})  & =\left(  2\pi\sigma^{2}\right)  ^{-\left.
n\right/  2}\exp\left[  -\frac{1}{2\sigma^{2}}\sum_{i}(x_{i}-\mu)^{2}\right]
\\
& \\
& =\left(  2\pi\sigma^{2}\right)  ^{-\left.  n\right/  2}\exp\left[  -\frac
{1}{2\sigma^{2}}\sum_{i}(x_{i}-\overline{x})^{2}+n(\overline{x}-\mu
)^{2}\right]  .
\end{align*}
Again this factorizes where $s(\mathbf{x})=\left(  \overline{x},\sum_{i}%
(x_{i}-\overline{x})^{2}\right)  ^{T}$, a vector valued function.\newline%
$\blacksquare$

\subsubsection{Large sample distribution of $\widehat{\theta}$}

From the data summary point of view, the m.l.e. $\widehat{\theta}$ and
$J\left(  \widehat{\theta}\right)  $have been thought of in terms of a
particular set of data. We now wish to think of $\widehat{\theta}$ in terms of
repeated sampling (\emph{i.e.} as a random variable).\newline\newline%
\ \newline\textbf{Main results}\medskip\ \newline In many situations and
subject to regularity conditions
\[
\widehat{\theta}\overset{D}{\rightarrow}N(\theta,I(\theta)^{-1}),
\]
We can apply the approach to obtaining confidence intervals developed in Mods,
for the CLT.\medskip\ \newline Then an approximate 95\% confidence interval
for $\theta$ is given by
\[
\widehat{\theta}\pm1.96I(\widehat{\theta})^{-1\left/  2\right.  }.
\]
[or $\widehat{\theta}\pm1.96J(\widehat{\theta})^{-1\left/  2\right.  }$,
regarded by many as better, but not in the books].\newline\newline\ \newline
In the multivariate case,
\[
\widehat{\mathbf{\theta}}\overset{D}{\rightarrow}N(\mathbf{\theta}%
,\mathbf{I}(\mathbf{\theta})^{-1})\mathrm{.}%
\]
~

\noindent\textbf{Example 2.23}\quad\emph{Exponential distribution}%
\medskip\ \newline For an exponential distribution with mean $\theta$,
\begin{align*}
L(\theta)  & =\theta^{-n}e^{-\sum_{i}x_{i}\left/  \theta\right.  },\quad
\theta>0,\\
l(\theta)  & =-n\log\theta-\sum_{i}x_{i}\left/  \theta\right.  ,
\end{align*}
so that
\[
U(\theta)=-\frac{n}{\theta}+\frac{\sum_{i}x_{i}}{\theta^{2}},\quad
J(\theta)=-\frac{n}{\theta^{2}}+\frac{2\sum_{i}x_{i}}{\theta^{3}}.
\]
Thus
\[
\widehat{\theta}=\overline{x},\quad J(\widehat{\theta})=\frac{n}{\overline
{x}^{2}}%
\]
and an approximate 95\% confidence interval is
\[
\overline{x}\pm1.96\overline{x}\left/  \sqrt{n}\right.
\]
$\blacksquare$\bigskip\ 

\noindent\textbf{Example 2.24}\quad\emph{Normal distribution\medskip
}\ \newline For a normal random sample,
\[
\widehat{\mu}=\overline{x},\quad\widehat{\sigma}^{2}=n^{-1}\sum_{i}\left(
x_{i}-\overline{x}\right)  ^{2},
\]%
\[
J\left(  \mu,\sigma^{2}\right)  =\left(
\begin{array}
[c]{cc}%
n\left/  \sigma^{2}\right.  & \sigma^{-4}\sum_{i}\left(  x_{i}-\mu\right) \\
\sigma^{-4}\sum_{i}\left(  x_{i}-\mu\right)  & \sigma^{-6}\sum_{i}\left(
x_{i}-\mu\right)  ^{2}-n\left/  2\sigma^{4}\right.
\end{array}
\right)  .
\]
Therefore
\[
I\left(  \widehat{\mu},\widehat{\sigma}^{2}\right)  =\left(
\begin{array}
[c]{cc}%
n\left/  \widehat{\sigma}^{2}\right.  & 0\\
0 & n\left/  2\widehat{\sigma}^{4}\right.
\end{array}
\right)  .
\]
An approximate 95\% confidence interval for $\mu$ is
\[
\overline{x}\pm1.96\widehat{\sigma}\left/  \sqrt{n}\right.  ,
\]
and for $\sigma^{2}$ is
\[
\widehat{\sigma}^{2}\pm1.96\widehat{\sigma}^{2}\sqrt{\frac{2}{n}}.
\]
Note that the estimators $\widehat{\mu}$ and $\widehat{\sigma}^{2}$ are
asymptotically uncorrelated.\newline$\blacksquare$\bigskip\newline%
\textbf{Lemma 2.5}\quad If $\widehat{\theta}$ is the maximum likelihood
estimator of a parameter $\theta$ based on a random sample, under suitable
regularity conditions%
\[
\widehat{\theta}\overset{D}{\rightarrow}N(\theta,I(\theta)^{-1}),
\]
where $I(\theta)$ is Fisher's information for the sample.\newline$\square
$\bigskip\ \newline\noindent\textbf{Sketch Proof\quad}Suppose $X_{1}%
,X_{2},\ldots,X_{n}$ is a random sample from a distribution with p.d.f.
$f(x;\theta)$. Then the log-likelihood, score and observed information are
\begin{align*}
l(\theta)  & =\sum_{i}\log f(x_{i};\theta),\\
U(\theta)  & =\sum_{i}\frac{\partial}{\partial\theta}\log f(x_{i};\theta),\\
J(\theta)  & =-\sum_{i}\frac{\partial^{2}}{\partial\theta^{2}}\log
f(x_{i};\theta).
\end{align*}
Let $U_{i}(\theta)$ be the random variable $U_{i}(\theta)=\dfrac{\partial
}{\partial\theta}\log f(X_{i};\theta)$, and, provided that conditions are such
that integration and differentiation are interchangeable,
\begin{align*}
E\left[  U_{i}(\theta)\right]   & =\int f(x;\theta)\frac{\partial}%
{\partial\theta}\log f(x;\theta)dx\\
& =\int\frac{\partial}{\partial\theta}f(x;\theta)dx\\
& =\frac{\partial}{\partial\theta}\int f(x;\theta)dx=\frac{\partial}%
{\partial\theta}1=0
\end{align*}
and
\begin{align*}
0  & =\frac{\partial}{\partial\theta}\int f(x;\theta)\frac{\partial}%
{\partial\theta}\log f(x;\theta)dx\\
& =\int f(x;\theta)\frac{\partial^{2}}{\partial\theta^{2}}\log f(x;\theta
)dx+\int\frac{\partial}{\partial\theta}f(x;\theta)\frac{\partial}%
{\partial\theta}\log f(x;\theta)dx\\
& =E\left[  \frac{\partial^{2}}{\partial\theta^{2}}\log f(X;\theta)\right]
+\int f(x;\theta)\left(  \frac{\partial}{\partial\theta}\log f(x;\theta
)\right)  ^{2}dx.
\end{align*}
So
\[
0=-i(\theta)+E\left[  U_{i}(\theta)^{2}\right]
\]
and, therefore, $V\left[  U_{i}(\theta)\right]  =i(\theta)$.\medskip\ \newline
It follows that $E\left[  U(\theta)\right]  =0$, $V\left[  U(\theta)\right]
=ni(\theta)=I(\theta)$, and the Central Limit Theorem shows that
\[
U(\theta)\overset{D}{\rightarrow}N\left(  0,I(\theta)\right)  .
\]
Now the m.l.e. is a solution of $U(\widehat{\theta})=0$, so that, Taylor
expanding about $\theta$,
\[
U(\theta)+U^{\prime}(\theta)(\widehat{\theta}-\theta)\simeq0
\]
or
\[
U(\theta)-J(\theta)(\widehat{\theta}-\theta)\simeq0.
\]
Re-arranging,
\[
\sqrt{I(\theta)}(\widehat{\theta}-\theta)\simeq U(\theta)\frac{\sqrt
{I(\theta)}}{J(\theta)}=\frac{U(\theta)}{\sqrt{I(\theta)}}\left/
\frac{J(\theta)}{I(\theta)}\right.  .
\]
From the CLT,
\[
\frac{U(\theta)}{\sqrt{I(\theta)}}\overset{D}{\rightarrow}N(0,1)
\]
and (from WLLN which you will meet in probability)
\[
\frac{J(\theta)}{I(\theta)}\overset{P}{\rightarrow}1.
\]
\ 

\noindent Slutsky's Theorem states that, if $S_{n}\overset{D}{\rightarrow}S$
and $U_{n}\overset{P}{\rightarrow}k$, where $k$ is a constant, then
\[
S_{n}+U_{n}\overset{D}{\rightarrow}S+k,\quad S_{n}U_{n}\overset{D}%
{\rightarrow}Sk.
\]
It therefore results in
\[
\sqrt{I(\theta)}(\widehat{\theta}-\theta)\overset{D}{\rightarrow}N(0,1)
\]
or
\[
\widehat{\theta}\overset{D}{\rightarrow}N(\theta,I(\theta)^{-1}).
\]
$\blacksquare$\bigskip

\noindent\textbf{Requirements of this proof}

\begin{enumerate}
\item[(i)] The true value of $\theta$ is interior to the parameter space.

\item[(ii)] Differentiation under the integral is valid, so that $E\left[
U(\theta)\right]  =0$ and $V\left[  U(\theta)\right]  =ni(\theta)$. This
allows a central limit theorem to apply to $U(\theta)$.

\item[(iii)] Taylor expansions are valid for the derivatives of the
log-likelihood, so that higher order terms may be neglected.

\item[(iv)] A weak law of large numbers applies to $J(\theta)$.
\end{enumerate}

\subsection{\noindent The $\delta$-method}

We often want the asymptotic distribution of some function of a random
variable. The basic method of doing this is the $\delta$\emph{-method}%
.\medskip\ \newline Let $X_{1},X_{2},\ldots,X_{n}$, be a random sample from a
distribution with mean $\mu$, variance $\sigma^{2}$.\medskip\medskip\ \newline
Then $E\left(  \overline{X}-\mu\right)  =0,\ V\left(  \overline{X}\right)
=\sigma^{2}/n$.\medskip\ \newline By Taylor's Theorem
\[%
\begin{array}
[c]{rcl}%
g\left(  \overline{X}\right)  & = & g\left(  \mu+(\overline{X}-\mu\right)  )\\
&  & \\
& = & g(\mu)+(\overline{X}-\mu)g^{\prime}(\mu)+\frac{1}{2}(\overline{X}%
-\mu)^{2}g^{\prime\prime}(\delta_{n}),\\
&  & \\
&  & \qquad\qquad\qquad\qquad\qquad\qquad0<\left\vert \delta_{n}%
-\mu\right\vert <\left\vert \overline{X}-\mu\right\vert ,
\end{array}
\]
and as $n$ becomes large
\[%
\begin{array}
[c]{rcl}%
E\left[  g\left(  \overline{X}\right)  \right]  & = & g(\mu)+O\left(
n^{-1}\right)  ,\\
&  & \\
V\left[  g\left(  \overline{X}\right)  \right]  & = & n^{-1}\sigma
^{2}g^{\prime}(\mu)^{2}+O\left(  n^{-\left.  3\right/  2}\right)  .
\end{array}
\]
\textbf{Example 2.25}\quad\emph{Exponential distribution}\medskip\ \ \newline
If
\[
f_{X}(x)=\lambda e^{-\lambda x},\quad x\geq0,
\]
then
\[
E(X)=\lambda^{-1},\ V(X)=\lambda^{-2},
\]
so that
\[
E(\overline{X})=\lambda^{-1},\ V(\overline{X})=\left.  1\right/  n\lambda^{2}.
\]
\medskip\ \newline Suppose we want $E(Y)$ and $V(Y)$, where $Y=\log
\overline{X}$.\medskip\ \newline Then
\[
g(x)=\log x,\ g^{\prime}(x)=\left.  1\right/  x
\]
and
\[
E(Y)\simeq-\log\lambda,\quad V(Y)\simeq\left.  \frac{1}{n\lambda^{2}}\left(
\frac{1}{x}\right)  ^{2}\right\vert _{x=\left.  1\right/  \lambda}=\frac{1}%
{n}.
\]
$\blacksquare$\bigskip\ 

\noindent\textbf{Example 2.26}\quad\emph{Variance stabilising transformations}%
\medskip\ \ \newline The aim of a variance stabilising transformation is to
find a transformation such that $g\left(  \overline{X}\right)  $ has a
variance which is approximately constant, \newline\emph{i.e.}
\[
\sigma^{2}g^{\prime}(\mu)^{2}\simeq c,\ \mathrm{a\ constant.}%
\]
Suppose we have a random sample $X_{1},X_{2},\ldots,X_{n}$ with mean $\mu$,
variance $V(\mu)$. Then
\[%
\begin{array}
[c]{crcl}
& g^{\prime}(\mu) & = & cV(\mu)^{-\left.  1\right/  2}\\
&  &  & \\
\Rightarrow & g(\mu) & = & \int^{\,\mu}V(u)^{-\left.  1\right/  2}du.
\end{array}
\]
\medskip\ \newline In practice, the transformation is usually applied to the
sample data directly to produce transformed data $g(X_{1}),g(X_{2}%
),\ldots,g(X_{n})$. \newline For a Poisson distribution, $E(X)=\mu,\ V(X)=\mu$
so that $g(\mu)=\mu^{\left.  1\right/  2}.$\newline Thus, if $X\sim
Poisson(\mu)$, then
\[
Y=\sqrt{X}%
\]
has a variance which is approximately constant.\medskip\ \newline For an
exponential distribution, $V(\mu)=\mu^{2}$, so
\[
g(\mu)=\log\mu.
\]
$\blacksquare$


\end{document}