Editing Maximum likelihood estimation (section)

== Examples ==
=== Discrete uniform distribution ===
{{Main|German tank problem}}
Consider a case where ''n'' tickets numbered from 1 to ''n'' are placed in a box and one is selected at random (''see [[Uniform distribution (discrete)|uniform distribution]]''); thus, the sample size is 1.  If ''n'' is unknown, then the maximum likelihood estimator <math>\widehat{n}</math> of ''n'' is the number ''m'' on the drawn ticket. (The likelihood is 0 for ''n''&nbsp;<&nbsp;''m'', {{frac|1|''n''}} for ''n''&nbsp;≥&nbsp;''m'', and this is greatest when ''n''&nbsp;=&nbsp;''m''. Note that the maximum likelihood estimate of ''n'' occurs at the lower extreme of possible values {''m'',&nbsp;''m''&nbsp;+&nbsp;1,&nbsp;...}, rather than somewhere in the "middle" of the range of possible values, which would result in less bias.) The [[expected value]] of the number ''m'' on the drawn ticket, and therefore the expected value of <math>\widehat{n}</math>, is (''n''&nbsp;+&nbsp;1)/2. As a result, with a sample size of 1, the maximum likelihood estimator for ''n'' will systematically underestimate ''n'' by (''n''&nbsp;−&nbsp;1)/2.

=== Discrete distribution, finite parameter space ===
Suppose one wishes to determine just how biased an [[unfair coin]] is.  Call the probability of tossing a '[[Obverse and reverse|head]]' ''p''. The goal then becomes to determine ''p''.

Suppose the coin is tossed 80 times: i.e. the sample might be something like ''x''<sub>1</sub>&nbsp;=&nbsp;H, ''x''<sub>2</sub>&nbsp;=&nbsp;T, ..., ''x''<sub>80</sub>&nbsp;=&nbsp;T, and the count of the number of [[Obverse and reverse|heads]] "H" is observed.

The probability of tossing [[Obverse and reverse|tails]] is 1&nbsp;−&nbsp;''p'' (so here ''p'' is ''θ'' above). Suppose the outcome is 49&nbsp;heads and 31&nbsp;[[Obverse and reverse|tails]], and suppose the coin was taken from a box containing three coins: one which gives heads with probability ''p''&nbsp;=&nbsp;{{frac|1|3}}, one which gives heads with probability ''p''&nbsp;=&nbsp;{{frac|1|2}} and another which gives heads with probability ''p''&nbsp;=&nbsp;{{frac|2|3}}. The coins have lost their labels, so which one it was is unknown. Using maximum likelihood estimation, the coin that has the largest likelihood can be found, given the data that were observed. By using the [[probability mass function]] of the [[binomial distribution]] with sample size equal to 80, number successes equal to 49 but for different values of ''p'' (the "probability of success"), the likelihood function (defined below) takes one of three values:

<math display="block">\begin{align}
\operatorname{\mathbb P}\bigl[\;\mathrm{H} = 49 \mid p=\tfrac{1}{3}\;\bigr] & = \binom{80}{49}(\tfrac{1}{3})^{49}(1-\tfrac{1}{3})^{31} \approx 0.000, \\[6pt]
\operatorname{\mathbb P}\bigl[\;\mathrm{H} = 49 \mid p=\tfrac{1}{2}\;\bigr] & = \binom{80}{49}(\tfrac{1}{2})^{49}(1-\tfrac{1}{2})^{31} \approx 0.012, \\[6pt]
\operatorname{\mathbb P}\bigl[\;\mathrm{H} = 49 \mid p=\tfrac{2}{3}\;\bigr] & = \binom{80}{49}(\tfrac{2}{3})^{49}(1-\tfrac{2}{3})^{31} \approx 0.054~.
\end{align}</math>

The likelihood is maximized when {{mvar|p}}&nbsp;=&nbsp;{{frac|2|3}}, and so this is the ''maximum likelihood estimate'' for&nbsp;{{mvar|p}}.

=== Discrete distribution, continuous parameter space ===
Now suppose that there was only one coin but its {{mvar|p}} could have been any value {{nowrap| 0 ≤ {{mvar|p}} ≤ 1 .}} The likelihood function to be maximised is
<math display="block">
L(p) = f_D(\mathrm{H} = 49 \mid p) = \binom{80}{49} p^{49}(1 - p)^{31}~,
</math>

and the maximisation is over all possible values {{nowrap|0 ≤ {{mvar|p}} ≤ 1 .}}

[[File:MLfunctionbinomial-en.svg|thumb|200px|Likelihood function for proportion value of a binomial process ({{mvar|n}}&nbsp;=&nbsp;10)]]

One way to maximize this function is by [[derivative|differentiating]] with respect to {{mvar|p}} and setting to zero:

<math display="block">\begin{align}
0 & = \frac{\partial}{\partial p} \left( \binom{80}{49} p^{49}(1-p)^{31} \right)~, \\[8pt]
0 & = 49 p^{48}(1-p)^{31} - 31 p^{49}(1-p)^{30} \\[8pt]
  & = p^{48}(1-p)^{30}\left[ 49 (1-p) - 31 p \right]  \\[8pt]
  & = p^{48}(1-p)^{30}\left[ 49 - 80 p \right]~.
\end{align}</math>

This is a product of three terms.  The first term is 0 when {{mvar|p}}&nbsp;=&nbsp;0.  The second is 0 when {{mvar|p}}&nbsp;=&nbsp;1.  The third is zero when {{mvar|p}}&nbsp;=&nbsp;{{frac|49|80}}. The solution that maximizes the likelihood is clearly {{mvar|p}}&nbsp;=&nbsp;{{frac|49|80}} (since {{mvar|p}}&nbsp;=&nbsp;0 and {{mvar|p}}&nbsp;=&nbsp;1 result in a likelihood of 0). Thus the ''maximum likelihood estimator'' for {{mvar|p}} is {{frac|49|80}}.

This result is easily generalized by substituting a letter such as {{mvar|s}} in the place of 49 to represent the observed number of 'successes' of our [[Bernoulli trial]]s, and a letter such as {{mvar|n}} in the place of 80 to represent the number of Bernoulli trials. Exactly the same calculation yields {{frac|{{mvar|s}}|{{mvar|n}}}} which is the maximum likelihood estimator for any sequence of {{mvar|n}} Bernoulli trials resulting in {{mvar|s}} 'successes'.

=== Continuous distribution, continuous parameter space ===
For the [[normal distribution]] <math>\mathcal{N}(\mu, \sigma^2)</math> which has [[probability density function]]

<math display="block">f(x\mid \mu,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}\ }
                               \exp\left(-\frac {(x-\mu)^2}{2\sigma^2} \right), </math>

the corresponding [[probability density function]] for a sample of {{mvar|n}} [[independent identically distributed]] normal random variables (the likelihood) is

<math display="block">f(x_1,\ldots,x_n \mid \mu,\sigma^2) = \prod_{i=1}^n f( x_i\mid  \mu, \sigma^2) = \left( \frac{1}{2\pi\sigma^2} \right)^{n/2} \exp\left( -\frac{ \sum_{i=1}^n (x_i-\mu)^2}{2\sigma^2}\right).</math>

This family of distributions has two parameters: {{math|''θ''&nbsp;{{=}} (''μ'',&nbsp;''σ'')}}; so we maximize the likelihood, <math>\mathcal{L} (\mu,\sigma^2) = f(x_1,\ldots,x_n \mid \mu, \sigma^2)</math>, over both parameters simultaneously, or if possible, individually.

Since the [[natural logarithm|logarithm]] function itself is a [[continuous function|continuous]] [[strictly increasing]] function over the [[range (statistics)|range]] of the likelihood, the values which maximize the likelihood will also maximize its logarithm (the log-likelihood itself is not necessarily strictly increasing). The log-likelihood can be written as follows:

<math display="block">
   \log\Bigl( \mathcal{L} (\mu,\sigma^2)\Bigr) = -\frac{\,n\,}{2} \log(2\pi\sigma^2)
   - \frac{1}{2\sigma^2} \sum_{i=1}^n (\,x_i-\mu\,)^2
</math>

(Note: the log-likelihood is closely related to [[information entropy]] and [[Fisher information]].)

We now compute the derivatives of this log-likelihood as follows.

<math display="block">\begin{align}
0 & = \frac{\partial}{\partial \mu} \log\Bigl( \mathcal{L} (\mu,\sigma^2)\Bigr) =
 0 - \frac{\;-2 n(\bar{x}-\mu)\;}{2\sigma^2}.
\end{align}</math>
where <math> \bar{x} </math> is the [[sample mean]]. This is solved by

<math display="block">\widehat\mu = \bar{x} = \sum^n_{i=1} \frac{\,x_i\,}{n}. </math>

This is indeed the maximum of the function, since it is the only turning point in {{mvar|μ}} and the second derivative is strictly less than zero. Its [[expected value]] is equal to the parameter {{mvar|μ}} of the given distribution,

<math display="block">\operatorname{\mathbb E}\bigl[\;\widehat\mu\;\bigr] = \mu, \, </math>

which means that the maximum likelihood estimator <math>\widehat\mu</math> is unbiased.

Similarly we differentiate the log-likelihood with respect to {{mvar|σ}} and equate to zero:

<math display="block">\begin{align}
0 & = \frac{\partial}{\partial \sigma} \log\Bigl( \mathcal{L} (\mu,\sigma^2)\Bigr) = -\frac{\,n\,}{\sigma} 
   + \frac{1}{\sigma^3} \sum_{i=1}^{n} (\,x_i-\mu\,)^2.
\end{align}</math>

which is solved by

<math display="block">\widehat\sigma^2 = \frac{1}{n} \sum_{i=1}^n(x_i-\mu)^2.</math>

Inserting the estimate <math>\mu = \widehat\mu</math> we obtain

<math display="block">\widehat\sigma^2 = \frac{1}{n} \sum_{i=1}^n (x_i - \bar{x})^2 = \frac{1}{n}\sum_{i=1}^n x_i^2 -\frac{1}{n^2}\sum_{i=1}^n\sum_{j=1}^n x_i x_j.</math>

To calculate its expected value, it is convenient to rewrite the expression in terms of zero-mean random variables ([[statistical error]])  <math>\delta_i \equiv \mu - x_i</math>. Expressing the estimate in these variables yields

<math display="block">\widehat\sigma^2 = \frac{1}{n} \sum_{i=1}^n (\mu - \delta_i)^2 -\frac{1}{n^2}\sum_{i=1}^n\sum_{j=1}^n (\mu - \delta_i)(\mu - \delta_j).</math>

Simplifying the expression above, utilizing the facts that <math>\operatorname{\mathbb E}\bigl[\;\delta_i\;\bigr] = 0 </math> and <math>\operatorname{E}\bigl[\;\delta_i^2\;\bigr] = \sigma^2 </math>, allows us to obtain

<math display="block">\operatorname{\mathbb E}\bigl[\;\widehat\sigma^2\;\bigr]= \frac{\,n-1\,}{n}\sigma^2.</math>

This means that the estimator <math>\widehat\sigma^2</math> is biased for <math>\sigma^2</math>. It can also be shown that <math>\widehat\sigma</math> is biased for <math>\sigma</math>, but that both <math>\widehat\sigma^2</math> and <math>\widehat\sigma</math> are consistent.

Formally we say that the ''maximum likelihood estimator'' for <math>\theta=(\mu,\sigma^2)</math> is

<math display="block">\widehat{\theta\,} = \left(\widehat{\mu},\widehat{\sigma}^2\right).</math>

In this case the MLEs could be obtained individually.  In general this may not be the case, and the MLEs would have to be obtained simultaneously.

The normal log-likelihood at its maximum takes a particularly simple form:

<math display="block">
   \log\Bigl( \mathcal{L}(\widehat\mu,\widehat\sigma)\Bigr) = \frac{\,-n\;\;}{2} \bigl(\,\log(2\pi\widehat\sigma^2) +1\,\bigr)
</math>

This maximum log-likelihood can be shown to be the same for more general [[least squares]], even for [[non-linear least squares]].  This is often used in determining likelihood-based approximate [[confidence interval]]s and [[confidence region]]s, which are generally more accurate than those using the asymptotic normality discussed above.