Editing Maximum likelihood estimation (section)

=== Example ===
<math>X_1,\ X_2,\ldots,\ X_m</math> are counts in cells / boxes 1 up to m; each box has a different probability (think of the boxes being bigger or smaller) and we fix the number of balls that fall to be <math>n</math>:<math>x_1+x_2+\cdots+x_m=n</math>. The probability of each box is <math>p_i</math>, with a constraint: <math>p_1+p_2+\cdots+p_m=1</math>. This is a case in which the <math>X_i</math> ''s'' are not independent, the joint probability of a vector <math>x_1,\ x_2,\ldots,x_m</math> is called the multinomial and has the form:

<math display="block">f(x_1,x_2,\ldots,x_m\mid p_1,p_2,\ldots,p_m)=\frac{n!}{\prod x_i!}\prod p_i^{x_i}= \binom{n}{x_1,x_2,\ldots,x_m} p_1^{x_1} p_2^{x_2} \cdots p_m^{x_m}</math>

Each box taken separately against all the other boxes is a binomial and this is an extension thereof.

The log-likelihood of this is:

<math display="block">\ell(p_1,p_2,\ldots,p_m)=\log n!-\sum_{i=1}^m \log x_i!+\sum_{i=1}^m x_i\log p_i</math>

The constraint has to be taken into account and use the Lagrange multipliers:

<math display="block">L(p_1,p_2,\ldots,p_m,\lambda)=\ell(p_1,p_2,\ldots,p_m)+\lambda\left(1-\sum_{i=1}^m p_i\right)</math>

By posing all the derivatives to be 0, the most natural estimate is derived

<math display="block">\hat{p}_i=\frac{x_i}{n}</math>

Maximizing log likelihood, with and without constraints, can be an unsolvable problem in closed form, then we have to use iterative procedures.