Editing Least squares (section)

==Problem statement==
The objective consists of adjusting the parameters of a model function to best fit a data set. A simple data set consists of ''n'' points (data pairs) <math>(x_i,y_i)\!</math>, ''i'' = 1, …, ''n'', where <math>x_i\!</math> is an [[independent variable]] and <math>y_i\!</math> is a [[dependent variable]] whose value is found by observation. The model function has the form <math>f(x, \boldsymbol \beta)</math>, where ''m'' adjustable parameters are held in the vector <math>\boldsymbol \beta</math>. The goal is to find the parameter values for the model that "best" fits the data. The fit of a model to a data point is measured by its [[errors and residuals in statistics|residual]], defined as the difference between the observed value of the dependent variable and the value predicted by the model: 
<math display="block">r_i = y_i - f(x_i, \boldsymbol \beta).</math>
[[File:Linear Residual Plot Graph.png|thumb|The residuals are plotted against corresponding <math>x</math> values. The random fluctuations about <math>r_i=0</math> indicate a linear model is appropriate.|251x251px]]

The least-squares method finds the optimal parameter values by minimizing the [[sum of squared residuals]], <math>S</math>:<ref name=":0" />
<math display="block">S = \sum_{i=1}^n r_i^2.</math>

In the simplest case <math>f(x_i, \boldsymbol \beta)= \beta</math> and the result of the least-squares method is the [[arithmetic mean]] of the input data.

An example of a model in two dimensions is that of the straight line. Denoting the y-intercept as <math>\beta_0</math> and the slope as <math>\beta_1</math>, the model function is given by <math>f(x,\boldsymbol \beta)=\beta_0+\beta_1 x</math>. See [[Linear least squares #Example|linear least squares]] for a fully worked out example of this model.

A data point may consist of more than one independent variable. For example, when fitting a plane to a set of height measurements, the plane is a function of two independent variables, ''x'' and ''z'', say. In the most general case there may be one or more independent variables and one or more dependent variables at each data point.  

To the right is a residual plot illustrating random fluctuations about <math>r_i=0</math>, indicating that a linear model<math>(Y_i = \beta_0 + \beta_1 x_i + U_i)</math> is appropriate. <math>U_i</math> is an independent, random variable.<ref name=":0">{{Cite book|title=A modern introduction to probability and statistics: understanding why and how|date=2005|publisher=Springer | others=Dekking, Michel, 1946-|isbn=978-1-85233-896-1|location=London|oclc=262680588}}</ref>   
[[File:Parabolic Residual Plot Graph.png|thumb|247x247px|The residuals are plotted against the corresponding <math>x</math> values. The parabolic shape of the fluctuations about <math>r_i=0</math> indicates a parabolic model is appropriate.]] 

If the residual points had some sort of a shape and were not randomly fluctuating, a linear model would not be appropriate. For example, if the residual plot had a parabolic shape as seen to the right, a parabolic model <math>(Y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + U_i)</math> would be appropriate for the data. The residuals for a parabolic model can be calculated via <math>r_i=y_i-\hat{\beta}_0-\hat{\beta}_1 x_i-\hat{\beta}_2 x_i^2</math>.<ref name=":0" /> <!-- Also, the residuals may be weighted to take into account differences in the reliability of the measurements.
<math> S = \sum_{i=1}^n w_i r_i^2 </math>.
This is called '''weighted least squares,''' in contrast to '''ordinary least squares''' in which unit weights are used. -->