Editing Overfitting (section)

==Statistical inference==
{{expand section|date=October 2017}}
In statistics, an [[Statistical inference|inference]] is drawn from a [[statistical model]], which has been [[model selection|selected]] via some procedure. Burnham&nbsp;& Anderson, in their much-cited text on model selection, argue that to avoid overfitting, we should adhere to the "[[Principle of Parsimony]]".<ref name="BA2002">{{Citation |last1=Burnham |first1=K. P. |last2=Anderson |first2=D. R. |year=2002 |title=Model Selection and Multimodel Inference |edition=2nd |publisher=Springer-Verlag }}.</ref> The authors also state the following.<ref name="BA2002" />{{rp|32–33}}
{{quote|text= Overfitted models ... are often free of bias in the parameter estimators, but have estimated (and actual) sampling variances that are needlessly large (the precision of the estimators is poor, relative to what could have been accomplished with a more parsimonious model). False treatment effects tend to be identified, and false variables are included with overfitted models. ... A best approximating model is achieved by properly balancing the errors of underfitting and overfitting.}}

Overfitting is more likely to be a serious concern when there is little theory available to guide the analysis, in part because then there tend to be a large number of models to select from. The book ''Model Selection and Model Averaging'' (2008) puts it this way.<ref>{{citation|last1=Claeskens|first1=G.|author1-link= Gerda Claeskens |author-link2=Nils Lid Hjort|last2=Hjort|first2=N.L.|year=2008|title=Model Selection and Model Averaging|publisher=[[Cambridge University Press]]}}.</ref>
{{quote| text=Given a data set, you can fit thousands of models at the push of a button, but how do you choose the best? With so many candidate models, overfitting is a real danger. Is the [[infinite monkey theorem|monkey who typed Hamlet]] actually a good writer?}}

===Regression===
In [[regression analysis]], overfitting occurs frequently.<ref name="RMS">{{citation| title= Regression Modeling Strategies | last= Harrell | first= F. E. Jr. | year= 2001 | publisher= Springer}}.</ref> As an extreme example, if there are ''p'' variables in a [[linear regression]] with ''p'' data points, the fitted line can go exactly through every point.<ref>{{cite web
| url=http://www.ma.utexas.edu/users/mks/statmistakes/ovefitting.html
| title=Overfitting
| author=Martha K. Smith
| date=2014-06-13
| publisher=[[University of Texas at Austin]]
| access-date=2016-07-31}}</ref> For [[logistic regression]] or Cox [[proportional hazards models]], there are a variety of rules of thumb (e.g. 5–9,<ref name="Vittinghoff et al. (2007)">{{cite journal |first1=E. |last1=Vittinghoff |first2=C. E. |last2=McCulloch |year=2007 |title=Relaxing the Rule of Ten Events per Variable in Logistic and Cox Regression |journal=[[American Journal of Epidemiology]] |volume=165 |issue=6 |pages=710–718 |doi=10.1093/aje/kwk052 |pmid=17182981|doi-access= }}</ref> 10<ref>{{cite book
| title = Applied Regression Analysis 
| edition= 3rd 
| last1 = Draper
| first1 = Norman R.
| last2 = Smith
| first2 = Harry
| publisher = [[John Wiley & Sons|Wiley]]
| year = 1998
| isbn = 978-0471170822}}</ref> and 10–15<ref>{{cite web
| url = http://blog.minitab.com/blog/adventures-in-statistics/the-danger-of-overfitting-regression-models
| title = The Danger of Overfitting Regression Models
| author = Jim Frost
| date = 2015-09-03
| access-date = 2016-07-31}}</ref> — the guideline of 10 observations per independent variable is known as the "[[one in ten rule]]"). In the process of regression model selection, the mean squared error of the random regression function can be split into random noise, approximation bias, and variance in the estimate of the regression function. The [[bias–variance tradeoff]] is often used to overcome overfit models.

With a large set of [[explanatory variable]]s that actually have no relation to the [[dependent variable]] being predicted, some variables will in general be falsely found to be [[statistically significant]] and the researcher may thus retain them in the model, thereby overfitting the model. This is known as [[Freedman's paradox]].