Editing Statistical hypothesis test (section)

===Clairvoyant card game===
A person (the subject) is tested for [[clairvoyance]]. They are shown the back face of a randomly chosen playing card 25 times and asked which of the four [[Suit (cards)|suits]] it belongs to. The number of hits, or correct answers, is called ''X''.

As we try to find evidence of their clairvoyance, for the time being the null hypothesis is that the person is not clairvoyant.<ref>{{cite book|last1=Jaynes|first1=E. T.|title=Probability theory : the logic of science|date=2007|publisher=Cambridge Univ. Press|isbn=978-0-521-59271-0|edition=5. print.|location=Cambridge [u.a.]}}</ref> The alternative is: the person is (more or less) clairvoyant.

If the null hypothesis is valid, the only thing the test person can do is guess. For every card, the probability (relative frequency) of any single suit appearing is 1/4. If the alternative is valid, the test subject will predict the suit correctly with probability greater than 1/4. We will call the probability of guessing correctly ''p''. The hypotheses, then, are:
* null hypothesis <math>\text{:} \qquad H_0: p = \tfrac 14</math> &nbsp;&nbsp;&nbsp;&nbsp;(just guessing)
and
* alternative hypothesis <math>\text{:} H_1: p > \tfrac 14</math> &nbsp;&nbsp;&nbsp;(true clairvoyant).

When the test subject correctly predicts all 25 cards, we will consider them clairvoyant, and reject the null hypothesis. Thus also with 24 or 23 hits. With only 5 or 6 hits, on the other hand, there is no cause to consider them so. But what about 12 hits, or 17 hits? What is the critical number, ''c'', of hits, at which point we consider the subject to be clairvoyant? How do we determine the critical value ''c''? With the choice ''c''=25 (i.e. we only accept clairvoyance when all cards are predicted correctly) we're more critical than with ''c''=10. In the first case almost no test subjects will be recognized to be clairvoyant, in the second case, a certain number will pass the test. In practice, one decides how critical one will be. That is, one decides how often one accepts an error of the first kind – a [[false positive]], or Type I error. With ''c'' = 25 the probability of such an error is:

:{{nowrap|<math>P(\text{reject }H_0 \mid H_0 \text{ is valid}) = P\left(X = 25\mid p=\frac 14\right)=\left(\frac 14\right)^{25}\approx10^{-15}</math>,}}

and hence, very small. The probability of a false positive is the probability of randomly guessing correctly all 25 times.

Being less critical, with ''c'' = 10, gives:

:{{nowrap|<math>P(\text{reject }H_0 \mid H_0 \text{ is valid}) = P\left(X \ge 10 \mid p=\frac 14\right) = \sum_{k=10}^{25}P\left(X=k\mid p=\frac 14\right) = \sum_{k=10}^{25} \binom{25}{k}\left( 1- \frac 14\right)^{25-k} \left(\frac 14\right)^k \approx 0.0713</math>.}}

Thus, ''c'' = 10 yields a much greater probability of false positive.

Before the test is actually performed, the maximum acceptable probability of a Type I error (''α'') is determined. Typically, values in the range of 1% to 5% are selected. (If the maximum acceptable error rate is zero, an infinite number of correct guesses is required.) Depending on this Type 1 error rate, the critical value ''c'' is calculated. For example, if we select an error rate of 1%, ''c'' is calculated thus:

:{{nowrap|<math>P(\text{reject }H_0 \mid H_0 \text{ is valid}) = P\left(X \ge c\mid p=\frac 14\right) \le 0.01</math>.}}

From all the numbers c, with this property, we choose the smallest, in order to minimize the probability of a Type II error, a [[false negative]]. For the above example, we select: <math>c=13</math>.
<!--
But what if the subject did not guess any cards at all? Having zero correct answers is clearly an oddity too. Without any clairvoyant skills the probability.

:<math>P(X=0 \mid H_0 \text{ is valid}) = P\left(X = 0\mid p=\frac 14\right) = \left(1-\frac 14\right)^{25} \approx 0.00075</math>.

This is highly unlikely (less than 1 in a 1000 chance). While the subject can't guess the cards correctly, dismissing H<sub>0</sub> in favour of H<sub>1</sub> would be an error. In fact, the result would suggest a trait on the subject's part of avoiding calling the correct card. A test of this could be formulated: for a selected 1% error rate the subject would have to answer correctly at least twice, for us to believe that card calling is based purely on guessing. -->