Hypothesis tests

The idea of a hypothesis test Paper 6

A hypothesis test asks whether data give real evidence against an accepted value of a parameter, or whether the result could easily be chance.

• The null hypothesis ${\mathrm{H}}_0$ states the accepted value, for example ${\mathrm{H}}_0:p=0.3$. The test is carried out assuming it is true.
• The alternative hypothesis ${\mathrm{H}}_1$ says how we suspect the value has changed: $p<0.3$, $p>0.3$, or $p\ne 0.3$.
• The test statistic is the quantity calculated from the sample: a count $X$, or a sample mean $\bar{X}$, or its standardised value $z$.
• The significance level $\alpha$, such as 5%, is the threshold for "too unlikely to be chance".

If, assuming ${\mathrm{H}}_0$, the probability of a result as extreme as the one observed, or more extreme, is less than $\alpha$, we reject ${\mathrm{H}}_0$: there is evidence for ${\mathrm{H}}_1$. Otherwise we do not reject ${\mathrm{H}}_0$: there is not enough evidence. Never say that ${\mathrm{H}}_0$ is proved true, or that ${\mathrm{H}}_1$ is proved: a test gives evidence, not certainty.

One tail or two. If ${\mathrm{H}}_1$ points in one direction ($p>0.3$ or $p<0.3$) the test is one-tailed. If it only says "has changed" ($p\ne 0.3$) it is two-tailed, and the significance level is split equally between the two ends: at 5%, compare the probability in the relevant tail with 2.5%. Choose the alternative from the wording of the question before looking at the data.

Regions. The critical region (or rejection region) is the set of values of the test statistic that lead to rejecting ${\mathrm{H}}_0$; the acceptance region is the rest. The critical value is the boundary between them.

Conclusions in context. Every test ends with a sentence about the real situation, with a non-definite word such as "evidence": "There is evidence at the 5% significance level that the proportion of customers buying dessert has decreased."

Tests using the binomial distribution Paper 6

For a single observation $X$ from $\mathrm{B}(n,p)$, the test is about $p$:

1. State ${\mathrm{H}}_0:p=p_0$ and ${\mathrm{H}}_1$, defining $p$ in context.
2. Under ${\mathrm{H}}_0$, $X\sim \mathrm{B}(n,p_0)$.
3. Find the probability of the observed value or more extreme, in the direction of ${\mathrm{H}}_1$: $\mathrm{P}(X⩽x)$ for ${\mathrm{H}}_1:p<p_0$, $\mathrm{P}(X⩾x)$ for ${\mathrm{H}}_1:p>p_0$.
4. Compare with the significance level (or half of it for a two-tailed test), decide, and conclude in context.

Use cumulative probabilities, never the single probability $\mathrm{P}(X=x)$: any one value is unlikely, so the question is whether the observation is far out in the tail.

Finding a critical region. Add up tail probabilities until the next one would exceed the significance level. For example, under $\mathrm{B}(25,0.3)$, $\mathrm{P}(X⩽3)=0.0332$ but $\mathrm{P}(X⩽4)=0.0905$, so at 5% the critical region for ${\mathrm{H}}_1:p<0.3$ is $X⩽3$. Its actual probability, 0.0332, is less than 5% because $X$ is discrete.

Tests using the Poisson distribution Paper 6

For a single observation from $\mathrm{Po}(\lambda )$ the method is the same, with hypotheses about the mean rate $\lambda$. Scale $\lambda$ to the interval observed first: if the usual rate is 1.5 per day and one week is observed, ${\mathrm{H}}_0:\lambda =10.5$ for that week, or state the hypotheses about the daily rate and use $\mathrm{Po}(10.5)$ for the weekly count.

Using a normal approximation Paper 6

For large samples, exact binomial or Poisson tails need many terms. Use the approximations from earlier topics:

• $\mathrm{B}(n,p)\approx \mathrm{N}(np,np(1-p))$ when $np>5$ and $n(1-p)>5$;
• $\mathrm{Po}(\lambda )\approx \mathrm{N}(\lambda ,\lambda )$ when $\lambda >15$;

in each case with a continuity correction. For an observed $x=66$ with ${\mathrm{H}}_1:p<p_0$, find $\mathrm{P}(X⩽66)\approx \mathrm{P}(Y<66.5)$. The parameters of the normal distribution come from ${\mathrm{H}}_0$, because the test assumes ${\mathrm{H}}_0$ is true.

Tests for a population mean Paper 6

To test ${\mathrm{H}}_0:\mu ={\mu }_0$ from a sample of size $n$, use the sample mean. Under ${\mathrm{H}}_0$,
$$\bar{X}\sim \mathrm{N}({\mu }_0,\frac{{\sigma }^2}{n}),$$
exactly if the population is normal with known variance, or approximately by the Central Limit Theorem if the sample is large, with ${\sigma }^2$ estimated by $s^2$. The test statistic is
$$z=\frac{\bar{x}-{\mu }_0}{\sigma /\sqrt{n}}\mathrm{.}$$
Compare $z$ with the critical value from MF19: for a one-tailed test at 5% it is 1.645 (or $-1.645$ for ${\mathrm{H}}_1:\mu <{\mu }_0$); for a two-tailed test at 5% it is $\pm 1.960$; at 1%, 2.326 one-tailed and $\pm 2.576$ two-tailed. Reject ${\mathrm{H}}_0$ if $z$ is beyond the critical value. Equivalently, compare the tail probability $\mathrm{P}(Z>|z|)$ with the significance level. Show the comparison explicitly.

A two-tailed test at 5%: the critical region is $z<-1.96$ or $z>1.96$, each tail holding 2.5%. For a one-tailed test the whole 5% sits in one tail, beyond 1.645 or $-1.645$.

Type I and Type II errors Paper 6

A test can go wrong in two ways:

• a Type I error is rejecting ${\mathrm{H}}_0$ when it is actually true;
• a Type II error is not rejecting ${\mathrm{H}}_0$ when it is actually false.

P(Type I error) $=\mathrm{P}(\text{test statistic in the critical region}|{\mathrm{H}}_0\text{true})$. For a normal test this is exactly the significance level. For a binomial or Poisson test it is the actual probability of the critical region, which is usually a little less than the stated level.

P(Type II error) needs a particular true value of the parameter, which the question gives: it is $\mathrm{P}(\text{test statistic in the acceptance region}|\text{that true value})$. So first find the critical region using ${\mathrm{H}}_0$, then find the probability of not being in it using the new value.

The two errors pull against each other: making the critical region smaller reduces the chance of a Type I error but increases the chance of a Type II error. A larger sample can reduce both.