Type I & Type II Errors and Power Simulator

The graph shows two Normal sampling distributions of the sample mean. The solid navy curve labelled H0 is centred at the null value $\mu_0$; the dashed coral curve labelled Ha is centred at the true mean. The gold dashed line marked x* is the critical value: a sample mean beyond it leads you to reject $H_0$. Both curves have standard deviation $\sigma/\sqrt{n}$, so the simulator treats $\sigma$ as known.

The controls:

Three shaded regions match three boxes under the graph. Gold is α, the part of the H0 curve inside the rejection region. Coral is β, the part of the Ha curve on the fail-to-reject side. Teal is power, the rest of the Ha curve. The α box repeats your chosen level, β and power are given to three decimal places, and the power number turns teal once it reaches 0.80 (it is coral below that).

At the default settings the curves hardly overlap and power reads 1.000, so lower the effect size or n before comparing settings. Try an effect size of 0: β reads 0.950 and power reads 0.050 at α = 0.05. When $H_0$ is actually true, the only way to reject it is by making a Type I error.

For a right-tailed test with known $\sigma$, which is what the simulator draws, the critical value and $\beta$ are:

Power rises when the effect size grows, n grows, $\sigma$ shrinks or $\alpha$ grows. The first three push the curves apart or make them narrower. A larger $\alpha$ slides x* toward the centre of H0, which buys power at the cost of more Type I errors. Of the four, increasing n is the usual way to gain power without accepting more Type I errors.

Problem: An engineer redesigns a battery and tests $H_0: \mu = 20$ hours against $H_a: \mu > 20$ at $\alpha = 0.05$. Battery life has $\sigma = 2$ hours. If the new design really averages 20.5 hours, what is the power of the test with 36 batteries? With 100?

Step 1: Standard deviation of $\bar{x}$. $\sigma/\sqrt{n} = 2/\sqrt{36} = 0.333$ hours.

Step 2: Critical value. For a right-tailed test at 0.05, $z^* = 1.645$, so $\bar{x}^* = 20 + 1.645(0.333) = 20.548$ hours.

Step 3: Type II error probability. $\beta = P(\bar{x} < 20.548 \text{ when } \mu = 20.5) = P\left(Z < \frac{20.548 - 20.5}{0.333}\right) = P(Z < 0.145) \approx 0.558$. Power $= 1 - 0.558 = 0.442$.

Step 4: Repeat with n = 100. $\sigma/\sqrt{n} = 0.2$, $\bar{x}^* = 20 + 1.645(0.2) = 20.329$, and $\beta = P(Z < (20.329 - 20.5)/0.2) = P(Z < -0.855) \approx 0.196$. Power $\approx 0.804$.

Interpretation: with 36 batteries the test misses a real half-hour improvement more often than it detects it. About 100 batteries are needed; n = 99 is the smallest sample size that gets power to 0.80.

Check it in the simulator: set effect size to 0.50, σ to 2.00 and n to 36, with α = 0.05 and Right tail. The boxes read β = 0.558 and power = 0.442. Slide n to 100 and they change to 0.196 and 0.804, and the power number turns teal. Go back to n = 36 and press α = 0.10: power climbs to 0.586, but the gold α area doubles. Switch to Two tail and power at n = 36 drops to 0.323.

This is topic 3.8, Potential Errors When Performing Tests, in Unit 3 (Inference for Categorical Data: Proportions). The same reasoning carries over to the tests for means in Unit 4. Expect to identify and describe Type I and Type II errors in context, to state a consequence of each, and to explain how changing n or α affects power.