Week 11 — Hypothesis Testing Logic

This week in a line: you’ll learn to test whether one specific claimed value is believable, using a test statistic, a p-value, and a clear decide / fail-to-reject rule.

Last week you built a confidence interval — a whole range of plausible values for an unknown population parameter. This week flips the question around. Instead of “what range of values is plausible?” you’ll ask something sharper: “is this one specific claimed value believable, given what my sample shows?” Your roommate insists the coin from the laundry room “seems unfair” — like it lands heads more often than tails. You can’t interrogate a coin, so you flip it, count, and reason from the data. Spelled out formally, that reasoning process is a hypothesis test, and it’s the backbone of almost every “does this actually work?” claim you’ll meet after this course — in a lab report, a business memo, or a news headline.

1Two competing claims: $H_0$ and $H_A$¶

Every hypothesis test starts with two claims about a population parameter that can’t both be true. One is the null hypothesis, $H_0$ — a skeptical, “nothing new going on” claim, always stated as an equality. The other is the alternative hypothesis, $H_A$ — what you suspect might be true instead, stated with $<$, $>$, or $\neq$. The logic mirrors a courtroom: you assume $H_0$ (innocence) unless the sample data gives strong enough evidence to abandon it.

If $H_A$ uses $\neq$, the test is two-sided (evidence in either direction counts against $H_0$). If $H_A$ uses $<$ or $>$, the test is one-sided (only evidence in the stated direction counts).

• Coin fairness. No reason to suspect a particular direction, so this is two-sided:

  $$H_0: p = 0.5 \qquad H_A: p \neq 0.5$$

• Study-skills workshop. The workshop is designed to raise scores, so this is one-sided:

  $$H_0: \mu = 70 \qquad H_A: \mu > 70$$

Figure 1. Two bell-shaped sampling-distribution sketches side by side. The left one, centered at $p_0=0.5$, has both tails shaded past a marked point on either side (two-sided evidence). The right one, centered at $\mu_0=70$, has only its right tail shaded past a marked point (one-sided evidence).

2The test statistic: how far from $H_0$, in SE units¶

Once you assume $H_0$ is true, ask: how surprising is the sample statistic under that assumption? The test statistic answers this the same way Week 6’s z-score did — by measuring distance in standard error units, using the standard errors you built in Weeks 9–10:

Here $\hat p$ and $\bar x$ are the sample statistic actually observed; $p_0$ and $\mu_0$ are the hypothesized values from $H_0$; and $\mathrm{SE}_0$ is the standard error computed as if $H_0$ were true. A large $|z|$ means the sample landed far from what $H_0$ predicted — evidence against it.

3The p-value: probability of data this extreme, if $H_0$ is true¶

The p-value is the probability of getting a test statistic at least as extreme as the one you observed, assuming $H_0$ is true. It is a conditional probability — conditioned on an assumption you haven’t confirmed.

The decision rule compares the p-value to a significance level $\alpha$ — the risk of a wrong rejection you agree to accept before seeing the data, usually $\alpha=0.05$ in this course:

Failing to reject $H_0$ is not the same as proving $H_0$ true. It only means the sample didn’t give strong enough evidence to abandon it — absence of evidence isn’t evidence of absence.

Worked Example 1 — Is the laundry-room coin fair? Simulating $n=100$ flips of a genuinely fair coin ($p_0=0.5$, set.seed(1209)) gives 42 heads, so $\hat p = 42/100 = 0.42$. Under $H_0$,

The sample proportion sits 1.60 SEs below the hypothesized 0.5. For a two-sided test, the p-value adds both tails beyond $|z|=1.60$: $p \approx 0.1096$. Since $0.1096 \geq 0.05 = \alpha$, we fail to reject $H_0$: this sample of 100 flips does not give strong evidence the coin is unfair. (It might still be biased — 100 flips just wasn’t enough to tell. More on that below.)

4One-sided tests, and the cost of being wrong¶

Worked Example 2 — Does the study-skills workshop raise scores? CSUB’s math-placement diagnostic (a computer-adaptive exam scored on a 0–200 scale) has historically averaged $\mu_0=70$ points with a known population standard deviation $\sigma=12$ (a stated benchmark built from many prior years, so $\sigma$ is treated as known here — the large-sample z story). This term, $n=40$ students completed a new study-skills workshop before testing; simulating their scores (set.seed(1209), centered 4 points above the historical mean to model a real but modest effect) gives $\bar x = 75.15$ and a sample spread $s=14.74$. Since $\sigma$ is treated as known, the test still uses $\sigma$, not $s$, in the standard error (Week 14’s t-test will use $s$ instead):

$H_A:\mu>70$ is one-sided, so the p-value is only the right tail beyond $z=2.71$: $p\approx 0.0033$. Since $0.0033<0.05=\alpha$, we reject $H_0$: this sample gives strong evidence the workshop group’s average diagnostic score is higher than the historical 70.

Statistical significance vs. practical importance. The average shift (about 5 points) is statistically clear, but the spread among individual students, $s\approx14.74$, is nearly three times that shift — a large n makes small average effects easy to detect statistically, but a program director still has to ask separately whether a 5-point average bump is worth the workshop’s cost and time across students who vary that much individually.

Type I and Type II errors. No decision rule is ever certain, because you never actually know whether $H_0$ is true. A Type I error is rejecting an $H_0$ that is actually true; by design, this happens with probability exactly $\alpha$. In Example 2, if the workshop truly had no effect, a batch of 40 students would still produce a “significant” result about 5% of the time purely by chance — the price of using $\alpha=0.05$. A Type II error is failing to reject an $H_0$ that is actually false. In Example 1, if the laundry-room coin actually is slightly biased, 100 flips may simply not be enough evidence to catch it — we’d fail to reject fairness even though it’s false. Which error costs more depends on context: a factory that stops a good production line (Type I) wastes time and money; a factory that lets a broken line keep running (Type II) ships defective product.

See it in R.

library(mosaic)
library(BSDA)

# Worked Example 1: coin fairness, two-sided --------------------------------
set.seed(1209)
flips <- rflip(100, prob = 0.5)          # 42 heads out of 100 this run
prop.test(x = 42, n = 100, p = 0.5, alternative = "two.sided", correct = FALSE)

# Worked Example 2: workshop diagnostic scores, one-sided --------------------
zsum.test(mean.x = 75.15, sigma.x = 12, n.x = 40, mu = 70, alternative = "greater")

prop.test prints X-squared = 2.56, p-value = 0.1096 — note $\sqrt{2.56}=1.60$, the same $z$ computed by hand. zsum.test prints z = 2.7143, p-value = 0.003321, matching Example 2 above.

5Check your understanding¶

1. A campus dining hall claims “at least 90% of students are satisfied with the new menu.” A student group suspects satisfaction is actually lower and surveys a random sample to test this. State $H_0$ and $H_A$ in symbols, and say whether the test is one-sided or two-sided.

2. Explain in your own words why “fail to reject $H_0$” is not the same as “$H_0$ is true.” Use the coin-fairness result from Worked Example 1 in your explanation.

3. A classmate looks at Worked Example 2’s result ($z=2.71$, $p\approx0.0033$) and says, “That means there’s only a 0.33% chance the workshop doesn’t work.” Explain what is wrong with that statement, and give the correct interpretation of the p-value.

4. Using the coin-fairness setup from Worked Example 1 ($p_0=0.5$, $n=100$, same $\mathrm{SE}_0=0.05$), suppose the sample had instead come out $\hat p = 0.61$. Compute the z test statistic.

5. A researcher sets $\alpha=0.05$ in advance and gets a p-value of 0.04. Should they reject or fail to reject $H_0$? Now suppose they had set $\alpha=0.01$ instead, for that same p-value — does the decision change? What does this tell you about the role of $\alpha$?

6. In one or two sentences, describe a real (non-classroom) situation where a Type I error and a Type II error would have different real-world costs — for example, a medical test, a spam filter, or a quality-control check. Name which error is more costly in your example and why.

6Key terms¶

• Null hypothesis, $H_0$ — the skeptical, “nothing new going on” claim about a parameter, always stated as an equality; assumed true unless the data give strong evidence otherwise.

• Alternative hypothesis, $H_A$ — the claim you suspect might be true instead, stated with $<$, $>$, or $\neq$.

• One-sided / two-sided test — a test where $H_A$ points in one direction ($<$ or $>$) vs. either direction ($\neq$).

• Test statistic — a standardized measure (here, $z$) of how far the sample statistic is from the value $H_0$ claims, in standard-error units.

• p-value — the probability of a test statistic at least this extreme, assuming $H_0$ is true.

• Significance level, $\alpha$ — the pre-chosen risk of a wrong rejection; commonly 0.05.

• Statistically significant — the decision reached when the p-value is less than $\alpha$.

• Type I error — rejecting a true $H_0$; occurs with probability $\alpha$ by design.

• Type II error — failing to reject a false $H_0$.