Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Chapter 12 — ANOVA: Comparing Many Means

MATH 2200 · Introduction to Statistical Concepts and Methods

1Why a fourth way to compare groups?

You already know how to compare two groups. In Chapter 10 you ran a two-sample tt-test: is the average PM2.5 at one Bakersfield monitor different from the average at another? But Bakersfield does not have two air monitors. It has three in the city limits — California Avenue, Golden / M Street, and the Airport (Planz Road) site — plus more in Oildale, Shafter, and the desert towns. A reasonable air-quality question is not “do these two differ?” but “do any of these monitors differ from the rest?”

Here is the real 2023 picture, computed live from the curated EPA dataset kern_airquality (data/codebooks/kern_airquality.md):

The three monitors’ 2023 average daily PM2.5 readings are 11.93, 13.69, and 12.47 µg/m³ (dataset-derived from kern_airquality; PM2.5 daily_mean for the 857 Bakersfield monitor-days). They are close but not identical. Is that gap real — a true difference in the air across the city — or just the random wobble you would expect from sampling a few hundred noisy days at each site?

The honest worry: if you ran a separate two-sample tt-test for every pair of monitors, each test carries its own 5% false-alarm risk, and the risk of at least one false alarm balloons as you add groups. ANOVA — the ANalysis Of VAriance — answers the single question “are these group means all equal?” with one test at one error rate. This chapter is about that test.

2Learning objectives

By the end of this chapter you will be able to:

  1. State the ANOVA hypotheses for comparing three or more group means and explain the role of between-group versus within-group variation.

  2. Conduct a one-way ANOVA in R with aov() + anova() and interpret the FF-statistic and its pp-value in context.

  3. Check the ANOVA conditions (independence, approximate normality, roughly equal variances) and judge whether the test is trustworthy.

  4. Explain why a single ANOVA is preferred to many pairwise tt-tests (the multiple-comparisons problem).

  5. Decide whether a follow-up pairwise comparison is warranted, and interpret it cautiously.

Prerequisite: Chapter 10 (inference for means: the tt procedures, degrees of freedom, conditions for means inference).

R skills introduced: one-way ANOVA with aov(y ~ g, data =) + anova(); group-means plots with gf_boxplot(); reading and reporting an FF/pp ANOVA table; notes on pairwise follow-up.

Durable skills: quantitative reasoning (decomposing variation) · critical thinking (the multiple-comparisons trap).


312.1 The big idea: comparing variation, not just means

3.1Intuition

The name is a little backwards. ANOVA compares means, but it does so by analyzing variance. Here is the trick that makes it work.

Picture three groups of dots scattered on a number line — exam scores for three class sections, say. There are two completely different reasons the dots are spread out:

Now the key insight. If the groups truly have the same population mean, then the only reason their sample averages differ at all is luck — the same luck that makes individuals within a group differ. In that world, between-group spread and within-group spread are two estimates of the same thing, and their ratio should sit near 1.

But if the groups really differ, the between-group spread gets an extra push from the genuine mean differences, while the within-group spread does not. The ratio climbs above 1. ANOVA is exactly that ratio:

F=between-group variation (signal + noise)within-group variation (noise alone).F = \frac{\text{between-group variation (signal + noise)}}{\text{within-group variation (noise alone)}}.

A big FF says “the gaps between group averages are too large to blame on the ordinary scatter inside the groups.” A small FF (near 1) says “everything I see is consistent with one common mean.”

3.2Formula

We compare kk groups (here kk is the number of groups, e.g. k=3k = 3 monitors). Let:

We split the total variation into the two pieces from the intuition.

Between-group sum of squares — how far each group mean sits from the grand mean, weighted by group size:

SSB=j=1knj(xˉjxˉ)2.\text{SSB} = \sum_{j=1}^{k} n_j \,(\bar{x}_j - \bar{x})^2 .

Within-group sum of squares — how far each observation sits from its own group mean:

SSW=j=1ki=1nj(xijxˉj)2.\text{SSW} = \sum_{j=1}^{k} \sum_{i=1}^{n_j} (x_{ij} - \bar{x}_j)^2 .

These add up to the total variation: SST=SSB+SSW\text{SST} = \text{SSB} + \text{SSW}.

A sum of squares is not yet a fair comparison, because SSB is built from kk groups and SSW from NN observations. We divide each by its degrees of freedom — the number of independent pieces of information it rests on:

dfbetween=k1,dfwithin=Nk.df_{\text{between}} = k - 1, \qquad df_{\text{within}} = N - k .

Dividing gives the two mean squares (these are the two variance estimates):

MSB=SSBk1,MSW=SSWNk.\text{MSB} = \frac{\text{SSB}}{k - 1}, \qquad \text{MSW} = \frac{\text{SSW}}{N - k}.

And the FF-statistic is their ratio:

F=MSBMSW\boxed{\,F = \dfrac{\text{MSB}}{\text{MSW}}\,}

Under the null hypothesis H0: μ1=μ2==μkH_0:\ \mu_1 = \mu_2 = \dots = \mu_k (all kk population means are equal, where μj\mu_j is the true mean of group jj), FF follows an FF-distribution with (k1, Nk)(k-1, \ N-k) degrees of freedom. The alternative is HAH_A: at least one group mean differs from the others — not that they are all different.

The pp-value is the upper-tail area: the probability of an FF at least as large as the one we observed, if H0H_0 were true. ANOVA is always one-sided on the right, because only large FF values are evidence against “all means equal.”

Finally, a plain effect size — eta-squared — reports the share of total variation explained by the grouping:

η2=SSBSST.\eta^2 = \frac{\text{SSB}}{\text{SST}} .

It runs from 0 (groups explain nothing) to 1 (groups explain everything).

3.3R

The mosaic/base-R workflow is two steps that follow the formula grammar you already know: fit the model with aov(response ~ group, data = D), then print the classic ANOVA table with anova() — sums of squares, degrees of freedom, mean squares, the FF-statistic, and its pp-value. Here is the Bakersfield question:

air <- read.csv("data/processed/kern_airquality.csv")
bak <- subset(air, pollutant == "PM2.5" & city == "Bakersfield")

# aov(response ~ group): daily_mean broken down by site_name.
fit <- aov(daily_mean ~ site_name, data = bak)

anova(fit) prints the ANOVA table. The one thing it does not print is the effect size η2\eta^2, so we compute that from the table’s sums of squares:

anova(fit)                       # SS, df, MS, F value, and Pr(>F)

# eta-squared effect size = SSB / SST, read from the ANOVA table:
ss <- anova(fit)[["Sum Sq"]]     # c(between, within)
eta_squared <- ss[1] / sum(ss)
round(eta_squared, 4)

The anova(fit) table reports F value = 4.0653 on 2 and 854 degrees of freedom with Pr(>F) = 0.01749, and the effect size works out to η2=0.0094\eta^2 = 0.0094 (all dataset-derived from kern_airquality). Working at a significance level of α=0.05\alpha = 0.05 (the false-alarm rate you decide to accept, from Chapter 8), we reject H0H_0: there is evidence that average daily PM2.5 differs across the three Bakersfield monitors — even though η2\eta^2 tells us the monitor explains under 1% of the day-to-day variation. (Air quality swings far more from day to day — winter inversions, summer dust — than it does from corner to corner of the city. Both facts are true at once, and ANOVA reports both.)


412.2 The multiple-comparisons problem

4.1Intuition

Why not just run a tt-test on each pair of monitors? With three monitors there are three pairs (California–Golden, California–Airport, Golden–Airport). The trouble is that every test you run is another roll of the false-alarm dice.

If a single test uses α=0.05\alpha = 0.05, it has a 5% chance of crying “difference!” when there is none. Run several independent tests and the chance that at least one of them raises a false alarm is

P(at least one false alarm)=1(1α)m,P(\text{at least one false alarm}) = 1 - (1 - \alpha)^{m},

where mm is the number of tests and α\alpha is the per-test error rate. Watch it grow:

m <- 1:10
fwer <- 1 - (1 - 0.05)^m
data.frame(tests = m, familywise_error = round(fwer, 3))

With m=3m = 3 pairwise tests the family-wise error rate is already about 0.143 — nearly triple the 5% you thought you were using. With ten groups (45 pairs) you are almost guaranteed a false positive. ANOVA sidesteps this by asking one question — “are all the means equal?” — at one error rate.

4.2Formula

The fix, when you do need pairwise answers, is to shrink each test’s threshold so the whole family of tests still totals 5%. The simplest is the Bonferroni adjustment: with mm comparisons, test each one at a stricter threshold α\alpha^{*} (read “alpha-star,” the adjusted per-test cutoff),

α=αm,\alpha^{*} = \frac{\alpha}{m},

so for three pairs you would require p<0.05/30.0167p < 0.05/3 \approx 0.0167 on each. A slightly more powerful, ANOVA-specific method is Tukey’s Honest Significant Difference (HSD), which adjusts all pairwise comparisons jointly. Either way, the principle is identical: more looks demand a stricter bar.

4.3R

You almost never compute family-wise rates by hand; you let the procedure do it. But seeing the arithmetic once builds trust:

alpha     <- 0.05
m         <- choose(3, 2)            # number of pairwise comparisons among 3 groups
alpha_adj <- alpha / m
c(comparisons = m, per_test_threshold = round(alpha_adj, 4))

512.3 Worked example 1 — Do Bakersfield monitors differ? (Kern data)

Intuition. Three city monitors, hundreds of daily PM2.5 readings each, three averages a couple of µg/m³ apart. Is the spread between the monitor averages big relative to the spread within each monitor’s daily readings?

Formula. We need SSB, SSW, the two mean squares, and F=MSB/MSWF = \text{MSB}/\text{MSW} on (k1,Nk)=(2,854)(k-1, N-k) = (2, 854) df.

Computation. First, look — always plot before you test:

air <- read.csv("data/processed/kern_airquality.csv")
bak <- subset(air, pollutant == "PM2.5" & city == "Bakersfield")

gf_boxplot(daily_mean ~ site_name, data = bak, fill = okabe_ito[1]) %>%
  gf_labs(title = "Daily PM2.5 by Bakersfield monitor (2023)",
          x = "Monitor",
          y = "Daily mean PM2.5 (µg/m³)") %>%
  gf_theme(theme_minimal(base_size = 13))
Side-by-side boxplots of daily PM2.5 in micrograms per cubic meter for three Bakersfield monitors — Airport (Planz), California, and Golden/M Street. All three medians fall between roughly 9 and 12, the boxes overlap substantially, and each monitor shows a long upper tail of high-pollution winter days reaching past 40 micrograms per cubic meter.

Daily PM2.5 by Bakersfield monitor, 2023. The Golden/M Street box sits slightly higher than the California Avenue box, but all three overlap heavily — a small between-monitor signal inside a large day-to-day spread.

Now the test:

fit1 <- aov(daily_mean ~ site_name, data = bak)
anova(fit1)

# Equal-variance condition: ratio of largest to smallest group SD.
sd_by_site <- favstats(daily_mean ~ site_name, data = bak)$sd
max(sd_by_site) / min(sd_by_site)

Reading the printed ANOVA table (with sums of squares carried to full precision), the computation is:

SourceSSdfMSFF
Between (monitor)582.192291.094.07
Within (residual)61150.7485471.61
Total61732.92856

(Sums of squares are dataset-derived from kern_airquality; grand mean xˉ=12.74\bar{x} = 12.74 µg/m³.)

Interpretation. F=4.07F = 4.07, p=0.0175<0.05p = 0.0175 < 0.05, so we reject H0H_0: average daily PM2.5 is not the same at all three Bakersfield monitors. To check the equal-variance condition, the second line of the cell above divides the largest group SD by the smallest: the ratio is about 1.04 — comfortably under 2, so the condition is met, and with hundreds of days per group the means are well-behaved. But η2=0.0094\eta^2 = 0.0094: the monitor explains under 1% of the variation in daily PM2.5. Statistically detectable, practically tiny — a distinction every honest analyst must report.


612.4 Worked example 2 — Which monitors differ? (pairwise follow-up)

Intuition. The ANOVA said “at least one differs.” Fine — which one? We compare pairs, but with a family-wise correction so the three comparisons together still risk only 5%.

Formula. Tukey’s HSD compares each pair’s mean difference xˉaxˉb\bar{x}_a - \bar{x}_b against a jointly-adjusted critical distance; equivalently it reports an adjusted pp-value for each pair that you compare to α\alpha directly.

Computation. In R the follow-up is TukeyHSD() on the same aov() fit — the workplace-standard idiom. It reports, for each pair, the difference in means, a family-wise-adjusted confidence interval, and an adjusted pp-value (p adj).

air <- read.csv("data/processed/kern_airquality.csv")
bak <- subset(air, pollutant == "PM2.5" & city == "Bakersfield")
bak$site_name <- factor(bak$site_name)   # treat site_name as labeled groups, not text

aov_fit <- aov(daily_mean ~ site_name, data = bak)
TukeyHSD(aov_fit)

Interpretation. Only one pair clears the bar after adjustment: California Avenue vs. Golden / M Street, with a mean difference of about 1.76 µg/m³ higher at Golden, adjusted p0.013p \approx 0.013 (dataset-derived from kern_airquality). The Airport site is statistically indistinguishable from either of the other two. So the omnibus “they differ” traces to a single contrast: Golden/M Street runs dustier than California Avenue. Notice the discipline — we only earned the right to run these pairwise tests because the overall ANOVA was significant first.


712.5 Worked example 3 — Crop yields (clear-cut signal, simulated data)

The dataset kern_crops_sim is simulated — its name carries the *_sim tag so you always know it is made-up (data/codebooks/kern_crops_sim.md). The magnitudes are plausible for Kern agriculture, but they are not measured values. We use it here to show what a large FF looks like.

Intuition. Compare yield per acre for almonds, pistachios, and oranges. Almonds and pistachios both yield about a ton per acre; oranges, a citrus fruit, yield many tons per acre. The group averages are miles apart compared with the scatter within each crop. Expect a huge FF.

Formula. Same machinery: F=MSB/MSWF = \text{MSB}/\text{MSW} on (k1,Nk)=(2,24)(k-1, N-k) = (2, 24) df, with k=3k = 3 crops and N=27N = 27 crop-years.

Computation.

crops <- read.csv("data/processed/kern_crops_sim.csv")
three <- subset(crops, commodity %in% c("ALMONDS", "PISTACHIOS", "ORANGES"))
three$commodity <- factor(three$commodity)

fit3 <- aov(yield_per_acre ~ commodity, data = three)
anova(fit3)

# effect size eta-squared = SSB / SST:
ss3 <- anova(fit3)[["Sum Sq"]]
round(c(eta_squared = ss3[1] / sum(ss3)), 4)

Interpretation. F591.9F \approx 591.9 with p<0.0001p < 0.0001 and η20.98\eta^2 \approx 0.98 (dataset-derived from kern_crops_sim): crop type explains about 98% of the variation in yield per acre. We reject H0H_0 overwhelmingly — unsurprising, because the groups barely overlap. Contrast this with the Bakersfield air example (F=4.07F = 4.07, η2=0.0094\eta^2 = 0.0094): both are “statistically significant,” yet one effect is enormous and the other is hairline. The pp-value tells you whether there is a difference; the effect size η2\eta^2 tells you how much it matters. Always report both.


812.6 Worked example 4 — A pencil-and-paper ANOVA table

Intuition. On an exam you will be handed a partial ANOVA table and asked to fill it in and decide. No data, no computer — just the definitions.

Setup. Three training programs, n=5n = 5 trainees each (N=15N = 15). You are told SSB=48\text{SSB} = 48 and SSW=72\text{SSW} = 72. Test at α=0.05\alpha = 0.05.

Formula + computation. With k=3k = 3 groups:

dfbetween=k1=2,dfwithin=Nk=12.df_{\text{between}} = k - 1 = 2, \qquad df_{\text{within}} = N - k = 12.
MSB=SSBdfbetween=482=24,MSW=SSWdfwithin=7212=6.\text{MSB} = \frac{\text{SSB}}{df_{\text{between}}} = \frac{48}{2} = 24, \qquad \text{MSW} = \frac{\text{SSW}}{df_{\text{within}}} = \frac{72}{12} = 6.
F=MSBMSW=246=4.0.F = \frac{\text{MSB}}{\text{MSW}} = \frac{24}{6} = 4.0.

The completed table:

SourceSSdfMSFF
Between482244.0
Within72126
Total12014

Decision. Compare F=4.0F = 4.0 to the critical value F0.05,2,12F_{0.05,\,2,\,12}:

qf(0.95, df1 = 2, df2 = 12)      # critical F
pf(4.0, df1 = 2, df2 = 12, lower.tail = FALSE)   # p-value

The critical value is F0.05,2,12=3.89F_{0.05,2,12} = 3.89 and the pp-value is 0.0467. Since 4.0>3.894.0 > 3.89 (equivalently p=0.0467<0.05p = 0.0467 < 0.05), we reject H0H_0: at least one training program’s mean differs. (These two values come from the FF-distribution, not from a dataset.)

Interpretation. This is the skeleton every ANOVA shares — fill the table, compare to the critical FF (or compare pp to α\alpha), state a conclusion in context. Practice it until the four boxes (SSB/SSW \rightarrow MSB/MSW \rightarrow FF \rightarrow decision) are automatic.


912.7 Checking the conditions

ANOVA’s FF-test is trustworthy when three conditions hold. You check all three yourself — the boxplot and the group standard deviations from favstats() make the last two quick.

  1. Independence. Observations are independent within and across groups. This comes from the study design, not from the data — a random sample or a randomized experiment earns it. (Daily air readings at one monitor are mildly autocorrelated day-to-day; we treat the monitor-days as approximately independent for teaching, and flag it honestly.)

  2. Approximate normality within each group — or large njn_j, in which case the Central Limit Theorem (Chapter 6) covers the group means. With hundreds of days per monitor we are safe.

  3. Roughly equal variances (homogeneity). Compare max(SD)/min(SD)\max(\text{SD}) / \min(\text{SD}) across the groups (the favstats() sd column, as in Worked Example 1); a ratio under 2 is the rule of thumb. For the Bakersfield monitors it was about 1.04 — fine.


10Try it


11Practice problems

Work these with mosaic and base R. Odd-numbered problems have short answers in the appendix (Appendix: Odd Answers); full worked solutions live in the instructor key. Datasets load with read.csv("data/processed/<name>.csv").

  1. In one or two sentences, state the null and alternative hypotheses for a one-way ANOVA comparing the mean PM2.5 of k=5k = 5 Kern monitors.

  2. A study compares k=4k = 4 groups with N=40N = 40 observations total. Give dfbetweendf_{\text{between}} and dfwithindf_{\text{within}}.

  3. An ANOVA reports SSB=90\text{SSB} = 90 and SSW=210\text{SSW} = 210. Compute η2\eta^2 and interpret it in one sentence.

  4. True or false, with a reason: “A significant ANOVA means every pair of group means is different.”

  5. Complete the ANOVA table: k=3k = 3, N=18N = 18, SSB=60\text{SSB} = 60, SSW=90\text{SSW} = 90. Find both dfdf, both MSMS, and FF.

  6. For the table in Problem 5, find the critical value F0.05,2,15F_{0.05,\,2,\,15} with qf() and state the decision.

  7. You run pairwise tt-tests on all pairs among k=5k = 5 groups at α=0.05\alpha = 0.05. How many comparisons is that, and what is the approximate family-wise error rate 1(10.05)m1 - (1-0.05)^m?

  8. Explain in your own words why ANOVA uses within-group variation as its yardstick for judging between-group variation.

  9. Load kern_airquality, keep PM2.5 rows for the three Bakersfield monitors, and reproduce the omnibus ANOVA with aov() + anova(). Report FF, the two dfdf, and pp.

  10. For the Problem 9 fit, report η2\eta^2 and write one sentence distinguishing statistical significance from practical importance here.

  11. Using kern_airquality PM2.5 rows, run a one-way ANOVA of daily_mean across all monitors with at least 100 days (site_name). Report FF and pp.

  12. Make a boxplot of daily_mean by site_name for the Problem 11 group with gf_boxplot(). In one sentence, does the picture agree with the test?

  13. A four-group ANOVA gives MSB=30\text{MSB} = 30 and MSW=12\text{MSW} = 12. Compute FF. If df=(3,36)df = (3, 36), is it significant at α=0.05\alpha = 0.05? (Use pf().)

  14. State the three ANOVA conditions and name the one that comes from study design rather than from the data.

  15. The SD-ratio (max ÷ min group SD) in an ANOVA is 3.1. Which condition is threatened, and what alternative test handles it?

  16. Load kern_crops_sim, keep ALMONDS, PISTACHIOS, and GRAPES, WINE, and run aov() + anova() on yield_per_acre (compute η2\eta^2 from the table). Report FF and η2\eta^2. (Remember: simulated data.)

  17. Why is it invalid to ANOVA almond yield (tons/acre) against cotton yield (bales/acre)? Answer in one sentence.

  18. A significant ANOVA is followed by Tukey HSD; exactly one of three pairs has adjusted p<0.05p < 0.05. Write the one-sentence conclusion.

  19. Compute the Bonferroni per-test threshold for all pairwise comparisons among k=6k = 6 groups at family-wise α=0.05\alpha = 0.05.

  20. Explain why ANOVA’s FF-test is one-sided (right tail only), referring to what a small FF near 1 means.

  21. For kern_airquality PM2.5, run a Tukey HSD on the three Bakersfield monitors (aov() + TukeyHSD()). Which single pair is significant after adjustment?

  22. An ANOVA has SST=500\text{SST} = 500 and SSB=25\text{SSB} = 25. Compute η2\eta^2 and comment on whether grouping explains much.

  23. Two analysts test the same 5-group data. One runs ANOVA; the other runs all 10 pairwise tt-tests at α=0.05\alpha = 0.05 and reports the one “significant” pair. Whose error rate is controlled, and why?

  24. Given k=3k = 3, N=30N = 30, and an observed F=2.10F = 2.10, find the pp-value with pf() and state the decision at α=0.05\alpha = 0.05.

  25. In one paragraph for a non-technical city official, summarize the Bakersfield PM2.5 ANOVA finding (§12.3–12.4): what differs, by how much, and how much it matters.


12Chapter summary

13FAQ

Q1. Why not just run a tt-test on every pair? Each test carries its own false-alarm risk, and the chance of at least one false alarm grows as 1(1α)m1-(1-\alpha)^m. ANOVA asks one question at one error rate.

Q2. What does a big FF actually mean? The gaps between group averages are large compared with the ordinary scatter inside the groups — too large to credibly blame on chance.

Q3. ANOVA was significant. Which groups differ? The omnibus test does not say. Run a family-wise-corrected pairwise procedure (Tukey HSD or Bonferroni) after a significant result to locate the difference.

Q4. Is a tiny pp-value the same as a big effect? No. The Bakersfield air example is significant (p=0.0175p = 0.0175) but η2<0.01\eta^2 < 0.01 — a real-but-tiny effect. Always report the effect size with the pp-value.

Q5. My groups have very different spreads. Is ANOVA still valid? If the SD ratio (max ÷ min) exceeds about 2, the equal-variance condition is shaky; switch to a Welch ANOVA (oneway.test(..., var.equal = FALSE)).

Q6. Why are there two degrees of freedom? The FF-distribution is a ratio of two variance estimates, each with its own df: k1k-1 for the numerator (between) and NkN-k for the denominator (within). Both are needed to find the critical value or pp-value.

Q7. Can ANOVA compare just two groups? Yes — and it gives exactly the (equal-variance) two-sample tt-test result, with F=t2F = t^2. ANOVA earns its keep when there are three or more groups.

Q8. Does a significant ANOVA prove the grouping caused the difference? No. Causation requires a randomized experiment. With observational data (like the air monitors) ANOVA establishes a difference, not its cause.

14Resumen en español