3. Inference for simple linear regression

In 2.2 Least squares from first principles we fit a line to the Toluca Company’s 25 production runs and got a slope of $b_1 = 3.5702$ hours per unit of lot size. The engineers were pleased, and then they asked the questions any careful person asks about a number: how sure are you? A different 25 runs would have given a slightly different slope. Could the true slope really be zero, meaning lot size does not affect hours at all and we are fooling ourselves? And the practical one, the reason they collected the data: if we schedule a new run of 100 units next week, how many hours should we budget, and how far off might that be?

Chapter 2 gave point estimates: single best guesses for the intercept, slope, and error variance. A point estimate with no measure of its uncertainty is half an answer. This chapter supplies the other half. We attach margins of error to the slope and intercept, test whether the slope is really different from zero, and, most importantly for planning, build two very different intervals around a prediction. One says where the average run of a given size lands. The other says where a single new run lands. Figure 1 shows both drawn around the Toluca line, and the gap between them is the single most useful idea in the chapter.

Everything here rests on one added assumption that Chapter 2 introduced but did not yet use for inference: the errors are normally distributed (2.6 Maximum likelihood under normal errors). With that assumption the sampling distributions sketched in 2.5 Sampling behavior and the Gauss-Markov theorem become exact $t$ and $F$ distributions, and honest intervals and tests follow.

3.1 The sampling distributions of $b_0$ and $b_1$¶

Intuition¶

We already know from 2.5 Sampling behavior and the Gauss-Markov theorem that the slope estimate $b_1$ is unbiased, with variance $\sigma^2/S_{xx}$, where $S_{xx} = \sum_i (X_i - \bar{X})^2$ is the spread of the predictor values about their mean $\bar{X}$. That tells us where $b_1$ centers and how much it spreads, but not the shape of its distribution. To build an interval we need the shape. The trick is that $b_1$ is a weighted sum of the responses, and a weighted sum of independent normal random variables is itself normal. So the moment we assume normal errors, $b_1$ is exactly normal, and we can standardize it the way intro statistics standardizes a sample mean.

There is one wrinkle. Standardizing needs the true error standard deviation $\sigma$, which we do not know. We replace it with the estimate $s = \sqrt{\mathrm{MSE}}$. Swapping a known constant for a noisy estimate adds a little extra uncertainty, and that extra uncertainty is exactly what turns the normal distribution into a $t$ distribution. Figure 2 confirms this: standardize 5000 simulated slopes by their own estimated standard errors, and the pile matches a $t$ curve, with slightly heavier tails than the normal.

Formula¶

Under the normal error model $Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i$ with $\varepsilon_i \overset{\text{iid}}{\sim} N(0, \sigma^2)$, the estimators are exactly normal:

In words: each estimator is centered on the parameter it estimates, with the variance we derived in Chapter 2, and now with a normal shape. To standardize either one we need the spread of the estimate. The estimated standard error of the slope (Definition 3.1) supplies it.

The estimated standard error $s\{b_1\}$ is our best guess at how much $b_1$ bounces from one sample to the next. Standardizing $b_1$ with it gives a $t$ statistic, the result stated next.

In words: subtract the true value and divide by the estimated standard error, and the result follows a $t$ distribution with $n-2$ degrees of freedom. The degrees of freedom are $n-2$ because that is the divisor in $\mathrm{MSE}$, the two lost degrees paying for $b_0$ and $b_1$ (2.4 Estimating the error variance). This single $t$ result powers every interval and test in the rest of the chapter.

That last sentence is worth a picture. Almost everything ahead is one idea, the $t$ statistic above, pointed at a different estimate each time. Figure 3 is the map of the chapter on one page: keep it in mind and each new section becomes “the same move, new target.”

Derivation¶

Derivation (the $t$ statistic for the slope). We assemble the result from three facts.

Normality. From 2.5 Sampling behavior and the Gauss-Markov theorem, $b_1 = \sum_i k_i Y_i$ with $k_i = (X_i - \bar{X})/S_{xx}$. Under the normal error model the $Y_i$ are independent normal variables, and any linear combination of independent normals is normal. With the mean and variance already derived, $b_1 \sim N(\beta_1, \sigma^2/S_{xx})$. Dividing by the true standard deviation gives a standard normal:

A chi-square, independent of the slope. It can be shown that

and that this quantity is statistically independent of $b_1$ (indeed of the whole vector $(b_0, b_1)$). The clean proof uses the hat matrix and a theorem on quadratic forms in normal vectors; we give it in full in 7.3 The hat matrix, once we have the matrix tools. For now we take these two facts on faith, and note that the simulation in Figure 2 is a direct check that the end result is right.

Combine into a $t$. A standard normal divided by the square root of an independent chi-square over its degrees of freedom is, by definition, a $t$ random variable:

The middle step just multiplies top and bottom out: the numerator is $Z$, and $s\{b_1\}/(\sigma/\sqrt{S_{xx}}) = \sqrt{\mathrm{MSE}/\sigma^2} = \sqrt{\chi^2_{n-2}/(n-2)}$. The identical argument with the weights $m_i$ from 2.5 Sampling behavior and the Gauss-Markov theorem gives the $t$ statistic for $b_0$. $\blacksquare$

The setup code below refits the Toluca model and recovers the standard errors we will use throughout.

toluca <- read.csv("data/toluca.csv")
fit <- lm(hours ~ lotsize, data = toluca)
n <- nrow(toluca)
x <- toluca$lotsize
y <- toluca$hours
Sxx <- sum((x - mean(x))^2)
MSE <- sum(residuals(fit)^2) / (n - 2)
round(c(n = n, Sxx = Sxx, MSE = MSE, s = sqrt(MSE),
        se_b1 = sqrt(MSE / Sxx)), 4)

         n        Sxx        MSE          s      se_b1 
   25.0000 19800.0000  2383.7156    48.8233     0.3470 

import numpy as np
import pandas as pd
import statsmodels.formula.api as smf

toluca = pd.read_csv("data/toluca.csv")
fit = smf.ols("hours ~ lotsize", data=toluca).fit()
n = len(toluca)
x = toluca["lotsize"].to_numpy()
y = toluca["hours"].to_numpy()
Sxx = np.sum((x - x.mean()) ** 2)
MSE = np.sum(fit.resid ** 2) / (n - 2)
print(round(MSE, 4), round(np.sqrt(MSE), 4), round(np.sqrt(MSE / Sxx), 4))

2383.7156 48.8233 0.347

The estimated standard error of the slope is $s\{b_1\} = 0.347$ hours per unit, the same number Chapter 2 computed from the variance formula. It is the yardstick for everything that follows.

3.2 Inference for the slope¶

Intuition¶

The slope is usually the number a study is about: does lot size affect hours, and by how much? Two questions come in a pair. A confidence interval (Definition 3.3) answers “how much?” by giving a range of plausible values for $\beta_1$. A hypothesis test answers “is there any effect at all?” by asking whether zero is a plausible value. Both are built from the one $t$ statistic of 3.1 The sampling distributions of $b_0$ and $b_1$, so they always agree: a 95% interval excludes zero exactly when the two-sided test rejects at the 5% level.

Formula¶

In words: take the estimate and reach out a multiplier’s worth of standard errors on each side. To test $H_0: \beta_1 = 0$ against $H_a: \beta_1 \ne 0$, form the test statistic

and reject $H_0$ when $|t^{\ast}| > t_{1-\alpha/2,\,n-2}$, or equivalently when the two-sided p-value $2\,P(t_{n-2} > |t^{\ast}|)$ is below $\alpha$. In words: if the estimate sits many standard errors away from zero, zero is not credible. Nothing here is special to zero; to test $H_0: \beta_1 = c$ for any value $c$, put $c$ in place of the 0.

The 95% in “95% confident” is a promise about repeated samples, not about the one interval sitting in front of you, and that is a hard thing to take on faith. The widget below draws the samples so you can watch the promise being kept, and occasionally broken.

3.3 Inference for the intercept¶

The intercept is where the fitted line crosses the vertical axis: the model’s predicted hours for a lot of zero units. It is the same kind of estimate as the slope, so it gets the exact same machinery. Follow the roadmap in Figure 3 one box over, and only the standard error changes.

The intercept gets the same treatment, with its own standard error $s\{b_0\} = \sqrt{\mathrm{MSE}\,(1/n + \bar{X}^2/S_{xx})}$. The confidence interval is $b_0 \pm t_{1-\alpha/2,\,n-2}\, s\{b_0\}$, and the test of $H_0: \beta_0 = 0$ uses $t^{\ast} = b_0/s\{b_0\}$.

The reason the intercept comes second is that it is usually less interesting and often not interpretable. For Toluca, $\beta_0$ is the mean hours at a lot size of zero, an extrapolation far outside the data (2.2 Least squares from first principles), so a test of $\beta_0 = 0$ answers a question nobody asked. The interval is still worth computing, if only to see how loosely the intercept is pinned down.

confint(fit, "(Intercept)", level = 0.95)

               2.5 %  97.5 %
(Intercept) 8.213711 116.518

print(fit.conf_int(alpha=0.05).loc["Intercept"])

0      8.213711
1    116.518006
Name: Intercept, dtype: float64

The 95% interval for the intercept runs from about 8 to 117 hours, a span of over 100 hours. That enormous width is the price of extrapolating to $X = 0$, which sits 70 units below the smallest observed lot. Compare it with the slope’s tight interval: the slope is anchored by data on both sides, the intercept is not.

3.4 Confidence interval for a mean response¶

Intuition¶

Now we move from the coefficients to a prediction. Ask: across all runs of lot size $X_h = 65$, what is the average number of hours? That average is the point on the true line, $E\{Y_h\} = \beta_0 + \beta_1 X_h$, and our estimate of it is the fitted value $\hat{Y}_h = b_0 + b_1 X_h$. Because $b_0$ and $b_1$ wobble from sample to sample, so does $\hat{Y}_h$, and we want an interval for the true mean.

The key feature is that the interval’s width depends on where $X_h$ sits. Near the center $\bar{X}$ the fitted line is pinned down tightly, so the interval is narrow. Far from $\bar{X}$ a small tilt in the slope swings the line up or down a lot, so the interval widens. This is the pinch we saw in the bundle of fitted lines in Figure 15, now made into a formula.

Formula¶

The estimated variance of the fitted mean at $X_h$ is

and the confidence interval for the mean response follows.

In words: the further $X_h$ is from the average predictor value $\bar{X}$, the larger the second term inside the parentheses, and the wider the interval. At $X_h = \bar{X}$ the second term vanishes and the interval is at its narrowest, width set only by $1/n$.

Derivation¶

Derivation (variance of the fitted mean). Write the fitted value in its pivoted form, using $b_0 = \bar{Y} - b_1 \bar{X}$:

The two pieces $\bar{Y}$ and $b_1$ are uncorrelated. To see it, recall $b_1 = \sum k_i Y_i$ and $\bar{Y} = \sum \tfrac{1}{n} Y_i$, so, using independence of the $Y_i$ and $\sum k_i = 0$,

Because the two pieces are uncorrelated, their variances add:

Replacing $\sigma^2$ by $\mathrm{MSE}$ gives $s^2\{\hat{Y}_h\}$. The $t$ interval then follows from the same standardize-and-divide argument as in 3.1 The sampling distributions of $b_0$ and $b_1$, since $(\hat{Y}_h - E\{Y_h\})/s\{\hat{Y}_h\} \sim t_{n-2}$. $\blacksquare$

Figure 5 plots $s\{\hat{Y}_h\}$ against $X_h$: a curve that dips to its minimum at $\bar{X} = 70$ and rises symmetrically on both sides. That curve is what gives the confidence band in Figure 1 its gentle bow-tie shape.

3.5 Prediction interval for a new observation¶

Intuition¶

Here is the distinction that trips up more practitioners than any other. The confidence interval of 3.4 Confidence interval for a mean response is about the average run of size $X_h$. But the engineers do not schedule an average run; they schedule one specific run next week, and they want to know where its hours will land. A single run scatters around the mean by its own random error $\varepsilon$, and no amount of data removes that scatter. So predicting a single new observation carries two sources of uncertainty: our uncertainty about where the line is (the same $s^2\{\hat{Y}_h\}$ as before), plus the new run’s own irreducible variance $\sigma^2$. The interval that accounts for both is the prediction interval (Definition 3.6), and it is always wider than the confidence interval.

The whole choice comes down to one question, and Figure 6 lays it out. Ask what you are bounding: the average outcome over many cases at $X_h$, or the outcome of one specific new case. The left answer is a confidence interval; the right answer is a prediction interval. Decide the question first, and the formula follows.

Formula¶

For a new observation $Y_{h(\text{new})}$ at $X_h$, the estimated variance of the prediction error is $s^2\{\text{pred}\} = \mathrm{MSE}(1 + 1/n + (X_h - \bar{X})^2/S_{xx})$, and the prediction interval follows.

In words: this is the mean-response variance with an extra $\mathrm{MSE}$ (the leading 1) added for the new run’s own scatter. That extra term is usually the dominant one, which is why the prediction interval is much wider and why, unlike the confidence interval, it never shrinks to zero even with infinite data.

Derivation¶

Derivation (variance of a prediction). The quantity we bound is the prediction error $Y_{h(\text{new})} - \hat{Y}_h$, the gap between the actual new run and our forecast. The new run is $Y_{h(\text{new})} = \beta_0 + \beta_1 X_h + \varepsilon_{\text{new}}$, and its error $\varepsilon_{\text{new}}$ is independent of the past data that produced $\hat{Y}_h$. Two independent pieces, so their variances add:

Replacing $\sigma^2$ by $\mathrm{MSE}$ gives $s^2\{\text{pred}\}$, and standardizing the prediction error by $s\{\text{pred}\}$ again yields a $t_{n-2}$ distribution. The $t$ interval follows. $\blacksquare$

The gap between the two intervals is not a modeling choice; it is a fact about the two questions, and each interval keeps its own 95% promise. We can check this by simulation, which is the surest way to see that a prediction interval really does trap a new observation 95% of the time. Figure 7 runs the experiment.

set.seed(4210)
b0_true <- 62.366; b1_true <- 3.5702; sigma <- 48.82
xh <- 100
mu_h <- b0_true + b1_true * xh
tcrit <- qt(0.975, n - 2)
ci_hits <- 0; pi_hits <- 0; N <- 5000
for (i in 1:N) {
  ysim <- b0_true + b1_true * x + rnorm(n, 0, sigma)
  f <- lm(ysim ~ x)
  pr <- predict(f, data.frame(x = xh), se.fit = TRUE)
  s_mean <- pr$se.fit
  s_pred <- sqrt(pr$se.fit^2 + sum(residuals(f)^2) / (n - 2))
  if (abs(pr$fit - mu_h) <= tcrit * s_mean) ci_hits <- ci_hits + 1
  ynew <- mu_h + rnorm(1, 0, sigma)
  if (abs(pr$fit - ynew) <= tcrit * s_pred) pi_hits <- pi_hits + 1
}
round(c(CI_covers_mean = ci_hits / N, PI_covers_new = pi_hits / N), 4)

CI_covers_mean  PI_covers_new 
        0.9496         0.9458 

from scipy import stats

rng = np.random.default_rng(4210)
b0_true, b1_true, sigma = 62.366, 3.5702, 48.82
xh = 100.0
mu_h = b0_true + b1_true * xh
tcrit = stats.t.ppf(0.975, n - 2)
ci_hits = pi_hits = 0
N = 5000
for _ in range(N):
    ysim = b0_true + b1_true * x + rng.normal(0, sigma, size=n)
    b1s = np.sum((x - x.mean()) * (ysim - ysim.mean())) / Sxx
    b0s = ysim.mean() - b1s * x.mean()
    yhat = b0s + b1s * xh
    mse = np.sum((ysim - (b0s + b1s * x)) ** 2) / (n - 2)
    s_mean = np.sqrt(mse * (1 / n + (xh - x.mean()) ** 2 / Sxx))
    s_pred = np.sqrt(mse * (1 + 1 / n + (xh - x.mean()) ** 2 / Sxx))
    if abs(yhat - mu_h) <= tcrit * s_mean:
        ci_hits += 1
    ynew = mu_h + rng.normal(0, sigma)
    if abs(yhat - ynew) <= tcrit * s_pred:
        pi_hits += 1
print(round(ci_hits / N, 4), round(pi_hits / N, 4))

0.9492 0.9544

Across 5000 simulated datasets, the confidence interval covers the true mean about 95% of the time and the prediction interval covers a fresh new run about 95% of the time. Each interval is honest about its own question. The prediction interval earns its extra width; it is not being cautious for the sake of caution.

The gap between the two intervals is easier to believe once you can open and close it yourself. The widget below puts the sample size and the prediction location under your finger, so you can find out which band answers to more data and which one does not.

3.6 The analysis of variance approach¶

Intuition¶

There is a second, equivalent way to see the same inference, and it scales up beautifully to the multiple regression of later chapters. The idea is bookkeeping for variation. The response values $Y_i$ vary; some of that variation the line explains, and some it leaves as residual scatter. If we can split the total variation cleanly into an “explained” part and a “leftover” part, we can compare them, and a big explained part relative to leftover is evidence that the slope is real.

Figure 9 shows the split for a single run: the total deviation from the mean, $Y_i - \bar{Y}$, breaks into the part the line accounts for, $\hat{Y}_i - \bar{Y}$, plus the residual, $Y_i - \hat{Y}_i$. Remarkably, when you square these and sum over all runs, the cross term vanishes and the totals split just as cleanly.

Formula¶

Define the three sums of squares:

In words: the total variation of $Y$ about its mean (SSTO) equals the variation the regression explains (SSR, the spread of the fitted values about the mean) plus the variation it does not (SSE, the error sum of squares from 2.4 Estimating the error variance). Each sum has degrees of freedom: $n-1$ for SSTO, 1 for SSR (one predictor), and $n-2$ for SSE. The degrees of freedom add the same way, $n-1 = 1 + (n-2)$. Dividing each sum of squares by its degrees of freedom gives a mean square (Definition 3.8).

In words: a mean square is a sum of squares averaged over its degrees of freedom, so it is a variance-like number. MSR and MSE are the two we will put in a ratio to test the slope.

Derivation¶

Derivation ($\mathrm{SSTO} = \mathrm{SSR} + \mathrm{SSE}$). Split each total deviation by routing it through the fitted value:

Square both sides and sum over $i$:

The cross term is zero. Write $Y_i - \hat{Y}_i = e_i$ and expand:

Both pieces vanish by the residual identities of 2.3 Fitted values and the properties of residuals: $\sum \hat{Y}_i e_i = 0$ (fitted values orthogonal to residuals) and $\sum e_i = 0$. So the cross term is zero and $\mathrm{SSTO} = \mathrm{SSR} + \mathrm{SSE}$. The residual identities we proved in Chapter 2 are doing the work again. $\blacksquare$

Derivation (expected mean squares). From 2.4 Estimating the error variance we already have $E\{\mathrm{MSE}\} = \sigma^2$. A similar expectation calculation, using $\mathrm{SSR} = b_1^2 S_{xx}$ and $E\{b_1^2\} = \operatorname{Var}\{b_1\} + \beta_1^2 = \sigma^2/S_{xx} + \beta_1^2$, gives

In words: MSE always estimates $\sigma^2$, while MSR estimates $\sigma^2$ plus a term that is positive exactly when $\beta_1 \ne 0$. So if the slope is zero, MSR and MSE estimate the same thing and their ratio should be near 1; if the slope is nonzero, MSR is inflated and the ratio is large. That ratio is the $F$ test. $\blacksquare$

The ANOVA table collects these quantities in a standard layout. This is the anchor later chapters extend to extra sums of squares (8.3 Extra sums of squares).

3.7 The F test and its equivalence to the t test¶

Intuition¶

The ANOVA table hands us a test almost for free. Under $H_0: \beta_1 = 0$, both MSR and MSE estimate the same $\sigma^2$, so their ratio $F^{\ast} = \mathrm{MSR}/\mathrm{MSE}$ hovers near 1. Under $H_a: \beta_1 \ne 0$, MSR is inflated by $\beta_1^2 S_{xx}$ and the ratio grows. A large $F^{\ast}$ is evidence against the null.

But we already tested $\beta_1 = 0$ with a $t$ test in 3.2 Inference for the slope and got $t^{\ast} = 10.29$. Do the two tests agree? They do, exactly and always: in simple linear regression the $F$ statistic is the square of the $t$ statistic. This is worth proving, because it shows the ANOVA $F$ test is not a new test, just the $t$ test wearing different clothes.

Formula¶

The $F$ test of $H_0: \beta_1 = 0$ uses

and rejects $H_0$ when $F^{\ast} > F_{1-\alpha;\,1,\,n-2}$. The claimed identity is

where $t^{\ast} = b_1/s\{b_1\}$ is the slope $t$ statistic. In words: the $F$ statistic and the squared $t$ statistic are the same number, and the tests give identical p-values.

Derivation¶

Derivation ($F^{\ast} = (t^{\ast})^2$). Start from the regression sum of squares. Since $\hat{Y}_i - \bar{Y} = b_1 (X_i - \bar{X})$,

With one regression degree of freedom, $\mathrm{MSR} = \mathrm{SSR} = b_1^2 S_{xx}$. Therefore

Now write the $t$ statistic and square it, using $s^2\{b_1\} = \mathrm{MSE}/S_{xx}$:

The two expressions are literally the same. Distributionally this matches a standard fact: the square of a $t_{n-2}$ random variable is an $F_{1,\,n-2}$ random variable, so squaring the slope $t$ statistic lands on the ANOVA $F$ statistic, and the critical values match too, $F_{1-\alpha;\,1,n-2} = (t_{1-\alpha/2,\,n-2})^2$. $\blacksquare$

Figure 10 confirms the distributional half: square 200,000 draws from a $t_{23}$ and the histogram lands on the $F_{1,23}$ density, with the 5% critical value 4.28 equal to 2.069².

The algebra above is three lines long, but the identity is far more convincing when you get to break the data yourself and the two numbers still refuse to disagree.

3.8 R-squared and its interpretation¶

Intuition¶

The ANOVA split invites a natural summary: what fraction of the total variation did the line explain? That fraction is $R^2$, the coefficient of determination (Definition 3.12). It runs from 0 (the line explains nothing beyond the mean) to 1 (the line passes through every point). It is the single most reported and most misread number in regression, so we will state plainly what it does and does not mean.

Formula¶

In words: $R^2$ is the proportion of the total variation in $Y$ that the regression on $X$ accounts for. In simple linear regression it also equals $r^2$, the square of the Pearson correlation between $X$ and $Y$, a link Chapter 4 develops (4.2 Correlation and the regression slope).

Figure 12 shows the two ingredients: the total scatter of hours about their mean (SSTO, left) shrinks to the much smaller residual scatter about the line (SSE, right). $R^2$ is how much of the vertical spread the line removed.

3.9 A confidence band for the whole line¶

Intuition¶

The confidence interval of 3.4 Confidence interval for a mean response is honest about one lot size you name in advance. But often you want to draw a band around the entire fitted line and claim, with 95% confidence, that the true line lies inside it everywhere at once. If you simply string together the pointwise intervals, the simultaneous confidence is less than 95%, because you are making many claims and some will fail by chance. The Working-Hotelling band (Definition 3.13) fixes this by using a slightly larger multiplier, so the whole-line claim holds at the stated level.

Formula¶

In words: the band has the same center and the same standard error $s\{\hat{Y}_h\}$ as the pointwise interval, but the multiplier $W$ replaces $t_{1-\alpha/2,\,n-2}$. Because $W > t$, the band is a little wider everywhere, and that extra width buys simultaneous coverage of the true line at all $X_h$ at once. The multiplier uses $F$ with 2 numerator degrees of freedom because the line has two parameters, $\beta_0$ and $\beta_1$, that can both vary.

W <- sqrt(2 * qf(0.95, 2, n - 2))
round(c(W = W, t = qt(0.975, n - 2)), 4)

     W      t 
2.6162 2.0687 

W = np.sqrt(2 * stats.f.ppf(0.95, 2, n - 2))
print(round(W, 4), round(stats.t.ppf(0.975, n - 2), 4))

2.6162 2.0687

For Toluca the whole-line multiplier is $W = 2.62$ against the pointwise $t = 2.07$, so the simultaneous band is about 26% wider than a single-point interval. Figure 13 draws both.

3.10 A prediction-focused example: advertising and sales¶

To see the machinery on a fresh dataset, turn to the advertising data: 200 markets’ TV advertising budgets and the resulting product sales (thousands of units). A manager wants to predict sales for a new market that will receive a TV budget of 150 thousand dollars, and to understand the average effect of the budget.

Figure 14 draws both bands. Notice the prediction band comfortably contains almost all 200 markets, while the confidence band, being about the mean, does not.

3.11 Chapter summary¶

You can now do inference for a simple linear regression, not just fit it. You saw why the standardized slope and intercept follow a $t$ distribution with $n-2$ degrees of freedom, and used it to build confidence intervals and hypothesis tests for both coefficients (the Toluca slope, $t^{\ast} = 10.29$, 95% CI $(2.85, 4.29)$). You learned to tell a confidence interval for a mean response from a prediction interval for a single new observation, to derive both variance formulas, and, above all, to say which one a question needs: for a 100-unit run the mean interval is $(390, 449)$ but the single-run prediction interval is $(314, 525)$. You derived the ANOVA decomposition, built the table, ran the $F$ test ($F^{\ast} = 105.9$), and proved it is the slope $t$ test squared. You can read $R^2 = 0.82$ correctly and widen a pointwise interval into a Working-Hotelling band ($W = 2.62$) when you need the whole line.

Key results at a glance.

Key terms. Estimated standard error, confidence interval for the slope, hypothesis test for the slope, confidence interval for a mean response, prediction interval, analysis of variance, total sum of squares, regression sum of squares, mean square, $F$ test, coefficient of determination, Working-Hotelling confidence band, degrees of freedom.

You should now be able to:

• Derive the sampling distributions of $b_0$ and $b_1$ and explain why inference uses $t_{n-2}$.

• Construct and interpret confidence intervals and hypothesis tests for the slope and intercept from software output.

• Distinguish a confidence interval for a mean response from a prediction interval for a new observation, derive both, and choose the right one.

• Derive the ANOVA decomposition $\mathrm{SSTO} = \mathrm{SSR} + \mathrm{SSE}$, build the ANOVA table, and run the $F$ test for the slope.

• Prove that $F^{\ast} = (t^{\ast})^2$ in simple linear regression.

• Interpret $R^2$ correctly, and name what it does and does not measure.

• Construct a Working-Hotelling confidence band for the whole regression line.

Where this fits. In the workflow spine of The modeling workflow (ASK, EXPLORE, FIT, CHECK, USE), this chapter is the heart of the USE stage: it turns a fitted line into decisions and honest forecasts with stated uncertainty. Chapter 2 gave the point estimates; here we attached intervals and tests. The prediction interval (3.5 Prediction interval for a new observation) returns in Chapter 12 when we validate models on held-out data and in Chapter 15 when we forecast, where we will confront why time-ordered data makes prediction intervals harder. The ANOVA table (3.6 The analysis of variance approach) and the $F$ test (3.7 The F test and its equivalence to the t test) generalize in Chapter 8 into the extra-sums-of-squares machinery (8.3 Extra sums of squares) and the general linear test (8.4 The general linear test), which can compare whole models at once. And the tiny slope p-value here is the benchmark Chapter 5 checks against a permutation test that assumes no normality at all (5.2 The permutation test for the slope), asking whether our $t$-based conclusion survives without the normal error model.

3.12 Frequently asked questions¶

Q1. What is the one-sentence difference between a confidence interval and a prediction interval? A confidence interval is for the mean response at $X_h$, averaged over all cases at that predictor value; a prediction interval is for a single new case, and is wider because it must also cover that case’s own random scatter $\sigma^2$.

Q2. Why does the interval widen as $X_h$ moves away from $\bar{X}$? Because the variance of the fitted mean carries a $(X_h - \bar{X})^2/S_{xx}$ term. Away from the center, a small change in the estimated slope pivots the line a large vertical distance, so the fitted value there is less certain. Every fitted line passes through $(\bar{X}, \bar{Y})$, so the uncertainty is smallest there.

Q3. Why use $t$ and not the normal distribution? Because we estimate $\sigma$ with $s = \sqrt{\mathrm{MSE}}$ rather than knowing it. That extra estimation makes the reference distribution $t_{n-2}$, with heavier tails than the normal. For large $n$ the two nearly coincide, but for $n = 25$ the difference matters.

Q4. The $F$ test and the $t$ test gave the same p-value. Do I need both? In simple linear regression, no: $F^{\ast} = (t^{\ast})^2$, so they are one test. Report the $t$ test for a single slope because it keeps the direction (sign) of the effect. The $F$ test earns its keep in multiple regression, where it can test several coefficients simultaneously.

Q5. My $R^2$ is high. Does that mean my model is good? Not by itself. A high $R^2$ can accompany a curved relationship, non-constant variance, or influential outliers, all of which a residual plot would reveal. $R^2$ measures variation explained, not correctness, not prediction accuracy, and not causation. Read it alongside $s$ and diagnostic plots (Chapter 9).

Q6. Can I use these intervals to predict hours for a lot of 500 units? No. That is extrapolation far beyond the data range (20 to 120 units). The formulas will happily produce an interval, but its stated confidence assumes the straight-line model holds out there, which the data cannot support. Trust intervals only inside, or just barely outside, the observed range of $X$.

Q7. What exactly does “95% confident” mean for the slope interval? It means the procedure traps the true slope in 95% of repeated samples, as Figure 15 shows with 100 simulated intervals. For your one interval $(2.85, 4.29)$, the true slope is either in it or not; the 95% describes the long-run reliability of the method, not a probability about this particular interval.

3.13 Practice problems¶

1. (A) In one sentence each, state what a confidence interval for the mean response and a prediction interval for a new observation are about, and say which is wider and why.

2. (A) The slope’s 95% confidence interval is $(2.85, 4.29)$. Explain what “95% confident” means here, and what it does not mean.

3. (A) Explain why the confidence interval for a mean response is narrowest at $X_h = \bar{X}$ and widens on both sides.

4. (A) Why does inference for the slope use a $t$ distribution with $n-2$ degrees of freedom rather than a standard normal? Where do the $n-2$ degrees of freedom come from?

5. (A) A report states “$R^2 = 0.82$, so the model is correct.” Give two distinct reasons this conclusion does not follow.

6. (A) The intercept’s 95% interval $(8.21, 116.52)$ is enormously wide compared with the slope’s. Explain why, referring to where $X = 0$ sits relative to the data.

7. (A) In simple linear regression the ANOVA $F$ test and the slope $t$ test always agree. State the exact relationship between the two statistics and one reason to still report the $t$ test.

8. (A) A colleague uses a 95% confidence interval for the mean response to budget a single upcoming production run. Explain why this understates the risk, and which interval they should use.

9. (B) Derive the $t$ statistic $(b_1 - \beta_1)/s\{b_1\} \sim t_{n-2}$ (Theorem 3.2), stating clearly which facts you take from Chapter 2, which you take on faith until Chapter 7, and why the combination is a $t$.

10. (B) Derive $\operatorname{Var}\{\hat{Y}_h\} = \sigma^2(1/n + (X_h - \bar{X})^2/S_{xx})$ (Theorem 3.5), including the step that $\bar{Y}$ and $b_1$ are uncorrelated.

11. (B) Derive the prediction-error variance $\sigma^2(1 + 1/n + (X_h - \bar{X})^2/S_{xx})$ (Theorem 3.7), explaining where the leading 1 comes from and why the new error is independent of $\hat{Y}_h$.

12. (B) Prove the ANOVA identity $\mathrm{SSTO} = \mathrm{SSR} + \mathrm{SSE}$ (Theorem 3.9) by showing the cross term vanishes, naming the residual identities you use.

13. (B) Prove that $F^{\ast} = (t^{\ast})^2$ in simple linear regression (Theorem 3.11), starting from $\mathrm{SSR} = b_1^2 S_{xx}$.

14. (B) Show that $E\{\mathrm{MSR}\} = \sigma^2 + \beta_1^2 S_{xx}$ (Theorem 3.10), and explain why this makes $\mathrm{MSR}/\mathrm{MSE}$ a sensible test statistic for $H_0: \beta_1 = 0$.

15. (B) Show that a $100(1-\alpha)\%$ confidence interval for $\beta_1$ excludes 0 exactly when the two-sided $t$ test of $H_0: \beta_1 = 0$ rejects at level $\alpha$.

16. (B) Starting from $R^2 = 1 - \mathrm{SSE}/\mathrm{SSTO}$ and $\mathrm{SSE} = S_{yy} - b_1^2 S_{xx}$, show that $R^2 = b_1^2 S_{xx}/S_{yy}$, and hence that $R^2 = r^2$ where $r = S_{xy}/\sqrt{S_{xx}S_{yy}}$.

17. (B) The Working-Hotelling multiplier is $W = \sqrt{2 F_{1-\alpha;2,n-2}}$ and the pointwise multiplier is $t_{1-\alpha/2,n-2}$. Explain, without a full proof, why $W$ must exceed $t$ and what the extra width buys.

18. (B) Show that the prediction interval half-width never shrinks below $t\, s$ even as $n \to \infty$, while the confidence-interval half-width shrinks to 0. What does this say about the limits of prediction?

19. (C) Fit the Toluca model in R or Python. Reproduce the slope’s $t$ statistic, its p-value, and its 95% confidence interval, and confirm they match the values in this chapter.

20. (C) Compute a 95% confidence interval for the mean hours at $X_h = 40$ units and at $X_h = 90$ units. Which is wider, and does the difference match the $(X_h - \bar{X})^2$ rule?

21. (C) Compute both the 95% confidence interval for the mean and the 95% prediction interval for a new run at $X_h = 50$ units. Report the ratio of their widths and explain it.

22. (C) Build the ANOVA table for Toluca in software. From it, extract $\mathrm{SSR}$, $\mathrm{SSE}$, $\mathrm{MSR}$, $\mathrm{MSE}$, and $F^{\ast}$, and verify $F^{\ast} = (t^{\ast})^2$ using the slope $t$ from problem 19.

23. (C) Using the advertising data, fit sales ~ TV, test the slope, and give a 95% prediction interval for a new market with a TV budget of 200. Interpret the interval for a manager.

24. (C) Write a seeded simulation (seed 4210, 2000 datasets from the fitted Toluca model at the observed lot sizes) that checks the empirical coverage of the 95% confidence interval for the mean at $X_h = 60$. Report the coverage and compare it with 0.95.

25. (C) Reproduce the Working-Hotelling multiplier $W$ and the pointwise $t$ multiplier for Toluca. Then compute both the pointwise 95% interval and the Working-Hotelling band half-width at $X_h = 120$, and report the ratio.

26. (C) Regress hours ~ lotsize and make a plot of the fitted line with the 95% confidence band and the 95% prediction band (as in Figure 1). Describe how the two bands differ in width and shape.

27. (B) A student claims that because $F^{\ast} = 105.9$ is “huge,” the model must predict individual runs accurately. Explain the confusion between the $F$ test (about the slope) and prediction accuracy (about $s$ and the prediction interval), using the Toluca numbers.

28. (C) Using the advertising data, compute $R^2$ two ways: from the ANOVA sums of squares and as the square of the correlation cor(adv$TV, adv$sales). Confirm they match, and interpret the value.

3.14 Exam practice¶

The problems above build skills one at a time. Exams ask you to combine them and, above all, to explain in full sentences: to read software output in context, to judge a claim, and to say what would change under a new condition. The five problems below are written in that exam voice. Work each one as if for full marks, in complete sentences, before opening the model answer. A bare number or a one-word verdict earns little credit on the real thing; the reasoning is the answer.

EP 3.1 (interpret output in context). A production scheduler has one specific new run of 80 units on next week’s calendar. Software fits the Toluca model and returns two intervals at a lot size of 80.

predict(fit, data.frame(lotsize = 80), interval = "confidence")
predict(fit, data.frame(lotsize = 80), interval = "prediction")

      fit      lwr      upr
1 347.982 326.5449 369.4191
      fit      lwr      upr
1 347.982 244.7333 451.2307

pr = fit.get_prediction(pd.DataFrame({"lotsize": [80]}))
print("mean CI ", pr.conf_int(alpha=0.05))
print("pred PI ", pr.conf_int(obs=True, alpha=0.05))

mean CI  [[326.54493825 369.41910215]]
pred PI  [[244.73334785 451.23069256]]

Interpret each interval in one sentence, with units and in the context of this problem. Then the scheduler proposes to budget next week’s single run at “between 327 and 369 hours, 95% sure,” using the first interval. Explain in full sentences whether that is the right interval to use, which one the scheduler should use instead, and why the two intervals differ by so much.

EP 3.2 (a student claims X, evaluate). A classmate writes: “Under the normal-error model the slope estimate $b_1$ is exactly normal, so a 95% confidence interval for the slope should use the standard normal multiplier 1.96.” The chapter instead uses $t_{0.975,\,23} = 2.069$. Explain in full sentences what is right and what is wrong in the classmate’s reasoning, what specific step forces the switch from 1.96 to 2.069, and what happens to the gap between the two multipliers as the sample size grows.

EP 3.3 (interpret output in context, evaluate a claim). The advertising data record TV budgets and sales for $n = 200$ markets. Fitting sales ~ TV gives the output below.

summary(fit_adv)$coefficients
anova(fit_adv)
c(R2 = summary(fit_adv)$r.squared, s = summary(fit_adv)$sigma)

             Estimate Std. Error  t value Pr(>|t|)
(Intercept)  7.032594 0.45784294 15.36028        0
TV           0.047537 0.00269061 17.66763        0
           Df Sum Sq Mean Sq F value    Pr(>F)
TV          1 3314.6  3314.6  312.14 < 2.2e-16
Residuals 198 2102.5    10.6
        R2          s 
    0.6119     3.2587 

print(sm.stats.anova_lm(fit_adv))
print(round(fit_adv.rsquared, 4), round(np.sqrt(fit_adv.scale), 4))

             df       sum_sq      mean_sq           F        PR(>F)
TV          1.0  3314.618167  3314.618167  312.144994  1.467390e-42
Residual  198.0  2102.530583    10.618841           NaN           NaN
0.6119 3.2587

First, verify from the numbers that the $F$ statistic equals the square of the TV slope’s $t$ statistic, and explain in one sentence why that identity must hold here. Then a manager reads $F = 312$ and $R^2 = 0.61$ and concludes, “the model is so significant that it will predict any one market’s sales to within a unit or two.” Explain in full sentences what the large $F$ actually establishes, what $R^2 = 0.61$ means, and which number in the output governs how far a single new market can fall from the line.

EP 3.4 (what would change if). Suppose the Toluca engineers gathered far more production data at the same spread of lot sizes, so that the sample size $n$ grows very large while $S_{xx}$ grows in proportion and $\mathrm{MSE}$ stays near 2384. Explain in full sentences what happens, as $n \to \infty$, to (a) the width of the 95% confidence interval for the mean hours at $X_h = 100$, and (b) the width of the 95% prediction interval for a single new 100-unit run. Say why the two behave differently, and what that difference reveals about the limits of prediction.

EP 3.5 (interpret a figure, explain why). Figure 5 plots two estimated-standard-error curves against lot size: the mean-response standard error $s\{\hat{Y}_h\}$ (blue) dips to a minimum near the mean lot size of 70 and rises toward both ends, while the prediction standard error $s\{\text{pred}\}$ (yellow) stays nearly flat near 49 hours across the whole range and sits above the blue curve everywhere. Explain in full sentences (a) why the blue curve is U-shaped with its lowest point at $\bar{X}$, and (b) why the yellow curve barely bends even though it lies above the blue one at every lot size.

3.15 Chapter game¶