8. Multiple regression in practice

We return to the Dwaine Studios cities of Chapters 6 and 7 (7.1 The model and least squares in matrix form), where a chain of portrait studios in mid-sized towns wants to decide, for a city it has never entered, how much a new studio will sell. Two recorded numbers about each of its 21 cities plausibly drive that answer: the target population targtpop (thousands of persons under 16, the pool of potential family-portrait customers) and the disposable income dispoinc (thousands of dollars per person, how much those families have to spend), together predicting annual sales (thousands of dollars).

In Chapter 6 we built this data into matrices, and in Chapter 7 we solved the normal equations $\mathbf{X}'\mathbf{X}\mathbf{b} = \mathbf{X}'\mathbf{Y}$ to get the least-squares coefficients $\mathbf{b} = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{Y} = (-68.86,\ 1.45,\ 9.37)$; see 7.1 The model and least squares in matrix form. That was the FIT stage. This chapter is about what comes after: reading those three numbers honestly, deciding which predictors are earning their place, and knowing the traps that wait for anyone who reports a coefficient without thinking. Figure 1 shows the raw material, sales against each predictor one at a time.

Both pictures slope up, so both predictors look useful. But the two predictors also move together (richer cities tend to have different age profiles), and that overlap is the whole difficulty of multiple regression. A coefficient in a multiple regression is not the slope you would see in a one-predictor scatterplot. It answers a more careful question, and this chapter is about asking that question correctly, testing whether a predictor belongs, and seeing the answer in a picture.

8.1 Reading the coefficients of a multiple regression¶

Intuition¶

The fit handed us three numbers. This section is about what each one honestly means, because the obvious reading is the wrong one. The short version: a number in a multiple regression answers a more careful question than the same-looking number in a one-predictor plot.

The Dwaine fit is $\widehat{\text{sales}} = -68.86 + 1.45\,\text{targtpop} + 9.37\,\text{dispoinc}$. It is tempting to read the 9.37 the same way we read a simple-regression slope: “raise income by one unit and sales go up 9.37.” That reading is wrong in a specific and important way. In a one-predictor world, moving income means moving whatever else travels with income. In the multiple regression, the coefficient on income has already accounted for target population. It measures what income adds after target population has said everything it can say.

The clean phrase is: a multiple regression coefficient is a partial slope (Definition 8.1). It is the change in the mean response for a one-unit change in that predictor, holding the other predictors fixed. “Holding fixed” is what makes it different from the marginal picture in Figure 1, where nothing is held fixed and every predictor drifts along with every other.

Formula¶

The multiple linear regression model with $p-1$ predictors is stated in Definition 8.2.

Term by term:

• $Y_i$ is the response for case $i$ (Dwaine sales for city $i$).

• $X_{ik}$ is the $k$-th predictor for case $i$.

• $\beta_k$ is the partial slope for predictor $k$: the change in $E\{Y\}$ per one-unit rise in $X_k$ with every other predictor held fixed.

• $\beta_0$ is the intercept, the mean response when all predictors equal zero.

• $p$ is the number of regression parameters including the intercept; here $p = 3$. With $n$ cases, the error variance is estimated with divisor $n - p$, that is $\mathrm{MSE} = \mathrm{SSE}/(n-p)$, exactly the estimator of 7.5 Estimating the error variance.

In words: each $\beta_k$ is the effect of its own predictor after the other predictors in the model have been given their say. Change what else is in the model and you change the question, so you can change the coefficient.

R¶

The fit itself is one line; the reading is the hard part. We start from the same 21 cities.

dwaine <- read.csv("data/dwaine.csv")
fit <- lm(sales ~ targtpop + dispoinc, data = dwaine)
round(coef(fit), 4)

(Intercept)    targtpop    dispoinc
   -68.8571      1.4546      9.3655

The fitted model is no longer a line but a plane sitting above the (targtpop, dispoinc) floor, as in Figure 2. Each coefficient is the tilt of that plane in one direction. The partial slope for income is how steeply the plane rises as you walk in the income direction while keeping your population coordinate fixed.

8.2 The limits of “holding the others fixed”¶

Intuition¶

“Holding the other predictors fixed” is the correct interpretation, and it is also where careful people get into trouble. The phrase is a statement about the arithmetic of the fitted plane: it is always true that the coefficient is the plane’s tilt in one direction with the other coordinates pinned. The trouble is that the sentence quietly promises something the data may not support, that you could actually find, or create, two cities identical in every predictor but one. When predictors move together, that promise is fiction.

Look again at the Dwaine predictors. Target population and disposable income have a correlation of 0.78. Cities with many young families are systematically different in income from cities with few. So “two cities with the same target population but different income” describes a thinner slice of reality than it sounds, and “hold population fixed and push income far” can walk you clean off the cloud of cities that ever existed. The coefficient is still computable and still meaningful as a description of the fitted surface. What weakens is the causal daydream that you could intervene on one predictor and leave the others undisturbed.

The mechanism: a coefficient depends on its companions¶

A coefficient is not a property of a predictor. It is a property of a predictor and the company it keeps. Add or remove another predictor and the coefficient can move, sometimes a little, sometimes enough to change sign. Figure 3 shows the Dwaine income story both ways: the steep marginal slope when income stands alone, and the gentler partial slope once population is accounted for.

When “holding fixed” describes empty space¶

The deeper limit is geometric. Holding population fixed and varying income traces a line across the (targtpop, dispoinc) floor. If the data cloud is a tilted ellipse, because the two predictors are correlated, then most of that line lies outside the cloud. The model will still return a fitted value there, but it is extrapolating into a region where no city has ever been observed, and no data constrain the answer. We return to this in 8.7 Two dangers: hidden extrapolation and lurking variables with a picture; for now, hold the warning: a coefficient can be perfectly estimated and still describe a comparison you can never actually make.

The widget below does the holding fixed for you, one population level at a time, so you can watch the gentle partial slope and the steep marginal slope stop looking like rivals.

8.3 Extra sums of squares¶

Intuition¶

We need a way to ask “how much does a predictor add?” that does not depend on staring at a coefficient. The natural measure is how much unexplained variation a predictor removes when it joins a model that already has the others. Fit the model without the predictor and record its error sum of squares; add the predictor and record the new, smaller error sum of squares; the drop is the predictor’s extra sum of squares (Definition 8.3). It is the variation that only this predictor could mop up, once the others have done their part.

Two facts make this work. First, adding a predictor can never make the error sum of squares larger (we prove this below), so the drop is never negative. Second, the total variation $\mathrm{SSTO}$ in $Y$ does not depend on the model, so a smaller $\mathrm{SSE}$ means a larger $\mathrm{SSR}$: the extra sum of squares is variation moved from the “unexplained” column to the “explained” column.

A picture makes the “only this predictor could mop up” idea concrete. Think of the response’s variation as a target, and each predictor as a circle that covers the part it can explain. When two predictors are correlated, their circles overlap, and the shared piece can be credited to only one of them: whichever entered the model first. The extra sum of squares of the second predictor is just its own crescent, the part of its circle that the first predictor’s circle did not already cover. Figure 5 draws this for the body-fat predictors we work with next.

Formula¶

For a model already containing a set of predictors, write $\mathrm{SSE}(\cdot)$ for its error sum of squares. The extra sum of squares of $X_2$ given $X_1$ is defined in Definition 8.3.

• $\mathrm{SSE}(X_1)$ is the error sum of squares of the model with $X_1$ only.

• $\mathrm{SSE}(X_1, X_2)$ is the error sum of squares once $X_2$ is added.

• The two right-hand expressions are equal because $\mathrm{SSTO} = \mathrm{SSR} + \mathrm{SSE}$ is the same total for both models (3.6 The analysis of variance approach), so any drop in $\mathrm{SSE}$ is an equal rise in $\mathrm{SSR}$.

In words: $\mathrm{SSR}(X_2 \mid X_1)$ is the extra explained variation you buy by adding $X_2$ to a model that already has $X_1$. The vertical bar reads “given” or “adjusted for.”

Extra sums of squares chain into a decomposition of the total regression sum of squares. Adding predictors one at a time,

which is exactly what the sequential (Type I) ANOVA table prints, one row per predictor in the order you list them. Because the order is a choice, so is the split: reorder the predictors and the pieces change, though their total $\mathrm{SSR}$ does not.

Derivation (adding a predictor cannot increase SSE)¶

Proof. Let the reduced model use the predictors in a set $R$ and the full model use $R$ together with one more predictor, so the full model’s columns span a larger space. By 7.2 The geometry of least squares, the fitted vector is the orthogonal projection of $\mathbf{Y}$ onto the column space of the design matrix, and $\mathrm{SSE}$ is the squared distance from $\mathbf{Y}$ to that space. Every fitted vector the reduced model can produce is also available to the full model (set the extra coefficient to zero), so the full model’s column space contains the reduced model’s. Projecting onto a larger subspace lands at least as close to $\mathbf{Y}$:

Therefore $\mathrm{SSR}(X_2 \mid X_1) = \mathrm{SSE}(X_1) - \mathrm{SSE}(X_1, X_2) \ge 0$, with equality only when the added predictor buys nothing (its fitted contribution is already in the reduced space). $\blacksquare$

R and Python¶

The classic worked example is not Dwaine but the ALRM body-fat data: 20 men, percent body fat measured by underwater weighing, against three tape-measure predictors, triceps skinfold, thigh circumference, and midarm circumference. It is built for this section because the predictors overlap heavily, so order matters a lot.

Figure 6 draws the decomposition as a stacked bar: the total variation in body fat split into the sequential pieces plus the leftover error.

8.4 The general linear test¶

Intuition¶

Extra sums of squares measure how much a predictor adds; now we ask whether that addition is more than noise would produce. The tool is a comparison of two models, one reduced (without the terms in question) and one full (with them). If the extra terms are worthless, the full model’s error sum of squares will be only trivially smaller than the reduced model’s, a drop no bigger than random scatter would give. If the extra terms matter, the drop will be large relative to the noise level. The general linear test (Definition 8.5) turns “large relative to noise” into a single $F$ statistic.

Formula¶

Let the full model have $\mathrm{SSE}(F)$ on $df_F = n - p_F$ degrees of freedom and the reduced model $\mathrm{SSE}(R)$ on $df_R = n - p_R$, where the reduced model is the full model with some coefficients forced to zero. The general linear test statistic and its null distribution are stated in Theorem 8.6.

• $\mathrm{SSE}(R) - \mathrm{SSE}(F)$ is the extra sum of squares explained by the terms the reduced model dropped: variation the full model can account for and the reduced model cannot.

• $df_R - df_F = p_F - p_R$ is the number of coefficients set to zero under the reduced model, call it $q$.

• The denominator $\mathrm{SSE}(F)/df_F = \mathrm{MSE}(F)$ is the full model’s estimate of $\sigma^2$.

In words: $F^{*}$ is the extra variation explained per dropped coefficient, measured in units of the noise variance. Under the reduced model (the dropped coefficients really are zero) with normal errors, $F^{*} \sim F(q,\ n - p_F)$, and a large $F^{*}$ is evidence against the reduced model. We reject the reduced model when $F^{*}$ exceeds the upper $F$ critical value, equivalently when its $p$-value is small.

Derivation (the test statistic and its distribution)¶

Proof. Write $\widehat{\mathbf{Y}}_F$ and $\widehat{\mathbf{Y}}_R$ for the fitted vectors of the full and reduced models. By 7.2 The geometry of least squares each is the orthogonal projection of $\mathbf{Y}$ onto a column space, and the reduced space is a subspace of the full space. Split the reduced model’s residual through the full fit:

The full residual is orthogonal to the entire full column space, and $\widehat{\mathbf{Y}}_F - \widehat{\mathbf{Y}}_R$ lies inside that space, so the two pieces are perpendicular. Pythagoras gives

so the numerator sum of squares $\mathrm{SSE}(R) - \mathrm{SSE}(F) = \lVert \widehat{\mathbf{Y}}_F - \widehat{\mathbf{Y}}_R\rVert^2$ is the squared length of the extra-fit vector, which lives in a subspace of dimension $q = p_F - p_R$. That is where its $q$ degrees of freedom come from. Figure 7 draws this right triangle. $\blacksquare$

The distributional claim needs one more layer. Under the reduced model with normal errors, the two perpendicular pieces are independent, and each squared length divided by $\sigma^2$ is a chi-square: $\mathrm{SSE}(F)/\sigma^2 \sim \chi^2_{n - p_F}$ and $(\mathrm{SSE}(R) - \mathrm{SSE}(F))/\sigma^2 \sim \chi^2_{q}$. Dividing each chi-square by its degrees of freedom and taking the ratio gives an $F(q,\ n-p_F)$ variable. The independence and chi-square facts rest on the normal-theory geometry of projections built in 7.3 The hat matrix. The full argument uses Cochran’s theorem on quadratic forms, which is beyond the matrix algebra this book proves. So we take those two distributional facts on the authority of that geometry, and verify the whole machine numerically below. The ALRM text, Chapter 7, gives the quadratic-form details for the reader who wants them.

Tests on a subset of coefficients¶

The savings data give a clean subset test, and they carry the second running thread of this chapter. In Chapter 4 you used these 50 countries to study correlation and its traps (4.2 Correlation and the regression slope); we now fit them as a multiple regression. The response sr is the aggregate personal savings rate, and the predictors are the percent of population under 15 (pop15), the percent over 75 (pop75), per-capita disposable income (dpi), and its growth rate (ddpi). Life-cycle savings theory says the two dependency measures, pop75 and dpi, might not matter once age structure and growth are in the model. That is a hypothesis about two coefficients at once, and the general linear test is built for it.

One coefficient: the test is exactly $t^2$¶

When the reduced model drops a single coefficient ($q = 1$), the general linear test must agree with the $t$-test that software already prints for that coefficient. It does, exactly.

Proof. From 7.4 The sampling distribution of b the estimated variance of $b_k$ is $s^2\{b_k\} = \mathrm{MSE}\cdot c_{kk}$, where $c_{kk}$ is the $k$-th diagonal entry of $(\mathbf{X}'\mathbf{X})^{-1}$. The extra sum of squares for dropping the single predictor $X_k$ works out to $\mathrm{SSR}(X_k \mid \text{others}) = b_k^2 / c_{kk}$. Substituting both into the general linear test with $q = 1$,

So the subset test with one dropped coefficient is the square of the ordinary $t$ statistic, and an $F(1, n-p)$ variable is the square of a $t_{n-p}$ variable. The two tests are one test. $\blacksquare$

Before you trust any single row of a sequential table, promote and demote the predictors yourself and watch $F^{*}$ re-decide whether the newest one earned its place.

8.5 Added-variable plots¶

Intuition¶

A coefficient is a single number; sometimes you want to see the partial relationship it summarizes. The obstacle is that you cannot simply plot $Y$ against $X_k$, because that picture mixes in every other predictor. The added-variable plot (Definition 8.8), also called a partial regression plot, removes the other predictors from both axes first, then plots what is left.

Here is the recipe. To see the partial effect of $X_k$: regress $Y$ on all the other predictors and keep the residuals $e_Y$, the part of $Y$ the others do not explain. Regress $X_k$ on all the other predictors and keep the residuals $e_X$, the part of $X_k$ the others do not explain. Now plot $e_Y$ against $e_X$. Both axes have been purged of the other predictors, so the plot shows the relationship between $Y$ and $X_k$ that is genuinely new. Its slope is exactly the multiple regression coefficient $b_k$, a fact we prove below, so the plot is a picture of the very number in the coefficient table.

The one rule people break is forgetting to clean the horizontal axis: they residualize $Y$ but then plot it against the raw predictor. Figure 9 lays out the three steps so both axes get the same treatment.

Formula¶

For predictor $X_k$ in a model whose other predictors are collected as $X_{(-k)}$,

and the added-variable plot is the scatter of $e_Y$ against $e_X$. The least-squares slope of that scatter equals $b_k$, the full-model coefficient on $X_k$.

• $e_Y$ is the response with the other predictors’ explanation removed.

• $e_X$ is predictor $k$ with the other predictors’ explanation removed, its unique part.

• The slope of $e_Y$ on $e_X$ is $b_k$; the scatter around the line shows how cleanly $X_k$’s unique part predicts $Y$’s unmatched part.

Derivation (the added-variable slope equals $b_k$)¶

Proof (Frisch-Waugh, two-predictor case). Take the full model $Y = b_0 + b_1 X_1 + b_2 X_2 + e$ and let $X_1$ play the role of “the other predictor” while we build the added-variable plot for $X_2$. Split $X_2$ into its projection onto $\{\mathbf{1}, X_1\}$ and the leftover: $X_2 = \widehat{X}_2 + e_X$, where $e_X \perp \{\mathbf{1}, X_1\}$ by construction. Substitute into the fitted equation and group the terms that live in $\operatorname{span}\{\mathbf{1}, X_1\}$:

Now form $e_Y$ by removing the $\{\mathbf{1}, X_1\}$ part of $Y$, that is, projecting $Y$ onto the orthogonal complement of $\operatorname{span}\{\mathbf{1}, X_1\}$. The first group vanishes under this projection. The term $e_X$ is already orthogonal to $\{\mathbf{1}, X_1\}$, so it survives untouched. The full residual $e$ is orthogonal to $\{\mathbf{1}, X_1, X_2\}$, hence to $\{\mathbf{1}, X_1\}$, so it survives too. Therefore

Finally regress $e_Y$ on $e_X$ through the origin (both residual vectors have mean zero). The slope is

because $\langle e, e_X\rangle = 0$: the full residual $e$ is orthogonal to $X_2$ and to $\{\mathbf{1}, X_1\}$, hence to $e_X = X_2 - \widehat{X}_2$. The added-variable slope is exactly the multiple regression coefficient. The same argument works with any number of other predictors in place of $X_1$. $\blacksquare$

R and Python¶

An added-variable plot can also reveal the opposite, a predictor with almost nothing left to add. Figure 11 shows the plot for growth rate ddpi in the savings model, whose partial slope is a positive but gentle 0.41.

Rather than take the recipe on faith, pull the cleaning slider below and watch the marginal slope turn into the coefficient in front of you.

8.6 $R^2$, adjusted $R^2$, and standardized coefficients¶

Intuition¶

We can now test which predictors belong; the next job is to summarize a whole model in one number and compare models of different sizes fairly. $R^2$ is the fraction of the response’s variation the model explains, and it has one seductive flaw: it never goes down when you add a predictor, even a column of random noise. That makes it useless for comparing models of different sizes, because the biggest model always wins. Adjusted $R^2$ (Definition 8.10) fixes this by charging rent for each predictor: it goes up only if the new predictor explains more than an average predictor’s worth of noise would. Standardized coefficients (Definition 8.11) solve a different problem: raw coefficients live in different units (dollars, thousands of people, percents), so you cannot compare their sizes; putting everything in standard-deviation units makes the comparison fair.

Formula¶

• $\mathrm{SSTO} = \sum (Y_i - \bar Y)^2$ is the total variation in $Y$, fixed regardless of the model.

• $R^2$ compares raw sums of squares; $R^2_a$ compares them per degree of freedom, so adding a parameter (raising $p$) is penalized through the factor $(n-1)/(n-p)$.

• A standardized coefficient is $b_k^{*} = b_k \cdot (s_{X_k}/s_Y)$, the coefficient you get by fitting the regression on variables first rescaled to have standard deviation one.

In words: $R^2$ measures explained fraction, $R^2_a$ measures explained fraction after paying for model size, and $b_k^{*}$ measures effect size in standard deviations, so different predictors can be ranked on one scale.

Derivation ($R^2$ never falls; $R^2_a$ can)¶

Proof. By the monotonicity result of 8.3 Extra sums of squares (Theorem 8.4), adding a predictor cannot raise $\mathrm{SSE}$, and $\mathrm{SSTO}$ does not depend on the model. Hence $\mathrm{SSE}/\mathrm{SSTO}$ cannot rise, so $R^2 = 1 - \mathrm{SSE}/\mathrm{SSTO}$ cannot fall: every added predictor leaves $R^2$ equal or higher, which is why $R^2$ alone cannot choose a model. Adjusted $R^2$ behaves differently because it uses $\mathrm{MSE} = \mathrm{SSE}/(n-p)$. Adding a useless predictor lowers $\mathrm{SSE}$ by a little but also lowers the divisor $n-p$ by one; if the shrinkage in $\mathrm{SSE}$ is smaller than the shrinkage in $n-p$ demands, $\mathrm{MSE}$ rises and $R^2_a$ falls. So $R^2_a$ can go down when a predictor does not earn its degree of freedom, which is exactly the behavior we want from a model-size-aware measure. $\blacksquare$

Figure 13 makes this concrete: starting from the real savings model, we add columns of pure random noise one at a time and watch $R^2$ climb while $R^2_a$ sags.

R and Python¶

Figure 14 ranks the four standardized coefficients on one axis, which the raw coefficients, living in incompatible units, could never share.

8.7 Two dangers: hidden extrapolation and lurking variables¶

Intuition¶

Two failures deserve their own section because they are invisible in the coefficient table and fatal to a conclusion. The first is hidden extrapolation (Definition 8.13): predicting at a combination of predictor values that is inside the range of each predictor separately but outside the region where they occur together. The second is the lurking variable (Definition 8.14): an omitted predictor that, if included, would move or reverse the coefficients you reported.

Hidden extrapolation¶

With one predictor, extrapolation is easy to police: stay inside the observed range of $X$. With several predictors, the observed region is not a box but a cloud, usually a tilted one because predictors are correlated. A query point can sit inside every predictor’s own range and still fall far outside the cloud. Figure 15 shows the Dwaine case: a city with target population 75 and disposable income 16 is unremarkable on each axis alone, yet no real city combines a population that large with income that low, so a prediction there is an extrapolation the one-variable ranges would never flag.

The practical check is to ask whether a query point lies within the joint region of the data, not merely within each margin. Chapter 9 makes this rigorous with the hat matrix: the leverage value $h_{ii}$ of 7.3 The hat matrix measures how far a point sits from the center of the predictor cloud, and a new point whose leverage value would exceed the observed ones is a hidden extrapolation.

Lurking variables¶

A regression can only adjust for the predictors you give it. If an important variable is missing and it correlates with both the response and an included predictor, the coefficient you report absorbs the missing variable’s influence and can be badly wrong, even reversed. This is the multiple regression version of the correlation-is-not-causation warning from 4.2 Correlation and the regression slope, and it is why an observational coefficient is always provisional: some unmeasured factor might be doing the work.

“Even reversed” sounds like an exaggeration until you see it. Figure 16 shows the mechanism on a schematic dataset. Ignore the lurking variable and the points march up from left to right, so the slope looks positive. Color the points by the hidden group and each group tips the other way: within any single group the slope is negative. A model that never saw the group reports the upward trend and gets the sign exactly backwards.

The honest posture is to name the lurking variables you can think of and admit the ones you cannot. In the Dwaine study, local competition, the age of each studio, and regional advertising are all plausibly related to both sales and the city characteristics; none is in the model, so each partial slope is “the best we can say from these three columns,” not a final truth. No amount of arithmetic inside a fixed set of predictors can rule out a variable you never measured.

8.8 Chapter summary¶

You can now take a fitted multiple regression and do the work that matters after the fit. You can read each coefficient as a partial slope, stating in words the controlled comparison it makes and the exact phrase “holding the other predictors fixed,” and you can explain why that phrase weakens when predictors are correlated and can describe a region with no data. You can compute extra sums of squares and read a sequential ANOVA table as a decomposition of $\mathrm{SSR}$. You can run the general linear test for one coefficient or a group, derive its $F$ statistic from the Pythagorean split of nested models, and show it collapses to $t^2$ for a single coefficient. You can build an added-variable plot and prove its slope is the multiple regression coefficient. You can compute $R^2$, adjusted $R^2$, and standardized coefficients, and explain why only the last two are safe for comparing models or effects. And you can spot the two dangers that a coefficient table hides: hidden extrapolation in the joint predictor space and the lurking variable no model ever measured.

Key results at a glance

Key terms

partial slope, multiple linear regression model, extra sum of squares, sequential (Type I) ANOVA, general linear test, full model, reduced model, added-variable plot, coefficient of determination ($R^2$), adjusted $R^2$, standardized coefficient, hidden extrapolation, lurking variable, multicollinearity.

You should now be able to

• Read each coefficient as a partial slope, with its units and the phrase “holding the other predictors fixed.”

• Explain why “holding fixed” weakens under correlated predictors and can describe a region with no data.

• Compute an extra sum of squares and read a sequential ANOVA table as a decomposition of $\mathrm{SSR}$.

• Run the general linear test for one coefficient or a group, and show it reduces to $F^{*} = t^2$ when $q = 1$.

• Build an added-variable plot and prove its slope equals the multiple regression coefficient.

• Compute and contrast $R^2$, adjusted $R^2$, and standardized coefficients, and say why $R^2$ never falls when a predictor is added.

• Spot hidden extrapolation in the joint predictor space and explain how a lurking variable can flip a coefficient’s sign.

Where this fits. In the workflow spine of The modeling workflow this chapter is mostly USE: it takes the fit that Chapter 7 produced for Dwaine in matrix form and turns it into interpretation, tests, and honest caveats, with the general linear test also serving FIT by deciding which terms belong. Chapter 7 gave the estimator and its variance; this chapter gave the reasoning that makes an estimate a claim. What we deferred is the CHECK stage: whether any single case is quietly running the fit, and whether the model’s assumptions hold at all. That is Chapter 9’s job. It returns to the savings data, singling out Libya as the case that bends the fit and Zambia as a large-residual outlier, and builds the influence diagnostics (9.3 Influence: which points actually change the fit) and the leverage statistic that 9.1 Leverage values and the hat matrix points to, which together decide how much of this chapter’s confidence survives contact with real data. The general linear test you learned here is the testing engine that Chapter 11 reuses for categorical predictors and interactions (11.1 From categories to numbers: indicator coding) and Chapter 12 for variable selection (12.2 Choosing among models: selection criteria), neither re-deriving it.

8.9 Frequently asked questions¶

Q1. Why is the multiple regression coefficient on income so different from the simple-regression slope? Because they answer different questions. The simple slope 31.17 compares cities that differ in income and lets population differ along with it; the partial slope 9.37 compares cities that differ in income but share population. When predictors are correlated, the two questions have different answers, and only the partial slope earns the phrase “holding population fixed.”

Q2. Does “holding the other predictors fixed” mean I can intervene on one predictor and expect that coefficient’s change? No. The coefficient describes a comparison across cases that already differ, not the effect of an action. With observational data, and especially with correlated predictors or a lurking variable, acting on a predictor need not reproduce the observed association. The coefficient is a description of the fitted surface, not a promise about interventions.

Q3. The sequential ANOVA gives different sums of squares when I reorder the predictors. Which order is right? None is uniquely right; the order is a choice about what “given the others” means. The first predictor gets its full simple-regression sum of squares, each later one gets only what it adds beyond those before it. The pieces always total the same $\mathrm{SSR}$. If you want each predictor adjusted for all the others at once, use its $t$-test (or the general linear test dropping just that predictor), which does not depend on order.

Q4. When should I use the general linear test instead of just reading $t$ statistics? Use it whenever you want to test several coefficients at once, such as “do these two predictors together add anything.” A group of individually non-significant $t$’s does not tell you whether the group as a whole matters, because the predictors may share their explanatory power; the general linear test asks the joint question directly. For a single coefficient, the general linear test and the $t$-test are identical ($F^{*} = t^2$).

Q5. Why does $R^2$ always go up when I add a predictor, even a useless one? Because adding a predictor can only lower (or not change) $\mathrm{SSE}$, and $\mathrm{SSTO}$ is fixed, so $R^2 = 1 - \mathrm{SSE}/\mathrm{SSTO}$ can only rise. That is exactly why $R^2$ cannot be used to choose between models of different sizes; use adjusted $R^2$, which penalizes each added parameter, or a formal test.

Q6. My added-variable plot slope matches the coefficient exactly. Is that a coincidence? No, it is a theorem (Frisch-Waugh). Removing the other predictors from both the response and the target predictor, then regressing the leftovers, reproduces the full-model coefficient exactly. That is what makes the added-variable plot a faithful picture of the coefficient rather than a rough sketch.

Q7. All three body-fat coefficients are non-significant, yet the overall $F$ is highly significant. How can the model be useful if no single predictor is? This is the fingerprint of multicollinearity. The three tape-measure predictors overlap so much that no one of them adds much given the other two, so each individual $t$ is small; but together they explain body fat well, so the overall $F$ is large. The predictors are substitutes, not independent contributors. Chapter 12 takes up this multicollinearity theme on a larger body-fat study and measures the overlap with the variance inflation factor.

8.10 Practice problems¶

1. (A) State in one sentence what the Dwaine coefficient 1.4546 on targtpop means, naming its units and what is held fixed.

2. (A) Explain the difference between the simple-regression slope of sales on dispoinc (31.17) and the multiple regression coefficient (9.37). Which holds population fixed?

3. (A) Define the extra sum of squares $\mathrm{SSR}(X_2 \mid X_1)$ in words, and explain why it can never be negative.

4. (A) A reduced model drops three predictors from a full model. State the numerator degrees of freedom of the general linear test comparing them.

5. (A) Why does $R^2$ never decrease when a predictor is added? Answer using the identity $R^2 = 1 - \mathrm{SSE}/\mathrm{SSTO}$.

6. (A) A colleague reports that adjusted $R^2$ fell when they added a variable, and concludes they made an error. Are they right? Explain.

7. (A) In your own words, what does an added-variable plot put on each axis, and why is neither axis the raw variable?

8. (A) Give an example, in the Dwaine context, of a lurking variable that could distort the income coefficient, and say why it would matter.

9. (A) Explain why a query point can be a hidden extrapolation even when each predictor value is inside its own observed range.

10. (A) All three body-fat coefficients are individually non-significant but the overall $F$ is highly significant. Explain how both can be true.

11. (B) Prove that adding a predictor to a model cannot increase $\mathrm{SSE}$ (Theorem 8.4), using the projection characterization of least squares from 7.2 The geometry of least squares.

12. (B) Show that $\mathrm{SSR}(X_2 \mid X_1) = \mathrm{SSR}(X_1, X_2) - \mathrm{SSR}(X_1)$ is equivalent to $\mathrm{SSE}(X_1) - \mathrm{SSE}(X_1, X_2)$, citing which quantity is model-independent.

13. (B) Derive the Pythagorean identity $\mathrm{SSE}(R) = \mathrm{SSE}(F) + \lVert \widehat{\mathbf{Y}}_F - \widehat{\mathbf{Y}}_R\rVert^2$ for nested models, stating where orthogonality is used.

14. (B) Starting from $s^2\{b_k\} = \mathrm{MSE}\cdot c_{kk}$ and $\mathrm{SSR}(X_k \mid \text{others}) = b_k^2/c_{kk}$, prove that the general linear test with $q = 1$ gives $F^{*} = t_k^2$ (Theorem 8.7).

15. (B) Prove that the slope of the added-variable plot for $X_k$ equals the full-model coefficient $b_k$, in the two-predictor case (Theorem 8.9, Frisch-Waugh).

16. (B) Derive $R^2_a = 1 - \frac{n-1}{n-p}(1 - R^2)$ from the definition $R^2_a = 1 - \frac{\mathrm{SSE}/(n-p)}{\mathrm{SSTO}/(n-1)}$, and explain when adding a predictor lowers it.

17. (B) Show that the standardized coefficient satisfies $b_k^{*} = b_k\,(s_{X_k}/s_Y)$ by relating the regression on standardized variables to the regression on raw variables.

18. (B) Explain why reordering predictors changes the sequential sums of squares but never changes their total or the full-model coefficients. Tie your answer to which quantities the order affects.

19. (B) The general linear test for the overall regression compares the full model to the intercept-only model. Show that its $F^{*}$ equals $\mathrm{MSR}/\mathrm{MSE}$ with $\mathrm{MSR} = \mathrm{SSR}/(p-1)$.

20. (C) Fit the Dwaine model in R or Python and reproduce $b_0, b_1, b_2$, $R^2$, and the residual standard error. Confirm $\mathrm{MSE} = \mathrm{SSE}/(n-p)$ from the residuals.

21. (C) On bodyfat3.csv, print the sequential ANOVA in both predictor orders (triceps-then-thigh and thigh-then-triceps) and report how $\mathrm{SSR}(\text{thigh})$ and $\mathrm{SSR}(\text{thigh} \mid \text{triceps})$ differ. Explain the difference.

22. (C) Run the general linear test dropping pop75 and dpi from the savings model, first with the built-in model comparison and then by hand from the two error sums of squares. Confirm they agree and report $F^{*}$ and its $p$-value.

23. (C) For the savings model, verify $F^{*} = t^2$ for the single predictor ddpi: compute the general linear test dropping ddpi and compare to the square of ddpi’s $t$ statistic.

24. (C) Build the added-variable plot for pop15 in the savings model. Report the slope, confirm it equals the full-model coefficient on pop15, and describe the scatter.

25. (C) Add three columns of standard normal noise (seed 4210) to the savings data, refit with the extra columns, and report how $R^2$ and adjusted $R^2$ change. Which one behaves sensibly?

26. (C) Standardize the Dwaine variables and refit; report the two standardized coefficients and say which predictor has the larger standardized effect. Does the ranking match the raw $t$ statistics?

27. (C) Predict Dwaine sales at target population 75 and income 16, and separately at the sample means. Argue from Figure 15 which prediction you would trust and why.

28. (C) Fit the full three-predictor body-fat model and report all three $t$ statistics and the overall $F$. Explain, using the correlations among the predictors, why the individual and joint verdicts disagree.

29. (B) A student claims that because $\mathrm{SSR}(X_2 \mid X_1) \ge 0$ always, adding any predictor always improves the model. Explain precisely what “improves” can and cannot mean here, distinguishing $\mathrm{SSE}$, $R^2$, adjusted $R^2$, and out-of-sample prediction.

30. (C) Using the Dwaine model, compute a 95% confidence interval for mean sales and a 95% prediction interval for a new city’s sales at target population 65.4 and income 17.6. Explain in one sentence why the prediction interval is wider, citing 3.5 Prediction interval for a new observation.

8.11 Exam practice¶

These five questions are written in the style of the course exams: each one asks you to explain your reasoning in full sentences, not just to produce a number. Read the output, decide what it says, and write the argument you would want a grader to read. Work each one before opening its model answer.

EP 8.1. The four-predictor savings model is fit to the 50 countries, and R prints the coefficient table below.

            Estimate Std. Error t value Pr(>|t|)
(Intercept)  28.5661     7.3545  3.8842   0.0003
pop15        -0.4612     0.1446 -3.1885   0.0026
pop75        -1.6915     1.0836 -1.5610   0.1255
dpi          -0.0003     0.0009 -0.3618   0.7192
ddpi          0.4097     0.1962  2.0882   0.0425

Interpret the coefficient on pop15 as a partial slope, giving its units and the exact phrase for what is held fixed. Then, using the $t$ value and $p$-value, explain whether pop15 is distinguishable from zero at the 5% level. Finally, a classmate reads the near-zero dpi coefficient $(-0.0003)$ and concludes that disposable income has no relationship with the savings rate; explain in full sentences why that conclusion overreaches.

EP 8.2. A student fits the three body-fat predictors in two different orders and prints the sequential (Type I) ANOVA table each time.

Response: bodyfat        (order: triceps, thigh, midarm)
          Df Sum Sq Mean Sq F value    Pr(>F)
triceps    1 352.27  352.27 57.2768 1.131e-06
thigh      1  33.17   33.17  5.3931   0.03373
midarm     1  11.55   11.55  1.8773   0.18956
Residuals 16  98.40    6.15

Response: bodyfat        (order: thigh, triceps, midarm)
          Df Sum Sq Mean Sq F value    Pr(>F)
thigh      1 381.97  381.97 62.1052 6.735e-07
triceps    1   3.47    3.47  0.5647   0.46330
midarm     1  11.55   11.55  1.8773   0.18956
Residuals 16  98.40    6.15

The student writes: “Thigh gets a sum of squares of 381.97 in the second table but only 33.17 in the first. The same variable cannot have two different sums of squares, so one of these tables must be a software error.” Evaluate this claim, and explain in full sentences what the two thigh numbers actually represent.

EP 8.3. In the savings model you want to know whether the two economic predictors, dpi and ddpi, can be dropped once the two demographic predictors pop15 and pop75 are in the model. The model comparison prints:

Model 1: sr ~ pop15 + pop75
Model 2: sr ~ pop15 + pop75 + dpi + ddpi
  Res.Df    RSS Df Sum of Sq     F  Pr(>F)
1     47 726.17
2     45 650.71  2    75.455 2.609 0.08471

State the null and alternative hypotheses this test evaluates and your decision at the 5% level. Then, recalling from the full-model summary in EP 8.1 that dpi has $t = -0.36$ ($p = 0.72$) while ddpi has $t = 2.09$ ($p = 0.0425$), explain in full sentences why it would be a mistake to react to this non-significant joint test by dropping both dpi and ddpi.

EP 8.4. For the savings model, the added-variable plot for pop15 is built by residualizing both the response and pop15 on the other three predictors, and its slope is computed.

AV slope pop15 = -0.4612   full coef pop15 = -0.4612

Explain why the added-variable plot slope must equal the full-model coefficient, naming the result that guarantees it. Then explain what would change if a student residualized only the response sr on the other predictors but plotted those residuals against the raw pop15 values instead of the residualized pop15: would the slope still be -0.4612, and what quantity would that botched slope represent?

EP 8.5. Starting from the four-predictor savings model, three columns of pure standard-normal noise (seed 4210) are added and the model is refit.

model                     R2       adj R2
sr ~ 4 predictors         0.3385   0.2797
sr ~ 4 predictors + 3 noise 0.3980   0.2977

Explain why $R^2$ rose from 0.3385 to 0.3980 even though the three added columns are pure noise. Then explain why this rise is not evidence that the larger model is better, and say what adjusted $R^2$ and out-of-sample prediction each tell you here.

8.12 Chapter game¶