4. Correlation

Why do some countries save a large share of their income while others save almost nothing? In the 1960s economists studying the life-cycle theory of saving collected a small cross-country table to look for an answer. For 50 countries they recorded the aggregate personal savings rate, averaged over 1960 to 1970. Alongside it they noted a few features of each economy: the fraction of the population under 15, the fraction over 75, per-capita income, and how fast that income was growing. The life-cycle idea predicts that people save during their working years and draw the savings down when young or old, so a country with many dependents, young or old, should save less.

Figure 1 shows the first slice of that table: the savings rate against the share of the population under 15. The cloud slopes downward. Countries with more children do tend to save less, just as the theory says. But how strong is that tendency? “Slopes downward” is a picture, not a number, and a picture cannot go into a report or a comparison. We want a single number that measures how tightly two quantities move together, one that does not depend on whether savings is in percent or income in dollars, and one we can attach a margin of error to.

That number is the correlation coefficient (defined formally in Definition 4.1), and this chapter is about reading it honestly. In Chapters 2 and 3 we built an entire line and tested every part of it, from the slope to the prediction interval; correlation asks a smaller and more symmetric question, how tightly two quantities move together, and Section 4.2 will show it is that same fit seen through a single number. Correlation is the workhorse summary of a straight-line relationship: it compresses a whole scatterplot into one value between -1 and +1. It is also one of the most abused numbers in all of statistics, blamed for causal claims it never made, inflated by outliers, hidden by curves, and shrunk by a badly chosen sample. By the end you will be able to compute it, test it, interval it, and, just as important, say when it is lying to you.

4.1 The correlation coefficient¶

Intuition¶

Here is the goal in one plain sentence: take a whole scatterplot and boil it down to a single number that says how tightly the two things move together, and make that number come out the same no matter what units you measured them in. That number is the correlation, and this section builds it from parts you already have.

The regression slope from Chapter 2 already measures how $Y$ moves with $X$, but it carries the units of the problem: for the savings data the slope is in percent of savings per percentage point of young population. Change savings to a fraction instead of a percent and the slope changes too. That makes slopes hard to compare across different pairs of variables. Correlation fixes the units problem by measuring association on a pure, dimensionless scale.

The trick is standardizing. Before comparing how two variables move, put both on the same footing: subtract each variable’s mean and divide by its standard deviation, so each becomes a set of z-scores with no units. A country that is one standard deviation above the mean in young population and half a standard deviation below the mean in savings contributes a product of $(+1)(-0.5) = -0.5$ to the tally. Add those products up and average them, and you have measured whether above-average $X$ tends to come with above-average $Y$ (positive), below-average $Y$ (negative), or neither (near zero). The average of the standardized products always lands between -1 and +1. That average is the correlation.

Figure 2 makes this bookkeeping visible on the savings data. Split the plot at the two means into four quadrants. A country in the upper-right or lower-left sits on the same side of both means, so its product is positive. A country in the other two quadrants sits on opposite sides, so its product is negative. Because more youth tends to come with less saving, most points fall into the two negative quadrants, and the average of all the products comes out negative. That negative average is the correlation.

Formula¶

Recall the deviation sums from 2.2 Least squares from first principles: $S_{xx} = \sum (X_i - \bar X)^2$, $S_{yy} = \sum (Y_i - \bar Y)^2$, and the cross-product $S_{xy} = \sum (X_i - \bar X)(Y_i - \bar Y)$. Standardizing the covariance by the two standard deviations gives the correlation.

• $s_{xy}$ is the covariance: the average product of paired deviations, positive when $X$ and $Y$ tend to sit on the same side of their means.

• $s_x = \sqrt{S_{xx}/(n-1)}$ and $s_y = \sqrt{S_{yy}/(n-1)}$ are the sample standard deviations of $X$ and $Y$.

• $r$ is the covariance rescaled by the two standard deviations, which cancels all units and pins the value into $[-1, 1]$.

In words: $r$ is the covariance of $X$ and $Y$ measured in standard-deviation units of each, so it says how many standard deviations $Y$ moves, on average, per standard deviation of $X$, capped at one. The $(n-1)$ factors in $s_{xy}$, $s_x$, and $s_y$ cancel in the ratio, which is why the second form of $r$ uses the raw deviation sums with no divisor at all.

Derivation (why $r$ lives in $[-1, 1]$)¶

Proof. Standardize both variables. Write $u_i = (X_i - \bar X)/s_x$ and $v_i = (Y_i - \bar Y)/s_y$. By construction $\sum u_i^2 = (n-1)$ and $\sum v_i^2 = (n-1)$ (the squared deviations of $X$ sum to $S_{xx}$, and dividing each by $s_x^2 = S_{xx}/(n-1)$ leaves $n-1$; same for $Y$), and the correlation is $r = \frac{1}{n-1}\sum u_i v_i$. Now use the fact that a sum of squares is never negative. For either choice of sign,

Divide by $2(n-1) > 0$. The plus-sign choice gives $0 \le 1 + r$, so $r \ge -1$; the minus-sign choice gives $0 \le 1 - r$, so $r \le 1$. Therefore $-1 \le r \le 1$. Equality holds only when the squared sum is exactly zero, that is when $v_i = \pm u_i$ for every $i$: the standardized points fall on a perfect line, so $r = \pm 1$ means an exact straight-line relationship and nothing less. $\blacksquare$

R¶

Reading a correlation is easier once you see a whole table of them. The cor function applied to several columns returns the correlation matrix, every pairwise correlation at once.

savings <- read.csv("data/savings.csv")
round(cor(savings[, c("sr", "pop15", "pop75", "dpi", "ddpi")]), 3)

          sr  pop15  pop75    dpi   ddpi
sr     1.000 -0.456  0.317  0.220  0.305
pop15 -0.456  1.000 -0.908 -0.756 -0.048
pop75  0.317 -0.908  1.000  0.787  0.025
dpi    0.220 -0.756  0.787  1.000 -0.129
ddpi   0.305 -0.048  0.025 -0.129  1.000

The diagonal is all ones (every variable correlates perfectly with itself), and the matrix is symmetric. The savings rate sr correlates -0.456 with the young-population share and +0.317 with the old-population share, exactly the two signs the life-cycle theory predicts. Notice also the -0.908 between pop15 and pop75: countries with many children have few elderly and the reverse, so those two predictors carry almost the same information. That near-duplication will matter when the savings data returns for multiple regression in Chapter 8 and for diagnostics in Chapter 9.

To calibrate your eye, Figure 3 shows six clouds with correlations from strongly negative to nearly perfect. A correlation of $\pm 0.3$ still looks like a formless blob to most people; you need to reach 0.7 or so before the linear trend jumps out. The savings correlation of -0.46 sits between the second and third panels.

Reading a correlation off a picture is a trained skill, and the six frozen panels above can only take you so far, so sweep the slider below through the whole range and calibrate your eye against a cloud that moves.

4.2 Correlation and the regression slope¶

Intuition¶

Correlation and the regression slope from Chapter 2 are measuring the same relationship, so they cannot be independent numbers. The slope $b_1$ answers “how many units of $Y$ per unit of $X$,” carrying units; the correlation $r$ answers “how many standard deviations of $Y$ per standard deviation of $X$,” carrying none. Convert one to the other by putting in or taking out the units, which means multiplying or dividing by the ratio of the two standard deviations. This section makes that exact, because the tie between $r$ and $b_1$ explains a fact you will meet again and again: the correlation squared is the fraction of variance the regression explains.

Formula¶

The slope and the correlation are linked by

• $b_1 = S_{xy}/S_{xx}$ is the least-squares slope of $Y$ on $X$ from 2.2 Least squares from first principles.

• $s_x, s_y$ are the sample standard deviations, so $s_y/s_x$ is the unit-conversion factor between the two scales.

• $R^2 = \mathrm{SSR}/\mathrm{SSTO}$ is the coefficient of determination from 3.8 R-squared and its interpretation, the share of the total variation in $Y$ that the fitted line accounts for.

In words: the slope is the correlation scaled up by how much more spread $Y$ has than $X$, and squaring the correlation gives exactly the proportion of $Y$’s variance that the regression explains.

Derivation (the exact link between $r$, $b_1$, and $R^2$)¶

Proof. Start from the two definitions and factor. Since $s_y/s_x = \sqrt{S_{yy}/S_{xx}}$ (the $(n-1)$ factors cancel),

Solving for $r$ gives $r = b_1\, s_x/s_y$. Both estimates share the sign of $S_{xy}$, so a positive slope always comes with a positive correlation and never disagrees in sign.

Now the connection to $R^2$. From the fitted line of 2.2 Least squares from first principles the regression sum of squares is $\mathrm{SSR} = \sum(\hat Y_i - \bar Y)^2 = b_1^2 S_{xx}$, and because $b_1 S_{xx} = S_{xy}$ this is $\mathrm{SSR} = b_1 S_{xy}$. Divide by $\mathrm{SSTO} = S_{yy}$:

So the coefficient of determination is nothing more than the correlation squared, in simple linear regression. $\blacksquare$

One more consequence is worth stating, because it explains a puzzle from Chapter 2. If you regress $X$ on $Y$ instead of $Y$ on $X$, the slope is $b_1' = S_{xy}/S_{yy}$. Multiply the two slopes:

The two regression slopes are reciprocals only when $r^2 = 1$, that is only when the fit is perfect. Otherwise regressing $Y$ on $X$ and $X$ on $Y$ give genuinely different lines, and their slopes multiply to $r^2$. Correlation is the one symmetric summary that does not care which variable you call the response.

4.3 Inference for the correlation¶

Intuition¶

The -0.46 is a sample correlation, computed from 50 countries that happened to be in the table. Behind it sits a population correlation $\rho$ (the Greek letter rho), the value we would get from every country that could exist under the same conditions. We have the same two questions as always: could the true $\rho$ be zero, with our -0.46 just sampling noise, and what range of $\rho$ values is consistent with the data? The first is a hypothesis test, the second a confidence interval. The test turns out to be one you already know in disguise, and the interval needs one clever change of variable.

Formula¶

To test $H_0: \rho = 0$ against $H_a: \rho \ne 0$, use

For a confidence interval we first change scale, using Fisher’s $z$ transformation.

Figure 5 shows what the transform does. Near the middle of the scale it barely changes $r$ at all: a step from 0.45 to 0.50 moves $z$ by only 0.06. But as $r$ nears the walls at $\pm 1$ the curve turns almost vertical, so the same 0.05 step from 0.90 to 0.95 moves $z$ by 0.36, more than five times as far. The transform gives the crowded values near the ceiling room to spread out, which is exactly what turns their lopsided pile into a symmetric bell.

On this scale the sampling distribution of $z$ is approximately normal with a spread that no longer depends on the unknown $\rho$, $z \approx N(\operatorname{arctanh}(\rho),\ 1/(n-3))$; the next derivation shows why, and Theorem 4.7 states it precisely.

• $t^{\ast}$ is the test statistic; under the null of no correlation it follows a $t$ distribution with $n-2$ degrees of freedom.

• $\operatorname{arctanh}$ is the inverse hyperbolic tangent; it stretches the crowded ends of the $[-1,1]$ scale out toward $\pm\infty$.

• The transformed statistic $z$ is approximately normal with a variance, $1/(n-3)$, that no longer depends on the unknown $\rho$.

In words: the correlation test is a $t$ test with $n-2$ degrees of freedom, and to build an interval we move to a scale where the sampling distribution is normal with a known spread, make the interval there, and transform back.

Derivation (the correlation $t$ test is the slope $t$ test)¶

Proof. From 3.7 The F test and its equivalence to the t test the slope is tested with $t = b_1 / s\{b_1\}$, where $s\{b_1\} = \sqrt{\mathrm{MSE}/S_{xx}}$ and $\mathrm{MSE} = \mathrm{SSE}/(n-2)$. Two facts from 4.2 Correlation and the regression slope do the work. First, $\mathrm{SSE} = \mathrm{SSTO} - \mathrm{SSR} = S_{yy}(1 - r^2)$, since $\mathrm{SSR} = r^2 S_{yy}$. Second, $b_1\sqrt{S_{xx}} = r\,(s_y/s_x)\sqrt{S_{xx}} = r\sqrt{S_{yy}}$. Substitute:

The $S_{yy}$ cancels, and the slope $t$ statistic and the correlation $t$ statistic are the identical number. Testing “is the slope zero” and “is the correlation zero” are the same test, as they must be, since $b_1$ and $r$ share a sign and vanish together. $\blacksquare$

Derivation sketch (Fisher’s $z$ transformation)¶

The $t$ test handles $\rho = 0$, but a confidence interval for a nonzero $\rho$ is harder, for two reasons. When $\rho$ is far from zero the sampling distribution of $r$ is skewed, because $r$ is trapped below the ceiling at 1 (or above the floor at -1) and piles up against it. And the variance of $r$ depends on $\rho$ itself, so we cannot even write down a fixed standard error. Fisher’s idea was to find a transformation $g(r)$ that cures both problems at once.

Proof (sketch). For a sample from a bivariate normal population, large-sample theory gives $r$ an approximate mean $\rho$ and variance

We want a function $g$ so that $g(r)$ has a variance free of $\rho$. The delta method (a first-order Taylor expansion, $g(r) \approx g(\rho) + g'(\rho)(r - \rho)$) says

To kill the $\rho$ dependence, demand $g'(\rho) = 1/(1-\rho^2)$. Integrating that derivative gives

and with this choice $\operatorname{Var}(g(r)) \approx 1/n$, constant at last. Fisher’s finer analysis replaces the $n$ by $n-3$ for a better small-sample match and shows the distribution of $z = \operatorname{arctanh}(r)$ is close to normal, not skewed. So $z \approx N(\operatorname{arctanh}(\rho),\, 1/(n-3))$. $\blacksquare$

Two honesty notes. The exact sampling distribution of $r$ under bivariate normality is known in closed form but is a messy special function; the normal approximation to $z$ is what everyone actually uses, and the next simulation shows why it is safe. And the whole argument assumes the data really are bivariate normal. When that is in doubt, the bootstrap of Chapter 5 (5.4 The bootstrap for regression) gives a confidence interval for $\rho$ that leans on no distributional assumption at all.

To build a 95 percent interval: compute $z = \operatorname{arctanh}(r)$, form $z \pm 1.96/\sqrt{n-3}$, and transform both ends back with $\tanh$.

The transformation, checked by simulation¶

The Fisher argument leaned on two claims: that $r$ is skewed while $z$ is nearly normal, and that the standard deviation of $z$ is about $1/\sqrt{n-3}$. Both are checkable without any more algebra. Draw many samples from a bivariate normal with a known $\rho$, compute $r$ each time, and look at the pile of results.

set.seed(4210)
simulate_r <- function(n, rho, reps = 10000) {
  replicate(reps, {
    z1 <- rnorm(n)
    z2 <- rnorm(n)
    xa <- z1
    ya <- rho * z1 + sqrt(1 - rho^2) * z2
    cor(xa, ya)
  })
}
rs <- simulate_r(n = 20, rho = 0.7)
zs <- atanh(rs)
round(c(mean_r = mean(rs), sd_r = sd(rs),
        mean_z = mean(zs), sd_z = sd(zs),
        theory_sd_z = 1 / sqrt(20 - 3)), 4)

     mean_r        sd_r      mean_z        sd_z theory_sd_z 
     0.6919      0.1250      0.8906      0.2412      0.2425 

rng = np.random.default_rng(4210)
def simulate_r(n, rho, reps=10000):
    out = np.empty(reps)
    for i in range(reps):
        z1 = rng.standard_normal(n)
        z2 = rng.standard_normal(n)
        xa = z1
        ya = rho * z1 + np.sqrt(1 - rho ** 2) * z2
        out[i] = np.corrcoef(xa, ya)[0, 1]
    return out

rs = simulate_r(20, 0.7)
zs = np.arctanh(rs)
print(round(rs.mean(), 4), round(rs.std(ddof=1), 4),
      round(zs.mean(), 4), round(zs.std(ddof=1), 4),
      round(1 / np.sqrt(20 - 3), 4))

0.6893 0.1254 0.8852 0.2405 0.2425

The simulated standard deviation of $z$ is 0.2412 (R) and 0.2405 (Python), both a hair under the theoretical $1/\sqrt{17} = 0.2425$, confirming the variance formula. The mean of $z$ lands near $\operatorname{arctanh}(0.7) = 0.867$, close to the simulated 0.89. Figure 6 tells the visual half of the story: the raw correlations bunch up and skew toward the ceiling at 1, while the transformed values sit under a symmetric normal curve. That skew is exactly why we never build the interval on the raw $r$ scale.

4.4 The bivariate normal model¶

Intuition¶

So far we have pictured one variable as a set of dials we fix in advance, with only the other left to vary, the regression setup of Chapter 2. Correlation has a second, more even-handed home, where both measurements are random and arrive together as a pair, like a random person’s height and weight measured at the same moment. In that picture the two variables are drawn together from a single joint distribution. The cleanest such distribution is the bivariate normal, whose contour lines are ellipses. When the correlation is zero the ellipse is a circle; as the correlation grows the circle stretches and tilts toward the 45-degree line, becoming a thin cigar as it approaches $\pm 1$. Figure 7 shows the progression.

Formula¶

In words: the formula is scarier than the idea. The density piles up highest at the center point $(\mu_X, \mu_Y)$ and falls off in oval rings around it, and the single number $\rho$ decides how much those rings tilt and stretch away from circles. Read the pieces one at a time.

• $\mu_X, \mu_Y$ are the two means and $\sigma_X, \sigma_Y$ the two standard deviations.

• $\rho$ is the population correlation; it is the only parameter that ties $X$ and $Y$ together, and $\rho = 0$ makes the density factor into two separate normals.

• The bracketed quadratic form is what draws the elliptical contours; its cross term carries the $\rho$.

The single most useful fact about this model is what it says about $Y$ once you know $X$. The conditional distribution of $Y$ given $X = x$ is again normal, with

In words: inside a bivariate normal, the regression of $Y$ on $X$ is exactly linear, with population slope $\rho\,\sigma_Y/\sigma_X$, and the leftover variance is the original variance shrunk by the factor $1 - \rho^2$.

Derivation (the hidden regression line)¶

Proof. The joint density factors as $f(x,y) = f_X(x)\, f(y \mid x)$, where $f_X$ is the $N(\mu_X, \sigma_X^2)$ marginal. Divide the full density by that marginal. The terms that depend only on $x$ cancel, and after completing the square in $y$ inside the exponent, what remains is a normal density in $y$ with mean $\mu_Y + \rho(\sigma_Y/\sigma_X)(x-\mu_X)$ and variance $\sigma_Y^2(1-\rho^2)$. The completing-the-square step is the same algebra used for the univariate normal; the cross term $-2\rho(x-\mu_X)(y-\mu_Y)/(\sigma_X\sigma_Y)$ is what pushes the center of the conditional distribution off $\mu_Y$ by an amount proportional to $(x - \mu_X)$. $\blacksquare$

Compare the population slope $\rho\,\sigma_Y/\sigma_X$ with the sample slope $b_1 = r\,s_y/s_x$ from 4.2 Correlation and the regression slope: they are the same formula, one with population quantities and one with their sample estimates. The regression line of Chapter 2 is the sample version of the bivariate normal’s conditional mean. And the variance identity $\operatorname{Var}\{Y \mid X\} = \sigma_Y^2(1 - \rho^2)$ is the population twin of $R^2 = r^2$: knowing $X$ removes the fraction $\rho^2$ of $Y$’s variance, leaving $1 - \rho^2$. When $\rho = 0.7$, knowing $X$ explains 0.49 of the variance and leaves just over half.

4.5 Spearman rank correlation¶

Intuition¶

Pearson’s $r$ measures straight-line association, and it trusts every value at face value, so a single wild point can swing it hard. Often you care about a gentler question: as $X$ goes up, does $Y$ tend to go up, whether or not the increase is along a straight line? That is a question about order, not distance, and it has its own coefficient. Spearman’s rank correlation (Definition 4.10) replaces each value by its rank (smallest is 1, next is 2, and so on) and then computes the ordinary Pearson correlation of those ranks. Because ranks ignore how far apart the values are and see only their order, Spearman is unbothered by a lone outlier and by curves that are monotone but not straight.

Figure 8 shows the curve case. On the left is a relationship that always rises but bends, so Pearson reads only 0.91, penalizing the bend. Replace each value by its rank, and the same points snap onto a perfect diagonal: rank 1 in $X$ pairs with rank 1 in $Y$, rank 2 with rank 2, all the way up. Because the order is perfect, even though the spacing is not, Spearman is exactly 1. Ranks straighten any monotone curve.

Formula¶

• $\operatorname{rank}(X_i)$ is the position of $X_i$ in the sorted list of the $X$ values, and likewise for $Y$.

• $r_S$ lies in $[-1, 1]$ just like Pearson’s $r$, equals +1 for any strictly increasing relationship (not only a straight line) and -1 for any strictly decreasing one.

In words: Spearman is Pearson applied to the rankings, so it measures whether the two variables rise and fall together in order, and it does not care about the exact spacing of the values.

R and Python¶

The savings data has a natural test case. The growth rate of income, ddpi, correlates with the savings rate, but one country, Libya, had an enormous income growth rate over the period and only a middling savings rate. That single point drags the Pearson correlation down. Spearman, seeing Libya merely as “the highest-growth country” rather than as a point far off to the right, is less disturbed.

Neither number is “right.” They answer different questions. Pearson asks about straight-line strength and counts Libya’s distance; Spearman asks about monotone order and does not. When they disagree, that disagreement is information: it says an outlier or a curve is present, and you should look at the plot before quoting either one.

The gap between the two coefficients is easiest to believe when you open and close it with your own finger, so drag the stand-in for Libya and watch which number reacts.

4.6 Four ways a correlation lies¶

A single number that compresses a scatterplot must throw information away, and sometimes it throws away the part that mattered. The founding demonstration is Anscombe’s quartet: four datasets, built by the statistician Frank Anscombe in 1973, that share the same means, the same standard deviations, the same correlation of 0.816, and the same fitted line, yet look nothing alike.

anscombe <- read.csv("data/anscombe.csv")
stats_row <- function(i) {
  xi <- anscombe[[paste0("x", i)]]
  yi <- anscombe[[paste0("y", i)]]
  f <- lm(yi ~ xi)
  c(set = i, mean_x = mean(xi), mean_y = mean(yi),
    r = cor(xi, yi), b0 = coef(f)[[1]], b1 = coef(f)[[2]])
}
round(t(sapply(1:4, stats_row)), 3)

     set mean_x mean_y     r    b0  b1
[1,]   1      9  7.501 0.816 3.000 0.5
[2,]   2      9  7.501 0.816 3.001 0.5
[3,]   3      9  7.500 0.816 3.002 0.5
[4,]   4      9  7.501 0.817 3.002 0.5

anscombe = pd.read_csv("data/anscombe.csv")
for i in range(1, 5):
    xi = anscombe[f"x{i}"]
    yi = anscombe[f"y{i}"]
    f = smf.ols(f"y{i} ~ x{i}", data=anscombe).fit()
    print(i, round(xi.mean(), 2), round(yi.mean(), 3),
          round(xi.corr(yi), 3), round(f.params.iloc[0], 3),
          round(f.params.iloc[1], 3))

1 9.0 7.501 0.816 3.0 0.5
2 9.0 7.501 0.816 3.001 0.5
3 9.0 7.5 0.816 3.002 0.5
4 9.0 7.501 0.817 3.002 0.5

The four rows are identical to three decimals. Figure 11 plots them, and only the plot reveals the truth. Set I is a genuine linear scatter, the honest case. Set II is a smooth curve, where a straight-line correlation is the wrong summary entirely. Set III is a perfect line with one outlier that tilts the fit and lowers $r$ from 1 to 0.82. Set IV is the most alarming: ten points stacked at a single $x$ value carry no information about slope at all, and one lone point far to the right invents the entire correlation. Delete that one point and $r$ is undefined. The lesson Anscombe drew, and the first habit of any honest analyst, is to plot the data before trusting the number.

The quartet is a museum piece, built to make the point. The same four failures show up in real data, so here they are one at a time, with the mechanism named.

Nonlinearity. Correlation measures straight-line association only. A relationship can be perfectly predictable and yet have a Pearson correlation near zero, as in a symmetric U-shape where $Y$ falls and then rises: the downhill and uphill halves cancel. A small $r$ never means “no relationship.” It means “no straight-line relationship,” and the only way to tell the difference is to look. Anscombe’s set II is this trap; the fix is a scatterplot and, if a curve appears, the transformations of Chapter 10.

Outliers. Because $r$ squares distances, one point far from the crowd can dominate the sum, either manufacturing a correlation (set IV) or destroying one (set III). The Libya example in Figure 9 is the mild, real-data version. The defenses are to plot the data, to check whether one or two points are driving the result by recomputing without them, and to report Spearman alongside Pearson when they disagree.

Restriction of range. Correlation depends on the spread of $X$. Squeeze that spread and the correlation shrinks toward zero, even though the underlying relationship has not changed at all. The Galton height data shows this cleanly.

galton <- read.csv("data/galton_heights.csv")
band <- galton[galton$midparentHeight >= 69 & galton$midparentHeight <= 71, ]
round(c(full_r = cor(galton$midparentHeight, galton$childHeight),
        full_n = nrow(galton),
        band_r = cor(band$midparentHeight, band$childHeight),
        band_n = nrow(band),
        sd_full = sd(galton$midparentHeight),
        sd_band = sd(band$midparentHeight)), 3)

 full_r  full_n  band_r  band_n sd_full sd_band 
  0.321 934.000   0.143 393.000   1.802   0.527 

Across all 934 children the correlation between midparent height and child height is 0.32. Keep only families whose midparent height is between 69 and 71 inches, cutting the standard deviation of the parent heights from 1.80 to 0.53, and the correlation falls by more than half to 0.14, as Figure 12 shows. Nothing about heredity changed; we just stopped looking at short and tall parents. This is why a correlation computed inside a narrow group, admitted students, hired employees, surviving patients, routinely understates the association in the full population. Whenever you read a weak correlation, ask what range of $X$ it was computed over.

Set the width of that band yourself, and see how much of a correlation you can destroy without touching a single data point.

Ecological correlation. A correlation computed on group averages is not the correlation for individuals, and it is usually larger, because averaging cancels the within-group scatter. Group the Galton children by family and correlate the family-average child height with the midparent height.

fam <- aggregate(cbind(midparentHeight, childHeight) ~ family,
                 data = galton, FUN = mean)
round(c(individual_r = cor(galton$midparentHeight, galton$childHeight),
        family_r = cor(fam$midparentHeight, fam$childHeight),
        n_families = nrow(fam)), 3)

individual_r     family_r   n_families 
       0.321        0.399      205.000 

The individual-level correlation is 0.32; the family-average correlation is 0.40, as Figure 14 shows. Averaging smooths away the sibling-to-sibling differences and leaves a tighter picture, so the aggregate number overstates how well one child’s height can be predicted from the parents. Inferring individual behavior from group averages is the ecological fallacy, and it has led to real errors: a correlation across countries, states, or precincts can be strong while the same correlation across the people inside them is weak or even reversed. The unit you correlate is part of the claim.

Anscombe’s claim only lands if you confirm that the numbers really do stay put while the picture changes, so step through all four sets yourself and keep your eye on the readouts rather than the plot.

4.7 Regression to the mean, and why correlation is not causation¶

There is a fifth confusion that deserves its own section, because it fooled the inventor of regression himself and still fills the news. When Francis Galton plotted children’s heights against their parents’ in the 1880s, he found that tall parents had children who were taller than average but not as tall as the parents, and short parents had children who were shorter than average but not as short. He first suspected some biological force pulling the population toward mediocrity. There is no such force. The pull toward the average is a mathematical certainty whenever the correlation is less than perfect, and it has a name: regression to the mean.

The identity from 4.2 Correlation and the regression slope makes it exact. In standard units, where both variables are z-scores, the standard deviations are both 1, so the regression slope is $b_1 = r\,(s_y/s_x) = r$. A parent who is 2 standard deviations above the mean has children predicted at only $r \times 2$ standard deviations above the mean. For the Galton data $r = 0.32$, so the children of very tall parents are predicted just $0.32 \times 2 = 0.64$ standard deviations up, well short of their parents.

Regression to the mean is not a special fact about heredity. It happens any time two measurements are imperfectly correlated, which is almost always. The student who scores highest on the midterm tends to score lower (still good, just less extreme) on the final. The rookie athlete featured on a magazine cover after a spectacular season tends to do worse the next year, the so-called cover jinx, with no jinx involved. A clinic enrolls patients with the highest blood pressure, treats them, and sees their pressure fall on average, partly because the most extreme readings were extreme partly by luck and drift back on their own. In every case the naive reading blames a cause, the pressure of fame, a miracle drug, when the imperfect correlation between two measurements already predicts the drift. Whenever you select cases because they were extreme and then measure them again, expect regression to the mean, and do not hand the credit or the blame to a story.

This is the deepest form of the oldest warning in statistics: correlation is not causation. A correlation between $X$ and $Y$ is consistent with $X$ causing $Y$, with $Y$ causing $X$, with some third variable driving both, and with pure coincidence. The savings correlation of -0.46 does not prove that having children makes a country thrifty or spendthrift; income, culture, and history sit behind both. Correlation earns its keep as a description and a first clue, not as a verdict. Sorting genuine causes from lookalikes needs either a designed experiment or the careful causal reasoning that Chapter 16 takes up (16.1 Drawing a causal story: path diagrams), and it is the single hardest thing this book asks you to keep straight.

4.8 Chapter summary¶

You can now describe a straight-line relationship with a single, unitless number and say how much to trust it. You met the Pearson correlation as a standardized covariance, proved it lives in $[-1, 1]$, tied it to the regression slope and to $R^2$, tested and interval-ed the population value $\rho$, placed it inside the bivariate normal model, softened it into Spearman’s rank version, and learned the five ways it misleads: nonlinearity, outliers, restriction of range, ecological aggregation, and the leap to causation.

Key results at a glance

Key terms. Correlation coefficient, sample covariance, correlation matrix, coefficient of determination, Fisher’s $z$ transformation, bivariate normal distribution, conditional distribution, Spearman rank correlation, restriction of range, ecological correlation (and the ecological fallacy), regression to the mean.

You should now be able to

• Compute the Pearson correlation $r$ by hand and with software, and read it as a unit-free standardized covariance.

• Derive the identity $b_1 = r\,s_y/s_x$ and show that $r^2 = R^2$.

• Test $\rho = 0$ with the $t$ statistic and build a confidence interval for $\rho$ with Fisher’s $z$, checking it by simulation.

• State the bivariate normal model and read its population regression line and $1 - \rho^2$ leftover variance.

• Compute Spearman’s rank correlation and decide when ranks are safer than raw values.

• Diagnose nonlinearity, outliers, restriction of range, and ecological correlation.

• Explain regression to the mean and why a strong correlation is still not evidence of causation.

Where this fits. In the modeling workflow of The modeling workflow, correlation serves the EXPLORE stage: before fitting anything, you look at how each pair of variables moves together, and the correlation matrix is the numeric companion to the scatterplot. It reaches into USE as well, through the inference for $\rho$ and the warnings that keep you from turning a described association into a claimed cause. It builds directly on the least-squares line of 2.2 Least squares from first principles and the ANOVA decomposition of 3.6 The analysis of variance approach, since $r$, $b_1$, and $R^2$ are three views of one fit. Its tools get used again soon: the $r$-versus-$b_1$ identity of 4.2 Correlation and the regression slope returns in Chapter 12 as a check on an inflating $R^2$ (12.2 Choosing among models: selection criteria); the same savings data becomes a second worked multiple regression in Chapter 8, where the strong pop15-pop75 correlation you saw in the matrix turns into a real collinearity problem; Libya’s outsized pull becomes a named influence diagnostic in Chapter 9 (9.3 Influence: which points actually change the fit); and the “correlation is not causation” thread is picked back up in Chapter 16 with the tools to reason carefully about it (16.1 Drawing a causal story: path diagrams).

4.9 Frequently asked questions¶

Q1. Is a correlation of 0.5 twice as strong as 0.25? Not in the sense that matters most. If “strength” means share of variance explained, then $r = 0.5$ gives $r^2 = 0.25$ and $r = 0.25$ gives $r^2 = 0.0625$, so the first explains four times as much variance, not twice. Correlation and variance-explained are different scales; always be clear which one you mean.

Q2. What is a “big” correlation? It depends entirely on the field. In a tightly controlled physics experiment $r = 0.95$ might be disappointing; in social science, where outcomes have many causes, $r = 0.3$ can be a real and useful finding. There is no universal threshold. Report the number, the interval, and the plot, and let the reader judge against the norms of the subject.

Q3. Correlation is symmetric but regression is not. Why? Correlation asks whether two variables move together and does not distinguish a response from a predictor, so $r(X, Y) = r(Y, X)$. Regression singles out one variable as the response and minimizes vertical distances to it, so swapping the roles changes which distances you minimize and gives a different line. Their slopes multiply to $r^2$, meeting only when the fit is perfect.

Q4. If $r = 0$, are $X$ and $Y$ independent? No. Zero correlation means no straight-line association, but a strong curved relationship (a U-shape, a circle) can have $r = 0$ while $X$ and $Y$ are anything but independent. Independence implies zero correlation; the reverse holds only inside special models like the bivariate normal, where $\rho = 0$ does force independence because the density factors.

Q5. Which should I report, Pearson or Spearman? Report Pearson when you care about straight-line strength and the data are roughly linear with no wild outliers. Reach for Spearman when the relationship is monotone but curved, when outliers are pulling Pearson around, or when the variables are ordinal ranks to begin with. If the two disagree a lot, that gap is a signal to plot the data and find out why, not to quietly pick the bigger number.

Q6. Why does the confidence interval for $\rho$ come out lopsided around $r$? Because the interval is built symmetrically on the Fisher $z$ scale and then bent back through $\tanh$, which is nonlinear near the ends. For the savings data $r = -0.46$ sits at the center of the symmetric $z$ interval, but after transforming back the bounds $(-0.65, -0.20)$ are not equidistant from -0.46. That asymmetry is a feature: it respects the hard walls at $\pm 1$ that a symmetric interval on the raw scale would ignore.

Q7. Does a significant correlation mean the relationship is strong? No. Significance and strength are different things. With a large enough sample, a tiny correlation of 0.05 can be statistically significant, meaning “probably not exactly zero,” while explaining a quarter of one percent of the variance. Always read the size of $r$ and its interval, not just the $p$-value; a small $p$ says the effect is real, not that it is big.

4.10 Practice problems¶

1. (A) In one sentence each, say what the sign and the size of a correlation tell you, and why $r$ has no units.

2. (A) A report gives $r = 0.8$ between two variables and calls it “80 percent agreement.” Explain what is wrong and give the correct interpretation of 0.8.

3. (A) The savings correlation between sr and pop15 is -0.46. Interpret both the sign and the magnitude for a reader who has never seen a correlation.

4. (A) Explain why a correlation of exactly 0 does not mean $X$ and $Y$ are unrelated. Sketch or describe a relationship with $r = 0$ that is nonetheless perfectly predictable.

5. (A) Give the four-question checklist you would run before trusting any reported correlation, and say which trap each question guards against.

6. (A) A weak correlation of 0.1 is found between SAT score and college GPA, computed only among admitted students at a selective school. Name the trap and predict whether the full-population correlation is larger or smaller.

7. (A) Explain regression to the mean in your own words, and give an everyday example that is not heights or test scores.

8. (A) Why are the two regression slopes (of $Y$ on $X$ and of $X$ on $Y$) generally different, and when are they reciprocals of each other?

9. (B) Prove that $-1 \le r \le 1$ (Theorem 4.2) using the fact that $\sum(u_i \pm v_i)^2 \ge 0$ for the standardized values $u_i, v_i$. State the condition for equality and what it means geometrically.

10. (B) Derive the identity $b_1 = r\,s_y/s_x$ from the definitions $b_1 = S_{xy}/S_{xx}$ and $r = S_{xy}/\sqrt{S_{xx} S_{yy}}$.

11. (B) Show that $R^2 = r^2$ in simple linear regression (Theorem 4.4), starting from $\mathrm{SSR} = b_1^2 S_{xx}$ and $\mathrm{SSTO} = S_{yy}$.

12. (B) Prove that the two regression slopes satisfy $b_1 \cdot b_1' = r^2$, where $b_1'$ is the slope of $X$ on $Y$. Explain why this forces $|r| \le \sqrt{|b_1 b_1'|}$ to be an equality, not an inequality.

13. (B) Derive the correlation $t$ statistic $t^{\ast} = r\sqrt{n-2}/\sqrt{1-r^2}$ (Theorem 4.6) from the slope $t$ statistic $b_1/s\{b_1\}$, using $\mathrm{SSE} = S_{yy}(1 - r^2)$.

14. (B) Explain the delta-method argument behind Theorem 4.7: starting from $\operatorname{Var}(r) \approx (1-\rho^2)^2/n$, show that $g(\rho) = \operatorname{arctanh}(\rho)$ makes $\operatorname{Var}(g(r))$ approximately constant, and carry out the integral $\int d\rho/(1-\rho^2)$.

15. (B) In the bivariate normal, use the conditional-mean result (Theorem 4.9) to show that the population regression slope of $Y$ on $X$ is $\rho\,\sigma_Y/\sigma_X$, and that the fraction of $Y$’s variance left after conditioning on $X$ is $1 - \rho^2$.

16. (B) Show that if every $Y_i = a + b X_i$ exactly (a perfect line, $b \ne 0$), then $r = \operatorname{sign}(b)$. Which of the bound-derivation’s equality conditions does this meet?

17. (B) A sample has $r = 0.6$ with $n = 19$. Compute the 95 percent Fisher confidence interval for $\rho$ by hand, showing the $z$, the standard error, and the back-transformation.

18. (B) Prove that Spearman’s rank correlation equals +1 for any strictly increasing relationship $Y = h(X)$, not only a linear one, by considering what the ranks of $X$ and $Y$ look like.

19. (C) Load savings.csv and reproduce the full correlation matrix of sr, pop15, pop75, dpi, ddpi. Identify the strongest positive and strongest negative correlations and explain each in words.

20. (C) For sr and pop75, compute $r$, run the $t$ test, and build the Fisher 95 percent interval in R or Python. State whether the correlation is significant and interpret the interval.

21. (C) Confirm on the savings data that $b_1 = r\,s_y/s_x$ and $R^2 = r^2$ for the regression of sr on dpi. Report all four numbers.

22. (C) Compute both Pearson and Spearman correlations between dpi and sr. They differ; make the scatterplot and explain which countries drive the gap.

23. (C) Using anscombe.csv, reproduce the table showing all four sets share the same mean, correlation, and fitted line, then plot all four and describe how each departs (or not) from a straight line.

24. (C) Using galton_heights.csv, demonstrate restriction of range: compute the correlation on all children, then on the subset with midparent height between 68 and 70 inches, and report how much the correlation and the standard deviation of the parent height each shrink.

25. (C) Using galton_heights.csv, fit childHeight ~ midparentHeight, report the standardized slope, and explain the number as regression to the mean. Predict the height of a child whose midparent height is 3 standard deviations above the mean, in standard units.

26. (C) Simulate the sampling distribution of $r$ for $n = 15$ and $\rho = 0.6$ with 5000 draws (seed 4210), and check that the standard deviation of $\operatorname{arctanh}(r)$ is close to $1/\sqrt{n-3}$. Report both numbers.

27. (C) Aggregate galton_heights.csv by family and compare the individual-level and family-average correlations between midparent and child height. Explain the direction of the difference as an ecological effect.

28. (C) Take any pair from the savings data with a moderate correlation and delete the single most extreme point. Recompute $r$ and report how much it moved, then say whether that point qualifies as influential.

4.11 Exam practice¶

These five questions are written in the style of the course exams. Every one asks you to explain your reasoning in full sentences, not just to report a number, because that is how the exams are graded: a bare value earns little credit, and clear reasoning with a small slip earns most of it. Work each one on paper before you open the model answer. Where a question shows software output, it is genuine output from the book’s datasets.

EP 4.1 Interpret the output in context¶

For the 50 countries in savings.csv, the savings rate sr is correlated with the share of the population over 75, pop75. Here is the cor.test output.

	Pearson's product-moment correlation

data:  savings$sr and savings$pop75
t = 2.3118, df = 48, p-value = 0.02513
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 0.04186153 0.54670273
sample estimates:
      cor 
0.3165211 

Report the sample correlation and interpret its sign and size in the context of the two variables. Say whether the correlation is significantly different from zero at the 5 percent level and how you read that from the output. Interpret the confidence interval, explain in one sentence why it is not symmetric about the sample correlation, and say whether the result shows that having more elderly people makes a country save more.

EP 4.2 A student’s claim¶

A classmate runs the correlation between the savings rate sr and the income growth rate ddpi and gets the following.

r = 0.3048   r^2 = 0.0929   t = 2.2171   p = 0.031385

She writes: “The correlation between income growth and saving is statistically significant ($p = 0.03$), so income growth is a strong driver of national saving.” Evaluate her claim, correcting every error you find.

EP 4.3 What would change if¶

Across all 934 children in galton_heights.csv, the correlation between midparent height and child height is as follows.

full_r = 0.321   full_n = 934   full_sd(midparent) = 1.802

Suppose you recomputed this correlation using only the families whose midparent height is between 69.5 and 70.5 inches. Predict whether the correlation would rise, fall, or stay the same, explain the mechanism, and say whether the underlying parent-child relationship has changed.

EP 4.4 Explain why¶

For income growth ddpi against the savings rate sr, the two correlation coefficients disagree, and one country stands out.

pearson  = 0.3048   spearman = 0.4082
country    ddpi    sr
Libya     16.71   8.89

Explain why the Pearson and Spearman correlations differ here, using Libya to make the mechanism concrete. Then say which of the two numbers is “correct” and what their disagreement should prompt you to do.

EP 4.5 A student’s claim about regression to the mean¶

The regression of child height on midparent height in galton_heights.csv gives the following.

r = 0.321   slope = 0.637   sd(midparent) = 1.802   sd(child) = 3.579

A student reasons: “Children of tall parents are on average shorter than their parents, so tall families are reverting toward the average, and over enough generations everyone’s height will converge to the mean.” Evaluate the reasoning, using the standardized slope, and say what will actually happen to the spread of heights across generations.

Chapter game¶