Appendix A: Formula Summary

This appendix collects every named formula from Chapters 1 through 16, in the order each chapter introduces it. Each formula keeps the ALRM notation used throughout the book: $Y_i$ for a response, $X_i$ for a predictor, $\beta_0, \beta_1, \dots$ for population parameters, $b_0, b_1, \dots$ for their least-squares estimates, $e_i$ for a residual, $\mathbf{X}$ for the design matrix, and a prime for transpose. One plain-language line under each formula states what it claims, so this page works as a quick lookup during homework and exams rather than a replacement for the chapter’s derivation. For full context, definitions, and worked examples, follow the chapter link in each heading.

A.1 Chapter 1: Regression as a way of thinking¶

Anatomy of a statistical model

Any response splits into a systematic part $f(X)$ that depends on the predictor and a random part $\varepsilon$ that does not.

Simple linear regression function

The average response at a given $X$ falls on a straight line with intercept $\beta_0$ and slope $\beta_1$.

Fitted line (estimates)

The fitted line uses sample estimates $b_0$ and $b_1$ in place of the unknown $\beta_0$ and $\beta_1$ to predict a response from $X$.

Galton’s fitted height line

A child’s predicted height rises by about 0.637 inch for each extra inch of average parent height, the historical example that gave regression its name.

A.2 Chapter 2: Simple linear regression¶

SLR model

Each observed response is the true line at $X_i$ plus a random error $\varepsilon_i$.

Error assumptions

Errors average to zero, share one variance $\sigma^2$ across all observations, and do not correlate with each other.

Mean response

The population line traces the average of $Y$ at each value of $X$.

Least-squares criterion

Least squares picks the intercept and slope that make the sum of squared vertical gaps to the data as small as possible.

Deviation sums

$S_{xx}$ measures spread in $X$, $S_{yy}$ measures spread in $Y$, and $S_{xy}$ measures how $X$ and $Y$ move together; every least-squares formula below is built from these three numbers.

Normal equations

Setting the calculus derivatives of $Q$ to zero gives two linear equations in $b_0$ and $b_1$ whose solution is the least-squares line.

Slope estimate

The estimated slope is the covariance of $X$ and $Y$ scaled by the spread of $X$.

Intercept estimate

The estimated intercept forces the fitted line through the point of averages $(\bar X, \bar Y)$.

Fitted value

The fitted value is where the estimated line sits above (or below) observation $i$’s predictor value.

Residual

The residual is the vertical gap between an observed response and its fitted value.

Residual identities

The least-squares residuals sum to zero, are uncorrelated with $X$ and with the fitted values, and preserve the total of $Y$, four automatic checks on any regression fit.

Error sum of squares

$\mathrm{SSE}$ totals the squared residuals, the leftover variation the line does not explain.

Error variance estimate

Dividing $\mathrm{SSE}$ by $n-2$, the degrees of freedom left after estimating two parameters, gives an unbiased estimate of the error variance $\sigma^2$.

Unbiasedness of MSE

On average, across repeated samples, $\mathrm{MSE}$ hits the true error variance exactly.

Mean and variance of $b_1$

The slope estimate is unbiased, and it grows less variable as the predictor spreads out more.

Mean and variance of $b_0$

The intercept estimate is unbiased, with variance that grows if $\bar X$ sits far from zero.

Estimated standard errors

Replacing $\sigma^2$ with $\mathrm{MSE}$ in the variance formulas gives the standard errors software reports.

$b_1$ as a linear estimator

The slope estimate is a weighted sum of the responses, which is the fact the Gauss-Markov proof builds on.

Normal likelihood

Under normal errors, this product measures how plausible a candidate $(\beta_0, \beta_1, \sigma^2)$ is given the observed data.

MLE of variance

Maximizing the likelihood over $\sigma^2$ gives a variance estimate that divides by $n$ instead of $n-2$, which is why it is slightly biased low.

A.3 Chapter 3: Inference for simple linear regression¶

Sampling distribution of $b_1$

Across repeated samples, the slope estimate follows a normal distribution centered at the true slope.

Sampling distribution of $b_0$

The intercept estimate is likewise normal and unbiased, with the variance derived in Chapter 2.

$t$ statistic for the slope

Standardizing the slope estimate by its estimated standard error gives a $t$ distribution with $n-2$ degrees of freedom, the basis for every slope interval and test.

Confidence interval for the slope

This interval is a range of plausible values for the true slope $\beta_1$, built from the estimate plus or minus a $t$ multiple of its standard error.

Slope test statistic

This ratio measures how many standard errors the estimated slope sits from zero, used to test whether $X$ predicts $Y$ at all.

Estimated standard error of the slope

This is the number software reports next to $b_1$, an estimate of how much the slope would bounce around across repeated samples.

Estimated standard error of the intercept

This is the corresponding uncertainty measure for the intercept estimate.

Variance of the fitted mean

The uncertainty in a fitted mean response grows the farther $X_h$ sits from the center of the data.

Confidence interval for a mean response

This interval traps the true average response at $X_h$, not a single new observation.

Variance of a prediction

Predicting one new observation carries an extra unit of variance beyond estimating the mean, because a single case also carries its own random error.

Prediction interval for a new observation

This wider interval is built to cover one future observation at $X_h$, not the population average.

ANOVA decomposition

Total variation in $Y$ splits exactly into the part the line explains ($\mathrm{SSR}$) and the part it leaves over ($\mathrm{SSE}$).

Mean squares

Dividing each sum of squares by its degrees of freedom converts it into an average squared contribution, ready to compare in an $F$ ratio.

Regression sum of squares

$\mathrm{SSR}$ measures how far the fitted values swing away from the overall mean, the credit the line earns for tracking $Y$.

Expected mean squares

$\mathrm{MSR}$ only exceeds $\mathrm{MSE}$ on average when the true slope is nonzero, which is the logic behind the $F$ test.

$F$ test statistic

A large ratio of explained to unexplained variance is evidence against the null hypothesis that $\beta_1 = 0$.

$F$ equals $t$ squared

In simple regression the $F$ test for the slope and the two-sided $t$ test for the slope are the same test written two ways.

Coefficient of determination

$R^2$ is the proportion of total variation in $Y$ that the line explains, between 0 and 1.

$R$-squared equals $r$-squared

In simple regression, the coefficient of determination is exactly the squared Pearson correlation between $X$ and $Y$.

Working-Hotelling band

This band covers the whole regression line at every $X_h$ simultaneously, wider at each single point than the pointwise confidence interval.

A.4 Chapter 4: Correlation¶

Sample covariance

Covariance measures the average product of how far $X$ and $Y$ each stray from their own means, positive when they tend to rise together.

Pearson correlation

Correlation rescales covariance by both standard deviations so the result never depends on the units of $X$ or $Y$.

Bounds of $r$

Correlation always lands between perfect negative and perfect positive linear association.

Correlation and slope

The regression slope and the correlation carry the same sign and convert into each other once you rescale by the ratio of standard deviations.

Determination equals $r$ squared

Squaring the correlation gives the share of variation the line explains.

Product of the two regression slopes

Regressing $Y$ on $X$ and regressing $X$ on $Y$ give two different lines whose slopes multiply to $r^2$, a reminder that “the” regression line depends on which variable you predict.

Correlation $t$ test

This statistic tests whether the true correlation $\rho$ is zero, and it equals the slope’s $t$ test exactly.

Fisher $z$ transformation

This transform straightens out $r$’s skewed sampling distribution into an approximately normal one, which makes confidence intervals for $\rho$ possible.

Confidence interval for $\rho$

Build the interval on the $z$ scale, then transform both endpoints back to the correlation scale.

Variance of $r$ (large sample)

Correlation estimates get more precise with more data and, for a fixed $n$, are least variable when the true correlation is near zero.

Bivariate normal density

This surface describes two jointly normal variables at once; its five parameters are the two means, two standard deviations, and the correlation $\rho$.

Bivariate normal conditional mean

Under bivariate normality, the average of $Y$ given $X = x$ is exactly a straight line in $x$, which is why simple linear regression is the natural model here.

Bivariate normal conditional variance

Knowing $X$ shrinks the spread of $Y$ by a factor of $1-\rho^2$, the same factor that shows up as $1 - R^2$ in regression.

Spearman rank correlation

Replacing each value with its rank before computing Pearson’s $r$ gives a correlation that detects any monotone relationship, not only a straight-line one.

A.5 Chapter 5: Randomization and bootstrap inference for regression¶

Slope estimate

The same least-squares slope from Chapter 2 is the statistic this chapter re-derives its uncertainty for, without assuming normal errors.

No-association null

The permutation test’s null hypothesis is that the pairing between $X_i$ and $Y_i$ carries no information, so shuffling one against the other should not matter.

Permuted slope

Recomputing the slope after randomly reshuffling the $Y$ values against the $X$ values gives one draw from the null distribution.

Exact permutation $p$-value

The exact $p$-value is the fraction of all possible shuffles that produce a slope as extreme as the one actually observed.

Monte Carlo permutation $p$-value

Since checking all $n!$ shuffles is usually impossible, this estimate uses $B$ random shuffles, with $M$ of them at least as extreme as the observed slope.

Permutation resolution floor

With only $B$ permutations, the smallest reportable $p$-value is $1/(B+1)$, so a very small true $p$-value needs many permutations to resolve.

Case-resampling bootstrap slope

Each bootstrap replicate refits the slope on a resample of whole $(X_i, Y_i)$ pairs drawn with replacement from the original data.

Residual-resampling bootstrap

This alternative bootstrap keeps the $X$ values fixed and builds new responses by adding randomly resampled residuals to the fitted line.

Bootstrap standard error

The spread of the bootstrap slope estimates across replicates estimates the standard error of $b_1$ without any normal-error formula.

Percentile interval

This interval simply reads off the lower and upper quantiles of the bootstrap distribution of $b_1^{\ast}$.

Basic (reflection) interval

This interval reflects the bootstrap quantiles around the original estimate, which corrects for skew that the plain percentile interval can miss.

Classical $t$ interval for slope

The Chapter 3 formula reappears here as the benchmark the permutation and bootstrap intervals are compared against.

A.6 Chapter 6: Matrix algebra for regression¶

Model in matrix form

Stacking every observation into vectors and matrices writes the whole regression model, with any number of predictors, in one equation.

Design matrix dimensions

$\mathbf{X}$ has one row per observation and one column per parameter, including a column of ones for the intercept, so $p$ counts parameters and $n$ counts observations.

Matrix product entry

Each entry of a matrix product is the dot product of a row of $\mathbf{A}$ with a column of $\mathbf{B}$.

Transpose of a product

Transposing a product reverses the order of the factors.

Cross-product matrix

Each entry of $\mathbf{X}'\mathbf{X}$ is a sum of products between two columns of $\mathbf{X}$, summarizing how the predictors relate to each other and to themselves.

Cross-product vector

This vector summarizes how each predictor column relates to the response.

Symmetry of $\mathbf{X}'\mathbf{X}$

The cross-product matrix always equals its own transpose, a fact the least-squares derivation relies on.

2x2 determinant and inverse

For a two-by-two matrix, the determinant $ad - bc$ tells you whether an inverse exists, and this formula hand-computes it when it does.

Matrix normal equations

The two-equation normal equations from Chapter 2 collapse into this single matrix equation that works for any number of predictors.

Least-squares estimator

Inverting $\mathbf{X}'\mathbf{X}$ and multiplying by $\mathbf{X}'\mathbf{Y}$ solves the normal equations in one step, giving every least-squares coefficient at once.

Hat matrix

$\mathbf{H}$ is the matrix that turns the observed responses directly into fitted values, “putting a hat on” $\mathbf{Y}$.

Fitted values and residuals

The hat matrix produces the fitted values directly, and what remains after subtracting them from $\mathbf{Y}$ is the residual vector.

Idempotency of $\mathbf{H}$

Applying the hat matrix twice does nothing more than applying it once, and its diagonal entries sum to $p$, the number of parameters.

SSE as a quadratic form

The error sum of squares can be written directly as a quadratic form in $\mathbf{Y}$, or equivalently as the squared length of the residual vector.

SS decomposition (matrix)

This matrix identity is the geometric version of $\mathrm{SSTO} = \mathrm{SSR} + \mathrm{SSE}$: three projection matrices that add up exactly.

Variance estimate

With $p$ parameters instead of two, the error variance estimate now divides by $n-p$ degrees of freedom.

Expectation of a linear map

Expectation passes through a fixed matrix and a fixed vector added afterward, the matrix analogue of $E\{aW + c\} = aE\{W\} + c$.

Covariance of a linear map

Transforming a random vector by a fixed matrix sandwiches the original covariance matrix between that matrix and its transpose.

Unbiasedness of $\mathbf{b}$

On average, the least-squares coefficient vector recovers the true parameter vector exactly.

Covariance of $\mathbf{b}$

This matrix holds the variance of every coefficient on its diagonal and the covariance between pairs of coefficients off the diagonal.

Estimated covariance of $\mathbf{b}$

Substituting $\mathrm{MSE}$ for $\sigma^2$ gives the covariance matrix software actually reports.

Sampling law of $\mathbf{b}$

Under normal errors, the whole coefficient vector follows a multivariate normal distribution centered at the truth.

Multivariate normal of $\mathbf{Y}$

The response vector itself is multivariate normal, with mean given by the regression surface and independent equal-variance errors.

A.7 Chapter 7: The general linear model¶

General linear model

The same matrix model from Chapter 6 is now the standing model for every remaining chapter, covering any number and kind of predictor.

Error moments

Errors average to zero, share one variance, and do not correlate with each other, the matrix form of the Chapter 2 assumptions.

Sum of squares objective

This is the squared length of the residual vector, the quantity least squares minimizes over $\mathbf{b}$.

Normal equations

Setting the gradient of $Q(\mathbf{b})$ to zero gives this equation, whose solution is the least-squares estimate.

Least-squares estimator

This is the same closed-form solution from Chapter 6, now the working formula for every model in the rest of the book.

Orthogonality of residual

The residual vector is perpendicular to every column of $\mathbf{X}$, the algebraic reason the residuals are uncorrelated with each predictor.

Hat matrix

$\mathbf{H}$ projects the response vector onto the column space of $\mathbf{X}$, the geometric meaning behind “fitting” a regression.

Fitted values

Applying the hat matrix to the data produces the fitted values, the closest point to $\mathbf{Y}$ that the model can reach.

Residual vector

The residuals are what is left of $\mathbf{Y}$ after removing the part the model can explain.

Hat matrix trace

The hat matrix’s diagonal sums to $p$, the number of fitted parameters, which leaves $n-p$ degrees of freedom for estimating error.

Mean of $\mathbf{b}$

Least squares is unbiased for any design matrix $\mathbf{X}$ of full column rank.

Covariance of $\mathbf{b}$

This is the general-model version of the coefficient covariance matrix, used to build every standard error, confidence interval, and test in multiple regression.

Sampling distribution of $\mathbf{b}$

Under normal errors, the coefficient vector is exactly multivariate normal, which underlies every $t$ and $F$ procedure in multiple regression.

Error sum of squares

$\mathrm{SSE}$ is the squared length of the residual vector, written two equivalent ways.

Error variance estimator

Dividing $\mathrm{SSE}$ by $n-p$ gives an unbiased estimate of $\sigma^2$ for a model with $p$ parameters.

Unbiasedness of MSE

This expectation is the algebraic reason dividing by $n-p$, not $n$, makes $\mathrm{MSE}$ unbiased.

Quadratic form expectation

This general tool computes the expected value of any quadratic form in a random vector, the technique behind proving $E\{\mathrm{SSE}\} = (n-p)\sigma^2$.

Estimated covariance of $\mathbf{b}$

Every coefficient’s standard error, in R’s or Python’s regression output, is the square root of a diagonal entry of this matrix.

Coefficient of determination

$R^2$ still measures the proportion of total variation explained, now for a model with any number of predictors.

A.8 Chapter 8: Multiple regression in practice¶

Multiple linear regression model

Each response depends on several predictors at once, with $p-1$ slopes and one intercept, for $p$ parameters total.

Partial slope (least squares)

Each entry of $\mathbf{b}$ is a partial slope: the effect of one predictor holding the others in the model fixed.

Error variance estimate

This is the multiple-regression version of the error variance estimate from Chapter 7, with $p$ counting every predictor plus the intercept.

Extra sum of squares

This measures how much extra variation $X_2$ explains once $X_1$ is already in the model.

Sequential decomposition of SSR

The total explained variation splits into a chain of extra contributions, one predictor added at a time in a chosen order.

General linear test statistic

Comparing a reduced model $R$ against a full model $F$, this ratio tests whether the extra predictors in $F$ improve the fit by more than chance.

Pythagoras for nested models

The reduced model’s error splits into the full model’s error plus a right-angle piece measuring the extra distance the bigger model’s fit travels.

One-coefficient test equals $t$ squared

Testing one coefficient with the general linear test gives the same answer as the usual $t$ test, squared.

Added-variable slope equals coefficient

Regressing the residuals of $Y$ against the residuals of $X_k$, after removing every other predictor from both, recovers $X_k$’s own coefficient $b_k$.

Coefficient of determination

$R^2$ measures the share of total variation the full set of predictors explains together.

Adjusted $R$ squared

This version penalizes adding predictors that do not pull their weight, so it can fall even when plain $R^2$ rises.

Standardized coefficient

Rescaling a coefficient by the ratio of standard deviations puts predictors measured in different units on a common scale for comparing their impact.

A.9 Chapter 9: Model diagnostics¶

Hat matrix

The same projection matrix from Chapter 7 now supplies the diagonal entries that flag unusual predictor values.

Leverage value

An observation’s leverage value measures how far its predictor values sit from the center of the predictor space, and how much pull it has on its own fitted value.

Leverage value bounds and sum

Every leverage value falls between zero and one, and the leverage values across all observations always add up to the number of parameters.

Leverage value in simple regression

In simple regression, the leverage value grows with squared distance from $\bar X$, confirming that points far from the center of $X$ carry more influence.

Leverage value cutoff

This rule flags an observation as a high-leverage point when its leverage value exceeds twice the average leverage value of $p/n$.

Residual covariance

Residuals do not all share the same variance; high-leverage points have smaller residual variance because the fit is pulled closer to them.

Semistudentized residual

Dividing a residual by the overall residual standard deviation gives a rough check for outliers that ignores the varying leverage values across observations.

Standardized residual

This version corrects for each observation’s own leverage value, giving residuals that are more directly comparable across the dataset.

Studentized deleted residual

This residual refits the model without observation $i$ before standardizing it, which gives an exact $t$ distribution for flagging outliers.

Deleted variance identity

This shortcut computes the error variance from the model with observation $i$ removed, without actually refitting.

Deleted residual

This is the gap between observation $i$’s response and the prediction a model trained without it would make.

Bonferroni outlier cutoff

Testing every observation for being an outlier at once inflates the false-positive rate, so this cutoff tightens the threshold by a factor of $n$ to compensate.

Leave-one-out coefficient update

This formula gives the exact shift in every coefficient caused by removing one observation, without refitting the model.

Sherman-Morrison downdate

This linear algebra identity updates a matrix inverse after removing one row, the engine behind every leave-one-out formula in this chapter.

Cook’s distance

Cook’s distance combines how large a residual is with how large its leverage value is, into a single number measuring an observation’s overall influence on all the fitted values.

DFFITS

This measures how many standard errors observation $i$’s own fitted value shifts when that observation is removed.

DFBETAS

This measures how many standard errors one specific coefficient shifts when observation $i$ is removed, pinpointing which parameter a case influences most.

Durbin-Watson statistic

This statistic compares consecutive residuals to detect leftover autocorrelation, with values near 2 indicating none.

Lack-of-fit decomposition

When some predictor values repeat, the error sum of squares splits into pure error, unavoidable scatter at repeated $X$’s, and lack of fit, systematic bias from a wrong functional form.

Pure error sum of squares

This totals the scatter of responses around their own group mean at each repeated predictor value, variation no model could ever explain.

Lack-of-fit sum of squares

This totals the gap between each group’s actual mean response and what the fitted line predicts there, the part attributable to a wrong model shape.

Lack-of-fit $F$ test

A large ratio here is evidence that the straight-line model misses real curvature or structure in the data.

A.10 Chapter 10: Remedial measures and transformations¶

Log-log model

Taking logs of both variables fits a model where the relationship is a straight line on the log scale, appropriate for multiplicative, curved relationships.

Log-linear model

Taking a log of only the response fits exponential growth or decay in the original units.

Log-log exact factor

Multiplying $X$ by a constant multiplies the predicted $Y$ by that constant raised to the power $\beta_1$, the defining property of a log-log model.

Log-log elasticity (approx)

The slope in a log-log model is approximately the percent change in $Y$ per one percent change in $X$, an elasticity.

Log-linear exact factor

Adding one unit to $X$ multiplies predicted $Y$ by $e^{\beta_1}$, which converts to a percent change.

Box-Cox family

This one family of transformations includes the raw scale ($\lambda=1$), the log scale ($\lambda=0$), and everything between, so the data can pick the best power.

Box-Cox full log-likelihood

This likelihood scores how well a given transformation power $\lambda$ makes the transformed data fit a normal-errors regression.

Box-Cox profile log-likelihood