13. Logistic regression

On the evening of January 27, 1986, engineers at Morton Thiokol and NASA argued on a conference call about whether to launch the space shuttle Challenger the next morning. The forecast for Cape Canaveral was cold, near freezing, colder than any previous launch. The concern was the rubber O-rings that sealed the joints of the solid rocket boosters. In the cold the rubber stiffens, and a stiff O-ring might not seal in time to hold back the burning gas. The engineers had data from 23 earlier flights: for each one, the temperature at launch and the number of O-rings, out of six, that showed heat damage afterward.

Figure 1 shows that data. Damage happened on both warm and cool flights, but the coldest launches carried the worst incidents, and the launch under discussion would be far colder than anything on the record. That is the hard part. Every flight that had ever flown did so at 53 degrees Fahrenheit or warmer. The predicted launch temperature was about 31 degrees, off the left edge of all experience. To say anything about the risk at 31 degrees, you have to extend a pattern past the last data point, which is exactly the move a statistician is trained to distrust.

The response here is not a number like work hours or savings rate. It is closer to a yes-or-no: did an O-ring fail or not. Straight-line regression, the tool of the last eleven chapters, is built for a continuous response and breaks in specific ways when the outcome is binary. This chapter builds the right tool. You will learn the logistic regression model, the odds and log-odds that make it linear, how to fit it by maximum likelihood when no formula gives the estimates, how to read its coefficients as odds ratios without the usual mistakes, how to test them, and how to judge how well the model separates the two outcomes. By the end you will be able to put a number on the risk the Challenger engineers were arguing about, and to say honestly how much to trust it.

13.1 Why a straight line fails, and what to use instead¶

Intuition¶

Start with what goes wrong. Suppose you code the response as $Y = 1$ for a positive outcome (an O-ring damaged, a diabetes test positive) and $Y = 0$ otherwise, and you fit the ordinary line $Y = \beta_0 + \beta_1 X + \varepsilon$. Three things break.

First, the fitted line is unbounded. A line keeps rising as $X$ grows, so it eventually predicts probabilities above 1 and, going the other way, below 0. Figure 2 shows the least-squares line through the diabetes data sliding straight out of the $[0, 1]$ range where a probability has to live.

Second, the spread is not constant. A $0/1$ response has variance $\pi(1 - \pi)$, where $\pi$ is the probability that $Y = 1$. That variance is largest near $\pi = 0.5$ and shrinks toward zero as $\pi$ approaches 0 or 1, the upside-down parabola in Figure 3. A coin near 50-50 is the hardest to call, so its outcome is the most variable; a coin that almost always lands heads is nearly a sure thing, so it barely varies. The spread is fixed by the mean. So the constant-variance assumption behind least squares (2.1 The simple linear regression model) is false by construction, and the weighting that would fix it changes with the mean, an idea we met in weighted least squares (10.4 Weighted least squares).

Third, the errors cannot be normal. If $Y$ is only ever 0 or 1, then for a given $X$ the “error” takes just two values, so the normal error model that justified our $t$ and $F$ inference simply does not apply.

The fix is to model the probability $\pi$ directly and to bend the line so it can never leave $[0, 1]$. The bend is the S-shaped logistic curve in the figure.

From probability to odds to log-odds¶

The bridge is the odds (Definition 13.1).

A probability of 0.5 is odds of 1 (even money). A probability of 0.75 is odds of 3 (three to one). As $\pi$ climbs toward 1 the odds shoots off to infinity, and as $\pi$ falls toward 0 the odds falls toward 0 but never goes negative. Figure 4 draws this mapping. Odds stretch the bounded $[0, 1]$ probability scale onto the half-line $[0, \infty)$.

Taking one more step, the log-odds or logit (Definition 13.2) is the natural logarithm of the odds:

In words: the logit is the log of the odds. The logarithm sends $[0, \infty)$ onto the whole real line $(-\infty, \infty)$, the same range a linear predictor $\beta_0 + \beta_1 X$ can take. That is the whole trick. We cannot set a probability equal to a line, because one is bounded and the other is not, but we can set the logit of the probability equal to a line.

The logistic regression model¶

The logistic regression model (Definition 13.3) for a binary response says the log-odds is linear in the predictors:

where $\pi_i = P(Y_i = 1 \mid X_i)$ is the probability that case $i$ has a positive outcome, the $\beta$’s are the regression parameters (with $p$ of them counting the intercept), and $X_{i1}, \dots, X_{i,p-1}$ are the predictors for case $i$. Writing $\eta_i = \beta_0 + \beta_1 X_{i1} + \dots$ for the linear predictor, we solve the logit equation for $\pi_i$ to get the model on the probability scale:

In words: the probability is the linear predictor passed through the logistic function $1/(1 + e^{-\eta})$, the S-curve of Figure 5. The function is squashed to lie strictly between 0 and 1, so no matter what the coefficients are, the predicted probability is always a legal probability.

The two links in the term generalized linear model are visible here. There is a random part, $Y_i$ being 0 or 1 with probability $\pi_i$, and a systematic part, the linear predictor $\eta_i$, joined by the link function logit that maps the mean onto the linear scale. Ordinary regression is the same picture with the identity link and a normal response; Chapter 14 makes the family explicit and adds the Poisson member (14.6 One family: the generalized linear model).

R and Python¶

For the O-ring data the response is grouped: each flight contributes six O-rings, of which $Y_i$ were damaged. That is a binomial count, and logistic regression handles it with the same machinery, modeling the probability $\pi_i$ that a single O-ring is damaged at temperature $X_i$. In R you pass the two-column response cbind(successes, failures) to glm with family = binomial; in Python you give statsmodels the two counts on the left of the formula.

That 0.99 is easier to weigh honestly once you have walked the curve down there yourself.

What to notice. Inside the data the curve is gentle, but the moment you cross 53 degrees into the shaded strip the fitted probability climbs from 0.550 to 0.993 with no observation underneath it. Try this. Park the slider at 31 degrees, read the “none: extrapolating” label, and ask what would have to stay true about the world for that number to hold. Back to Section 13.1.

13.2 Fitting by maximum likelihood¶

Intuition¶

In simple regression we had formulas: $b_1 = S_{xy}/S_{xx}$ and $b_0 = \bar Y - b_1 \bar X$. Logistic regression has no such closed form. The reason is the S-curve: the probability $\pi_i$ is a nonlinear function of the coefficients, so setting derivatives to zero gives equations you cannot solve with algebra. Instead we choose the coefficients that make the observed data most probable, the principle of maximum likelihood we first met for the normal model in Chapter 2, and we find them by climbing the likelihood hill numerically.

The likelihood and log-likelihood¶

Let case $i$ have $m_i$ trials and $Y_i$ successes (for a plain binary response $m_i = 1$ and $Y_i \in \{0, 1\}$; for the O-rings $m_i = 6$). Given the probabilities $\pi_i$, the successes are independent binomial counts, so the likelihood, the probability of the observed data as a function of the coefficients, is

where each $\pi_i = 1/(1 + e^{-\eta_i})$ depends on $\beta$ through the linear predictor $\eta_i = X_i'\beta$. In words: the likelihood multiplies together, across all cases, the binomial chance of seeing exactly the successes we observed, so larger values point to coefficients the data favor. Taking logs turns the product into a sum, the log-likelihood:

where $C$ collects the binomial coefficients $\log\binom{m_i}{Y_i}$, which do not involve $\beta$ and so do not affect where the maximum sits. In words: the log-likelihood rewards coefficients that put high probability on the successes we saw and high non-probability on the failures we saw.

The score equations¶

Proof. We want the $\beta$ that maximizes $\ell$, so we differentiate and set the result to zero. Two facts about the logistic link make the algebra collapse. First, from $\pi_i = e^{\eta_i}/(1 + e^{\eta_i})$,

Substituting these into $\ell$ and collecting terms,

Second, differentiate $\log(1 + e^{\eta_i})$ with respect to $\eta_i$ to get exactly $\pi_i$. Using the chain rule with $\partial \eta_i / \partial \beta_j = X_{ij}$ (the $j$-th predictor value for case $i$, with $X_{i0} = 1$ for the intercept),

Setting all $p$ partial derivatives to zero gives the score equations. Stacking them with the design matrix $\mathbf{X}$ (rows $X_i'$), the fitted mean vector $\boldsymbol{\mu}$ with entries $\mu_i = m_i \pi_i$, and the response vector $\mathbf{Y}$,

In words: at the maximum-likelihood estimate, each predictor is orthogonal to the residuals $Y_i - \mu_i$, exactly the condition the least-squares normal equations $\mathbf{X}'(\mathbf{Y} - \mathbf{X}\mathbf{b}) = \mathbf{0}$ imposed in 7.1 The model and least squares in matrix form. The difference is that $\boldsymbol{\mu}$ here bends through the logistic link, so the equations are nonlinear in $\beta$ and cannot be solved in closed form. $\blacksquare$

Newton-Raphson and IRLS¶

Proof. To solve $\mathbf{X}'(\mathbf{Y} - \boldsymbol{\mu}) = \mathbf{0}$ we use Newton’s method, which needs the matrix of second derivatives. Differentiating the score once more, and using $\partial \pi_i / \partial \eta_i = \pi_i(1 - \pi_i)$,

so the Hessian is $-\mathbf{X}'\mathbf{W}\mathbf{X}$ with the diagonal weight matrix $\mathbf{W} = \operatorname{diag}\!\big(m_i \pi_i (1 - \pi_i)\big)$. Each weight $w_i = m_i \pi_i(1 - \pi_i)$ is the variance of $Y_i$, largest for cases near $\pi_i = 0.5$ and small for cases the model is already sure about. A Newton step from a current guess $\beta^{(t)}$ is

with $\mathbf{W}$, $\boldsymbol{\mu}$ evaluated at $\beta^{(t)}$. Now the tidy part. Define the working response $z_i = \eta_i + (Y_i - \mu_i)/w_i$. Then

using $\boldsymbol{\eta} = \mathbf{X}\beta^{(t)}$. Multiplying the Newton step by $\mathbf{X}'\mathbf{W}\mathbf{X}$ shows it is equivalent to

That is the formula for a weighted least squares fit of the working response $\mathbf{z}$ on $\mathbf{X}$ with weights $w_i$ (10.4 Weighted least squares). Because $\mathbf{W}$ and $\mathbf{z}$ change each time $\beta$ updates, we recompute and refit, which is why the method is called iteratively reweighted least squares (IRLS). Starting from any reasonable guess, the iterates climb the concave log-likelihood to its single maximum. $\blacksquare$

R and Python¶

The centerpiece is worth doing by hand once, so IRLS stops being a black box inside glm.

IRLS climbs the log-likelihood in six passes; climb it by hand once and the algorithm stops being a black box.

What to notice. The log-likelihood is always negative and has a single peak, so every move away from the best setting costs you something, which is the concavity the proof relied on. Try this. Get the log-likelihood as close to zero as you can by hand, then set $b_1 = 0$ and watch the S-curve flatten and the odds ratio drop to 1. Back to Section 13.2.

13.3 Reading coefficients as odds ratios¶

Intuition¶

A slope in ordinary regression is easy to say out loud: one more unit of $X$ adds $b_1$ to the predicted response. The logistic slope is not that, and saying it that way is the single most common mistake in applied logistic regression. The coefficient lives on the log-odds scale, so before interpreting it we have to undo the log.

Here is the wrong reading first, because you will hear it constantly. For the diabetes model below, the glucose coefficient is about 0.04. The tempting sentence is “each extra unit of glucose raises the probability of a positive test by 0.04.” That is wrong twice over. The coefficient is not a change in probability at all, and the effect on probability is not even constant: it depends on where you start on the S-curve, tiny out in the flat tails and largest in the steep middle. What is constant is the effect on the odds. Figure 10 makes the split visible: one fixed step along the curve gives three different probability jumps but always the same odds multiplier.

Formula¶

Exponentiate the coefficient. Because $\log(\text{odds}) = \beta_0 + \beta_1 X$, raising $X$ by one unit changes the log-odds by $\beta_1$, so it multiplies the odds by $e^{\beta_1}$:

The quantity $e^{\beta_1}$ is the odds ratio (Definition 13.6) for a one-unit increase in $X$. In words: a one-unit rise in $X$ multiplies the odds of a positive outcome by $e^{\beta_1}$, whatever the starting odds were. An odds ratio of 1 means no effect; above 1 the odds go up, below 1 they go down. This is the same “logs make it multiplicative” logic as the log-transformed response in 10.2 The log transformation and reading its coefficients, moved onto the odds scale. For a step of $c$ units the odds ratio is $e^{c\beta_1}$, which is how you report an effect per 10 units or per decade of age.

R and Python¶

Figure 11 shows the single-predictor fit. The probability climbs smoothly with glucose, steeply through the middle of the range and flattening at both ends, the S-curve doing its job.

The split between a constant odds ratio and a moving probability change is worth watching move under your finger.

What to notice. Hold $c$ at 40 and slide the starting point: the change in probability runs from about 0.12 at glucose 60 up past 0.33 near glucose 100 and back down, while the odds ratio never leaves 5.080. Try this. Hunt for the starting glucose with the largest probability jump, and notice it sits in the steep middle of the S-curve, never in a tail. Back to Section 13.3.

13.4 Testing coefficients: Wald and likelihood-ratio¶

Intuition¶

The first section fit the model; the obvious next question is which predictors are pulling their weight. There are two standard tests, and they answer the same question by different routes. The Wald test asks how many standard errors a coefficient sits from zero, using the fitted model alone. The likelihood-ratio test asks how much the fit gets worse when you delete the predictor and refit, comparing two nested models the way the general linear test compared them for ordinary regression (8.4 The general linear test).

Formula¶

For the Wald test of $H_0: \beta_j = 0$, the statistic is

In words: the Wald statistic counts how many standard errors the estimate $b_j$ sits from zero, and under the null that distance follows a standard normal curve. Here $s\{b_j\}$ is the standard error printed by the fit. You can compare $z_j^2$ to a chi-square with one degree of freedom instead, which gives the same test. The Wald confidence interval for $\beta_j$ is $b_j \pm z^* s\{b_j\}$, and exponentiating its endpoints gives a confidence interval for the odds ratio. These standard errors and the normal reference are large-sample approximations, exact only as $n \to \infty$. With a small sample or a near-perfect predictor they can mislead.

The likelihood-ratio test uses the deviance (Definition 13.7). For a fitted model with maximized log-likelihood $\ell(\hat\beta)$, the deviance is

where $\ell_{\text{sat}}$ is the log-likelihood of the saturated model that fits every observation perfectly ($\hat\mu_i = Y_i$), and $\hat\mu_i = m_i \hat\pi_i$ are the fitted counts.

In words: the deviance measures how far the fitted model sits from a perfect fit, smaller being better; it is the logistic stand-in for the residual sum of squares. To compare a full model to a nested reduced model that drops $q$ predictors, the likelihood-ratio statistic is the increase in deviance,

with $q$ the number of coefficients set to zero. The special case comparing the full model to the intercept-only model uses the null deviance as $D_{\text{reduced}}$ and tests whether any predictor matters at all.

R and Python¶

Figure 13 plots the odds ratios and their intervals on a log scale, each rescaled to an interpretable step. An interval that clears the vertical line at 1 flags a predictor distinguishable from no effect; age’s interval straddles it.

The Wald and likelihood-ratio tests usually agree, but not always. The likelihood-ratio test is generally the more reliable of the two, because it uses the actual shape of the log-likelihood rather than a single quadratic approximation at the estimate. In the awkward case of a very large coefficient (a predictor that nearly separates the two outcomes), the Wald standard error can inflate and pull its $z$ toward zero, hiding a strong effect. The likelihood-ratio test does not suffer that failure. When the two disagree, trust the likelihood-ratio test. Figure 14 shows the reason in one picture: the Wald test replaces the true log-likelihood with a symmetric parabola matched at the peak, and when the true curve is lopsided the parabola drifts away from it, so the two tests measure different drops down to the null.

13.5 How good is the fit? Classification, ROC, and data checks¶

Intuition¶

A model can have significant coefficients and still classify poorly, so the last step is to ask how well the fitted probabilities actually separate the two outcomes. We keep this light: a classification table, an ROC curve, and, first, a look at whether the data deserve trust at all.

Data first: the disguised zeros¶

Recall the impossible zeros from 13.3 Reading coefficients as odds ratios. Figure 15 counts them. Serum insulin is zero for 374 of 768 women and skinfold thickness for 227; a living person has neither. These are missing values recorded as zeros, and a model that swallows them whole will estimate, for instance, a nonsense insulin slope driven by a spike of fake zeros. This is why we set glucose and BMI zeros to NA before fitting and why we left insulin and triceps out of the model entirely. The lesson generalizes: look at your predictors before you trust any coefficient, because the software will fit whatever you feed it.

Classification table and ROC¶

To turn probabilities into yes-or-no predictions, pick a threshold (0.5 is the default) and predict positive when $\hat\pi_i$ exceeds it. Cross-tabulating predictions against truth gives a classification table, from which sensitivity (the fraction of true positives caught) and specificity (the fraction of true negatives correctly cleared) follow (Definition 13.8). Because one threshold is an arbitrary choice, the ROC curve sweeps every threshold at once, plotting sensitivity against one-minus-specificity; the area under the curve (AUC) summarizes the whole sweep as the probability that the model scores a random positive above a random negative (Definition 13.9).

The 0.5 in that table is a choice you made, not something the model handed you, so the fair thing to do is move it.

What to notice. Dropping the threshold from 0.50 to 0.30 lifts sensitivity from 0.568 to 0.784 and cuts the missed positives from 114 to 57, while false alarms climb from 57 to 140 and accuracy falls. Try this. Find the threshold with the highest accuracy, count how many positive cases it still misses, then decide whether you would ship it as a screening test. Back to Section 13.5.

13.6 Chapter summary¶

This chapter built a regression model for a binary or binomial response. Ordinary least squares fails for a yes-or-no outcome in three ways (unbounded fits, non-constant variance, non-normal errors), and logistic regression fixes all three by making the log-odds linear. Because the S-curve makes the likelihood equations nonlinear, there is no closed-form estimate: maximum likelihood finds the coefficients, and IRLS computes them by repeated weighted least squares. Each coefficient reads as an odds ratio, tested by the Wald and likelihood-ratio tests, and the fitted probabilities are judged by deviance, a classification table, and an ROC curve, after the data are checked for disguised missing values.

Key results at a glance

Key terms

logistic regression, odds, log-odds (logit), linear predictor, link function, maximum likelihood, likelihood, log-likelihood, score equations, iteratively reweighted least squares (IRLS), working response, odds ratio, Wald test, likelihood-ratio test, deviance, saturated model, classification table, sensitivity, specificity, ROC curve, area under the curve (AUC).

You should now be able to

• Explain why ordinary least squares is the wrong model for a binary response, naming its three failures.

• State the logistic regression model and move among probability, odds, and log-odds.

• Derive the score equations (Theorem 13.4) and describe how IRLS (Theorem 13.5) solves them.

• Interpret a logistic coefficient as an odds ratio and correct the common misreadings.

• Test coefficients with the Wald and likelihood-ratio tests, and explain when they disagree.

• Compute the deviance and use it to compare nested models.

• Evaluate a fitted model with a classification table and an ROC curve, after diagnosing data problems.

Where this fits. In the workflow of The modeling workflow this chapter is mostly FIT and USE for a new kind of response: we ASK a yes-or-no question, EXPLORE with the same plots (now of proportions), FIT by maximum likelihood instead of least squares, CHECK with deviance and the disguised-zero audit, and USE the fitted probabilities to interpret odds ratios and to classify. The machinery carries the earlier chapters with it: the score equations echo the normal equations of 7.1 The model and least squares in matrix form, IRLS is weighted least squares (10.4 Weighted least squares) run in a loop, the likelihood-ratio test is the general linear test (8.4 The general linear test) in deviance form, categorical predictors enter through the dummy coding of 11.1 From categories to numbers: indicator coding, and honest evaluation needs the validation mindset of 12.4 Validation: train, test, and cross-validate. Chapter 14 takes the last step, keeping the maximum-likelihood and deviance machinery but swapping the binomial family for the Poisson to model counts, and names the generalized linear model family that holds linear, logistic, and Poisson regression together (14.6 One family: the generalized linear model).

13.7 Frequently asked questions¶

Q1. Why maximum likelihood instead of least squares here? Least squares minimizes squared error, which is the right criterion when the response is continuous with constant-variance normal noise. A binary response has neither, so squared error is no longer the natural loss. Maximum likelihood asks which coefficients make the observed 0/1 pattern most probable under the logistic model, which is the principled choice for this response, and for the normal model it happens to reproduce least squares anyway (Chapter 2).

Q2. Is an odds ratio the same as a relative risk? No, and confusing them is a common error. Relative risk is a ratio of probabilities; the odds ratio is a ratio of odds. When the outcome is rare (small $\pi$) the two are close, because odds $\approx$ probability there, but for a common outcome they diverge, and the odds ratio is always the more extreme number. Report an odds ratio as an odds ratio.

Q3. What does a negative coefficient mean? It means the predictor lowers the log-odds, so its odds ratio $e^{\beta_j}$ is below 1 and the probability of a positive outcome falls as the predictor rises. The O-ring temperature coefficient is negative: warmer launches have lower damage odds.

Q4. Why is the deviance the logistic version of the residual sum of squares? Both measure how far the fitted model sits from the data. In ordinary regression, twice the negative log-likelihood gap between your model and a perfect fit is exactly the residual sum of squares (up to a constant); for logistic regression that same gap is the deviance. Smaller deviance is a better fit, and differences in deviance between nested models follow a chi-square distribution, just as differences in the sum of squares gave $F$ statistics before.

Q5. Can I use $R^2$ for a logistic model? Not the ordinary one, because there is no residual sum of squares to divide. Several “pseudo-$R^2$” measures exist (McFadden’s, Cox-Snell, Nagelkerke), each built from log-likelihoods, and software reports them, but they do not have the clean “fraction of variance explained” meaning of the linear $R^2$. For judging a logistic model, deviance, the likelihood-ratio test, and the AUC are more informative.

Q6. My predicted probability at an extreme $X$ is 0.999. Should I believe it? Treat it the way you would any extrapolation. If the extreme $X$ is inside the range of your data, the probability is as trustworthy as the fit. If it is outside, as with the O-rings at 31 degrees, the S-curve is being asked to keep bending where you have no evidence about its shape, and the tidy number hides real uncertainty. Report it, but say plainly that it is an extrapolation.

Q7. Why did glm drop 16 observations from the diabetes model? Because we set the impossible zeros in glucose (5 of them) and BMI (11) to NA, and glm uses only complete cases by default. Those 16 women are missing a predictor the model needs. Dropping them is defensible here, but for a serious analysis you would consider whether the missingness is related to the outcome, which can bias the fit, and possibly impute rather than delete.

13.8 Practice problems¶

1. (A) In one sentence each, name the three ways ordinary least squares fails for a binary response.

2. (A) Convert these probabilities to odds: 0.2, 0.5, 0.8. Then convert these odds to probabilities: 0.25, 1, 4.

3. (A) The O-ring temperature coefficient is -0.2162. State its odds ratio for a one-degree increase and say in words what that number means.

4. (A) Explain the difference between the error $Y_i - \pi_i$ that logistic regression works with and the fitted probability $\hat\pi_i$. Which is observed?

5. (A) A student writes “the glucose odds ratio is 1.04, so a positive test is 4 percent more likely per unit.” Identify the misconception and correct it.

6. (A) Why does logistic regression have no closed-form formula for its coefficients, unlike simple linear regression?

7. (A) The AUC of a model is 0.5. What does that say about the model’s ability to rank cases?

8. (A) Give the sensitivity and specificity from the Pima classification table in Example 13.5, and say which kind of error a diabetes screening test should most want to avoid.

9. (A) The diabetes-pedigree odds ratio is 2.51 with 95 percent interval $(1.39, 4.53)$. Is its effect distinguishable from no effect? How can you tell from the interval?

10. (A) Explain why the age coefficient can be non-significant in the multiple model even though older women are, marginally, more likely to test positive.

11. (B) Starting from $\ell(\beta) = \sum_i [Y_i \log \pi_i + (m_i - Y_i)\log(1 - \pi_i)]$ and $\pi_i = 1/(1 + e^{-\eta_i})$, show that $\ell = \sum_i [Y_i \eta_i - m_i \log(1 + e^{\eta_i})] + C$.

12. (B) Differentiate the log-likelihood in Problem 11 to derive the score equations $\sum_i (Y_i - m_i \pi_i) X_{ij} = 0$ (Theorem 13.4), stating where you use $\partial \pi_i / \partial \eta_i = \pi_i(1 - \pi_i)$.

13. (B) Show that the second derivative of the log-likelihood is $-\sum_i m_i \pi_i(1 - \pi_i) X_{ij} X_{ik}$, and conclude that the Hessian is $-\mathbf{X}'\mathbf{W}\mathbf{X}$ with $\mathbf{W} = \operatorname{diag}(m_i \pi_i(1 - \pi_i))$. Explain why this makes $\ell$ concave.

14. (B) Derive the IRLS update $\beta^{(t+1)} = (\mathbf{X}'\mathbf{W}\mathbf{X})^{-1}\mathbf{X}'\mathbf{W}\mathbf{z}$ (Theorem 13.5) from the Newton step, identifying the working response $z_i$.

15. (B) Using the intercept score equation, prove that a logistic model with an intercept has $\sum_i Y_i = \sum_i m_i \hat\pi_i$. Interpret this for a plain binary response.

16. (B) Derive the odds ratio $e^{c\beta_1}$ for a $c$-unit increase in a predictor, starting from $\operatorname{logit}(\pi) = \beta_0 + \beta_1 X$.

17. (B) Show that for a rare outcome (small $\pi$) the odds ratio and the relative risk are approximately equal, by expanding both for small probabilities.

18. (B) Write the deviance for a binary ($m_i = 1$) logistic model and explain why each term is zero exactly when $\hat\pi_i$ equals the observed $Y_i$, so the saturated model has deviance 0.

19. (B) Explain, in terms of the shape of the log-likelihood, why the likelihood-ratio test can disagree with the Wald test when a coefficient is very large, and which test to trust.

20. (B) The logistic function is $\pi(\eta) = 1/(1 + e^{-\eta})$. Show that $\pi'(\eta) = \pi(\eta)(1 - \pi(\eta))$, and explain why this makes the probability change per unit of $\eta$ largest at $\eta = 0$.

21. (C) Fit the O-ring model in R or Python and reproduce $b_0 = 11.663$ and $b_1 = -0.2162$. Predict the damage probability at 50 and 75 degrees and interpret both.

22. (C) Refit the O-ring model treating each flight as a single binary “any damage” outcome (damage > 0) instead of the count of six. Compare the temperature coefficient to the grouped fit and comment on what changed.

23. (C) On the Pima data, fit test ~ glucose and test ~ glucose + bmi. Report the glucose odds ratio in each and explain why it changes when BMI is added.

24. (C) Reproduce the five-predictor Pima classification table at the 0.5 threshold, then recompute sensitivity and specificity at thresholds 0.3 and 0.7. Describe the trade-off as the threshold moves.

25. (C) Bin age into a factor with levels “under 30”, “30 to 45”, and “over 45”, add it to test ~ glucose + bmi as a categorical predictor (as in 11.1 From categories to numbers: indicator coding), and interpret the odds ratios of its levels relative to the reference.

26. (C) Compute the whole-model likelihood-ratio statistic for test ~ glucose + bmi + age + diabetes + pregnant from its null and residual deviances, and confirm your number against drop1 or a manual null-model comparison.

27. (C) Carry out a rough validation: split the Pima data 70/30 (seed 4210), fit the five-predictor model on the training part, and compute the AUC on the held-out part. Compare it to the in-sample AUC of 0.84 and explain any gap using 12.4 Validation: train, test, and cross-validate.

28. (C) Draw the fitted probability curve for test ~ glucose and add the observed positive rate within glucose deciles as points. Does the curve track the binned rates? What would a poor fit look like here?

13.9 Exam practice¶

These five questions are written in the style of the course exams. Every one asks you to explain your reasoning in full sentences, not to report a bare number: on the real exam a correct number with no supporting words earns little credit, and clear reasoning with a small arithmetic slip earns most of it. Where a question shows software output, it was produced on the course machine (R 4.6.0) from the same CSV files in data/ that you used all term; Python with statsmodels gives the same numbers. Work each one before opening its model answer.

EP 13.1 (interpret output in context). A clinic fits a logistic model for a positive diabetes test on the cleaned pima data (impossible zeros in glucose and bmi set to NA and dropped), using plasma glucose, body mass index, and number of pregnancies.

pima <- read.csv("data/pima.csv")
pima$glucose[pima$glucose == 0] <- NA
pima$bmi[pima$bmi == 0] <- NA
fit <- glm(test ~ glucose + bmi + pregnant, family = binomial, data = pima)
summary(fit)

Coefficients:
             Estimate Std. Error z value Pr(>|z|)
(Intercept) -8.780034   0.684606 -12.825  < 2e-16 ***
glucose      0.037079   0.003459  10.720  < 2e-16 ***
bmi          0.089899   0.014598   6.158 7.35e-10 ***
pregnant     0.131273   0.027299   4.809 1.52e-06 ***
---
    Null deviance: 974.75  on 751  degrees of freedom
Residual deviance: 714.58  on 748  degrees of freedom

Interpret the pregnant coefficient as an odds ratio in words a clinician could use. Then, for a woman with glucose = 150, bmi = 30, and pregnant = 4, the reported linear predictor is $\hat\eta = 0.004$; compute her estimated probability of a positive test, show the arithmetic, and say in one sentence why that probability is not four times the odds-ratio story.

EP 13.2 (a student claims something; evaluate it). Looking at the fit in EP 13.1, a student writes: “The BMI coefficient is 0.0899, and its odds ratio is $e^{0.0899} = 1.094$, so a woman with a BMI of 45 is about 9 percent more likely to test positive than a woman with a BMI of 44.” Evaluate the claim. Say precisely what is right, what is wrong, and give the corrected sentence.

EP 13.3 (explain why). Simple linear regression has closed-form formulas for its coefficients, $b_1 = S_{xy}/S_{xx}$ and $b_0 = \bar Y - b_1 \bar X$, but logistic regression has no such formula and is fit by iteratively reweighted least squares instead. Explain why no closed form exists, what IRLS actually does at each step, and why the procedure is guaranteed to climb to a single best set of coefficients rather than getting stuck.

EP 13.4 (what would change if). The five-predictor Pima model of Example 13.4 classifies at the default 0.5 probability cutoff with the table on the left below. A screening clinic proposes lowering the cutoff to 0.3, which gives the table on the right.

   cutoff 0.5                        cutoff 0.3
          actual 0  actual 1                  actual 0  actual 1
predict 0      431       114        predict 0      348        57
predict 1       57       150        predict 1      140       207

Explain what changes when the clinic moves the cutoff from 0.5 to 0.3. Compute the sensitivity and specificity at each cutoff, describe the trade-off in plain terms, and say whether the move is a good idea for a screening test and why.

EP 13.5 (interpret output in context). To ask whether age, diabetes pedigree, and number of pregnancies add anything beyond glucose and BMI, an analyst runs a likelihood-ratio test on the cleaned pima data and also reports the in-sample AUC of the larger model.

Model 1: test ~ glucose + bmi                     Residual deviance 738.51 on 749 df
Model 2: test ~ glucose + bmi + age + diabetes + pregnant   Residual deviance 703.24 on 746 df
Likelihood-ratio test: deviance drop = 35.27 on 3 df, p-value = 1.1e-07
in-sample AUC (Model 2) = 0.843

State the null hypothesis, show where the statistic 35.27 comes from, give its reference distribution and conclusion, and then explain why the AUC of 0.843 is likely optimistic and what tool from Chapter 12 would give an honest estimate.

Chapter game¶