Conditional Probability, Independence, and Bayes’ Rule

1Conditional Probability¶

1.1The Intuition: Shrinking the Sample Space¶

Let’s start with a concrete situation before we write any formulas.

A local coffee shop tracked 500 customers over a week and recorded two things: whether the customer ordered a pastry, and whether the customer visited in the morning (before noon) or afternoon.

Suppose we pick a customer at random. What is the probability they ordered a pastry?

Now here’s the twist. Someone tells you: “The customer visited in the morning.” Does this change the probability they ordered a pastry?

Yes, it does. If we know the customer visited in the morning, we’re no longer looking at all 500 customers. We’ve shrunk the sample space to the 300 morning customers. Among those 300, how many ordered a pastry? It’s 140. So:

That vertical bar “$\mid$” is read “given.” The expression $P(\text{pastry} \mid \text{morning})$ means: “the probability of a pastry order, given that we know the customer came in the morning.”

Notice what happened. The overall probability of ordering a pastry is 0.40. But among morning customers, it jumps to about 0.467. Knowing the time of visit changed our assessment. That’s what conditioning does — it updates your probability by restricting your attention to a subset of outcomes.

1.2The Formal Definition¶

Let’s translate the intuition into mathematics. Look again at what we computed:

Divide both the numerator and denominator by 500 (the total number of customers):

This is the general pattern.

Three things to notice immediately:

1. The denominator is $P(B)$, not $P(S)$. Conditioning on $B$ means $B$ becomes our new universe. Everything gets rescaled so that $P(B|B) = 1$.

2. $P(B)$ must be positive. If $P(B) = 0$, the definition doesn’t apply — you can’t condition on an event that never happens. (Division by zero is undefined.)

3. $P(A|B)$ is itself a legitimate probability. It satisfies all three axioms of probability. This means every rule you learned in Chapter 2 — complements, unions, inclusion-exclusion — works inside a conditional universe too. We’ll prove this next.

1.3Conditional Probability Satisfies the Axioms¶

1.4Worked Examples¶

EXAMPLE 3.1: Quality Control at a Bottling Plant

A bottling plant inspects bottles for two types of defects: underfilling (less than the labeled volume) and mislabeling (wrong label applied). Records show:

• $P(\text{underfill}) = 0.08$

• $P(\text{mislabel}) = 0.05$

• $P(\text{underfill} \cap \text{mislabel}) = 0.02$

A bottle is pulled off the line and found to be mislabeled. What is the probability that it is also underfilled?

Solution.

Step 1: Identify what we need. We want $P(\text{underfill} \mid \text{mislabel})$.

Step 2: Apply the definition.

Step 3: Interpret. Among mislabeled bottles, 40% are also underfilled. Compare this to the overall underfill rate of 8%. Knowing a bottle is mislabeled dramatically increases the chance it’s underfilled — perhaps both defects share a common cause (a malfunctioning machine).

EXAMPLE 3.2: The Asymmetry That Destroys Courtroom Arguments

An auto insurance company classifies 1,000 policyholders:

Compute both $P(\text{claim} | \text{under 25})$ and $P(\text{under 25} | \text{claim})$.

Solution.

These are not the same. The first says: “Among drivers under 25, 37.5% file a claim.” The second says: “Among all claimants, 33.3% are under 25.” Different questions, different denominators, different answers.

This is the exact confusion that destroyed Sally Clark’s life. $P(\text{evidence} \mid \text{innocent})$ is not the same as $P(\text{innocent} \mid \text{evidence})$. Getting these backwards has a name: the Prosecutor’s Fallacy (also called the fallacy of the transposed conditional or confusion of the inverse). We’ll resolve it formally in Section 3.5 with Bayes’ Rule.

EXAMPLE 3.3: Drawing Without Replacement

A box contains 8 red balls and 4 blue balls. Two balls are drawn one at a time without replacement. What is the probability that the second ball is red, given that the first ball was blue?

Solution.

Let $R_2$ = “second ball is red” and $B_1$ = “first ball is blue.”

After drawing a blue ball first, the box contains 8 red and 3 blue balls — a total of 11 balls. So:

This is higher than the unconditional probability $P(R_2) = 8/12 = 2/3 \approx 0.667$. Why? Because removing a blue ball on the first draw left a higher proportion of red balls remaining.

We can verify this with the formal definition. With 12 balls total and two draws without replacement, there are $12 \times 11 = 132$ ordered outcomes.

IN R: Simulation

# Simulate drawing without replacement
set.seed(42)
n_sim <- 100000
box <- c(rep("red", 8), rep("blue", 4))

first_blue <- 0; second_red_given_first_blue <- 0
for (i in 1:n_sim) {
  draw <- sample(box, 2, replace = FALSE)
  if (draw[1] == "blue") {
    first_blue <- first_blue + 1
    if (draw[2] == "red") second_red_given_first_blue <- second_red_given_first_blue + 1
  }
}
cat("Simulated P(R2 | B1) =", round(second_red_given_first_blue / first_blue, 4), "\n")
cat("Theoretical P(R2 | B1) = 8/11 =", round(8/11, 4), "\n")

EXAMPLE 3.4: The Monty Hall Problem — Solved Formally

In Chapter 1, we introduced the Monty Hall Problem and claimed that switching wins 2/3 of the time. Now we have the tools to prove it.

Setup. Three doors. A car is behind one (chosen uniformly at random); goats behind the other two. You pick Door 1. The host (Monty), who knows where the car is, opens a door revealing a goat — say he opens Door 3. Should you switch to Door 2?

Let $C_i$ = “car is behind door $i$” and $D_3$ = “Monty opens door 3.” We want $P(C_2 | D_3)$ — the probability that the car is behind Door 2, given that Monty opened Door 3.

The key probabilities we need:

• $P(C_1) = P(C_2) = P(C_3) = 1/3$ (car is equally likely behind any door).

• $P(D_3 | C_1) = 1/2$ (if the car is behind Door 1, Monty can open either Door 2 or Door 3; assume he picks uniformly).

• $P(D_3 | C_2) = 1$ (if the car is behind Door 2, Monty must open Door 3 — it’s the only goat door he can open, since you picked Door 1).

• $P(D_3 | C_3) = 0$ (if the car is behind Door 3, Monty can’t open Door 3 — he’d reveal the car).

Apply the definition of conditional probability:

We need $P(D_3)$ by the Law of Total Probability (Section 3.4 — we’re previewing it here):

Therefore:

Switching wins with probability $\boldsymbol{2/3}$. Staying wins with probability $P(C_1 | D_3) = 1/3$.

Why does switching work? Because Monty’s action is informative. When the car is behind Door 2, Monty is forced to open Door 3 (probability 1). When the car is behind Door 1, Monty opens Door 3 only half the time. So observing “Monty opened Door 3” is twice as likely under $C_2$ as under $C_1$. Conditioning on Monty’s choice shifts the probability.

1.5Section 3.1 Exercises¶

Computation

1. If $P(A) = 0.6$, $P(B) = 0.4$, and $P(A \cap B) = 0.15$, find $P(A|B)$ and $P(B|A)$.

2. A campus health clinic surveyed 800 students. Of these, 320 received a flu vaccine and 96 got the flu. Among vaccinated students, 16 got the flu. If a student is selected at random and found to have gotten the flu, what is the probability they were vaccinated?

3. A deck of 52 playing cards is shuffled and two cards are drawn without replacement. Find the probability that the second card is an ace, given that the first card was not an ace.

4. In a class of 40 students, 12 are math majors, 10 are CS majors, and 5 are double-majoring in both. A student is selected at random and found to be a math major. What is the probability they are also a CS major?

5. Events $A$ and $B$ satisfy $P(A) = 0.5$, $P(B) = 0.3$, and $P(A \cup B) = 0.65$. Find $P(A|B)$, $P(B|A)$, and $P(A|A \cup B)$.

6. In the Monty Hall problem, suppose the host is not random when the car is behind your door — instead, Monty always prefers to open Door 3 when he has a choice ($P(D_3|C_1) = 1$ instead of $1/2$). Recompute $P(C_2|D_3)$. Does switching still help?

Conceptual

7. Explain in your own words why $P(A|B)$ and $P(B|A)$ are generally different. Give a real-world example (not from this textbook) where confusing the two would lead to a bad decision.

8. Can $P(A|B)$ ever be larger than $P(A)$? Can it ever equal zero even when $P(A) > 0$? Give examples.

9. If $P(A|B) = P(A)$, what does this tell you about the relationship between $A$ and $B$? (We’ll formalize this in Section 3.2.)

Proof

10. Prove that if $A \subset B$ and $P(B) > 0$, then $P(A|B) = P(A)/P(B)$ and (if $P(A) > 0$) $P(B|A) = 1$.

11. Prove that for any events $A$, $B$ with $P(B) > 0$: $P(A^c | B) = 1 - P(A|B)$.

12. Prove: if $P(A|B) > P(A)$, then $P(B|A) > P(B)$. (Hint: write both in terms of $P(A \cap B)$ and rearrange.) Interpret in words.

2Independence of Events¶

2.1When Conditioning Changes Nothing¶

Look back at Definition 3.1. If $P(A|B) = P(A)$, then learning $B$ occurred provides zero information about $A$. The probability of $A$ is the same whether $B$ happened or not.

What does this imply? If $P(A|B) = P(A)$, then:

This is symmetric — if $P(A \cap B) = P(A) \cdot P(B)$ and $P(A) > 0$, then:

So if $A$ doesn’t affect $B$, then $B$ doesn’t affect $A$ either. Independence is a two-way street.

2.2Checking Independence: Verify, Don’t Assume¶

EXAMPLE 3.5: Weather and Transit — Not Independent

A transportation survey of 1,000 commuters:

Let $R$ = “rainy day” and $T$ = “took public transit.” Are $R$ and $T$ independent?

Check: $P(R) \cdot P(T) = 0.30 \times 0.50 = 0.15.$ But $P(R \cap T) = 180/1000 = 0.18.$

Since $0.18 \neq 0.15$, not independent. People are more likely to take transit when it rains.

EXAMPLE 3.6: Two Dice — Surprisingly Independent

Roll a fair die twice. Let $A$ = “first roll is 4” and $B$ = “sum of both rolls is 7.”

(Only (4,3) has first die = 4 AND sum = 7.)

Check: $P(A) \cdot P(B) = \frac{1}{6} \cdot \frac{1}{6} = \frac{1}{36} = P(A \cap B).$ $\checkmark$

Independent! This might be surprising — the sum involves the first roll. But no matter what the first die shows, exactly one value of the second die makes the sum 7.

2.3Independence vs. Mutually Exclusive: The Great Confusion¶

2.4Independence of Three or More Events¶

For two events, one equation ($P(A \cap B) = P(A)P(B)$) suffices. For three events, things get more demanding.

EXAMPLE 3.7: Pairwise but Not Mutually Independent

Toss a fair coin twice. Define:

• $A$ = “first toss is heads”

• $B$ = “second toss is heads”

• $C$ = “both tosses give the same result” (both H or both T)

Then $P(A) = P(B) = P(C) = 1/2$.

Check pairwise: $A \cap B$ = {HH}, so $P(A \cap B) = 1/4 = (1/2)(1/2)$. $\checkmark$ Similarly for $A \cap C$ and $B \cap C$.

But: $A \cap B \cap C$ = {HH}, so $P(A \cap B \cap C) = 1/4 \neq 1/8 = (1/2)^3$. $\boldsymbol{\times}$

Pairwise independent but NOT mutually independent. Knowing any one event tells you nothing about any other single event. But knowing two of them determines the third with certainty.

2.5When Independence Fails: Real-World Disasters¶

The most devastating failures of the independence assumption come from common-cause failures — when events that look independent share a hidden trigger.

The Fukushima Daiichi Nuclear Disaster (2011). The plant had 13 emergency diesel generators, designed as a redundant backup system. Under an independence assumption, $P(\text{all 13 fail}) = (0.001)^{13}$ — effectively impossible. But the 9.0-magnitude Tōhoku earthquake triggered a 13–15 meter tsunami (the design basis was only 5.7 meters), which flooded the generator rooms and destroyed 12 of 13 generators simultaneously. The single shared cause — the tsunami — made the “independent” backup systems fail together. This is called common-cause failure, and it’s why engineers must always ask: “What shared event could knock out all of my redundancy at once?”

The lesson for probability students: The multiplication rule $P(A \cap B) = P(A) \cdot P(B)$ is only valid when $A$ and $B$ are truly independent. When events share a common cause, the joint probability can be orders of magnitude larger than the product suggests. Always justify the independence assumption — never just assume it.

2.6Section 3.2 Exercises¶

Computation

1. If $P(A) = 0.3$, $P(B) = 0.5$, and $A$, $B$ are independent, find $P(A \cap B)$, $P(A \cup B)$, and $P(A|B)$.

2. A system has three components in series (all must work). Each operates independently with reliability 0.95. What is the probability the system functions?

3. Events $A$ and $B$ satisfy $P(A) = 0.7$, $P(B) = 0.4$, $P(A \cap B) = 0.25$. Independent? Justify.

4. A password requires a digit (0–9) then a letter (A–Z), chosen independently and uniformly. $P(\text{starts with 7 and ends with Z})$?

5. A fair die is rolled three independent times. $P(\text{at least one 6})$?

Conceptual

6. True or False: If $A$ and $B$ are mutually exclusive with $P(A) > 0$ and $P(B) > 0$, then they are independent. Justify.

7. Can an event be independent of itself? For what values of $P(A)$?

8. “I flipped heads 10 times in a row, so tails is due.” Use independence to explain why this is wrong.

9. Give a real-world example of two events that seem unrelated but are actually dependent due to a shared confounder. (Hint: think about ice cream sales and drowning rates.)

Proof

10. Prove: if $A$ and $B$ are independent with $P(A) > 0$ and $P(B) > 0$, then $A$ and $B$ cannot be mutually exclusive.

11. Prove: any event $A$ is independent of $S$. That is, $P(A \cap S) = P(A) \cdot P(S)$.

12. Prove: any event $A$ is independent of $\emptyset$. That is, $P(A \cap \emptyset) = P(A) \cdot P(\emptyset)$.

3The Multiplication Rule¶

3.1The Rule¶

Extension to three events:

Read this as a chain: start with $P(A)$, then multiply by the probability of $B$ given $A$ happened, then multiply by the probability of $C$ given both $A$ and $B$ happened.

General form for $k$ events:

3.2Tree Diagrams: The Multiplication Rule Made Visual¶

EXAMPLE 3.8: Airport Security Screening

An airport checkpoint uses two phases. Phase 1: 85% pass, 15% flagged. Phase 2 (for flagged passengers only): 70% cleared, 30% detained.

About 4.5% of passengers are detained.

EXAMPLE 3.9: Genetics — Cystic Fibrosis Carrier Probability

Cystic fibrosis (CF) is caused by a recessive allele. A carrier has genotype $Ff$ (one normal allele $F$, one CF allele $f$). If both parents are carriers ($Ff \times Ff$), the Punnett square gives:

Each cell has probability 1/4. So $P(\text{affected}) = 1/4$, $P(\text{carrier}) = 2/4 = 1/2$, $P(\text{unaffected, not carrier}) = 1/4$.

Now the real-world question: In the U.S. population of European descent, approximately 1 in 25 people are CF carriers. If two people from this population have a child, what is the probability the child is affected?

Using the multiplication rule with independent carrier status:

This matches the observed prevalence: approximately 1 in 2,500 live births.

EXAMPLE 3.10: Three Cards Without Replacement

Three cards are drawn from a standard 52-card deck, one at a time, without replacement. $P(\text{all three hearts})$?

About 1 in 77. Notice how each conditional probability adjusts for cards already drawn — this is why the multiplication rule uses conditional probabilities, not unconditional ones.

IN R

# Three hearts in a row
p_all_hearts <- (13/52) * (12/51) * (11/50)
cat("P(all three hearts) =", round(p_all_hearts, 6), "≈ 1 in", round(1/p_all_hearts), "\n")

# Simulation verification
set.seed(123); n_sim <- 100000
deck <- rep(c("heart", "other"), c(13, 39))
count <- sum(replicate(n_sim, all(sample(deck, 3) == "heart")))
cat("Simulated:", count/n_sim, "\n")

3.3Section 3.3 Exercises¶

1. A box has 6 green and 4 yellow marbles. Two are drawn without replacement. $P(\text{both green})$?

2. A student answers three independent questions, each correct with probability 0.8. $P(\text{all three correct})$? $P(\text{at least one wrong})$?

3. In a best-of-three playoff, Team A wins each game with probability 0.6 (independent). Draw a tree diagram and find $P(\text{Team A wins the series})$.

4. A manufacturing line has four independent inspection stages, each catching a defective item with probability 0.90. If an item is defective, $P(\text{passes all four undetected})$?

5. Two cards from a deck of 52, without replacement. Find $P(\text{both aces})$, $P(\text{first ace, second king})$, and $P(\text{neither is a face card})$.

6. Both parents are carriers of an autosomal recessive condition ($Ff \times Ff$). They have three children. Assuming independent inheritance: (a) $P(\text{all three affected})$? (b) $P(\text{none affected})$? (c) $P(\text{at least one affected})$?

7. (Proof) Prove the multiplication rule for four events: $P(A \cap B \cap C \cap D) = P(A) \cdot P(B|A) \cdot P(C|A \cap B) \cdot P(D|A \cap B \cap C)$.

4The Law of Total Probability¶

4.1Partitions¶

The simplest partition is $\{B, B^c\}$. But partitions can have any number of pieces.

4.2The Law¶

4.3Worked Examples¶

EXAMPLE 3.11: Insurance Claim Rates

An insurance company operates in three regions:

$P(\text{claim}) = (0.08)(0.50) + (0.05)(0.35) + (0.03)(0.15) = 0.040 + 0.0175 + 0.0045 = 0.0620.$

The overall claim rate is 6.2% — not the simple average of 8%, 5%, and 3% (which would be 5.33%). The weighted average is higher because the region with the highest claim rate also has the most policies.

EXAMPLE 3.12: Email Spam Filtering

Of all emails, 40% are spam. A spam filter catches 95% of spam (true positive rate) and incorrectly flags 3% of legitimate emails (false positive rate). $P(\text{marked as spam})$?

About 39.8% of all emails get flagged. We’ll use this in Section 3.5 to answer the reverse question: given an email is flagged, what’s the probability it’s actually spam?

4.4Simpson’s Paradox: When Aggregation Lies¶

EXAMPLE 3.13: The Kidney Stone Paradox (Real Data)

This is a real medical study (Charig et al., BMJ, 1986). Two kidney stone treatments were compared:

Treatment B looks better overall. But now stratify by stone size:

Treatment A wins in BOTH groups — but loses overall!

How is this possible? Doctors assigned the harder cases (large stones) disproportionately to Treatment A. Since large stones have lower success rates regardless of treatment, Treatment A’s aggregate was dragged down by having more difficult cases.

This is Simpson’s Paradox: a trend that appears in aggregated data reverses when the data are stratified by a confounding variable.

The mathematical explanation is the Law of Total Probability with unequal weights:

• Treatment A: $P(\text{success}) = (0.93)(87/350) + (0.73)(263/350) = 0.231 + 0.548 = 0.780.$

• Treatment B: $P(\text{success}) = (0.87)(270/350) + (0.69)(80/350) = 0.671 + 0.158 = 0.829.$

Treatment B has more weight on the easy group (small stones: 270/350 = 77%) while Treatment A has more weight on the hard group (large stones: 263/350 = 75%). The weights — not the conditional success rates — drive the aggregate reversal.

The lesson: Never compare aggregate rates without asking “what are the rates conditional on relevant subgroups?” Simpson’s Paradox is the Law of Total Probability warning you that the weighting scheme matters.

4.5Section 3.4 Exercises¶

1. A factory has two lines. Line A: 60% of output, 4% defect rate. Line B: 40%, 6% defects. $P(\text{random item defective})$?

2. A disease affects 2% of a population. A test has 97% sensitivity and 4% false positive rate. $P(\text{positive test})$?

3. 30% of commuters drive, 50% use transit, 20% bike. During rain, lateness rates are 15%, 25%, 40% respectively. $P(\text{late during rain})$?

4. A student guesses on all exam questions. Multiple choice (60% of exam): 1/4 correct. True/false (40%): 1/2 correct. Overall fraction correct?

5. Three suppliers: S1 (50%, 1% defective), S2 (30%, 2%), S3 (20%, 5%). Overall defect rate?

6. (Simpson’s Paradox — Jeter vs. Justice) In 1995–1996, Derek Jeter batted .250 (12/48) in 1995 and .314 (183/582) in 1996. David Justice batted .253 (104/411) in 1995 and .321 (45/140) in 1996. Justice has a higher average in BOTH years. Yet Jeter’s combined average over both years is higher. Verify this numerically. Which player was actually better? Explain the paradox.

7. A politician claims: “Our schools outperform the national average in every demographic group, so we must outperform overall.” Can this be false? Explain using the Law of Total Probability.

5Bayes’ Rule¶

5.1A Brief History: The Theorem That Almost Died¶

5.2Motivating Example: The Rare Disease Test¶

A disease affects 1 in 1,000 people. A test has 99% sensitivity (true positive rate) and 95% specificity (5% false positive rate). You test positive. $P(\text{you have the disease})$?

Most people — including most physicians (a study by David Eddy found 95 out of 100 got this wrong) — guess “about 99%.” Let’s compute the actual answer.

Let $D$ = “has disease,” $+$ = “tests positive.”

• $P(D) = 0.001$, $P(D^c) = 0.999$.

• $P(+ | D) = 0.99$ (sensitivity).

• $P(+ | D^c) = 0.05$ (false positive rate).

First, $P(+)$ by the Law of Total Probability:

Now:

The probability you have the disease is about 1.9% — NOT 99%.

5.3Understanding the Surprise: Natural Frequency Reframing¶

The result above shocks most people. The psychologist Gerd Gigerenzer showed that reframing the problem using natural frequencies (counts instead of probabilities) dramatically improves comprehension. Let’s try it:

Imagine 100,000 people from the general population get tested.

• 100 have the disease (prevalence = 0.001).

• Of these 100, 99 test positive (sensitivity = 99%).

• 99,900 do NOT have the disease.

• Of these 99,900, 4,995 test positive (false positive rate = 5%).

Total positive tests: $99 + 4{,}995 = 5{,}094$.

Of those 5,094 positives, only 99 actually have the disease: $99/5{,}094 \approx 1.9\%$.

The 99 true positives are drowning in a sea of 4,995 false alarms. The “99% accurate” test produces a positive result that’s a false alarm 98% of the time — because the disease is rare.

This is why base rates matter. A test can be excellent at detecting a disease but terrible as a screening tool when the disease is uncommon. This is also why a positive screening test always needs confirmation with a second, more specific test.

5.4The Formal Statement¶

5.5The Vocabulary of Bayes’ Rule¶

In words: posterior = (likelihood × prior) / evidence.

Bayes’ Rule is a learning machine. You start with a prior belief, observe data, and update.

5.6Worked Examples¶

EXAMPLE 3.14: Resolving the Prosecutor’s Fallacy

Return to Sally Clark. The prosecution presented $P(\text{two SIDS deaths} | \text{innocent}) = 1/73{,}000{,}000$. But the jury needed $P(\text{innocent} | \text{two deaths})$.

Bayes’ Rule:

The prosecution compared $P(\text{evidence} | \text{innocent})$ to $P(\text{evidence} | \text{guilty})$ but ignored the prior: double infanticide is also extremely rare. The statistician Ray Hill later showed that the likelihood ratio $P(\text{two SIDS})/P(\text{two murders})$ actually favored SIDS by 4.5:1 to 9:1. The “1 in 73 million” number was not only wrong (it assumed independence) but also irrelevant without the Bayesian comparison.

The Prosecutor’s Fallacy is precisely the error of presenting $P(\text{evidence}|\text{cause})$ as if it were $P(\text{cause}|\text{evidence})$. Bayes’ Rule is the only correct way to invert the conditional.

EXAMPLE 3.15: Identifying the Source of a Defective Part

Three suppliers:

A defective component is found. Which supplier most likely produced it?

Step 1: $P(D) = (0.01)(0.50) + (0.03)(0.30) + (0.05)(0.20) = 0.005 + 0.009 + 0.010 = 0.024.$

Step 2: Bayes’ Rule for each:

Gamma is most likely (41.7%) even though it provides only 20% of parts — because it has the highest defect rate. Bayes’ Rule shifts blame toward the supplier with the worst quality.

Check: $0.208 + 0.375 + 0.417 = 1.000.$ $\checkmark$

IN R

# Bayes' Rule: Supplier identification
suppliers <- c("Alpha", "Beta", "Gamma")
p_supplier <- c(0.50, 0.30, 0.20)
p_defect_given <- c(0.01, 0.03, 0.05)

p_defect <- sum(p_defect_given * p_supplier)
posterior <- (p_defect_given * p_supplier) / p_defect

for (i in 1:3) cat("P(", suppliers[i], "| defect) =", round(posterior[i], 4), "\n")
cat("Most likely source:", suppliers[which.max(posterior)], "\n")

EXAMPLE 3.16: The Spam Filter Revisited

From Example 3.12, $P(\text{marked as spam}) = 0.398$. If an email is marked as spam, is it actually spam?

A flagged email has a 95.5% chance of actually being spam. The remaining 4.5% are innocent emails caught in the net.

Historical note: In August 2002, programmer Paul Graham published “A Plan for Spam,” describing a Bayesian spam filter that computed $P(\text{spam} | \text{words in email})$ using Bayes’ Rule with word frequencies as likelihoods. His approach — essentially the Law of Total Probability applied token by token — achieved over 99.5% accuracy and became the basis for SpamAssassin, Mozilla Thunderbird’s filter, and most modern spam detection.

5.7Why the Base Rate Matters: A Simulation¶

# Rare disease simulation: see Bayes' Rule in action
set.seed(789)
n <- 100000; prevalence <- 0.001; sensitivity <- 0.99; fpr <- 0.05

has_disease <- rbinom(n, 1, prevalence)
test_positive <- ifelse(has_disease, rbinom(n, 1, sensitivity), rbinom(n, 1, fpr))

cat("People with disease:", sum(has_disease), "\n")
cat("Total positive tests:", sum(test_positive), "\n")
cat("  True positives:", sum(test_positive & has_disease), "\n")
cat("  False positives:", sum(test_positive & !has_disease), "\n")
cat("\nSimulated P(disease|+) =", round(sum(test_positive & has_disease)/sum(test_positive), 4), "\n")
cat("Theoretical P(disease|+) =", round(0.00099/0.05094, 4), "\n")

5.8Section 3.5 Exercises¶

1. A disease affects 5% of a population. A test has 90% sensitivity and 8% false positive rate. $P(\text{disease} | +)$?

2. In a forensic investigation, 0.1% of the population matches a suspect’s DNA profile. The DNA test has a 0.01% false positive rate. A random person tests positive. $P(\text{actual source})$?

3. A retailer sells from brands X (40%), Y (35%), Z (25%). Return rates: 5%, 8%, 12%. A product is returned. $P(\text{from Brand Z})$?

4. A friend picks one of two coins randomly: fair ($P(H) = 0.5$, chosen with probability 0.6) or biased ($P(H) = 0.8$, chosen with probability 0.4). The coin lands heads. $P(\text{fair coin})$?

5. Repeat the rare disease calculation with prevalence increased to 10%. How does $P(\text{disease}|+)$ change? Why?

6. (Sequential Bayes) Using the rare disease from the opening example (prevalence 0.001, sensitivity 0.99, FPR 0.05): a person tests positive, then takes a second independent test with the same characteristics, which is also positive. What is $P(\text{disease} | \text{both positive})$? (Hint: use the posterior from the first test as the new prior.)

7. (The Cab Problem — Kahneman & Tversky) A city has 85% green cabs and 15% blue cabs. A witness to an accident says the cab was blue. Testing shows the witness correctly identifies cab color 80% of the time. $P(\text{cab is actually blue} | \text{witness says blue})$?

8. Prove: posteriors from Bayes’ Rule always sum to 1. That is, $\sum_{j=1}^{k} P(B_j|A) = 1$.

9. Prove: if $P(A|B) > P(A)$, then $P(B|A) > P(B)$. Interpret.

10. (Natural Frequency Exercise) Reframe Problem 1 using natural frequencies: start with 10,000 people, compute counts of true positives, false positives, true negatives, and false negatives, then derive $P(\text{disease}|+)$ from the counts. Verify it matches your formula answer.

6The Event-Composition Method¶

6.1The Strategy¶

1. Define the experiment and identify the target event.

2. Identify the component events you can work with.

3. Express the target as a composition using $\cup$, $\cap$, complements.

4. Apply probability laws (additive, multiplicative, complement) to compute the answer.

The hardest step is (3). Always verify your composition contains the same sample points as the target event.

6.2Worked Examples¶

EXAMPLE 3.17: Parallel and Series System

A fire suppression system: two sprinklers ($A$, $B$) in parallel (at least one must work) and an alarm ($C$) in series (must work). Reliabilities: $P(A) = 0.95$, $P(B) = 0.90$, $P(C) = 0.99$. All independent.

System works $= C \cap (A \cup B)$.

About 98.5% — better than any individual sprinkler, thanks to the parallel design.

EXAMPLE 3.18: “At Least One” via Complement

Five friends each independently attempt a puzzle with success probabilities 0.3, 0.4, 0.5, 0.6, 0.7.

The complement method turned a five-event union into a simple product. This is the most powerful trick in event composition.

6.3Section 3.6 Exercises¶

1. Three independent relays in series, each with reliability 0.98. $P(\text{message gets through})$?

2. Two independent servers in parallel: uptimes 0.97 and 0.93. $P(\text{website accessible})$?

3. A student applies to four grad programs independently with acceptance probabilities 0.20, 0.30, 0.15, 0.40. $P(\text{at least one acceptance})$?

4. A circuit: Path 1 has two switches in series (each 0.9). Path 2 is a single switch (0.8). Paths in parallel. $P(\text{current flows})$?

5. Ten independent smoke detectors, each with $P(\text{detect}) = 0.95$. $P(\text{at least one detects a fire})$?

6. (Proof) Prove: for $n$ independent events each with probability $p$, $P(\text{at least one occurs}) = 1 - (1-p)^n$. Then show that as $n \to \infty$ with $p > 0$, this probability approaches 1.

7Random Variables: A First Look¶

7.1From Events to Numbers¶

Flip a coin three times. The sample space is $S = \{HHH, HHT, HTH, HTT, THH, THT, TTH, TTT\}$.

Now define $X$ = number of heads. $X$ is a function from $S$ to $\{0, 1, 2, 3\}$:

We can now ask: $P(X = 2) = 3/8$ (three outcomes give exactly 2 heads).

IN R

# Random variable: number of heads in 3 flips
set.seed(321); n_sim <- 100000
x <- rbinom(n_sim, size = 3, prob = 0.5)
cat("Simulated distribution of X:\n"); print(round(table(x)/n_sim, 4))
cat("\nTheoretical: P(X=0)=1/8, P(X=1)=3/8, P(X=2)=3/8, P(X=3)=1/8\n")

8Chapter Summary¶

1. Conditional probability $P(A|B) = P(A \cap B)/P(B)$ measures $P(A)$ in the restricted universe where $B$ occurred.

2. $P(A|B) \neq P(B|A)$ in general. Confusing these is the Prosecutor’s Fallacy.

3. Independence means $P(A \cap B) = P(A)P(B)$. Equivalent: $P(A|B) = P(A)$.

4. Independence ≠ mutually exclusive. These are nearly opposite when both events have positive probability.

5. The multiplication rule: $P(A \cap B) = P(A) \cdot P(B|A)$. Extends to chains. Visualized via tree diagrams.

6. The Law of Total Probability: $P(A) = \sum P(A|B_i) P(B_i)$ — a weighted average over a partition. Simpson’s Paradox is what happens when you forget the weights.

7. Bayes’ Rule: $P(B_j|A) = P(A|B_j)P(B_j)/P(A)$. Updates prior beliefs with evidence. Base rates matter enormously.

8. Event-composition method: Express complex events using $\cup$, $\cap$, complements. The complement method ($1 - P(\text{none})$) is especially powerful.

9. A random variable is a numerical function on the sample space — the bridge to Chapters 4–7.

9Chapter Refresh Exercises¶

1. $P(A) = 0.6$, $P(B) = 0.4$, $P(A|B) = 0.3$. Find $P(A \cap B)$, $P(B|A)$, $P(A \cup B)$. Are $A$ and $B$ independent?

2. A factory uses Material 1 (70%) and Material 2 (30%). Pass rates: 92% and 85%. (a) Overall pass rate? (b) A product fails. $P(\text{Material 2})$?

3. Three independent sensors detect a leak with probability 0.85 each. (a) $P(\text{all three detect})$? (b) $P(\text{at least one detects})$? (c) Alarm triggers when ≥2 detect. $P(\text{alarm})$?

4. 1% of a population uses an illegal substance. A urine test: 98% true positive, 3% false positive. (a) $P(+)$? (b) $P(\text{user} | +)$? (c) Is one positive test enough to convict? Discuss.

5. Two cards from a 52-card deck, without replacement. (a) $P(\text{both aces})$? (b) $P(\text{second ace} | \text{first not ace})$? (c) $P(\text{same suit})$?

6. $P(A)=0.5$, $P(B)=0.4$, $P(C)=0.3$, $P(A \cap B)=0.2$, $P(A \cap C)=0.15$, $P(B \cap C)=0.12$, $P(A \cap B \cap C)=0.06$. (a) $A$, $B$ independent? (b) $A$, $C$ independent? (c) All three mutually independent?

7. Medical lab: Hospital 1 sends 60% of samples (8% positive). Hospital 2 sends 40% (12% positive). (a) Overall positive rate? (b) A positive sample — $P(\text{Hospital 2})$? (c) Two independent samples — $P(\text{both positive})$?

8. (Monty Hall Variant) In the Monty Hall problem, suppose Monty doesn’t know where the car is and opens a door at random. If he happens to reveal a goat (Door 3), should you switch? Compute $P(C_2|D_3)$ under this “Ignorant Monty” variant.

9. (Proof) If $A$, $B$, $C$ are mutually independent, prove $A$ is independent of $B \cup C$.

10. (Genetics) A woman’s mother was a known carrier of hemophilia A (X-linked recessive). The woman has probability 1/2 of being a carrier. She has three healthy sons. Using sequential Bayesian updating ($P(\text{healthy son} | \text{carrier}) = 1/2$, $P(\text{healthy son} | \text{not carrier}) = 1$), compute $P(\text{carrier} | \text{3 healthy sons})$. (Answer: $1/(1+2^3) = 1/9$.)

11. Building with two independent fire exits. Exit 1 blocked with probability 0.1; Exit 2 blocked with probability 0.15. (a) $P(\text{successful evacuation})$ (at least one clear)? (b) Exit 1 is found blocked. Now $P(\text{evacuation})$?

12. (Simpson’s Paradox) A hospital reports that Treatment A has a higher survival rate than Treatment B among both young patients and elderly patients. But overall, Treatment B has a higher survival rate. Explain how this is possible. Construct a numerical example.

10Test Your Knowledge: Chapter 3 Quiz¶