Multivariate Probability Distributions - Probability Theory (MATH 3200)

The Opening Problem

Portfolio Risk. An investor holds two stocks. Stock A has an expected annual return of 12% with a standard deviation of 20%. Stock B has an expected return of 8% with a standard deviation of 15%. The investor puts 60% in A and 40% in B. What is the expected return and risk (standard deviation) of the portfolio?

If the stocks move independently, the answer is one number. If they tend to rise and fall together (positive correlation), the portfolio is riskier. If they move in opposite directions (negative correlation), the portfolio is safer — diversification works.

To answer this question precisely, we need to describe the joint behavior of two random variables simultaneously. That is what this chapter is about. Every concept from Chapters 4 and 5 — expected value, variance, distributions — generalizes from one variable to many.

1Joint Probability Distributions¶

1.1Discrete Case¶

EXAMPLE: Smartphone Defects. A quality inspector checks smartphones for two types of defects: screen defects ( $Y_1$ ) and battery defects ( $Y_2$ ). Each phone has 0 or 1 of each type. The joint distribution is:

	$y_2 = 0$	$y_2 = 1$	Row Total
$y_1 = 0$	0.80	0.05	0.85
$y_1 = 1$	0.10	0.05	0.15
Col Total	0.90	0.10	1.00

Reading: $p(0, 0) = 0.80$ means 80% of phones have neither defect. $p(1, 1) = 0.05$ means 5% have both.

1.2Continuous Case¶

EXAMPLE: Joint Continuous Distribution. Let $f(y_1, y_2) = 2(1 - y_1)$ for $0 \leq y_1 \leq 1$ , $0 \leq y_2 \leq 1$ .

Verification: $\int_0^1\int_0^1 2(1-y_1)\, dy_2\, dy_1 = \int_0^1 2(1-y_1)[y_2]_0^1\, dy_1 = \int_0^1 2(1-y_1)\, dy_1 = [2y_1 - y_1^2]_0^1 = 2 - 1 = 1. \checkmark$

Find $P(Y_1 + Y_2 \leq 1)$ :

P(Y_1 + Y_2 \leq 1) = \int_0^1 \int_0^{1-y_1} 2(1-y_1)\, dy_2\, dy_1 = \int_0^1 2(1-y_1)^2\, dy_1 = \left[-\frac{2(1-y_1)^3}{3}\right]_0^1 = \frac{2}{3}

1.3Non-Rectangular Support Regions: Where Students Get Lost¶

The example above had a rectangular support: $0 \leq y_1 \leq 1$ , $0 \leq y_2 \leq 1$ . The limits of integration were constants, and life was easy. Most interesting joint distributions don’t have rectangular support. The region where $f(y_1, y_2) > 0$ is often a triangle, wedge, or other shape — and getting the integration limits right is the single most important skill in this chapter.

📐 Calculus Flashback: Setting Up Double Integral Limits

When the support region is not a rectangle, follow this procedure:

Step 1. Sketch the region in the $(y_1, y_2)$ -plane. Shade it.

Step 2. Choose the order of integration. If integrating $dy_2$ first (inner) then $dy_1$ (outer):

Outer limits (for $y_1$ ): constants — the leftmost and rightmost extent of the region.
Inner limits (for $y_2$ ): functions of $y_1$ — for a fixed $y_1$ , what are the lowest and highest $y_2$ values in the region?

Step 3. Write: $\int_{y_1 = a}^{b} \int_{y_2 = g(y_1)}^{h(y_1)} f(y_1, y_2)\, dy_2\, dy_1$

You can also integrate in the opposite order ( $dy_1$ first). Sometimes one order is much easier than the other.

Example 1 (EXAMPLE M.1: Triangular Support — The Classic Case)

The Problem.

Let $f(y_1, y_2) = 6y_1$ for $0 < y_1 < y_2 < 1$ , and $f = 0$ otherwise.

(a) Verify this is a valid PDF. (b) Find $P(Y_2 < 0.5)$ .

Solution.

The support region is the triangle where $y_1 < y_2$ , both between 0 and 1. Sketch it: it’s the triangle above the line $y_1 = y_2$ in the unit square.

(a) Verify $\int\int f = 1$ .

Integrate $dy_2$ first (inner). For a fixed $y_1$ , $y_2$ ranges from $y_1$ to 1 (above the diagonal). Then $y_1$ ranges from 0 to 1:

\int_0^1 \int_{y_1}^{1} 6y_1\, dy_2\, dy_1 = \int_0^1 6y_1(1 - y_1)\, dy_1 = \int_0^1 (6y_1 - 6y_1^2)\, dy_1

= \left[3y_1^2 - 2y_1^3\right]_0^1 = 3 - 2 = 1 \quad \checkmark

(b) $P(Y_2 < 0.5)$ : the region is $0 < y_1 < y_2 < 0.5$ .

P(Y_2 < 0.5) = \int_0^{0.5} \int_{y_1}^{0.5} 6y_1\, dy_2\, dy_1 = \int_0^{0.5} 6y_1(0.5 - y_1)\, dy_1 = \int_0^{0.5}(3y_1 - 6y_1^2)\, dy_1

= \left[\frac{3y_1^2}{2} - 2y_1^3\right]_0^{0.5} = \frac{3}{8} - \frac{1}{4} = \frac{1}{8} = 0.125

Example 2 (EXAMPLE M.2: Finding the Constant

c

)

The Problem.

Find the constant $c$ that makes $f(y_1, y_2) = c(y_1 + y_2)$ a valid joint PDF on the triangular region $0 \leq y_1 \leq 1$ , $0 \leq y_2 \leq 1 - y_1$ .

Solution.

The support is the triangle with vertices $(0,0)$ , $(1,0)$ , $(0,1)$ . For a fixed $y_1$ , $y_2$ ranges from 0 to $1 - y_1$ .

Set $\int\int f = 1$ :

c\int_0^1 \int_0^{1-y_1} (y_1 + y_2)\, dy_2\, dy_1 = 1

Inner integral (treat $y_1$ as constant):

\int_0^{1-y_1}(y_1 + y_2)\, dy_2 = \left[y_1 y_2 + \frac{y_2^2}{2}\right]_0^{1-y_1} = y_1(1-y_1) + \frac{(1-y_1)^2}{2}

= (1-y_1)\left[y_1 + \frac{1-y_1}{2}\right] = (1-y_1) \cdot \frac{1+y_1}{2} = \frac{1-y_1^2}{2}

Outer integral:

c \int_0^1 \frac{1-y_1^2}{2}\, dy_1 = \frac{c}{2}\left[y_1 - \frac{y_1^3}{3}\right]_0^1 = \frac{c}{2} \cdot \frac{2}{3} = \frac{c}{3} = 1

Therefore $c = 3$ .

Example 3 (EXAMPLE M.3: Infinite Wedge Support (Exponential-Type))

The Problem.

The total time at a service window is $Y_1$ and the waiting time in line is $Y_2$ , with joint density $f(y_1, y_2) = e^{-y_1}$ for $0 \leq y_2 \leq y_1 < \infty$ .

(a) Verify this is a valid PDF. (b) Find $P(Y_1 > 2)$ .

Solution.

The support is the infinite wedge below the line $y_2 = y_1$ in the first quadrant. Since $y_2 \leq y_1$ , for a fixed $y_1$ , $y_2$ ranges from 0 to $y_1$ .

(a) $\int_0^{\infty}\int_0^{y_1} e^{-y_1}\, dy_2\, dy_1 = \int_0^{\infty} y_1 e^{-y_1}\, dy_1$

This is $E(Y)$ for $Y \sim \text{Exponential}(1)$ , which equals 1. $\checkmark$

(Or by integration by parts: $\int_0^{\infty} y_1 e^{-y_1}\, dy_1 = [-y_1 e^{-y_1}]_0^{\infty} + \int_0^{\infty} e^{-y_1}\, dy_1 = 0 + 1 = 1$ .)

(b) $P(Y_1 > 2) = \int_2^{\infty}\int_0^{y_1} e^{-y_1}\, dy_2\, dy_1 = \int_2^{\infty} y_1 e^{-y_1}\, dy_1$

By integration by parts: $= [-y_1 e^{-y_1}]_2^{\infty} + \int_2^{\infty} e^{-y_1}\, dy_1 = 2e^{-2} + e^{-2} = 3e^{-2} \approx 0.4060$ .

Example 4 (EXAMPLE M.4: Choosing the Integration Order)

The Problem.

Let $f(y_1, y_2) = 24y_1 y_2$ for $0 < y_1 < 1$ , $0 < y_2 < 1$ , $y_1 + y_2 < 1$ . Find $P(Y_1 > Y_2)$ .

Solution.

The support is the triangle with vertices $(0,0)$ , $(1,0)$ , $(0,1)$ — the same region as Example M.2. The additional constraint for this probability is $y_1 > y_2$ .

The region where $y_1 > y_2$ AND $y_1 + y_2 < 1$ is the triangle below $y_2 = y_1$ and below $y_2 = 1 - y_1$ .

Order 1: $dy_2$ first. For fixed $y_1$ , $y_2$ ranges from 0 to $\min(y_1, 1-y_1)$ . When $y_1 < 0.5$ : upper limit is $y_1$ . When $y_1 \geq 0.5$ : upper limit is $1 - y_1$ . This requires splitting into two integrals — messy.

Order 2: $dy_1$ first. For fixed $y_2$ , $y_1$ ranges from $y_2$ (since $y_1 > y_2$ ) to $1 - y_2$ (since $y_1 + y_2 < 1$ ). Then $y_2$ ranges from 0 to 0.5 (at $y_2 = 0.5$ , $y_1 = y_2$ meets $y_1 + y_2 = 1$ ). One integral — much cleaner.

P(Y_1 > Y_2) = \int_0^{0.5}\int_{y_2}^{1-y_2} 24y_1 y_2\, dy_1\, dy_2

Inner: $\int_{y_2}^{1-y_2} 24y_1 y_2\, dy_1 = 24y_2 \cdot \frac{y_1^2}{2}\Big|_{y_2}^{1-y_2} = 12y_2[(1-y_2)^2 - y_2^2] = 12y_2(1-2y_2)$

Outer: $\int_0^{0.5} 12y_2(1-2y_2)\, dy_2 = 12\int_0^{0.5}(y_2 - 2y_2^2)\, dy_2 = 12\left[\frac{y_2^2}{2} - \frac{2y_2^3}{3}\right]_0^{0.5}$

= 12\left[\frac{1}{8} - \frac{1}{12}\right] = 12 \cdot \frac{1}{24} = \frac{1}{2}

By symmetry of $24y_1 y_2$ in $y_1$ and $y_2$ on this symmetric region, $P(Y_1 > Y_2) = 1/2$ makes perfect sense.

1.4Section Exercises: Joint Distributions¶

J.1. Let $f(y_1, y_2) = c$ for $0 \leq y_2 \leq y_1 \leq 2$ . Find $c$ , then find $P(Y_1 \leq 1)$ .

J.2. Let $f(y_1, y_2) = 2$ for $0 < y_1 < 1$ , $0 < y_2 < 1-y_1$ . Verify it integrates to 1. Find $P(Y_1 > 0.5)$ .

J.3. Let $f(y_1, y_2) = e^{-(y_1+y_2)}$ for $y_1 > 0$ , $y_2 > 0$ . (a) Verify it’s a valid joint PDF. (b) Is the support rectangular? (c) Find $P(Y_1 + Y_2 < 1)$ .

J.4. Let $f(y_1, y_2) = 6(1-y_2)$ for $0 < y_1 < y_2 < 1$ . (a) Verify. (b) Find $P(Y_1 < 0.25, Y_2 > 0.5)$ .

2Marginal Distributions¶

Definition: Marginal Distributions

Discrete: $p_1(y_1) = \sum_{y_2} p(y_1, y_2)$ and $p_2(y_2) = \sum_{y_1} p(y_1, y_2)$

Continuous: $f_1(y_1) = \int_{-\infty}^{\infty} f(y_1, y_2)\, dy_2$ and $f_2(y_2) = \int_{-\infty}^{\infty} f(y_1, y_2)\, dy_1$

Key point for non-rectangular support: The “ $-\infty$ to $\infty$ ” limits really mean “over all $y_2$ values where $f(y_1, y_2) > 0$ for this particular $y_1$ .” When the support is a triangle, these limits are functions of $y_1$ .

EXAMPLE (Discrete marginals). From the smartphone table, the marginal of $Y_1$ (screen defects) is:

p_1(0) = 0.80 + 0.05 = 0.85, \quad p_1(1) = 0.10 + 0.05 = 0.15

This is the row totals column — the distribution of screen defects ignoring battery status.

EXAMPLE (Rectangular support). For $f(y_1, y_2) = 2(1-y_1)$ on $[0,1]^2$ :

f_1(y_1) = \int_0^1 2(1-y_1)\, dy_2 = 2(1-y_1), \quad 0 \leq y_1 \leq 1

f_2(y_2) = \int_0^1 2(1-y_1)\, dy_1 = [2y_1 - y_1^2]_0^1 = 1, \quad 0 \leq y_2 \leq 1

So $Y_2 \sim \text{Uniform}(0,1)$ but $Y_1$ has the triangular density $2(1-y_1)$ .

Example 5 (EXAMPLE M.5: Marginals from a Triangular Support)

The Problem.

From Example M.1: $f(y_1, y_2) = 6y_1$ for $0 < y_1 < y_2 < 1$ . Find $f_1(y_1)$ and $f_2(y_2)$ .

Solution.

Finding $f_1(y_1)$ : Integrate out $y_2$ . For a fixed $y_1$ , $y_2$ ranges from $y_1$ to 1 (the support constraint $y_1 < y_2 < 1$ ):

f_1(y_1) = \int_{y_1}^{1} 6y_1\, dy_2 = 6y_1(1 - y_1), \quad 0 < y_1 < 1

Common error: Writing $\int_0^1 6y_1\, dy_2 = 6y_1$ — WRONG! The lower limit is $y_1$ , not 0, because $y_2 > y_1$ in this support.

Finding $f_2(y_2)$ : Integrate out $y_1$ . For a fixed $y_2$ , $y_1$ ranges from 0 to $y_2$ (the constraint $0 < y_1 < y_2$ ):

f_2(y_2) = \int_0^{y_2} 6y_1\, dy_1 = 6 \cdot \frac{y_1^2}{2}\Big|_0^{y_2} = 3y_2^2, \quad 0 < y_2 < 1

Verification: $\int_0^1 6y_1(1-y_1)\, dy_1 = [3y_1^2 - 2y_1^3]_0^1 = 1 \checkmark$ and $\int_0^1 3y_2^2\, dy_2 = [y_2^3]_0^1 = 1 \checkmark$ .

Example 6 (EXAMPLE M.6: Marginals from the Infinite Wedge)

The Problem.

From Example M.3: $f(y_1, y_2) = e^{-y_1}$ for $0 \leq y_2 \leq y_1 < \infty$ . Find both marginals.

Solution.

$f_1(y_1)$ : Integrate out $y_2$ from 0 to $y_1$ :

f_1(y_1) = \int_0^{y_1} e^{-y_1}\, dy_2 = y_1 e^{-y_1}, \quad y_1 > 0

This is a $\text{Gamma}(2, 1)$ distribution. Check: $\int_0^{\infty} y_1 e^{-y_1}\, dy_1 = 1 \checkmark$ .

$f_2(y_2)$ : Integrate out $y_1$ from $y_2$ to $\infty$ (since $y_1 \geq y_2$ ):

f_2(y_2) = \int_{y_2}^{\infty} e^{-y_1}\, dy_1 = e^{-y_2}, \quad y_2 \geq 0

This is an $\text{Exponential}(1)$ distribution. Check: $\int_0^{\infty} e^{-y_2}\, dy_2 = 1 \checkmark$ .

Interpretation: The total service time ( $Y_1$ ) follows a Gamma(2,1), while the waiting time ( $Y_2$ ) follows an Exponential(1).

2.1Section Exercises: Marginal Distributions¶

MG.1. For $f(y_1, y_2) = 3(y_1 + y_2)$ on the triangle $0 \leq y_1 \leq 1$ , $0 \leq y_2 \leq 1-y_1$ (from Example M.2 with $c = 3$ ), find $f_1(y_1)$ and $f_2(y_2)$ . Verify each integrates to 1.

MG.2. For $f(y_1, y_2) = 24y_1 y_2$ on $y_1 > 0$ , $y_2 > 0$ , $y_1 + y_2 < 1$ , find $f_1(y_1)$ .

MG.3. For $f(y_1, y_2) = 2e^{-y_1-y_2}$ on $0 < y_1 < y_2 < \infty$ , find both marginals. (Hint: For $f_1$ , $y_2$ ranges from $y_1$ to $\infty$ .)

3Conditional Distributions¶

EXAMPLE (Discrete). From the smartphone data: $P(Y_1 = 1 | Y_2 = 1) = \frac{p(1,1)}{p_2(1)} = \frac{0.05}{0.10} = 0.50.$

Among phones with battery defects, 50% also have screen defects — much higher than the unconditional rate of 15%. The two defect types are associated.

Example 7 (EXAMPLE M.7: Conditional Distribution — Triangular Support)

The Problem.

For $f(y_1, y_2) = 6y_1$ on $0 < y_1 < y_2 < 1$ (Example M.1), find $f(y_1 | y_2)$ and compute $E(Y_1 | Y_2 = 0.8)$ .

Solution.

From Example M.5, $f_2(y_2) = 3y_2^2$ .

f(y_1 | y_2) = \frac{f(y_1, y_2)}{f_2(y_2)} = \frac{6y_1}{3y_2^2} = \frac{2y_1}{y_2^2}, \quad 0 < y_1 < y_2

Verify it’s a valid density:

\int_0^{y_2} \frac{2y_1}{y_2^2}\, dy_1 = \frac{2}{y_2^2} \cdot \frac{y_1^2}{2}\Big|_0^{y_2} = \frac{y_2^2}{y_2^2} = 1 \quad \checkmark

Conditional expectation:

E(Y_1 | Y_2 = y_2) = \int_0^{y_2} y_1 \cdot \frac{2y_1}{y_2^2}\, dy_1 = \frac{2}{y_2^2}\int_0^{y_2} y_1^2\, dy_1 = \frac{2}{y_2^2} \cdot \frac{y_2^3}{3} = \frac{2y_2}{3}

At $y_2 = 0.8$ : $E(Y_1 | Y_2 = 0.8) = \frac{2(0.8)}{3} = \frac{1.6}{3} \approx 0.533$ .

Notice: $E(Y_1 | Y_2 = y_2) = \frac{2y_2}{3}$ is a function of $y_2$ , not a single number. The larger $Y_2$ is, the larger $Y_1$ tends to be — which makes sense because $Y_1 < Y_2$ .

Example 8 (EXAMPLE M.8: Verifying the Tower Property)

The Problem.

Using Example M.7, verify that $E[E(Y_1 | Y_2)] = E(Y_1)$ .

Solution.

We found $E(Y_1 | Y_2 = y_2) = \frac{2y_2}{3}$ , so $E(Y_1 | Y_2) = \frac{2Y_2}{3}$ as a random variable.

E\left[\frac{2Y_2}{3}\right] = \frac{2}{3} E(Y_2) = \frac{2}{3}\int_0^1 y_2 \cdot 3y_2^2\, dy_2 = \frac{2}{3}\int_0^1 3y_2^3\, dy_2 = \frac{2}{3} \cdot \frac{3}{4} = \frac{1}{2}

Check directly: $E(Y_1) = \int_0^1 y_1 \cdot 6y_1(1-y_1)\, dy_1 = \int_0^1 (6y_1^2 - 6y_1^3)\, dy_1 = 2 - \frac{3}{2} = \frac{1}{2} \quad \checkmark$

The Tower Property holds: $E[E(Y_1|Y_2)] = E(Y_1) = 1/2$ .

3.1Section Exercises: Conditional Distributions¶

CD.1. For $f(y_1,y_2) = e^{-y_1}$ on $0 \leq y_2 \leq y_1 < \infty$ (Example M.3), find $f(y_2|y_1)$ and $E(Y_2|Y_1 = y_1)$ .

CD.2. For $f(y_1,y_2) = 3(y_1+y_2)$ on the triangle $0 \leq y_1 \leq 1$ , $0 \leq y_2 \leq 1-y_1$ , find $f(y_1|y_2)$ . What is its support for a given $y_2$ ?

CD.3. If $f(y_1|y_2) = f_1(y_1)$ for all $y_2$ , prove that $Y_1$ and $Y_2$ are independent.

4Independence of Random Variables¶

Checking the smartphone example: $p(0,0) = 0.80$ but $p_1(0) \cdot p_2(0) = 0.85 \times 0.90 = 0.765 \neq 0.80$ . Not independent. The defects are associated.

Checking $f(y_1,y_2) = 2(1-y_1)$ on $[0,1]^2$ : Support is rectangular ✓. Does $f$ factor? $f = 2(1-y_1) \cdot 1 = g(y_1) \cdot h(y_2)$ ✓. Independent!

Checking $f(y_1,y_2) = 6y_1$ on $0 < y_1 < y_2 < 1$ : Support is triangular (NOT rectangular). Dependent — no calculation needed! We confirmed this: $E(Y_1|Y_2 = y_2) = 2y_2/3$ , which depends on $y_2$ .

Theorem: Independence and Expectations

If $Y_1$ and $Y_2$ are independent, then for any functions $g_1$ and $g_2$ :

E[g_1(Y_1) \cdot g_2(Y_2)] = E[g_1(Y_1)] \cdot E[g_2(Y_2)]

Proof. $E[g_1(Y_1)g_2(Y_2)] = \int\int g_1(y_1)g_2(y_2)f(y_1,y_2)\,dy_1\,dy_2$ . Since $f = f_1 \cdot f_2$ by independence:

= \int\int g_1(y_1)g_2(y_2)f_1(y_1)f_2(y_2)\,dy_1\,dy_2 = \left[\int g_1(y_1)f_1(y_1)\,dy_1\right]\left[\int g_2(y_2)f_2(y_2)\,dy_2\right]

= E[g_1(Y_1)] \cdot E[g_2(Y_2)]. \quad \blacksquare

The factoring step works because $g_1(y_1)f_1(y_1)$ does not depend on $y_2$ , so it factors out of the $dy_2$ integral.

5Expected Values for Multivariate Distributions¶

EXAMPLE. For the smartphone data, $E(Y_1 + Y_2) = \sum\sum (y_1+y_2)p(y_1,y_2)$ :

= (0+0)(0.80)+(1+0)(0.10)+(0+1)(0.05)+(1+1)(0.05) = 0 + 0.10 + 0.05 + 0.10 = 0.25

Note: $E(Y_1) = 0(0.85)+1(0.15) = 0.15$ and $E(Y_2) = 0(0.90)+1(0.10) = 0.10$ . So $E(Y_1+Y_2) = 0.25 = 0.15 + 0.10 = E(Y_1) + E(Y_2)$ . Linearity of expectation holds — even when the variables are not independent.

6Covariance and Correlation¶

Proof of the Shortcut Formula

Strategy: Expand the product $(Y_1 - \mu_1)(Y_2 - \mu_2)$ and apply linearity of expectation.

E[(Y_1-\mu_1)(Y_2-\mu_2)] = E[Y_1 Y_2 - \mu_2 Y_1 - \mu_1 Y_2 + \mu_1\mu_2]

By linearity: $= E(Y_1 Y_2) - \mu_2 E(Y_1) - \mu_1 E(Y_2) + \mu_1\mu_2$

Substituting $E(Y_1) = \mu_1$ , $E(Y_2) = \mu_2$ :

= E(Y_1 Y_2) - \mu_1\mu_2 - \mu_1\mu_2 + \mu_1\mu_2 = E(Y_1 Y_2) - \mu_1\mu_2 \quad \blacksquare

Key fact: If $Y_1$ and $Y_2$ are independent, then $E(Y_1 Y_2) = E(Y_1)E(Y_2)$ , so $\text{Cov}(Y_1, Y_2) = 0$ .

The converse is FALSE. Zero covariance does not imply independence. Covariance only detects linear association.

EXAMPLE (Smartphone). $E(Y_1 Y_2) = (0)(0)(0.80)+(1)(0)(0.10)+(0)(1)(0.05)+(1)(1)(0.05) = 0.05$ .

$\text{Cov}(Y_1, Y_2) = 0.05 - (0.15)(0.10) = 0.05 - 0.015 = 0.035 > 0.$

Positive covariance: screen and battery defects tend to occur together.

7Variance of Linear Combinations¶

Theorem: E and V of Linear Combinations

Let $Y_1, \ldots, Y_n$ be random variables (not necessarily independent) with means $\mu_i$ and variances $\sigma_i^2$ . Let $a_1, \ldots, a_n$ be constants.

Expected value (always, no conditions):

E\left(\sum_{i=1}^n a_i Y_i\right) = \sum_{i=1}^n a_i \mu_i

Variance (general case):

V\left(\sum_{i=1}^n a_i Y_i\right) = \sum_{i=1}^n a_i^2 \sigma_i^2 + 2\sum_{i<j} a_i a_j \text{Cov}(Y_i, Y_j)

If $Y_1, \ldots, Y_n$ are independent (all covariances are zero):

V\left(\sum_{i=1}^n a_i Y_i\right) = \sum_{i=1}^n a_i^2 \sigma_i^2

Proof (for two variables)

Expected value: $E(a_1 Y_1 + a_2 Y_2) = a_1 E(Y_1) + a_2 E(Y_2) = a_1\mu_1 + a_2\mu_2$ . (Linearity.)

Variance: Let $W = a_1 Y_1 + a_2 Y_2$ . Then $E(W) = a_1\mu_1 + a_2\mu_2$ .

V(W) = E[(W - E(W))^2] = E[(a_1(Y_1-\mu_1) + a_2(Y_2-\mu_2))^2]

Expanding the square:

= E[a_1^2(Y_1-\mu_1)^2 + 2a_1 a_2(Y_1-\mu_1)(Y_2-\mu_2) + a_2^2(Y_2-\mu_2)^2]

By linearity:

= a_1^2 E[(Y_1-\mu_1)^2] + 2a_1 a_2 E[(Y_1-\mu_1)(Y_2-\mu_2)] + a_2^2 E[(Y_2-\mu_2)^2]

= a_1^2 \sigma_1^2 + 2a_1 a_2 \text{Cov}(Y_1, Y_2) + a_2^2 \sigma_2^2 \quad \blacksquare

EXAMPLE: Portfolio Risk (Solving the Opening Problem).

Stock A: $\mu_A = 0.12$ , $\sigma_A = 0.20$ . Stock B: $\mu_B = 0.08$ , $\sigma_B = 0.15$ . Portfolio: $R = 0.6Y_A + 0.4Y_B$ .

Expected return: $E(R) = 0.6(0.12) + 0.4(0.08) = 0.072 + 0.032 = 0.104 = 10.4\%$

Variance (if independent, $\rho = 0$ ):

V(R) = (0.6)^2(0.04) + (0.4)^2(0.0225) = 0.0144 + 0.0036 = 0.018

\sigma_R = \sqrt{0.018} = 0.1342 = 13.42\%

Variance (if $\rho = 0.5$ ): $\text{Cov}(Y_A, Y_B) = \rho \sigma_A \sigma_B = 0.5(0.20)(0.15) = 0.015$

V(R) = 0.018 + 2(0.6)(0.4)(0.015) = 0.018 + 0.0072 = 0.0252

\sigma_R = \sqrt{0.0252} = 0.1587 = 15.87\%

Positive correlation increases risk. This is why diversification (low or negative $\rho$ ) reduces portfolio volatility.

8Conditional Expectation¶

9Conditional Variance and the Law of Total Variance¶

Definition: Conditional Variance

The conditional variance of $Y_1$ given $Y_2 = y_2$ is:

V(Y_1 | Y_2 = y_2) = E\!\left[(Y_1 - E(Y_1|Y_2=y_2))^2 \,\middle|\, Y_2 = y_2\right] = E(Y_1^2 | Y_2 = y_2) - [E(Y_1|Y_2 = y_2)]^2

This is the variance of the conditional distribution of $Y_1$ given $Y_2 = y_2$ . It measures the remaining uncertainty in $Y_1$ after observing $Y_2$ . If $Y_1$ and $Y_2$ are independent, the conditional variance equals the unconditional variance: knowing $Y_2$ doesn’t reduce our uncertainty about $Y_1$ .

Theorem: The Law of Total Variance (Eve’s Law)

For any random variables $Y_1$ and $Y_2$ with $V(Y_1) < \infty$ :

V(Y_1) = E\!\left[V(Y_1 | Y_2)\right] + V\!\left(E(Y_1 | Y_2)\right)

In words:

\text{Total variance} = \underbrace{E[\text{conditional variance}]}_{\text{average within-group spread}} + \underbrace{V(\text{conditional mean})}_{\text{between-group spread of means}}

The first term measures the average spread within groups defined by $Y_2$ . The second term measures how much the group means differ from each other. Together, they account for all the variability in $Y_1$ .

Proof of the Law of Total Variance

Strategy: Write both terms on the right in terms of $E(Y_1^2)$ and $E(Y_1)^2$ , and show they sum to $V(Y_1) = E(Y_1^2) - [E(Y_1)]^2$ .

Let $g(Y_2) = E(Y_1 | Y_2)$ . By the Tower Property, $E[g(Y_2)] = E(Y_1)$ .

First term: $E[V(Y_1|Y_2)] = E[E(Y_1^2|Y_2) - g(Y_2)^2] = E[E(Y_1^2|Y_2)] - E[g(Y_2)^2]$

By the Tower Property applied to $Y_1^2$ : $E[E(Y_1^2|Y_2)] = E(Y_1^2)$ . So:

E[V(Y_1|Y_2)] = E(Y_1^2) - E[g(Y_2)^2] \tag{i}

Second term: V(E(Y_1|Y_2)) = V(g(Y_2)) = E[g(Y_2)^2] - [E(g(Y_2))]^2 = E[g(Y_2)^2] - [E(Y_1)]^2 \tag{ii}

Adding (i) and (ii): The $E[g(Y_2)^2]$ terms cancel:

E[V(Y_1|Y_2)] + V(E(Y_1|Y_2)) = E(Y_1^2) - [E(Y_1)]^2 = V(Y_1) \quad \blacksquare

Example 9 (EXAMPLE M.9: Eve’s Law — Triangular Joint Density)

The Problem.

For $f(y_1, y_2) = 6y_1$ on $0 < y_1 < y_2 < 1$ (our running example), verify Eve’s Law.

Solution.

From earlier examples: $E(Y_1|Y_2 = y_2) = \frac{2y_2}{3}$ and $f_2(y_2) = 3y_2^2$ .

Step 1: Find $V(Y_1|Y_2 = y_2)$ .

$E(Y_1^2|Y_2=y_2) = \int_0^{y_2} y_1^2 \cdot \frac{2y_1}{y_2^2}\, dy_1 = \frac{2}{y_2^2}\cdot\frac{y_2^4}{4} = \frac{y_2^2}{2}$

$V(Y_1|Y_2=y_2) = \frac{y_2^2}{2} - \left(\frac{2y_2}{3}\right)^2 = \frac{y_2^2}{2} - \frac{4y_2^2}{9} = \frac{y_2^2}{18}$

Step 2: Compute $E[V(Y_1|Y_2)]$ .

$E\!\left[\frac{Y_2^2}{18}\right] = \frac{1}{18}\int_0^1 y_2^2 \cdot 3y_2^2\, dy_2 = \frac{1}{18}\int_0^1 3y_2^4\, dy_2 = \frac{1}{18}\cdot\frac{3}{5} = \frac{1}{30}$

Step 3: Compute $V(E(Y_1|Y_2))$ .

$V\!\left(\frac{2Y_2}{3}\right) = \frac{4}{9}V(Y_2)$

$E(Y_2) = \int_0^1 y_2 \cdot 3y_2^2\, dy_2 = \frac{3}{4}$ , $\quad E(Y_2^2) = \int_0^1 y_2^2 \cdot 3y_2^2\, dy_2 = \frac{3}{5}$

$V(Y_2) = \frac{3}{5} - \frac{9}{16} = \frac{48-45}{80} = \frac{3}{80}$

$V(E(Y_1|Y_2)) = \frac{4}{9}\cdot\frac{3}{80} = \frac{12}{720} = \frac{1}{60}$

Step 4: Verify.

$E[V(Y_1|Y_2)] + V(E(Y_1|Y_2)) = \frac{1}{30} + \frac{1}{60} = \frac{2}{60} + \frac{1}{60} = \frac{3}{60} = \frac{1}{20}$

Check directly: $E(Y_1^2) = \int_0^1 \int_{y_1}^1 y_1^2 \cdot 6y_1\, dy_2\, dy_1 = \int_0^1 6y_1^3(1-y_1)\, dy_1 = \frac{6}{4} - \frac{6}{5} = \frac{3}{10}$

$V(Y_1) = \frac{3}{10} - \left(\frac{1}{2}\right)^2 = \frac{3}{10} - \frac{1}{4} = \frac{1}{20} \quad \checkmark$

9.1Section Exercises: Conditional Variance¶

CV.1. For independent $Y_1, Y_2$ with $V(Y_1) = \sigma^2$ , show that Eve’s Law gives $V(Y_1) = \sigma^2$ (the first term equals $\sigma^2$ and the second equals 0).

CV.2. Let $N \sim \text{Poisson}(\lambda)$ and given $N = n$ , let $X | N=n \sim \text{Binomial}(n, p)$ . Use Eve’s Law to find $V(X)$ . (Hint: $E(X|N) = Np$ , $V(X|N) = Np(1-p)$ .)

10The Bivariate Normal Distribution¶

Definition: Bivariate Normal Distribution

Random variables $Y_1$ and $Y_2$ have a bivariate normal distribution with parameters $\mu_1, \mu_2, \sigma_1^2, \sigma_2^2, \rho$ (five parameters) if their joint PDF is:

f(y_1, y_2) = \frac{1}{2\pi\sigma_1\sigma_2\sqrt{1-\rho^2}} \exp\!\left\{-\frac{1}{2(1-\rho^2)}\left[\left(\frac{y_1 - \mu_1}{\sigma_1}\right)^2 - 2\rho\left(\frac{y_1-\mu_1}{\sigma_1}\right)\left(\frac{y_2-\mu_2}{\sigma_2}\right) + \left(\frac{y_2-\mu_2}{\sigma_2}\right)^2\right]\right\}

for all $(y_1, y_2) \in \mathbb{R}^2$ .

Parameter meanings:

$\mu_1, \mu_2$ : the means of $Y_1$ and $Y_2$
$\sigma_1^2, \sigma_2^2$ : the variances of $Y_1$ and $Y_2$
$\rho = \text{Corr}(Y_1, Y_2)$ : the correlation between $Y_1$ and $Y_2$ , with $-1 < \rho < 1$

We write $(Y_1, Y_2) \sim \text{BVN}(\mu_1, \mu_2, \sigma_1^2, \sigma_2^2, \rho)$ .

The formula looks intimidating, but you rarely need to evaluate it directly. What matters are the properties.

10.1Key Properties of the Bivariate Normal¶

Theorem: Properties of the Bivariate Normal

If $(Y_1, Y_2) \sim \text{BVN}(\mu_1, \mu_2, \sigma_1^2, \sigma_2^2, \rho)$ , then:

1. Marginals are normal. $Y_1 \sim N(\mu_1, \sigma_1^2)$ and $Y_2 \sim N(\mu_2, \sigma_2^2)$ .

2. Conditionals are normal. The conditional distribution of $Y_1$ given $Y_2 = y_2$ is:

Y_1 | Y_2 = y_2 \sim N\!\left(\mu_1 + \rho\frac{\sigma_1}{\sigma_2}(y_2 - \mu_2), \quad \sigma_1^2(1-\rho^2)\right)

In particular:

E(Y_1|Y_2 = y_2) = \mu_1 + \rho\frac{\sigma_1}{\sigma_2}(y_2 - \mu_2)

V(Y_1|Y_2 = y_2) = \sigma_1^2(1-\rho^2)

3. Zero correlation implies independence. If $\rho = 0$ , then $Y_1$ and $Y_2$ are independent.

4. Linear combinations are normal. $aY_1 + bY_2$ is normally distributed for any constants $a, b$ .

Why Property 3 is remarkable: For general distributions, $\text{Cov} = 0$ does NOT imply independence — it only rules out linear association. The bivariate normal is special: uncorrelated does mean independent. This is because the joint PDF factors when $\rho = 0$ :

When $\rho = 0$ , the exponent becomes $-\frac{1}{2}\left[\left(\frac{y_1-\mu_1}{\sigma_1}\right)^2 + \left(\frac{y_2-\mu_2}{\sigma_2}\right)^2\right]$ , which is a sum of two terms — one depending only on $y_1$ , the other only on $y_2$ . So $f(y_1,y_2) = f_1(y_1) \cdot f_2(y_2)$ .

Why Property 2 matters for regression: The conditional mean $E(Y_1|Y_2 = y_2) = \mu_1 + \rho\frac{\sigma_1}{\sigma_2}(y_2 - \mu_2)$ is a linear function of $y_2$ . This is the theoretical foundation of simple linear regression — the “best prediction” of $Y_1$ given $Y_2$ is a straight line with slope $\rho\sigma_1/\sigma_2$ .

Example 10 (EXAMPLE M.10: Heights of Married Couples)

The Problem.

Suppose the heights of married couples follow a bivariate normal distribution. Wives’ heights: $\mu_1 = 64$ inches, $\sigma_1 = 2.5$ inches. Husbands’ heights: $\mu_2 = 70$ inches, $\sigma_2 = 3$ inches. Correlation: $\rho = 0.50$ .

(a) If a husband is 76 inches tall, what is the expected height of his wife? (b) What is the conditional standard deviation? (c) What proportion of wives of 76-inch husbands are taller than 66 inches?

Solution.

(a) By Property 2:

E(Y_1 | Y_2 = 76) = 64 + 0.50 \cdot \frac{2.5}{3}(76 - 70) = 64 + 0.50 \cdot \frac{2.5}{3} \cdot 6 = 64 + 2.5 = 66.5 \text{ inches}

(b) Conditional standard deviation:

\sigma_{Y_1|Y_2} = \sigma_1\sqrt{1-\rho^2} = 2.5\sqrt{1-0.25} = 2.5\sqrt{0.75} = 2.5 \times 0.8660 = 2.165 \text{ inches}

Notice: the conditional SD (2.165) is smaller than the unconditional SD (2.5). Knowing the husband’s height reduces our uncertainty about the wife’s height.

(c) $Y_1 | Y_2 = 76 \sim N(66.5, 2.165^2)$ .

P(Y_1 > 66) = P\!\left(Z > \frac{66 - 66.5}{2.165}\right) = P(Z > -0.231) = 1 - \Phi(-0.231) = \Phi(0.231) \approx 0.591

About 59.1% of wives of 76-inch husbands are taller than 66 inches.

Example 11 (EXAMPLE M.11: Eve’s Law for the Bivariate Normal)

The Problem.

Verify Eve’s Law for $(Y_1, Y_2) \sim \text{BVN}(\mu_1, \mu_2, \sigma_1^2, \sigma_2^2, \rho)$ .

Solution.

$V(Y_1|Y_2) = \sigma_1^2(1-\rho^2)$ — a constant (does not depend on $Y_2$ ). So:

$E[V(Y_1|Y_2)] = \sigma_1^2(1-\rho^2)$

$E(Y_1|Y_2) = \mu_1 + \rho\frac{\sigma_1}{\sigma_2}(Y_2 - \mu_2)$ , so:

$V(E(Y_1|Y_2)) = V\!\left(\rho\frac{\sigma_1}{\sigma_2}(Y_2-\mu_2)\right) = \rho^2\frac{\sigma_1^2}{\sigma_2^2} V(Y_2) = \rho^2\frac{\sigma_1^2}{\sigma_2^2}\sigma_2^2 = \rho^2\sigma_1^2$

Sum: $\sigma_1^2(1-\rho^2) + \rho^2\sigma_1^2 = \sigma_1^2 - \rho^2\sigma_1^2 + \rho^2\sigma_1^2 = \sigma_1^2 = V(Y_1) \quad \checkmark$

Interpretation: Of the total variance $\sigma_1^2$ in $Y_1$ , the fraction $\rho^2$ is “explained” by the linear relationship with $Y_2$ , and the fraction $1-\rho^2$ remains “unexplained.” This is why $R^2 = \rho^2$ in simple linear regression.

#| label: bvn-sim
# Simulate bivariate normal to visualize
set.seed(2026)
n <- 5000
mu1 <- 64; mu2 <- 70; s1 <- 2.5; s2 <- 3; rho <- 0.50

Z1 <- rnorm(n); Z2 <- rnorm(n)
Y1 <- mu1 + s1 * Z1
Y2 <- mu2 + s2 * (rho * Z1 + sqrt(1 - rho^2) * Z2)

plot(Y2, Y1, pch = '.', col = rgb(0.2,0.4,0.8,0.3),
     xlab = "Husband height (in)", ylab = "Wife height (in)",
     main = paste("Bivariate Normal, ρ =", rho))
abline(lm(Y1 ~ Y2), col = "red", lwd = 2)
cat("Sample cor =", round(cor(Y1, Y2), 3), " (theory:", rho, ")\n")
cat("Regression: slope =", round(coef(lm(Y1~Y2))[2], 3),
    " theory: rho*s1/s2 =", round(rho*s1/s2, 3), "\n")

10.2Section Exercises: Bivariate Normal¶

BN.1. If $(Y_1, Y_2) \sim \text{BVN}(10, 20, 4, 9, 0.6)$ , find $E(Y_1|Y_2=25)$ and $V(Y_1|Y_2=25)$ .

BN.2. Show that when $\rho = 0$ , the bivariate normal PDF factors into $f_1(y_1) \cdot f_2(y_2)$ , confirming independence.

BN.3. For the heights example (M.10), find $E(Y_2|Y_1 = 60)$ — the expected husband’s height given a 60-inch wife.

BN.4. What fraction of the total variance of $Y_1$ is “explained” by $Y_2$ when $\rho = 0.8$ ? When $\rho = 0.3$ ?

11The Multinomial Distribution (Optional)¶

Definition: Multinomial Distribution

If each of $n$ independent trials results in one of $k$ categories with probabilities $p_1, p_2, \ldots, p_k$ ( $\sum p_i = 1$ ), then the joint distribution of the counts $(Y_1, Y_2, \ldots, Y_k)$ is:

p(y_1, y_2, \ldots, y_k) = \frac{n!}{y_1! y_2! \cdots y_k!} p_1^{y_1} p_2^{y_2} \cdots p_k^{y_k}

where $\sum y_i = n$ .

Properties: $E(Y_i) = np_i$ , $V(Y_i) = np_i(1-p_i)$ , $\text{Cov}(Y_i, Y_j) = -np_i p_j$ for $i \neq j$ .

Note the negative covariance: if more trials fall in category $i$ , fewer are available for category $j$ .

12Seeing It in R: Joint Distributions¶

#| label: joint-sim
#| fig-cap: "Simulated joint distribution with triangular support"

# Simulate from f(y1,y2) = 6*y1 on 0 < y1 < y2 < 1
# Method: rejection sampling
set.seed(2026)
n_sim <- 50000
y1 <- runif(n_sim); y2 <- runif(n_sim)
accept <- (y1 < y2) & (runif(n_sim) < 6*y1 / 6)  # max of 6y1 is 6
y1_acc <- y1[accept]; y2_acc <- y2[accept]

# Scatter plot shows the triangular support
plot(y1_acc[1:2000], y2_acc[1:2000], pch = '.', col = "steelblue",
     xlab = expression(y[1]), ylab = expression(y[2]),
     main = "Joint distribution: f = 6y1 on 0 < y1 < y2 < 1")
abline(0, 1, col = "red", lwd = 2)  # y1 = y2 boundary

# Verify marginals
cat("Simulated E(Y1) =", round(mean(y1_acc), 4), " Theory: 0.5000\n")
cat("Simulated E(Y2) =", round(mean(y2_acc), 4), " Theory: 0.7500\n")
cat("Simulated Cov(Y1,Y2) =", round(cov(y1_acc, y2_acc), 4), "\n")
cat("Simulated Cor(Y1,Y2) =", round(cor(y1_acc, y2_acc), 4), "\n")

#| label: portfolio-sim
# Portfolio simulation: see diversification in action
set.seed(42)
n <- 100000
mu_A <- 0.12; sigma_A <- 0.20
mu_B <- 0.08; sigma_B <- 0.15

# Independent stocks
A <- rnorm(n, mu_A, sigma_A)
B_ind <- rnorm(n, mu_B, sigma_B)

# Correlated stocks (rho = 0.5)
Z1 <- rnorm(n); Z2 <- rnorm(n)
B_corr <- mu_B + sigma_B * (0.5*Z1 + sqrt(1-0.25)*Z2)
A_corr <- mu_A + sigma_A * Z1

R_ind <- 0.6*A + 0.4*B_ind
R_corr <- 0.6*A_corr + 0.4*B_corr

cat("Independent: E(R) =", round(mean(R_ind),4), " SD(R) =", round(sd(R_ind),4), "\n")
cat("Corr (ρ=0.5): E(R) =", round(mean(R_corr),4), " SD(R) =", round(sd(R_corr),4), "\n")
cat("Theory (ind): E=0.104, SD=0.1342\n")
cat("Theory (ρ=.5): E=0.104, SD=0.1587\n")

Common Mistakes — Multivariate Distributions

Wrong integration limits on non-rectangular support. If the support is $0 < y_1 < y_2 < 1$ , the marginal of $y_1$ integrates $y_2$ from $y_1$ to 1, NOT from 0 to 1. Sketch the region first. Always.
Confusing $f(y_1, y_2)$ with $f(y_1 | y_2)$ . The joint density is NOT the conditional density. The conditional divides by the marginal: $f(y_1|y_2) = f(y_1,y_2)/f_2(y_2)$ .
Assuming Cov = 0 implies independence. Covariance measures only LINEAR association. Two variables can have $\text{Cov} = 0$ and still be highly dependent (e.g., $Y_2 = Y_1^2$ when $Y_1$ is symmetric around 0).
$V(Y_1 - Y_2) = V(Y_1) + V(Y_2)$ , not minus. The variance of a difference ADDS variances (when independent). The signs inside don’t affect the spread.
Forgetting the cross-term. $V(aY_1 + bY_2) = a^2\sigma_1^2 + b^2\sigma_2^2 + 2ab\text{Cov}(Y_1,Y_2)$ . When the variables are dependent, the $2ab\text{Cov}$ term can be large. Dropping it underestimates (or overestimates) the variance.
Non-rectangular support ⟹ dependent. If the range of $Y_1$ depends on $Y_2$ , they cannot be independent. Don’t waste time checking the factoring condition — the support shape alone rules it out.

13Chapter Summary¶

Joint distributions describe the probability behavior of multiple random variables simultaneously. Joint PMFs sum to 1; joint PDFs integrate to 1.
Marginal distributions are obtained by summing out (discrete) or integrating out (continuous) the other variable(s).
Conditional distributions update probabilities given the value of another variable: $f(y_1|y_2) = f(y_1,y_2)/f_2(y_2)$ .
Independence means $f(y_1, y_2) = f_1(y_1) f_2(y_2)$ for all values — the joint factors into marginals.
Covariance $= E(Y_1 Y_2) - E(Y_1)E(Y_2)$ measures linear association. Zero for independent RVs, but zero covariance does NOT imply independence.
Correlation $\rho = \text{Cov}(Y_1,Y_2)/(\sigma_1\sigma_2) \in [-1,1]$ standardizes covariance.
Variance of a linear combination: $V(\sum a_i Y_i) = \sum a_i^2 \sigma_i^2 + 2\sum_{i<j} a_i a_j \text{Cov}(Y_i, Y_j)$ .
The Multinomial distribution generalizes the Binomial to $k > 2$ categories.