Formula Sheet

This is the course formula sheet. It collects every named formula in the book, in the order the chapters introduce them, and it is meant to sit open beside you while you work.

Nothing on this page needs calculus. Nothing on this page assumes you remember high-school algebra either. If a symbol is unfamiliar, it has a row in a table telling you how to say it and what it means.

Each entry has six parts, always in this order¶

In words. What the formula does, with no symbols at all.
The formula, on its own line.
The symbols. A table with three columns: the symbol, how to say it out loud, and what it means. Every symbol gets a row, including the operators, including the ones you are sure you already know.
Out loud. The whole formula read as one English sentence.
Worked, step by step. A small numeric instance with every multiplication and every division written out, so you can reproduce it on a phone calculator.
Sanity check. How to tell whether you got it right, and what going wrong looks like.

Each entry also names the chapter it comes from.

How to read a formula¶

A formula is not a sentence to be memorised. It is a set of instructions. You can read any formula in this book by asking three questions, in this order.

Question 1: what goes in? Look for the letters on the right-hand side of the equals sign. Those are the numbers you have to supply. In the softmax formula below, the things going in are the scores, written $z_1, z_2, z_3$ and so on. If you cannot say where a number comes from, you cannot use the formula yet, and that is the first thing to fix.

Question 2: what comes out? Look at the left-hand side of the equals sign. That is the single thing the formula produces. In softmax it is $p_i$ , one probability. Ask yourself two follow-up questions about it. Is the answer one number or a list of numbers? And what range can the answer live in? For softmax the answer is one number between 0 and 1. Knowing the range is what lets you catch a mistake later.

Question 3: what do the operators do? An operator is a symbol that tells you to perform an action: add, multiply, divide, take a square root, add up a whole list. Work from the inside out, the way you would peel an onion. In softmax the innermost action is “raise $e$ to a power”, then “add those results up”, then “divide”. Three actions, in that order.

Here is the habit in one worked pass, using softmax on three scores.

Question	Answer for softmax
What goes in?	a list of scores, one per word. Here: 2, 1, 0.
What comes out?	one probability per word, each between 0 and 1, all adding to 1.
What do the operators do?	raise $e$ to each score; add the results; divide each result by that total.

Now do the three actions. Raise $e$ to each score: 7.389056, 2.718282, 1.000000. Add them: 11.107338. Divide each by that total: 0.665241, 0.244728, 0.090031. Check the range you predicted in Question 2: all three sit between 0 and 1, and they add to 1. Done.

That is the whole method, and it works on every formula on this page.

1. Notation used on this page¶

These are the letters that appear more than once. Each entry below repeats the ones it uses, so you never have to scroll back up.

Symbol	How to say it out loud	What it means
$=$	“equals”	the thing on the left is the same number as the thing on the right
$z_i$	“z sub i”	logit: the raw score a model gives token $i$ before any conversion to probability. Can be negative. First used in Ch. 4.
$V$	“vee”	vocabulary size: how many distinct tokens the model can choose from. $V = 151{,}936$ for the model used throughout this book. Ch. 2.
$p_i$	“p sub i”	probability the model assigns to token $i$ . Always between 0 and 1. Ch. 4.
$T$	“tee”	temperature: a positive number you divide the logits by before converting them. Ch. 5.
$H$	“aitch”	entropy, measured in bits: one number for how spread out a probability distribution is. Ch. 5.
$N$	“en”	number of parameters in a model, or in part of one. $N = 494{,}032{,}768$ for Qwen2.5-0.5B-Instruct. Ch. 3.
$w_i$	“w sub i”	one weight, that is, one of those parameters, stored as a number. Ch. 3.
$b$	“bee”	bits used to store one weight. Ch. 6.
$s$	“ess”	quantization scale: the size of one step on the grid of values a quantized weight is allowed to take. Ch. 7.
$B$	“capital bee”	block size: how many consecutive weights share one scale. Ch. 7.
$\mathbf{u}, \mathbf{v}, \mathbf{q}, \mathbf{d}$	“bold u”, “bold v”, “bold q”, “bold d”	vectors, written as lists of coordinates such as $(3, 4)$ . Bold type marks a list of numbers rather than one number. Ch. 8.
$D$	“capital dee”	number of coordinates in a vector. $D = 2$ when you can draw it; $D = 384$ for the sentence embeddings in this book. Ch. 8.
$n$	“en”	number of questions on a test. Ch. 11.
$x$	“ex”	number of those questions answered correctly. Ch. 11.
$\hat{p}$	“p hat”	sample proportion, here the measured accuracy $x/n$ . The hat marks a number you measured rather than a number you know. Ch. 11.
$\bar{d}$	“d bar”	the bar marks an average. $\bar{d}$ is the average of the $d$ values. Ch. 13.
$SE$	“ess ee”	standard error: how much an estimate would move if you drew a different sample. Ch. 12.
$z^\star$	“z star”	critical value, a fixed multiplier set by the confidence level. $z^\star = 1.96$ for 95%, and that is the only value this book uses. Ch. 12.

2. Counting what is in the model¶

In words. A model is built out of named parts. This works out what fraction of the whole model one of those parts accounts for, by dividing the size of the part by the size of the whole thing.

The formula.

\text{share} = \frac{N_{\text{component}}}{N_{\text{total}}}

The symbols.

Symbol	How to say it out loud	What it means
$\text{share}$	“share”	the answer: a number between 0 and 1, which you turn into a percentage by multiplying by 100
$=$	“equals”	everything on the left is the same number as everything on the right
$N_{\text{component}}$	“en sub component”	how many parameters are in one named part of the model, such as the vocabulary table, or all the attention layers together
$N_{\text{total}}$	“en sub total”	how many parameters are in the whole model
the fraction bar	“divided by”	divide the number on top by the number underneath
the subscript, as in $N_{\text{total}}$	“sub total”	a label attached to a letter. It is not multiplication and it is not a power. It tells you which $N$ this is.

Out loud. “The share held by a component is the number of parameters in that component, divided by the number of parameters in the whole model.”

Worked, step by step. The vocabulary table of Qwen2.5-0.5B-Instruct is a grid with one row for each of the 151,936 tokens and 896 columns.

Step 1, count the parameters in the component. That is rows times columns.

$151{,}936 \times 896$

Break the multiplication into two easy pieces if you are working on paper:

$151{,}936 \times 900 = 136{,}742{,}400$

$151{,}936 \times 4 = 607{,}744$

$136{,}742{,}400 - 607{,}744 = 136{,}134{,}656$

Step 2, write down the total for the whole model: $N_{\text{total}} = 494{,}032{,}768$ .

Step 3, divide.

$136{,}134{,}656 \div 494{,}032{,}768 = 0.275558$

Step 4, turn it into a percentage by multiplying by 100.

$0.275558 \times 100 = 27.5558$ , which is 27.56% to two decimal places.

Sanity check. A share must land between 0 and 1. If you get a number bigger than 1 you divided the wrong way round; swap the top and the bottom. If you work out the share for every component and add them up, the total has to be 1, or 100%. For this model the vocabulary table is 27.56%, the MLP blocks are 63.51%, and all of attention is 8.92%. Attention is the famous part and it is under a tenth of the model.

Chapter. Chapter 3, What is a parameter?

2.2 Size of a stored matrix (USE)¶

In words. A weight matrix is a rectangle of numbers. To count how many numbers are in it, multiply how many rows it has by how many columns it has.

The formula.

N_{\text{matrix}} = (\text{rows}) \times (\text{columns})

The symbols.

Symbol	How to say it out loud	What it means
$N_{\text{matrix}}$	“en sub matrix”	the answer: how many numbers the rectangle holds
$=$	“equals”	the two sides are the same number
$\text{rows}$	“rows”	how many horizontal lines of numbers the rectangle has
$\times$	“times”	multiply the thing on the left by the thing on the right
$\text{columns}$	“columns”	how many vertical lines of numbers the rectangle has
the round brackets	“bracket”	they group a quantity together so you can see it counts as one thing

Out loud. “The number of parameters in a matrix is its number of rows multiplied by its number of columns.”

Worked, step by step. The first attention query matrix in Qwen2.5-0.5B-Instruct has 896 rows and 896 columns.

Step 1, write down the two numbers: 896 and 896.

Step 2, multiply. On paper, split it:

$896 \times 900 = 806{,}400$

$896 \times 4 = 3{,}584$

$806{,}400 - 3{,}584 = 802{,}816$

So that one matrix holds 802,816 numbers.

Sanity check. The answer must be a whole number, and it must be at least as large as the bigger of the two sides. If your answer is smaller than 896, you added instead of multiplying. The measured statistics of this particular matrix drive all of Section 4: its standard deviation is 0.066741, its largest magnitude is 1.2266, and half of its weights are smaller in magnitude than 0.02698.

Chapter. Chapter 3

3. The probability of the next word¶

3.1 Softmax (USE)¶

In words. Softmax takes a list of scores, where a bigger score means the model likes that word more, and turns them into percentages that add up to 100%.

The formula.

p_i = \frac{e^{z_i}}{\displaystyle\sum_{j=1}^{V} e^{z_j}}

The symbols.

Symbol	How to say it out loud	What it means
$p_i$	“p sub i”	the answer: the probability of word number $i$ , a number between 0 and 1
$=$	“equals”	the two sides are the same number
$i$	“eye”	a counter. $i = 1$ means the first word, $i = 2$ means the second word.
$z_i$	“z sub i”	the score the model gave to word number $i$ . Called a logit. It can be negative.
$e$	“e”	a fixed number, $2.718282\ldots$ , in the same way that $\pi$ is a fixed number
$e^{z_i}$	“e to the z sub i”	raise $e$ to the power $z_i$ . On a calculator this is the key marked `exp` or $e^x$ .
the raised position, as in $e^{z_i}$	“to the power of”	a number written up and to the right of another is an exponent, meaning a power
$\sum$	“sum”, or “sigma”	add up everything that follows, once for each value of the counter
$j=1$ underneath the $\sum$	“j equals one”	start the counter at 1
$V$ above the $\sum$	“vee”	stop the counter at $V$ , the vocabulary size. For our model $V = 151{,}936$ .
$j$	“jay”	a second counter, used inside the sum so it does not clash with $i$
the fraction bar	“divided by”	divide the top by the bottom

Out loud. “The probability of a word is $e$ raised to that word’s score, divided by the sum of $e$ raised to every word’s score.”

Worked, step by step, with three scores: 2, 1, and 0. These three numbers are made up, and chosen so the arithmetic is short enough to check on paper.

Step 1, raise $e$ to each score.

$e^{2} = 7.389056$

$e^{1} = 2.718282$

$e^{0} = 1.000000$

Step 2, add those three results together. This is what the $\sum$ sign asked for.

$7.389056 + 2.718282 = 10.107338$

$10.107338 + 1.000000 = 11.107338$

Step 3, divide each result from Step 1 by that total.

$p_1 = 7.389056 \div 11.107338 = 0.665241$

$p_2 = 2.718282 \div 11.107338 = 0.244728$

$p_3 = 1.000000 \div 11.107338 = 0.090031$

step	word 1	word 2	word 3	total
score $z_i$	2	1	0
$e^{z_i}$	7.389056	2.718282	1.000000	11.107338
$p_i = e^{z_i} \div 11.107338$	0.665241	0.244728	0.090031	1.000000

Sanity check. Add your three probabilities. They must come to 1.

$0.665241 + 0.244728 = 0.909969$

$0.909969 + 0.090031 = 1.000000$

If they do not add to 1, you divided by the wrong total. Two more things must always be true. No probability can be negative, because $e$ raised to any power is positive. And the word with the largest score must end up with the largest probability, so the order 2, 1, 0 has to survive as 0.665, 0.245, 0.090. The real run does this same arithmetic 151,936 times.

Chapter. Chapter 4, The probability of the next word

3.2 The ratio form (READ)¶

In words. If you want to know how many times more likely one word is than another, you do not need the whole vocabulary. You need only the difference between their two scores.

The formula.

\frac{p_i}{p_j} = e^{\,z_i - z_j}

The symbols.

Symbol	How to say it out loud	What it means
$p_i$	“p sub i”	the probability of word $i$
$p_j$	“p sub j”	the probability of some other word, word $j$
the fraction bar	“divided by”	divide the top by the bottom. The answer says how many times bigger $p_i$ is than $p_j$ .
$=$	“equals”	the two sides are the same number
$e$	“e”	the fixed number $2.718282\ldots$
$z_i - z_j$	“z sub i minus z sub j”	the gap between the two scores
$-$	“minus”	subtract the number on the right from the number on the left
the raised position	“to the power of”	raise $e$ to whatever is written up there, gap and all

Out loud. “The ratio of two probabilities is $e$ raised to the difference between those two words’ scores.”

Worked, step by step, on the made-up scores from 3.1. Word 1 scored 2 and word 2 scored 1.

Step 1, subtract the scores. $2 - 1 = 1$ .

Step 2, raise $e$ to that. $e^{1} = 2.718282$ .

Step 3, check it against the probabilities you already computed in 3.1.

$0.665241 \div 0.244728 = 2.718287$

The two answers agree to five decimal places. The last digit differs because the probabilities in 3.1 were rounded to six decimal places before being divided. That is rounding, not error.

Worked, step by step, on the real model run. On the prompt The capital of France is the top two logits were 17.21729 for the token ' Paris' and 16.31964 for the token ' ______'.

Step 1, subtract. $17.21729 - 16.31964 = 0.89765$ .

Step 2, raise $e$ to that. $e^{0.89765} = 2.45383$ .

Step 3, check against the recorded probabilities, $30.219\%$ and $12.315\%$ .

$0.30219 \div 0.12315 = 2.4538$

They agree to four decimal places.

Sanity check. If the score on top is the bigger one, your answer must be bigger than 1. If the two scores are equal, the difference is 0, and $e^{0} = 1$ , so the ratio is exactly 1 and the two words are equally likely. An answer below 1 means you put the lower-scoring word on top. This formula also explains something useful: the enormous denominator in 3.1 cancels out, so you can compare two words without knowing anything about the other 151,934.

Chapter. Chapter 4

3.3 Softmax with temperature (USE)¶

In words. Temperature is a dial. Before you turn the scores into probabilities, you divide every score by the same number. A small number spreads the scores apart, which makes the leading word even more likely. A large number squashes the scores together, which shares the probability out more evenly.

The formula.

p_i(T) = \frac{e^{\,z_i/T}}{\displaystyle\sum_{j=1}^{V} e^{\,z_j/T}}

The symbols.

Symbol	How to say it out loud	What it means
$p_i(T)$	“p sub i of tee”	the answer: the probability of word $i$ when the temperature dial is set to $T$
the round brackets in $p_i(T)$	“of”	they say the answer depends on $T$ . They do not mean multiply.
$T$	“tee”	the temperature, a positive number you choose. $T = 1$ is the model’s own setting.
$z_i$	“z sub i”	the score for word $i$
$z_i/T$	“z sub i over tee”	divide the score by the temperature. This happens before the exponential.
the slash $/$	“divided by”	divide the number on the left by the number on the right
$e$	“e”	the fixed number $2.718282\ldots$
the raised position	“to the power of”	raise $e$ to whatever is written up there
$\sum_{j=1}^{V}$	“the sum from j equals one to vee”	add up what follows, once for every word in the vocabulary
the fraction bar	“divided by”	divide the top by the bottom
$=$	“equals”	the two sides are the same number

Out loud. “The probability of a word at temperature $T$ is $e$ raised to that word’s score divided by $T$ , divided by the sum of $e$ raised to every word’s score divided by $T$ .”

Worked, step by step, at $T = 0.5$ , on the same made-up scores 2, 1, 0.

Step 1, divide every score by $T = 0.5$ . Dividing by 0.5 is the same as multiplying by 2.

$2 \div 0.5 = 4$

$1 \div 0.5 = 2$

$0 \div 0.5 = 0$

Step 2, raise $e$ to each of those.

$e^{4} = 54.598150$

$e^{2} = 7.389056$

$e^{0} = 1.000000$

Step 3, add them up.

$54.598150 + 7.389056 = 61.987206$

$61.987206 + 1.000000 = 62.987206$

Step 4, divide each by that total.

$p_1 = 54.598150 \div 62.987206 = 0.866813$

$p_2 = 7.389056 \div 62.987206 = 0.117310$

$p_3 = 1.000000 \div 62.987206 = 0.015876$

Worked again, at $T = 2.0$ .

Step 1, divide every score by 2: 1, 0.5, 0.

Step 2, raise $e$ to each: $e^{1} = 2.718282$ , $e^{0.5} = 1.648721$ , $e^{0} = 1.000000$ .

Step 3, add them: $2.718282 + 1.648721 = 4.367003$ , then $4.367003 + 1.000000 = 5.367003$ .

Step 4, divide each by 5.367003: 0.506480, 0.307196, 0.186324.

Put the three settings side by side.

	word 1	word 2	word 3
$p_i$ at $T = 0.5$	0.866813	0.117310	0.015876
$p_i$ at $T = 1.0$	0.665241	0.244728	0.090031
$p_i$ at $T = 2.0$	0.506480	0.307196	0.186324

Sanity check. Each row must add to 1. Check the top row: $0.866813 + 0.117310 = 0.984123$ , then $0.984123 + 0.015876 = 0.999999$ , which is 1 apart from rounding in the sixth decimal place. Then check the ordering. Word 1 is first in every row and word 3 is last in every row. If your ordering changed, you made an arithmetic mistake, because temperature cannot reorder words. Section 3.4 shows why.

Chapter. Chapter 5, Temperature, and what it does not do

3.4 Temperature as a power, the equivalent form (READ)¶

In words. You can get the temperature-adjusted probabilities from the ordinary probabilities alone, without ever seeing the scores. Raise each probability to the power one-over-the-temperature, then divide everything by the new total so it adds to 1 again.

The formula.

p_i(T) = \frac{\big(p_i(1)\big)^{1/T}}{\displaystyle\sum_{j=1}^{V} \big(p_j(1)\big)^{1/T}}

The symbols.

Symbol	How to say it out loud	What it means
$p_i(T)$	“p sub i of tee”	the answer: the probability of word $i$ at temperature $T$
$p_i(1)$	“p sub i of one”	the probability of word $i$ at $T = 1$ , that is, plain softmax from 3.1
$1/T$	“one over tee”	the reciprocal of the temperature. At $T = 0.5$ this is 2; at $T = 2$ it is 0.5.
the big round brackets	“bracket”	they group the whole probability together, so the power applies to all of it
the raised $1/T$	“to the power one over tee”	raise the bracketed number to that power. A power of 2 means square it; a power of 0.5 means take its square root.
$\sum_{j=1}^{V}$	“the sum from j equals one to vee”	add up the same quantity for every word in the vocabulary
the fraction bar	“divided by”	divide the top by the bottom. This is the renormalising step.
$=$	“equals”	the two sides are the same number

Out loud. “The probability of a word at temperature $T$ is that word’s ordinary probability raised to the power one over $T$ , divided by the sum of every word’s ordinary probability raised to the power one over $T$ .”

Worked, step by step, at $T = 2$ . Here $1/T = 1 \div 2 = 0.5$ , and raising to the power 0.5 is taking a square root. Start from the $T = 1$ probabilities computed in 3.1.

Step 1, take the square root of each probability.

$\sqrt{0.665241} = 0.815623$

$\sqrt{0.244728} = 0.494700$

$\sqrt{0.090031} = 0.300052$

Step 2, add the three roots.

$0.815623 + 0.494700 = 1.310323$

$1.310323 + 0.300052 = 1.610375$

Step 3, divide each root by that total.

$0.815623 \div 1.610375 = 0.506480$

$0.494700 \div 1.610375 = 0.307196$

$0.300052 \div 1.610375 = 0.186324$

Compare with 3.3, where dividing the scores by 2 first gave 0.506480, 0.307196, 0.186324. The two routes agree. Carried out at full machine precision they agree to better than one part in a trillion: both give 0.50648039, 0.30719589, 0.18632372.

Worked again, at $T = 0.5$ , where $1/T = 2$ and raising to the power 2 means squaring.

$0.665241 \times 0.665241 = 0.44254559$

$0.244728 \times 0.244728 = 0.05989179$

$0.090031 \times 0.090031 = 0.00810558$

$0.44254559 + 0.05989179 = 0.50243738$ , then $0.50243738 + 0.00810558 = 0.51054296$

$0.44254559 \div 0.51054296 = 0.866814$

$0.05989179 \div 0.51054296 = 0.117310$

$0.00810558 \div 0.51054296 = 0.015876$

Compare that with 3.3, which gave 0.866813, 0.117310, 0.015876. The first answer differs by one in the sixth decimal place, because you started from probabilities that had already been rounded to six decimal places and then squared them. That is rounding carried forward, not an error in the formula.

Check it on the real run. At $T = 1$ the top token is 2.45383 times as probable as the runner-up, which is what 3.2 worked out. Squaring that gives $2.45383 \times 2.45383 = 6.0213$ . The recorded probabilities at $T = 0.5$ are $73.430\%$ and $12.195\%$ , and $0.73430 \div 0.12195 = 6.0213$ . They match.

Sanity check. Your answers must still add to 1 after Step 3, and the order must not have changed. Here is the reason the order cannot change, and it is worth reading twice. Raising every probability to the same positive power keeps them in the same order, because a bigger number stays bigger when you square it, and a bigger number stays bigger when you take its square root. Dividing them all by the same total keeps them in the same order too. So nothing in this formula can promote one word above another. That is the cleanest proof that temperature cannot reorder tokens.

Chapter. Chapter 5

3.5 Entropy in bits (USE)¶

In words. Entropy is one number saying how undecided the model is. If the model is certain of one word, entropy is zero. The more candidates share the probability, the bigger entropy gets.

The formula.

H = -\sum_{i=1}^{V} p_i \log_2 p_i

The symbols.

Symbol	How to say it out loud	What it means
$H$	“aitch”	the answer: the entropy, measured in bits
$=$	“equals”	the two sides are the same number
the leading $-$	“minus”, or “negative”	flip the sign of everything that follows. It is there because the logarithm of a number below 1 is negative, and entropy is reported as a positive number.
$\sum$	“sum”, or “sigma”	add up what follows, once for each value of the counter
$i=1$ underneath	“i equals one”	start the counter at the first word
$V$ above	“vee”	stop at the last word in the vocabulary
$p_i$	“p sub i”	the probability of word $i$
$\log_2$	“log base two”	the logarithm base 2. It answers the question “2 raised to what power gives this number?” So $\log_2 8 = 3$ , because $2^3 = 8$ .
$\log_2 p_i$	“log base two of p sub i”	apply that question to the probability. Because every $p_i$ is below 1, every answer is negative.
the space between $p_i$ and $\log_2 p_i$	“times”	two things written next to each other are multiplied

Out loud. “Entropy is minus the sum, over every word, of that word’s probability multiplied by the logarithm base two of that same probability.”

Worked, step by step, on three made-up probabilities: 0.5, 0.25, 0.25. These were chosen because every logarithm lands on a whole number.

Step 1, take the logarithm base 2 of each probability.

$\log_2 0.50 = -1$ , because $2^{-1} = 0.5$

$\log_2 0.25 = -2$ , because $2^{-2} = 0.25$

$\log_2 0.25 = -2$

Step 2, multiply each probability by its own logarithm.

$0.50 \times (-1) = -0.50$

$0.25 \times (-2) = -0.50$

Step 3, add those three products.

$-0.50 + (-0.50) = -1.00$ , then $-1.00 + (-0.50) = -1.50$

Step 4, flip the sign, because of the minus at the front of the formula.

$H = 1.5$ bits

Worked again, on the three probabilities from 3.1. These do not land on whole numbers, so use the log button on a calculator and divide the answer by $\log 2$ , or use a log2 button if your calculator has one.

$p_i$	$\log_2 p_i$	$p_i \times \log_2 p_i$	$-p_i \log_2 p_i$
0.665241	-0.588051	-0.391196	0.391196
0.244728	-2.030746	-0.496981	0.496981
0.090031	-3.473441	-0.312716	0.312716

$0.391196 + 0.496981 = 0.888177$

$0.888177 + 0.312716 = 1.200893$

So $H = 1.200893$ bits. Repeating that for the other two temperature settings gives the table that makes the temperature lesson quantitative.

$T$	$p_1$	$p_2$	$p_3$	entropy $H$ (bits)
0.5	0.866813	0.117310	0.015876	0.636311
1.0	0.665241	0.244728	0.090031	1.200893
2.0	0.506480	0.307196	0.186324	1.471825

Sanity check. Entropy can never be negative. If yours is, you left off the minus sign at the front. Entropy also has a ceiling: it can never exceed $\log_2$ of the number of choices. With three choices the ceiling is $\log_2 3 = 1.584963$ bits, and the largest number in the table above is 1.471825, which sits under it. Two values worth committing to memory: a fair coin gives exactly 1 bit, and a certain outcome gives exactly 0 bits.

What the real run gives. On the same prompt, entropy over the full 151,936-token vocabulary was 0.237 bits at $T = 0.25$ , 4.450 bits at $T = 1.0$ , and 13.367 bits at $T = 2.0$ . The ceiling for this vocabulary is $\log_2 151{,}936 = 17.2131$ bits, which only a perfectly flat distribution would reach.

Chapter. Chapter 5

3.6 How many tokens hold 90% of the probability (USE)¶

In words. Sort the words from most likely to least likely, then start adding their probabilities from the top. Count how many words you need before the running total reaches nine tenths. That count is an honest answer to “how many words is the model really considering?”

The formula.

k_{0.9} = \min\Big\{\,k \ : \ p_{(1)} + p_{(2)} + \cdots + p_{(k)} \ \ge\ 0.9 \,\Big\}

The symbols.

Symbol	How to say it out loud	What it means
$k_{0.9}$	“k sub nought point nine”	the answer: a whole number, how many words it takes to reach 90%
$=$	“equals”	the two sides are the same number
$\min$	“min”, short for “minimum”	the smallest value that works
the curly brackets	“the set of”	they hold a description of a collection of numbers
the colon	“such that”	it separates which numbers we are talking about from the condition they must meet
$k$	“kay”	a counter: how many words you have added so far
$p_{(1)}$	“p bracket one”	the largest probability. The brackets around the number mean the list has been sorted, largest first.
$p_{(2)}$	“p bracket two”	the second largest probability, and so on down the list
$+$	“plus”	add
$\cdots$	“and so on”	keep adding in the same pattern
$\ge$	“is greater than or equal to”	the thing on the left is at least as big as the thing on the right
0.9	“nought point nine”	nine tenths, which is 90%

Out loud. “ $k$ sub nought point nine is the smallest number of words you can take from the top of the sorted list whose probabilities add to at least nine tenths.”

Worked, step by step, on the made-up probabilities at $T = 1$ : 0.665241, 0.244728, 0.090031. They are already sorted from largest to smallest.

Step 1, take the top one. Running total $= 0.665241$ . Is $0.665241 \ge 0.9$ ? No.

Step 2, add the next one. $0.665241 + 0.244728 = 0.909969$ . Is $0.909969 \ge 0.9$ ? Yes.

Step 3, stop, and report how many words you used. $k_{0.9} = 2$ .

Worked again, on the same words at $T = 2$ : 0.506480, 0.307196, 0.186324.

Step 1, running total $= 0.506480$ . Below 0.9.

Step 2, $0.506480 + 0.307196 = 0.813676$ . Still below 0.9.

Step 3, $0.813676 + 0.186324 = 1.000000$ . At or above 0.9, so $k_{0.9} = 3$ .

Raising the temperature from 1 to 2 took the count from 2 to 3. That is the same movement the entropy showed in 3.5, in a unit anyone can picture.

Sanity check. $k_{0.9}$ must be a whole number, at least 1, and no larger than the number of words. It can never go down when you raise the temperature. If your running total passes 1, you are adding probabilities that were never a valid distribution, so go back and check that they added to 1 in the first place.

What the real run gives. At $T = 0.25$ the top token alone carries $96.799\%$ , which already clears $90\%$ , so $k_{0.9} = 1$ . One word holds nine tenths of the probability. At $T = 2.0$ on the same logits it takes 41,274 tokens to reach the same threshold. That pair of numbers is what temperature actually does.

Chapter. Chapter 5

4. Making models small¶

4.1 Symmetric linear quantization: the scale (USE)¶

In words. Storing a weight with fewer bits means you can only keep a small number of different values. Line those allowed values up evenly on either side of zero, like marks on a ruler. This works out how far apart the marks have to be so that the largest weight in the group still lands on the ruler.

The formula.

s = \frac{\displaystyle\max_i \lvert w_i \rvert}{2^{\,b-1} - 1}

The symbols.

Symbol	How to say it out loud	What it means
$s$	“ess”	the answer: the scale, the gap between two neighbouring marks on the ruler
$=$	“equals”	the two sides are the same number
$w_i$	“w sub i”	weight number $i$ in the group being quantized together
$\lvert w_i \rvert$	“the absolute value of w sub i”, or “mod w sub i”	the size of the weight with its minus sign thrown away. $\lvert -3 \rvert = 3$ and $\lvert 3 \rvert = 3$ .
the two upright bars	“absolute value”	they are the throw-away-the-sign instruction, nothing to do with division
$\max_i$	“the maximum over i”	look at every weight in the group and keep the biggest answer
$b$	“bee”	how many bits you are spending on each stored weight
$b-1$	“bee minus one”	one bit is spent recording whether the weight is positive or negative, so $b-1$ bits are left for the size
$2^{\,b-1}$	“two to the bee minus one”	2 multiplied by itself $b-1$ times. This is how many different sizes those bits can express.
$-\,1$	“minus one”	subtract one, because one of the levels has to be zero itself
the fraction bar	“divided by”	divide the top by the bottom

Out loud. “The scale is the largest weight size in the group, divided by the number of whole steps that fit on each side of zero.”

Worked, step by step, at 8 bits. The real weight matrix from 2.2 has $\max_i \lvert w_i \rvert = 1.2265625$ .

Step 1, work out $b - 1$ . With $b = 8$ bits, $8 - 1 = 7$ .

Step 2, raise 2 to that power. That means 2 multiplied by itself seven times, so $2^{7} = 128$ .

Step 3, subtract 1. $128 - 1 = 127$ . So 127 whole steps fit on each side of zero.

Step 4, divide. $1.2265625 \div 127 = 0.00965797$ .

Worked again, at 4 bits.

Step 1, $4 - 1 = 3$ .

Step 2, $2^{3} = 2 \times 2 \times 2 = 8$ .

Step 3, $8 - 1 = 7$ .

Step 4, $1.2265625 \div 7 = 0.17522321$ .

Every bit width, on the same matrix:

$b$	$2^{b-1}$	steps each side, $2^{b-1}-1$	scale $s$
8	128	127	0.0096580
6	32	31	0.0395665
4	8	7	0.1752232
3	4	3	0.4088542
2	2	1	1.2265625

Hold on to the 4-bit scale, 0.1752232. It is used in 4.2 and 4.3.

Sanity check. The scale is always positive, and it always gets bigger as $b$ gets smaller, because fewer bits means fewer marks on the same ruler, so the marks must be further apart. Here is the direct check: divide the largest weight by the scale and you must land exactly on the top step. $1.2265625 \div 0.00965797 = 127.0$ , which is the number from Step 3. If you get something else, you used the wrong $b$ .

Chapter. Chapter 7, Quantization: what you lose, what it buys

4.2 Quantize, then dequantize (USE)¶

In words. Take each weight, work out which mark on the ruler it is nearest to, and store only the number of that mark. Later, when the model runs, multiply the mark number by the gap between marks to get a weight back. The number you get back is close to the original, and the original is gone for good.

The formula.

q_i = \operatorname{round}\!\left(\frac{w_i}{s}\right), \qquad \hat{w}_i = q_i \, s

The symbols.

Symbol	How to say it out loud	What it means
$q_i$	“q sub i”	the whole number actually stored in place of weight $i$ . It sits between $-(2^{\,b-1}-1)$ and $+(2^{\,b-1}-1)$ .
$=$	“equals”	the two sides are the same number
$\operatorname{round}$	“round”	round to the nearest whole number. 2.4 rounds to 2; 2.6 rounds to 3; -1.9 rounds to -2.
$w_i$	“w sub i”	the original weight
$s$	“ess”	the scale from 4.1
the fraction bar	“divided by”	divide the top by the bottom
the comma	no sound	it separates the two formulas. This entry has two, done in order.
$\hat{w}_i$	“w hat sub i”	the dequantized weight: the value the model actually uses once it has been shrunk
the hat	“hat”	it marks a number that stands in for another number, rather than being the real thing
the space between $q_i$ and $s$	“times”	two things written next to each other are multiplied

Out loud. “The stored number is the weight divided by the scale, rounded to the nearest whole number; the weight you get back is that stored number multiplied by the scale.”

Worked, step by step, at 8 bits. Take the typical weight of that matrix, $w = -0.02697754$ . It is the median magnitude, so half of the matrix is smaller than this.

Step 1, write down the 8-bit scale from 4.1: $s = 0.00965797$ .

Step 2, divide the weight by the scale. $-0.02697754 \div 0.00965797 = -2.79329$

Step 3, round to the nearest whole number. -2.79329 sits between -3 and -2, and it is nearer to -3, so $q = -3$ .

Step 4, multiply back to get the dequantized weight. $-3 \times 0.00965797 = -0.02897391$

Step 5, work out how far off you are. $0.02897391 - 0.02697754 = 0.00199637$

Worked again, at 4 bits.

Step 1, the 4-bit scale from 4.1: $s = 0.17522321$ .

Step 2, $-0.02697754 \div 0.17522321 = -0.15396$ .

Step 3, round. -0.15396 is nearer to 0 than to -1, so $q = \mathbf{0}$ .

Step 4, $0 \times 0.17522321 = \mathbf{0}$ .

Step 5, the error is the whole weight: 0.02697754.

$b$	$s$	$w \div s$	$q$	$\hat{w} = q \times s$	error
8	0.00965797	-2.79329	-3	-0.02897391	0.00199637
4	0.17522321	-0.15396	$\mathbf{0}$	$\mathbf{0}$	0.02697754

At 8 bits the weight survives with a small error. At 4 bits it rounds to zero and is erased. The lab run records -0.02897392 for the 8-bit case, which matches the hand arithmetic to seven decimal places; the last digit differs because the scale was rounded to eight places above.

Sanity check. $q$ must be a whole number, and it must sit inside the range $-(2^{b-1}-1)$ to $+(2^{b-1}-1)$ . At 4 bits that range is -7 to +7, and 0 is inside it. The dequantized weight must have the same sign as the original, or be zero. And the error can never be bigger than half the scale: at 4 bits half the scale is $0.17522321 \div 2 = 0.08761161$ , and the error 0.02697754 is under that.

Chapter. Chapter 7

4.3 Quantization error, measured against the spread of the weights (READ)¶

In words. Work out how far off each weight is on average, then compare that to how spread out the weights were in the first place. Reporting the error as a share of the spread makes “how bad is 4-bit?” a question with an answer that does not depend on the units.

The formula.

\text{relative error} = \frac{\dfrac{1}{N}\displaystyle\sum_{i=1}^{N} \big\lvert \hat{w}_i - w_i \big\rvert}{\operatorname{sd}(w)}

The symbols.

Symbol	How to say it out loud	What it means
$\text{relative error}$	“relative error”	the answer: a number usually written as a percentage
$=$	“equals”	the two sides are the same number
$N$	“en”	how many weights are in the group
$\dfrac{1}{N}$	“one over en”	dividing by $N$ after adding $N$ things up is how you take an average
$\sum_{i=1}^{N}$	“the sum from i equals one to en”	add up what follows, once for every weight
$\hat{w}_i$	“w hat sub i”	the weight you get back after quantizing, from 4.2
$w_i$	“w sub i”	the original weight
$\hat{w}_i - w_i$	“w hat sub i minus w sub i”	the rounding error on that one weight. It can be positive or negative.
the two upright bars	“absolute value”	throw away the minus sign, so errors in opposite directions do not cancel each other out
$\operatorname{sd}(w)$	“the standard deviation of w”	one number saying how spread out the original weights were. A big standard deviation means the weights are far apart.
the big fraction bar	“divided by”	divide the average error on top by the spread underneath

Out loud. “The relative error is the average size of the rounding errors, divided by the standard deviation of the weights themselves.”

Worked, step by step, on four made-up weights. Take 0.10, -0.06, 0.03, -0.02697754, quantized at 4 bits with the whole-matrix scale $s = 0.17522321$ from 4.1. The last of the four is the real typical weight from 4.2; the other three are chosen so the arithmetic is short.

Step 1, quantize and dequantize each one, using 4.2.

$w_i$	$w_i \div s$	$q_i$	$\hat{w}_i$	$\lvert \hat{w}_i - w_i \rvert$
0.10000000	0.5707	1	0.17522321	0.07522321
-0.06000000	-0.3424	0	0	0.06000000
0.03000000	0.1712	0	0	0.03000000
-0.02697754	-0.1540	0	0	0.02697754

Step 2, add the four errors.

$0.07522321 + 0.06000000 = 0.13522321$

$0.13522321 + 0.03000000 = 0.16522321$

$0.16522321 + 0.02697754 = 0.19220075$

Step 3, divide by $N = 4$ to get the average error.

$0.19220075 \div 4 = 0.04805019$

Step 4, work out the standard deviation of the four original weights. First their average.

$0.10 + (-0.06) + 0.03 + (-0.02697754) = 0.04302246$

$0.04302246 \div 4 = 0.01075562$

Step 5, subtract that average from each weight and square the result.

$(0.10000000 - 0.01075562)^2 = (0.08924438)^2 = 0.00796456$

$(-0.06000000 - 0.01075562)^2 = (-0.07075562)^2 = 0.00500636$

$(0.03000000 - 0.01075562)^2 = (0.01924438)^2 = 0.00037035$

$(-0.02697754 - 0.01075562)^2 = (-0.03773316)^2 = 0.00142379$

Step 6, add those four squares.

$0.00796456 + 0.00500636 = 0.01297092$

$0.01297092 + 0.00037035 = 0.01334127$

$0.01334127 + 0.00142379 = 0.01476506$

Step 7, divide by $N - 1 = 3$ , then take the square root.

$0.01476506 \div 3 = 0.00492169$

$\sqrt{0.00492169} = 0.07015476$

Step 8, divide the average error by the standard deviation.

$0.04805019 \div 0.07015476 = 0.684917$

$0.684917 \times 100 = \mathbf{68.5\%}$

What the real matrix gives. Those four weights are a made-up sample, not the matrix. Across all 802,816 weights, the measured mean absolute error at 4 bits with one scale for the whole matrix is 0.031098, and the measured standard deviation of the weights is 0.066741.

$0.031098 \div 0.066741 = 0.4660$ , which is the 46.6% quoted in Chapter 7.

With one scale per 32 weights the mean error drops to 0.004831, and $0.004831 \div 0.066741 = 0.0724$ , or 7.2%.

bits	one scale per matrix	one scale per 32 weights
8	3.6%	0.4%
6	14.7%	1.6%
4	46.6%	7.2%
3	58.9%	16.8%
2	62.7%	44.7%

Sanity check. Relative error can never be negative, because both the top and the bottom are built from things that cannot be negative. It should also fall as you spend more bits: reading either column of that table upwards, the numbers get smaller, which is the direction you would expect. If your relative error rises when you add bits, you used the wrong scale somewhere.

Chapter. Chapter 7

4.4 Blockwise quantization (USE)¶

In words. Instead of one ruler for the whole matrix, chop the weights into short runs of 32 and give each run its own ruler, sized by the largest weight in that run alone. Then one unusually large weight can only spoil the 31 weights next to it, rather than all 802,816.

The formula.

s^{(k)} = \frac{\displaystyle\max_{i \,\in\, \text{block } k} \lvert w_i \rvert}{2^{\,b-1} - 1}, \qquad q_i = \operatorname{round}\!\left(\frac{w_i}{s^{(k)}}\right), \qquad \hat{w}_i = q_i \, s^{(k)}

The symbols.

Symbol	How to say it out loud	What it means
a block	“block”	a short run of $B$ weights that sit next to each other. $B = 32$ throughout this book.
$k$	“kay”	which block you are in: block 1, block 2, and so on
$s^{(k)}$	“ess bracket kay”	that block’s own scale. The raised $(k)$ is a label, not a power.
the raised bracketed $(k)$	“bracket kay”	a label saying which block. It is not an exponent.
$\max_{i \in \text{block } k}$	“the maximum over i in block kay”	look only at the weights inside block $k$ and keep the biggest answer
$\in$	“in”, or “is an element of”	says which weights you are allowed to look at
$\lvert w_i \rvert$	“absolute value of w sub i”	the weight with its minus sign thrown away
$2^{\,b-1} - 1$	“two to the bee minus one, minus one”	the number of whole steps on each side of zero, exactly as in 4.1
$q_i$	“q sub i”	the whole number stored for weight $i$
$\hat{w}_i$	“w hat sub i”	the weight you get back
$\operatorname{round}$	“round”	round to the nearest whole number
the fraction bar	“divided by”	divide the top by the bottom
the commas	no sound	they separate three formulas carried out in order

Out loud. “Each block gets its own scale, which is the largest weight size inside that block divided by the number of steps; then every weight in the block is divided by that scale and rounded, and multiplied back by that same scale to be read.”

Worked, step by step, on a block of four weights at 4 bits. The block is 0.10, -0.06, 0.03, -0.02697754, the same four as in 4.3. Real blocks hold 32 weights; four keeps the arithmetic short.

Step 1, find the largest magnitude inside the block. The four magnitudes are 0.10, 0.06, 0.03 and 0.02697754, so the largest is 0.10.

Step 2, work out the number of steps, exactly as in 4.1. At $b = 4$ this is $2^{3} - 1 = 8 - 1 = 7$ .

Step 3, divide to get this block’s scale. $s^{(k)} = 0.10 \div 7 = 0.01428571$

Step 4, quantize and dequantize each weight with that scale.

$w_i$	$w_i \div s^{(k)}$	$q_i$	$\hat{w}_i = q_i \times s^{(k)}$	error
0.10000000	7.00000	7	0.09999997	0.00000003
-0.06000000	-4.20000	-4	-0.05714284	0.00285716
0.03000000	2.10000	2	0.02857142	0.00142858
-0.02697754	-1.88843	-2	-0.02857142	0.00159388

Take the fourth row line by line. $-0.02697754 \div 0.01428571 = -1.88843$ , which is nearer to -2 than to -1, so $q = -2$ . Then $-2 \times 0.01428571 = -0.02857142$ , and the error is $0.02857142 - 0.02697754 = 0.00159388$ .

Step 5, redo the relative error from 4.3 on these new numbers.

$0.00000003 + 0.00285716 = 0.00285719$

$0.00285719 + 0.00142858 = 0.00428577$

$0.00428577 + 0.00159388 = 0.00587965$

$0.00587965 \div 4 = 0.00146991$

The standard deviation of the four weights has not changed, because the weights have not changed: it is still 0.07015476 from 4.3.

$0.00146991 \div 0.07015476 = 0.020952$ , which is 2.1%.

Compare with 4.3, where the same four weights under the whole-matrix scale gave 68.5%. The arithmetic is identical apart from which maximum you divided by. That one change took the error from 68.5% of the spread to 2.1% of it.

Sanity check. Inside a block, the largest weight must come back exactly, or very nearly, so check it first: 0.10 came back as 0.09999997. A block scale can never be larger than the whole-matrix scale, because the largest weight in one block cannot exceed the largest weight overall. So blockwise error can never be worse than whole-matrix error. If yours is, you took the maximum over the wrong set of weights.

Chapter. Chapter 7

4.5 Bytes per weight, including the block scales (USE)¶

In words. Every block needs its own scale written down somewhere, and that scale takes up room too. This counts the honest storage cost: the bits for the weight itself, plus each weight’s share of its block’s scale, converted from bits into bytes.

The formula.

\text{bytes per weight} = \frac{b + \dfrac{b_{\text{scale}}}{B}}{8}

The symbols.

Symbol	How to say it out loud	What it means
$\text{bytes per weight}$	“bytes per weight”	the answer. It can be less than 1, which is the point of the exercise.
$=$	“equals”	the two sides are the same number
$b$	“bee”	bits spent on one stored weight
$+$	“plus”	add
$b_{\text{scale}}$	“bee sub scale”	bits spent on one block scale. It is 16 in this book, because the scale is kept in half precision.
$B$	“capital bee”	how many weights share that one scale. It is 32 in this book.
the small fraction bar	“divided by”	dividing the scale’s bits by the number of weights sharing it gives each weight’s share of the bookkeeping
the big fraction bar	“divided by”	divide everything above it by the 8 below it
8	“eight”	there are 8 bits in a byte, so dividing by 8 turns bits into bytes

Out loud. “Bytes per weight is the bits used for a weight, plus the bits used for a block scale shared out across the weights in that block, all divided by eight.”

Worked, step by step, for 4-bit weights with a 16-bit scale per 32 weights.

Step 1, work out the bookkeeping share. $16 \div 32 = 0.5$ bits per weight

Step 2, add it to the bits spent on the weight itself. $4 + 0.5 = 4.5$ bits per weight

Step 3, turn bits into bytes by dividing by 8. $4.5 \div 8 = \mathbf{0.5625}$ bytes per weight

Step 4, compare with the cost if you ignored the scales. $4 \div 8 = 0.5$ bytes per weight, and $0.5625 \div 0.5 = 1.125$ , so the bookkeeping costs 12.5% on top. That is the whole overhead.

scheme	$b$	$b_{\text{scale}} \div B$	bytes per weight
FP32	32	0	4.0000
FP16	16	0	2.0000
INT8	8	0	1.0000
4-bit, 16-bit scale per 32	4	0.5	0.5625
4-bit, no overhead counted	4	0	0.5000

Sanity check. Multiply your answer by 8 and you must get back the total bits per weight. For the 4-bit blockwise row, $0.5625 \times 8 = 4.5$ , which is the number from Step 2. When there are no block scales the middle term is 0 and the formula becomes $b \div 8$ , so 32-bit storage gives $32 \div 8 = 4$ bytes, which is what FP32 means.

Chapter. Chapter 6, Bits, precision, and rounding and Chapter 7

4.6 Model size on disk (USE)¶

In words. Multiply how many parameters the model has by how many bytes each one takes up. That gives you the size of the file you are about to download.

The formula.

\text{size in bytes} = N \times (\text{bytes per weight})

The symbols.

Symbol	How to say it out loud	What it means
$\text{size in bytes}$	“size in bytes”	the answer, in bytes
$=$	“equals”	the two sides are the same number
$N$	“en”	the parameter count. $N = 494{,}032{,}768$ for Qwen2.5-0.5B-Instruct.
$\times$	“times”	multiply
$\text{bytes per weight}$	“bytes per weight”	the answer from 4.5
the round brackets	“bracket”	they group the quantity so you can see it counts as one number

Throughout this book GB means 10⁹ bytes, which is how model files are usually quoted.

Out loud. “The size in bytes is the number of parameters multiplied by the number of bytes each parameter takes up.”

Worked, step by step, at FP16.

Step 1, write down the parameter count: $494{,}032{,}768$ .

Step 2, write down the bytes per weight from 4.5: FP16 is 16 bits, and $16 \div 8 = 2$ bytes.

Step 3, multiply. $494{,}032{,}768 \times 2 = 988{,}065{,}536$ bytes

Step 4, turn bytes into gigabytes by dividing by $1{,}000{,}000{,}000$ . $988{,}065{,}536 \div 1{,}000{,}000{,}000 = 0.988$ GB

That is why a “0.5B model” arrives as a file a little under a gigabyte.

Worked again, at 4 bits with block scales.

Step 1, $N = 494{,}032{,}768$ .

Step 2, bytes per weight from 4.5: 0.5625.

Step 3, $494{,}032{,}768 \times 0.5625 = 277{,}893{,}432$ bytes.

Step 4, $277{,}893{,}432 \div 1{,}000{,}000{,}000 = 0.278$ GB.

scheme	bytes per weight	size in bytes	size
FP32	4.0000	1,976,131,072	1.976 GB
FP16	2.0000	988,065,536	0.988 GB
INT8	1.0000	494,032,768	0.494 GB
4-bit, 16-bit scale per 32	0.5625	277,893,432	0.278 GB
4-bit, no overhead counted	0.5000	247,016,384	0.247 GB

Sanity check. Halving the bytes per weight must halve the file size. Read the table down from FP32 to FP16 to INT8: 1.976, then 0.988, then 0.494, each one half the last. If your numbers do not halve, check whether you divided by 8 when converting bits to bytes. One more check on the size of the answer: at FP32 this model is about 2 GB, and 2 GB for half a billion numbers at 4 bytes each is the right order of magnitude.

Chapter. Chapter 3, Chapter 6, Chapter 7

5. Meaning as geometry¶

A vector is an ordered list of numbers, such as $(3, 4)$ . When there are two numbers in the list you can draw it as an arrow on graph paper: go 3 across, then 4 up. The models in this book use lists of 384 numbers, which nobody can draw, but every formula below works the same way no matter how long the list is.

5.1 Dot product (USE)¶

In words. Take two lists of numbers of the same length. Multiply the first number of one list by the first number of the other, then the second by the second, and so on. Add up all those products. The answer is one number.

The formula.

\mathbf{u} \cdot \mathbf{v} = \sum_{i=1}^{D} u_i v_i = u_1 v_1 + u_2 v_2 + \cdots + u_D v_D

The symbols.

Symbol	How to say it out loud	What it means
$\mathbf{u}$ , $\mathbf{v}$	“bold u”, “bold v”	two vectors, each a list of $D$ numbers. Bold type marks a list rather than a single number.
$\cdot$	“dot”	the dot product operation itself. It is the name of the whole procedure, not plain multiplication.
$=$	“equals”	the two sides are the same number
$\sum_{i=1}^{D}$	“the sum from i equals one to capital dee”	add up what follows, once for each position in the list
$i$	“eye”	which position in the list you are looking at
$u_i$	“u sub i”	the number in position $i$ of the list $\mathbf{u}$
$v_i$	“v sub i”	the number in position $i$ of the list $\mathbf{v}$
$u_i v_i$	“u sub i times v sub i”	two things written next to each other are multiplied
$D$	“capital dee”	how many numbers are in each list. Both lists must have the same $D$ .
$+$	“plus”	add
$\cdots$	“and so on”	keep going in the same pattern to the end of the list

Out loud. “The dot product of two vectors is the sum, over every position in the list, of the number in that position of the first vector multiplied by the number in that position of the second.”

Worked, step by step, with $\mathbf{a} = (3, 4)$ and $\mathbf{b} = (4, 3)$ . Here $D = 2$ .

Step 1, multiply the first numbers. $3 \times 4 = 12$ .

Step 2, multiply the second numbers. $4 \times 3 = 12$ .

Step 3, add the two products. $12 + 12 = \mathbf{24}$ .

Worked again, with $\mathbf{a} = (3, 4)$ and $\mathbf{c} = (-4, 3)$ .

Step 1, $3 \times (-4) = -12$ . A positive times a negative gives a negative.

Step 2, $4 \times 3 = 12$ .

Step 3, $-12 + 12 = \mathbf{0}$ .

Worked once more, with $\mathbf{a} = (3, 4)$ and $\mathbf{d} = (6, 8)$ .

Step 1, $3 \times 6 = 18$ .

Step 2, $4 \times 8 = 32$ .

Step 3, $18 + 32 = \mathbf{50}$ .

Sanity check. The answer is one number, never a list. If you finished with a list, you forgot to add the products up. A dot product of exactly zero means the two arrows meet at a right angle, which is what happened with $\mathbf{a}$ and $\mathbf{c}$ . A large positive answer means the arrows point roughly the same way and are long; a negative answer means they point in opposing directions. And the order does not matter: $\mathbf{a} \cdot \mathbf{b}$ and $\mathbf{b} \cdot \mathbf{a}$ both give 24.

Chapter. Chapter 8, A sentence is an arrow

5.2 Length of a vector (USE)¶

In words. Square every number in the list, add the squares up, and take the square root of the total. In two dimensions this is the length of the arrow you would draw, and the formula is Pythagoras.

The formula.

\lVert \mathbf{u} \rVert = \sqrt{\mathbf{u} \cdot \mathbf{u}} = \sqrt{\sum_{i=1}^{D} u_i^{2}}

The symbols.

Symbol	How to say it out loud	What it means
$\lVert \mathbf{u} \rVert$	“the norm of u”, or “the length of u”	the answer: how long the arrow is. Also called the magnitude.
the double upright bars	“norm”, or “length”	they are the length instruction. Single bars mean absolute value; double bars mean vector length.
$=$	“equals”	the two sides are the same number
$\sqrt{\phantom{x}}$	“the square root of”	the number which, multiplied by itself, gives what is underneath. $\sqrt{25} = 5$ because $5 \times 5 = 25$ .
the bar across the top of the root sign	no sound	it shows how far the square root reaches. Everything underneath it goes inside.
$\mathbf{u} \cdot \mathbf{u}$	“u dot u”	the dot product of the vector with itself, from 5.1
$\sum_{i=1}^{D}$	“the sum from i equals one to capital dee”	add up what follows, once for each position in the list
$u_i^{2}$	“u sub i squared”	the number in position $i$ , multiplied by itself
the raised 2	“squared”	multiply the thing by itself once

Out loud. “The length of a vector is the square root of the sum of the squares of its numbers.”

Worked, step by step, with $\mathbf{a} = (3, 4)$ .

Step 1, square each number. $3^{2} = 3 \times 3 = 9$ , and $4^{2} = 4 \times 4 = 16$ .

Step 2, add the squares. $9 + 16 = 25$ .

Step 3, take the square root. $\sqrt{25} = \mathbf{5}$ , because $5 \times 5 = 25$ .

Worked again, with $\mathbf{b} = (4, 3)$ . $4^{2} = 16$ , $3^{2} = 9$ , $16 + 9 = 25$ , $\sqrt{25} = \mathbf{5}$ . Same length as $\mathbf{a}$ , pointing a different way.

Worked once more, with $\mathbf{d} = (6, 8)$ . $6^{2} = 36$ , $8^{2} = 64$ , $36 + 64 = 100$ , $\sqrt{100} = \mathbf{10}$ .

Sanity check. A length can never be negative, because squaring removes every minus sign and the square root of a positive number is positive. A length can only be zero if every number in the list is zero. And a length is always at least as big as the largest single number in the list, ignoring signs: for $\mathbf{d} = (6, 8)$ the length 10 is bigger than 8, as it must be. Notice also that $\mathbf{d}$ is exactly twice $\mathbf{a}$ , and its length is exactly twice as big. That fact is the whole point of 5.3.

Chapter. Chapter 8

5.3 Cosine similarity (USE)¶

In words. The dot product gets bigger when the arrows point the same way, but it also gets bigger when either arrow is longer. Dividing by both lengths removes the effect of length and leaves only direction. That is what you want when you are asking whether two pieces of text mean the same thing, because a longer sentence should not count as a better match.

The formula.

\cos(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u} \cdot \mathbf{v}} {\lVert \mathbf{u} \rVert \, \lVert \mathbf{v} \rVert}

The symbols.

Symbol	How to say it out loud	What it means
$\cos(\mathbf{u}, \mathbf{v})$	“cosine of u and v”	the answer: a number between -1 and +1
the round brackets and comma	“of ... and ...”	they hold the two vectors the answer depends on. They do not mean multiply.
$=$	“equals”	the two sides are the same number
$\mathbf{u} \cdot \mathbf{v}$	“u dot v”	the dot product from 5.1
$\lVert \mathbf{u} \rVert$	“the length of u”	the length from 5.2
$\lVert \mathbf{v} \rVert$	“the length of v”	the length of the other vector
the space between the two lengths	“times”	multiply the two lengths together
the fraction bar	“divided by”	divide the dot product on top by the product of the lengths underneath

Out loud. “The cosine similarity of two vectors is their dot product, divided by the length of the first multiplied by the length of the second.”

Worked, step by step, with $\mathbf{a} = (3, 4)$ and $\mathbf{b} = (4, 3)$ .

Step 1, the dot product, from 5.1. $3 \times 4 + 4 \times 3 = 12 + 12 = 24$ .

Step 2, the two lengths, from 5.2. $\lVert \mathbf{a} \rVert = 5$ and $\lVert \mathbf{b} \rVert = 5$ .

Step 3, multiply the lengths. $5 \times 5 = 25$ .

Step 4, divide. $24 \div 25 = \mathbf{0.9600}$ .

Worked again, with $\mathbf{a} = (3, 4)$ and $\mathbf{c} = (-4, 3)$ .

Step 1, dot product: $3 \times (-4) + 4 \times 3 = -12 + 12 = 0$ .

Step 2, lengths: 5 and 5.

Step 3, $5 \times 5 = 25$ .

Step 4, $0 \div 25 = \mathbf{0.0000}$ .

Worked once more, with $\mathbf{a} = (3, 4)$ and $\mathbf{d} = (6, 8)$ . This is the row that explains why the formula divides at all.

Step 1, dot product: $3 \times 6 + 4 \times 8 = 18 + 32 = 50$ .

Step 2, lengths: $\lVert \mathbf{a} \rVert = 5$ and $\lVert \mathbf{d} \rVert = 10$ .

Step 3, $5 \times 10 = 50$ .

Step 4, $50 \div 50 = \mathbf{1.0000}$ , exactly.

pair	dot product	lengths	product of lengths	cosine
$\mathbf{a} = (3,4)$ , $\mathbf{b} = (4,3)$	24	5 and 5	25	$24 \div 25 = \mathbf{0.9600}$
$\mathbf{a} = (3,4)$ , $\mathbf{c} = (-4,3)$	0	5 and 5	25	$0 \div 25 = \mathbf{0.0000}$
$\mathbf{a} = (3,4)$ , $\mathbf{d} = (6,8)$	50	5 and 10	50	$50 \div 50 = \mathbf{1.0000}$

Look at the last row. $\mathbf{d}$ is exactly $2\mathbf{a}$ : same direction, twice as long. Its dot product with $\mathbf{a}$ is 50, more than double the 24 in the first row, so a dot product on its own would rank $\mathbf{d}$ as the better match purely for being longer. Dividing by the lengths returns exactly 1. Length is not meaning, and that is the entire reason for the denominator.

Sanity check. The answer must land between -1 and +1. If it does not, you made an arithmetic slip, most often by forgetting to take a square root in Step 2. A cosine of exactly 1 means the two vectors point in exactly the same direction. A cosine of 0 means a right angle. A negative cosine means they point in opposing directions, and real embeddings do produce negative values: two unrelated sentences in the lab run scored -0.017.

Chapter. Chapter 9, Similarity is geometry

5.4 Angle between two vectors (READ)¶

In words. A cosine similarity can be turned back into the angle you would measure with a protractor if you drew the two arrows on paper. This is for intuition; the labs work with the cosine directly.

The formula.

\theta = \arccos\big(\cos(\mathbf{u}, \mathbf{v})\big)

The symbols.

Symbol	How to say it out loud	What it means
$\theta$	“theta”	the answer: the angle. It is a Greek letter, used for angles by long habit.
$=$	“equals”	the two sides are the same number
$\arccos$	“arc cosine”, or “inverse cosine”	the undo button for cosine. Give it a number between -1 and 1 and it hands back the angle that produces it.
the calculator key $\cos^{-1}$	“cos inverse”	the same thing on a calculator. The raised -1 here means “undo”, not “one over”.
the round brackets	“of”	they hold the number you are feeding to $\arccos$
$\cos(\mathbf{u}, \mathbf{v})$	“cosine of u and v”	the cosine similarity from 5.3

Out loud. “Theta is the inverse cosine of the cosine similarity of the two vectors.”

Worked, step by step, on the three pairs from 5.3. Put your calculator in degrees mode first, otherwise it will answer in radians and every number will look wrong.

Step 1, for $\mathbf{a}$ and $\mathbf{b}$ , the cosine was 0.96. Press $\cos^{-1}$ on 0.96: $\theta = 16.26^\circ$ .

Step 2, for $\mathbf{a}$ and $\mathbf{c}$ , the cosine was 0. Press $\cos^{-1}$ on 0: $\theta = 90.00^\circ$ , a right angle.

Step 3, for $\mathbf{a}$ and $\mathbf{d}$ , the cosine was 1. Press $\cos^{-1}$ on 1: $\theta = 0.00^\circ$ . No angle at all, because they point the same way.

Sanity check. The answer must land between $0^\circ$ and $180^\circ$ . If your calculator gives something like 0.2838 for the first one, it is in radians; switch to degrees. If it gives an error, you fed it a number outside -1 to 1, which means the cosine in 5.3 was already wrong.

Chapter. Chapter 9

5.5 Retrieval: pick the nearest passage (USE)¶

In words. Turn the question into a list of numbers. Turn every passage in your collection into a list of numbers the same way. Work out the cosine similarity between the question and each passage, and hand the model whichever passage scores highest.

The formula.

\text{best passage} = \arg\max_{k} \ \cos(\mathbf{q}, \mathbf{d}_k)

The symbols.

Symbol	How to say it out loud	What it means
$\text{best passage}$	“best passage”	the answer: which passage to retrieve. It is a passage, not a score.
$=$	“equals”	the two sides are the same thing
$\arg\max_{k}$	“arg max over kay”	“the value of $k$ that makes what follows as large as possible”. $\max$ would give you the best score; $\arg\max$ gives you which one scored it.
$k$	“kay”	a counter over the passages: passage 1, passage 2, and so on
$\mathbf{q}$	“bold q”	the embedding of the question: the list of $D$ numbers a model produces to stand for that text. $D = 384$ for the model used here.
$\mathbf{d}_k$	“bold d sub k”	the embedding of passage number $k$
$\cos(\mathbf{q}, \mathbf{d}_k)$	“cosine of q and d sub k”	the cosine similarity from 5.3, between the question and passage $k$

Out loud. “The best passage is the one whose embedding has the largest cosine similarity with the question’s embedding.”

Worked, step by step, on made-up two-number embeddings. Suppose the question comes out as $\mathbf{q} = (3, 4)$ , and there are three passages, $\mathbf{d}_1 = (4, 3)$ , $\mathbf{d}_2 = (-4, 3)$ and $\mathbf{d}_3 = (6, 8)$ .

Step 1, cosine with passage 1. Dot product $3 \times 4 + 4 \times 3 = 24$ ; lengths 5 and 5; $24 \div 25 = 0.9600$ .

Step 2, cosine with passage 2. Dot product $3 \times (-4) + 4 \times 3 = 0$ ; lengths 5 and 5; $0 \div 25 = 0.0000$ .

Step 3, cosine with passage 3. Dot product $3 \times 6 + 4 \times 8 = 50$ ; lengths 5 and 10; $50 \div 50 = 1.0000$ .

Step 4, pick the largest. The three scores are 0.9600, 0.0000 and 1.0000, so the largest is the third. Retrieve passage 3.

Worked on the real run. The query Which California county is Bakersfield in? was scored against six sentences. The cosines came back, sorted from largest:

rank	cosine
1	0.9101
2	0.7191
3	0.0893
4	0.0615
5	0.0556
6	0.0116

The largest is rank 1, so that passage is retrieved.

Sanity check. Every score has to sit between -1 and 1, because each one is a cosine. The useful check is not the top score on its own but the gap. Here rank 2 scores 0.7191 and rank 3 scores 0.0893, a drop of 0.6298. The two relevant passages separate cleanly from the four irrelevant ones, which is what makes retrieval work. If the top few scores are all close together, the method has not found anything; it has ranked noise.

Chapter. Chapter 10, Retrieval: the open-book exam

6. Judging a model like a statistician¶

6.1 Sample proportion, which is what accuracy is (USE)¶

In words. Count how many questions the model got right, divide by how many questions there were, and you have its accuracy.

The formula.

\hat{p} = \frac{x}{n}

The symbols.

Symbol	How to say it out loud	What it means
$\hat{p}$	“p hat”	the answer: the accuracy, a number between 0 and 1
the hat	“hat”	it marks a number you measured rather than a number you know for certain. It is the difference between “what this model scored on these 20 questions” and “how good this model is”.
$=$	“equals”	the two sides are the same number
$x$	“ex”	how many questions the model answered correctly
$n$	“en”	how many questions were on the test
the fraction bar	“divided by”	divide the top by the bottom

Out loud. “P hat is the number of questions answered correctly, divided by the number of questions asked.”

Worked, step by step. Qwen2.5-0.5B-Instruct answered 5 of 20 questions correctly under the naive scoring procedure.

Step 1, write down $x = 5$ and $n = 20$ .

Step 2, divide. $5 \div 20 = 0.25$ .

Step 3, turn it into a percentage by multiplying by 100. $0.25 \times 100 = \mathbf{25.0\%}$ .

Worked again, for the set piece used in 6.3 and 6.4: $x = 8$ out of $n = 10$ .

Step 1, $x = 8$ , $n = 10$ .

Step 2, $8 \div 10 = 0.8$ .

Step 3, $0.8 \times 100 = \mathbf{80.0\%}$ .

Sanity check. Accuracy must land between 0 and 1, which is 0% and 100%. If you get a number above 1 you divided the wrong way round. Here is the part that matters more than the arithmetic: because your questions are a sample of the questions you could have asked, $\hat{p}$ is a random variable, not a fixed property of the model. Run a different 20 questions and you get a different number. Chance on a four-option multiple-choice test is 25%, which is exactly what this model scored, and that coincidence is the observation that starts Chapter 13.

Chapter. Chapter 11, Can a model take a test?

6.2 Standard error of a proportion (USE)¶

In words. An accuracy measured on a handful of questions would come out differently if you had asked a different handful. This puts a size on that wobble, in the same units as the accuracy itself.

The formula.

SE = \sqrt{\frac{\hat{p}\,(1 - \hat{p})}{n}}

The symbols.

Symbol	How to say it out loud	What it means
$SE$	“ess ee”	the answer: the standard error, an estimate of how far $\hat{p}$ would typically move if you drew a fresh set of $n$ questions of the same kind
$=$	“equals”	the two sides are the same number
$\sqrt{\phantom{x}}$	“the square root of”	the number which, multiplied by itself, gives what is underneath
$\hat{p}$	“p hat”	the measured accuracy, from 6.1
$1 - \hat{p}$	“one minus p hat”	the share the model got wrong
the round brackets	“bracket”	they say to work out $1 - \hat{p}$ first, before multiplying
the space between $\hat{p}$ and the bracket	“times”	multiply
$n$	“en”	how many questions were on the test
the fraction bar	“divided by”	divide the top by the bottom

Out loud. “The standard error is the square root of the accuracy multiplied by one minus the accuracy, all divided by the number of questions.”

Worked, step by step, with $\hat{p} = 0.25$ and $n = 20$ .

Step 1, work out $1 - \hat{p}$ . $1 - 0.25 = 0.75$ .

Step 2, multiply. $0.25 \times 0.75 = 0.1875$ .

Step 3, divide by $n$ . $0.1875 \div 20 = 0.009375$ .

Step 4, take the square root. $\sqrt{0.009375} = \mathbf{0.096825}$ .

Nearly ten percentage points of wobble on a 20-question test.

Worked again, with $\hat{p} = 0.8$ and $n = 10$ .

Step 1, $1 - 0.8 = 0.2$ .

Step 2, $0.8 \times 0.2 = 0.16$ .

Step 3, $0.16 \div 10 = 0.016$ .

Step 4, $\sqrt{0.016} = \mathbf{0.126491}$ .

Sanity check. The standard error can never be negative, and it can never be larger than 0.5. It is largest when $\hat{p} = 0.5$ , because $0.5 \times 0.5 = 0.25$ is the biggest that top can get, and it shrinks toward zero as the accuracy approaches 0 or 1. Here is the fact worth carrying around: because $n$ sits under a square root, cutting the standard error in half takes four times as many questions, not twice as many.

Chapter. Chapter 12, Is that score real?

6.3 Wald confidence interval for a proportion (USE)¶

In words. Take the accuracy you measured and put a range around it, stretching about two standard errors either side. This is the textbook interval, the first one anyone learns, and the first one to fail.

The formula.

\hat{p} \ \pm\ z^\star \sqrt{\frac{\hat{p}\,(1 - \hat{p})}{n}}

The symbols.

Symbol	How to say it out loud	What it means
$\hat{p}$	“p hat”	the measured accuracy, from 6.1. It sits at the centre of the interval.
±	“plus or minus”	do the calculation twice: once adding, once subtracting. You get two endpoints, a lower and an upper.
$z^\star$	“z star”	the critical value, a fixed multiplier set by how confident you want to be. $z^\star = 1.96$ for 95%, and that is the only value this book uses.
the square root and what is under it	“the standard error”	this is exactly the $SE$ from 6.2
$n$	“en”	how many questions were on the test

Out loud. “The interval runs from the accuracy minus 1.96 standard errors, to the accuracy plus 1.96 standard errors.”

Conditions. The usual rule is at least 10 correct and at least 10 incorrect. Check this before you compute anything.

Worked, step by step, instance 1: $\hat{p} = 0.25$ , $n = 20$ .

Step 1, the standard error, from 6.2: $SE = 0.096825$ .

Step 2, multiply by $z^\star$ . $1.96 \times 0.096825 = 0.189777$ .

Step 3, subtract for the lower end. $0.25 - 0.189777 = 0.060223$ .

Step 4, add for the upper end. $0.25 + 0.189777 = 0.439777$ .

The interval is $\mathbf{0.0602}$ to $\mathbf{0.4398}$ , that is $6.0\%$ to $44.0\%$ . The honest statement about that model is not “25%”; it is “somewhere between 6% and 44%”, which is wide enough to be almost useless. That uselessness is the lesson about benchmark sizes. Note also that this instance already breaks its own conditions: there were only 5 correct answers, and the rule asks for at least 10.

Worked, step by step, instance 2, the one to remember: $x = 8$ , $n = 10$ .

Step 1, the accuracy, from 6.1. $8 \div 10 = 0.8$ .

Step 2, the standard error, from 6.2. $1 - 0.8 = 0.2$ ; $0.8 \times 0.2 = 0.16$ ; $0.16 \div 10 = 0.016$ ; $\sqrt{0.016} = 0.126491$ .

Step 3, multiply by $z^\star$ . $1.96 \times 0.126491 = 0.247922$ .

Step 4, subtract. $0.8 - 0.247922 = 0.552078$ .

Step 5, add. $0.8 + 0.247922 = \mathbf{1.047922}$ .

The interval is $\mathbf{0.5521}$ to $\mathbf{1.0479}$ .

Sanity check. The interval must sit inside 0 and 1, because an accuracy outside that range is meaningless. Instance 2 fails that check, and the failure is the lesson.

Chapter. Chapter 12

6.4 Wilson confidence interval for a proportion (READ)¶

In words. The same job as the Wald interval, done in a way that cannot produce an impossible answer. It shifts the centre of the interval away from the measured accuracy and toward the middle, by an amount that shrinks as you ask more questions, and it can never run below 0 or above 1.

The formula.

\text{centre} = \frac{x + \dfrac{z^{\star 2}}{2}}{n + z^{\star 2}}, \qquad \text{half-width} = \frac{z^\star}{n + z^{\star 2}} \sqrt{\hat{p}(1 - \hat{p})\,n + \frac{z^{\star 2}}{4}}

\text{interval} = \text{centre} \ \pm\ \text{half-width}

The symbols.

Symbol	How to say it out loud	What it means
$\text{centre}$	“centre”	the middle of the interval. It is not $\hat{p}$ : it is $\hat{p}$ pulled toward 0.5.
$\text{half-width}$	“half width”	how far the interval reaches on each side of the centre
$x$	“ex”	how many questions the model got right
$n$	“en”	how many questions there were
$\hat{p}$	“p hat”	the measured accuracy $x/n$ , from 6.1
$z^\star$	“z star”	the critical value, 1.96 for 95%
$z^{\star 2}$	“z star squared”	$1.96 \times 1.96 = 3.8416$
the raised 2	“squared”	multiply the thing by itself
$\dfrac{z^{\star 2}}{2}$	“z star squared over two”	$3.8416 \div 2 = 1.9208$ . Adding this to $x$ is what pulls the centre toward the middle.
$\dfrac{z^{\star 2}}{4}$	“z star squared over four”	$3.8416 \div 4 = 0.9604$
$+$	“plus”	add
±	“plus or minus”	do it twice, once subtracting and once adding
$\sqrt{\phantom{x}}$	“the square root of”	the number which, multiplied by itself, gives what is underneath
the fraction bars	“divided by”	divide the top by the bottom
the comma between the two formulas	no sound	it separates two quantities you work out before combining them

Out loud. “The centre is the number correct plus half of z star squared, divided by the number of questions plus z star squared. The half-width is z star divided by that same bottom, multiplied by the square root of the accuracy times one minus the accuracy times the number of questions, plus a quarter of z star squared. The interval is the centre plus or minus the half-width.”

Worked, step by step, on the same $x = 8$ , $n = 10$ .

Step 1, square the critical value. $1.96 \times 1.96 = 3.8416$ .

Step 2, build the bottom of both fractions. $10 + 3.8416 = 13.8416$ .

Step 3, build the top of the centre. $3.8416 \div 2 = 1.9208$ , then $8 + 1.9208 = 9.9208$ .

Step 4, divide to get the centre. $9.9208 \div 13.8416 = 0.716738$ .

Step 5, build what goes under the square root. $\hat{p} = 0.8$ , so $\hat{p}(1-\hat{p}) = 0.8 \times 0.2 = 0.16$ ; $0.16 \times 10 = 1.6$ ; $3.8416 \div 4 = 0.9604$ ; $1.6 + 0.9604 = 2.5604$ .

Step 6, take the square root. $\sqrt{2.5604} = 1.600125$ .

Step 7, work out the multiplier in front. $1.96 \div 13.8416 = 0.141602$ .

Step 8, multiply to get the half-width. $0.141602 \times 1.600125 = 0.226581$ .

Step 9, subtract and add. $0.716738 - 0.226581 = \mathbf{0.490157}$ $0.716738 + 0.226581 = \mathbf{0.943319}$

The interval is $\mathbf{0.4902}$ to $\mathbf{0.9433}$ .

Put the two methods side by side. This is the comparison to carry out of Chapter 12.

$x = 8$ , $n = 10$ , 95%	lower	upper	inside 0 to 1?
Wald (6.3)	0.5521	1.0479	no
Wilson (6.4)	0.4902	0.9433	yes

The Wilson interval is not the Wald interval with the top trimmed off. It is lower at both ends, because it recognises that a model scoring 8 of 10 is more likely to be a mediocre model that got lucky than an excellent one that got unlucky.

Worked again, on the 20-question run: $x = 5$ , $n = 20$ .

Step 1, $z^{\star 2} = 3.8416$ .

Step 2, $20 + 3.8416 = 23.8416$ .

Step 3, $5 + 1.9208 = 6.9208$ .

Step 4, $6.9208 \div 23.8416 = 0.290283$ .

Step 5, $0.25 \times 0.75 = 0.1875$ ; $0.1875 \times 20 = 3.75$ ; $3.75 + 0.9604 = 4.7104$ .

Step 6, $\sqrt{4.7104} = 2.170346$ .

Step 7, $1.96 \div 23.8416 = 0.082209$ .

Step 8, $0.082209 \times 2.170346 = 0.178422$ .

Step 9, $0.290283 - 0.178422 = 0.111860$ and $0.290283 + 0.178422 = 0.468705$ .

Wilson gives $\mathbf{0.1119}$ to $\mathbf{0.4687}$ , against Wald’s 0.0602 to 0.4398. Both are wide. Neither rescues a 20-question benchmark.

Sanity check. Both endpoints must land inside 0 and 1, every time, for any $x$ and any $n$ . That is the whole reason this formula exists, so if an endpoint escapes that range, you made an arithmetic slip. The centre must also sit between $\hat{p}$ and 0.5: for $x = 8$ , $n = 10$ the centre 0.7167 sits between 0.5 and 0.8, as it should. Report Wilson, and let software compute it; the arithmetic above is here so you can see there is no magic in it.

Chapter. Chapter 12

6.5 Bootstrap percentile interval (USE)¶

In words. Instead of trusting a formula, ask what would have happened if the luck of the draw had fallen differently. Build a new test by drawing questions at random from the ones you already asked, allowing repeats, and score it using the answers the model already gave. Do that ten thousand times, sort the ten thousand scores, and read off the value 2.5% of the way up and the value 97.5% of the way up.

The formula.

\text{interval} = \Big(\ \hat{p}^{*}_{(\lceil 0.025\,B \rceil)}, \ \ \hat{p}^{*}_{(\lceil 0.975\,B \rceil)}\ \Big)

The symbols.

Symbol	How to say it out loud	What it means
a resample	“resample”	a new set of $n$ questions drawn at random from your $n$ questions, with replacement, so a question can appear twice or not at all
$B$	“capital bee”	how many resamples you take. $B = 10{,}000$ in this book.
$\hat{p}^{*}$	“p hat star”	an accuracy computed on a resample rather than on the real test. The star marks it as resampled.
$\hat{p}^{*}_{(r)}$	“p hat star bracket r”	the $r$ -th smallest of the $B$ resampled accuracies, once they are sorted from smallest to largest. The brackets around $r$ mean the list has been sorted.
$\lceil\ \rceil$	“ceiling”	round up to the next whole number. $\lceil 4.1 \rceil = 5$ and $\lceil 4.0 \rceil = 4$ .
0.025	“nought point nought two five”	2.5%, the share of resamples left below the lower end
0.975	“nought point nine seven five”	97.5%, the share of resamples left below the upper end
the space between 0.025 and $B$	“times”	multiply
the outer round brackets and the comma	“the interval from ... to ...”	they hold the two endpoints

Out loud. “The interval runs from the resampled accuracy two and a half percent of the way up the sorted list, to the resampled accuracy ninety-seven and a half percent of the way up.”

Worked, step by step, part one: what one resample looks like. Take a made-up 5-question test where the model got questions 1, 2 and 4 right and questions 3 and 5 wrong. Write that as $1, 1, 0, 1, 0$ , so the real score is $3 \div 5 = 0.6$ .

Step 1, draw 5 question numbers at random from 1 to 5, allowing repeats. Suppose you draw 2, 2, 5, 1, 4.

Step 2, look up the model’s answer for each of those questions: question 2 was right (1), question 2 again (1), question 5 was wrong (0), question 1 was right (1), question 4 was right (1).

Step 3, score the resample. $1 + 1 + 0 + 1 + 1 = 4$ , so $\hat{p}^{*} = 4 \div 5 = 0.8$ .

One resample, one number. The computer does this $B$ times.

Worked, step by step, part two: reading off the endpoints. This is the part you do by hand.

Step 1, write down $B = 10{,}000$ .

Step 2, work out the lower rank. $0.025 \times 10{,}000 = 250$ . Rounding 250 up leaves it at 250.

Step 3, work out the upper rank. $0.975 \times 10{,}000 = 9{,}750$ .

Step 4, sort the 10,000 resampled accuracies from smallest to largest, and read off the 250th value and the 9,750th value.

On the lab run, seeded 20260912 so it reproduces exactly, those two values are 0.05 and 0.45: an interval of $\mathbf{5.0\%}$ to $\mathbf{45.0\%}$ , with the 10,000 resampled accuracies having a standard deviation of 0.0974.

Sanity check. Both ranks must be whole numbers between 1 and $B$ , and the lower rank must be smaller than the upper one. The interval itself must land inside 0 and 1 automatically, because every resampled accuracy is a real accuracy computed on a real number of questions. Then compare it with the Wald interval on the same data, which was $6.0\%$ to $44.0\%$ . Two different methods agreeing closely is a good sign; two methods disagreeing wildly means one of them is being used outside its conditions.

Chapter. Chapter 12

6.6 Paired difference and its standard error (USE)¶

In words. When two models sat the same test, compare them question by question instead of score against score. For each question write down whether model B beat model A on that one question. Average those per-question results, and the difficulty of the questions stops being a source of noise.

The formula.

d_i = c_i^{(B)} - c_i^{(A)}, \qquad \bar{d} = \frac{1}{n}\sum_{i=1}^{n} d_i

s_d = \sqrt{\frac{1}{n-1}\sum_{i=1}^{n}\big(d_i - \bar{d}\big)^2}, \qquad SE_{\bar{d}} = \frac{s_d}{\sqrt{n}}, \qquad \text{interval} = \bar{d} \pm z^\star\, SE_{\bar{d}}

The symbols.

Symbol	How to say it out loud	What it means
$c_i^{(A)}$	“c sub i bracket A”	1 if model A got question $i$ right, 0 if not. The raised $(A)$ is a label naming the model, not a power.
$c_i^{(B)}$	“c sub i bracket B”	the same thing for model B
$d_i$	“d sub i”	the difference on question $i$ . It can only be -1, 0 or +1.
$-$	“minus”	subtract
$\bar{d}$	“d bar”	the average of all the $d_i$ values. It always equals the difference in the two accuracies.
the bar over the $d$	“bar”	it marks an average
$\dfrac{1}{n}$	“one over en”	dividing by $n$ after adding $n$ things up is how you take an average
$\sum_{i=1}^{n}$	“the sum from i equals one to en”	add up what follows, once for every question
$s_d$	“ess sub d”	the standard deviation of the differences: how much the per-question results vary
$\big(d_i - \bar{d}\big)^2$	“bracket d sub i minus d bar, squared”	how far one question’s difference sits from the average difference, squared so the minus signs cannot cancel
$n-1$	“en minus one”	you divide by one fewer than the number of questions when computing a standard deviation from a sample
$\sqrt{\phantom{x}}$	“the square root of”	the number which, multiplied by itself, gives what is underneath
$SE_{\bar{d}}$	“ess ee sub d bar”	the standard error of the average difference
$\sqrt{n}$	“the square root of en”	the square root of the number of questions
$z^\star$	“z star”	1.96, for 95%
±	“plus or minus”	do it twice, once subtracting and once adding

Out loud. “The difference on a question is one model’s result minus the other’s. The average difference is the sum of those divided by the number of questions. The standard deviation of the differences is the square root of the sum of their squared distances from that average, divided by one fewer than the number of questions. The standard error is that standard deviation divided by the square root of the number of questions. The interval is the average difference plus or minus 1.96 standard errors.”

Worked, step by step. On 20 questions, Qwen2.5-3B was right on 19 and Qwen2.5-1.5B on 14. The contingency table says where the differences are.

	3B correct	3B wrong
1.5B correct	14	0
1.5B wrong	5	1

So on 5 questions the 3B won and the 1.5B lost, giving $d_i = +1$ . On 14 questions both were right and on 1 question both were wrong, giving $d_i = 0$ fifteen times. On no question did the 1.5B win.

Step 1, average the differences. $\bar{d} = 5 \div 20 = 0.25$ . Check it against the accuracies: $0.95 - 0.70 = 0.25$ . They agree, as they always must.

Step 2, work out each squared distance from that average. For the 5 questions where $d_i = 1$ : $(1 - 0.25)^2 = (0.75)^2 = 0.5625$ . For the 15 questions where $d_i = 0$ : $(0 - 0.25)^2 = (-0.25)^2 = 0.0625$ .

Step 3, add them all up. $5 \times 0.5625 = 2.8125$ $15 \times 0.0625 = 0.9375$ $2.8125 + 0.9375 = 3.75$

Step 4, divide by $n - 1 = 19$ , then take the square root. $3.75 \div 19 = 0.197368$ $\sqrt{0.197368} = 0.444262$

Step 5, divide by the square root of $n$ . $\sqrt{20} = 4.472136$ $0.444262 \div 4.472136 = \mathbf{0.099340}$

Step 6, build the interval. $1.96 \times 0.099340 = 0.194706$ $0.25 - 0.194706 = 0.055294$ $0.25 + 0.194706 = 0.444706$

The interval is 0.0553 to 0.4447, that is $\mathbf{+5.5}$ to $\mathbf{+44.5}$ percentage points.

Sanity check. $\bar{d}$ must equal the difference in the two accuracies; if it does not, you mis-tallied the contingency table. Every $d_i$ must be -1, 0 or +1, and the four cells of the table must add to $n$ : $14 + 0 + 5 + 1 = 20$ . The questions both models got right, and the questions both got wrong, contribute $d_i = 0$ , which pulls the standard deviation down. That is the payoff from pairing, and 6.7 measures how big it is.

Chapter. Chapter 13, Measure your measurement

6.7 The unpaired standard error, shown for contrast (READ)¶

In words. This is the standard error you would use if the two models had sat two different tests. Work out the wobble for each model separately, add the two wobbles, and take the square root. Using it on paired data throws away the fact that the questions were the same, and you pay for that in a wider interval.

The formula.

SE_{\text{unpaired}} = \sqrt{\frac{\hat{p}_A(1 - \hat{p}_A)}{n_A} + \frac{\hat{p}_B(1 - \hat{p}_B)}{n_B}}

The symbols.

Symbol	How to say it out loud	What it means
$SE_{\text{unpaired}}$	“ess ee sub unpaired”	the answer: the standard error of the difference, computed the wrong way for this data
$\hat{p}_A$	“p hat sub A”	the accuracy of model A
$\hat{p}_B$	“p hat sub B”	the accuracy of model B
$n_A$ , $n_B$	“en sub A”, “en sub B”	the two test sizes, both 20 here
$1 - \hat{p}_A$	“one minus p hat sub A”	the share model A got wrong
$+$	“plus”	add the two pieces together, inside the square root
the fraction bars	“divided by”	divide each top by its own bottom
$\sqrt{\phantom{x}}$	“the square root of”	applies to the whole sum, not to one piece of it

Out loud. “The unpaired standard error is the square root of model A’s accuracy times one minus it, divided by its number of questions, plus model B’s accuracy times one minus it, divided by its number of questions.”

Worked, step by step, with $\hat{p}_A = 0.70$ , $\hat{p}_B = 0.95$ , and $n = 20$ for both.

Step 1, model A’s piece. $1 - 0.70 = 0.30$ ; $0.70 \times 0.30 = 0.21$ ; $0.21 \div 20 = 0.010500$ .

Step 2, model B’s piece. $1 - 0.95 = 0.05$ ; $0.95 \times 0.05 = 0.0475$ ; $0.0475 \div 20 = 0.002375$ .

Step 3, add the two pieces. $0.010500 + 0.002375 = 0.012875$ .

Step 4, take the square root. $\sqrt{0.012875} = \mathbf{0.113468}$ .

Step 5, build the interval around the same difference of 0.25. $1.96 \times 0.113468 = 0.222397$ $0.25 - 0.222397 = 0.027603$ $0.25 + 0.222397 = 0.472397$

That is $\mathbf{+2.8}$ to $\mathbf{+47.2}$ percentage points.

method	$SE$	95% interval, percentage points	width
paired (6.6)	0.099340	+5.5 to +44.5	39.0
unpaired (6.7)	0.113468	+2.8 to +47.2	44.4

Divide one standard error by the other: $0.113468 \div 0.099340 = 1.142$ . The unpaired interval is 1.14 times wider on exactly the same data.

Sanity check. The unpaired standard error should come out larger than the paired one on the same data, so if yours comes out smaller, check that you added inside the square root rather than outside it. Say the size of the gain plainly: 1.14 times is a modest gain, not a dramatic one. Pairing pays most when two systems agree often and their disagreements are balanced, and here the disagreements all fall one way.

Chapter. Chapter 13

6.8 McNemar’s exact test (READ)¶

In words. Throw away every question the two models agreed on, because a question they both got right, or both got wrong, tells you nothing about which model is better. Look only at the questions where they disagreed. If the two models were equally good, each disagreement would fall either way like a coin toss. Work out how unlikely it would be for a run of coin tosses to lean as hard as yours did.

First build the table of counts.

Table 1:Two models, twenty shared questions. The diagonal from top left to bottom right holds the concordant pairs, where the models agreed; the other diagonal holds the discordant pairs.

	model B right	model B wrong
model A right	$a$	$c$
model A wrong	$b$	$d$

The formula.

p = 2 \sum_{k=0}^{\min(b,\,c)} \binom{b+c}{k} \left(\frac{1}{2}\right)^{b+c}, \qquad \text{capped at } 1

The symbols.

Symbol	How to say it out loud	What it means
$p$	“p”, or “the p-value”	the answer: how likely a lean this big would be if the two models were equally good. A small $p$ is evidence they are not.
$=$	“equals”	the two sides are the same number
2	“two”	multiply by 2 to make the test two-sided, because a lean in either direction would have been interesting
$\sum_{k=0}^{\min(b,c)}$	“the sum from k equals zero to the minimum of b and c”	add up what follows for $k = 0$ , then $k = 1$ , and so on, stopping at whichever of $b$ and $c$ is smaller
$a$	“ay”	how many questions both models answered correctly
$d$	“dee”	how many questions both models got wrong
$b$	“bee”	how many questions only model B got right
$c$	“see”	how many questions only model A got right
$b+c$	“b plus c”	the number of discordant pairs: the questions where the two models disagreed
$\min(b, c)$	“the minimum of b and c”	whichever of the two is smaller
$\binom{b+c}{k}$	“b plus c choose k”	how many different ways there are to pick $k$ items out of $b+c$ . For example $\binom{5}{2} = 10$ .
$\left(\frac{1}{2}\right)^{b+c}$	“a half, to the power b plus c”	the chance of one particular run of $b+c$ coin tosses. The $\tfrac{1}{2}$ is the null hypothesis: each disagreement is as likely to fall either way as a fair coin.
the raised $b+c$	“to the power of”	multiply $\tfrac{1}{2}$ by itself $b+c$ times
“capped at 1”	“capped at one”	if the arithmetic gives an answer above 1, report 1, because no probability exceeds 1

Out loud. “The p-value is twice the sum, over every outcome at least as lopsided as the one you saw, of the number of ways that outcome could happen multiplied by the chance of one particular run of coin tosses.”

Worked, step by step. The lab’s table, with Qwen2.5-1.5B as model A and Qwen2.5-3B as model B.

	3B right	3B wrong
1.5B right	$a = 14$	$c = 0$
1.5B wrong	$b = 5$	$d = 1$

Step 1, read off the two discordant counts. $b = 5$ and $c = 0$ . All 5 disagreements favour the larger model and none favours the smaller.

Step 2, add them to get the number of coin tosses. $b + c = 5 + 0 = 5$ .

Step 3, find where the sum stops. $\min(b, c) = \min(5, 0) = 0$ , so the sum has exactly one term, the one with $k = 0$ .

Step 4, work out that term’s “choose” part. $\binom{5}{0} = 1$ , because there is exactly one way to pick nothing out of five.

Step 5, work out the coin-toss part. That means a half multiplied by itself five times.

\left(\frac{1}{2}\right)^{5} = \frac{1}{2} \times \frac{1}{2} \times \frac{1}{2} \times \frac{1}{2} \times \frac{1}{2} = \frac{1}{32} = 0.03125

Step 6, multiply the two parts. $1 \times 0.03125 = 0.03125$ .

Step 7, double it, because the test is two-sided. $2 \times 0.03125 = \frac{2}{32} = \mathbf{0.0625}$

Five coin tosses all landing the same way. That is what a 25-point gap amounts to here.

Sanity check. The $p$ -value must land between 0 and 1, which is why the formula says to cap it. A quick check on Step 4: the “choose” numbers for 5 are $1, 5, 10, 10, 5, 1$ , and they add to 32, which is 2⁵. If your “choose” numbers do not add to $2^{b+c}$ , you computed one of them wrongly. The other check is on the four cells: $a + b + c + d$ must equal $n$ , so $14 + 0 + 5 + 1 = 20$ .

Chapter. Chapter 13

7. What a run costs¶

These three belong to the sustainability thread and are stated here for completeness. The full accounting, including everything these numbers leave out, is in the cost model appendix.

7.1 Energy per token (READ)¶

In words. Time the model while it is writing, and measure how much power the graphics card is drawing while it works. Multiply the power by the time to get the total energy used, then divide by how many words came out. The answer is the energy cost of one token.

The formula.

E_{\text{token}} = \frac{\bar{P} \times t}{n_{\text{tokens}}}

The symbols.

Symbol	How to say it out loud	What it means
$E_{\text{token}}$	“E sub token”	the answer, in joules per token. One joule is one watt drawn for one second.
$=$	“equals”	the two sides are the same number
$\bar{P}$	“P bar”	the mean power draw, in watts, over the window in which the model was generating
the bar over the $P$	“bar”	it marks an average. Power moves up and down while the model runs, so this is the average of many readings.
$\times$	“times”	multiply
$t$	“tee”	how many seconds the generation took
$n_{\text{tokens}}$	“en sub tokens”	how many tokens came out
the fraction bar	“divided by”	divide the total energy on top by the token count underneath

Out loud. “The energy per token is the average power in watts multiplied by the number of seconds, divided by the number of tokens produced.”

Worked, step by step. The 0.5B model generated 85 tokens in 3.0677 seconds while the graphics card drew a mean of 21.254 watts.

Step 1, multiply power by time to get total energy in joules. $21.254 \times 3.0677 = 65.2009$ joules

Step 2, divide by the number of tokens. $65.2009 \div 85 = \mathbf{0.7671}$ joules per token

Worked again, for the 3B model, which generated 88 tokens in 5.6539 seconds at a mean of 30.203 watts.

Step 1, $30.203 \times 5.6539 = 170.7647$ joules.

Step 2, $170.7647 \div 88 = \mathbf{1.9405}$ joules per token.

Sanity check. The answer must be positive, and it must fall if the model gets faster at the same power draw, because the same energy is then spread over more tokens. A quick reality check on the size: the card was drawing about 21 watts and producing about 28 tokens per second, and $21 \div 28$ is about 0.75, which is close to the 0.7671 computed above. If your answer is out by a factor of a thousand, you probably mixed up watts with milliwatts, or seconds with milliseconds.

Chapter. Chapter 1, Chapter 7, Chapter 14

7.2 Joules per token to watt-hours per 1,000 tokens (USE)¶

In words. Joules are a unit almost nobody has a feel for. Watt-hours are the unit on an electricity bill. This converts one into the other and scales it up to a thousand tokens, which is roughly a long answer.

The formula.

\text{Wh per 1{,}000 tokens} = \frac{E_{\text{token}} \times 1000}{3600}

The symbols.

Symbol	How to say it out loud	What it means
$\text{Wh per 1{,}000 tokens}$	“watt hours per thousand tokens”	the answer, in watt-hours
$=$	“equals”	the two sides are the same number
$E_{\text{token}}$	“E sub token”	the joules per token from 7.1
$\times$	“times”	multiply
1000	“one thousand”	scale up from one token to a thousand tokens
3600	“three thousand six hundred”	the number of seconds in an hour, which is what turns joules into watt-hours
the fraction bar	“divided by”	divide the top by the bottom

Out loud. “Watt-hours per thousand tokens is the joules per token multiplied by one thousand, divided by three thousand six hundred.”

Worked, step by step, for the 0.5B model.

Step 1, start from $E_{\text{token}} = 0.7671$ joules per token.

Step 2, multiply by 1,000. $0.7671 \times 1000 = 767.1$ joules per thousand tokens.

Step 3, divide by 3,600. $767.1 \div 3600 = \mathbf{0.2131}$ watt-hours per 1,000 tokens.

Worked again, for the 3B model.

Step 1, $E_{\text{token}} = 1.9405$ joules per token.

Step 2, $1.9405 \times 1000 = 1940.5$ .

Step 3, $1940.5 \div 3600 = \mathbf{0.5390}$ watt-hours per 1,000 tokens.

Step 4, compare the two. $1.9405 \div 0.7671 = 2.53$ . The 3B model costs 2.53 times the energy per token of the 0.5B.

Sanity check. Watt-hours per thousand tokens will always be a smaller number than joules per token, because 3,600 is much bigger than 1,000. If your answer came out larger, you divided and multiplied the wrong way round. For scale: a 60-watt light bulb uses 60 watt-hours in an hour, so 0.2131 watt-hours is that bulb running for about thirteen seconds.

Chapter. Chapter 7, Chapter 14

7.3 Marginal energy above idle (READ)¶

In words. A graphics card uses power even when nothing is running. Subtract that background draw before working out the cost of a token, and you get the extra energy that the model actually caused, rather than the cost of the machine being switched on.

The formula.

E_{\text{above idle}} = \frac{\big(\bar{P} - P_{\text{idle}}\big) \times t}{n_{\text{tokens}}}

The symbols.

Symbol	How to say it out loud	What it means
$E_{\text{above idle}}$	“E sub above idle”	the answer: the extra joules per token caused by running the model
$=$	“equals”	the two sides are the same number
$\bar{P}$	“P bar”	the mean power in watts while the model was generating
$P_{\text{idle}}$	“P sub idle”	the power the same hardware draws with nothing running. Measured at 13.834 watts on the machine used for this book.
$-$	“minus”	subtract
the round brackets	“bracket”	they say to do the subtraction first, before multiplying by $t$
$\times$	“times”	multiply
$t$	“tee”	how many seconds the generation took
$n_{\text{tokens}}$	“en sub tokens”	how many tokens came out
the fraction bar	“divided by”	divide the top by the bottom

Out loud. “The energy above idle is the average power minus the idle power, multiplied by the number of seconds, divided by the number of tokens.”

Worked, step by step, for the 0.5B model.

Step 1, subtract the idle power from the mean power. Do this first, because of the brackets. $21.254 - 13.834 = 7.420$ watts

Step 2, multiply by the time. $7.420 \times 3.0677 = 22.7623$ joules

Step 3, divide by the token count. $22.7623 \div 85 = \mathbf{0.2678}$ joules per token above idle

Step 4, compare with the full figure from 7.1. $0.2678 \div 0.7671 = 0.3491$

So about 35% of the energy is the model working, and the other 65%, roughly two thirds, is the graphics card being powered on at all.

Sanity check. This answer must always be smaller than the answer from 7.1, because you subtracted something positive before dividing. If it is not, check that you did the subtraction inside the brackets first. If the answer comes out negative, the mean power you measured was below idle, which means the measurement window was wrong: it probably included time before the model started working.

Chapter. Chapter 7

8. Use it or read it, in one table¶

§	Formula	Chapter	Tag
2.1	Share of parameters in a component	3	USE
2.2	Size of a stored matrix	3	USE
3.1	Softmax	4	USE
3.2	Ratio form, $p_i/p_j = e^{z_i - z_j}$	4	READ
3.3	Softmax with temperature	5	USE
3.4	Temperature as a power of the probabilities	5	READ
3.5	Entropy in bits	5	USE
3.6	Tokens holding 90% of the probability	5	USE
4.1	Quantization scale	7	USE
4.2	Quantize and dequantize	7	USE
4.3	Quantization error relative to the weight spread	7	READ
4.4	Blockwise quantization	7	USE
4.5	Bytes per weight, with block-scale overhead	6, 7	USE
4.6	Model size on disk	3, 6, 7	USE
5.1	Dot product	8	USE
5.2	Length of a vector	8	USE
5.3	Cosine similarity	9	USE
5.4	Angle from a cosine	9	READ
5.5	Retrieval by nearest cosine	10	USE
6.1	Sample proportion	11	USE
6.2	Standard error of a proportion	12	USE
6.3	Wald confidence interval	12	USE
6.4	Wilson confidence interval	12	READ
6.5	Bootstrap percentile interval	12	USE
6.6	Paired difference and its standard error	13	USE
6.7	Unpaired standard error, for contrast	13	READ
6.8	McNemar’s exact test	13	READ
7.1	Energy per token	1, 7, 14	READ
7.2	Watt-hours per 1,000 tokens	7, 14	USE
7.3	Marginal energy above idle	7	READ

Counting them: 30 formulas, 21 to use and 9 to read. If you can carry out the twenty-one and explain the nine, you can do every calculation this course asks for.

Where the numbers come from

Every measured number on this page was produced by a script in lab/ and written to a JSON file in lab/out/, and every worked instance above was recomputed and checked against that file by lab/appendix_formulas_checks.py, which writes lab/out/appendix_formulas_checks.json. Nothing here was recalled or estimated.

Numbers	Source
Logits, softmax by temperature, entropy, 90% mass	`we2_softmax.json`
Parameter counts, component shares, weight-matrix statistics, quantization scales	`we3_params_quant.json`
Per-tensor and block-32 quantization error by bit width	`we3b_quant.json`
Cosines, angles, retrieval scores	`we5_embeddings.json`
20-question run, Wald and bootstrap intervals	`we6_eval.json`
Paired comparison, McNemar	`we7_paired.json`
Power, joules per token, idle draw	`theme_s_energy.json`
Storage table, blockwise dequantized values, by-hand arithmetic	`_research/00-lab-verified-findings.md`

Some worked examples on this page use made-up inputs, and each one says so where it appears, because small numbers are easier to check by hand than real ones. They are: the three scores 2, 1, 0 used in 3.1 through 3.6; the three probabilities 0.5, 0.25, 0.25 in 3.5; the four weights in 4.3 and 4.4, of which only the last is a real measured weight; the two-number vectors in 5.1 through 5.5; and the five-question test in 6.5. Every other number is a measurement.

The critical value $z^\star = 1.96$ is used throughout, which is what the lab scripts used, so the intervals here reproduce the recorded ones to the last digit.

Each entry has six parts, always in this order¶

How to read a formula¶

1. Notation used on this page¶

2. Counting what is in the model¶

2.1 Share of the parameters held by one component (USE)¶

2.2 Size of a stored matrix (USE)¶

3. The probability of the next word¶

3.1 Softmax (USE)¶

3.2 The ratio form (READ)¶

3.3 Softmax with temperature (USE)¶

3.4 Temperature as a power, the equivalent form (READ)¶

3.5 Entropy in bits (USE)¶

3.6 How many tokens hold 90% of the probability (USE)¶

4. Making models small¶

4.1 Symmetric linear quantization: the scale (USE)¶

4.2 Quantize, then dequantize (USE)¶

4.3 Quantization error, measured against the spread of the weights (READ)¶

4.4 Blockwise quantization (USE)¶

4.5 Bytes per weight, including the block scales (USE)¶

4.6 Model size on disk (USE)¶

5. Meaning as geometry¶

5.1 Dot product (USE)¶

5.2 Length of a vector (USE)¶

5.3 Cosine similarity (USE)¶

5.4 Angle between two vectors (READ)¶

5.5 Retrieval: pick the nearest passage (USE)¶

6. Judging a model like a statistician¶

6.1 Sample proportion, which is what accuracy is (USE)¶

6.2 Standard error of a proportion (USE)¶

6.3 Wald confidence interval for a proportion (USE)¶

6.4 Wilson confidence interval for a proportion (READ)¶

6.5 Bootstrap percentile interval (USE)¶

6.6 Paired difference and its standard error (USE)¶

6.7 The unpaired standard error, shown for contrast (READ)¶

6.8 McNemar’s exact test (READ)¶

7. What a run costs¶

7.1 Energy per token (READ)¶

7.2 Joules per token to watt-hours per 1,000 tokens (USE)¶

7.3 Marginal energy above idle (READ)¶

8. Use it or read it, in one table¶