The Math Toolkit

What this page is for¶

This book uses mathematical notation. Notation is a shorthand. Someone invented each mark on this page to save writing, and every one of them can be said out loud in ordinary English.

This appendix says all of them out loud.

You can read it straight through in about an hour, or you can ignore it until a chapter sends you here. Every chapter in this book links to the exact section you need at the moment you need it. When Chapter 4 writes $\sum$ for the first time, it links to Section 10 of this page, and Section 10 starts from “add these up.”

Three things are true about this page and worth saying now.

It starts from nothing. It does not assume you remember high-school algebra. It does not assume you have seen a subscript before. If you last did algebra eleven years ago, this page is written for you and not for someone else.

Every worked example is arithmetic you can check. The numbers are small on purpose and the steps are all written down. A phone calculator is enough. If your answer differs from the one printed here, one of us made a slip, and finding out which one is a useful twenty seconds.

Nothing here is optional background. A student who can read $\hat{p} = x/n$ and say “p-hat equals x over n, which is the number right divided by the number asked” has already done the hard part of Chapter 11.

The index of anchors¶

Every section below has a permanent link name. Chapters link straight to them, and you can bookmark them. The middle column is what a chapter author types.

Four sections inside those twenty have their own anchor as well, because chapters link straight to them:

Two more tables sit at the end of the page: a one-page symbol summary of everything in the book, and twenty problems with full solutions.

How a formula is presented in this book¶

Every formula in this book, here and in all fifteen chapters, arrives in the same six parts, in the same order. Knowing the shape in advance makes each one easier to walk into.

1. In words. What the formula does, in a sentence with no symbols in it at all.

2. The formula, on its own line.

3. The symbols. A table with one row per symbol, including the operators, including the ones you already know. Three columns: the symbol, how to say it out loud, what it means.

4. Out loud. The whole formula read as one English sentence.

5. Worked. A numeric example with every step shown. Nothing skipped.

6. Check it. A test you can apply to your own answer, or a description of what a wrong answer looks like.

If you ever meet a formula in this book missing one of the six, that is a mistake in the book. Tell your instructor.

1. A letter standing for a number, and why we bother¶

What it is¶

In mathematics a letter can stand for a number. When you see

it means: the letter $p$ and the number 0.25 are the same thing here. Wherever $p$ appears in this discussion, you are allowed to cross it out and write 0.25 instead.

That is the whole idea. A letter is a container. It is not a puzzle and it is not hiding anything from you.

The equals sign¶

$=$ does not mean “now compute this.” It means “these two are the same.” So $3 + 4 = 7$ and $7 = 3 + 4$ say exactly the same thing, in the way that “Bakersfield is the largest city in Kern County” and “the largest city in Kern County is Bakersfield” say the same thing.

Why bother¶

Because a letter lets you say something true about every number at once.

Suppose tacos cost $2.50 each. You could write out a price list: one taco $2.50, two tacos $5.00, three tacos $7.50, and on forever. Or you could write one line that covers every case:

the cost is the number of tacos times the price of one taco.

To write that down compactly you need a name for “the number of tacos” and a name for “the price of one taco.” That is what the letters are for.

The same move is what lets this book talk about a language model at all. The model Qwen2.5-0.5B-Instruct contains 494,032,768 numbers (real, from lab/out/we3_params_quant.json). Nobody is going to write 494,032,768 separate sentences. So the book writes one sentence about “a weight, call it $w$,” and that sentence is true of all 494,032,768 of them.

Formula 1: the cost of several identical things¶

In words. The total cost is how many you bought, multiplied by the price of one.

The formula.

The symbols.

Out loud. “C equals n times p.” In full English: “the total cost equals the number of items times the price of one item.”

Worked, with numbers made up for practice. Four tacos at $2.50 each.

Step 1, write down what each letter is. $n = 4$ $p = 2.50$

Step 2, put those numbers into the formula in place of the letters. $C = 4 \times 2.50$

Step 3, do the multiplication. $C = 10.00$

Four tacos cost $10.00.

Check it. The answer has to be bigger than the price of one taco and smaller than the price of a hundred tacos. If you got $0.625 you divided instead of multiplying. If you got $6.50 you added instead of multiplying.

Letters used for a fixed number¶

Most letters stand for a number that could change. Two letters in this book stand for a number that never changes, no matter what:

These are the same kind of thing as the letter $n$ above, with one difference: they are already filled in. $e$ is a specific number in the same way that 7 is a specific number. It has a letter because writing $2.718281828459045\ldots$ every time would be unbearable.

What goes wrong¶

Assuming a letter means the same thing everywhere. It does not. $n$ means “number of tacos” above, and “number of questions on a test” in Chapter 11. A chapter always tells you what its letters mean before it uses them. If you cannot find the definition, that is a defect in the chapter, not a gap in you.

Reading $ab$ as a word. Two letters side by side means multiply them. See Section 3.

Trying to “solve for” the letter. Most of the time this book is not asking you to solve anything. It is handing you a recipe, and the letters are the ingredients.

2. Subscripts: $z_1$, $z_2$, $z_i$¶

What it is¶

A subscript is a small number or letter written low and to the right of another letter:

It is a house number. The letter $z$ names the street, and the subscript says which house on it.

The name for the small letter itself is index. In $z_i$, the letter $i$ is the index.

Why it exists¶

Because you often have a long list of numbers that are all the same kind of thing, and you need to talk about them one at a time without inventing a new letter for each.

Here is a real list. When you type The capital of France is into Qwen2.5-0.5B-Instruct, the model produces one score for every word it could say next. The eight highest scores were these (real, from lab/out/we2_softmax.json):

That list does not stop at eight. It runs to 151,936 entries, one for every token in the model’s vocabulary. Without subscripts you would need 151,936 different letters, and the Latin and Greek alphabets together give you about fifty.

The book calls these scores logits. Chapter 4 explains where they come from.

Worked example, with real numbers¶

Work out the gap between the model’s first and second choices.

Step 1, read the two values off the table. $z_1 = 17.2173$ $z_2 = 16.3196$

Step 2, subtract the second from the first. $z_1 - z_2 = 17.2173 - 16.3196$

Step 3, do the subtraction. $17.2173 - 16.3196 = 0.8977$

The gap is 0.8977. That one number decides how much more likely ' Paris' is than ' ______', and Chapter 4 turns it into the answer “2.4538 times as likely.”

Check it. $z_1$ is the biggest score in the list, so $z_1 - z_2$ has to come out positive. A negative answer means you subtracted the wrong way round.

A subscript is not an exponent¶

This is the most common mix-up on this page, so it gets its own table.

Low and small means “which one.” High and small means “multiply it by itself this many times.” Exponents get Section 5.

Two subscripts at once¶

Sometimes a number sits in a grid rather than a list, and then it needs two house numbers:

says “the number in row 3, column 5 of the grid called $W$.” Rows first, then columns, always.

This shows up in Chapter 3. One real grid inside Qwen2.5-0.5B-Instruct has 896 rows and 896 columns, so it holds 802,816 numbers, and $W_{3,5}$ is one of them (real, from lab/out/we3_params_quant.json).

What goes wrong¶

Reading $z_1$ as “z times 1.” It is not a multiplication. There is no operation happening at all. It is a name.

Losing track of where the counting starts. This book starts its lists at 1, so $z_1$ is the first item. Python, the programming language used in the code cells, starts at 0, so the first item is z[0]. That mismatch is a genuine nuisance, and every chapter that has both will point it out at the moment it matters.

3. Multiplication, written four ways¶

What it is¶

Multiplication has four different written forms in this book. They all mean the same thing.

Why four? History, mostly. The $\times$ sign is clear, but it looks like the letter x, which becomes a problem once letters start standing for numbers. So mathematics quietly dropped it and started writing $3 \cdot 4$, or $3(4)$, or nothing at all.

In the Python code in this book, multiplication is always written *, because a keyboard has no $\times$ key and a computer cannot guess what $ab$ is supposed to mean.

Worked example, with real numbers¶

The vocabulary table inside Qwen2.5-0.5B-Instruct is a grid with one row for each of its 151,936 tokens and 896 columns. How many numbers is that?

Step 1, write the multiplication. $151{,}936 \times 896$

Step 2, break it into pieces you can do one at a time. Notice that $896 = 900 - 4$. $151{,}936 \times 900 = 136{,}742{,}400$ $151{,}936 \times 4 = 607{,}744$

Step 3, subtract the second from the first. $136{,}742{,}400 - 607{,}744 = 136{,}134{,}656$

So the vocabulary table holds 136,134,656 numbers. The whole model holds 494,032,768 (real, from lab/out/we3_params_quant.json).

Check it. Round both numbers and multiply the round versions: $150{,}000 \times 900 = 135{,}000{,}000$. Your exact answer should land near that. It did. Rounding first is the fastest way to catch a slipped digit, and it is worth doing every time.

What goes wrong¶

Reading $3(4)$ as “three, then four.” It is 12.

Reading $ab$ as a two-letter name. It is $a$ times $b$. When a book wants a two-letter name it writes it in a different typeface, like $\text{rate}$, or spells the whole word out.

Confusing $\cdot$ with a decimal point. The multiplication dot sits at mid-height: $3 \cdot 4 = 12$. The decimal point sits on the line: 3.4, which is a number between 3 and 4. Where there is any risk of confusion, this book writes $\times$.

4. The fraction bar is division¶

What it is¶

A horizontal bar between two numbers means divide the top by the bottom.

Three ways of writing the same instruction:

The top of a fraction is the numerator. The bottom is the denominator. You do not have to memorise those two words, but statisticians say them constantly, so they are worth knowing.

The bar has invisible brackets¶

This matters more than anything else in this section.

The bar groups everything above it and everything below it. Finish the top, finish the bottom, then divide once, at the end.

That is not the same as $2 + 4/3$, which comes to $2 + 1.333 = 3.333$. The bar did work that you have to do with brackets when you type it on one line: (2 + 4) / 3.

Chapter 4 leans on this. The softmax formula has a whole long sum underneath the bar, and every term in that sum has to be added up before you divide even once.

Worked example, with real numbers¶

The vocabulary table holds 136,134,656 numbers. The whole model holds 494,032,768. What share of the model is the vocabulary table?

Step 1, write it as a fraction. Part on top, whole underneath.

Step 2, divide. On a calculator: type 136134656, press divide, type 494032768, press equals. $136{,}134{,}656 \div 494{,}032{,}768 = 0.275558$

Step 3, turn the decimal into a percentage by multiplying by 100. See Section 11. $0.275558 \times 100 = 27.5558$

So 27.56% of this model is a lookup table for words (real, from lab/out/we3_params_quant.json). All of the attention machinery, which is the part everyone writes about, is 8.92%.

Check it. A part divided by a whole has to land between 0 and 1. If you get a number bigger than 1, you divided the whole by the part. Here 0.2756 is a bit more than a quarter, and a quarter of 494 million is about 123 million, which is close to the 136 million on top. Right neighbourhood.

Dividing by zero¶

You cannot. $\frac{5}{0}$ is not a number, and no calculator will give you one; it returns an error. This matters in Chapter 9, where cosine similarity divides by the length of a vector, and a vector of all zeros has length zero. Chapter 9 says what to do about it.

What goes wrong¶

Dividing the wrong way round. “A out of B” is always $A/B$. Five questions right out of twenty is $5/20 = 0.25$, not $20/5 = 4$. A proportion above 1 is the warning sign.

Forgetting the invisible brackets. If your calculator has no fraction key, put brackets round the whole top and brackets round the whole bottom before you divide.

5. Exponents¶

What it is¶

A small number written high and to the right means “multiply the big number by itself this many times.”

The big number at the bottom is the base. The small number up top is the exponent, also called the power. “Two to the power of three” and “two to the three” are the same phrase.

Formula 2: a number raised to a whole-number power¶

In words. Write the base down that many times, with multiplication signs between them, and work it out.

The formula.

The symbols.

Out loud. “a to the n means a multiplied by itself, n times over.”

Worked, with numbers made up for practice. Compute 2¹⁰ the slow way, because the slow way is the one that shows you what is happening.

$2^1 = 2$ $2^2 = 2 \times 2 = 4$ $2^3 = 4 \times 2 = 8$ $2^4 = 8 \times 2 = 16$ $2^5 = 16 \times 2 = 32$ $2^6 = 32 \times 2 = 64$ $2^7 = 64 \times 2 = 128$ $2^8 = 128 \times 2 = 256$ $2^9 = 256 \times 2 = 512$ $2^{10} = 512 \times 2 = 1024$

Each line is the line above it, doubled. Ten doublings takes you from 2 to 1024.

Check it. Every step up the ladder doubles the value. If two of your lines are not a factor of two apart, the slip is at that line. On a calculator, look for a key marked x^y or y^x or ^: type 2, that key, 10, equals. You should see 1024.

Why $2^0 = 1$¶

This is the step most people want a reason for, so here is the reason.

Read the ladder above from the bottom up instead of the top down. Each step down divides by 2.

$2^3 = 8$ $2^2 = 4$, which is $8 \div 2$ $2^1 = 2$, which is $4 \div 2$ $2^0 = ?$, which must be $2 \div 2$

$2 \div 2 = 1$. So $2^0 = 1$.

The same argument works for any base. $10^0 = 1$. $7^0 = 1$. The rule holds for every base except zero itself, where the ladder has nothing to stand on and 0⁰ is left undefined.

$2^0 = 1$ is not a special exception someone bolted on. It is what the pattern already says. And it is load-bearing in Chapter 4, where $e^0 = 1$ shows up in the very first softmax you compute by hand.

What goes wrong¶

Reading 2³ as $2 \times 3$. It is 8, not 6. The exponent counts how many copies, it is not one of the things being multiplied.

Reading 2³ as 3². Those are different: $2^3 = 8$ and $3^2 = 9$. Base first, exponent second.

Expecting 2¹⁰ to be near 20. Exponents grow fast, and that speed is the reason Chapter 6 can store a model in far fewer bytes than you would guess.

6. Negative exponents¶

What it is¶

A minus sign in the exponent means “one divided by.”

Formula 3: a negative exponent¶

In words. A negative exponent means take the positive version and put it underneath a one.

The formula.

The symbols.

Out loud. “a to the minus n equals one over a to the n.”

Worked, with numbers made up for practice. Compute 2^-4.

Step 1, drop the minus sign and work out the positive version. $2^{4} = 2 \times 2 \times 2 \times 2 = 16$

Step 2, put that underneath a 1. $2^{-4} = \dfrac{1}{16}$

Step 3, do the division to get a decimal. $1 \div 16 = 0.0625$

Check it. A negative exponent on a base bigger than 1 always gives an answer between 0 and

1. If your answer is bigger than 1, the minus sign got lost. And $2^{-4} = 0.0625$ should be smaller than $2^{-3} = 0.125$, because you divided by a bigger number. It is.

Why it works, from the ladder¶

Keep walking down the ladder in Section 5, dividing by 2 each time.

$2^{2} = 4$ $2^{1} = 2$ $2^{0} = 1$ $2^{-1} = 1 \div 2 = 0.5$ $2^{-2} = 0.5 \div 2 = 0.25$ $2^{-3} = 0.25 \div 2 = 0.125$

Nothing new happened at zero. The pattern carried straight on through it.

Where this book uses it¶

Chapter 6 stores numbers inside a computer, and a computer stores a fraction as a sum of negative powers of two. The number 0.75, for instance, is $2^{-1} + 2^{-2}$, which is $0.5 + 0.25$. That is the reason Chapter 6 can tell you exactly which numbers a computer can hold and which ones it can only approximate.

Negative exponents also appear in scientific notation, where $1.69 \times 10^{-5}$ is a compact way of writing 0.0000169. See Section 17.

7. The number $e$, and what $e^x$ means¶

What it is¶

$e$ is a fixed number:

The dots mean the decimals carry on forever without repeating, the same way $\pi$’s do. For every calculation in this book, $e = 2.718282$ is close enough.

$e$ is not standing in for anything you are supposed to work out. It is a number, in the way that 7 is a number. It has a letter because nobody wants to write 2.718281828459045 repeatedly.

Where it is on a calculator¶

• Phone calculator, iPhone or Android: turn the phone sideways to get the scientific keypad. Look for a key marked e^x. On some phones it is the second function of the ln key, so you press 2nd or Shift first.

• Windows Calculator: switch to Scientific mode from the menu, then use the e^x key.

• A physical scientific calculator: e^x is nearly always printed above the ln key.

• Google: type e^2 into the search box.

• Python, which this book uses: import math then math.exp(2).

A quick test that you have found the right key: enter 1. You should get 2.718282. Enter 0. You should get exactly 1.

Formula 4: the exponential function¶

In words. Take the fixed number 2.718282 and raise it to whatever power you are given.

The formula.

The symbols.

Out loud. “y equals e to the x,” or in full, “y is the number 2.718282 raised to the power x.”

Worked, with numbers made up for practice. The three values that Chapter 4 uses in its first softmax, computed from the scores 2, 1 and 0.

Step 1, $e^{2}$. Press e^x, enter 2. $e^{2} = 7.389056$

Step 2, $e^{1}$. This is $e$ itself. $e^{1} = 2.718282$

Step 3, $e^{0}$. Any number to the power zero is 1, from Section 5. $e^{0} = 1.000000$

Step 4, add them, because softmax is going to need the total. $7.389056 + 2.718282 + 1.000000 = 11.107338$

Check it. Three sanity tests, in order of usefulness:

• $e^{x}$ is always positive, whatever $x$ is. A negative answer means a slip.

• $e^{0} = 1$ exactly, and $e^{1} = 2.718282$. If your calculator disagrees on either, you are on the wrong key.

• Bigger $x$ gives bigger $e^{x}$, always. If your list is not in the same order as your inputs, check the arithmetic.

What $e^x$ does to differences¶

There is one property worth meeting now, because Chapter 4 is built on it. Raising $e$ to a power turns differences into ratios.

Here it is with the real numbers from the lab. The top two logits were $z_1 = 17.2173$ for ' Paris' and $z_2 = 16.3196$ for ' ______' (real, lab/out/we2_softmax.json).

Step 1, take the difference. $17.2173 - 16.3196 = 0.8977$

Step 2, raise $e$ to that difference. $e^{0.8977} = 2.4538$

Step 3, compare that with the two probabilities the model actually assigned, which were $30.219\%$ and $12.315\%$ (real, same file). $0.30219 \div 0.12315 = 2.4538$

The two agree to four decimal places. A gap of 0.8977 in the scores is exactly a factor of 2.4538 in the probabilities. Chapter 4 explains why that has to happen.

What goes wrong¶

Reading $e^2$ as $e \times 2$. That would be 5.437. It is 7.389056.

Using the EXP or EE key by mistake. On many calculators EXP or EE means “times ten to the power of”, for scientific notation, which is Section 17. It is a different key from e^x and it will give you a wildly wrong answer. Test with $e^1 = 2.718282$ before you trust a key.

8. Logarithms: the reverse of an exponent¶

What it is¶

A logarithm answers one question: what exponent was used?

Exponents go one way:

“2 raised to the power 3 is 8.”

Logarithms go back the other way:

“To get 8 from 2, the exponent you need is 3.”

Written down, that second sentence is

Formula 5: the definition of a logarithm¶

In words. The log base two of a number is the power you have to raise two to in order to get that number.

The formula.

The symbols.

Out loud. “Log base two of x equals y” says the same thing as “two raised to the power y equals x.” They are one fact written in two directions.

Worked, with numbers made up for practice. Compute $\log_2(1024)$.

Step 1, ask the question in plain words. Two to what power gives 1024?

Step 2, climb the doubling ladder from Section 5 and count the steps. $2^1 = 2$ $2^2 = 4$ $2^3 = 8$ $2^4 = 16$ $2^5 = 32$ $2^6 = 64$ $2^7 = 128$ $2^8 = 256$ $2^9 = 512$ $2^{10} = 1024$

Step 3, read off the exponent that landed on 1024. It is 10. $\log_2(1024) = 10$

Check it. Put the answer back through the exponent: 2¹⁰ should equal your starting number. $2^{10} = 1024$. It does. That reverse check works on every logarithm and it is the fastest way to catch a mistake.

Doing it on a calculator that has no $\log_2$ key¶

Most calculators have log (base 10) and ln (base $e$) but no base-2 key. Either one gets you there:

Worked, with numbers made up for practice. $\log_2(1024)$ again, this time with the base-10 key.

Step 1, take the log of the top number. $\log(1024) = 3.010300$

Step 2, take the log of 2. $\log(2) = 0.301030$

Step 3, divide. $3.010300 \div 0.301030 = 10.000000$

Same answer, 10. In Python this book writes math.log2(1024), which does it in one step.

What logarithms do to size¶

A logarithm shrinks big numbers down to manageable ones.

That last row is why a logarithm belongs in this book. Chapter 5 measures how uncertain the model is, and the natural unit for uncertainty is bits, which is a base-2 logarithm. A number as big as 151,936 turns into a number as small as 17.213, which you can hold in your head and compare against another.

Three rules that catch mistakes¶

That third row surprises people, so here is why. $\log_2(0.5)$ asks “two to what power gives 0.5?” From Section 6, $2^{-1} = 0.5$. The exponent is -1. All probabilities are between 0 and 1, so every logarithm of a probability comes out negative. Chapter 5 puts a minus sign in front of the entropy formula for exactly that reason, and Section 10.4 shows the arithmetic.

What goes wrong¶

Pressing log and getting a different answer. The plain log key is base 10 on nearly every calculator, and ln is base $e$. If you want base 2, either use the division trick above or use a log2 function. Test yourself: $\log_2(8)$ must be 3. If your key gives 0.903, that is base 10, and 0.903 is $\log_{10}(8)$.

Expecting a logarithm of a negative number. There is no exponent you can put on 2 that produces a negative answer, so there is no logarithm of a negative number. Your calculator will say so.

9. Square roots, and why standard deviation has one¶

What it is¶

A square root undoes a squaring.

The long bar over the top of the number is part of the symbol. Everything under the bar goes inside the root, exactly like the invisible brackets on a fraction bar in Section 4. So $\sqrt{9 + 16} = \sqrt{25} = 5$, which is not the same as $\sqrt{9} + \sqrt{16} = 3 + 4 = 7$.

Some square roots are tidy and most are not¶

On a calculator the key is marked $\sqrt{\ }$ and usually sits near the division key. On a phone, turn it sideways for the scientific keypad.

Why standard deviation has a square root in it¶

Here is the reason, in plain terms, before any formula.

A standard deviation is one number that says how spread out a list of numbers is. To build it you measure how far each value sits from the average. Some of those distances are positive and some are negative, and if you added them up as they stand they would cancel out and always give zero, which tells you nothing.

The fix is to square every distance first. Squaring turns every negative into a positive, so nothing cancels.

But squaring has a side effect. If your numbers were measured in joules, squaring them gives you joules squared, which is not a unit anybody wants. The average of those squares is a number in the wrong units and on the wrong scale.

So at the very end you take the square root, which undoes the squaring and puts the answer back into the original units. A standard deviation of 0.066741 is in the same units as the weights it describes, and you can compare it against them directly.

Square first so nothing cancels, square root last so the units come back. That is the whole story, and Section 10.3 does the arithmetic.

Formula 6: the standard error of a proportion¶

Here is a formula from Chapter 12 that uses a square root and needs nothing else on this page yet.

In words. The standard error of an accuracy score says how much that score would wobble if you had asked a different set of questions of the same size.

The formula.

The symbols.

Out loud. “The standard error is the square root of p-hat times one minus p-hat, all divided by n.”

Worked, with real numbers. In the lab, Qwen2.5-0.5B-Instruct answered 5 of 20 questions correctly on the course’s statistics bank, so $\hat{p} = 0.25$ and $n = 20$ (real, from lab/out/we6_eval.json).

Step 1, work out $1 - \hat{p}$. $1 - 0.25 = 0.75$

Step 2, multiply the two. $0.25 \times 0.75 = 0.1875$

Step 3, divide by $n$. $0.1875 \div 20 = 0.009375$

Step 4, take the square root. $\sqrt{0.009375} = 0.0968$

The standard error is 0.0968, which is 9.68 percentage points. A score of 25% that could easily have been 15% or 35% is not a score you should report on its own, and Chapter 12 is about what to report instead.

Check it. The standard error must be a positive number smaller than the accuracy scale itself. Two more tests: it gets smaller as $n$ gets bigger, and it is largest when $\hat{p} = 0.5$. If you take the square root and get something bigger than the number you started with, check whether that number was less than 1. The square root of a number between 0 and 1 is always bigger than the number, which is correct and often surprising: $\sqrt{0.009375} = 0.0968$, and 0.0968 is bigger than 0.009375.

What goes wrong¶

Splitting a root across a plus sign. $\sqrt{9 + 16} = 5$, and $\sqrt{9} + \sqrt{16} = 7$. Finish everything under the bar before you press the root key.

Taking the root of a negative number. There is no ordinary number that multiplies by itself to give a negative, so your calculator will report an error. If you are computing a standard error and land under a root sign with a negative number, the mistake happened earlier.

10. Sigma notation: “add these up”¶

This is the symbol that stops more readers than any other in the book, so it gets built in four steps, from an English sentence to the real formula.

Step 1: the English sentence¶

You have five numbers and you want their total. In English:

add them all up.

Step 2: write them out¶

That works. It stops working when there are 151,936 numbers.

Step 3: give the list a name and use subscripts¶

Call the list $x$, and number the entries with subscripts from Section 2:

Now the total is

Better, because it no longer mentions the actual values. But it still writes out every term.

Step 4: the sigma¶

That is the same total, written once. Read it as: “add up $x_i$, starting at $i = 1$ and stopping at $i = 5$.”

The anatomy of the symbol¶

The symbol has four parts and they are always in the same places.

To expand a sigma back into a plain sum, write out the thing on the right once for each value the counter takes, and put plus signs between them.

Nothing more is happening. Every sigma in this book can be unfolded that way, and when a sigma confuses you, unfolding it is the move.

Formula 7: the general sum¶

In words. Add up every number in a list, from the first to the last.

The formula.

The symbols.

Out loud. “The sum, from i equals one to n, of x sub i.” In full English: “add up all the numbers in the list $x$, from the first to the last.”

Worked, with numbers made up for practice. Take the five numbers 3, 1, 4, 1, 5, so $n = 5$.

Step 1, unfold the sigma. $\sum_{i=1}^{5} x_i = x_1 + x_2 + x_3 + x_4 + x_5$

Step 2, put in the values. $= 3 + 1 + 4 + 1 + 5$

Step 3, add them one at a time, left to right. $3 + 1 = 4$ $4 + 4 = 8$ $8 + 1 = 9$ $9 + 5 = 14$

Check it. The total has to be at least as big as the largest single number, which is 5, and no bigger than 5 times the largest, which is 25. Fourteen sits inside that range. If your total is smaller than the biggest number in your list, you dropped a term.

Formula 8: the mean¶

In words. The average of a list of numbers is their total, shared out equally among them.

The formula.

The symbols.

Out loud. “x-bar equals one over n, times the sum of all the x values.” In plain English: “the average is the total divided by how many there are.”

Worked, with numbers made up for practice. The same five numbers, 3, 1, 4, 1, 5.

Step 1, find the total. $3 + 1 + 4 + 1 + 5 = 14$

Step 2, count how many numbers there are. $n = 5$

Step 3, divide. $14 \div 5 = 2.8$

Check it. An average always lands between the smallest and largest values in the list. The smallest here is 1 and the largest is 5, and 2.8 sits between them. An average outside that range means either the total or the count is wrong.

Formula 9: the standard deviation¶

This is the formula that uses everything so far: subscripts, sigma, a fraction bar and a square root. It is the longest formula on this page, and you can walk through it one piece at a time.

In words. The standard deviation is a typical distance from the average. Measure how far each value is from the average, square those distances so the negatives do not cancel, average the squares, then take a square root to undo the squaring.

The formula.

The symbols.

Out loud. “s is the square root of, one over n minus one, times the sum of each value minus the average, all squared.”

Worked, with numbers made up for practice. The same five numbers: 3, 1, 4, 1, 5.

Step 1, find the average. From Formula 8, $\bar{x} = 2.8$.

Step 2, subtract the average from each number. $3 - 2.8 = 0.2$ $1 - 2.8 = -1.8$ $4 - 2.8 = 1.2$ $1 - 2.8 = -1.8$ $5 - 2.8 = 2.2$

Step 3, square each of those. Squaring makes every one positive. $0.2 \times 0.2 = 0.04$ $-1.8 \times -1.8 = 3.24$ $1.2 \times 1.2 = 1.44$ $-1.8 \times -1.8 = 3.24$ $2.2 \times 2.2 = 4.84$

Step 4, add the squares. This is the sigma. $0.04 + 3.24 + 1.44 + 3.24 + 4.84 = 12.80$

Step 5, divide by $n - 1$, which is $5 - 1 = 4$. $12.80 \div 4 = 3.20$

Step 6, take the square root. $\sqrt{3.20} = 1.7889$

Check it. Four tests, any one of which catches most mistakes:

• The five differences in step 2 must add to zero: $0.2 - 1.8 + 1.2 - 1.8 + 2.2 = 0$. They do. If they do not, the average is wrong.

• Every squared value in step 3 must be positive. A negative one means a sign error.

• The standard deviation should be roughly the size of a typical distance from the average. The distances here were 0.2, 1.8, 1.2, 1.8, 2.2, and 1.7889 sits comfortably among them.

• The standard deviation can never be bigger than the full range of the data, which is $5 - 1 = 4$ here.

A real one for scale. One weight matrix inside Qwen2.5-0.5B-Instruct holds 802,816 numbers with an average of -0.000017 and a standard deviation of 0.066741 (real, from lab/out/we3_params_quant.json). The same six steps were carried out, 802,816 times over, by a computer.

Formula 10: entropy¶

This one is here because Chapter 5 needs it, and because it is the place where sigma, logarithms and negative numbers all meet.

In words. Entropy is one number saying how spread out a set of probabilities is. It is small when the model is committed to one answer and large when the model is spreading its bets.

The formula.

The symbols.

Out loud. “H equals minus the sum, over every token, of that token’s probability times the log base two of that probability.”

Worked, with numbers made up for practice. Four possible words, each equally likely, so each has probability 0.25. Here $V = 4$.

Step 1, work out $\log_2(0.25)$. Ask: two to what power gives 0.25? From Section 6, $2^{-2} = 0.25$, so the answer is -2. $\log_2(0.25) = -2$

Step 2, multiply each probability by its own log. $0.25 \times (-2) = -0.5$

Step 3, add that up four times, once per token. $-0.5 + (-0.5) + (-0.5) + (-0.5) = -2.0$

Step 4, apply the minus sign on the front of the formula. $H = -(-2.0) = 2.0$

The entropy is 2.0 bits. That is the sensible answer: with four equally likely choices you need exactly two yes-or-no questions to pin down which one it is, and two yes-or-no questions is two bits. Section 18 unpacks that link.

A real one. With the temperature set to 1.0, the entropy of the real distribution over all 151,936 tokens was 4.450 bits, and 25 tokens together held 90% of the probability. Turn the temperature down to 0.25 and the entropy drops to 0.237 bits, with a single token holding 90% of the probability (real, from _research/00-lab-verified-findings.md, produced by lab/we2_softmax.py). Low entropy is a committed model. High entropy is a hesitant one.

Check it. Three tests:

• Entropy is never negative. If yours is, you dropped the minus sign on the front.

• Entropy is zero when one probability is 1 and the rest are 0, because the model has no uncertainty left at all.

• Entropy is at its largest when everything is equally likely, and that largest value is $\log_2(V)$. For $V = 4$ that is 2 bits, which is exactly what the worked example gave. For $V = 151{,}936$ it would be 17.213 bits, so any entropy you compute for our model has to land between 0 and 17.213.

11. Percentages, decimals and fractions¶

This section is the longest on the page, because this is what trips up the most people, and because nearly every number in this book is a percentage.

One number, three costumes¶

These three are the same number:

They are one quantity written three ways, the way a single distance can be written 1 mile or 1.609 kilometres. Nothing changes except the costume.

What “per cent” means¶

“Per cent” is Latin for “per hundred.” The symbol $\%$ means “out of a hundred.”

So the $\%$ sign is a compressed instruction: it says “there is a division by 100 hiding here.” Once you read $\%$ as “divided by a hundred,” every conversion on this page becomes one step.

Formula 11: decimal to percentage, and back¶

In words. To turn a decimal into a percentage, multiply by a hundred. To go back the other way, divide by a hundred.

The formula.

The symbols.

Out loud. “A percentage is the decimal times one hundred.” And: “a decimal is the percentage divided by one hundred.”

Worked, with real numbers. Both directions, using the measured accuracy of Qwen2.5-0.5B-Instruct on the course’s 20-question bank.

Direction one, decimal to percentage. The lab recorded 0.25 (real, from lab/out/we6_eval.json). Step 1, multiply by 100. $0.25 \times 100 = 25$ Step 2, write the symbol on the end. $25\%$

Direction two, percentage to decimal. The lab also recorded the rotation-debiased score as $15.0\%$ (real, from lab/out/we6b_eval_debiased.json). Step 1, divide by 100. $15 \div 100 = 0.15$ Step 2, that decimal is what you use in any further arithmetic. 0.15

Check it. Multiplying by 100 makes a number bigger, so the percentage is always the bigger of the two. 0.25 becomes 25, not 0.0025. If your percentage came out smaller than your decimal, you divided when you should have multiplied.

The shortcut, and why it works¶

Multiplying by 100 moves the decimal point two places to the right. Dividing by 100 moves it two places to the left.

Going the other way:

This is not a separate rule to memorise. Our number system is built on tens, so multiplying by $100 = 10 \times 10$ shifts every digit two columns. The shortcut and the multiplication are the same act.

Fraction to decimal: divide the top by the bottom¶

That is all there is to it, and it is why Section 4 comes before this one. On a calculator: 5, divide, 20, equals.

The last three rows are real scores from the lab. Three models sat the same twenty questions under the same scoring procedure, the rotation-debiased one from Chapter 13, and scored 3 of 20, 14 of 20 and 19 of 20 in order of size (real, from lab/out/lab4_size_ladder.json).

The phrase “under the same scoring procedure” is doing real work in that sentence. Change the procedure and the numbers change: scored the naive way, the smallest model gets 5 of 20 rather than 3 (real, from lab/out/we6_eval.json). Chapter 13 is about that, and this book never prints an accuracy without naming the procedure that produced it.

Formula 12: a percentage of something¶

In words. To find a percentage of a quantity, turn the percentage into a decimal, then multiply.

The formula.

The symbols.

Out loud. “The part equals the percentage divided by a hundred, times the whole.”

Worked, with real numbers. The vocabulary table is 27.56% of Qwen2.5-0.5B-Instruct, which has 494,032,768 parameters. How many parameters is that?

Step 1, turn the percentage into a decimal. $27.56 \div 100 = 0.2756$

Step 2, multiply by the whole. $0.2756 \times 494{,}032{,}768 = 136{,}155{,}431$

Step 3, compare with the exact count, which Section 3 worked out as $151{,}936 \times 896 = 136{,}134{,}656$.

The two differ by about 20,000, which is a rounding effect: 27.56% is the rounded version of 27.5558%. Section 16 is about exactly this, and Chapter 3 always quotes the exact count rather than reconstructing it from a rounded share.

Check it. A percentage below 100 must give you a part smaller than the whole. Rough check: 27.56% is a bit over a quarter, and a quarter of 494 million is 123.5 million, so an answer near 136 million is right. An answer near 13.6 billion means the decimal point moved the wrong way.

Formula 13: percentage change¶

In words. Percentage change says how much bigger or smaller a new number is than an old one, as a share of the old one.

The formula.

The symbols.

Out loud. “Percentage change equals new minus old, all divided by old, times one hundred.”

Worked, with real numbers. In the lab, Qwen2.5-0.5B-Instruct used 0.767 joules of GPU energy per token it produced, and Qwen2.5-3B-Instruct used 1.941 joules per token (real, from lab/out/theme_s_energy.json). By what percentage did the energy go up?

Step 1, subtract. The old value is the small model, because that is what we are measuring the change from. $1.941 - 0.767 = 1.174$

Step 2, divide by the old value. $1.174 \div 0.767 = 1.5306$

Step 3, multiply by 100. $1.5306 \times 100 = 153.06$

The 3B model costs about 153% more energy per token. Said the other way, it costs $1.941 \div 0.767 = 2.53$ times as much, which the lab notebook reports as $2.53\times$.

Check it. “153% more” and “2.53 times as much” must agree, and they do: a 153% increase means you kept the original 100% and added 153% on top, which is 253% of the original, which is 2.53 times. If your percentage increase and your multiplier do not line up that way, one of them is wrong.

One more check, and it is the honest one. The lab stored those two energy figures to full precision, as 0.7670694643464218 and 1.940526247555462. Run the same three steps on the full numbers and you get 152.98%, not 153.06%. The gap of 0.08 percentage points came from rounding the inputs to three decimal places before dividing. That is why this book says “about 153%” rather than quoting five digits, and it is the whole argument of Section 16, visible in one calculation.

Percentage points, which is not the same as percent¶

This distinction matters in Chapters 12, 13 and 14, and getting it wrong changes the meaning of a result.

Two real numbers from the lab, both under the rotation-debiased procedure of Chapter 13: Qwen2.5-1.5B-Instruct scored 70.0% on the twenty-question bank and Qwen2.5-3B-Instruct scored 95.0% on the same twenty questions (real, from lab/out/we7_paired.json).

The difference in percentage points. Subtract one from the other. $95.0 - 70.0 = 25.0$ The gap is 25.0 percentage points. This is what the book reports.

The difference in percent. Divide, using Formula 13. $(95.0 - 70.0) \div 70.0 = 0.3571$ $0.3571 \times 100 = 35.71$ The larger model scored 35.7% higher than the smaller one, in relative terms.

Both statements describe the same pair of numbers. They are different by a factor of 1.43, and a reader who is told “25%” when the truth is “25 percentage points” has been given the wrong number.

The rule. When you subtract two percentages, the answer is in percentage points. When you divide them, the answer is in percent. Say which one you mean, every time.

The trap of going down and then back up¶

A 50% cut followed by a 50% rise does not return you to where you started.

Step 1, start at 100 and cut it by 50%. $100 \times 0.50 = 50$

Step 2, now raise that by 50%. The base has changed; it is 50 now, not 100. $50 \times 1.50 = 75$

You end at 75, not 100. The percentages were taken on different bases, so they do not cancel. This shows up whenever someone claims a model “lost 30% then recovered 30%.”

Common mistakes with percentages¶

1. Confusing 0.25 with 0.25%. 0.25 is $25\%$. $0.25\%$ is 0.0025, which is a hundred times smaller. When a chapter prints a decimal, it has no $\%$ sign on it.

2. Adding percentages that have different bases. 50% of the students in one class plus 50% of the students in another class is not 100% of anything.

3. Reporting a percentage without saying what of. “Accuracy went up 10%” leaves out the base. Every number in running prose in this book carries a unit or a referent for exactly this reason.

4. Forgetting that a percentage can exceed 100. If the new value is more than double the old, the percentage change is above 100%. That is a real answer, not a mistake.

5. Rounding a percentage before using it in the next calculation. Carry the full decimal through, round at the end. Section 16.

12. Proportions, and what “out of” means¶

What it is¶

A proportion is a part divided by a whole. It answers “what share of the total is this?”

The phrase “5 out of 20” always means the division $5 \div 20$. The word “out of” is the fraction bar spoken aloud.

A proportion is always a number between 0 and 1. Zero means none of them. One means all of them.

Formula 14: a sample proportion¶

In words. Accuracy is the number of questions the model got right, divided by the number of questions you asked.

The formula.

The symbols.

Out loud. “p-hat equals x over n,” or in full English, “the measured accuracy is the number right divided by the number asked.”

Worked, with real numbers. Three models sat the same twenty-question bank in the lab, scored the same way each time, with the rotation-debiased procedure of Chapter 13. All three counts are real, from lab/out/lab4_size_ladder.json.

Qwen2.5-0.5B-Instruct got 3 right. $\hat{p} = 3 \div 20 = 0.15$, which is 15.0%

Qwen2.5-1.5B-Instruct got 14 right. $\hat{p} = 14 \div 20 = 0.70$, which is 70.0%

Qwen2.5-3B-Instruct got 19 right. $\hat{p} = 19 \div 20 = 0.95$, which is 95.0%

Check it. Every one of those is between 0 and 1. A proportion above 1 means the count and the total swapped places. A negative proportion is not possible at all.

“Out of” and the base¶

“Out of” names the whole, and changing the whole changes the meaning even when the top number stays the same.

• 5 out of 20 is 25%.

• 5 out of 100 is 5%.

• 5 out of 5 is 100%.

The number 5 did not move. The base did. Whenever this book reports a proportion it says what the base was, and when a chapter says “n = 20” that is the base being named.

Scaling a proportion to a different base¶

Because a proportion is a share, you can rewrite it over any base you like.

Step 1, work out what turns 20 into 100. $100 \div 20 = 5$. Step 2, multiply the top by the same amount. $5 \times 5 = 25$. Step 3, so $\frac{5}{20}$ and $\frac{25}{100}$ are the same share, and the second is a percentage by definition.

Multiplying top and bottom by the same number never changes a fraction’s value, because you are multiplying by $\frac{5}{5}$, which is 1.

“One in four” and other spoken forms¶

English has several ways to say a proportion, and they mean the same thing.