Answers

What is on this page, and what is not¶

Every number on this page was computed. Nothing was recalled, estimated, or rounded into looking cleaner than it is. Where a solution reports a lab measurement, the solution names the JSON file in lab/out/ that holds it, so you can open the file and check. Where a solution is pure arithmetic, every step is printed, so you can redo it on paper and get the same digits. That is the same policy the rest of the book runs on: we never publish a number we did not compute.

How to use these answers¶

Four things. The second one is the one that matters.

1. Try the problem first, all the way to a number. Reading a solution before you have attempted the problem feels like learning and is not. You will recognise every step and be unable to produce any of them a week later. Check after, not before.

2. An answer you can reproduce is worth more than one you recognise. The test is not whether the solution below makes sense while you read it. The test is whether you can close this page, take a blank sheet, and get the same digits. If you cannot, you have not finished the problem; you have finished reading about it.

3. A mismatch is information. When your number differs from the one here, do not scan for the line where the text disagrees with you. Redo the problem slowly from the start. In this course the three most common slips are dividing by $T$ after exponentiating instead of before, forgetting that a length is a square root, and rounding at an intermediate step.

4. Keep full precision until the last line. Round once, at the end, to the precision the chapter asks for. Rounding $e^{2}$ to 7.4 partway through a softmax will move your final percentage by more than you expect.

Numbering convention¶

Practice problems are numbered from 1 within each chapter, in the order they appear. “Odd numbered” means problems 1, 3, 5, 7, and so on, of that chapter’s set. A reference such as “Ch. 12, #7” means the seventh practice problem in Chapter 12, not a book-wide running count.

Some solutions below are not yet keyed to problem numbers. Those are labelled in bold by what they compute, and the numbers are attached when the chapter’s problem set is fixed.

Status of this appendix¶

All fifteen chapters now carry a finished practice set. This appendix does not yet carry a finished answer set, and the table below says exactly where the gap is, chapter by chapter, so you never have to guess whether an answer is missing or not yet written.

The count, as of this build. The fifteen chapters hold 404 practice problems between them, of which 203 are odd-numbered and therefore belong on this page. This page currently holds 10 worked solutions, none of them keyed to a problem number. So 15 of 15 chapters have odd-numbered problems whose answers are still missing.

“Standing computation” means the calculation the course is built on, the one that appears in the chapter’s worked example, again in its practice set, and again on the concept check. Those were written here first because they are the ones worth being able to reproduce. Keying them to the problem numbers they answer, and writing the remaining odd-numbered solutions, is the work left on this page.

Chapter 1. Where is this running, and what did it cost?¶

These problems drill the two measuring sticks of the course: turning a parameter count into bytes and gigabytes, turning tokens and seconds and watts into rates and joules, and scoring an output by counting claims instead of reacting to prose.

Warm-up: can you do the arithmetic?¶

1. Asked: how many bytes does a 1,543,714,304-parameter model take when each parameter is stored at FP16?

This is Formula 1.2, size in bytes equals $N \times (\text{bytes per weight})$, where $N$ is the parameter count. Multiplication is written four ways and they all mean the same thing; see Toolkit 3.

Step 1, write down the parameter count.

$N = 1{,}543{,}714{,}304$

Step 2, work out the bytes per weight. FP16 means 16 bits for each parameter. There are 8 bits in one byte, so divide.

$16 \div 8 = 2$ bytes per parameter

Step 3, multiply. Splitting the number into two easy pieces keeps it on a phone calculator.

$1{,}543{,}714{,}304 = 1{,}500{,}000{,}000 + 43{,}714{,}304$

$1{,}500{,}000{,}000 \times 2 = 3{,}000{,}000{,}000$

$43{,}714{,}304 \times 2 = 87{,}428{,}608$

$3{,}000{,}000{,}000 + 87{,}428{,}608 = \mathbf{3{,}087{,}428{,}608}$ bytes

Answer: 3,087,428,608 bytes.

Check it. This is the same number the chapter gets for the 1.5B model in Worked Example 1.2, and it is size_fp16_gb in lab/out/lab4_size_ladder.json, recorded there as 3.087428608 gigabytes.

Where this goes wrong. The tempting answer is 1,543,714,304 bytes, which is about 1.54 GB. It is tempting because “1.5B parameters” and “1.5 GB” share a number, and because it feels natural that one number takes one slot. One parameter is not one byte. At FP16 it is two. The other common slip is dividing by 2 instead of multiplying, which gives 771,857,152 and a file smaller than the parameter count, which cannot happen at any format this course uses.

3. Asked: add four file sizes, one line at a time. Made up for practice.

This is Formula 1.1, $B_{\text{total}} = \sum_{i=1}^{k} b_i$, where $b_i$ is the size of file number $i$ and $k$ is how many files there are. Here $k = 4$. The symbol $\sum$ means “add up everything that comes after me”; see Toolkit 10.

Add them left to right, smallest first, so every line can be checked on its own.

$512 + 2{,}048 = 2{,}560$

$2{,}560 + 65{,}536 = 68{,}096$

$68{,}096 + 990{,}000{,}000 = \mathbf{990{,}068{,}096}$

Answer: 990,068,096 bytes.

Check it. The chapter’s sanity check for a total: it must be bigger than the largest single file and smaller than the number of files multiplied by the largest file. The largest file is 990,000,000 bytes and there are 4 files, so the total must land between 990,000,000 and $4 \times 990{,}000{,}000 = 3{,}960{,}000{,}000$. Our total sits inside that range.

Where this goes wrong. The tempting answer is 990,000,000, from deciding that three files of 512, 2,048 and 65,536 bytes are too small to matter next to a file of 990 million. They are small, and that is the honest reading of the result, but dropping them is a different act from measuring them and then reporting that they are small. In this course you add them and then say they came to 68,096 bytes out of 990,068,096. The second common slip is a column misalignment: 65,536 written one column across becomes 655,360, and the total comes out as 990,657,920. Nothing in the sanity check catches that, which is why adding one line at a time is worth the extra writing.

5. Asked: a model produces 64 tokens in 4.0 seconds. What is its rate? Made up for practice.

This is Formula 1.7, $r = n_{\text{tokens}} \div t$, where $n_{\text{tokens}}$ is how many tokens came out and $t$ is how many seconds it took.

Step 1, write down the two numbers.

$n_{\text{tokens}} = 64$ and $t = 4.0$ seconds

Step 2, divide.

$64 \div 4.0 = \mathbf{16.0}$

Answer: 16.0 tokens per second.

Check it. Multiply back. $16.0 \times 4.0 = 64$, which is the token count you started with, so the division is right.

Where this goes wrong. The tempting answer is $4.0 \div 64 = 0.0625$. That number is real and it has a name: seconds per token. It is not wrong arithmetic, it is a different quantity. The sign that you flipped the fraction is that you expected a number in the tens and got a number near zero. Tokens go on top when you want a rate in tokens per second.

7. Asked: a card draws a mean of 25.0 watts for 8.0 seconds and produces 100 tokens. What did one token cost in energy? Made up for practice.

This is Formula 1.8, $E_{\text{token}} = (\bar{P} \times t) \div n_{\text{tokens}}$. Here $\bar{P}$, said “P bar”, is the mean power in watts over the window the model was generating, $t$ is the seconds, and $n_{\text{tokens}}$ is the token count. A watt is one joule every second, so watts multiplied by seconds leaves joules.

Step 1, multiply power by time to get the total energy.

$25.0 \times 8.0 = 200.0$ joules

Step 2, divide by the token count.

$200.0 \div 100 = \mathbf{2.0}$

Answer: 2.0 joules per token.

Check it. The chapter’s sanity check is that energy per token should be roughly power divided by speed. The speed here is $100 \div 8.0 = 12.5$ tokens per second, and $25.0 \div 12.5 = 2.0$. The two routes agree.

Where this goes wrong. The tempting answer is $25.0 \div 100 = 0.25$ joules per token, from dividing power by tokens and never multiplying by the time. It is tempting because the two numbers you were handed are the two that look most like the question. Watts are not joules. You have to spend a watt for some number of seconds before any energy exists. The other classic slip, named in the chapter’s Common mistakes list, is a sensor that reports milliwatts: forget to divide by 1,000 and you would report 2,000 joules per token.

9. Asked: 5 checkable claims, 3 survive. What is the score? Made up for practice.

This is Formula 1.6, $\hat{p} = x \div n$. The symbol $\hat{p}$ is said “p hat”. The letter $p$ is for proportion, which means “how many out of how many”; see Toolkit 12. The little hat marks a number you measured rather than one you know for certain. $x$ is how many claims survived and $n$ is how many you checked.

Step 1, write down the counts.

$x = 3$ and $n = 5$

Step 2, divide.

$\hat{p} = 3 \div 5 = 0.6$

Step 3, multiply by 100 to turn the decimal into a percentage. Toolkit 11 builds that step from scratch.

$0.6 \times 100 = \mathbf{60\%}$

Answer: $\hat{p} = 0.6$, which is 60%.

Check it. A proportion has to land between 0 and 1, which is between 0% and 100%. It does.

Where this goes wrong. Two tempting wrong answers. The first is $5 \div 3 = 1.667$, from dividing upside down; the sign is a score above 1, which no proportion can be. The second is $2 \div 5 = 0.4$, or 40%, from counting the claims that failed instead of the ones that survived. Both fractions are meaningful, but only one of them is the score the chapter defines. Write down which count goes on top before you divide.

Practice: can you apply it?¶

11. Asked: express safetensors_header_bytes as a percentage of the model.safetensors file, and say in one sentence what that percentage tells you about whether the header is worth worrying about.

This is Formula 1.5, a ratio, followed by the multiply-by-100 step from Toolkit 11.

Step 1, read both numbers out of lab/out/ch01_first_run.json.

header $= 32{,}288$ bytes, and model.safetensors $= 988{,}097{,}824$ bytes

Step 2, divide the header by the file.

$32{,}288 \div 988{,}097{,}824 = 0.0000326769\ldots$

Step 3, round to three significant figures.

0.0000327

Step 4, multiply by 100.

$0.0000327 \times 100 = \mathbf{0.00327\%}$

Answer: the header is 0.00327% of the weight file, which is about three thousandths of one per cent. A number with that many leading zeros is easier to read as $3.27 \times 10^{-5}$ before the percentage step; Toolkit 17 explains that shorthand.

The one sentence the problem asks for. The header costs about three thousandths of one per cent of the file, so it is never worth worrying about as storage; the reason to know the number at all is that it explains the 32,288-byte gap between the size Formula 1.2 predicts and the size the disk reports, and an unexplained gap is worth chasing even when it is tiny.

Check it. A rough version in your head: 32 thousand out of 988 million is about 32 out of a million, and 32 per million is 0.0032%. Our answer is 0.00327%. Another way to feel the size: the file is $988{,}097{,}824 \div 32{,}288 = 30{,}602.6\ldots$ times the header, so about thirty thousand times.

Where this goes wrong. The tempting wrong answers are 0.0327% and 3.27%, both off by a factor of ten or a thousand. They come from multiplying by 100 twice, or from losing a zero when reading a decimal with five of them in a row. The fix is the rough check above: if you cannot get near your answer with round numbers in your head, the decimal point moved.

13. Asked: how many times more energy per token does the 1.5B cost than the 0.5B, and the 3B than the 1.5B? Then: which step up the ladder is the more expensive one, per unit of model?

This is Formula 1.5, $R = A \div B$, applied twice. The numbers come from the table in the thread that runs to Week 15: 0.767, 1.133 and 1.941 joules per token, all from lab/out/theme_s_energy.json.

Step 1, the first rung. Bigger on top.

$1.133 \div 0.767 = 1.4771838\ldots$

Rounded to two decimal places, $\mathbf{1.48}$.

Step 2, the second rung.

$1.941 \div 1.133 = 1.7131509\ldots$

Rounded to two decimal places, $\mathbf{1.71}$.

Answer to the first two parts: the 1.5B costs about 1.48 times the 0.5B per token, and the 3B costs about 1.71 times the 1.5B per token. Both are bare numbers with no unit, because joules divided by joules leaves a comparison rather than a quantity.

Check it. Two rungs multiplied should give the whole ladder. $1.4772 \times 1.7132 = 2.5307$, and the direct ratio is $1.941 \div 0.767 = 2.5306$. They agree to three decimal places, and the small difference is the rounding you did on the way.

Step 3, the parameter multipliers, from the same table.

$1{,}543{,}714{,}304 \div 494{,}032{,}768 = 3.1247204\ldots$, which rounds to 3.12

$3{,}085{,}938{,}688 \div 1{,}543{,}714{,}304 = 1.9990348\ldots$, which rounds to 2.00

Step 4, read the two rungs side by side.

Answer to the third part: the 1.5B to 3B step is the more expensive one. It is more expensive on the plain reading, because 1.71 is bigger than 1.48. It is also more expensive on the per-parameter reading, because the first step tripled the model for 1.48 times the energy while the second step only doubled it and still cost 1.71 times the energy. Both readings point the same way, which is why the ambiguity does not change the answer here. It could change the answer on a different ladder, which is why you name your reading.

Where this goes wrong. The tempting move is to subtract instead of divide: $1.133 - 0.767 = 0.366$ joules and $1.941 - 1.133 = 0.808$ joules. Those differences are real quantities, in joules per token, and they answer the question “how much more”. The question asked “how many times more”, and that phrase means division. Here the two methods happen to rank the steps the same way, so the slip hides. It stops hiding in Chapter 12, where a percentage and a percentage point are different things; see the Toolkit on percentage points.

15. Asked: how many joules does a card drawing 13.834 watts use in one minute of doing nothing, and what does the comparison with the Bakersfield answer’s 20.71 joules suggest?

The idea is the one that separates power from energy in Section 1.5. Power is a rate, in watts. Energy is an amount, in joules. Multiply the rate by the seconds and you get the amount, because one watt is one joule every second.

Step 1, turn the minute into seconds, because a watt is defined per second.

1 minute $= 60$ seconds

Step 2, multiply. Splitting it keeps the arithmetic easy to follow.

$13.834 \times 6 = 83.004$

$83.004 \times 10 = \mathbf{830.04}$ joules

Answer: 830.04 joules.

Step 3, compare with the 20.71 joules that the 27-token Bakersfield answer cost, using Formula 1.5. Bigger on top.

$830.04 \div 20.71 = 40.0791887\ldots$

Rounded to two decimal places, $\mathbf{40.08}$.

The other direction says the same thing as a percentage.

$20.71 \div 830.04 = 0.0249506\ldots$

$0.0249506 \times 100 = \mathbf{2.50\%}$

What the comparison suggests. One minute of a graphics card sitting idle costs about 40 times what the whole Bakersfield answer cost. Put the other way, that answer was about 2.50% of a minute of doing nothing. On a lightly used laptop, almost all of the graphics card’s energy goes to the card being powered on, not to any model. This is the same fact as the 65% figure in Section 1.5, seen from further away, and it is why the chapter insists that you name which column your energy figure comes from.

Check it. There is a second route to the 40. Worked Example 1.1 shows that the Bakersfield answer was worth 1.50 seconds of idle card, and $60 \div 1.50 = 40$. Two routes, one answer.

Where this goes wrong. The tempting answer is 13.834 joules, from multiplying by 1 because the question said “one minute”. A watt is a joule per second, not a joule per minute, so the unit you multiply by has to be seconds. The sign that this happened: your answer for a whole idle minute came out smaller than the 20.71 joules that one short answer cost, which cannot be true when that answer took under a second to write.

17. Asked: score the sentence “The Kern River flows through Bakersfield and is the longest river in California”, stating your splitting rule first. Made up for practice.

This is Formula 1.6, $\hat{p} = x \div n$, wrapped in the discipline from Section 1.3: state the procedure, then count, then divide.

Step 1, the splitting rule, written before any scoring. One claim for each assertion that could be true or false on its own. Where a sentence joins two assertions with “and”, split at the “and” if each half could stand alone as a sentence and be checked by itself.

Step 2, apply the rule.

Step 3, count. Two claims checked, so $n = 2$. One survived, so $x = 1$.

Step 4, divide.

$\hat{p} = 1 \div 2 = 0.5$

Step 5, multiply by 100.

$0.5 \times 100 = \mathbf{50\%}$

Answer: $\hat{p} = 0.5$, which is 50%, under the splitting rule stated in Step 1.

Notice the shape. This is the same score as the Bakersfield sentence in Section 1.3, and for the same reason: a true local fact welded to a false headline claim, in one fluent sentence, with nothing in the grammar marking where the true half stops.

A note on what this problem needs. Claim 2 cannot be checked from anything in Chapter 1. You have to look it up, and that is the point of the exercise rather than a gap in it. Checking is the habit the whole course is built on.

Where this goes wrong. Two tempting answers. The first is 0%, from treating the sentence as one claim, finding a false part, and marking the whole thing false. That is defensible if and only if you stated that rule first, which is exactly what the problem asks you to do. The second is 100%, from passing claim 2 because it sounds authoritative and the rest of the sentence is right. A superlative such as “longest”, “largest” or “first” is a specific ranking claim, and a specific ranking claim always needs a specific source.

19. Asked: a student writes “The model is 0.5 billion parameters, so the file is 0.5 GB.” Name the error, give the correct FP16 size, and name the missing information.

This is Formula 1.2 followed by Formula 1.3.

The error. The student treated a count of parameters as a count of bytes. Those are two different things. A parameter is one of the numbers inside the model. A byte is a unit of storage, eight bits, enough to hold one ordinary English character. How many bytes one parameter takes is a separate fact, and the student’s sentence never states it.

The correct size at FP16.

Step 1, the real parameter count, from lab/out/ch01_first_run.json.

$N = 494{,}032{,}768$

Step 2, the bytes per weight. FP16 is 16 bits per parameter, and there are 8 bits in a byte.

$16 \div 8 = 2$ bytes per parameter

Step 3, multiply, which is Formula 1.2.

$494{,}032{,}768 \times 2 = 988{,}065{,}536$ bytes

Step 4, divide by a billion, which is Formula 1.3.

$988{,}065{,}536 \div 1{,}000{,}000{,}000 = 0.988065536$ GB

Answer: about 0.99 GB at FP16, not 0.5 GB. The student’s figure is short by roughly a factor of two.

The missing information: how many bytes each parameter is stored in, which is the storage format. Without it, a parameter count fixes no file size at all. The same 494,032,768 parameters are 1.98 GB at FP32, 0.99 GB at FP16, and 0.49 GB at INT8.

Where this goes wrong, and why it is so tempting. At 1 byte per parameter, which is INT8, the file really would be $494{,}032{,}768 \div 1{,}000{,}000{,}000 = 0.494$ GB, or about 0.5 GB. So the student’s number is the right answer to a question about a different format, which is the hardest kind of wrong answer to catch. The second reason it is tempting is that “0.5 billion” and “0.5 GB” contain the same digit, and a number that appears twice feels like a confirmation. It is a coincidence of labels. Worth adding: 494,032,768 is not exactly 0.5 billion either. “0.5B” is a rounded name, not a measurement.

Stretch: can you reason with it?¶

21. Asked: explain, in your own words and without formulas, why the model that spends the larger share of its energy on the card merely being switched on is the 0.5B at 65% rather than the 3B at 46%.

This is a written argument, so there is no single right paragraph. Here is what a strong response contains, and then one worked example of such a response.

What a strong response contains.

1. The idea that idle power is a flat background charge, paid for every second the card is on, whatever the card is doing. On this laptop it is 13.834 watts.

2. The observation that the share is the background draw measured against the average draw during the run, and nothing else. The chapter’s Try it 1.4 shows the seconds and the token count cancelling out.

3. The two mean power figures, which are what actually separate the models: the 0.5B ran at a mean of 21.254 watts, so only $21.254 - 13.834 = 7.420$ watts of that was the model’s own work. The 3B ran at 30.203 watts, so $30.203 - 13.834 = 16.369$ watts was its own work.

4. A correction of the causal story the question invites. Speed is not the cause. Per token, the fast small model pays less background energy than the big one, not more.

5. No formulas, as the problem asks.

One worked example of a strong response.

The graphics card charges a flat background fee for every second it is switched on, whether or not anything is running. On this laptop that fee is 13.834 watts. The small model barely lifts the card above it. While the small model was writing, the card averaged 21.254 watts, so only about 7.4 watts of the draw was the model’s own work. The big model pushes the card to an average of 30.2 watts, so about 16.4 watts of that is its own work. The share of the bill charged to the background fee is that fee measured against the average draw, and the big model has a much larger average to measure it against. So the small model’s share is larger because it works the card so lightly, not because it works quickly. Being quick pushes the other way: the small model finishes a token in about 0.036 seconds and the big one takes about 0.064 seconds, so per token the small model pays about 0.4993 joules of background energy and the big one pays about 0.8888 joules. The small model pays less background energy per token and still hands a bigger fraction of its bill to it, because its own work costs so little that there is almost nothing else in the bill.

Where this goes wrong. The tempting paragraph runs: “the small model is faster, so it spends less time doing arithmetic, so a bigger share of its energy is idle.” It sounds right, it uses the word the question put in front of you, and it is backwards. Finishing sooner buys you fewer idle seconds per token, which lowers the idle energy per token. The two per-token figures in the worked response show that directly: 0.4993 joules against 0.8888. The cause is how hard the card is being worked, not how long.

A note on the problem’s wording. The word faster points at a true fact about the two models and at a false explanation of the result. Say so in your answer rather than working around it. Spotting that a correct observation is not the cause is the skill this problem is really testing.

23. Asked: if you measured the whole laptop at the wall instead of the graphics card, would joules per token go up, go down, or stay the same? And could the two figures fairly sit in the same table?

The answer to the first part is: it goes up.

What a strong response contains.

1. The direction, up, with the reason stated as a boundary rather than as a guess. A wall-plug meter counts everything the graphics-card sensor counts, and then also counts the processor, the memory sticks, the screen, the fans, and the energy lost inside the charger turning wall current into laptop current. Every one of those was running while the model wrote.

2. The reason it cannot go down. The graphics card sits inside the wall boundary. You are adding more things to the top of the calculation while the token count on the bottom stays the same, and nothing you add draws negative power.

3. The reason it cannot stay the same. Staying the same would need every other part of the laptop to draw exactly zero watts, which no laptop does.

4. The chapter’s own word for this, from Definition 1.5: every energy figure in this book is a lower bound on the energy of local inference, not a total.

5. An answer to the table question. The two figures should not share a column. They are not two measurements of one quantity; they are measurements of two different quantities that happen to carry the same unit. They can share a table if, and only if, the boundary is a named column, so a reader can see why they differ before comparing them.

6. No invented number. This course did not take a wall-plug measurement, so a response saying “it would be roughly double” is inventing a result.

One worked example of a strong response.

Joules per token would go up. The graphics-card sensor sees one component. The wall meter sees that component plus the processor, the memory, the screen, the fans and the charger’s own losses, all of which were drawing power during the same run. The token count is unchanged, so a larger energy divided by the same number of tokens gives a larger figure. It cannot go down, because the graphics card is inside the wall boundary, and it cannot stay equal unless everything else in the laptop drew nothing. This is what Definition 1.5 means when it calls the book’s figure a lower bound. The two numbers should not go in the same column of a table, because they measure different boundaries. I would put them in one table with a column headed “what the meter covers”, reading “GPU board only” on one row and “whole laptop at the wall” on the other, so nobody reads the difference as a difference between models. I am not giving a number for the wall figure, because we did not measure one.

Where this goes wrong. The tempting answer is “it stays the same, because the model did the same work and produced the same tokens”. It is tempting because the model and the token count genuinely did not change. Joules per token is not a property of the model. It is a property of the model, the machine, and the boundary of the meter, and the question changed the third one. The second tempting answer is “it goes down, because the wall meter measures the whole laptop so the model’s share gets smaller”. That confuses a share with a total. The figure is a total divided by tokens, not a share of anything.

25. Asked: name the single variable the environmental argument and the access argument are both reading, then describe one situation in which the two arguments would point at different models, and say what would have to be true of the hardware or the task.

The variable is the parameter count: how many numbers the model holds. Everything in the Week 15 thread table is that one variable seen twice.

• Read as energy. More parameters means more multiplying and adding for every token, so more joules per token: 0.767, then 1.133, then 1.941.

• Read as bytes. More parameters means more numbers to store and hold in memory, so more gigabytes: 0.99, then 3.09, then 6.17 at FP16.

What a strong response contains. The variable named as the parameter count, not as “size” and not as “energy”, because those are two of its consequences rather than the thing itself. Then one concrete situation, with arithmetic where the chapter supplies the numbers and an honest gap where it does not. Then a sentence naming the condition on the hardware or the task that makes the two arguments come apart.

Worked situation A: change the bytes per weight, and only the access side moves.

Store the 3B model at 4 bits per parameter instead of 16. Bytes per weight becomes $4 \div 8 = 0.5$.

$3{,}085{,}938{,}688 \times 0.5 = 1{,}542{,}969{,}344$ bytes

A card advertised as 4 GB holds 4 gibibytes, which is Formula 1.4 country.

$4 \times 1{,}073{,}741{,}824 = 4{,}294{,}967{,}296$ bytes

$1{,}542{,}969{,}344 \div 4{,}294{,}967{,}296 = 0.3592505\ldots$, and $0.3593 \times 100 = \mathbf{35.9\%}$ of the card

At FP16 the same model needed 143.7% of that card and did not fit. At 4 bits it uses about a third of it. The access argument has moved all the way from “you must rent this model” to “this runs on a student laptop”.

The energy argument has not moved with it, and this is where the response has to be honest. The amount of arithmetic per token did not change; only the number of bits each number is written in changed. This book has no measured energy figure for a quantized model, so the correct statement is: the access argument moved, and we have no measurement saying the energy argument moved with it. The only measured energy figure for the 3B is still 1.941 joules per token, which is 2.53 times the 0.5B.

Two honest footnotes. The 0.5 bytes per weight above is the ideal case. Real quantizers store a small scale for each block of weights, which Chapter 7 covers, and at 0.5625 bytes per weight the same model is $3{,}085{,}938{,}688 \times 0.5625 = 1{,}735{,}840{,}512$ bytes, which is $1{,}735{,}840{,}512 \div 4{,}294{,}967{,}296 = 0.4042$, or 40.4% of the card. It still fits. And this arithmetic is weights only; running a model also needs working memory, as Worked Example 1.2 says.

What would have to be true for this to happen: the binding limit on the hardware is memory rather than power, and the change you made moves bytes per parameter without moving the arithmetic per token.

Worked situation B: a task that needs retries.

Suppose the task has a checkable right answer, the 0.5B gets it right about one time in five, and the 3B gets it first time. Each answer is 27 tokens. Made up for practice, because this chapter measures neither model’s hit rate on any task.

The 0.5B, at five attempts:

$5 \times 27 = 135$ tokens

$135 \times 0.767 = 103.545$ joules

The 3B, at one attempt:

$1 \times 27 = 27$ tokens

$27 \times 1.941 = 52.407$ joules

$103.545 \div 52.407 = 1.9757856\ldots$, which rounds to $\mathbf{1.98}$

Per useful answer, the big model is about twice as cheap, while the access argument still points at the small one, because the 3B needs 143.7% of a 4 GB card at FP16 and cannot run there at all. What would have to be true: the task has a right answer you can check, and the small model’s hit rate is low enough that its retries cost more than its per-token saving.

Where this goes wrong. The tempting answer to the first part is “energy” or “size” or “how big the model is”. Those are readings of the variable, not the variable, and a response that names one of them has not said the thing the question is after: that one number, the parameter count, sits upstream of both columns. The tempting answer to the second part is to describe a situation where one argument is merely stronger than the other. The question asks for the two to point at different models, which needs something that moves one column without moving the other. Changing the bytes per weight does that. Changing how many attempts a task needs does that. Buying a faster graphics card does not, because it moves both columns the same way.

Chapter 2. Numbers all the way down: tokens¶

These problems drill four things: dividing one count by another to get characters per token, multiplying rows by columns to price a vocabulary table, multiplying a token count by a measured energy rate, and rebuilding a written number from its digits and their powers of ten.

Unless a solution says otherwise it uses the chapter’s standing values: $V = 151{,}936$, $d = 896$, $N_{\text{total}} = 494{,}032{,}768$ and $p = 0.767$ joules per token, all measured for Qwen2.5-0.5B-Instruct on the course lab machine and stored in lab/out/we1_tokens.json, lab/out/we3_params_quant.json, lab/out/ch03_parameter_anatomy.json and lab/out/theme_s_energy.json.

Warm-up¶

1. What is being asked. Work out how many characters each token of Bakersfield carried, to four decimal places.

What it tests. Formula 2.4, characters per token, $r = \dfrac{c}{n}$, from Section 2.3 of Chapter 2. The fraction bar means divide the top number by the bottom number; toolkit section 4 starts from the beginning.

The work.

Step 1, write down the two counts. The problem gives both. $c = 11$ characters and $n = 3$ tokens.

Step 2, divide the characters by the tokens.

The 6s never stop. Your calculator will show something like 3.666666667.

Step 3, round to four decimal places. Look at the fifth decimal place, which is a 6. Six is 5 or more, so the fourth decimal place goes up from 6 to 7.

Answer. 3.6667 characters per token.

Check it. Multiply back. $3.6667 \times 3 = 11.0001$. That is 11, with a leftover of 0.0001 that comes from the rounding in step 3 and from nowhere else.

Where this goes wrong. The tempting error is dividing the other way round, $3 \div 11 = 0.2727$. It is tempting because the problem says “3 tokens” and the 3 is the smaller, friendlier number to put first. Two things catch it. The name of the quantity is characters per token, and the word before “per” goes on top. And for this tokenizer no token is shorter than one character, so $r$ can never drop below 1. An answer of 0.2727 fails that test on sight.

3. What is being asked. Count how many individual numbers sit in a vocabulary table with 50,000 rows and 768 columns.

What it tests. Formula 2.2, $N_{\text{table}} = V \times d$, from Section 2.2 of Chapter 2, and the same formula in the formula summary, section 2.2. The symbol $\times$ means multiply the number on its left by the number on its right; toolkit section 3 shows the four ways this book writes multiplication.

The work.

Step 1, name the two numbers. $V = 50{,}000$ rows. $d = 768$ columns in every row.

Step 2, write down the multiplication.

Step 3, split it into a small multiplication and some zeros, because 50,000 is 5 followed by four zeros.

Step 4, put the four zeros back on the end of 3,840.

Answer. $N_{\text{table}} = 38{,}400{,}000$ numbers.

Check it. Two checks. Divide back: $38{,}400{,}000 \div 768 = 50{,}000$, which is the row count you started with. And rough it out first: 50,000 times 800 is 40,000,000, so an answer near 38 million is the right size.

Where this goes wrong. Two slips, both common. Adding instead of multiplying gives $50{,}000 + 768 = 50{,}768$, which is tempting because the two numbers sit side by side and addition is the reflex. The answer must be far larger than either number you started with, and 50,768 is barely larger than one of them. The second slip is losing a zero and writing 3,840,000. Counting the zeros out loud before you write them down is worth the two seconds.

Labelled honestly. The 50,000 and the 768 are made up for practice, as the problem set’s preamble says. Our model’s real values are $V = 151{,}936$ and $d = 896$.

5. What is being asked. Find the energy a model spends writing 250 tokens.

What it tests. Formula 2.6, $E = n \times p$, from Section 2.5 of Chapter 2, and the formula summary, section 7.1.

The work.

Step 1, name the two numbers. $n = 250$ tokens. $p = 0.767$ joules per token (measured, from lab/out/theme_s_energy.json).

Step 2, write down the multiplication.

Step 3, break 0.767 into three easy pieces: 0.7, then 0.06, then 0.007.

Step 4, add the three pieces, two at a time.

Answer. $E = 191.75$ joules.

Check it. Three checks, and the third is the good one. The answer must be much larger than 0.767, because 250 tokens cost more than one token does. Rough it out: 0.767 is close to 0.75, and $250 \times 0.75 = 187.5$, so an answer near 191 is the right size. And an independent one: the lab recorded 0.213 watt-hours per 1,000 tokens for this model. 250 tokens is a quarter of 1,000, so the cost should be a quarter of 0.213, which is $0.213 \div 4 = 0.05325$ watt-hours. Convert your own answer by dividing by 3,600, since one watt-hour is 3,600 joules: $191.75 \div 3600 = 0.053264$ watt-hours, rounded to six decimal places. Two separate measurements agreeing to four decimal places is a strong sign the arithmetic is right.

Where this goes wrong. The tempting error is dividing, $250 \div 0.767 = 325.9$. It is tempting because the rate has the word “per” in it, and “per” often signals division. Hold on to the units instead. The rate is joules per token, so tokens times joules per token leaves joules, and joules is the unit the question wants. The second slip is writing the answer as a bare “191.75”. A number without a unit is most of an error waiting to happen. Write it as 191.75 joules.

One limit to carry with it. The 0.767 figure is GPU board power only. No processor, no memory, no power-supply losses, no screen, no cooling. It is a lower bound on the energy of a local run, not a total.

7. What is being asked. Write five powers of ten as ordinary numbers.

What it tests. Exponent notation, used in Formula 2.5 in Section 2.4 of Chapter 2. The small raised number says how many times to multiply 10 by itself; toolkit section 5 builds it up from nothing.

The work, one line at a time.

$10^{0} = 1$. Anything raised to the power zero is 1. There are no 10s to multiply, and the answer is 1 rather than 0.

$10^{1} = 10$. One 10, on its own.

$10^{2} = 10 \times 10 = 100$

$10^{3} = 10 \times 10 \times 10 = 1{,}000$

$10^{4} = 10 \times 10 \times 10 \times 10 = 10{,}000$

Answer.

Check it. The exponent is the number of zeros written after the 1. 10⁴ has four zeros. 10⁰ has none, which leaves the bare 1. Each line is ten times the line above it, and it is.

Where this goes wrong. Almost everybody who gets one of these wrong gets 10⁰ wrong, and writes 0. It looks right, because a zero is sitting there in the exponent. Here is why it is 1. Going down the list, each step removes one 10 and therefore divides by 10. From $10^{2} = 100$ to $10^{1} = 10$ is a division by 10. Take the next step down, from $10^{1} = 10$, and you divide by 10 again, which gives 1. The pattern has no choice in the matter. The second slip is writing $10^{1} = 1$ by reading the exponent as the answer.

Practice¶

9. What is being asked. Find characters per token for the string ' spaces', which has two spaces in front of the word, then say why the six letters spaces can cost one token or two.

What it tests. Formula 2.4 again, $r = \dfrac{c}{n}$, together with Definition 2.1, which says a token can be a space followed by a word.

The work.

Step 1, write down the two counts, which the problem gives. $c = 8$ characters: two spaces plus the six letters s-p-a-c-e-s. $n = 2$ tokens, the pieces ' ' and ' spaces' (measured, from lab/out/we1_tokens.json, ids 220 and 12621).

Step 2, divide.

Step 3, write it to four decimal places, as the chapter does.

Answer. 4.0000 characters per token.

The one sentence the problem asks for. The six letters spaces cost one token when a single space sits in front of them, because ' spaces' with its space attached is one row of the vocabulary, and they cost two when a second space sits in front, because the leftover space has no word to attach itself to and needs a row of its own.

Check it. Multiply back: $4.0000 \times 2 = 8$, which is the character count. This division comes out exactly, so there is no rounding crumb this time.

Where this goes wrong. The tempting error is counting the string as 6 characters and getting $6 \div 2 = 3.0000$. It is tempting because spaces are invisible and the eye counts letters. Spaces are characters. len() in Python counts them, the tokenizer sees them, and a service billing you by the token is charging you for them. Count every character between the quotation marks, including the ones you cannot see.

11. What is being asked. Find characters per token for a 28-letter word and for a seven-digit string, then say why the longer one is cheaper per character.

What it tests. Formula 2.4, $r = \dfrac{c}{n}$, and the point of Section 2.3 of Chapter 2 that length is not what decides the cost.

The work, one string at a time.

String one, antidisestablishmentarianism. Here $c = 28$ and $n = 6$.

The 6s never stop. Round to four decimal places: the fifth decimal is 6, which is 5 or more, so the fourth goes up from 6 to 7.

String two, 1234567. Here $c = 7$ and $n = 7$.

Written to four decimal places, $r = 1.0000$. This one comes out exactly.

Answer. antidisestablishmentarianism is 4.6667 characters per token. 1234567 is 1.0000 characters per token.

The one sentence the problem asks for. The long word is cheaper per character because its six pieces are 'ant', 'idis', 'establish', 'ment', 'arian' and 'ism' (measured, from lab/out/we1_tokens.json), chunks the counting procedure met often enough to give each one a row, and each of those rows carries three to nine letters at once, while a bare digit only ever earned a single-character row, so seven digits have to buy seven rows.

Check it. Multiply back. $4.6667 \times 6 = 28.0002$, which is 28 plus a rounding crumb. $1.0000 \times 7 = 7$ exactly. And 1.0000 is the smallest value $r$ can take for this tokenizer, since no token is shorter than one character, so 1234567 is the worst possible result rather than merely a bad one.

Where this goes wrong. The tempting conclusion is that longer text costs more per character. These two rows say the opposite: 28 characters came to 4.6667 per token and 7 characters came to 1.0000 per token. What decides the cost is how often each chunk turned up in the pile of text the vocabulary was counted from, and nothing else. The arithmetic slip to watch for is rounding 4.666666 to 4.67 and then multiplying by 6, which gives 28.02 rather than 28.

13. What is being asked. Count the vocabulary rows the tokenizer never produces, then count how many numbers those rows take up.

What it tests. The three vocabulary counts from Section 2.2 of Chapter 2 and Cell 4, then Formula 2.2, $N_{\text{table}} = V \times d$, applied to the unused rows only. The sign $-$ means take the number on the right away from the number on the left.

The work.

Step 1, subtract the larger tokenizer count from the model’s row count.

Step 2, every row is $d = 896$ numbers wide, so multiply.

Step 3, break it up, because 896 is 900 minus 4.

Step 4, subtract the second from the first.

Answer. 271 rows are never emitted, and they occupy 242,816 numbers.

Check it. Rough out step 3: 271 times 900 is 243,900, so an answer near 243 thousand is the right size. Then put it in proportion with Formula 2.3, to see whether it is worth worrying about: $242{,}816 \div 494{,}032{,}768 = 0.000491$, rounded to six decimal places, and $0.000491 \times 100 = 0.0491$, so the waste is about 0.05% of the model. Real, measurable, and small.

Where this goes wrong. The tempting error is subtracting 151,643 instead of 151,665, which gives $151{,}936 - 151{,}643 = 293$ rows and $293 \times 896 = 262{,}528$ numbers. It is tempting because 151,643 is the number tokenizer.vocab_size prints, and that looks like the official one. It is the count of ordinary tokens only. The question asks about ids the tokenizer can ever produce, and it can also produce 22 special markers, so the count to use is len(tokenizer), which is 151,665. This is common mistake 4 from the chapter, and it is the mistake that turns 27.56% into 27.51% elsewhere.

15. What is being asked. Use place value to rebuild the number that the digit string 50607 spells out, and say what the two zeros do.

What it tests. Formula 2.5, $N = \sum_{k=0}^{m} d_k \times 10^{k}$, from Section 2.4 of Chapter 2. The symbol $\sum$ is the capital Greek letter sigma and means “add up everything that follows”; toolkit section 10 builds it up slowly.

Step 0, state $m$. The string has five digits, and $m$ is one less than the number of digits, so $m = 4$.

Step 1, number the digits from the right. Position 0 is the rightmost digit, and the position numbers grow as you move left. The digit written first has the largest position number.

$d_4 = 5$, $d_3 = 0$, $d_2 = 6$, $d_1 = 0$, $d_0 = 7$

Step 2, write out each power of ten.

$10^{4} = 10{,}000$

$10^{3} = 1{,}000$

$10^{2} = 100$

$10^{1} = 10$

$10^{0} = 1$

Step 3, multiply each digit by its power of ten.

$5 \times 10{,}000 = 50{,}000$

$0 \times 1{,}000 = 0$

$6 \times 100 = 600$

$0 \times 10 = 0$

$7 \times 1 = 7$

Step 4, add them, one at a time, from the top down.

$50{,}000 + 0 = 50{,}000$

$50{,}000 + 600 = 50{,}600$

$50{,}600 + 0 = 50{,}600$

$50{,}600 + 7 = 50{,}607$

Answer. $N = 50{,}607$.

What the two zeros contribute. Nothing to the total and everything to the positions. Each zero term added 0 to the sum, so you could delete both of those lines and the total would not move. What you cannot delete is the places they hold. Take the zeros out of the string and you are left with 567, which is a different number by a factor of about ninety. The zeros are placeholders: their job is to keep the 5 in the ten-thousands position and the 6 in the hundreds position.

Check it. The digits of the answer read 5, 0, 6, 0, 7, in that order, matching the string you started with. And the first term on its own, 50,000, must be larger than everything else added together, because each position is worth ten times the one to its right. Everything else comes to $0 + 600 + 0 + 7 = 607$, and 50,000 is larger.

Where this goes wrong. Two tempting errors. The first is skipping the zero terms because they contribute nothing, then renumbering the remaining positions, which turns 50607 into 567. Write the zero lines out even though they are boring; they are what keeps the numbering honest. The second is counting positions from the left, which gives $5 \times 1 + 0 \times 10 + 6 \times 100 + 0 \times 1{,}000 + 7 \times 10{,}000 = 70{,}605$. That is the digits read backwards. Formula 2.5 counts positions from the right-hand end, starting at zero, every time.

17. What is being asked. Compute characters per token for Bakersfield under two different tokenizers, then explain why the bigger vocabulary gave the longer token list.

What it tests. Formula 2.4, $r = \dfrac{c}{n}$, together with the argument in Section 2.2 of Chapter 2 that a vocabulary size is a decision rather than a measure of quality.

The work.

Bakersfield is 11 characters either way, so $c = 11$ on both lines.

Tokenizer one, our model, $V = 151{,}936$. The pieces are 'B', 'akers' and 'field', so $n = 3$ (measured, from lab/out/we1_tokens.json).

Rounded to four decimal places, $r = 3.6667$.

Tokenizer two, all-MiniLM-L6-v2, $V = 30{,}522$. The pieces are 'baker' and '##sfield', so $n = 2$ (measured, same file).

Written to four decimal places, $r = 5.5000$.

Answer. Our model gets 3.6667 characters per token on this word. all-MiniLM-L6-v2 gets 5.5000 characters per token on the same word, from a vocabulary about a fifth the size: $151{,}936 \div 30{,}522 = 4.9779$, rounded to four decimal places.

The two sentences the problem asks for. A vocabulary size says how many chunks earned a row; it says nothing about which chunks earned one, and only the which decides how any particular word gets cut. MiniLM’s 30,522 rows happen to include baker, and our 151,936 rows include no chunk that covers Bakers, so on this one word the smaller table wins, which it could not do if size alone decided the answer.

Check it. Multiply back. $5.5000 \times 2 = 11$ exactly, and $3.6667 \times 3 = 11.0001$, which is 11 plus a rounding crumb. Both land on the 11 characters you started with.

Where this goes wrong. The tempting conclusion is that MiniLM is the better tokenizer, or that a smaller vocabulary is better in general. One word is one word. The honest statement is that two working systems disagree about this word, which is what makes the vocabulary size a design choice rather than a fact about English. There is also a detail worth naming rather than hiding: MiniLM lowercases the text before cutting it, which is why its first piece is baker and not Baker. The two tokenizers are not doing quite the same job, and part of how MiniLM got away with two pieces is that it never needed a separate row for a capital letter.

Stretch¶

19. What is being asked. Add the four measured shares of this model, then say how you would tell a rounding gap from a forgotten part, and which of the two should worry you more.

What it tests. Formula 2.3, the share of a model held by one part, from Section 2.2 of Chapter 2, and the rule that a complete list of parts has to come to 100%. All four shares are measured, from lab/out/ch03_parameter_anatomy.json.

Part one, the addition. Add them two at a time, as printed.

Answer. The four printed shares come to 99.9989%, which is 100.00% once you round to the two decimal places the first three shares already carry.

A note on the wording, because the problem can be read two ways. Add the four figures exactly as printed and you get 99.9989%. Round the small one to two decimal places first, as the chapter does, and you get $99.99 + 0.01 = 100.00\%$ exactly. Both routes are defensible and both land on 100.00% at two decimal places, so say which route you took. The full-precision shares in the lab file are 27.555794841527597, 63.51529459681508, 8.920023701747654 and 0.00888685990966494, and those four add to exactly 100.0. The 0.0011 missing from 99.9989 is rounding, and it is nothing else.

Part two, the two sentences. A strong answer contains three things: a way to turn the gap into a count of parameters, a bound on how large a rounding gap can possibly be, and a clear statement of which failure is worse. One worked version:

Turn the gap into parameters before arguing about it: 0.01% of 494,032,768 is $0.0001 \times 494{,}032{,}768 = 49{,}403$ numbers, so I would look for a named part of the model about that size, and the layer norms at 43,904 numbers are exactly that. Then bound the rounding: three numbers each rounded to two decimal places can each be off by at most 0.005, so rounding alone can never move the total by more than $3 \times 0.005 = 0.015$ percentage points, which means a gap of 0.01 could be rounding while a gap of 0.1 could not.

Which one should worry you more. The forgotten part, and it is not close. A rounding gap is bounded, harmless, and goes away the moment you keep more decimal places. A missing part means your list of components is wrong, which means you do not know what the model is made of, and every share you computed was divided by a total you cannot account for. The gap is small either way. What differs is what the gap is telling you about your own bookkeeping.

Where this goes wrong. The tempting move is to write “99.99, close enough to 100” and stop. That habit is what would have hidden the layer norms. Chase every gap to a named cause. The opposite move is tempting too: assuming any gap must be a missing part. Compute the rounding bound first, since it costs one multiplication and it settles the question.

21. What is being asked. Turn “the model is bad at maths because it does not really understand numbers” into a claim about the input that Cell 7 could check, then say what result would prove your version wrong.

What it tests. The distinction in Section 2.4 of Chapter 2 between a measurement and a hypothesis. This is a written answer, so there is no single right wording. What follows is what a strong response contains, then one example of such a response.

What a strong response contains. Four things.

1. It talks about the input, not about the model’s mind. Words like “understand” and “really” cannot be measured and have to go.

2. It names a count somebody could produce, on a named tokenizer, with a named string.

3. It needs no model weights. Cell 7 loads a tokenizer, which is a few megabytes, and prints a number.

4. Its falsifier is a measurement, not a counter-argument.

One worked example of such a response.

Rewritten claim. When the tokenizer of Qwen2.5-0.5B-Instruct is given the string 1234567, it returns seven token ids, one per digit, and no id in that list carries the position of its digit inside the number.

What would show it wrong. Run Cell 7 and get fewer than seven tokens: one id standing for the whole of 1234567, or three ids standing for 123, 456 and 7. Then multi-digit chunks have rows of their own, the grouping is in the input, and my claim is false for that tokenizer. A second result would also kill it. If the id standing for '3' inside 1234567 differed from the id standing for '3' inside 37, then position would be riding along in the input after all. It does not. '3' is id 18 in both, and the digits '0' through '9' occupy the consecutive ids 15 through 24 whatever sits around them (measured, from lab/out/we1_tokens.json).

What the rewrite gives up, and why that is the point. The rewritten claim says nothing about whether the model adds well. It cannot, because this course has not measured that. “Digits arrive one token each” is a measurement you can reproduce in ten seconds. “Therefore the model is bad at maths” is a separate claim that needs a question bank, a scoring procedure and a confidence interval, which is Chapter 12. Giving up the bigger claim is what makes the smaller one worth something.

Where this goes wrong. The tempting rewrite keeps the conclusion and changes only the tone, for example “the model has no internal representation of quantity”. That is the original claim in a lab coat. It names nothing you can count, and Cell 7 cannot touch it. Ask two questions of your own sentence: what number would I print, and what value of that number would make me withdraw the sentence? If you cannot answer both, rewrite again.

23. What is being asked. List three things you would have to measure before anyone could claim “Spanish speakers pay more to use AI”, and say which one you would do first and why.

What it tests. Common mistake 6 from Chapter 2: characters per token belongs to one tokenizer and one text, not to a language. This is a written answer, so there is no single right list.

What the evidence actually is right now. One Spanish string, ¿Cómo estás?, measured 12 characters in 5 tokens, so $r = 12 \div 5 = 2.40$ characters per token (measured, from lab/out/we1_tokens.json). One string. A sample of one is not a study of a language.

What a strong response contains. Three measurements that are each genuinely measurable, a recognition that the tokenizer is part of the claim and not a neutral instrument, and a recognition that “pay more” is about money for a delivered message, not about a ratio.

One worked example of such a response.

Measurement 1. Many matched pairs, not one string. Take a few hundred documents that exist in both English and Spanish, the same content in each, and run every pair through the same tokenizer. Report the mean token count for each language with a confidence interval, the way Chapter 12 does. One pair is an anecdote; a few hundred pairs with an interval is a measurement.

Measurement 2. Tokens per message, not characters per token. Characters per token is the wrong quantity for a claim about money. A Spanish sentence with the same meaning is usually longer in characters than its English version, so a language could score well on characters per token and still cost more per message. Bills are per token, so measure the token count of the whole message.

Measurement 3. The tokenizers people are actually billed by. The 2.40 figure belongs to Qwen2.5-0.5B-Instruct. A person paying for a commercial service meets a different table, and Section 2.2 already shows two tokenizers disagreeing about a single word. Repeat measurements 1 and 2 on the tokenizers of the services the claim is about.

Which one first, and why. Measurement 1. Two reasons, and the second is the better one. It is the cheapest: a tokenizer is a few megabytes, no model weights are loaded, and a few hundred pairs run in seconds on a laptop. And it is the one most likely to kill the claim early. If the difference does not survive a few hundred matched pairs on the one tokenizer already sitting on the lab machine, nobody needs to go and price three commercial services. Run the measurement that can end the argument fastest, and run it before you pay for the expensive ones.

Where this goes wrong. The tempting move is to accept the direction of the claim and argue only about its size, for instance “yes, but it is probably a small difference”. That still treats one string as evidence. The honest position is that the current evidence does not support the claim in either direction. The second tempting move is to set 2.40 against 3.6667 as though two languages were being weighed. They are not. One is a Spanish greeting and the other is a California place name, and a place name is an unusual word in any language.

Chapter 3. What is a parameter?¶

These problems drill five moves: counting the numbers in a rectangle, turning a part into a share, turning a count of numbers into a count of bytes, describing a pile of weights with a middle and a spread, and writing very large and very small numbers in scientific notation.

Warm-up¶

1. What is asked. How many numbers are in a weight matrix whose shape is (896, 4864).

What it is testing. Formula 3.1, the count of numbers in a matrix: $N_{\text{matrix}} = (\text{rows}) \times (\text{columns})$. The $\times$ sign is multiplication, from Math Toolkit Section 3. The same formula is stated in the formula summary, section 2.2.

The work. The shape is written (rows, columns), so rows $= 896$ and columns $= 4{,}864$. The problem asks you to split 896 into $900 - 4$, because multiplying by 900 is easy and multiplying by 896 is not.

Step 1, multiply by 900. First multiply by 9, then put the two zeros back on.

$4{,}864 \times 9 = 43{,}776$

$4{,}864 \times 900 = 4{,}377{,}600$

Step 2, multiply by 4.

$4{,}864 \times 4 = 19{,}456$

Step 3, subtract, because you used 900 where you wanted 896.

$4{,}377{,}600 - 19{,}456 = 4{,}358{,}144$

The answer. $N_{\text{matrix}} = 4{,}358{,}144$ parameters.

Check it. The answer must be a whole number, and it must be far bigger than either side of the rectangle. It is. A second check: this is the same rectangle as mlp.gate_proj in Worked example 3.2, which has shape (4864, 896), the same rectangle turned on its side. Turning a rectangle on its side does not change how many numbers are inside it, and Worked example 3.2 also gets 4,358,144. One more: there are 24 layers, and $4{,}358{,}144 \times 24 = 104{,}595{,}456$, which is exactly the mlp.gate_proj row printed by Cell 3.

Where this goes wrong. The tempting wrong answer is $896 + 4{,}864 = 5{,}760$. It is tempting because the shape hands you two numbers and addition is the first thing people do with two numbers. Spot it by size: an answer of 5,760 is about the same size as one side of the rectangle, and the count of numbers in a rectangle has to be very much bigger than either side. This is Common mistake 6 in the chapter.

3. What is asked. Work out $104{,}595{,}456 \div 494{,}032{,}768$ and give it as a percentage to four decimal places.

What it is testing. Formula 3.2, the share of the parameters held by one part: $\text{share} = N_{\text{component}} / N_{\text{total}}$, and then turning a decimal into a percentage, from Math Toolkit Section 11.

The work. The part on top is 104,595,456, which is one of the three MLP matrices counted across all 24 layers. The whole underneath is 494,032,768, the parameter count of the model.

Step 1, divide. Type it into a calculator in the order it is written.

$104{,}595{,}456 \div 494{,}032{,}768 = 0.2117176$

Your calculator will keep going, 0.21171764865 and onwards.

Step 2, multiply by 100 to turn the decimal into a percentage. Multiplying by 100 moves the decimal point two places to the right.

$0.21171764 \times 100 = 21.171764$

Step 3, round to four decimal places. Keep four digits after the point, 21.1717, then look at the next digit. It is a 6, which is 5 or more, so round the last kept digit up.

$21.171764 \rightarrow 21.1718$

The answer. $21.1718\%$ of the model.

Check it. A share has to land between 0 and 1 before you convert it, and between 0% and 100% after. This one does. A second check: 21.1718% is exactly the mlp.gate_proj share printed by Cell 3, and there are three matrices that size, so $21.1718 \times 3 = 63.5154$, next to the measured 63.5153% for all three MLP matrices together. The gap of 0.0001 percentage points is rounding, not a mistake, and the chapter’s warning box on rounded shares says why.

Where this goes wrong. The tempting wrong answer is $494{,}032{,}768 \div 104{,}595{,}456 = 4.7233$, which becomes 472.33% if you carry on and multiply by 100. It is tempting because the bigger number feels like it belongs on top. A share above 100% is always this mistake, and the fix is to swap the two numbers. This is Common mistake 4 in the chapter. The other slip is stopping at 0.2118 and writing a percent sign after it. That is the share, not the percentage; it still needs the $\times 100$.

5. What is asked. The size of a 494,032,768-parameter model in bytes when each parameter takes four bytes, and then that size in gigabytes.

What it is testing. Formula 3.5, $\text{size in bytes} = N \times (\text{bytes per weight})$, and the unit rule from Definition 3.7: GB means 10⁹ bytes, which is one thousand million bytes.

The work.

Step 1, write down the two pieces. $N = 494{,}032{,}768$, and bytes per weight $= 4$. Four bytes per number is FP32, because FP32 gives each number 32 bits and $32 \div 8 = 4$.

Step 2, multiply by 4. Multiplying by 4 is doubling twice, which you can do on paper.

$494{,}032{,}768 \times 2 = 988{,}065{,}536$

$988{,}065{,}536 \times 2 = 1{,}976{,}131{,}072$

Step 3, turn bytes into gigabytes by dividing by one thousand million.

$1{,}976{,}131{,}072 \div 1{,}000{,}000{,}000 = 1.976131072$

Step 4, round to three decimal places. The fourth decimal digit is 1, which is under 5, so the third decimal stays as it is.

$1.976131072 \rightarrow 1.976$

The answer. 1,976,131,072 bytes, which is 1.976 GB.

Check it. Halving the bytes per weight has to halve the file exactly, so FP16 at two bytes should come out at half of this: $1.976131072 \div 2 = 0.988065536$, and 0.988 GB is the FP16 figure printed by Cell 9 and recorded in lab/out/ch03_parameter_anatomy.json. A rough check also works: about half a billion numbers at four bytes each is about two thousand million bytes, which is about 2 GB.

Where this goes wrong. The tempting wrong answer is 1,976.131 GB, from dividing by 1,000,000 instead of by 1,000,000,000. It is tempting because both divisions feel like “move the decimal point a lot” and the two look alike on a calculator screen. That answer is the size in megabytes, not gigabytes. MB is 10⁶ bytes and GB is 10⁹ bytes, three zeros apart. The other slip is reading FP32 as 32 bytes and multiplying by 32, which gives a file eight times too large. This is Common mistake 8 in the chapter.

7. What is asked. Write 151,936, the vocabulary size, in scientific notation, keeping every digit.

What it is testing. Formula 3.6, $x = a \times 10^{b}$ with $1 \le a < 10$. The full treatment, from nothing, is in Math Toolkit Section 17.

The work.

Step 1, put the decimal point after the first digit. The first digit of 151,936 is 1, so the mantissa is 1.51936.

Step 2, count how many places the point moved. In 151,936 the decimal point sits at the far right, after the 6. To reach 1.51936 it moved left past five digits: the 6, the 3, the 9, the 1 and the 5.

Step 3, moving the point to the left gives a positive exponent, and the exponent is the number of places it moved.

The answer. $1.51936 \times 10^{5}$ tokens. No digit was dropped, because the problem asked for every digit. Had it asked for three significant figures, the answer would be $1.52 \times 10^{5}$.

Check it. Move the point back the way you came, five places to the right: 1.51936 becomes 15.1936, then 151.936, then 1519.36, then 15193.6, then 151936. You land on the number you started with. A second check: Cell 11 prints this number as 1.51936e+05, and e+05 on a screen means “times ten to the power of 5”.

Where this goes wrong. Two tempting wrong answers. The first is $15.1936 \times 10^{4}$. That is arithmetically equal to 151,936, and it is still not scientific notation, because the mantissa has to be at least 1 and less than 10. The second is $1.51936 \times 10^{-5}$, from getting the sign of the exponent backwards. That value is 0.0000151936, which is not a count of tokens. A number bigger than 10 always has a positive exponent.

9. What is asked. The mean of five numbers, 2, 4, 4, 6 and 9, showing the running total and the division.

What it is testing. Formula 3.3, the mean: $\bar{w} = \frac{1}{N}\sum_{i=1}^{N} w_i$. The $\sum$ sign means “add these up”, from Math Toolkit Section 10.

The work. These five numbers are small and made up for practice. The steps are the same ones the code runs over 802,816 real weights.

Step 1, add them one at a time, writing down the running total each time.

2

$2 + 4 = 6$

$6 + 4 = 10$

$10 + 6 = 16$

$16 + 9 = 25$

The total is 25.

Step 2, count how many numbers there were. $N = 5$.

Step 3, divide the total by the count.

$25 \div 5 = 5$

The answer. The mean is 5.

Check it. A mean always lands between the smallest number in the list and the largest. The smallest here is 2 and the largest is 9, and 5 sits between them. If your answer is bigger than every number in the list, you forgot to divide by 5.

Where this goes wrong. The tempting wrong answer is 4. The list arrives already sorted, 2, 4, 4, 6, 9, and the middle one is 4. That makes 4 the median, not the mean. It is tempting because 4 appears twice and sits in the middle, and because the two words get used as if they meant the same thing. They do not. Section 3.4 is built on the fact that the mean and the median of the same list can be different numbers, and here they are: the mean is 5 and the median is 4.

Practice¶

11. What is asked. The FP32 size, in gigabytes, of a model with 1,543,714,304 parameters.

What it is testing. Formula 3.5 again, on a larger count. The parameter count of Qwen2.5-1.5B-Instruct is measured and recorded in lab/out/theme_s_energy.json.

The work.

Step 1, bytes per weight. FP32 gives each number 32 bits, and a byte is 8 bits, so $32 \div 8 = 4$ bytes per weight.

Step 2, multiply, by doubling twice.

$1{,}543{,}714{,}304 \times 2 = 3{,}087{,}428{,}608$

$3{,}087{,}428{,}608 \times 2 = 6{,}174{,}857{,}216 \text{ bytes}$

Step 3, divide by one thousand million to get gigabytes.

$6{,}174{,}857{,}216 \div 1{,}000{,}000{,}000 = 6.174857216$

Step 4, round to three decimal places. The fourth decimal digit is 8, which is 5 or more, so the third decimal rounds up from 4 to 5.

$6.174857216 \rightarrow 6.175$

The answer. 6,174,857,216 bytes, which is 6.175 GB.

Check it. The number at the end of step 2’s first line, 3,087,428,608 bytes, is the FP16 size, and 3.087 GB is exactly what Try it 3.3 worked out. FP32 has to be exactly double FP16, and $3.087428608 \times 2 = 6.174857216$. It is. A second check: this model holds $1{,}543{,}714{,}304 \div 494{,}032{,}768 = 3.1247$ times the parameters of the small one, and the small one came to 1.976131072 GB at FP32, so this one should come to about $1.976131072 \times 3.1247 = 6.1748$ GB. It does, and the last digit differs only because 3.1247 is itself a rounded figure.

Where this goes wrong. The tempting wrong answer is 3.087 GB. It is tempting because that figure is already worked out in Try it 3.3 and is sitting on the page. Try it 3.3 asked for FP16, two bytes per number. This problem asks for FP32, four bytes. Read the format the question names before you pick the multiplier.

13. What is asked. What share of the 494,032,768-parameter model is held by the self_attn.q_proj matrices across all 24 layers, which hold 19,289,088 parameters, as a percentage to four decimal places.

What it is testing. Formula 3.2, $\text{share} = N_{\text{component}} / N_{\text{total}}$.

The work.

Step 1, the part goes on top and the whole goes underneath. Divide.

$19{,}289{,}088 \div 494{,}032{,}768 = 0.0390441$

The calculator carries on, 0.039044146966 and onwards.

Step 2, multiply by 100, which moves the decimal point two places to the right.

$0.03904414 \times 100 = 3.904414$

Step 3, round to four decimal places. Keep 3.9044, then look at the next digit. It is a 1, which is under 5, so 3.9044 stays.

The answer. $3.9044\%$ of the model.

Check it. It matches the self_attn.q_proj row printed by Cell 3 and stored in lab/out/ch03_parameter_anatomy.json. You can also rebuild the count from the shapes. q_proj is an 896 by 896 matrix, so $896 \times 896 = 802{,}816$ numbers, plus a bias holding one number for each of the 896 outputs:

$802{,}816 + 896 = 803{,}712$ per layer

$803{,}712 \times 24 = 19{,}289{,}088$ across all 24 layers

which is the count the problem gave you.

Where this goes wrong. Two tempting wrong answers. The first is 0.0390%, from leaving out the $\times 100$ and writing a percent sign on the decimal anyway. The second is 3.9001%, which is the share of self_attn.o_proj. The two look like the same matrix, both 896 by 896, and they hold different counts: q_proj carries a bias and o_proj does not, so q_proj holds 21,504 more parameters across the 24 layers. If your answer comes out at 3.9001%, check which row of Cell 3 you read.

15. What is asked. How many times larger the vocabulary table, at 136,134,656 parameters, is than all of attention, at 44,067,840 parameters, to four decimal places.

What it is testing. The comparison used at the end of Section 3.2: to ask how many times larger one count is than another, divide the larger by the smaller. The fraction bar and $\div$ give the same instruction, from Math Toolkit Section 4.

The work.

Step 1, put the larger count on top and the smaller underneath.

$136{,}134{,}656 \div 44{,}067{,}840 = 3.0892064$

The calculator carries on, 3.08920645986 and onwards.

Step 2, round to four decimal places. Written out further, the answer is 3.08920645986. The four decimals you are keeping are 0, 8, 9 and 2, giving 3.0892. The next digit along is a 0, which is under 5, so 3.0892 stays as it is.

The answer. The vocabulary table is 3.0892 times the size of all of attention. The answer carries no units, because a count divided by a count leaves nothing behind.

Check it. Multiply back. $3.0892 \times 44{,}067{,}840 = 136{,}134{,}371$, which is 285 short of 136,134,656. That small gap is there because 3.0892 has been rounded to four decimal places, and multiplying a rounded number back will not land exactly. A rough check is cleaner: 44 million times 3 is 132 million, and the vocabulary table is 136 million, so the answer should be a little over 3. It is.

Where this goes wrong. The tempting wrong answer is 0.3237, from $44{,}067{,}840 \div 136{,}134{,}656$. It is tempting because the two counts are handed to you in the order “vocabulary table, then attention” and it feels natural to divide in the order you read them. An answer below 1 means you divided the smaller by the larger, and “how many times larger” can never come out below 1. If you did get 0.3237, it is a true statement standing on its head: all of attention is 0.3237 times the size of the vocabulary table.

17. What is asked. The FP32 size, in megabytes, of a model with 22,713,216 parameters.

What it is testing. Formula 3.5 with a different unit at the end. Definition 3.7 sets the unit: MB means 10⁶ bytes, which is one million bytes. The parameter count of all-MiniLM-L6-v2 is measured and recorded in _research/00-lab-verified-findings.md, section 6.

The work.

Step 1, bytes per weight. FP32 is 32 bits, and $32 \div 8 = 4$ bytes.

Step 2, multiply by 4 by doubling twice.

$22{,}713{,}216 \times 2 = 45{,}426{,}432$

$45{,}426{,}432 \times 2 = 90{,}852{,}864 \text{ bytes}$

Step 3, divide by one million to get megabytes.

$90{,}852{,}864 \div 1{,}000{,}000 = 90.852864$

Step 4, round to three decimal places. The digit after 90.852 is an 8, which is 5 or more, so the last kept digit rounds up.

$90.852864 \rightarrow 90.853$

The answer. 90,852,864 bytes, which is 90.853 MB.

Check it. Compare it with something already worked out. The chapter’s model holds $494{,}032{,}768 \div 22{,}713{,}216 = 21.75$ times as many parameters, and at the same four bytes each it came to 1,976,131,072 bytes, which is 1,976.131 MB. So this model should be about 21.75 times smaller, and $1{,}976.131 \div 21.75 = 90.857$ MB. That sits next to 90.853 MB, and the small gap is there because 21.75 is a rounded figure. Another way to state the same answer: 90,852,864 bytes is 0.090852864 GB, under a tenth of a gigabyte.

Where this goes wrong. The tempting wrong answer is 90.853 GB, or its mirror image 0.0909 MB. Both come from dividing by the wrong power of ten. MB is 10⁶, which is 1,000,000, and GB is 10⁹, which is 1,000,000,000. Spot it by asking whether the answer is believable: a 22.7 million parameter model taking up 90 gigabytes would be larger than every model in this course put together.

19. What is asked. How many standard deviations above the mean the largest weight sits, given a largest weight of 1.1719, a mean of -0.000017 and a standard deviation of 0.066741, to two decimal places.

What it is testing. The distance-in-standard-deviations formula from Section 3.4: $\text{distance in standard deviations} = \dfrac{w - \bar{w}}{s}$. All three measured values are in lab/out/ch03_parameter_anatomy.json.

The work. The brackets in the code line (weight_max - weight_mean) / weight_sd are there to force the subtraction to happen before the division. Do the top of the fraction first.

Step 1, subtract the mean from the weight. The mean is a negative number, and subtracting a negative number is the same as adding the positive version of it.

$1.1719 - (-0.000017) = 1.1719 + 0.000017 = 1.171917$

Step 2, divide that by the standard deviation.

$1.171917 \div 0.066741 = 17.559177$

Step 3, round to two decimal places. The digit after 17.55 is a 9, which is 5 or more, so the second decimal rounds up from 5 to 6.

$17.559177 \rightarrow 17.56$

The answer. The largest weight sits 17.56 standard deviations above the mean. The answer carries no units. It is a count of rulers, and the ruler is the standard deviation, 0.066741.

Check it. It matches the second line printed by Cell 7. Two tests on the formula itself: a weight sitting exactly at the mean gives 0, because the top of the fraction is 0, and a weight sitting one standard deviation above the mean gives exactly 1. So an ordinary member of this pile should come out near 1, and 17.56 says this one is nothing like ordinary.

Where this goes wrong. The risky slip is the minus sign in step 1. Write $1.1719 - 0.000017 = 1.171883$ instead and you get $1.171883 \div 0.066741 = 17.558667$, which still rounds to 17.56 at two decimal places. The wrong method hides behind the rounding here, and it will not hide when the mean is bigger. Build the habit now: subtracting a negative adds.

The other tempting wrong answer is 68,936.29, from dividing by the mean, 0.000017, instead of by the standard deviation, 0.066741. It is tempting because both numbers are small and sit beside each other in the printout. Spot it by size: an answer in the tens of thousands of standard deviations is not a description of a weight, it is a division by the wrong number.

Stretch¶

21. What is asked. Explain, in three or four sentences, why none of the bars in the Section 3.2 simulation move when the storage slider goes from FP32 to 4-bit, even though the model’s size falls from 1.976 GB to 0.247 GB.

What it is testing. The difference between a count of numbers and a count of bytes, which is Common mistake 5 in the chapter, and Formula 3.5.

What a strong answer contains. This one asks for a written argument, so the wording is yours. A strong answer has these four pieces, and the first two are the heart of it.

1. The bars measure how many parameters each part holds. That is a count of numbers.

2. The storage format decides how many bytes each number gets. That is a separate decision, made after the counting, and it changes nothing about the counting.

3. Nothing is added and nothing is thrown away when the format changes. The model still holds 494,032,768 parameters at every rung of the ladder.

4. The strongest answers go one step further. Even if the bars were redrawn in bytes instead of in parameters, the shape of the chart would still not change, because every part would shrink by the same factor. Four bytes down to half a byte is a factor of $4 \div 0.5 = 8$, and the whole model falls by that same factor: $1.976131072 \div 8 = 0.247016384$. Every bar shrinking by 8 leaves the picture identical.

One worked example of such an answer.

The bars in that chart are counts of parameters, and the slider does not change any count. It changes how many bytes each parameter is written down in, which is a different measurement of the same object. The model holds 494,032,768 numbers at FP32 and the same 494,032,768 numbers at 4 bits, so the vocabulary table is still 136,134,656 of them and all of attention is still 44,067,840 of them. What changes is the room each number gets, from four bytes down to half a byte, a factor of 8, which is why 1.976 GB becomes 0.247 GB while the picture stays where it is.

Where this goes wrong. The tempting wrong answer says the bars do not move because the change is too small to see. The change is not small: the file drops to an eighth of its size. The bars do not move because they are not measuring the thing that changed.

23. What is asked. Two sentences a careful reader should say back to the claim “7 billion parameters, fourteen times the size of the small one”. One sentence about what the count does and does not tell you, and one that uses something you measured in this chapter.

What a strong answer contains. This is a written argument and there is no single right wording. A strong answer has these pieces.

• It says that a parameter count is a count of numbers in a file. It is not a measure of how good the answers are, and it is not a file size until somebody says how many bytes each number gets.

• It notices that “fourteen times” is a claim you can check. Divide: $7{,}000{,}000{,}000 \div 494{,}032{,}768 = 14.1691$. As a ratio of counts, “fourteen times” is about right, and a careful reader says which quantity is fourteen times what.

• It uses a measured number from this chapter. Any of these works: the inventory at 27.56% vocabulary table, 63.51% MLP blocks and 8.92% attention; or the file size, since $7{,}000{,}000{,}000 \times 2 = 14{,}000{,}000{,}000$ bytes at FP16, which is 14 GB, against the 0.988 GB of the model in this chapter.

• It stops short of claiming the larger model is better. Nothing measured in this chapter says that.

One worked example of such an answer.

A parameter count tells you how many numbers are in the file and nothing about how good the replies are, and it is not a file size at all until you say how many bytes each number gets: at two bytes each, 7 billion parameters is 14,000,000,000 bytes, which is 14 GB, against the 0.988 GB of the model I have been counting. In that small model I measured 27.56% of the parameters sitting in a lookup table for tokens and 8.92% sitting in all of attention, so fourteen times as many numbers does not mean fourteen times as much of the part people write papers about.

Where this goes wrong. The tempting wrong reply is “so it must be fourteen times better”. Nothing in a count supports that. The other slip is agreeing that 7 billion parameters is “fourteen times the size” without asking size of what. Fourteen times the count of numbers and fourteen times the bytes on disk are two different claims, and they match only when both models use the same storage format.

25. What is asked. Explain why “attention is 8.92% of the parameters, so attention does 8.92% of the work” does not follow, without claiming to know how much of the work attention does.

What a strong answer contains. This is a written argument. A strong answer has these pieces.

• It names the two different things being measured. The 8.92% is a share of stored numbers, measured once, from the file on disk. “The work” is something that happens while the model runs, and this chapter measured nothing about that.

• It gives at least one reason the two can come apart. The clearest: a share of the file says nothing about how many times a number is used. A weight read once per token and a weight read many times take up the same room in the file.

• The strongest answers reach for a second reason. What attention costs depends on how long your text is, because attention looks back at earlier tokens, while the parameter count does not change with the length of the text at all. So the two quantities are not even measured against the same thing, and one of them moves when you change the prompt.

• It refuses to supply a replacement number. The honest ending is “this chapter did not measure that”, and a measurement of run-time cost would be a different experiment.

One worked example of such an answer.

The 8.92% is a share of the numbers stored in the file, counted once and fixed. Work happens when the model runs, and this chapter never measured that, so no number in the inventory can be carried across to it. The two can come apart because a parameter’s share of the file says nothing about how often that parameter gets used, and because what attention costs grows with how long your text is while the parameter count stays at 494,032,768 no matter what you type. I do not know what share of the work attention does, and neither does anyone reading it off the inventory.

Where this goes wrong. The tempting move is to answer a different question and say “attention actually does more of the work than 8.92%”. That may well be true, and this chapter gives you no way to know it, so saying it repeats the classmate’s mistake in the other direction. The problem asks you to reject the inference, not to replace the number.

27. What is asked. Explain what the 32,288 bytes are for, given that model.safetensors is 988,097,824 bytes while the weights alone come to 988,065,536 bytes, and why Cell 5 could not have reached layers[0].self_attn.q_proj.weight without them.

What it is testing. Reading the file census in Section 3.5. The byte counts are measured by lab/ch03_parameter_anatomy.py and stored in lab/out/ch03_parameter_anatomy.json, where the 32,288 figure has its own entry, safetensors_index_overhead_bytes.

The work. Start by confirming the two numbers the problem hands you.

Step 1, the weights alone, at two bytes each, because the published file is half precision.

$494{,}032{,}768 \times 2 = 988{,}065{,}536 \text{ bytes}$

Step 2, subtract that from the size of the file.

$988{,}097{,}824 - 988{,}065{,}536 = 32{,}288 \text{ bytes}$

Step 3, see how small that is as a share of the file, using Formula 3.2.

$32{,}288 \div 988{,}097{,}824 = 0.0000326769$

$0.0000326769 \times 100 = 0.0032677\%$

The answer. Those 32,288 bytes are the file’s index: a list written at the front of the file, naming every block of numbers in the model, saying what shape each block is, and saying where in the file its bytes start and stop. It is 0.0033% of the file, about three thousandths of one per cent.

Cell 5 asked for one block by name. Without the index, the file is 988,065,536 bytes in a row with no marks on it. There is nothing in the numbers themselves that says where one matrix ends and the next begins, or which of them is layers[0].self_attn.q_proj.weight, or that it is 896 rows by 896 columns rather than 802,816 numbers in a line. The index is what turns that run of bytes into named, findable objects. With it, the loader looks the name up, reads the start and stop positions, and goes straight there, which is why Cell 5 came back in a moment instead of reading 988 megabytes to find one matrix.

Check it. The 32,288 bytes cannot be parameters, because $988{,}065{,}536 = 494{,}032{,}768 \times 2$ exactly, with nothing left over. Every parameter is accounted for before those bytes are counted. That exact division is also the proof that the published file gives each number two bytes, not four.

Where this goes wrong. The tempting wrong answer is that the 32,288 bytes are 16,144 extra weights, from dividing 32,288 by the two bytes each weight gets. It is tempting because the rest of the file is nothing but weights. The count is fixed at 494,032,768 and it is already matched exactly by 988,065,536 bytes, so there is no room for extra weights. The second tempting answer is to mix these 32,288 bytes up with the 1,338 bytes from the folder census earlier in the same section. Those are different bytes doing a different job: the 32,288 sit inside model.safetensors and belong to the model file, while the 1,338 are bookkeeping files the download tool wrote beside the seven model files, and no chapter uses them.

Chapter 4. The probability of the next word¶

These problems drill one thing from three directions: turning a list of raw scores into a list of probabilities with softmax, checking that the result is legal, and reading what the shape of that result says about the model.

Two housekeeping notes before the solutions.

Rounding. Every solution keeps six decimal places through the middle of the work and rounds once at the end, which is what Toolkit 16 asks for. When six-place numbers are added back up, the total sometimes prints as 1.000001 or 0.999999 instead of 1.000000. That is the rounding, not an error, and each solution says so where it happens.

Where the numbers come from. Problems marked real are checked against lab/out/we2_softmax.json and lab/out/ch04_softmax_chapter.json. Problems marked made up for practice use invented scores, and their answers are also recorded in lab/out/ch04_softmax_chapter.json under by_hand_made_up_scores, so you can open the file and compare.

Warm-up: can you do the arithmetic?¶

1. Asked: work out $e^{3}$ to six decimal places.

Idea being tested: Formula 4.2, raising $e$ to a power. If the e^x key is new, Toolkit 7 says where to find it on a phone, on Windows Calculator, and in Google.

One press. Find the e^x key, enter 3.

If you want to see where that comes from rather than trust a key, $e^{3}$ means $e$ multiplied by itself three times. Do it in two steps.

$2.718282 \times 2.718282 = 7.389057$

$7.389057 \times 2.718282 = 20.085541$

That second route gives 20.085541, and the calculator gives 20.085537. The two disagree in the fifth decimal place because $e$ was rounded to 2.718282 before being multiplied three times, and each multiplication carries that rounding forward. The calculator answer is the one to report.

Answer. $e^{3} = 20.085537$. It is a plain number with no units.

Where students go wrong. The tempting wrong answer is $2.718282 \times 3 = 8.154846$. It is tempting because “e to the three” gets read at speed as “e times three”. It is mistake 2 in the chapter’s Common mistakes list. Test your key before you trust it: $e^{1}$ must print 2.718282 and $e^{0}$ must print exactly 1.

3. Asked: write down $e^{0}$ with no calculator, and say in one sentence why it has that value.

Idea being tested: exponents, from Toolkit 5, and the same fact the chapter uses in Worked Example 4.1.

Answer. $e^{0} = 1$.

The sentence. An exponent counts how many times you multiply, so an exponent of zero asks for no multiplications at all, and what is left is 1, the number every multiplication starts from.

If you would rather see it forced than asserted, Formula 4.6b does it in three lines. That rule says $e^{a} \div e^{b} = e^{a-b}$. Put $a = 1$ and $b = 1$.

Left-hand side: $2.718282 \div 2.718282 = 1.000000$, because any number divided by itself is 1.

Right-hand side: $1 - 1 = 0$, so the right-hand side is $e^{0}$.

The two sides are the same number, so $e^{0} = 1$.

Where students go wrong. The tempting wrong answer is 0. “To the power zero” sounds like “nothing there”, and nothing sounds like zero. It is not. Zero is what you get from $e \times 0$, which is a different operation. This one is worth checking on your calculator once, because it turns up in almost every softmax you will do by hand.

5. Asked: made up for practice. Three tokens have scores 1, 0 and 0. Find all three probabilities and check that they add to 1.

Idea being tested: Formula 4.4, softmax, in full. Two of the three scores are equal, which is the part worth noticing.

Step 1, raise $e$ to each score.

$e^{1} = 2.718282$

$e^{0} = 1.000000$

$e^{0} = 1.000000$

Step 2, add the three results. Add two at a time, left to right, so every line can be checked.

$2.718282 + 1.000000 = 3.718282$

$3.718282 + 1.000000 = 4.718282$

The total is $S = 4.718282$.

Step 3, divide each result by that total.

$p_1 = 2.718282 \div 4.718282 = 0.576117$

$p_2 = 1.000000 \div 4.718282 = 0.211942$

$p_3 = 1.000000 \div 4.718282 = 0.211942$

Check it.

$0.576117 + 0.211942 = 0.788059$

$0.788059 + 0.211942 = 1.000001$

That prints 1.000001, not 1.000000. Nothing is wrong. Each of the three was rounded to six decimal places before being added, and three rounded numbers do not have to add to a round total. Kept unrounded, the three come to exactly 1.

As percentages, using Toolkit 11: multiply each by 100.

$0.576117 \times 100 = 57.6117\%$

$0.211942 \times 100 = 21.1942\%$

$0.211942 \times 100 = 21.1942\%$

Answer. $p = (0.576117,\; 0.211942,\; 0.211942)$, which is 57.6117 per cent, 21.1942 per cent and 21.1942 per cent.

A second check worth doing. Formula 4.6 says the ratio of two probabilities is $e$ raised to the gap between their scores. The gap between token 1 and token 2 is $1 - 0 = 1$, so the ratio should be $e^{1} = 2.718282$. Divide: $0.576117 \div 0.211942 = 2.718277$. That agrees to four decimal places, and the fifth differs only because the two probabilities were rounded first.

Where students go wrong. The commonest slip is giving tokens 2 and 3 different probabilities, usually by dividing the first one properly and then “spreading the rest” by hand. Equal scores must give equal probabilities, because $e^{0} = e^{0}$ and both get divided by the same $S$. If your two zeros came out different, you did not divide the second one; you subtracted.

7. Asked: does the list $(0.5,\; 0.4,\; 0.2)$ pass both tests in Formula 4.1? If not, which test does it fail, and by how much?

Idea being tested: Formula 4.1, the two conditions a list must meet before anyone may call it a probability distribution. It is also in the formula summary, section 3.1.

Test one: is any of them below zero? 0.5 is above zero. 0.4 is above zero. 0.2 is above zero. Passed.

Test two: do they add to exactly 1?

$0.5 + 0.4 = 0.9$

$0.9 + 0.2 = 1.1$

The total is 1.1, not 1. Failed.

By how much. Subtract.

$1.1 - 1.0 = 0.1$

Turn that into percentage points by multiplying by 100.

$0.1 \times 100 = 10$

Answer. The list passes test one and fails test two. It is not a probability distribution. The total is 0.1 too big, which is 10 percentage points: the list is claiming 110 per cent of the certainty.

Where students go wrong. The tempting answer is “1.1 is close to 1, so it is near enough”. It is not, and the size of the gap is how you tell. Section 4.3 of Chapter 4 shows a real machine printing a total of 1.000062 instead of 1. That miss is 0.000062, which lives in the fifth decimal place, and it comes from 32-bit storage. A miss of 0.1 lives in the first decimal place and comes from a mistake. An error you can see without looking hard is yours.

9. Asked: write the probability 0.665241 as a percentage.

Idea being tested: converting a decimal to a percentage, from Toolkit 11. One multiplication.

The sign $\%$ means “out of a hundred”, so a decimal becomes a percentage when you multiply it by 100.

$0.665241 \times 100 = 66.5241$

Answer. 66.5241 per cent.

Where students go wrong. Two tempting wrong answers, and they are opposite mistakes. The first is $0.665241\%$, which leaves the number alone and adds a per cent sign to it; that would be a probability of about 1 in 150, not about two thirds. The second is $0.00665241\%$, which divides by 100 instead of multiplying, moving the decimal point the wrong way. The sanity check that catches both: a probability between 0.5 and 1 has to land between 50 per cent and 100 per cent. 66.5241 does. The other two do not.

Practice: can you apply it?¶

11. Asked: made up for practice. Run softmax on the scores 3, 2 and 1, compare with the scores 2, 1 and 0 from Worked Example 4.3, and say in one sentence why the answers came out the way they did.

Idea being tested: Formula 4.4, softmax, and then Formula 4.7, the rule that adding the same number to every score changes nothing.

Step 1, raise $e$ to each score.

$e^{3} = 20.085537$

$e^{2} = 7.389056$

$e^{1} = 2.718282$

Step 2, add them, two at a time.

$20.085537 + 7.389056 = 27.474593$

$27.474593 + 2.718282 = 30.192875$

The total is $S = 30.192875$.

Step 3, divide each by that total.

$p_1 = 20.085537 \div 30.192875 = 0.665241$

$p_2 = 7.389056 \div 30.192875 = 0.244728$

$p_3 = 2.718282 \div 30.192875 = 0.090031$

Check it.

$0.665241 + 0.244728 = 0.909969$

$0.909969 + 0.090031 = 1.000000$

Answer. $p = (0.665241,\; 0.244728,\; 0.090031)$, which is 66.5241 per cent, 24.4728 per cent and 9.0031 per cent. These are identical, to all six decimal places, to the answers for the scores 2, 1 and 0 in Worked Example 4.3. The same three values are recorded in lab/out/ch04_softmax_chapter.json under by_hand_made_up_scores.3_2_1, next to the ones for 2_1_0.

The one sentence. The list 3, 2, 1 is the list 2, 1, 0 with 1 added to every entry, and Formula 4.7 says adding the same number to every score leaves every probability exactly as it was.

Where students go wrong. The tempting expectation is that bigger scores give bigger probabilities. Every exponential did get bigger. So did the total, and by the same factor. Divide the two totals and you can watch it happen:

$30.192875 \div 11.107338 = 2.718282$

The new total is $e^{1} = 2.718282$ times the old one, which is exactly the factor every top grew by, so every fraction is unchanged. A logit on its own carries no information about how likely a word is. Only the gaps do.

13. Asked: made up for practice. Run softmax on the scores 4, 2 and 0, showing every step.

Idea being tested: Formula 4.4 again, with wider gaps between the scores. You have met these three scores before: they are the ones in Try it 4.3.

Step 1, raise $e$ to each score.

$e^{4} = 54.598150$

$e^{2} = 7.389056$

$e^{0} = 1.000000$

Step 2, add them, two at a time.

$54.598150 + 7.389056 = 61.987206$

$61.987206 + 1.000000 = 62.987206$

The total is $S = 62.987206$.

Step 3, divide each by that total.

$p_1 = 54.598150 \div 62.987206 = 0.866813$

$p_2 = 7.389056 \div 62.987206 = 0.117310$

$p_3 = 1.000000 \div 62.987206 = 0.015876$

Check it.

$0.866813 + 0.117310 = 0.984123$

$0.984123 + 0.015876 = 0.999999$

That prints 0.999999 rather than 1.000000 because all three were rounded to six places before being added. Kept unrounded they come to exactly 1.

Step 4, as percentages.

$0.866813 \times 100 = 86.6813\%$

$0.117310 \times 100 = 11.7310\%$

$0.015876 \times 100 = 1.5876\%$

Answer. $p = (0.866813,\; 0.117310,\; 0.015876)$, which is 86.6813 per cent, 11.7310 per cent and 1.5876 per cent. Recorded in lab/out/ch04_softmax_chapter.json under by_hand_made_up_scores.4_2_0.

A second check worth doing. Formula 4.6 says the ratio of the top two probabilities is $e$ raised to their score gap. The gap is $4 - 2 = 2$, so the ratio should be $e^{2} = 7.389056$. Divide: $0.866813 \div 0.117310 = 7.389080$. That agrees to four decimal places; the rest is six-place rounding.

Where students go wrong. Two slips. The first is reading $e^{4}$ as $e \times 4 = 10.873128$ instead of 54.598150. The second is being surprised that the leader took 86.6813 per cent here when it took only 66.5241 per cent on the scores 2, 1 and 0. The scores 4, 2, 0 have gaps of 2 where 2, 1, 0 had gaps of 1, and raising $e$ to a power stretches the top of a list away from the bottom. Doubling the gaps does not double the lead; it multiplies the ratios.

15. Asked: made up for practice. Four tokens all have the score 10. Find all four probabilities.

Idea being tested: symmetry, plus Formula 4.6, the ratio form. This one can be answered without exponentiating anything.

The short route, with no exponentials. All four scores are the same number. Formula 4.6 says the ratio of any two probabilities is $e$ raised to the gap between their two scores, and here every gap is $10 - 10 = 0$, and $e^{0} = 1$. A ratio of 1 means the two are equal. So all four probabilities are equal. Formula 4.1 says they must add to 1, and four equal numbers add to 1 only if each one is a quarter of 1.

$1 \div 4 = 0.25$

The long route, as a check. Raise $e$ to each score. All four are the same.

$e^{10} = 22026.465795$

Add the four. Adding the same number four times is multiplying it by 4.

$22026.465795 \times 4 = 88105.863180$

Divide any one of them by that total.

$22026.465795 \div 88105.863180 = 0.250000$

Check it. $0.25 \times 4 = 1.00$.

Answer. All four probabilities are 0.25, which is 25 per cent each.

Where students go wrong. The tempting move is to treat 10 as a “high” score and expect high probabilities. Scores have no fixed zero, which is the whole of Section 4.5 and Formula 4.7. Give the four tokens scores of 10, or 0, or -3, or 1000, and as long as all four are the same number the answer is 25 per cent each. The other slip is forgetting the divide and reporting 22026.465795 as a probability. A probability is never above 1.

17. Asked: real. The top eight tokens hold 72.056 per cent of the probability. How much is left, and across how many tokens is it spread? Then one sentence on what that means for reading a “top 5” display.

Idea being tested: Definition 4.9, probability mass, and Formula 4.1, which says the whole list adds to 1 and therefore to 100 per cent.

How much is left. The whole distribution is 100 per cent, so subtract.

$100 - 72.056 = 27.944$

Across how many tokens. The vocabulary is 151,936 tokens and eight of them are in the table, so subtract.

$151{,}936 - 8 = 151{,}928$

Answer. 27.944 per cent of the probability is left, spread across 151,928 tokens. Both are real, from lab/out/ch04_softmax_chapter.json, where the leftover is the field tail_mass and the vocabulary size is vocab_size.

How thin is that, per token? Divide.

$27.944 \div 151{,}928 = 0.000184$

An average token out of that tail carries about 0.000184 per cent, which is about one chance in 540,000.

The one sentence. A “top 5” display shows you the five tallest bars and says nothing at all about more than a quarter of the model’s belief, so the honest reading of that display is “the model gives Paris 30.219 per cent”, not “the model is choosing between these five”.

Where students go wrong. The tempting description of the leftover 27.944 per cent is “noise” or “rounding”. It is neither. Compare it with the model’s own second choice, ______, at 12.315 per cent:

$27.944 \div 12.315 = 2.2691$

The part of the distribution no display ever shows you holds more than twice as much probability as the model’s second-favourite token. It is thin per token and large in total, and those two facts are not in conflict.

19. Asked: a classmate raises one token’s score by 1 and leaves every other score in the vocabulary alone. Does that token’s probability go up, go down, or stay the same? One sentence, and name the formula that settles it.

Idea being tested: Formula 4.6, the ratio form, and the difference between changing one score and changing all of them.

Answer. It goes up.

The one sentence. Formula 4.6 says $\frac{p_i}{p_j} = e^{\,z_i - z_j}$, so raising $z_i$ by 1 while every other score stays put raises every one of those gaps by 1 and multiplies every one of those ratios by $e^{1} = 2.718282$; a token that has become 2.718282 times more likely than each of the other 151,935 tokens, in a list that still has to add to 1, must have gained probability.

A worked check, with numbers made up for practice. Take the scores 2, 1 and 0 from Worked Example 4.3, where the probabilities were 0.665241, 0.244728 and 0.090031. Raise the middle score by 1, so the scores become 2, 2 and 0, and leave the other two alone.

Step 1, raise $e$ to each new score.

$e^{2} = 7.389056$

$e^{2} = 7.389056$

$e^{0} = 1.000000$

Step 2, add them.

$7.389056 + 7.389056 = 14.778112$

$14.778112 + 1.000000 = 15.778112$

Step 3, divide each by that total.

$p_1 = 7.389056 \div 15.778112 = 0.468311$

$p_2 = 7.389056 \div 15.778112 = 0.468311$

$p_3 = 1.000000 \div 15.778112 = 0.063379$

Check: $0.468311 + 0.468311 = 0.936622$, and $0.936622 + 0.063379 = 1.000001$, which is 1 to within six-place rounding.

Token 2 went from 0.244728 to 0.468311. Up, and by a lot. The other two went down, from 0.665241 to 0.468311 and from 0.090031 to 0.063379, because the one unit of certainty has to come from somewhere.

A note on the wording. The problem asks you to “name the formula that settles it”. Formula 4.6 is the chapter’s own tool for comparing two tokens, and it is the answer being looked for. Formula 4.4, softmax itself, also settles it: the token’s own top grew by a factor of $e$ while the tops of all the others did not, so its share of the total had to rise. Either name, with the reasoning attached, is a complete answer.

Where students go wrong. The tempting wrong answer is “it stays the same, by Formula 4.7”. Formula 4.7 is about adding the same number $c$ to every score in the vocabulary. Here one score moved and 151,935 did not. That is not a shift, and Formula 4.7 has nothing to say about it. The two situations look similar on the page and behave completely differently, which is why the chapter states Formula 4.7 with “every” in bold.

Stretch: can you reason with it?¶

21. Asked: use Formula 4.6a, in the form $e^{z + c} = e^{z} \times e^{c}$, to show that adding the same number $c$ to every score leaves every softmax probability unchanged. Write it out for a vocabulary of three tokens, with every step visible.

Idea being tested: Formula 4.7, proved rather than asserted, out of Formula 4.6a. The letter $c$ stands for any fixed number you like, and it is the same number for all three tokens.

Setup. Three tokens with scores $z_1$, $z_2$, $z_3$. Their probabilities before any shift are

Now add $c$ to every score, giving $z_1 + c$, $z_2 + c$ and $z_3 + c$.

Step 1, write each new top using Formula 4.6a. That rule, read from right to left, turns an added exponent into a multiplication.

$e^{z_1 + c} = e^{z_1} \times e^{c}$

$e^{z_2 + c} = e^{z_2} \times e^{c}$

$e^{z_3 + c} = e^{z_3} \times e^{c}$

Step 2, build the new total. Add the three new tops.

Step 3, take the shared factor out. The same number $e^{c}$ multiplies all three terms, so it can be written once outside a bracket.

Step 4, write the new probability of token 1.

Step 5, cancel. There is a factor of $e^{c}$ on the top and the same factor of $e^{c}$ on the bottom. Formula 4.6b with $a = b = c$ says $e^{c} \div e^{c} = e^{c-c} = e^{0} = 1$, so that factor contributes a 1 and disappears.