Where is this running, and what did it cost?

Chapter 1. Where is this running, and what did it cost? Module A: a model is a measurable object.

What you need before this chapter¶

Not much, and everything on this list is written out somewhere you can reach in one click.

No programming. You have not been asked to write Python before this page and you are not asked to write it now. The code in this chapter is shown to you, line by line, with a comment on every line. Reading it is the assignment. Running it is optional this week.

No calculus, and no algebra you have to remember. This chapter uses adding, subtracting, multiplying and dividing. That is the whole list. Everything else is explained in place.

Five short pieces of notation, all of them in the Math Toolkit, which starts from nothing and works arithmetic you can check on a phone:

Five more, used lightly, and worth a glance if a number in this chapter looks strange: Toolkit 12, proportions, which is where “how many out of how many” is built up from nothing, Toolkit 16, rounding, Toolkit 17, scientific notation, Toolkit 18, powers of two, which is where the difference between a gigabyte and a gibibyte comes from, and Toolkit 19, inequality signs, which is where the symbol $\approx$, meaning “is about”, is explained.

A browser. That is the hardware floor. If you also have a laptop with about two gigabytes of free disk space, you can run everything yourself. If you do not, the chapter and the lab both have a route for you, and it is not a lesser route.

If a symbol in this chapter stops you, that is the symbol’s fault, and there is a page in the back that fixes it. This chapter will keep sending you there.

Setting up¶

Here is the whole toolkit for this chapter, in one block. Every line has a comment saying what it is for. You do not need to understand any of it yet. It is here so that nothing later in the chapter appears from nowhere.

import os                                      # for asking the operating system about files on disk
os.environ["HF_HOME"] = r"C:\math3219\models"   # the folder your models live in; MUST come before the transformers lines
import json                                    # for reading the .json result files this book ships
import time                                    # for a stopwatch, so we can time how long things take
import torch                                   # the numerical library the model's arithmetic runs on
from transformers import AutoTokenizer         # turns your text into numbers the model can read
from transformers import AutoModelForCausalLM  # loads the model itself, the pile of learned numbers

model_name = "Qwen/Qwen2.5-0.5B-Instruct"      # the smallest model in this course

Six imports, one setting and one name. The setting on line two tells the model library which folder on your disk the downloaded models are kept in, and it has to come before the two transformers lines, because that folder name is read once, at the moment those lines run. The last line does nothing except write down which model we mean, so that the rest of the chapter can say model_name instead of typing that string out again.

Two words in that block are technical, and both get proper definitions later in this chapter: tokenizer, the thing that cuts your text into pieces, and model, the pile of numbers that does the predicting.

A sentence about Bakersfield¶

Late one afternoon in September, on a laptop in a classroom on the CSU Bakersfield campus, a file finished downloading. It was 988,097,824 bytes, a little under one gigabyte, about the size of a long film at low quality. It took 2.54 seconds to read that file off the disk and hand it to the machine’s graphics card.

Then it was asked one question:

In one sentence, what is Bakersfield, California known for?

In 0.943 seconds it wrote back 27 pieces of text. Here is exactly what it said, copied out of lab/out/ch01_first_run.json with nothing changed:

Bakersfield, California is known as the “City of Butterflies” due to its diverse butterfly population and rich agricultural heritage.

Read that again, because two different things are true about it at once, and holding both in your head is the entire first week of this course.

The second half is fair. Kern County is one of the largest agricultural producers in the United States. Almonds, grapes, citrus, carrots, milk. If you grew up here, you did not need a computer to tell you that.

The first half is not true. Bakersfield is not called the City of Butterflies. There is no such nickname. You can check this yourself in about thirty seconds, and you should, because checking is the habit this course is built on. The city is widely associated with the Bakersfield Sound in country music, with oil, and with agriculture. Butterflies are not on the list.

Nobody lied to you. There is nothing inside that file that decided to be misleading. The file contains 494,032,768 numbers, and those numbers, run through some multiplication and addition, produced a sentence that sounds exactly like a true sentence. Fluent and wrong are not opposites. This chapter does not yet give you a name for what happened, or a system for sorting it. That comes much later, in Chapter 13, and it comes later deliberately: you are supposed to meet this before anyone hands you a word for it.

What you get today is smaller and more useful. Two questions, asked of every run, every week, from now until the capstone.

Question one: did it work? Not “was it impressive”. Not “did it sound confident”. Did the thing it said survive being checked.

Question two: what did it cost? That laptop was plugged into a wall. The graphics card drew measurable power the entire time it was writing about butterflies. On this machine, on a separate measured run, one piece of text out of this same model cost 0.767 joules. Twenty-seven pieces is about 20.7 joules. That is a real quantity of electricity, it came from somewhere, and by Week 15 you will be expected to put a number on it without being reminded.

Two questions. Both of them get an actual number. That is the course.

Learning objectives¶

By the end of this chapter you will be able to:

1. Describe what a language model physically is: a folder of files on a disk, most of it one file of learned numbers, and locate that folder on a machine.

2. Convert between parameters, bytes, gigabytes and gibibytes, and explain why a “0.5 billion parameter” model arrives as a file of about one gigabyte.

3. Distinguish running a model locally from sending your text to a company’s computer, and state what leaves your machine in each case.

4. Compute the two measuring sticks of this course for a real run: a score for whether it worked, and tokens per second and joules per token for what it cost.

5. Write a short honest report of a run, naming the machine, the units, and at least one thing the measurement does not cover.

This lesson at a glance¶

• A model is a file. One folder, seven files, 999,586,347 bytes, and 98.85% of that is a single file holding 494,032,768 learned numbers.

• It ran here. The prompt never left the laptop. Loading took 2.54 seconds, answering took 0.943 seconds, and both numbers are properties of that machine rather than of the model.

• Two measuring sticks. Did it work, scored as a fraction; what did it cost, measured in tokens per second and joules per token.

• One paragraph of honesty. Lab 0 asks you for one thing the model did that surprised you, quoted exactly, with the prompt that produced it.

The vocabulary of this chapter¶

Every technical word this chapter uses, defined before it is used. Read it once now. Come back whenever a word stops making sense; that is what it is for.

1.1 A model is a file on a disk¶

Intuition¶

Start with the least mystical possible description, because it is also the true one.

A language model is a folder on a hard drive. Inside the folder are a few files. One of them is enormous and the rest are small. The enormous one is a very long list of numbers. There is nothing else in there. No rules, no facts written in English, no little database of cities and their nicknames. Numbers.

An analogy that holds up better than most. Think of a very large printed table, like the interest tables that used to sit in the back of accounting books, or a tide table for the year. The table does not know anything. It is a grid of numbers. But if you know the procedure for reading it, look up this row, look up this column, multiply, the table gives you a useful answer. A model is that, scaled up until the table has 494,032,768 entries and the procedure for reading it takes a graphics card.

The word for one of those entries is parameter. People also say weights when they mean all of them together. They were decided once, by somebody else, on somebody else’s hardware, in a process called training that finished before you downloaded anything. Nothing in the file changes when you use it. You cannot wear it out. Asking it a question does not teach it anything, and asking it the same question tomorrow gets the same answer, provided nobody added randomness on purpose.

This matters for a practical reason and a political one.

The practical reason is that files have sizes, and sizes are facts you can check. Nobody has to take a vendor’s word for how big a model is. You right-click the file.

The political reason arrives in Week 7 and it is worth planting now. A file small enough to fit on an ordinary laptop is a file a public university can hand to every student. A file too big to fit is a file you have to rent access to. That is the whole access argument, and it turns out to be a question about bytes.

The mathematics¶

Four small pieces of arithmetic, each with everything spelled out. None of them is harder than multiplying and dividing. The point is not difficulty; it is that you should be able to check every size claim anyone makes about a model, including ours.

Here is the actual folder, read off the disk by lab/ch01_first_run.py and recorded in lab/out/ch01_first_run.json.

Seven files. One of them is almost all of the weight. That claim deserves arithmetic rather than the word “almost”, so here it is. Divide the big file by the total, then multiply the answer by 100 to turn the decimal into a percentage. Toolkit 11 explains that second step from scratch if it is not automatic. The sign $\div$ means “divided by”. It is the same instruction as the fraction bar you will meet in Formula 1.3, written on one line so that a step fits on a line. See Toolkit 4.

$988{,}097{,}824 \div 999{,}586{,}347 = 0.9885067\ldots$

The decimal carries on past six places, so round it there: 0.988507. Now multiply by 100.

$0.988507 \times 100 = 98.8507\%$

Round that to two decimal places and you have $\mathbf{98.85\%}$. Both roundings were done on purpose, and Toolkit 16 shows how rounding works if it is not automatic.

So about 98.85% of the folder is one file. That single fact is the chapter.

Formula 1.1: adding up a list of file sizes

In words. To find the total size of a folder, add up the sizes of the files inside it.

That sentence is the whole formula. The only reason to write it in symbols is that you are about to meet the symbol $\sum$ many times this semester, and meeting it for the first time attached to something this obvious is much kinder than meeting it attached to something hard.

The formula.

The symbols.

Out loud. “The total number of bytes is the sum, from file one to file $k$, of the size of file $i$.” In plainer English: “add up the size of every file.”

Worked, with the seven real files above. Add them one at a time, left to right, so you can check each line separately.

$659 + 242 = 901$

$901 + 1{,}671{,}839 = 1{,}672{,}740$

$1{,}672{,}740 + 7{,}031{,}645 = 8{,}704{,}385$

$8{,}704{,}385 + 7{,}305 = 8{,}711{,}690$

$8{,}711{,}690 + 2{,}776{,}833 = 11{,}488{,}523$

$11{,}488{,}523 + 988{,}097{,}824 = \mathbf{999{,}586{,}347}$ bytes

That final number is total_bytes_on_disk in lab/out/ch01_first_run.json. Open the file and look at it.

Check it. A total has to be bigger than the largest single item and smaller than the number of items multiplied by the largest item. The largest file is 988,097,824 bytes and there are seven files, so the total has to land between 988,097,824 and $7 \times 988{,}097{,}824$, which is 6,916,684,768. Our total, 999,586,347, sits comfortably inside. If your total came out smaller than the biggest file, you dropped a line.

Formula 1.2: the size of the weights, from the parameter count

In words. Count how many numbers the model has, and multiply by how many bytes it takes to store one of them. That is how many bytes the weights occupy.

The formula.

The symbols.

Out loud. “The size in bytes is the number of parameters, multiplied by the number of bytes each parameter takes up.”

Worked, for the real model, stored at FP16.

Step 1, write down the parameter count. It was measured, not guessed: $N = 494{,}032{,}768$. It is parameters in lab/out/ch01_first_run.json and params in lab/out/lab4_size_ladder.json, two separate scripts that agree.

Step 2, write down the bytes per weight. FP16 means 16 bits per parameter. There are 8 bits in a byte, so $16 \div 8 = 2$ bytes per parameter.

Step 3, multiply.

$494{,}032{,}768 \times 2 = \mathbf{988{,}065{,}536}$ bytes

Check it, and find something. Compare that with the real file: model.safetensors is 988,097,824 bytes. Those are not the same number. Subtract them.

$988{,}097{,}824 - 988{,}065{,}536 = \mathbf{32{,}288}$ bytes

The file is 32,288 bytes bigger than the numbers inside it. That gap is not an error. A .safetensors file starts with a short written index saying which block of numbers has which name and which shape, and that index, together with the marker giving its length, accounts for those 32,288 bytes. The script measured that index directly and recorded it as safetensors_header_bytes in lab/out/ch01_first_run.json, where it reads 32,288, so the gap we predicted and the header we measured are the same number.

How big a slice of the file is that? Divide the gap by the file, then multiply by 100 to get a percentage.

$32{,}288 \div 988{,}097{,}824 = 0.0000326769\ldots$

Round that to three significant figures and you get 0.0000327. Now multiply by 100.

$0.0000327 \times 100 = \mathbf{0.00327\%}$

Rounded to two significant figures that is 0.0033% of the file, which is three thousandths of one per cent. A number with that many leading zeros is easier to read written as $3.27 \times 10^{-5}$, and Toolkit 17 explains that shorthand from scratch.

This is worth pausing on, because it is the shape of a lot of honest measurement. The prediction was close but not exact. The gap was small. Instead of rounding the gap away, we looked at it, found a reason, and the reason was real. That is the move. Chapter 12 does the same move with statistics and it is much harder there.

Formula 1.3: bytes into gigabytes

In words. A gigabyte is a billion bytes. To turn a number of bytes into gigabytes, divide by a billion.

The formula.

The symbols.

Out loud. “The number of gigabytes is the number of bytes divided by one billion.”

Worked, for the weights.

$988{,}065{,}536 \div 1{,}000{,}000{,}000 = 0.988065536$ GB

Rounded for a table: 0.99 GB.

Worked again, for the whole folder.

$999{,}586{,}347 \div 1{,}000{,}000{,}000 = 0.999586347$ GB

The entire folder, tokenizer and all, is 0.9996 gigabytes. It misses being exactly one gigabyte by 413,653 bytes, which is a coincidence and a pleasing one.

Check it. Dividing by a billion moves the decimal point nine places to the left. Count the digits in 988,065,536: there are nine. So the answer must start with “0.” followed by those digits, and it does. If your answer came out as 988.065536 you divided by a million; if it came out as 0.000988 you divided by a trillion. Count the zeros.

Formula 1.4: bytes into gibibytes, and why anyone would

In words. Computer memory is sold in units of two multiplied by itself thirty times, not in units of a billion. That unit is called a gibibyte, and it is about 7% bigger than a gigabyte. To turn bytes into gibibytes, divide by 1,073,741,824.

The formula.

The symbols.

Out loud. “The number of gibibytes is the number of bytes divided by two to the thirtieth power.”

Worked, for the whole folder.

$999{,}586{,}347 \div 1{,}073{,}741{,}824 = 0.93093733\ldots$

Rounded to seven decimal places, that is $\mathbf{0.9309373}$ GiB.

So the same folder is 0.9996 GB and 0.9309 GiB. One folder, two correct numbers, because they are two different units.

Why you have to care. Graphics cards are advertised in “GB” and measured in GiB. A card sold as 4 GB actually holds

$4 \times 1{,}073{,}741{,}824 = 4{,}294{,}967{,}296$ bytes

which, in the decimal unit that model files are quoted in, is 4.295 GB.

Where does the 7% come from? Divide one gibibyte by one gigabyte, subtract 1 so that only the extra part is left, then multiply by 100 to make it a percentage.

$1{,}073{,}741{,}824 \div 1{,}000{,}000{,}000 = 1.073741824$

$1.073741824 - 1 = 0.073741824$

$0.073741824 \times 100 = 7.3741824\%$

Rounded to two decimal places, $\mathbf{7.37\%}$.

So the card has about 7% more room than the label suggests. That 7% decides real cases. Chapter 7 uses exactly this arithmetic to work out which of three models fits on a modest student laptop, and the answer would change if you mixed the units up.

Check it. A gibibyte is bigger than a gigabyte, so the same pile of bytes must come out as a smaller number of gibibytes. Our folder: 0.9996 GB against 0.9309 GiB. Smaller, as required. If your GiB number came out larger than your GB number, you multiplied where you should have divided.

Python¶

This is the code that produced the file table above. It is the first four lines of real work in the course, and every line has a comment.

# The folder the model was downloaded into. Yours will be somewhere else.
snapshot_folder = "lab/models/hub/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/7ae557604adf67be50417f59c2c2f167def9a775"

# os.listdir asks the operating system for the names of the files in a folder.
# sorted() puts those names in alphabetical order so the output is the same every time.
file_names = sorted(os.listdir(snapshot_folder))

# Start a running total at zero. We will add each file's size to it.
total_bytes_on_disk = 0

# Look at each file in turn. One trip round this loop per file.
for file_name in file_names:
    # Glue the folder name and the file name together into a full path.
    full_path = os.path.join(snapshot_folder, file_name)
    # os.stat asks the operating system about a file; .st_size is its size in bytes.
    size_in_bytes = os.stat(full_path).st_size
    # Add this file's size to the running total. This line is Formula 1.1.
    total_bytes_on_disk = total_bytes_on_disk + size_in_bytes
    # Print the name and the size, padded so the columns line up.
    print(f"{file_name:<28}{size_in_bytes:>14,} bytes")

# After the loop has finished, print the total we accumulated.
print(f"{'TOTAL':<28}{total_bytes_on_disk:>14,} bytes")
# Divide by a billion to get gigabytes. This line is Formula 1.3.
print(f"which is {total_bytes_on_disk / 1e9:.6f} GB")

The output, exactly as it was printed:

config.json                            659 bytes
generation_config.json                 242 bytes
merges.txt                       1,671,839 bytes
model.safetensors              988,097,824 bytes
tokenizer.json                   7,031,645 bytes
tokenizer_config.json                7,305 bytes
vocab.json                       2,776,833 bytes
TOTAL                          999,586,347 bytes
which is 0.999586 GB

Four things in that code are worth naming, because they come back every week.

The for loop. The lines indented under for file_name in file_names: run once for each file. Seven files, so seven trips. Each trip, file_name holds a different name. The indentation is not decoration; it is how Python knows which lines repeat.

The running total. total_bytes_on_disk starts at 0 and grows by one file’s size on each trip. The line total_bytes_on_disk = total_bytes_on_disk + size_in_bytes looks strange if you read = as “equals”. Read it as “becomes”: the total becomes what it was, plus this file. That line is Formula 1.1 with the sigma unrolled into steps.

The f before the quotes. It lets you drop a value into a piece of text. {size_in_bytes:>14,} means “put the number here, right-aligned in a column 14 wide, with commas between the thousands”.

Nothing here loaded the model. We asked the operating system about files. The model was not opened, no arithmetic was done, and no electricity worth measuring was spent. A model is a file, and you can weigh a file without opening it.

1.2 Local, or somebody else’s computer¶

Intuition¶

There are two ways to get an answer out of a language model, and they are different in a way that has nothing to do with the mathematics and everything to do with who is in the room.

Hosted. You type into a box in a browser. Your text travels over the internet to a building full of computers owned by a company. A model runs there. The answer travels back. The model is often very large, far too large for your laptop, and you never see the file. You are renting. What you typed is now on somebody else’s disk, subject to somebody else’s policy, for somebody else’s chosen length of time.

Local. The file is on your machine. When you press enter, the arithmetic happens on your own processor or your own graphics card. Nothing goes out. You could unplug the network cable and it would still work, because there is nothing to reach. Turn off the computer and the model is still a file, sitting there, doing nothing, costing nothing.

The measured runs in this book are all local, and this book is going to keep saying so, because almost every number in this course only means something once you know where the arithmetic happened.

Three consequences follow, and the course cares about all three.

Privacy. A local run cannot leak what you typed, because what you typed never moved. If a student wants to ask a model about a medical bill, a legal notice, or a piece of their own writing, local is not a preference. It is the only version of that action that is private.

Cost. Hosted models are usually billed per token, or bundled into a subscription. Local models are billed in electricity and in the hardware you already own. Both are real costs. Only one of them is a cost you can measure with a meter you control, which is why this course measures that one and says so.

Capability. This is the honest part. The hosted model is usually better. Frontier models are hundreds of times larger than the file on your laptop, and it shows. Our 0.5 billion parameter model invented a nickname for a city in the first sentence it wrote. This course is not going to pretend the small one is secretly better. The question worth asking is narrower and more useful: how much do you actually lose, measured, and is what you gain worth it? You will answer that with numbers in Week 7 and again in Week 11.

The mathematics¶

There is one piece of arithmetic in this section, and it is the one people use wrongly more than any other: comparing two numbers by dividing.

Here is the comparison worth making. The model weights are 988,065,536 bytes and they stay on your disk forever. The question you asked is 59 bytes and it is the only thing that would have travelled if this had been a hosted run. How different are those two sizes?

Formula 1.5: how many times as big

In words. To find out how many times bigger one thing is than another, divide the bigger one by the smaller one.

The formula.

The symbols.

Out loud. “R is A divided by B”, or in English, “A is R times as big as B.”

A ratio has no unit, and that is the point of it. Bytes divided by bytes leaves a bare number. “16.7 million times” is not 16.7 million of anything. It is a comparison.

Worked, with the two real byte counts. The weights are $A = 988{,}065{,}536$ bytes. The prompt is $B = 59$ bytes, recorded as prompt_bytes under text_sizes, then local, in lab/out/ch01_text_bytes.json.

Step 1, write the division down.

$R = 988{,}065{,}536 \div 59$

Step 2, divide.

$R = 16{,}746{,}873.4915\ldots$, which to two decimal places is $16{,}746{,}873.49$

Step 3, round it much harder, because no decision rests on the decimal.

$R \approx 16{,}700{,}000$, which is about 16.7 million.

Read that as a sentence. The model that stayed on your laptop is about 16.7 million times bigger than the question that would have left it.

Check it, two ways. First, direction. You divided the big number by the small one, so the answer must be bigger than 1. It is. If you get a number below 1, you divided upside down; flip it and try again.

Second, size. A rough estimate should land near the real answer. A billion divided by sixty is about 16 million, since $1{,}000{,}000{,}000 \div 60 \approx 16{,}666{,}667$. Our exact answer is 16,746,873. Close to the estimate, so no decimal point went missing. Doing the rough version first, in your head, is the single most effective way to catch an arithmetic slip, and it takes five seconds.

What the ratio is actually telling you. It is the shape of the whole hosted-versus-local question. In a hosted run, a tiny thing (your words) travels a long way to meet a huge thing (the model), and the huge thing never moves. In a local run, the huge thing was moved to you once, as a download, and after that nothing travels at all.

So the two arrangements have different cost profiles, not different amounts of cost:

Neither column is the right answer. Week 7 is where the course puts numbers in these boxes and makes you argue for one.

Python¶

Here is the code that measured the sizes of the text. No model is loaded. It reads the run that was already recorded and measures one more thing about it.

# Open the JSON file that ch01_first_run.py wrote, and read it into Python.
first_run = json.load(open("lab/out/ch01_first_run.json"))

# Pull out the record of the Bakersfield question.
local_run = first_run["runs"]["local"]

# The prompt, as a piece of text. Printing it confirms we have the right one.
prompt_text = local_run["prompt"]
print("prompt:", prompt_text)

# .encode("utf-8") turns text into the actual bytes a computer would store or send.
# len() counts them. UTF-8 is the encoding the whole web uses.
prompt_bytes = len(prompt_text.encode("utf-8"))
print("prompt bytes:", prompt_bytes)

# The same for what the model wrote back.
answer_text = local_run["text"]
answer_bytes = len(answer_text.encode("utf-8"))
print("answer bytes:", answer_bytes)

# The weights, in bytes, from the same file. This was measured, not assumed.
weight_bytes = first_run["weight_bytes_fp16"]
print("weight bytes:", weight_bytes)

# Formula 1.5: divide the big one by the small one.
ratio_weights_to_prompt = weight_bytes / prompt_bytes
print("the weights are", round(ratio_weights_to_prompt), "times the size of the prompt")

The output, exactly as it was printed:

prompt: In one sentence, what is Bakersfield, California known for?
prompt bytes: 59
answer bytes: 133
weight bytes: 988065536
the weights are 16746873 times the size of the prompt

Three things to take from that output.

The prompt is 59 bytes and it has 59 characters. Count them if you like; the spaces and the question mark count too. In UTF-8, every ordinary English character is one byte, so for plain English text the two numbers match. They stop matching the moment you use an accented letter or an emoji, which take two or more bytes each.

The answer is 133 bytes. Slightly longer than the question, as answers usually are, and still nothing next to a gigabyte.

16,746,873. The bare ratio. If you send a 59-byte question to a hosted model, 59 bytes of you leave your machine. If you run locally, 0 bytes leave. That is the entire privacy argument, and it is arithmetic rather than opinion.

1.3 The first measuring stick: did it work?¶

Intuition¶

You now have an answer from a model. The temptation, and it is a strong one, is to judge it on how it reads. Fluent, well-punctuated, confident, sounds like a person who knows. That is not a measurement. That is a reaction to a style.

Here is the uncomfortable fact underneath this whole course. A language model is trained to produce text that is likely, meaning text that looks like the text it was trained on. It is not trained to produce text that is true. Most of the time likely and true point the same way, because most sentences people write are about right. When they come apart, nothing in the machine notices, because nothing in the machine was ever checking.

Look again at the sentence from the opening.

Bakersfield, California is known as the “City of Butterflies” due to its diverse butterfly population and rich agricultural heritage.

Everything about the form of that sentence is correct. It is the right length. The grammar is clean. The structure, “known as X due to Y and Z”, is exactly how a person would write a sentence like this. A nickname in quotation marks is exactly what belongs in that slot. The model filled the slot with something that had the shape of a city nickname. Whether any human being has ever called Bakersfield that was not a question the machine was in a position to ask.

So “did it work” has to mean something you can check, and checking has to be something you do to the claims rather than to the prose. That turns out to be a small amount of work and a large amount of discipline.

The discipline is this. Before you look at the answer, decide what would count as right. After you look, check each claim separately, and write down how many survived. It is a slower way to read and it is the only way to get a number out of the end.

The mathematics¶

Break the sentence into claims that can be checked one at a time.

Two claims, one survives.

Formula 1.6: a score, as a proportion

In words. Count how many claims survived checking, divide by how many you checked, and that is the score.

The formula.

The symbols.

Out loud. “P hat is the number that checked out, divided by the number you checked.”

Worked, on the Bakersfield sentence.

Step 1, write down the two counts. One claim survived, so $x = 1$. Two were checked, so $n = 2$.

Step 2, divide.

$\hat{p} = 1 \div 2 = 0.5$

Step 3, turn it into a percentage by multiplying by 100. See Toolkit 11 if that step is not automatic.

$0.5 \times 100 = \mathbf{50\%}$

Worked again, on the second prompt. The same model was asked, in the same session:

What is 17 times 24? Answer with just the number.

It replied, in full: 408. That is correct. $17 \times 24 = 408$, and you can check it by hand: $17 \times 24 = 17 \times 20 + 17 \times 4 = 340 + 68 = 408$.

Step 1, $x = 1$ claim survived, $n = 1$ claim checked.

Step 2, $\hat{p} = 1 \div 1 = 1.0$, which is $\mathbf{100\%}$.

Check it. A proportion must land between 0 and 1, which is between 0% and 100%. If you get a number above 1, you divided the wrong way round: put the count of survivors on top. If you get a negative number, something has gone badly wrong, because you cannot check a negative number of claims.

Now the honest part, which matters more than the arithmetic. You have two scores: 50% on one prompt and 100% on another. Do not write those down as a verdict on the model. They rest on one claim and two claims. A single question tells you almost nothing about how a model will do on the next question, in the same way that one coin flip tells you almost nothing about the coin.

This is not a caution to be polite about. It is measurable, and this course measures it. In Chapter 12 a model scores 25.0% on a bank of twenty questions, and the honest interval around that number runs from 6.0% to 44.0%. That interval gets written $(6.0\%, 44.0\%)$, and Toolkit 19 unpacks the brackets. Twenty questions is already far too few to settle anything. Two claims is not in the same postcode as enough.

What the 50% is good for is starting the habit: name your procedure, count your cases, report a fraction, and say how many cases it rests on. Chapter 11 scales it to twenty questions, and Chapter 12 tells you how much to trust the result.

Notice one more thing before moving on, because it will surprise people who have been told otherwise. The model got the arithmetic right and the fact wrong. A common claim is that these models “cannot do maths”. This one did $17 \times 24$ correctly on the first try. Chapter 2 will show you the mechanical reason arithmetic is hard for them, which is that every digit becomes its own token, and Chapter 2 will also show you that “hard” does not mean “always fails”. Measure it rather than assume it. That is the rule.

Python¶

Here is how the second prompt was run and recorded. The first block of the cell opens the model, because this is the first time in the chapter that we need it in memory rather than on the disk. Section 1.4 comes back to that opening and puts a stopwatch on it.

# Ask torch whether this machine has an NVIDIA graphics card it can use.
# "cuda" means the graphics card; "cpu" means the main processor, which works but is slower.
if torch.cuda.is_available():
    device_name = "cuda"
else:
    device_name = "cpu"

# Load the tokenizer: the rules for cutting text into tokens.
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Load the model itself, at FP16, and move it onto the graphics card if there is one.
# .eval() switches it into answering mode rather than training mode.
model = AutoModelForCausalLM.from_pretrained(model_name, dtype=torch.float16)
model = model.to(device_name)
model = model.eval()

# The question, written out as text.
arithmetic_prompt = "What is 17 times 24? Answer with just the number."

# An instruction-following model expects its input wrapped in a particular way,
# with markers saying "here is the user speaking". This builds that wrapper.
messages = [{"role": "user", "content": arithmetic_prompt}]
chat_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# The tokenizer cuts the text into tokens and gives each token its id number.
# return_tensors="pt" asks for the result in the format torch wants.
# .to(device_name) puts those numbers wherever the model is: the graphics card if
# there is one, the main processor if there is not. device_name was set at the top.
input_ids = tokenizer(chat_text, return_tensors="pt").input_ids.to(device_name)

# Start the stopwatch.
generate_start_time = time.time()

# Generate. do_sample=False is greedy decoding: no randomness, same answer every run.
# max_new_tokens=60 stops it after 60 tokens even if it wants to keep going.
with torch.no_grad():
    output_ids = model.generate(input_ids, max_new_tokens=60, do_sample=False,
                                pad_token_id=tokenizer.eos_token_id)

# Stop the stopwatch and work out how long it took.
generate_seconds = time.time() - generate_start_time

# The output contains the prompt as well as the answer. Count the prompt tokens,
# then keep only the tokens that come after them.
prompt_token_count = input_ids.shape[1]
generated_ids = output_ids[0][prompt_token_count:]
generated_token_count = len(generated_ids)

# Turn the token ids back into readable text.
generated_text = tokenizer.decode(generated_ids, skip_special_tokens=True)

print("tokens generated:", generated_token_count)
print("seconds:", round(generate_seconds, 3))
print("the model said:", repr(generated_text))

The output, exactly as it was printed:

tokens generated: 4
seconds: 0.141
the model said: '408'

Two details in that output repay a second look.

Four tokens produced three characters. The answer 408 is three characters long, and the model emitted four tokens to say it. Tokens are not characters and they are not words. Chapter 2 is about nothing else.

do_sample=False is why this is reproducible. With that setting the model takes its single best guess at every step, which is called greedy decoding. Run the script again tomorrow on the same machine and you get 408 again. Turn sampling on and you are rolling dice, and two runs can disagree. Chapter 5 is where the dice come out on purpose. Until then, everything in this book is greedy, so that when a number changes you know the change came from the thing you changed.

The middle line is the exception. 0.141 seconds is a stopwatch reading on one laptop with one graphics card, and your own run will print something else. The token count and the text are the reproducible parts. Section 1.4 is about exactly that difference.

1.4 The second measuring stick, part one: time¶

Intuition¶

“What did it cost” has an easy half and a hard half. The easy half is time, and time is where this course starts counting, because a stopwatch is a meter everybody already owns.

There are two different times and confusing them wastes a lot of people’s afternoons.

Load time is how long it takes to get the model off the disk and ready to work. You pay it once, when you start. On the run in this chapter it was 2.54 seconds. It is mostly a fact about your disk: how fast it can read a gigabyte.

Generation time is how long the model takes to write an answer once it has started. You pay it every single time you ask a question. On the Bakersfield question it was 0.943 seconds for 27 tokens.

Quoting generation time on its own is not useful, because it depends on how long the answer was. A model that takes four seconds to write a paragraph is faster than one that takes two seconds to write a sentence. So the honest measure is a rate: how many tokens per second. That number can be compared across answers, across models, and across machines.

You already understand rates. Miles per hour is a rate. It is not how far you went, and it is not how long you took. It is one divided by the other, and the reason we use it is that it lets you compare a short drive with a long one.

One caution up front, because it is the mistake this section exists to prevent. Tokens per second is a fact about your machine at least as much as about the model. The same model on a gaming desktop, on a five-year-old laptop, and on a phone gives three different numbers, all correct. A speed with no machine named beside it is not a result.

The mathematics¶

Formula 1.7: tokens per second

In words. Count how many tokens came out, count how many seconds it took, and divide the first by the second.

The formula.

The symbols.

Out loud. “The rate is the number of tokens produced, divided by the number of seconds it took.”

Worked, on the Bakersfield run. From lab/out/ch01_first_run.json: 27 tokens in 0.9426095 seconds.

Step 1, write down the two numbers. $n_{\text{tokens}} = 27$ and $t = 0.9426095$.

Step 2, divide.

$r = 27 \div 0.9426095 = 28.6438869\ldots$

Step 3, round to two decimal places for reporting.

$r \approx \mathbf{28.64}$ tokens per second

Worked again, on the arithmetic run. 4 tokens in 0.1405993 seconds.

Step 1, $n_{\text{tokens}} = 4$ and $t = 0.1405993$.

Step 2, $4 \div 0.1405993 = 28.4496437\ldots$

Step 3, $r \approx \mathbf{28.45}$ tokens per second.

Check it, and then look at what the check reveals. Dividing by a number smaller than 1 makes the answer bigger than what you started with. That is why 27 divided by 0.94 came out as 28.6 rather than something smaller than 27. People find this counter-intuitive and it is worth saying out loud: dividing by 0.9426 is the same as asking “how many lots of 0.9426 seconds fit in one second, multiplied by 27”, and slightly more than one lot fits.

If you divided upside down you would get $0.9426095 \div 27 = 0.0349114\ldots$, which is seconds per token. That is also a real and useful quantity, but it is a different one, and a number near 0.03 when you expected a number near 30 is the sign you flipped the fraction.

Now the reveal. The two runs gave 28.64 and 28.45 tokens per second. Same model, same machine, same session, minutes apart, and the answers differ. Neither is wrong. A computer is doing many things at once, the second run was much shorter so the timing has less room to average out, and hardware speeds itself up and slows itself down constantly.

Here is a third measurement of the same quantity. The energy experiment, lab/out/theme_s_energy.json, timed this same model generating 85 tokens in 3.0677 seconds:

$85 \div 3.0677 = 27.70805\ldots$, which rounds to 27.708 tokens per second

Three measurements of one thing: 28.64, 28.45, 27.71. They cluster around 28 and they are not equal. A single measurement of a rate is an estimate with wobble in it, and this book prints the wobble instead of averaging it into a tidier-looking sentence. The whole of Chapter 12 is about what to do with that wobble.

The same disagreement shows up in load time, and more sharply. This chapter’s run recorded 2.54 seconds to load the model. A separate script, lab/lab4_size_ladder.py, loaded the same model from the same disk and recorded 3.93 seconds. Divide them. Use the unrounded seconds the two JSON files actually hold, 3.9297843 and 2.5354536, rather than the two-decimal versions printed above, because a ratio built from rounded inputs drifts.

$3.9297843 \div 2.5354536 = 1.5499334\ldots$

Turn that into a percentage the same way as the gibibyte gap in Formula 1.4. Subtract 1, so that only the extra part is left, then multiply by 100.

$1.5499334 - 1 = 0.5499334$

$0.5499334 \times 100 = 54.99334\%$

Rounded to the nearest whole percent, $\mathbf{55\%}$.

Here is why the unrounded inputs were worth the extra digits. Divide the rounded seconds instead and you get $3.93 \div 2.54 = 1.5472440\ldots$, which is 54.7% rather than 55.0%. The difference is small and it is real, and it is the whole reason this book rounds at the end of a calculation rather than at the start.

One run took 55% longer than the other, for the same model and the same file. The most likely reason is the operating system’s disk cache: the second time you read a file, part of it is already in memory and does not have to come off the disk. Neither number is a lie. They are two measurements of a quantity that genuinely varies, and the reason you are being shown both is that the alternative, showing you one and calling it “the load time”, would be dishonest.

The rule this establishes, and it holds all semester: report the run you did, name the machine, and if you have two numbers for the same thing, print both.

Python¶

First, the load itself, timed. This is where the 2.54 seconds came from. The stopwatch goes either side of the lines that read the weights off the disk.

# Ask torch whether this machine has an NVIDIA graphics card it can use.
# "cuda" means the graphics card; "cpu" means the main processor, which works but is slower.
# Written as a plain if and else, so you can read it top to bottom.
if torch.cuda.is_available():
    device_name = "cuda"
else:
    device_name = "cpu"

# Start the stopwatch before anything is read off the disk.
load_start_time = time.time()

# Load the tokenizer: the rules for cutting text into tokens. It is small and quick.
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Load the model itself. This is the line that reads 988,097,824 bytes off the disk.
# dtype=torch.float16 says to keep each parameter at FP16, two bytes each.
model = AutoModelForCausalLM.from_pretrained(model_name, dtype=torch.float16)

# Move the loaded numbers onto the graphics card, if there is one.
model = model.to(device_name)

# .eval() switches the model into answering mode rather than training mode.
model = model.eval()

# Stop the stopwatch and subtract to get the elapsed seconds.
load_seconds = time.time() - load_start_time

# Count the parameters by adding up how many numbers are in each piece of the model.
# An explicit loop, so you can see the addition happening one piece at a time.
parameter_count = 0
for one_parameter in model.parameters():
    parameter_count = parameter_count + one_parameter.numel()

print("device:", device_name)
print("load time:", round(load_seconds, 2), "seconds")
print("parameters:", f"{parameter_count:,}")

The output, exactly as it was printed:

device: cuda
load time: 2.54 seconds
parameters: 494,032,768

The 2.54 seconds is a cold load on the machine described in the callout above: one laptop, one NVIDIA RTX 3500 Ada GPU, one run. Section 1.3 already opened this model, so on your own machine the operating system will still be holding the file in memory and your number will be smaller. That is the point of Definition 1.3: a time is a fact about a machine, not about a model. The device and the parameter count, though, are facts you should be able to reproduce.

Two lines deserve attention. time.time() returns the number of seconds since a fixed date in 1970, which sounds strange until you notice that the only thing done with it here is subtracting one reading from another, and the fixed date cancels. That is the whole stopwatch. And the parameter count was counted, not read off a label: the loop walks through every piece of the loaded model adding up how many numbers are in it, and arrives at 494,032,768.

Now the timings, pulled out of the recorded run so you can see the two rates computed side by side.

# Read the recorded run back in.
first_run = json.load(open("lab/out/ch01_first_run.json"))

# The load time, in seconds, measured with the same stopwatch.
load_seconds = first_run["load_seconds"]
print("load time:", round(load_seconds, 2), "seconds")

# The Bakersfield run: how many tokens, and how long.
local_tokens = first_run["runs"]["local"]["generated_tokens"]
local_seconds = first_run["runs"]["local"]["seconds"]
# Formula 1.7: tokens divided by seconds.
local_rate = local_tokens / local_seconds
print("local run:", local_tokens, "tokens in", round(local_seconds, 3), "s")
print("local rate:", round(local_rate, 2), "tokens per second")

# The arithmetic run: the same two numbers, and the same division.
# This is written out again on purpose rather than tucked into a function,
# because seeing the same three lines twice is how the pattern sticks.
arithmetic_tokens = first_run["runs"]["arithmetic"]["generated_tokens"]
arithmetic_seconds = first_run["runs"]["arithmetic"]["seconds"]
arithmetic_rate = arithmetic_tokens / arithmetic_seconds
print("arithmetic run:", arithmetic_tokens, "tokens in", round(arithmetic_seconds, 3), "s")
print("arithmetic rate:", round(arithmetic_rate, 2), "tokens per second")

The output, exactly as it was printed:

load time: 2.54 seconds
local run: 27 tokens in 0.943 s
local rate: 28.64 tokens per second
arithmetic run: 4 tokens in 0.141 s
arithmetic rate: 28.45 tokens per second

Two remarks about the code, and one about what it does not do.

The repetition is deliberate. Lines 11 to 13 and lines 18 to 20 are the same three lines with different names. A professional programmer would fold them into a function. This book does not, because the aim here is for you to see the identical shape twice, not to write the shortest program. Repetition is a teaching tool, and this course uses it on purpose.

round(local_rate, 2) rounds to two decimal places. The underlying value is 28.64388552078351. Nothing about this run justifies fourteen digits, and printing them would suggest a precision the measurement does not have. See Toolkit 16.

Nothing here measures energy. A stopwatch tells you how long the card was working. It cannot tell you how hard. For that you need a different instrument, and it is the next section.

1.5 The second measuring stick, part two: energy¶

Intuition¶

The laptop was plugged into the wall. While the model was writing about butterflies, current was flowing, the graphics card was warming up, and the fan was working harder. That is not a metaphor. It is measurable, and this section measures it.

Two words have to be separated first, because they get used interchangeably in ordinary speech and they are not the same thing.

Power is a rate, measured in watts. It is how hard something is drawing right now. A bright household LED bulb is around 10 watts. A laptop charger might be 65 watts.

Energy is an amount, measured in joules. It is power multiplied by how long. One watt drawn for one second is one joule. Leave that 10 watt bulb on for a minute and it has used $10 \times 60 = 600$ joules.

The distinction matters because power tells you nothing on its own. A card drawing 30 watts for half a second and a card drawing 15 watts for a full second used the same energy. What you are charged for, and what the planet is charged for, is energy.

Now, the honest shape of the measurement. The instrument here is a sensor inside the graphics card, read through something called NVML, which is the same sensor the tool nvidia-smi reports from. It reports board power: the whole card, its chip and its memory and its electronics. The script reads it about fifty times a second while the model is generating and takes the average.

That sensor sees the graphics card. It does not see the processor, the memory sticks, the screen, the fans, or the energy lost in the charger converting wall current. All of those were also running. So every energy figure in this book is a lower bound on what a local run costs. It is not a total. Saying so is not modesty; it is the difference between a measurement and a claim.

One more thing, and it is the one that surprises people. A graphics card that is switched on and doing nothing still draws power. On this machine, with nothing running, it drew 13.834 watts. That background draw gets charged to whatever happens to be running, which means a fast small model looks worse than it deserves unless you account for it. The next formula but one does exactly that.

The mathematics¶

Two formulas. The first gives the cost of a token. The second gives the cost of choosing to run the model, which is a different question with a different answer.

Two words for summing up a pile of readings are about to do a lot of work, so here they are. The mean is the ordinary average: add every reading up, then divide by how many readings there were. The median is the middle reading once you have lined them all up from smallest to largest. One freak reading pulls a mean towards it and leaves a median where it was.

Both formulas use the same measured run: the 0.5B model generating 85 tokens in 3.0677 seconds while the card drew a mean of 21.254 watts, with idle measured at 13.834 watts. Every one of those figures is in lab/out/theme_s_energy.json.

Formula 1.8: energy per token

In words. Multiply the average power by the number of seconds to get the total energy, then divide by how many tokens came out. The answer is what one token cost.

The formula.

The symbols.

Out loud. “The energy per token is the average power in watts, multiplied by the number of seconds, divided by the number of tokens produced.”

Worked, step by step.

Step 1, multiply power by time. Watts are joules per second, so watts multiplied by seconds leaves joules.

$21.254 \times 3.0677 = 65.2008958$ joules

Rounded to four decimal places, 65.2009 joules.

Step 2, divide by the token count.

$65.2009 \div 85 = 0.76706941\ldots$

Rounded to four decimal places, $\mathbf{0.7671}$ joules per token.

That is j_per_token in lab/out/theme_s_energy.json, recorded there as 0.7670694643464218. Our four-digit answer matches it. Note what our two starting numbers were: 21.254 watts and 3.0677 seconds are themselves the file’s longer figures rounded, so the last digit of our answer carries that rounding with it. Here it lands on the same four digits anyway.

Check it. The answer has to be positive, and it has to be roughly power divided by speed. The card drew about 21 watts and produced about 28 tokens each second, and $21 \div 28 = 0.75$, which is close to 0.7671. If your answer is out by a factor of a thousand, you probably mixed watts with milliwatts or seconds with milliseconds, which is the most common slip with this formula.

Formula 1.9: energy per token above idle

In words. Take away the power the card would have drawn anyway, then do the same calculation. The answer is the extra energy the model caused, rather than the cost of having the machine switched on.

The formula.

The symbols.

Out loud. “The energy above idle is the average power minus the idle power, multiplied by the number of seconds, divided by the number of tokens.”

Worked, step by step.

Step 1, subtract inside the brackets first.

$21.254 - 13.834 = 7.420$ watts

Step 2, multiply by the time.

$7.420 \times 3.0677 = 22.762334$ joules

Rounded to four decimal places, 22.7623 joules.

Step 3, divide by the token count.

$22.7623 \div 85 = 0.26779176\ldots$

Rounded to four decimal places, $\mathbf{0.2678}$ joules per token above idle.

That matches j_per_token_above_idle in the JSON file, 0.2677937560020182, to four decimal places.

Step 4, compare the two answers, using Formula 1.5.

$0.2678 \div 0.7671 = 0.34910702\ldots$, which rounds to 0.3491

Step 5, multiply by 100 to turn that decimal into a percentage. See Toolkit 11 if that step is not automatic.

$0.3491 \times 100 = \mathbf{34.91\%}$

So about 35% of what a token cost was the model doing arithmetic. Whatever is left over out of 100% was the graphics card being switched on at all, so subtract.

$100 - 34.91 = \mathbf{65.09\%}$

That is about 65%. Two thirds of the bill is the light being on in an empty room.

Check it. The above-idle answer must always be smaller than the total, because you subtracted something positive before dividing. If it is not, check that you did the subtraction inside the brackets before multiplying. If it comes out negative, your measurement window included time before the model started working, so the average power dropped below idle.

Worked Example 1.1: what the Bakersfield answer cost

Put the two measuring sticks together on one run, which is the thing the course asks you to do every week from now on.

Read the honesty note before the arithmetic. This calculation joins two separate measured runs. The token count, 27, is from lab/out/ch01_first_run.json. The per-token energy, 0.7671 joules, is from lab/out/theme_s_energy.json, which measured the same model on the same machine but on a different prompt on a different occasion. No meter was attached to the Bakersfield run itself. So what follows is an estimate built from two measurements, not a third measurement, and it is labelled that way wherever it appears.

Step 1, write down what you have.

• Tokens produced: $n_{\text{tokens}} = 27$

• Energy per token, measured separately: $E_{\text{token}} = 0.7671$ joules

Step 2, multiply. Energy per token, multiplied by tokens, gives energy.

$27 \times 0.7671 = 20.7117$ joules

Rounded to two decimal places, $\mathbf{20.71}$ joules.

Step 3, make 20.71 joules mean something, using a second measured number from the same file. The idle card draws 13.834 watts, which is 13.834 joules every second. How many seconds of an idle graphics card does 20.71 joules buy?

$20.71 \div 13.834 = 1.4970362\ldots$

Rounded to two decimal places, $\mathbf{1.50}$ seconds.

The result, stated honestly in one sentence. Writing that 27-token sentence about Bakersfield cost about 20.7 joules of graphics-card energy, which is about the same as leaving the same card switched on and doing nothing for 1.50 seconds, and this figure covers the graphics card only.

Check it. The answer should be larger than the per-token figure and smaller than the per-token figure multiplied by a hundred, because 27 sits between 1 and 100. It should also be roughly “about three quarters of a joule, twenty-seven times”, and three quarters of 27 is about 20. It is.

Why this number is worth nothing on its own, and everything in a series. Twenty point seven joules is a tiny amount of energy. You would not notice it on a bill. Every honest version of this course has to say so rather than dress it up.

It becomes interesting in three ways, and each one is a later week:

1. Multiplied. One answer is nothing. A campus of students, several times a day, for a semester, is a quantity. Week 14 does that arithmetic and does it carefully.

2. Compared. The 3B model in this course costs 1.941 joules per token, which is 2.53 times the 0.5B. Same question, same machine, different file. Week 7 puts that ratio against what the bigger model buys you in accuracy.

3. Bounded. This figure is graphics card only. A hosted model adds the building it lives in. Week 7 names what that adds and refuses to guess a number for it.

Python¶

This is the core of lab/theme_s_energy.py, reduced to the lines that matter. It is a separate script rather than a continuation of this chapter, so it carries its own three imports at the top. It runs in two halves. The first half reads the sensor. Running that half needs an NVIDIA card, and Lab 0 has routes for people who do not have one.

# pynvml is the Python door into the graphics card's own sensors.
# It is the same source the command-line tool nvidia-smi reads from.
import pynvml                          # the library that reads the graphics card's sensors
import statistics                      # for median() and mean()
import time                            # for the short pause between readings

pynvml.nvmlInit()                      # wake up the measurement library
gpu_handle = pynvml.nvmlDeviceGetHandleByIndex(0)   # ask for graphics card number 0

# STEP 1: the idle baseline, taken with nothing else running.
# Fifty readings, 0.02 seconds apart, so one second of sampling in all.
# We take the MEDIAN, not the mean: one spike from a background program
# moves a mean and does not move a median.
idle_samples = []
idle_reading_number = 0
while idle_reading_number < 50:
    idle_samples.append(pynvml.nvmlDeviceGetPowerUsage(gpu_handle) / 1000.0)   # mW to W
    time.sleep(0.02)                   # 0.02 seconds is 20 ms, so about 50 readings a second
    idle_reading_number = idle_reading_number + 1

idle_watts_on_this_machine = statistics.median(idle_samples)
print("readings taken:", len(idle_samples))
print("median idle power on this machine:", round(idle_watts_on_this_machine, 3), "W")

The reading count is fixed, so the first line is the same for everybody. The second line is a fact about one graphics card at one moment, so yours will be a different number. On the laptop this book was written on, with the model already loaded and nothing else asking for the card, it printed this:

readings taken: 50
median idle power on this machine: 13.834 W

The second half is the arithmetic, and it is the same for everybody, because it runs on the numbers that were written down rather than on the card in front of you. The real script samples the card in the background while the model generates, which needs machinery this chapter does not need you to read. What it wrote down is in the JSON file.

# The recorded run, read back off the disk.
energy_records = json.load(open("lab/out/theme_s_energy.json"))
small_model_record = energy_records["Qwen/Qwen2.5-0.5B-Instruct"]

# The median idle watts, from step 1 of the real run.
idle_watts = small_model_record["idle_w"]
# The MEAN watts over the generation window. The mean is right here, because we
# want the average over a window where power rises and falls.
mean_watts = small_model_record["mean_w"]
# The stopwatch reading and the token count, taken around the same generate() call.
elapsed_seconds = small_model_record["seconds"]
tokens_generated = small_model_record["tokens"]

# STEP 3: the arithmetic. These three lines are Formulas 1.8 and 1.9.
total_joules = mean_watts * elapsed_seconds                       # watts x seconds = joules
joules_per_token = total_joules / tokens_generated                # Formula 1.8
above_idle_joules_per_token = (mean_watts - idle_watts) * elapsed_seconds / tokens_generated

print("idle power:", round(idle_watts, 3), "W")
print("mean power:", round(mean_watts, 3), "W")
print("joules per token:", round(joules_per_token, 4))
print("joules per token above idle:", round(above_idle_joules_per_token, 4))

The output, matching lab/out/theme_s_energy.json for the 0.5B model:

idle power: 13.834 W
mean power: 21.254 W
joules per token: 0.7671
joules per token above idle: 0.2678

Four decisions inside that code are the actual lesson, and every one of them would change the answer if you got it wrong.

Median for idle, mean for generation. These are different on purpose. The idle baseline should not be moved by one background process waking up, so a median is the safer summary. The generation figure genuinely is an average over a window where power rises and falls, so a mean is correct there. Choosing a summary statistic is a decision, not a default.

Milliwatts divided by 1000. The sensor reports milliwatts. Forget that division and every energy figure you report is a thousand times too big. This is the single most common error in this measurement and it is the reason the sanity check in Formula 1.8 is written the way it is.

Twenty millisecond sampling. Fifty readings a second. Sample too slowly and you miss the peaks; sample faster and the measuring itself starts costing energy.

A warm-up that is not shown here. The real script generates eight tokens and throws them away before the sampler starts, because the first generation after a model loads also pays for one-time setup inside the graphics card. Skip the warm-up and every model looks worse than it is.

The thread that runs to Week 15¶

One short section, planting one idea, so that it is not a surprise in Week 7.

Twenty point seven joules for a sentence about butterflies. Nobody’s electricity bill changes. So why does a mathematics course spend a semester on it?

Because of what happens when you put the number next to a different number. Here are the three models this course uses, with two measured quantities each. Parameter counts and file sizes come from lab/out/lab4_size_ladder.json; energy comes from lab/out/theme_s_energy.json.

Read the table twice, once for each column, and notice that you are reading the same story.

Read as energy, it says that the bigger model costs more for every token it writes. That is an environmental fact and it is measurable.

Read as bytes, it says the bigger model needs more memory to run at all. A graphics card sold as 4 GB holds 4,294,967,296 bytes. The 1.5B model’s weights are 3,087,428,608 bytes and they fit. The 3B model’s weights are 6,171,877,376 bytes and they do not. That is an access fact, and it is also measurable, in the same table, on the same day.

These are not two separate findings that happen to agree. They are one variable, parameter count, seen twice. A model small enough to be cheap to run is a model cheap enough to be widely available. The environmental argument and the access argument are the same argument in different units, and this course is built the way it is because both halves can be computed instead of asserted.

Worked Example 1.2: which of these fits on a 4 GB laptop graphics card?

This is the access half of the argument, done as arithmetic rather than asserted. It uses Formula 1.2 for the byte counts and Formula 1.4 for the card’s true capacity.

Step 1, find out how much room the card actually has. A card advertised as “4 GB” holds 4 gibibytes, not 4 gigabytes.

$4 \times 1{,}073{,}741{,}824 = 4{,}294{,}967{,}296$ bytes

Step 2, write down each model’s weights in bytes, using Formula 1.2 with 2 bytes per parameter.

$494{,}032{,}768 \times 2 = 988{,}065{,}536$ bytes

$1{,}543{,}714{,}304 \times 2 = 3{,}087{,}428{,}608$ bytes

$3{,}085{,}938{,}688 \times 2 = 6{,}171{,}877{,}376$ bytes

Step 3, for each one, divide the weights by the card’s capacity to get the share of the card it would use. This is Formula 1.5 again, and multiplying by 100 turns it into a percentage, which is the step Toolkit 11 builds from scratch. Each division below is rounded to four decimal places before it is multiplied, and each percentage is then rounded to one decimal place.

$988{,}065{,}536 \div 4{,}294{,}967{,}296 = 0.2300519\ldots$, and $0.2300 \times 100 = 23.00\%$, so $23.0\%$ of the card

$3{,}087{,}428{,}608 \div 4{,}294{,}967{,}296 = 0.7188479\ldots$, and $0.7188 \times 100 = 71.88\%$, so $71.9\%$ of the card

$6{,}171{,}877{,}376 \div 4{,}294{,}967{,}296 = 1.4370021\ldots$, and $1.4370 \times 100 = 143.70\%$, so $143.7\%$ of the card

Step 4, read the answers. Anything above 100% does not fit.

Check it. A share above 1, which is above 100%, means the thing is bigger than the container. The 3B model’s weights are 6.17 GB and the card holds 4.29 GB, so a share above 100% is what you should have expected before dividing. If your 3B share came out below 100%, you most likely divided the card by the model instead of the model by the card.

The caveat that has to travel with this table. These are the weights only. Running a model also needs room for its working memory, which grows as the conversation gets longer. So 71.9% is a floor for the 1.5B, not a ceiling, and a model at 95% of a card on paper will not run in practice. Chapter 7 puts numbers on the difference.

What the table means. A line runs between the 1.5B and the 3B, and it is drawn by a hardware budget rather than by anything about language. On one side of that line is a model a student can run on a laptop they already own. On the other side is a model they have to rent. That line is what Theme S of this course is about, and you computed where it falls in four steps of division.

That is the claim. It is not proved yet, and nothing in this chapter proves it. Week 7 measures what shrinking a model costs you in quality. Week 14 works out who pays. Week 15 asks you to do the whole accounting yourself for a model you chose.

Today all you owe it is the habit: every time you run something, write down what it cost.

Lab 0: first run and cost baseline¶

60 points, 6% of the course grade. Due at the start of Week 2.

Lab 0 has one purpose: to get a model running on a machine you can reach, and to produce the first honest measurement of the semester. It is graded on completeness and honesty rather than on whether your numbers are impressive. A slow machine loses no marks. A missing unit does.

Pick a route¶

All three routes get full marks. Choose on what you have, not on what you think looks better.

Route A, the full local stack. Python, transformers, and the 0.5B model downloaded to your own disk. The Python reference on installing the tools walks through it. Budget an hour, most of it downloading.

Route B, the no-install local route. Ollama runs a model from a single command with no Python environment to build. You still get a local run, a real file on your own disk, and real timings. You will not get the full probability distribution this way, which matters from Week 4, and by then you will have had time to set up Route A.

Route C, the measured-data route. If you do not have a machine that can run a model, or you do not have one you are allowed to install software on, you work from the recorded runs in lab/out/. You do the same arithmetic on real measured numbers, and you write the same report. Nobody is required to buy hardware for this course.

What to hand in¶

Five things, in one document, in this order.

1. Evidence it ran (10 points). A screenshot or a copied terminal transcript showing the model producing text. Include the prompt you used. Route C hands in the JSON file and names the script that wrote it.

2. The model as files (10 points). A table of the files in your model folder with their sizes in bytes, a total, and that total converted to gigabytes using Formula 1.3. Then check your parameter count against your weight file using Formula 1.2 and say what the gap is, as Section 1.1 does.

3. Did it work (15 points). Two prompts of your own choosing. For each: the exact prompt, the exact output quoted with nothing tidied, your split of the output into checkable claims, which survived, and the score $\hat{p}$ from Formula 1.6. Write your splitting rule down before you score. Marks are for the procedure being stated, not for the model doing well.

4. What did it cost (15 points). For each prompt: tokens generated, seconds taken, and tokens per second from Formula 1.7. Then your energy figure, and this is where the honesty lives:

• If you have an NVIDIA card, use Formula 1.8 and Formula 1.9, and report both columns.

• If you have a wall-plug energy meter, report whole-system power, and write one sentence saying it is a larger boundary than a graphics-card figure and is not comparable to ours.

• If you have neither, report the two legs that need no meter, speed and memory, and say plainly which quantity you could not measure.

Every figure needs a unit and a named machine. Finish with one sentence naming something your measurement does not cover. That sentence is worth marks on its own.

5. What surprised you (10 points). One paragraph. Not two.

Describe one thing the model did that you did not expect. Quote the exact prompt and the exact output. Say what you expected instead, and say why you think the two came apart. You are not being asked to diagnose it correctly; the course has not given you the tools yet, and it will not until Week 13. You are being asked to notice it precisely and record it exactly.

A note on Theme S¶

Lab 0 is one of the assignments that carries this course’s sustainability and justice thread, and it does so through Part 4. Producing a cost figure, naming its units, and stating what it leaves out is the smallest possible version of the accounting you will be asked for in the capstone. It is small on purpose. It is not optional.

Common mistakes¶

Seven things that go wrong in this chapter’s arithmetic, with the sign that each one has happened.

1. Reading GB when the number is GiB. A card advertised as 4 GB holds 4,294,967,296 bytes, which is 4.295 GB in the decimal unit that model files use. The sign: a model that “should not fit” fits, or a fit you were sure about fails by a few percent. The fix: write the unit next to every size, every time, and convert once at the start rather than in the middle.

2. Dividing tokens per second upside down. The sign: you expected a number near 30 and got 0.035. You computed seconds per token. Both quantities are real, and each one is 1 divided by the other, which is what the word “reciprocal” means. Divide tokens by seconds for a rate.

3. Forgetting that milliwatts are not watts. The sensor reports milliwatts. Forget the division by 1000 and your energy figure is a thousand times too large. The sign: joules per token in the hundreds. A token costs less than a joule on this hardware, not more than a hundred.

4. Reporting an energy figure without saying which column. Total and above idle differ by a factor of nearly three on the small model. The sign: two classmates report different numbers for the same model and neither can say why. The fix: name the column and say what question it answers.

5. Treating one run as a benchmark. This chapter has three measurements of one model’s speed, 27.71, 28.45 and 28.64 tokens per second, and two measurements of one load time, 2.54 and 3.93 seconds. The sign: a sentence containing “the load time is” with no qualifier. One run is a measurement. Several runs, reported with their spread, start to be a benchmark. Chapter 12 makes this precise.

6. Confusing the parameter count with the file size. The model has 494,032,768 parameters and the weight file is 988,065,536 bytes. They differ by a factor of exactly 2 because each parameter takes 2 bytes at FP16. The sign: a claimed size of about half a gigabyte for this model, or about 2 GB. The fix: always say which format the parameters are stored in.

7. Judging an answer on how it reads. The Bakersfield sentence is well-written, correctly punctuated, confident, and half false. The sign: a report that says an output was “good” without naming a single claim that was checked. The fix: score the claims, not the prose.

What to remember¶

A model is a folder of files on a disk, almost all of it one file holding 494,032,768 learned numbers, which is 988,065,536 bytes at two bytes each. Running it locally means that arithmetic happens on your machine and your words never leave it; running it hosted means they do. Every run gets measured twice, once for whether its claims survive checking and once for what it cost in time and in joules. Every measured number needs a unit, a named machine, and a sentence saying what the measurement does not cover. Fluent and true are different properties, and only one of them is something the machine was ever trying to produce.

Practice problems¶

Answers to the odd-numbered problems are in the answers appendix.

Every number you need is in this chapter or in the JSON files it names. Where a problem invents numbers so that the arithmetic stays checkable by hand, it says made up for practice.

Warm-up: can you do the arithmetic?¶

1. A model has 1,543,714,304 parameters and is stored at FP16, which is 2 bytes per parameter. Use Formula 1.2 to find its size in bytes.

2. Convert your answer to problem 1 into gigabytes using Formula 1.3.

3. A folder contains four files of 512, 2,048, 65,536 and 990,000,000 bytes. Use Formula 1.1 to find the total, adding one line at a time. Made up for practice.

4. Convert 6,171,877,376 bytes into gigabytes, and then into gibibytes. State which of the two numbers you would use to decide whether the file fits on a card advertised as 8 GB, and why.

5. A model produces 64 tokens in 4.0 seconds. Use Formula 1.7 to find its rate in tokens per second. Made up for practice.

6. The same model produces 15 tokens in 0.5 seconds. Find that rate too, and say whether the two rates agree. Made up for practice.

7. A graphics card draws a mean of 25.0 watts for 8.0 seconds while producing 100 tokens. Use Formula 1.8 to find the joules per token. Made up for practice.

8. For the run in problem 7, idle power is 12.0 watts. Use Formula 1.9 to find the joules per token above idle. Made up for practice.

9. A model answers a question with a sentence containing 5 checkable claims, of which 3 survive checking. Use Formula 1.6 to find $\hat{p}$, as a decimal and as a percentage. Made up for practice.

10. Using Formula 1.5, how many times bigger is 6,171,877,376 bytes than 988,065,536 bytes?

Practice: can you apply it?¶

11. Open lab/out/ch01_first_run.json. Find safetensors_header_bytes. Express it as a percentage of model.safetensors, and say in one sentence what that percentage tells you about whether the header is worth worrying about.

12. The 1.5B model has 1,543,714,304 parameters. If it were stored at 4 bits per parameter instead of 16, how many bytes would the weights take? Give your answer in GB, and state the one assumption you had to make.

13. Using the table in the thread that runs to Week 15, work out how many times more energy per token the 1.5B model costs than the 0.5B. Then do the same for the 3B against the 1.5B. Which step up the ladder is the more expensive one, per unit of model?

14. A classmate reports “my model ran at 45 tokens per second”. List three pieces of information you would need before that number could be compared with the 28.64 in this chapter. Made up for practice.

15. The idle draw on the machine in this book is 13.834 watts. How many joules does that card use in one minute of doing nothing? Compare it with the 20.71 joules that the Bakersfield answer cost, and say what the comparison suggests about where the energy goes on a lightly used laptop.

16. A hosted service charges per token. A local run charges in electricity. Write two sentences naming one cost that appears in each arrangement and not in the other. Neither sentence should contain a number, because you have not measured either.

17. Score this output, made up for practice: “The Kern River flows through Bakersfield and is the longest river in California.” Split it into claims, mark each, and report $\hat{p}$ with your splitting rule stated first.

18. The chapter reports 2.54 seconds and 3.93 seconds as two load times for the same model from the same disk. Suppose you had to publish one number. Write the sentence you would publish, including whatever qualification you think it needs.

19. A student writes: “The model is 0.5 billion parameters, so the file is 0.5 GB.” Identify the error, state the correct size at FP16, and name the piece of information the student left out.

20. Using Formula 1.8 and the 1.5B row of lab/out/theme_s_energy.json (120 tokens in 4.9619 seconds at a mean of 27.409 watts), compute the joules per token and check your answer against j_per_token in the file.

Stretch: can you reason with it?¶

21. On the 0.5B model, 65% of the energy per token was the card merely being switched on. On the 3B model the same share was 46%. Explain, in your own words and without formulas, why the faster model is the one that spends the larger share of its energy budget on being switched on rather than on doing arithmetic.

22. A vendor advertises a model as “70% smaller, with only a 5% drop in quality”. Made up for practice, because no vendor measured on our machine said this. Using only what this chapter gives you, write down the three questions you would have to ask before that sentence could be checked. For each question, name the quantity that would answer it and the unit it would be measured in.

23. This chapter’s energy figures cover the graphics card only. Suppose you also had a wall-plug meter measuring the whole laptop. Would the joules-per-token figure go up, go down, or stay the same? Explain your reasoning, and say whether the two figures could fairly be put in the same table.

24. The Bakersfield sentence was half true. Suppose a model produced a sentence that was entirely true but entirely useless, such as “Bakersfield is a city in California.” Design a scoring procedure that separates true-and-useful from true-and-empty. State your procedure precisely enough that a classmate applying it to the same sentence would get your score.

25. The course claims the environmental argument and the access argument are “the same variable seen twice”. Using the table in the thread that runs to Week 15, state the variable, and then describe one situation in which the two arguments would point at different models. What would have to be true of the hardware or the task for that to happen?

26. You are asked to compare two models fairly on one laptop. List every quantity you would hold fixed and every quantity you would let vary, and say for each one why. Then name the one thing on your fixed list that you think would be hardest to actually hold fixed, and why.