Numbers all the way down: tokens

Chapter 2, Numbers all the way down: tokens¶

What you need before this chapter¶

This is an honest list. If any line on it makes you uneasy, follow the link, read that short section, then come back. Nothing here takes long.

From this book.

• Chapter 1, Where is this running, and what did it cost?. You have run a model once and written down what the run cost. This chapter opens up the first thing that happens inside that run.

From the Math Toolkit. Six short sections, and you will use all six in this chapter.

Three sections are useful but not required. Section 10, sigma notation appears once, in Section 2.4, and the text writes the sum out in full beside it so you can follow either version. Section 16, rounding and decimal places explains why this chapter prints 3.6667 rather than 3.67. Section 18, powers of two sits behind one remark in Section 2.2 about why 151,936 is a convenient number of rows. That remark explains itself where it appears, so the toolkit section is there only if you want more.

What you do not need. You do not need to have written Python before. You do not need to know what a language model is made of; that is Chapter 3. You do not need algebra beyond “a letter can stand for a number”, and the toolkit link above starts from there.

Setup¶

Run this block once, at the top of your session, before anything else in this chapter. Every line has a comment saying what it is for. Nothing here downloads a whole model. A tokenizer is a small file, a few megabytes, and it loads in about a second even on a laptop with no graphics card.

# Cell 1. The setup for Chapter 2. Run this once per session.

import os                                      # lets Python read and change settings on your computer
os.environ["HF_HOME"] = r"C:\math3219\models"  # where downloaded files are kept; MUST come before the next line

from transformers import AutoTokenizer         # the tool that turns text into numbers, and numbers back into text

# The name of the model whose tokenizer we are borrowing. This is the smallest model in the
# course. Writing the name down once, here, means every later cell can say MODEL_NAME instead
# of repeating a long string.
MODEL_NAME = "Qwen/Qwen2.5-0.5B-Instruct"

# Fetch the tokenizer the first time you run this, then read it from your own disk afterwards.
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

print("Tokenizer ready for:", MODEL_NAME)

What that prints:

Tokenizer ready for: Qwen/Qwen2.5-0.5B-Instruct

If you are on a Mac or on Linux, change the folder in the second line to something that exists on your machine, such as /Users/yourname/math3219/models. The r in front of the Windows path tells Python to read the backslashes as backslashes. If that line fails, the Python reference has the fix for every operating system.

Bakersfield is three pieces¶

Type your city into a language model and it does not see your city.

The word Bakersfield has eleven letters and one meaning. A person reading it thinks of a place: the Kern River, the 99, the smell of the fields off Highway 58 in July. A language model does not receive eleven letters, and it certainly does not receive a place. It receives this:

[33, 8312, 2566]

Three whole numbers. That is the entire input. The model has no access to the letters, no access to the spelling, and no access to the fact that those three numbers came from one word. Turn each number back into the piece of text it stands for and you get 'B', then 'akers', then 'field'. The city gets cut into three pieces because the list of pieces the model knows about was built by counting which chunks of text show up often in a very large pile of writing, and the chunk Bakersfield did not show up often enough to earn a place on that list. The chunk akers did, and it has its own row, id 8312 (measured). It is not a word in any language. It is a run of letters that a counting procedure found worth keeping.

This is worth sitting with before the mathematics starts, because almost everything strange about these systems begins here. A model that has never seen the word Bakersfield as one thing is a model that has to rebuild the idea of your city out of three fragments, every single time.

Now compare some other places, all cut by the same tokenizer on the same machine (measured, from lab/out/we1_tokens.json).

Read the last row, then look back up at Sacramento. Fresno is six characters against Sacramento’s ten, so it is the shorter name, and it costs more tokens rather than fewer. Three against two. Length is not what is being measured here, and neither is importance. Sacramento is the state capital and it is cut in two. What is being measured is how often each run of letters turned up in a pile of text that nobody in Kern County chose, and the answer got written into a file and frozen. By the end of this chapter you will have opened that file, counted its rows, worked out what those rows cost, and printed the numbers your own name turns into.

Then the same machinery will do something worse, and the measurement will be even plainer. Hand the model the number 1234567 and it receives seven separate tokens, one per digit. Not a quantity. Seven symbols in a row. Everything people find puzzling about language models and arithmetic starts at that line, and it is not a puzzle at all once you have seen the count.

Learning objectives¶

By the end of this chapter you will be able to:

1. Explain, without notation, why text must become numbers before a language model can do anything with it, and say what a token, a token id and a vocabulary are.

2. Read and use the formula for tokenizing a string, $\text{tokenize}(s) = (t_1, t_2, \ldots, t_n)$, including saying every symbol in it out loud. Like every formula in this book, it arrives in six parts, and you will use all six of them.

3. Compute characters per token for any piece of text by hand, and use it to compare how efficiently a tokenizer handles two different strings.

4. Compute the size of a vocabulary table and its share of a whole model, and explain from that arithmetic why a vocabulary size of 151,936 is a decision somebody made rather than a fact about language.

5. Give the mechanical account of why arithmetic is hard for these models, anchored to the measured result that 1234567 becomes seven tokens, and say which half of that account is a measurement and which half is a hypothesis still waiting for a test.

This lesson at a glance¶

• Four ideas. Text becomes a list of whole numbers; those numbers point into a fixed table; the length of the list is what you are billed and charged energy for; and digits go one token each.

• Six formulas, each given in six parts: a plain sentence, the formula, a table defining every symbol, how to read it aloud, arithmetic with every step shown, and a way to check your own answer.

• One simulation and nine code cells. Nothing in this chapter loads a model’s weights, so everything runs on any laptop in under a minute.

• One number to leave with. 1234567 is seven tokens. It is a measurement, it takes four lines of code to reproduce, and it is where every honest account of these models and arithmetic has to start.

The vocabulary of this chapter¶

Every term this chapter uses is collected here, in one line each, before the numbered sections begin. Three of them, token, tokenizer and vocabulary, already turned up in the opening pages; this is where they get pinned down. Come back to this table whenever a word stops meaning anything. Each term is defined again, more carefully, at the point where it does its work.

The simulation for this chapter¶

Press each button. The chips are tokens, the small number on top of a chip is its position in the text, and the number underneath is its id. Watch what happens when you go from cat to cats, and watch the token count when you press 1234567.

You can come back to this at any point in the chapter. Everything the simulation shows is measured, and the same numbers are printed in the sections below.

2.1 Text has to become numbers first¶

Intuition¶

A computer stores numbers. That is not a limitation somebody decided to impose; it is what the hardware is. Underneath every photograph, every song and every document on your phone there is a long line of numbers, and there is nothing else. When you open a text message, a piece of software is turning numbers into shapes on glass for you. The numbers were always there.

So a language model, which lives on that hardware, cannot receive letters. Something has to happen first. The text has to be converted into numbers, and the conversion has to be reversible, because the model’s answer comes back the same way and has to be turned into letters again for you to read it.

Here is the everyday version of the same problem. Think of a very large phrasebook, printed once and never revised. Every entry has a number and a chunk of text: entry 4616 is cat, entry 2566 is field, entry 33 is the single capital letter B. To send a message you look up each chunk of what you want to say and write down its entry number. To read a message you look the numbers back up. The phrasebook is fixed. If the chunk you want is not in it, you have to spell your meaning out of smaller chunks that are.

That is precisely what happens to Bakersfield. The phrasebook has no entry for the whole city, so the text gets built out of three entries that do exist: 33, 8312 and 2566. Nothing has been lost, because the three chunks glued back together spell the word exactly. But something has been changed. The model’s view of your city is now a sequence of three unrelated entries, and the model must work out for itself, from having read a great deal of text, that this particular sequence tends to be about a place in California.

The cutting program is called a tokenizer, and the pieces it produces are called tokens. The whole process runs before the model is involved at all. It is not intelligent, it has no opinions, and it produces exactly the same answer every single time you give it the same text. That last property is worth holding on to. In a subject full of things that change from run to run, tokenization does not.

The mathematics¶

Now the formula. It arrives in the six parts every formula in this book arrives in, and the six parts are described on the Math Toolkit if you want the shape in advance.

Formula 2.1, tokenizing a piece of text¶

In words. Tokenizing takes one piece of text and gives back a list of whole numbers. The list is in the same order as the text, and its length is however many pieces the text broke into.

The formula.

The symbols.

Out loud. “Tokenize of ess equals the list t sub one, t sub two, and so on, up to t sub en.” In full English: “cutting up the text ess gives you a list of $n$ whole numbers, in order.”

Worked, with the measured result for Bakersfield.

Step 1. Write down the text. $s =$ Bakersfield.

Step 2. The tokenizer cuts it into pieces. For this model the pieces are 'B', 'akers' and 'field'. There are three of them, so $n = 3$.

Step 3. Look up each piece and write down its id.

$t_1 = 33$, because that is the row for 'B'

$t_2 = 8312$, because that is the row for 'akers'

$t_3 = 2566$, because that is the row for 'field'

Step 4. Put them in the list, in the order the pieces appeared.

Check it. There is one check and it is complete. Turn every id back into its piece, glue the pieces together in order, and you must get the text you started with, character for character.

The plus sign on that line is not addition. Sitting between two pieces of text it means “write the second piece straight after the first, with nothing in between”. It is the one place in this book where $+$ means something other than adding two numbers together, and Cell 3 below uses it in exactly this way.

Eleven characters in, eleven characters out. If your rebuilt text is missing a space, you dropped a token or you glued the pieces in the wrong order. If it has an extra space, you inserted one that was not there; spaces live inside tokens in this tokenizer, and you never add your own.

Python¶

Now watch it happen. Two short cells. The first one tokenizes, the second one checks.

The first cell does three things: it hands the text to the tokenizer, it turns each id back into its piece with an explicit loop so you can see every step, and it prints the five facts worth knowing.

# Cell 2. Tokenize one word and look at everything that comes out.

# A piece of text in Python is written inside quotation marks and is called a string.
city_name = "Bakersfield"

# Hand the string to the tokenizer. The part called .input_ids is the list of whole numbers.
bakersfield_ids = tokenizer(city_name).input_ids

# Turn each id back into the piece of text it stands for, one id at a time.
# An empty list first, to collect the pieces into.
bakersfield_pieces = []
for one_id in bakersfield_ids:
    one_piece = tokenizer.decode([one_id])   # decode wants a list, so the id goes in brackets
    bakersfield_pieces.append(one_piece)     # .append puts the piece on the end of the list

# len() counts how many things are in something. On a string it counts characters.
# On a list it counts entries.
print("text       :", city_name)
print("characters :", len(city_name))
print("tokens     :", len(bakersfield_ids))
print("ids        :", bakersfield_ids)
print("pieces     :", bakersfield_pieces)

What that prints:

text       : Bakersfield
characters : 11
tokens     : 3
ids        : [33, 8312, 2566]
pieces     : ['B', 'akers', 'field']

Read that output line by line. The text went in with 11 characters. It came out as 3 tokens. The three ids are 33, 8312 and 2566, which is exactly the list in Formula 2.1. The three pieces are 'B', 'akers' and 'field', and the square brackets with quotation marks inside are Python’s way of showing a list of strings.

One detail in the code is worth naming, because it catches people. tokenizer.decode([one_id]) has square brackets inside the round ones. The decode step expects a list of ids, even when there is only one, so the single id is wrapped in a list of length one. Leave the brackets out and you get an error.

Now the check from Formula 2.1, run as code rather than done on paper.

# Cell 3. Glue the pieces back together and see whether we get the original text.

# Start with an empty piece of text and add one piece at a time.
rebuilt_text = ""
for one_piece in bakersfield_pieces:
    rebuilt_text = rebuilt_text + one_piece   # the plus sign joins two strings end to end

print("rebuilt    :", rebuilt_text)
print("same again :", rebuilt_text == city_name)   # two equals signs ask "are these the same?"

What that prints:

rebuilt    : Bakersfield
same again : True

True is Python’s word for yes. The double equals sign is a question, not an instruction: one equals sign would mean “make this be that”, and two means “is this the same as that?”. This is the single most common beginner mistake in Python and it is worth the extra second every time.

Nothing was lost. The tokenizer took eleven characters apart into three numbered pieces and the pieces put the eleven characters back. That is the whole of Section 2.1: the text is still all there, it is now three numbers, and the model will only ever see the numbers.

2.2 The vocabulary, and why its size is a decision¶

Intuition¶

The phrasebook from Section 2.1 has a real name. It is called the vocabulary, and for our model it has 151,936 entries.

Three things about that number are worth separating, because they get confused constantly.

First, the vocabulary is finished. It was built once, before the model was trained, by running a counting procedure over an enormous pile of text and keeping the chunks that came up most often. Then it was frozen. Nothing the model does afterwards can add a row. If a word becomes famous next year, this model will spell it out of fragments forever.

Second, the vocabulary is a list of chunks, not a list of words. Some rows are ordinary words. Some are word fragments such as akers, which nobody would call a word. Some are single characters. Some are a space plus a word, such as ' County', which is a different row from County with no space. That last point surprises people, and it is why the same six letters can cost you one token or two depending on what sits in front of them.

Third, 151,936 is a choice. This is the part of the chapter to slow down for. There is no law of language that says a vocabulary should have about 152,000 entries. Somebody picked it, and picking a different number would have produced a different, also working, model. You can see that it is a choice from three directions, and all three are measurable.

You can see it from the number itself. 151,936 is not a round number in the way people count, but divide it by 128 and you get exactly 1,187, with nothing left over. Two signs turn up in that sentence, so here they are. The sign $\div$ is said “divided by” and means split the number on its left into equal parts, as many parts as the number on its right says; toolkit section 4 starts from the beginning. The sign $\times$ is said “times” and means multiply the number on its left by the number on its right; toolkit section 3 shows the four ways this book writes it. Now the claim, and its check.

The second line is how you check the first. Multiplying the answer back must return the number you started with, and it does, so there is no remainder hiding anywhere.

The reason 128 is the number worth trying is that it is a power of two: a number you get by multiplying 2 by itself over and over. Seven 2s in a row gives it,

and computer hardware is built out of powers of two all the way down, so a table whose number of rows divides evenly by one of them is convenient to store and to move about. Toolkit section 18 builds that idea up from nothing if it is new to you. A number chosen for convenience is a choice.

You can see it from the spare rows. The tokenizer for this model only ever produces 151,665 different ids, counting everything including its special markers. The model’s table has 151,936 rows. Take the smaller from the larger,

and that leaves 271 rows that are never used, sitting there, costing memory. A law of nature does not leave 271 empty seats.

And you can see it from a second model. A different model sitting on the same lab machine, all-MiniLM-L6-v2, uses a vocabulary of 30,522 entries, which is about a fifth the size. Divide the two to see where “a fifth” comes from: $151{,}936 \div 30{,}522 = 4.9779$, rounded to four decimal places, so our table has close to five rows for every one of theirs. That smaller table cuts Bakersfield into two pieces rather than three: 'baker' and '##sfield'. Same word, same English, different table, different answer. Two working systems disagree about the right answer, so there is no single right answer to disagree about.

What a bigger vocabulary buys you is shorter lists of tokens, because more chunks get a row of their own. What it costs you is parameters, and the next part measures exactly how many.

The mathematics¶

A vocabulary of $V$ entries is not stored as a list of words. It is stored as a grid of numbers with one row per token, and every row is the same width. That grid is what costs parameters, and its size is the arithmetic behind “151,936 is a design choice”.

Formula 2.2, the size of the vocabulary table¶

In words. The vocabulary table is a rectangle of numbers with one row for every token and the same number of columns in every row. To find out how many numbers are in it, multiply the number of rows by the number of columns.

The formula.

The symbols.

Out loud. “En sub table equals vee times dee.” In full English: “the number of values in the vocabulary table is the number of tokens multiplied by the width of one row.”

Worked, with the measured values for our model. $V = 151{,}936$ and $d = 896$.

Step 1, write down the multiplication.

$151{,}936 \times 896$

Step 2, break it into two easy pieces, because 896 is 900 minus 4. Multiplying by 900 and by 4 is easier by hand than multiplying by 896.

$151{,}936 \times 900 = 136{,}742{,}400$

$151{,}936 \times 4 = 607{,}744$

Step 3, subtract the second from the first, because $900 - 4 = 896$.

$136{,}742{,}400 - 607{,}744 = 136{,}134{,}656$

So the vocabulary table holds

Check it. Two checks. The answer must be a whole number, and it must be bigger than either of the two things you multiplied. Rough it out first: 150,000 times 900 is 135,000,000, so an answer near 136 million is the right size. If your answer came out near 136 thousand, you lost three zeros somewhere. If it came out smaller than 151,936, you added instead of multiplying.

Formula 2.3, the share of a model held by one part¶

In words. To find out what fraction of the whole model one part accounts for, divide the size of the part by the size of the whole model. Multiply by 100 to say it as a percentage.

The formula.

The symbols.

Out loud. “Share equals en sub part divided by en sub total.” In full English: “the share held by a part is the size of that part divided by the size of the whole thing.”

Worked, with measured values. The whole model is 494,032,768 numbers (measured, from lab/out/we3_params_quant.json). The vocabulary table is the 136,134,656 from Formula 2.2.

Step 1, write down the division.

$136{,}134{,}656 \div 494{,}032{,}768$

Step 2, do the division. On a phone calculator, type the top number, the divide key, the bottom number, then equals.

$136{,}134{,}656 \div 494{,}032{,}768 = 0.275558$

Your calculator will show more digits than that. The full answer starts 0.2755579484, and 0.275558 is that answer rounded to six decimal places. Most divisions never end, so every division in this chapter that does not come out exactly is shown rounded, and the line it sits on says to how many places. Shares and energy totals get six decimal places, because Cell 5 and Cell 9 print six. Characters per token gets four, because Cell 6 prints four. Toolkit section 16 explains what rounding does and what it costs.

Step 3, turn the decimal into a percentage by multiplying by 100. Multiplying by 100 moves the decimal point two places to the right.

$0.275558 \times 100 = 27.5558$

Step 4, round to two decimal places.

Check it. A share must land between 0 and 1 before you multiply by 100, and between 0% and 100% after. If you got 3.629, you divided the wrong way round; swap the top and the bottom and try again. There is a second check available here: Chapter 3 measures every part of this model, and all of the parts together must add to 100%. The vocabulary table is 27.56%, the MLP blocks are 63.51%, and all of the attention machinery is 8.92% (measured, all three from lab/out/ch03_parameter_anatomy.json). Add them, two at a time.

$27.56 + 63.51 = 91.07$

$91.07 + 8.92 = 99.99$

That is 99.99%, and 100% is what a complete list of parts has to come to, so 0.01% is somewhere else. It is not a rounding slip. There is a fourth part this chapter has not named: the layer norms, which come to 0.0089% of the model, and 0.0089 rounded to two decimal places is 0.01 (measured, same file). Put it in and the four parts come to $99.99 + 0.01 = 100.00$. Chase a gap like that down every time. Sometimes it is rounding and sometimes it is a piece you forgot, and the arithmetic will not tell you which until you look.

Python¶

Two cells. The first counts the rows, and it prints three different numbers on purpose, because they are three different things and confusing them is a real mistake rather than a hypothetical one.

# Cell 4. How many rows does the vocabulary have? Three answers, all correct, all different.

# What the tokenizer object itself reports: the ordinary tokens it can produce.
print("tokenizer.vocab_size :", tokenizer.vocab_size)

# The same count including the special markers added afterwards, such as the
# end-of-text marker. len() on a tokenizer counts every id it can ever emit.
print("len(tokenizer)       :", len(tokenizer))

# The number of rows in the model's own table, which comes from the model's config file.
# This is the number the rest of this book uses, because it is how many scores the model
# produces every time it predicts a next token.
model_vocabulary_size = 151936
print("model vocabulary     :", model_vocabulary_size)

# Rows that exist in the table but that the tokenizer never produces.
never_used_rows = model_vocabulary_size - len(tokenizer)
print("rows never emitted   :", never_used_rows)

What that prints:

tokenizer.vocab_size : 151643
len(tokenizer)       : 151665
model vocabulary     : 151936
rows never emitted   : 271

Three numbers, and the difference between them is the point. 151,643 is how many ordinary tokens the tokenizer knows. 151,665 adds the special markers, which are things like the end-of-text signal rather than pieces of English. Subtract to see how many of those there are: $151{,}665 - 151{,}643 = 22$. Then 151,936 is how many rows the model actually built its table with, and it is larger than either. The final line of the cell subtracts again to show the second gap, $151{,}936 - 151{,}665 = 271$, so 271 rows exist in the model and are never filled by anything the tokenizer produces.

When this book says “the vocabulary is 151,936”, it means the third number, because that is how many scores come out of the model on every step, and Chapter 4 is going to need all of them.

Now price the table.

# Cell 5. How big is the vocabulary table, and how much of the model is it?

# The width of one row, from the model's config file. Chapter 3 explains what the 896
# numbers in a row are for.
row_width = 896

# Rows times columns.
vocabulary_table_numbers = model_vocabulary_size * row_width

# The size of the whole model, measured in Chapter 3 and stored in lab/out/we3_params_quant.json
total_parameters = 494032768

# The share: the part divided by the whole. A single slash is division in Python.
vocabulary_share = vocabulary_table_numbers / total_parameters

print("numbers in the vocabulary table :", vocabulary_table_numbers)
print("numbers in the whole model      :", total_parameters)
print("share, as a decimal             :", round(vocabulary_share, 6))
print("share, as a percentage          :", round(vocabulary_share * 100, 2))

What that prints:

numbers in the vocabulary table : 136134656
numbers in the whole model      : 494032768
share, as a decimal             : 0.275558
share, as a percentage          : 27.56

Those are the same four numbers as Formulas 2.2 and 2.3, produced by a machine instead of by hand. Getting the same answer twice, by two routes, is the point of doing it both ways.

Read the last line again, because it is the most surprising fact in Chapter 2 and it sets up Chapter 3. Over a quarter of this model is a lookup table for chunks of text. It does no reasoning. It holds no grammar. It is the phrasebook, and it is bigger than the attention machinery that gets all the attention, which comes to 8.92% (measured, from lab/out/ch03_parameter_anatomy.json).

2.3 How long is your text, in tokens?¶

Intuition¶

Here is a question with a practical edge. How long is a piece of text?

You would probably answer in words, or in characters if you were filling in a form with a limit. Neither is the answer a language model uses. A model counts in tokens, and so does every service that charges you money for one. A company that bills “per thousand tokens” is billing you for the length of the list from Formula 2.1. A model also has a ceiling on how much text it can hold in front of it at once, called its context length, and that ceiling is counted in tokens too. No amount of careful wording gets you past it; only a shorter token list does. The context lengths of the three models this course uses are [to be measured], and the glossary entry says so rather than guessing.

So it is worth knowing how many characters of your text a single token tends to carry. That number has a name in this chapter: characters per token. It is one count divided by another count, and it tells you how efficiently this particular tokenizer handled this particular text.

The measured values are not all alike, and the spread is the interesting part. For our model, cat is 3 characters in 1 token, which is 3 characters per token. Bakersfield is 11 characters in 3 tokens, which is about 3.67. The twenty-eight-letter word antidisestablishmentarianism is 28 characters in 6 tokens, which is about 4.67, so the longest word here is the most efficient. And 1234567 is 7 characters in 7 tokens, which is exactly 1.00, the worst possible result for this tokenizer.

Length is not what decides this. A twenty-eight-letter word cost 6 tokens and a seven-character string cost 7. What decides it is how common each chunk was in the pile of text the vocabulary was counted from. establish is a common chunk and earned a row. A bare digit gets a row too, but only as a single character, so seven digits cost seven rows.

There is a warning attached to this, and it matters more than the arithmetic. Characters per token belongs to a tokenizer, not to a language and not to a person. Our model gave the Spanish string ¿Cómo estás? 5 tokens for 12 characters, which is 2.40 characters per token, lower than the 3.67 it gave Bakersfield. One string is not a study of a language and this book will not pretend it is. It is a reason to measure your own text rather than trusting a rule of thumb, particularly if somebody is charging you by the token.

The mathematics¶

Formula 2.4, characters per token¶

In words. Count the characters in your text. Count the tokens it turned into. Divide the first by the second. The answer is how many characters, on average, each token carried.

The formula.

The symbols.

Out loud. “Arr equals see over en.” In full English: “characters per token is the number of characters divided by the number of tokens.”

Worked, with three measured texts. All three counts come from lab/out/we1_tokens.json.

Text one, Bakersfield. Here $c = 11$ and $n = 3$.

$r = 11 \div 3 = 3.666666\ldots$

Rounded to four decimal places, $r = 3.6667$.

Text two, antidisestablishmentarianism. Here $c = 28$ and $n = 6$.

$r = 28 \div 6 = 4.666666\ldots$

Rounded to four decimal places, $r = 4.6667$.

Text three, 1234567. Here $c = 7$ and $n = 7$.

$r = 7 \div 7 = 1.0000$

Check it. For this tokenizer no token is shorter than one character, so $r$ can never drop below 1. If you compute a value under 1, you have more tokens than characters, which cannot happen; check whether you divided the wrong way round. A second check: multiply your answer back by $n$ and you must get $c$. For Bakersfield, $3.6667 \times 3 = 11.0001$. That is 11, plus a leftover of 0.0001 that comes from the rounding you did in the last step and from nowhere else. Rounding always leaves a crumb like that behind, and it is why this book keeps four decimal places rather than two. Keep only two and the same check gives $3.67 \times 3 = 11.01$, which is a hundred times further from 11.

Python¶

One cell, a loop over eight strings. This is the cell to change and re-run on your own text.

# Cell 6. Characters per token, for eight pieces of text.

# A list of strings to measure. Add your own to the end of this list and run it again.
texts_to_measure = ["cat", "cats", "Bakersfield", "Kern County", "ChatGPT",
                    "antidisestablishmentarianism", "1234567",
                    "The capital of France is"]

# A heading line, so the columns of numbers underneath have names.
print("text                              chars  tokens  chars/token")

# Take each string in turn. Everything inside the loop happens once per string.
for one_text in texts_to_measure:
    one_text_ids = tokenizer(one_text).input_ids   # the list of token ids for this string
    character_count = len(one_text)                # how many characters, spaces included
    token_count = len(one_text_ids)                # how many tokens it became
    characters_per_token = character_count / token_count   # Formula 2.4, r = c / n
    # The f in front of the quotation marks lets us put values inside { } braces.
    # The :<34 pads the text out to 34 characters wide. The :>13.4f puts the number on
    # the right of a 13-wide column with 4 decimal places.
    print(f"{one_text:<34}{character_count:>5}{token_count:>8}{characters_per_token:>13.4f}")

What that prints:

text                              chars  tokens  chars/token
cat                                   3       1       3.0000
cats                                  4       1       4.0000
Bakersfield                          11       3       3.6667
Kern County                          11       3       3.6667
ChatGPT                               7       3       2.3333
antidisestablishmentarianism         28       6       4.6667
1234567                               7       7       1.0000
The capital of France is             24       5       4.8000

Eight rows, and five of them are worth stopping on, in four groups.

cat and cats. Both are one token. The tokenizer did not take cat and add an s. The word cats has its own row in the vocabulary, which happens to be row 37,452, while cat is row 4,616. Those two numbers have nothing to do with each other.

ChatGPT at 2.3333. Seven characters, three tokens: 'Chat', 'G', 'PT'. A product name famous enough to be in every headline in 2026 does not have a row of its own, because the vocabulary was counted before it was famous, and the vocabulary is frozen.

antidisestablishmentarianism at 4.6667. The longest string in the table is close to the most efficient one, and it is beaten only by the ordinary English sentence in the last row at 4.8000. A twenty-eight-letter word costs less per character than ChatGPT does. Length does not decide the cost.

The capital of France is at 4.8000. Twenty-four characters, five tokens. Ordinary English sentences do well here, because ordinary English is what the counting procedure saw most of. This particular sentence is the prompt Chapter 4 uses, and its five ids are [785, 6722, 315, 9625, 374] (measured, from lab/out/we2_softmax.json).

2.4 Seven tokens for seven digits¶

Intuition¶

This is the section the chapter was built for.

Hand our model the string 1234567 and it becomes seven tokens, one for every digit: ids 16, 17, 18, 19, 20, 21 and 22, standing for '1', '2', '3', '4', '5', '6' and '7'. The model does not receive the number one million two hundred thirty-four thousand five hundred and sixty-seven. It receives seven symbols in a row, and nothing in that row says which symbol is in the hundreds place.

Think about what you do when you add two numbers on paper. You line the numbers up on the right, you add the rightmost column, and if the total is ten or more you carry one into the next column. Every step of that depends on knowing which column each digit is in. Lining up the columns is not a preliminary to the arithmetic; lining up the columns is the arithmetic. That knowledge is called place value, and a person carries it around as a fact about how written numbers work.

A model handed seven unrelated symbols has no such fact. It has an order, because the tokens arrive in order, and that is all. Whether the first symbol means millions or means one is something it has to work out for itself from having read a great deal of text in which the answer happened to be written down. Place value is not given to it. Place value has to be rebuilt, every time, from the order the symbols arrived in.

That is the honest, mechanical explanation for something people usually explain badly. When a language model fumbles a multiplication, the reason is not that it is “not really thinking” or that it “lacks true understanding”. Those sentences explain nothing and cannot be tested. This one can: the input never contained place value, so place value had to be inferred. You can check the input side of that claim yourself in ten seconds with Cell 7, and you can check it on any tokenizer you like.

Being testable is the point, and it cuts both ways, so here is what this book has and has not done. It has measured the input: 1234567 is seven tokens, one per digit. It has not measured how this model’s accuracy on arithmetic changes as numbers get longer, and it will not claim a pattern it did not run. That experiment is a good capstone project, and Chapter 11 gives you the machinery to run it: pick a range of digit lengths, write a bank of questions, score them, and report the result with an interval. Predict first, then measure. If the prediction that longer numbers are harder turns out to be wrong, the measurement wins.

The mathematics¶

To see exactly what the token list is missing, write down what place value actually says. Here is the rule as a formula, with every symbol defined.

Formula 2.5, the value of a written number from its digits¶

In words. A written number is built by multiplying each digit by a power of ten that depends on how far that digit sits from the right-hand end, and then adding all of those products together.

The formula.

Written out in full, with no sigma, for a seven-digit number:

The symbols.

Out loud. “En equals the sum, from kay equals zero to em, of dee sub kay times ten to the kay.” In full English: “the value of a written number is found by multiplying each digit by ten raised to the power of its position, counting positions from zero at the right, and adding all the results.”

Worked, on 1234567, every step. The digits, numbered from the right, are $d_6 = 1$, $d_5 = 2$, $d_4 = 3$, $d_3 = 4$, $d_2 = 5$, $d_1 = 6$, $d_0 = 7$. Note the reversal: the digit written first, on the left, has the largest position number.

Step 1, work out each power of ten.

$10^6 = 1{,}000{,}000$

$10^5 = 100{,}000$

$10^4 = 10{,}000$

$10^3 = 1{,}000$

$10^2 = 100$

$10^1 = 10$

$10^0 = 1$

Step 2, multiply each digit by its power of ten.

$1 \times 1{,}000{,}000 = 1{,}000{,}000$

$2 \times 100{,}000 = 200{,}000$

$3 \times 10{,}000 = 30{,}000$

$4 \times 1{,}000 = 4{,}000$

$5 \times 100 = 500$

$6 \times 10 = 60$

$7 \times 1 = 7$

Step 3, add them one at a time, from the top down.

$1{,}000{,}000 + 200{,}000 = 1{,}200{,}000$

$1{,}200{,}000 + 30{,}000 = 1{,}230{,}000$

$1{,}230{,}000 + 4{,}000 = 1{,}234{,}000$

$1{,}234{,}000 + 500 = 1{,}234{,}500$

$1{,}234{,}500 + 60 = 1{,}234{,}560$

$1{,}234{,}560 + 7 = 1{,}234{,}567$

Check it. The result must have the same digits, in the same order, as the string you started with. If your total has six digits, you dropped a term. A faster check: the first term on its own, $1{,}000{,}000$, must be bigger than everything else added together, because each position is worth ten times the one to its right. It is.

Why this formula is here. Look at what Formula 2.5 needs and what Formula 2.1 supplies. Formula 2.5 needs two things for every digit: the digit’s value, and the digit’s position counted from the right. Formula 2.1 supplies a list of ids in the order they appeared, counted from the left, with no ending in sight until the list runs out. The token list carries the digits. It does not carry $10^k$. To use place value at all, a model must first work out how long the number is, then count backwards from the end, then attach a power of ten to each position, and it must do all of that from patterns in text rather than from a rule it was given. That is the gap, written as arithmetic instead of as an opinion.

Python¶

One cell, and it is the shortest in the chapter.

# Cell 7. What the model actually receives when you type a seven-digit number.

digit_string = "1234567"
digit_ids = tokenizer(digit_string).input_ids

print("text   :", digit_string)
print("tokens :", len(digit_ids))
print("ids    :", digit_ids)

# Show what each id stands for, one line at a time.
# repr() puts quotation marks round the piece so you can see it is a piece of text,
# not a number.
for one_id in digit_ids:
    print("   id", one_id, "stands for", repr(tokenizer.decode([one_id])))

What that prints:

text   : 1234567
tokens : 7
ids    : [16, 17, 18, 19, 20, 21, 22]
   id 16 stands for '1'
   id 17 stands for '2'
   id 18 stands for '3'
   id 19 stands for '4'
   id 20 stands for '5'
   id 21 stands for '6'
   id 22 stands for '7'

Seven characters, seven tokens, one per digit. That is the measurement the whole section rests on, and it took four lines of code.

One thing in that output invites a wrong conclusion, so it is worth heading off. The ids 16 through 22 are consecutive, and it is tempting to read something into that. All ten digits do land on consecutive rows in this tokenizer: '0' is id 15 and '9' is id 24 (measured, from lab/out/we1_tokens.json). But that is a fact about how this particular table was built, and it does not make the ids into quantities. The next cell shows why not.

# Cell 8. Token ids are labels, not amounts.

cat_ids = tokenizer("cat").input_ids
cats_ids = tokenizer("cats").input_ids

print("id for 'cat'  :", cat_ids)
print("id for 'cats' :", cats_ids)
print("difference    :", cats_ids[0] - cat_ids[0])   # [0] takes the first entry of a list

What that prints:

id for 'cat'  : [4616]
id for 'cats' : [37452]
difference    : 32836

The difference is 32,836 and it means nothing at all. Adding one letter to a word moved its id by thirty-two thousand. Two words with almost the same meaning sit thirty-two thousand rows apart, while two digits with completely different values sit one row apart. Nearness in id has no relationship to nearness in meaning. Ids are seat numbers in a theatre: seat 12 and seat 13 are adjacent, and that tells you nothing about the two people sitting in them.

Chapter 8 introduces the thing that does carry meaning as a number, which is a row of the vocabulary table rather than the row’s number. Until then, hold the line: a token id is a name.

2.5 What a token costs¶

Intuition¶

Every week of this course ends the same way. You run something, you watch what it does, you explain the arithmetic, and then you write down what the run cost. This is the fourth beat, and in this chapter it is short, because you now have the unit that everything else is measured in.

A model does not spend energy per word, per sentence, or per question. It spends energy per token. Producing one token means a pass through the whole machine, and the machine draws power from a wall while it does that. Chapter 1 measured it. On this course’s lab machine, an NVIDIA RTX 3500 Ada laptop GPU, the smallest model in the course spent 0.767 joules per token it wrote (measured, from lab/out/theme_s_energy.json).

A joule is a small unit. Lifting an apple one metre takes about one joule. So 0.767 joules is a small amount of energy, and that is honest to say: one token is cheap. The reason the number matters is not that it is large. It is that it is a per-token number, and token counts get large quickly. Work one out from a rate you have already measured. An ordinary English sentence came to 4.8 characters per token in Section 2.3. A page of about 2,500 characters at that rate is $2{,}500 \div 4.8 = 520.8333$ tokens, where the division never ends and 520.8333 is that answer rounded to four decimal places. Part of a token is not a thing you can have, so round again, to 521 whole tokens. At 0.767 joules each that is $521 \times 0.767 = 399.607$ joules, call it 399.6. That estimate uses a measured rate and a made up for practice page length, and it is the shape of every cost claim in this course: a count you made, times a rate you can point at.

This is also the exact place where the two halves of this course’s argument meet. Fewer tokens means less energy, which is the environmental side. Fewer tokens also means less time and less memory, which is the access side: it is what decides whether a model runs on a laptop somebody already owns rather than on hardware they would have to buy. Chapter 7 makes that argument quantitative. This chapter gives you the unit it is made of.

Three limits have to travel with this number every time it is used, and they are not fine print. It is GPU board power only, with no processor, no memory, no power-supply losses, no screen and no cooling, so it is a lower bound on what a local run costs rather than a total. It came from one greedy run of one prompt on one machine. And a meaningful share of it is nothing more than the cost of having the graphics card switched on. The idle draw was 13.8 watts and the average draw during the run was 21.3 watts, so the work itself added $21.3 - 13.8 = 7.5$ watts. A watt is one joule every second, and the model wrote 27.7 tokens every second, so the work cost $7.5 \div 27.7 = 0.2708$ joules per token, rounded to four decimal places. Those three figures are themselves rounded: the lab file holds 13.834 watts idle, 21.254066 watts mean and 27.708138 tokens per second. Run the same subtraction and division on the full values and you get 0.268 joules per token (measured, same file), which is the number this book uses. The gap between 0.2708 and 0.268 is entirely the rounding, and it is worth seeing once. Most of the 0.767 was the card being on.

The mathematics¶

Formula 2.6, the energy in a piece of text¶

In words. To find the energy a model spent writing something, multiply how many tokens it wrote by how much energy one token costs.

The formula.

The symbols.

Out loud. “Ee equals en times pee.” In full English: “total energy is the number of tokens multiplied by the energy per token.”

Worked, twice.

First, the small case. The word Bakersfield is 3 tokens (measured). So $n = 3$ and $p = 0.767$ joules per token.

$E = 3 \times 0.767$

$3 \times 0.7 = 2.1$

$3 \times 0.067 = 0.201$

$2.1 + 0.201 = 2.301$

Second, a longer answer. Suppose a model writes a 500-token reply, which is roughly a page. (The 500 is made up for practice; the 0.767 is measured.) So $n = 500$.

$E = 500 \times 0.767$

$500 \times 0.7 = 350$

$500 \times 0.06 = 30$

$500 \times 0.007 = 3.5$

$350 + 30 + 3.5 = 383.5$

To put that in the unit an electricity bill uses, divide by 3,600, because one watt-hour is 3,600 joules.

$383.5 \div 3600 = 0.106528$ watt-hours, rounded to six decimal places.

Check it. Three checks, and the third is the good one. The answer must be larger than $p$ whenever $n$ is more than 1, and it must carry the unit joules, never “0.767” on its own with no unit attached. Rough it out: 0.767 is close to 0.75, and 500 times 0.75 is 375, so an answer near 383 is the right size. And the independent check: the lab recorded 0.213 watt-hours per 1,000 tokens for this model. Half of 1,000 tokens should cost half of 0.213, which is 0.1065, and that is what the division gave. Two different measurements agreeing to four decimal places is a strong sign the arithmetic is right.

Python¶

One cell. It uses the token count Cell 2 already produced, so run this after Cell 2.

# Cell 9. What it costs to write a piece of text.

# Measured on the lab GPU in Chapter 1 and recorded in lab/out/theme_s_energy.json.
# Joules of energy spent per token generated, by Qwen2.5-0.5B-Instruct.
joules_per_token = 0.767

# The small case: the three tokens of "Bakersfield" from Cell 2.
bakersfield_joules = len(bakersfield_ids) * joules_per_token
print("tokens in 'Bakersfield' :", len(bakersfield_ids))
print("joules to write it      :", bakersfield_joules)

# A longer answer. The 500 is a round number chosen for the example, not a measurement.
essay_tokens = 500
essay_joules = essay_tokens * joules_per_token

# One watt-hour is 3600 joules, because an hour is 3600 seconds.
essay_watt_hours = essay_joules / 3600

print("tokens in a short essay :", essay_tokens)
print("joules to write it      :", essay_joules)
print("watt-hours to write it  :", round(essay_watt_hours, 6))

What that prints:

tokens in 'Bakersfield' : 3
joules to write it      : 2.301
tokens in a short essay : 500
joules to write it      : 383.5
watt-hours to write it  : 0.106528

Those are the same two answers as Formula 2.6, computed the other way. The variable names carry their units in them on purpose: joules_per_token, essay_joules, essay_watt_hours. A number without a unit is most of an error waiting to happen, and naming the unit in the variable is the cheapest way to stop it.

Notice what this cell did not do. It did not run a model. It multiplied a token count by a measurement somebody else made on a documented machine, and it said which file that measurement came from in a comment. That is the honest shape of a cost estimate, and it is the shape Lab 1 asks for: a count you made, times a rate you can cite, with the limits of the rate written down next to it.

Common mistakes¶

Six things go wrong in this chapter, and they go wrong often enough to be worth naming. Each one comes with the symptom you would actually see.

1. Treating a token as a word. Token counts and word counts are different numbers, and they are not close for names, numbers or technical terms. Symptom: you predict a 10-word sentence will be 10 tokens and the count comes back 14. Fix: count tokens with the tokenizer, every time, and never estimate from word count in written work for this course.

2. Doing arithmetic on token ids. Ids are categorical labels. Adding, averaging, sorting or comparing them produces a number with no meaning. Symptom: you notice '1' is id 16 and '2' is id 17 and conclude ids track values. They do not; cat is 4,616 and cats is 37,452. Fix: treat an id as a seat number.

3. Adding a space by hand when rebuilding text. In this tokenizer a space belongs to the token that follows it, so ' County' already carries its space. Symptom: your rebuilt string has two spaces where the original had one, and the check in Cell 3 prints False. Fix: glue the decoded pieces together with nothing between them.

4. Mixing up the three vocabulary counts. 151,643, 151,665 and 151,936 are three correct answers to three different questions. Symptom: your share calculation gives 27.51% rather than 27.56%. Fix: for anything involving the model’s table, including every calculation in Chapters 3 and 4, use 151,936.

5. Rounding partway through. Rounding 3.666666 to 3.67 and then multiplying by 3 gives 11.01 rather than 11. Symptom: your answer is close to the printed one but wrong in the second or third decimal place. Fix: keep all the digits your calculator shows until the last line, then round once. See toolkit section 16.

6. Quoting a characters-per-token figure as though it were a fact about a language. The value 2.40 that this chapter measured for one Spanish string describes that string as cut by that tokenizer. Symptom: a sentence in your lab report of the form “Spanish costs more tokens than English”. Fix: say which tokenizer, which text, and how many strings you measured, or do not make the claim.

What to remember¶

A computer stores only numbers, so your text is cut into pieces called tokens and every piece is swapped for a whole number called its id. Those ids are row numbers in a fixed list called the vocabulary, which has 151,936 rows in our model and takes up 27.56% of it. Somebody chose that size; a different model on the same machine uses 30,522 rows and cuts Bakersfield into two pieces instead of three. Length is counted in tokens rather than words, and characters per token, $r = c/n$, says how efficiently one tokenizer handled one text. Every digit of 1234567 is its own token, so place value is missing from the input and has to be rebuilt from the order, and that measurement is where an honest account of these models and arithmetic starts.

Practice problems¶

Twenty-four problems in three tiers. Warm-up asks whether you can do the arithmetic. Practice asks whether you can apply it. Stretch asks whether you can reason with it. Odd-numbered answers are in the answers appendix.

Unless a problem says otherwise, use $V = 151{,}936$, $d = 896$, $N_{\text{total}} = 494{,}032{,}768$ and $p = 0.767$ joules per token, all of which are measured values for Qwen2.5-0.5B-Instruct on this course’s lab machine. Any other number a problem hands you, such as a made-up vocabulary of 50,000 rows or a price per million tokens, is made up for practice. It is there so the arithmetic fits on a phone, and it is not a report of any model anyone built or ran.

Warm-up¶

1. Bakersfield is 11 characters and 3 tokens. Compute $r = c/n$ to four decimal places.

2. ChatGPT is 7 characters and 3 tokens. Compute $r$ to four decimal places.

3. A vocabulary has $V = 50{,}000$ rows and a row width of $d = 768$. Use Formula 2.2 to find $N_{\text{table}}$.

4. Using your answer to problem 3 and a model total of 110,000,000 numbers, use Formula 2.3 to find the share, as a decimal to six places and as a percentage to two.

5. A model writes 250 tokens. Use Formula 2.6 with $p = 0.767$ to find the energy in joules.

6. Convert your answer to problem 5 into watt-hours, remembering that one watt-hour is 3,600 joules.

7. Write out 10⁰, 10¹, 10², 10³ and 10⁴ as ordinary numbers.

8. Use Formula 2.5 to find the value of the three-digit string 907. State $m$ first, then show every step.

Practice¶

9. The tokenizer gives ' spaces' (two leading spaces, then the word) the pieces ' ' and ' spaces', so $c = 8$ and $n = 2$. Compute $r$, then explain in one sentence why the same six letters spaces can cost either one token or two.

10. A team wants to add rows for 2,500 place names to the vocabulary, at $d = 896$ each. How many numbers does that add, and what percentage of 494,032,768 is it?

11. antidisestablishmentarianism is 28 characters in 6 tokens and 1234567 is 7 characters in 7 tokens. Compute $r$ for each and write one sentence saying why the longer string is the cheaper one per character.

12. The prompt The capital of France is is 5 tokens. If a model answers with 40 tokens, what is the total energy for prompt plus answer at $p = 0.767$? State your assumption about whether reading the prompt costs the same per token as writing the answer, and say why that assumption might be wrong.

13. The model’s table has 151,936 rows and the tokenizer emits at most 151,665 ids. How many rows are never emitted, and how many numbers do those unused rows occupy at $d = 896$?

14. Two token ids are 4,616 and 37,452. A classmate computes their average, 21,034, and asks what word it is. Write two sentences explaining what is wrong with the question.

15. Use Formula 2.5 to find the value of 50607. State $m$, show each power of ten, and say what the two zeros contribute.

16. A service charges $0.50 per million tokens. A document is 4,200 tokens. What does it cost? Now suppose the same document is measured with a tokenizer whose vocabulary is 30,522 rows and it comes to 5,900 tokens. What does it cost then, and what does the difference tell you about token-based pricing?

17. all-MiniLM-L6-v2 has $V = 30{,}522$ and cuts Bakersfield into 2 pieces, while our model has $V = 151{,}936$ and cuts it into 3. Compute $r$ for each, then explain in two sentences why a larger vocabulary did not produce the shorter token list here.

18. A model produces 3,000 tokens over a session. Compute the energy at the all-in rate $p = 0.767$ and at the above-idle rate $p = 0.268$. Express the second as a percentage of the first, to one decimal place.

Stretch¶

19. The vocabulary table is 27.56% of this model. Chapter 3 reports the MLP blocks at 63.51%, all of attention at 8.92% and the layer norms at 0.0089%. Add the four. Now suppose you had only the first three and saw a total of 99.99%. Write two sentences saying how you would work out, without being told, whether a gap that small is rounding or a part you forgot, and say which of the two answers would worry you more.

20. Suppose you could redesign the vocabulary with a budget of 494,032,768 numbers held fixed, so that every row you add to the table must be paid for by taking numbers away from somewhere else. Write a paragraph arguing either for a larger vocabulary or for a smaller one. Your argument must name what is gained, what is given up, and one measurement you would make to decide.

21. A classmate says “the model is bad at maths because it does not really understand numbers”. Rewrite that claim as a statement about the input that could be checked with Cell 7. Then describe, in two sentences, what result would show your rewritten claim to be wrong.

22. Formula 2.5 needs the position of each digit counted from the right-hand end. Formula 2.1 supplies tokens in order from the left. Explain, in a short paragraph, why that mismatch is harder for long numbers than for short ones, and name one change to the input, not to the model, that would remove the problem.

23. Characters per token for one Spanish string measured 2.40, and for Bakersfield it measured 3.6667. A news article claims from this that “Spanish speakers pay more to use AI”. List three things you would need to measure before that claim could be made, and say which single one you would do first and why.

24. Energy per token for this model is 0.767 joules all-in and 0.268 joules above idle. A vendor quotes the smaller number, a critic quotes the larger. Write a paragraph saying which number answers which question, and give one question about MATH 3219’s own use of these models where the two would lead to different decisions.