A sentence is an arrow

Chapter 8. A sentence is an arrow.

This chapter has no language model in it until the last section. It has graph paper, two arrows, and arithmetic small enough to do on your phone. That is deliberate. The geometry is the part people find hard, and meeting it inside a 22-million-parameter model at the same time as meeting it for the first time is two problems at once.

So the arrows come first.

What you need before this chapter¶

An honest list. If any line below is unfamiliar, the link takes you to a short section of the Math Toolkit that starts from nothing and works arithmetic you can check.

From earlier in this book you need one idea and one number. The idea is that a model stores numbers it learned, which Chapter 3 calls parameters. The number is that Qwen2.5-0.5B-Instruct holds 494,032,768 of them, and that 27.56 per cent of them sit in a single lookup table of words. This chapter explains what is stored in a table like that.

You do not need anything from Chapter 4, Chapter 5, Chapter 6 or Chapter 7. No softmax, no probability, no bits. If Module C went badly for you, this chapter is a fresh start.

The setup code for this chapter¶

Every piece of Python in this chapter runs after this one block. Run it once, at the top, and then leave it alone. This course puts every import in the first cell, one per line, each with a comment saying what it is for, so that you can read one block and know everything the rest of the chapter depends on.

# Cell 1. The imports for Chapter 8. Run this once, before anything else on this page.

import os                                      # lets Python read and change settings on your computer
os.environ["HF_HOME"] = r"C:\math3219\models"  # the folder where downloaded models are kept
                                               # this line MUST come before the model import below

import math                                    # gives us math.sqrt, which takes a square root
from sentence_transformers import SentenceTransformer   # turns a whole sentence into one list of numbers

Three of those four lines do nothing visible. That is normal. An import is a way of telling Python which toolboxes you intend to open, and Python opens them quietly.

The one line worth reading twice is the second. os.environ["HF_HOME"] sets the folder where downloaded models are stored on your machine. Models are large, and the default folder is usually hidden somewhere inconvenient. Setting it yourself means you can find the files, check their size, and delete them when the semester ends. The letter r sitting in front of the opening quotation mark stands for raw. It tells Python to take every character between the quotes exactly as typed. That matters on Windows, because the backslash in a folder path has a second job inside Python quotes, and the r switches that second job off. Change C:\math3219\models to a path that exists on your own computer. On a Mac or on Linux that would look like /Users/yourname/math3219/models instead.

If the last line fails with a message about sentence_transformers not being found, the library is not installed yet. Section 2 of the Python Reference walks through installing it, one command at a time, with the expected output of each command printed underneath.

Giving directions in Bakersfield¶

Someone stops you outside the library at CSU Bakersfield and asks how to get to a place a few minutes away. You will probably answer with two numbers. Go this many blocks that way, then this many blocks that other way. Two numbers and a starting point, and you have named a location exactly, without a map, without a street name, without a single adjective.

That is the whole idea of this chapter, and you already had it.

Now make the two numbers do something harder. Suppose two people both start outside the library. One walks 3 blocks east and 4 blocks north. The other walks 4 blocks east and 3 blocks north. Neither of them has gone the same way as the other. But they have not gone in wildly different directions either. They are both heading roughly north-east, and they end up roughly the same distance from where they began. Something in you already knows that “3 east and 4 north” and “4 east and 3 north” are similar journeys, and that “3 east and 4 north” and “4 west and 3 north” are not.

This chapter turns that feeling into one number you can compute.

Here is why that matters for a language model. A model cannot read. It has no access to meaning, only to arithmetic. So the trick every modern language system uses is to turn a piece of text into a list of numbers, in such a way that texts with similar meanings get similar lists. Then “do these two sentences mean roughly the same thing?” becomes “do these two lists of numbers point roughly the same way?”, and that second question is arithmetic.

The lab for this course ran a real model over six sentences. Two of them were these:

“Bakersfield is in Kern County, California.”

“Kern County’s largest city is Bakersfield.”

Those two sentences share three words and disagree about which of the two places is the subject of the sentence. The model scored them at 0.833303 against each other, where 1 would mean identical direction. Two other sentences in the same run were:

“The cat sat on the mat.”

“A kitten rested on the rug.”

Those two share no content words at all. Not one. “Cat” and “kitten” are different words, “sat” and “rested” are different words, “mat” and “rug” are different words. They scored 0.612421. A method that only counted shared content words would have scored that pair zero.

And a third pair, “The stock market fell sharply on Tuesday.” against “Photosynthesis converts light into chemical energy.”, came out at -0.016586, a negative number.

By the end of this chapter you will know exactly what arithmetic produced those three numbers, and you will have done the same arithmetic yourself on arrows you can draw.

Learning objectives¶

By the end of this chapter you will be able to:

1. Plot a point from a pair of numbers, name the origin, and say out loud what each number in the pair means.

2. Compute the length of a two-dimensional vector using Pythagoras, showing the squaring, the addition and the square root as three separate steps.

3. Compute a dot product by hand, and state in plain English what a positive answer, a zero answer and a negative answer each tell you about the two arrows.

4. Explain what “384 dimensions” means to someone who can only picture three, and say why the length formula and the dot product formula do not change when the list gets longer.

5. Pull a real 384-number sentence vector out of a model in Python, check that its length is 1.000000, and compute a dot product between two of them with an explicit loop.

This lesson at a glance¶

• A vector is one object with two faces: an ordered list of numbers, which is how you compute with it, and an arrow drawn from the origin, which is how you picture it.

• The length of the arrow comes from Pythagoras. Square each number, add the squares, take the square root. For $(3,4)$ that is $\sqrt{9+16} = \sqrt{25} = 5$.

• The dot product multiplies matching coordinates and adds the results. For $(3,4)$ and $(4,3)$ it is $12 + 12 = 24$. For $(3,4)$ and $(-4,3)$ it is $-12 + 12 = 0$, and zero is what a right angle looks like in arithmetic.

• A real sentence becomes a list of 384 numbers instead of 2. Every formula on this page works on that list without a single change.

The vocabulary of this chapter¶

Every term below is used later on this page. They are collected here first so that no word arrives unexplained. Each one is defined again, more slowly, at the point where it is needed.

8.1 Where things are: axes, points, and the origin¶

Intuition¶

Take a sheet of graph paper. Draw a horizontal line across the middle of it and a vertical line down the middle. You now have two reference lines crossing at one spot, and that is the entire apparatus.

The point where they cross is the starting place for everything. It is called the origin, which is the same word as in “the origin of a story”: the place things begin. The corner outside the CSUB library in the opening of this chapter was an origin. Choosing where it sits is up to you, and once you have chosen you must stop moving it.

From the origin you can name any spot on the paper with two numbers: how far across, then how far up. Always across first. This is not a deep truth about the universe. It is a convention, agreed a long time ago so that two people writing $(3,4)$ mean the same place. Conventions like this are worth learning once and then never thinking about again.

Two things about the numbers themselves. A number can be negative, and a negative first number means go left instead of right. A negative second number means go down instead of up. And a number does not have to be a whole number. The point $(2.5, -1.25)$ is a real place: two and a half across, one and a quarter down.

Here is the part that people find genuinely strange the first time, so it is worth saying plainly. The order in the pair carries information. $(3,4)$ and $(4,3)$ use the same two digits and they are two different places on the paper. If you write the numbers in the wrong order you have not made a small slip. You have named somewhere else. That is why the pair is called an ordered pair, and the word “ordered” is doing real work.

This chapter needs you to be comfortable with exactly that much: two axes, one origin, and a pair of numbers that names a spot. Everything after this is built on it.

The mathematics¶

A point is written as two numbers inside round brackets with a comma between them.

There is no formula here yet, only notation, so this section defines the marks rather than working arithmetic.

Four points, read out in full, so that nothing is left implied.

• $(3, 4)$ is 3 to the right of the origin and 4 above it.

• $(4, 3)$ is 4 to the right of the origin and 3 above it. This is not the same point as $(3,4)$.

• $(-4, 3)$ is 4 to the left of the origin and 3 above it. The minus sign only affects the direction of the first move.

• $(6, 8)$ is 6 to the right and 8 above.

Those four points are the ones this whole chapter uses. They are made up for practice, chosen because every calculation they produce comes out a whole number.

In Python¶

Python writes an ordered pair almost the way mathematics does. Type this into a cell of its own and run it.

# Two places on a grid, each written as a pair of numbers.
# First number: how far across. Second number: how far up.

first_place = (3, 4)      # 3 across from the origin, then 4 up
second_place = (4, 3)     # 4 across from the origin, then 3 up

print("first place: ", first_place)
print("second place:", second_place)

# The two numbers inside a pair are its coordinates.
# Python counts positions starting from 0, so position 0 holds the FIRST coordinate.
print("east coordinate of the first place: ", first_place[0])
print("north coordinate of the first place:", first_place[1])

# The order matters. These two pairs hold the same digits and name different places.
print("are the two places the same place?", first_place == second_place)

Output:

first place:  (3, 4)
second place: (4, 3)
east coordinate of the first place:  3
north coordinate of the first place: 4
are the two places the same place? False

Four things are happening there, and one of them catches almost everybody.

first_place = (3, 4) creates the pair and gives it a name. The equals sign in Python is not the equals sign in mathematics. In mathematics $=$ announces that two things are the same number. In Python = is an instruction: make this name refer to this value. It is closer to “let” than to “equals”.

print(...) displays whatever is inside the brackets. The text inside double quotes is printed exactly as written, and the name outside the quotes is replaced by its value. That is why the first line of output shows the label and then the pair.

first_place[0] reaches inside the pair and pulls out one number. This is the thing that catches people. Python counts positions from 0, not from 1. So position 0 is the first coordinate and position 1 is the second. Meanwhile the mathematics in the next section counts from 1 and calls them $a_1$ and $a_2$. Both conventions are correct inside their own world and neither is going to change for our convenience. The rule to hold on to is: $a_1$ in the maths is a[0] in the Python. Write that on the inside cover of your notes. Almost every mistake in this chapter’s code is that mistake.

first_place == second_place asks a question rather than giving an instruction. The double equals sign is Python’s way of asking “are these the same?”, and the answer comes back as True or False. Here it is False, because $(3,4)$ and $(4,3)$ are different places, which is what the last paragraph of the intuition section claimed. The code agrees with the claim.

One more small thing. The extra spaces inside "first place: " are there so that the two lines of output line up under each other. Lining up output is not decoration. When you have twelve numbers on the screen, a column that lines up is a column you can read.

8.2 An arrow from the origin: what a vector is¶

Intuition¶

Go back to the point $(3,4)$ on your graph paper. Now draw a straight line from the origin to it, and put an arrowhead on the end that touches $(3,4)$.

You have not added any information. The arrow goes to the same place the point was. But you have changed what the object feels like, and the change is useful. A point is a location, and locations sit still. An arrow has a direction and a length, and both of those are things you can compare between two arrows.

That is the whole move of this section. The same two numbers, read a different way.

• Read as a location: “the spot 3 across and 4 up”.

• Read as an arrow: “a journey of a certain length, heading up and to the right”.

The arrow version is called a vector. And a vector is genuinely both things at once. When you need to calculate, you use the list of numbers. When you need to think, you use the arrow. Swapping between the two pictures inside a single paragraph is the skill this chapter is teaching, and it is normal for that to feel unsteady for a week.

Why does a language model want arrows rather than locations? Because the question a model needs to answer is “are these two texts about the same thing?”, and that is a question about direction. Two arrows pointing the same way are making the same claim about where to go. Two arrows pointing opposite ways are disagreeing. Two arrows at a square corner to each other are neither agreeing nor disagreeing; they are talking about unrelated things.

You cannot ask a location whether it agrees with another location. You can ask an arrow.

One rule, and it holds everywhere in this chapter and the next two. Every arrow starts at the origin. Vectors in general do not have to, but in this book they always do, which means an arrow is completely determined by where its tip lands. Tip at $(3,4)$, and the arrow is fixed.

The mathematics¶

A vector is written as a list of numbers, and the whole list gets a single bold letter as its name.

Formula 8.1: a vector as a list of numbers.

1. In words. A vector is a list. Give the list one name, and give each number in the list that same name with a small number written low and to the right saying which position it occupies.

2. The formula.

3. The symbols.

4. Out loud. “Vector a is the list whose first number is a sub one, whose second number is a sub two, and so on up to a sub capital D.”

5. Worked, on the vector $\mathbf{a} = (3, 4)$, made up for practice.

Step 1, count the numbers in the list. There are two of them, so $D = 2$.

Step 2, name the first one. The first number is 3, so $a_1 = 3$.

Step 3, name the second one. The second number is 4, so $a_2 = 4$.

Step 4, check that the last one is also $a_D$. Since $D = 2$, the symbol $a_D$ means $a_2$, which is 4. Both names point at the same number, which is what we wanted.

Worked once more, on $\mathbf{c} = (-4, 3)$, also made up for practice. Here $D = 2$, $c_1 = -4$ and $c_2 = 3$. The minus sign belongs to the number, not to the subscript. The first component of $\mathbf{c}$ is negative four, not four.

6. Check it. A subscript is never an exponent. $a_2$ means “the second number in the list $\mathbf{a}$”. It does not mean $a$ multiplied by itself. The position of the small number tells you which: low and to the right is a subscript, raised and to the right is an exponent. If you ever find yourself trying to calculate $a_2$, stop, because $a_2$ is already a number and there is nothing to work out. A second check: a vector is never a single number. If your answer to “what is $\mathbf{a}$?” is one number, you have reported a component, not the vector.

Three pieces of notation you will meet in other books, mentioned once so they do not startle you later. Some authors write a vector with an arrow over it, $\vec{a}$, instead of in bold. Some write the list vertically in a tall bracket instead of horizontally. Some use square brackets, $[3, 4]$, instead of round ones. All three mean exactly what $\mathbf{a} = (3,4)$ means here. This book uses bold and round brackets throughout.

In Python¶

Python has no bold type, so a vector is stored as a list, written with square brackets. Type this into a new cell.

# Four vectors, written as Python lists. Square brackets, numbers separated by commas.
# All four are made up for practice; they are chosen so the arithmetic stays whole.

vector_a = [3, 4]      # 3 across, 4 up
vector_b = [4, 3]      # 4 across, 3 up
vector_c = [-4, 3]     # 4 to the LEFT, 3 up
vector_d = [6, 8]      # 6 across, 8 up. Notice this is exactly twice vector_a.

print("vector a:", vector_a)
print("vector b:", vector_b)
print("vector c:", vector_c)
print("vector d:", vector_d)

# len() counts how many numbers are in the list. That count is the dimension, D.
print("how many numbers are in vector a?", len(vector_a))

# The subscript trap, spelled out. a_1 in the maths is vector_a[0] in the Python.
print("a_1, the first number of a, is", vector_a[0])
print("a_2, the second number of a, is", vector_a[1])

Output:

vector a: [3, 4]
vector b: [4, 3]
vector c: [-4, 3]
vector d: [6, 8]
how many numbers are in vector a? 2
a_1, the first number of a, is 3
a_2, the second number of a, is 4

The names are long on purpose. vector_a is more typing than a, and it is worth every extra character. In six weeks you will open this file again and vector_a will still tell you what it holds. This course never uses x, temp or df as a name.

len(vector_a) returns 2. The word len is short for length, which is unfortunate, because in this chapter “length” is about to mean something completely different. len() in Python counts how many numbers are in the list. The length in the mathematics is how far the arrow reaches, which is the subject of the next section. For $\mathbf{a} = (3,4)$ those two ideas give 2 and 5 respectively, and they are not the same question. Keep them apart in your head. Where this book means the Python count, it says “how many numbers”; where it means the arrow, it says “length”.

Look at vector_d = [6, 8] and compare it with vector_a = [3, 4]. Every number in $\mathbf{d}$ is exactly twice the matching number in $\mathbf{a}$. Drawn on paper, $\mathbf{d}$ points in the identical direction and goes twice as far. Hold on to that pair. It is the reason Chapter 9 exists, and it is going to cause trouble in section 8.5 before it gets fixed.

8.3 How long is the arrow? Pythagoras, one step at a time¶

Intuition¶

You have an arrow from the origin to $(3,4)$. How long is it?

The tempting first answer is 7, because you walked 3 blocks and then 4 blocks. That is the length of your walk. It is not the length of the arrow. The arrow is the straight line from where you started to where you finished, and a straight line is shorter than any route that turns a corner. You know this already from crossing a car park diagonally.

To get the straight-line distance, draw the picture and look at what you have made. Go 3 across. Turn. Go 4 up. Draw the arrow straight back to the start. Those three lines form a triangle, and the corner where you turned is a square corner, because “across” and “up” are at right angles by construction.

A triangle with a square corner in it is a right triangle, and there is a rule about right triangles that has been known for about two and a half thousand years. The rule says: square the two short sides, add those squares together, and the answer is the square of the long side. The long side is the one opposite the square corner, which is exactly our arrow.

So the recipe has three steps, and they are three separate steps.

1. Square each coordinate.

2. Add the squares together.

3. Take the square root of the total.

That last step undoes the squaring. Squaring turned lengths into areas, and the square root turns the area back into a length. If that sentence does not land yet, the arithmetic below will carry you anyway; you do not need the geometry to be vivid in order to do the calculation correctly.

One reassurance before the formula. Squaring a negative number gives a positive answer, because a negative times a negative is a positive. So a vector pointing left has the same length as the matching vector pointing right, which is what your eyes expect. Length never comes out negative. If yours does, you have made a sign error, and the most common one is squaring -4 and writing -16 instead of 16.

The mathematics¶

Formula 8.2: the length of a two-dimensional vector.

1. In words. Square each of the two numbers, add the two squares together, and take the square root of the total. The answer is how long the arrow is.

2. The formula.

3. The symbols.

4. Out loud. “The length of a is the square root of, a sub one squared plus a sub two squared.”

5. Worked, on $\mathbf{a} = (3, 4)$, made up for practice. Every step is written out. You can follow along on a phone calculator.

Step 1, square the first number.

Step 2, square the second number.

Step 3, add the two squares.

Step 4, take the square root of the total.

So $\lVert \mathbf{a} \rVert = 5$.

Worked again, on $\mathbf{b} = (4, 3)$. $4^{2} = 16$. Then $3^{2} = 9$. Then $16 + 9 = 25$. Then $\sqrt{25} = 5$. So $\lVert \mathbf{b} \rVert = 5$. Same length as $\mathbf{a}$, pointing a different way.

Worked again, on $\mathbf{c} = (-4, 3)$. This one has a minus sign in it, which is the step people fumble. $(-4)^{2} = (-4) \times (-4) = 16$. A negative times a negative is a positive. Then $3^{2} = 9$. Then $16 + 9 = 25$. Then $\sqrt{25} = 5$. So $\lVert \mathbf{c} \rVert = 5$ as well. Three vectors, three directions, one length.

Worked once more, on $\mathbf{d} = (6, 8)$. $6^{2} = 36$. Then $8^{2} = 64$. Then $36 + 64 = 100$. Then $\sqrt{100} = 10$. So $\lVert \mathbf{d} \rVert = 10$. Exactly twice the length of $\mathbf{a}$, which is what you would hope, since every one of its numbers is twice the matching number in $\mathbf{a}$.

6. Check it. Three checks, each of which catches a different slip.

A length can never be negative. Squaring removes every minus sign, and the square root sign in this formula always means the positive root. A negative answer is a sign error.

A length is always at least as large as the biggest single number in the list, ignoring the minus signs. For $\mathbf{d} = (6,8)$ the length 10 is bigger than 8, as it must be. If your length comes out smaller than one of the coordinates, you have added when you should have squared, or you have taken the root too early.

A length can only be zero when every number in the list is zero. There is exactly one vector with length zero and it is $(0,0)$, the arrow that does not go anywhere.

Now the same idea for a list of any length at all. Nothing changes except how many things you add up, so the formula needs a way to say “and keep adding”. That is what the sigma symbol is for.

Formula 8.3: the length of a vector in $D$ dimensions.

1. In words. Square every number in the list, add all the squares together, and take the square root of the total.

2. The formula.

3. The symbols.

4. Out loud. “The length of a is the square root of the sum, from i equals 1 to capital D, of a sub i squared.”

5. Worked, on $\mathbf{a} = (3,4)$ again, so you can see the sigma unfold into Formula 8.2.

Here $D = 2$, so the counter $i$ takes the value 1 and then the value 2, and then stops.

Step 1, write out what the sigma is asking for, one term per value of $i$.

Step 2, put the numbers in. $a_1 = 3$ and $a_2 = 4$.

Step 3, do the squaring. $3^2 = 9$ and $4^2 = 16$.

Step 4, take the square root.

Same answer as before, 5, from what looks like a more frightening formula. That is the point of showing both. Formula 8.3 is Formula 8.2 with the number of terms left unspecified.

Worked once more with $D = 4$, on the made-up vector $\mathbf{v} = (1, 2, 2, 4)$, to prove that nothing breaks when the list gets longer.

Step 1, square each of the four numbers. $1^2 = 1$, $2^2 = 4$, $2^2 = 4$, $4^2 = 16$.

Step 2, add all four squares. $1 + 4 = 5$. Then $5 + 4 = 9$. Then $9 + 16 = 25$.

Step 3, take the square root. $\sqrt{25} = 5$.

So $\lVert \mathbf{v} \rVert = 5$, even though $\mathbf{v}$ has four numbers in it and cannot be drawn on paper.

6. Check it. The sum under the square root is a sum of squares, so it can never be negative, so the square root is always a real number. If you find yourself asked for the square root of a negative number, you have subtracted somewhere instead of adding. And the answer is still a single number, no matter how long the list was. A length is always one number.

In Python¶

Here is the length of $\mathbf{a}$, computed the slow, visible way, with the running total printed after every step. An explicit for loop is used rather than any shortcut, because the loop is the thing being taught.

# The length of vector a, computed one coordinate at a time.
# Step 1 of the recipe: square each number and keep a running total.

sum_of_squares_for_a = 0                       # start the running total at zero
for one_coordinate in vector_a:                # visit 3, then visit 4
    sum_of_squares_for_a = sum_of_squares_for_a + one_coordinate * one_coordinate
    print("  running total after squaring", one_coordinate, "is", sum_of_squares_for_a)

# Step 2 of the recipe: take the square root of that total.
length_of_a = math.sqrt(sum_of_squares_for_a)

print("sum of squares for a:", sum_of_squares_for_a)
print("length of a:", length_of_a)

Output:

  running total after squaring 3 is 9
  running total after squaring 4 is 25
sum of squares for a: 25
length of a: 5.0

Read the loop out loud and it says what the formula says. “Start the total at zero. For each coordinate in the vector, add that coordinate times itself to the total.” The line for one_coordinate in vector_a: means “do the indented lines below once for each number in vector_a, and while you are doing them, let one_coordinate be that number”. The first time through, one_coordinate is 3. The second time, it is 4. Then the list runs out and the loop stops.

The line sum_of_squares_for_a = sum_of_squares_for_a + one_coordinate * one_coordinate looks circular and is not. The right-hand side is worked out first, using the current value, and the answer is then stored back under the same name. Inside that right-hand side the multiplication happens before the addition. That is the ordinary rule of arithmetic, the one that makes $2 + 3 \times 4$ equal 14 rather than 20, and Python obeys it. So the coordinate is squared first, and the square is added to the total second. First pass: $0 + 3 \times 3 = 9$. Second pass: $9 + 4 \times 4 = 25$. The printed running total lets you watch it happen.

math.sqrt(25) gives 5.0 rather than 5. The dot-zero is Python telling you the answer is a decimal number rather than a whole number, because a square root usually is one. It has not changed the value.

Now the other three vectors. The same six lines are written out again for each one rather than being folded into a reusable piece of code. Repeating them is the point: you get to see that nothing secret is happening and that each vector is handled identically.

# The same recipe for vector c. Watch what -4 squared does: it comes out positive.

sum_of_squares_for_c = 0
for one_coordinate in vector_c:
    sum_of_squares_for_c = sum_of_squares_for_c + one_coordinate * one_coordinate
length_of_c = math.sqrt(sum_of_squares_for_c)
print("sum of squares for c:", sum_of_squares_for_c)
print("length of c:", length_of_c)

# And again for vector d, which is exactly twice vector a.

sum_of_squares_for_d = 0
for one_coordinate in vector_d:
    sum_of_squares_for_d = sum_of_squares_for_d + one_coordinate * one_coordinate
length_of_d = math.sqrt(sum_of_squares_for_d)
print("sum of squares for d:", sum_of_squares_for_d)
print("length of d:", length_of_d)

Output:

sum of squares for c: 25
length of c: 5.0
sum of squares for d: 100
length of d: 10.0

Both answers match the by-hand arithmetic above: 5 for $\mathbf{c}$ and 10 for $\mathbf{d}$. The $\mathbf{c}$ case is the useful one to stare at. Its first coordinate is -4, and the running total still went up, because -4 * -4 is 16. Python applies the same rule about negatives that you applied on paper.

8.4 The dot product: one number for “do these two point the same way?”¶

Intuition¶

You now have two arrows and you want one number that says how much they agree.

Think about what agreement should mean. Two arrows pointing in exactly the same direction agree completely. Two arrows at a square corner to each other share nothing: moving along one gets you no distance at all in the direction of the other. Two arrows pointing in opposite directions actively disagree; one is undoing the other.

So the number you want should be big when they line up, around zero when they are at a square corner, and negative when they oppose each other. That is a lot to ask of a single number.

The dot product delivers it, and the recipe is much simpler than the description. Line up the two lists of numbers. Multiply the first number of one by the first number of the other. Multiply the second by the second. Add the two results. Done.

Why does that work? Look at what multiplication does to signs. If both arrows go right, their first coordinates are both positive, and a positive times a positive is a positive contribution. If one goes right and the other goes left, the first coordinates have opposite signs, and a positive times a negative is a negative contribution. Each coordinate votes, positively or negatively, on whether the two arrows agree along that direction, and the dot product collects the votes.

Two warnings before the arithmetic, because they save trouble later.

The dot product also grows when either arrow gets longer, not only when they line up better. A long arrow has big numbers in it, and big numbers make big products. So a large dot product can mean “these agree strongly” or it can mean “one of these is enormous”. Separating those two causes is the entire job of Chapter 9. This chapter builds the tool; the next chapter fixes its blind spot.

And the answer is one plain number, not a list. You put in two vectors and you get out a scalar, which is the word for an ordinary single number when you want to stress that it is not a list. The arrows are gone once you have the answer.

The mathematics¶

Formula 8.4: the dot product in two dimensions.

1. In words. Multiply the two first numbers together. Multiply the two second numbers together. Add those two answers.

2. The formula.

3. The symbols.

4. Out loud. “a dot b is a sub one times b sub one, plus a sub two times b sub two.”

5. Worked, on $\mathbf{a} = (3, 4)$ and $\mathbf{b} = (4, 3)$, made up for practice.

Step 1, write down the matching pairs so nothing gets crossed over. $a_1 = 3$ pairs with $b_1 = 4$. $a_2 = 4$ pairs with $b_2 = 3$.

Step 2, multiply the first pair.

Step 3, multiply the second pair.

Step 4, add the two products.

That 24 is a measured, recorded value in this course. It sits in lab/out/appendix_formulas_checks.json under vectors_2d, as dot_ab, and in _research/00-lab-verified-findings.md, section 6. The arithmetic you did on paper and the number the lab recorded are the same number, which is the modest and reassuring situation you want to be in with a formula this simple.

6. Check it. Four checks.

The answer is a single number. If you finished holding a list, you did the multiplications and forgot to add them up.

The order does not matter. $\mathbf{b} \cdot \mathbf{a} = 4 \times 3 + 3 \times 4 = 12 + 12 = 24$, the same answer. Swapping the two vectors can never change a dot product.

The pairing does matter. Multiplying $a_1$ by $b_2$ is a different calculation and gives a different, wrong answer: $3 \times 3 + 4 \times 4 = 9 + 16 = 25$, which is not 24. First with first, second with second, always.

Both vectors must be the same size. A vector with 2 numbers has no dot product with a vector of 3 numbers, because one of the pairs would have no partner. There is nothing clever to do about this; the operation is undefined and Python will refuse in its own way.

Now the general version, for lists of any length.

Formula 8.5: the dot product in $D$ dimensions.

1. In words. Walk along both lists together, position by position. At each position multiply the two numbers you find there. Add up all of those products.

2. The formula.

3. The symbols.

4. Out loud. “a dot b is the sum, from i equals 1 to capital D, of a sub i times b sub i.” Said less formally: “for every position, multiply the two numbers there, then add up everything.”

5. Worked, with $D = 4$, on the made-up vectors $\mathbf{p} = (2, 0, 1, 3)$ and $\mathbf{q} = (1, 5, 4, 2)$. Four dimensions cannot be drawn, and the arithmetic does not notice.

Step 1, position 1. $p_1 = 2$ and $q_1 = 1$, so $2 \times 1 = 2$. Running total: 2.

Step 2, position 2. $p_2 = 0$ and $q_2 = 5$, so $0 \times 5 = 0$. Running total: $2 + 0 = 2$.

Step 3, position 3. $p_3 = 1$ and $q_3 = 4$, so $1 \times 4 = 4$. Running total: $2 + 4 = 6$.

Step 4, position 4. $p_4 = 3$ and $q_4 = 2$, so $3 \times 2 = 6$. Running total: $6 + 6 = 12$.

Notice position 2 contributed exactly nothing, because one of its numbers was zero. A zero in one vector silently deletes whatever the other vector had at that position. That is worth remembering when you look at a 384-number sentence vector later and see numbers close to zero.

6. Check it. Count your multiplications before you add. For two vectors of $D$ numbers each there must be exactly $D$ products and then $D - 1$ additions. With $D = 4$ that is 4 multiplications and 3 additions, which is what the four steps above did. With $D = 384$ it is 384 multiplications and 383 additions, which is why you will hand that one to a computer.

In Python¶

The dot product in code is one loop with one line inside it. Here it is written the slow way, printing each product as it is formed, so you can match the screen against the paper.

# The dot product of vector a and vector b, one position at a time.
# range(len(vector_a)) produces the position numbers 0 and 1, in that order.

dot_product_of_a_and_b = 0                            # start the running total at zero
for coordinate_position in range(len(vector_a)):      # coordinate_position becomes 0, then 1
    one_product = vector_a[coordinate_position] * vector_b[coordinate_position]
    print("  position", coordinate_position, ":",
          vector_a[coordinate_position], "times",
          vector_b[coordinate_position], "=", one_product)
    dot_product_of_a_and_b = dot_product_of_a_and_b + one_product

print("a dot b =", dot_product_of_a_and_b)

Output:

  position 0 : 3 times 4 = 12
  position 1 : 4 times 3 = 12
a dot b = 24

This loop is shaped differently from the one in section 8.3, and the difference matters.

In section 8.3 the loop was for one_coordinate in vector_a, which handed you the numbers one at a time. That was enough, because the length formula only ever looks at one vector.

Here the loop is for coordinate_position in range(len(vector_a)), which hands you the position numbers one at a time: first 0, then 1. You need positions rather than numbers because you have to reach into two lists at the same position and pair them up. Handing you one number from vector_a would leave you with no way to find its partner in vector_b.

range(len(vector_a)) reads from the inside out. len(vector_a) is 2. range(2) produces the positions 0, 1. It stops before 2, which is exactly right, because a two-number list has positions 0 and 1 and nothing at position 2. Python’s habit of stopping one short looks like an off-by-one error until you notice that it makes range(len(...)) land on precisely the valid positions and no others.

The printed lines show $3 \times 4 = 12$ and $4 \times 3 = 12$, and the total shows 24. Those are the same three numbers you wrote on paper in step 2, step 3 and step 4 of the worked example. The code has no extra knowledge. It has patience.

8.5 Zero is a right angle, and what the sign is telling you¶

Intuition¶

Take $\mathbf{a} = (3,4)$, which points up and to the right. Now take $\mathbf{c} = (-4,3)$, which points up and to the left.

Draw both. The corner between them looks square, and it is. If you rotate $\mathbf{a}$ by a quarter turn anticlockwise, you land exactly on $\mathbf{c}$. That is not a coincidence about these particular numbers; swapping the two coordinates and flipping the sign of one of them is what a quarter turn does.

Their dot product is zero, and the arithmetic showing why is worth watching closely. The first coordinates are 3 and -4, which disagree, and they contribute -12. The second coordinates are 4 and 3, which agree, and they contribute +12. The agreement and the disagreement are exactly equal, so they cancel, and nothing is left.

That is what a right angle is, in arithmetic. Not “a small amount of agreement”. Not “almost nothing in common”. It is the precise balance point between leaning together and leaning apart, the fence between the two.

This gives you a reading of the sign of any dot product, and it is the single most useful thing to carry out of this chapter.

• Positive: the arrows lean the same way. The larger it is, the more they agree, or the longer they are, or both.

• Zero: square corner. They share no direction whatsoever.

• Negative: they lean apart. They are pointing, to some degree, against each other.

Now the trap, and this chapter is going to walk you straight into it on purpose. Compare $\mathbf{a} \cdot \mathbf{b} = 24$ with $\mathbf{a} \cdot \mathbf{d} = 50$. The second is much larger. Does that mean $\mathbf{d}$ agrees with $\mathbf{a}$ more than $\mathbf{b}$ does?

It does, and for an uninteresting reason. $\mathbf{d} = (6,8)$ is $\mathbf{a}$ doubled. It points in exactly the same direction and is twice as long. The 50 is partly agreement in direction and partly nothing more than size. A raw dot product cannot tell you which portion is which, and for comparing the meanings of sentences that is a real defect, because a longer sentence should not count as a better match. Chapter 9 removes the size effect by dividing by both lengths. Notice the shape of the problem now, so that the fix lands as a fix rather than as a ritual.

The mathematics¶

Here is the sign rule written as a small table, because it deserves to be somewhere you can find it again.

Worked example 8.1: the three dot products of this chapter, side by side.

All three use $\mathbf{a} = (3,4)$, made up for practice. All three answers are recorded in lab/out/appendix_formulas_checks.json under vectors_2d.

With $\mathbf{b} = (4, 3)$:

With $\mathbf{c} = (-4, 3)$:

With $\mathbf{d} = (6, 8)$:

Three answers, 24, 0 and 50, from three arrows that all have the same starting point. The lengths are $\lVert \mathbf{a} \rVert = 5$, $\lVert \mathbf{b} \rVert = 5$, $\lVert \mathbf{c} \rVert = 5$ and $\lVert \mathbf{d} \rVert = 10$, all computed in section 8.3. The only vector with a different length is $\mathbf{d}$, and $\mathbf{d}$ is the one with the surprising dot product. That is not an accident and Chapter 9 is about it.

There is one more fact worth having, because it ties this section back to section 8.3 and because it is the cleanest sanity check in the chapter.

Formula 8.6: a vector dotted with itself is its length squared.

1. In words. If you take the dot product of a vector with itself, you get the same number you would get by measuring its length and then squaring that.

2. The formula.

3. The symbols.

4. Out loud. “a dot a equals the length of a, squared.”

5. Worked, on $\mathbf{a} = (3,4)$, made up for practice. Both sides computed separately, so you can see them meet.

Left-hand side, using the dot product recipe with $\mathbf{a}$ in both slots. Step 1, $a_1 \times a_1 = 3 \times 3 = 9$. Step 2, $a_2 \times a_2 = 4 \times 4 = 16$. Step 3, add. $9 + 16 = 25$. So $\mathbf{a} \cdot \mathbf{a} = 25$.

Right-hand side, using the length from section 8.3 and then squaring it. Step 1, the length. $\lVert \mathbf{a} \rVert = \sqrt{3^2 + 4^2} = \sqrt{25} = 5$. Step 2, square it. $5^{2} = 5 \times 5 = 25$. So $\lVert \mathbf{a} \rVert^{2} = 25$.

Both sides give 25. They agree.

6. Check it. They must agree, always, for every vector, because the dot product of a vector with itself squares each coordinate and adds them, and the length squares each coordinate, adds them, takes a square root, and then squares it again, which undoes the root. If your two sides disagree, one of the two calculations has a slip in it, and this formula has told you so without needing to know which.

This also explains why a vector can never have a negative dot product with itself. Every term is a number multiplied by itself, and that is never negative.

In Python¶

Here are the other two dot products, computed with the same loop as before, written out again in full.

# The dot product of vector a and vector c, printing each product as it is formed.

dot_product_of_a_and_c = 0
for coordinate_position in range(len(vector_a)):
    one_product = vector_a[coordinate_position] * vector_c[coordinate_position]
    print("  position", coordinate_position, ":",
          vector_a[coordinate_position], "times",
          vector_c[coordinate_position], "=", one_product)
    dot_product_of_a_and_c = dot_product_of_a_and_c + one_product
print("a dot c =", dot_product_of_a_and_c)

# The dot product of vector a and vector d, without the per-position printing this time.

dot_product_of_a_and_d = 0
for coordinate_position in range(len(vector_a)):
    one_product = vector_a[coordinate_position] * vector_d[coordinate_position]
    dot_product_of_a_and_d = dot_product_of_a_and_d + one_product
print("a dot d =", dot_product_of_a_and_d)

Output:

  position 0 : 3 times -4 = -12
  position 1 : 4 times 3 = 12
a dot c = 0
a dot d = 50

The two printed products for $\mathbf{a}$ and $\mathbf{c}$ are -12 and +12, and the total is 0. The cancellation described in the intuition section is right there on the screen. It is not that the products were small. They were as large as the ones that produced 24. They pointed opposite ways and destroyed each other.

Now the check from Formula 8.6, which is the single most valuable line of code in this chapter, because it will catch a slip in either of your two recipes.

# Formula 8.6, checked: a dot a should equal the length of a, squared.

dot_product_of_a_and_a = 0
for coordinate_position in range(len(vector_a)):
    one_product = vector_a[coordinate_position] * vector_a[coordinate_position]
    dot_product_of_a_and_a = dot_product_of_a_and_a + one_product

print("a dot a =", dot_product_of_a_and_a)
print("length of a, squared =", length_of_a * length_of_a)

Output:

a dot a = 25
length of a, squared = 25.0

Both sides give 25. One of them prints as a whole number and one prints as 25.0, because length_of_a came out of math.sqrt and carries a decimal point with it. The values are the same; the display is different. Do not read 25 and 25.0 as a disagreement.

Run this same check on any vector you invent for yourself. If the two sides ever fail to match, one of the two loops has a typo in it, usually a vector_b where a vector_a should be.

The simulation: drag the arrows and watch the number move¶

Everything in the last three sections is one picture, and the picture moves. Drag the tip of either arrow and watch the readouts change underneath.

Three things to do with it, in this order.

First, confirm the arithmetic you already did. The arrows start at $\mathbf{a} = (3,4)$ and $\mathbf{b} = (4,3)$, and the Dot product readout shows 24.00. That is your $12 + 12$.

Second, press the button marked “Set b to (-4, 3)”. The dot product falls to 0.00, and the arc drawn between the two arrows becomes a square corner. Check the arithmetic against the screen: $3 \times (-4) = -12$, $4 \times 3 = 12$, and the two cancel.

Third, drag the tip of $\mathbf{b}$ slowly all the way around the origin and watch only the sign. It stays positive while $\mathbf{b}$ leans toward $\mathbf{a}$, passes through zero at the square corner, and goes negative once $\mathbf{b}$ leans away. The number does not lurch at zero. It slides through it. Zero is a crossing, not a cliff.

The panel also shows readouts labelled Cosine and Angle. Leave those alone for now. They are what Chapter 9 is about, and they are on the screen so that you have already seen them once before that chapter asks you to care about them.

8.6 Three hundred and eighty-four directions: a real sentence vector¶

Intuition¶

Everything so far used two numbers because two numbers fit on graph paper. A real model does not use two. The one in this chapter uses 384.

Start with what “how many numbers” has meant so far, and keep walking.

One number locates you on a line. How far along a street you are.

Two numbers locate you on a map. Blocks east, blocks north.

Three numbers locate you in a room. East, north, and how high off the floor.

Three is where drawing stops. There is no fourth direction to point your pencil in, and there is not going to be one.

Four numbers still describe something perfectly ordinary. A day in Kern County could be summarised by four numbers: the high temperature, the humidity, the wind speed, and the air quality index. That is a list of four numbers standing for one day. Two days with similar lists were similar days. Nobody needs to draw a four-dimensional picture in order to say that.

That is the entire idea, and it is smaller than it sounds. “Dimension” means “how many numbers are in the list.” It does not mean a direction you could walk in. It does not mean a parallel universe. When someone says a sentence vector lives in 384-dimensional space, they are saying the list has 384 numbers in it. The word “space” is a habit of speech borrowed from the two-dimensional and three-dimensional cases where you really can point.

So: you cannot picture 384 directions at once. Nobody can, including the people who build these models. What you can do is add up 384 products, and every formula in this chapter is written so that it does not care how many there are. Formula 8.3 says $\sum_{i=1}^{D} a_i^2$ with $D$ left open. Formula 8.5 says $\sum_{i=1}^{D} a_i b_i$ with $D$ left open. Put $D = 2$ and you get graph paper. Put $D = 384$ and you get a sentence. The formula does not change and neither does the recipe.

One more thing before the arithmetic, so that the last section does not oversell itself. The individual numbers in a sentence vector do not mean anything on their own. There is no column for “is about cats” and no column for “mentions a county”. The 384 numbers were produced by a training process that nobody hand-designed, and reading number 57 tells you nothing. The meaning lives in the whole pattern, which is why the only sensible question is a comparison between two patterns, and why the dot product is the tool.

The mathematics¶

Normalising matters here because of a measured fact. The model used in this chapter hands back rows that are already unit vectors. The lab checked this directly: for the first sentence, the sum of the squares of all 384 numbers came out as 1.000000, so the length came out as 1.000000. That is recorded in lab/out/appendix_python_reference_checks.json under the key normalise, as sum_of_squares and vector_length.

It is not luck. The same JSON file records the three pieces this model is built from, under modules_in_embedding_model: Transformer, Pooling, and Normalize. The third piece does the normalising, every time, as the last thing the model does.

Formula 8.7: normalising a vector.

1. In words. Divide every number in the list by the length of the whole list. The result points the same way and has length 1.

2. The formula.

3. The symbols.

4. Out loud. “a hat is the vector a divided by the length of a, which means every single number in a gets divided by that one length.”

5. Worked, on $\mathbf{a} = (3, 4)$, made up for practice.

Step 1, find the length. From section 8.3, $\lVert \mathbf{a} \rVert = \sqrt{9 + 16} = \sqrt{25} = 5$.

Step 2, divide the first number by that length.

Step 3, divide the second number by the same length.

Step 4, write the answer.

6. Check it. Compute the length of your answer. It must be exactly 1.

Step 1, $0.6^{2} = 0.6 \times 0.6 = 0.36$. Step 2, $0.8^{2} = 0.8 \times 0.8 = 0.64$. Step 3, $0.36 + 0.64 = 1.00$. Step 4, $\sqrt{1.00} = 1$.

The length is 1, so the normalising worked. If your answer’s length is not 1, you divided by something other than the true length, and the usual culprit is dividing by the sum of squares (25 here) instead of by its square root (5 here).

Two consequences of unit length, both of which you should carry into Chapter 9.

The first is that the largest a dot product between two unit vectors can ever be is 1, reached when the two vectors are identical. The smallest it can be is -1, reached when one is exactly the reverse of the other. So for unit vectors, the dot product is already trapped between -1 and 1, and it is already a clean measure of direction with no size effect left in it.

The second follows from the first. When both vectors have length 1, dividing by the two lengths means dividing by $1 \times 1$, which changes nothing. So for this particular model, the dot product of two rows is already the similarity score. Chapter 9 introduces the division step properly and shows what goes wrong without it. This chapter can read the numbers off directly because the model has done the normalising already.

In Python¶

Now the real model. This block downloads about 87 megabytes the first time you run it and is instant every time after that.

# Load the embedding model. The first run downloads it into the HF_HOME folder from Cell 1.
embedding_model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

# Six sentences, chosen for this course. Three pairs: two about pets, two about Kern County,
# and two about nothing in common with anything.
sentence_list = [
    "The cat sat on the mat.",
    "A kitten rested on the rug.",
    "Bakersfield is in Kern County, California.",
    "Kern County's largest city is Bakersfield.",
    "The stock market fell sharply on Tuesday.",
    "Photosynthesis converts light into chemical energy.",
]

# encode() turns every sentence in the list into its own row of numbers, all at once.
sentence_vectors = embedding_model.encode(sentence_list)

print("shape:", sentence_vectors.shape)
print("first six numbers of sentence 0:", sentence_vectors[0][:6])

# Count the model's parameters: the numbers it learned during training.
embedding_parameter_count = 0
for one_parameter_block in embedding_model.parameters():
    embedding_parameter_count = embedding_parameter_count + one_parameter_block.numel()
print("parameters in the embedding model:", f"{embedding_parameter_count:,}")

Output:

shape: (6, 384)
first six numbers of sentence 0: [ 0.13023718 -0.01577282 -0.03671669  0.05798642 -0.05979175  0.0330537 ]
parameters in the embedding model: 22,713,216

Read that output slowly, because it is the payoff for the whole chapter.

shape: (6, 384) says the result is a grid with 6 rows and 384 columns. One row per sentence, 384 numbers per row. “The cat sat on the mat.” has six words and “Photosynthesis converts light into chemical energy.” has six words, and a thirty-word sentence would also have come back as 384 numbers. That fixed width is what makes the arithmetic possible: two lists of different lengths have no dot product, so every sentence has to produce the same count.

sentence_vectors[0][:6] reads as “row 0, then the first six columns of it”. The colon inside the square brackets is a slice, and [:6] means “from the start, up to but not including position 6”. Six of the 384 are printed so the line fits on the page. Look at them: some positive, some negative, all small. Not one of them means anything by itself.

The last four lines of that cell count the parameters, and they use two words that belong to the model rather than to Python. embedding_model.parameters() hands back the model’s stored blocks of learned numbers, one block at a time, so the for loop visits them in turn. .numel() is short for “number of elements”, and it reports how many numbers are inside one block. The loop adds those counts up, starting from zero, which is the same running-total pattern you used for the sum of squares in section 8.3. The last line prints the total. The f in front of the quotation marks and the :, inside the braces are Python’s instruction to write the number with commas between the thousands, so that 22713216 arrives on the screen as 22,713,216 and can be read.

22,713,216 is how many numbers this model learned during training. Compare that with Qwen2.5-0.5B-Instruct from Chapter 3, which holds 494,032,768. Divide the larger count by the smaller one, 494,032,768 divided by 22,713,216, and the answer is 21.75. So the embedding model is about one twenty-second the size, and it takes 87 megabytes of disk (lab/out/appendix_python_reference_checks.json, key disk_usage, 91,578,455 bytes). That number is worth holding on to for the cost conversation this course runs every week. Turning a sentence into a vector is a far cheaper operation than generating text, and Chapter 10 builds a whole retrieval system on exactly that asymmetry.

Now check the claim about unit length, on the real row, with the same loop you used on $(3,4)$.

# The length of row 0, computed the same way as the length of vector a in section 8.3.
# The only difference is that this loop runs 384 times instead of 2.

sum_of_squares_for_row_0 = 0.0
for one_number in sentence_vectors[0]:
    sum_of_squares_for_row_0 = sum_of_squares_for_row_0 + float(one_number) * float(one_number)

length_of_row_0 = math.sqrt(sum_of_squares_for_row_0)

print("sum of squares for row 0: %.6f" % sum_of_squares_for_row_0)
print("length of row 0:          %.6f" % length_of_row_0)

Output:

sum of squares for row 0: 1.000000
length of row 0:          1.000000

The loop is the same loop. The recipe is the same recipe. It ran 384 times instead of twice and it did not need to be told anything new.

float(one_number) converts each entry into an ordinary Python decimal number before multiplying. The model hands its numbers back in a compact storage format, and converting keeps the arithmetic here in plain Python where you can see it. "%.6f" % value prints a number to exactly six decimal places, which is how this book shows that 1.000000 is a measured value rather than a rounded 1.

Finally, the number the chapter opened with.

# The dot product of the two Kern County sentences: row 2 and row 3.
# Both rows have length 1, so this dot product is already a similarity score between -1 and 1.

dot_product_of_row_2_and_row_3 = 0.0
for coordinate_position in range(len(sentence_vectors[2])):
    dot_product_of_row_2_and_row_3 = (dot_product_of_row_2_and_row_3
        + float(sentence_vectors[2][coordinate_position])
        * float(sentence_vectors[3][coordinate_position]))
print("row 2 dot row 3: %.6f" % dot_product_of_row_2_and_row_3)

# The two pet sentences, which share no content words at all.
dot_product_of_row_0_and_row_1 = 0.0
for coordinate_position in range(len(sentence_vectors[0])):
    dot_product_of_row_0_and_row_1 = (dot_product_of_row_0_and_row_1
        + float(sentence_vectors[0][coordinate_position])
        * float(sentence_vectors[1][coordinate_position]))
print("row 0 dot row 1: %.6f" % dot_product_of_row_0_and_row_1)

# A pet sentence against a Kern County sentence: unrelated.
dot_product_of_row_0_and_row_2 = 0.0
for coordinate_position in range(len(sentence_vectors[0])):
    dot_product_of_row_0_and_row_2 = (dot_product_of_row_0_and_row_2
        + float(sentence_vectors[0][coordinate_position])
        * float(sentence_vectors[2][coordinate_position]))
print("row 0 dot row 2: %.6f" % dot_product_of_row_0_and_row_2)

# The stock market against photosynthesis: the one that comes out negative.
dot_product_of_row_4_and_row_5 = 0.0
for coordinate_position in range(len(sentence_vectors[4])):
    dot_product_of_row_4_and_row_5 = (dot_product_of_row_4_and_row_5
        + float(sentence_vectors[4][coordinate_position])
        * float(sentence_vectors[5][coordinate_position]))
print("row 4 dot row 5: %.6f" % dot_product_of_row_4_and_row_5)

Output:

row 2 dot row 3: 0.833303
row 0 dot row 1: 0.612421
row 0 dot row 2: 0.084388
row 4 dot row 5: -0.016586

Worked example 8.2: reading the four measured numbers.

Every one of these four numbers is the sum of 384 products, and every one of them is recorded in lab/out/we5_embeddings.json in the similarity_matrix.

0.833303, the two Kern County sentences. These two sentences are about the same fact and they share three words. The model puts them close to the same direction. This is the largest score among the six sentences, other than each sentence with itself.

0.612421, “The cat sat on the mat.” against “A kitten rested on the rug.” These two sentences share no content words. Not “cat”, not “sat”, not “mat”. A method that counted overlapping content words would report zero here and be wrong. The model reports 0.612421, which is most of the way to the Kern County pair, and it got there without a dictionary, a thesaurus or a rule about cats. This single number is why the course spends three weeks on geometry.

0.084388, a cat sentence against a Kern County sentence. Small and positive. Both are ordinary English sentences about physical things in the world, and that shared quality is worth a little. It is not zero and it should not be. Two unrelated English sentences still have more in common than an English sentence and a random list of numbers.

-0.016586, the stock market against photosynthesis. Negative. The two arrows lean slightly apart. Nothing dramatic is being claimed by a number that small; the honest reading is “these two have nothing to do with each other, and the model’s estimate of how much they share landed a hair below zero”. What matters is the contrast with 0.833303. The gap between the biggest and the smallest of these four numbers is what makes them useful for sorting.

Common mistakes¶

Seven things that go wrong in this chapter, with how to spot each one.

1. Adding the coordinates instead of squaring them. For $(3,4)$ this gives $3 + 4 = 7$ rather than 5. How to spot it: your length is bigger than it should be, and it equals the sum of the coordinates. The straight line between two points is always shorter than a route with a corner in it, so a length of 7 for a journey of 3 then 4 cannot be right.

2. Forgetting the square root. For $(3,4)$ this gives 25 rather than 5. How to spot it: your length is much larger than the biggest coordinate. A length is normally in the same neighbourhood as the coordinates, not five times bigger.

3. Squaring a negative number and keeping the minus sign. Writing $(-4)^2 = -16$ instead of 16. How to spot it: your sum of squares comes out negative, or smaller than it should, and then the square root fails or the length is too small. A square is never negative.

4. Pairing the wrong coordinates in a dot product. Multiplying $a_1$ by $b_2$. For $\mathbf{a} = (3,4)$ and $\mathbf{b} = (4,3)$ this gives 25 instead of 24, which is close enough to look plausible. How to spot it: check your answer against $\mathbf{b} \cdot \mathbf{a}$, computed from scratch. Swapping the vectors cannot change a correct dot product, and it does change a mispaired one.

5. The Python index shift. Writing vector_a[1] when you mean $a_1$. How to spot it: your code runs happily and gives the wrong answer, which is the worst kind of bug. $a_1$ is vector_a[0], $a_2$ is vector_a[1]. Write it down somewhere you can see it.

6. Confusing len() with length. len(vector_a) is 2, the count of numbers. $\lVert \mathbf{a} \rVert$ is 5, the reach of the arrow. How to spot it: your “length” comes out as a small whole number that is the same for every 2-D vector you try. If every vector you test has length 2, you have computed len.

7. Treating a big dot product as proof of strong agreement. $\mathbf{a} \cdot \mathbf{d} = 50$ is bigger than $\mathbf{a} \cdot \mathbf{b} = 24$, and the reason is that $\mathbf{d}$ is twice as long, not that it agrees twice as well. How to spot it: compute the lengths. If one vector is much longer than the other, the raw dot product is not yet a fair comparison. Chapter 9 is the fix.

What to remember¶

A vector is an ordered list of numbers and an arrow from the origin, and those are two ways of holding the same object. The length of the arrow is Pythagoras: square every number, add the squares, take the square root, so $(3,4)$ has length $\sqrt{9+16} = \sqrt{25} = 5$. The dot product multiplies matching coordinates and adds the results, giving one number that is positive when two arrows lean the same way, zero when they meet at a right angle, and negative when they lean apart. Every one of these formulas is written with $D$ left open, so a list of 384 numbers is handled by the identical arithmetic as a list of 2, which is how a real model scored two sentences sharing no words at 0.612421. The raw dot product still grows when either arrow grows, and removing that size effect is the whole business of Chapter 9.

Practice problems¶

Twenty-six problems in three tiers. Warm-up asks whether you can do the arithmetic. Practice asks whether you can apply it. Stretch asks whether you can reason with it.

Answers to the odd-numbered problems are in the Answers appendix. Every vector invented in these problems is made up for practice unless the problem names a file in lab/out/, in which case the number is a real measurement and you can open the file and check it.

Warm-up¶

1. Write down the coordinates of the point that is 7 to the right of the origin and 2 below it.

2. Say which of $(5, -2)$ and $(-2, 5)$ is farther to the right, and which is higher up.

3. Compute $\lVert \mathbf{u} \rVert$ for $\mathbf{u} = (6, 8)$. Show the squaring, the addition and the square root as three separate lines.

4. Compute $\lVert \mathbf{u} \rVert$ for $\mathbf{u} = (0, 7)$.

5. Compute $\lVert \mathbf{u} \rVert$ for $\mathbf{u} = (-3, -4)$. State what the two minus signs do to the answer.

6. Compute $\mathbf{p} \cdot \mathbf{q}$ for $\mathbf{p} = (2, 5)$ and $\mathbf{q} = (3, 1)$.

7. Compute $\mathbf{p} \cdot \mathbf{q}$ for $\mathbf{p} = (1, 0)$ and $\mathbf{q} = (0, 1)$. Say in one sentence what your answer means about the two arrows.

8. Compute $\mathbf{p} \cdot \mathbf{q}$ for $\mathbf{p} = (4, -2)$ and $\mathbf{q} = (-1, 3)$.

9. Compute $\mathbf{v} \cdot \mathbf{v}$ for $\mathbf{v} = (2, 3)$, then compute $\lVert \mathbf{v} \rVert$ and square it. Confirm that Formula 8.6 holds.

10. Compute the dot product of the four-dimensional vectors $\mathbf{s} = (1, 2, 0, 4)$ and $\mathbf{t} = (3, 1, 7, 2)$. Show all four products and the running total.

Practice¶

11. Normalise $\mathbf{u} = (0, 5)$ using Formula 8.7. Then verify that the length of your answer is exactly 1.

12. Normalise $\mathbf{u} = (3, 4)$ and then normalise $\mathbf{d} = (6, 8)$. Compare the two answers and explain in one sentence why they came out the way they did.

13. Find a vector perpendicular to $(7, -2)$. Prove it is perpendicular with a dot product. Then find a second, different perpendicular vector.

14. Show that $\mathbf{a} \cdot \mathbf{d} = 50$ for $\mathbf{a} = (3,4)$ and $\mathbf{d} = (6,8)$, that $\mathbf{a} \cdot \mathbf{b} = 24$ for $\mathbf{b} = (4,3)$, and that $\mathbf{a} \cdot \mathbf{a} = 25$. Then explain, in two sentences, why the gap between 50 and 24 is not by itself a measurement of direction. Use the third answer in your explanation: $\mathbf{a}$ points in exactly its own direction and still scores only 25, so most of the 50 is length rather than direction.

15. A vector $\mathbf{g}$ has $\lVert \mathbf{g} \rVert = 5$. Every number in $\mathbf{g}$ is then multiplied by 3, giving a new vector $\mathbf{h}$. Without knowing what $\mathbf{g}$'s numbers are, state $\lVert \mathbf{h} \rVert$ and explain how you know.

16. Two vectors have $D = 384$ and both have length 1. State the largest value their dot product could possibly take, the smallest, and what each of those two extremes would mean about the two sentences.

17. Open lab/out/we5_embeddings.json and find the similarity matrix. Read off the value in row 0, column 3, and the value in row 3, column 0. Explain why those two numbers are the same.

18. Using the same file, list the six sentences in order of how similar they are to “Bakersfield is in Kern County, California.”, most similar first. Exclude the sentence’s comparison with itself.

19. State how many multiplications and how many additions are needed for one dot product between two vectors with $D = 384$. Then state the same two counts for $D = 2$, and say what stayed the same between the two cases.

20. Open the simulation and drag $\mathbf{b}$ until the Dot product readout is as close to 0.00 as you can get it. Record the coordinates of $\mathbf{b}$ and the Angle readout. Then check your coordinates by computing the dot product by hand.

Stretch¶

21. Invent two two-dimensional vectors whose dot product is negative, with neither vector having a zero in it. Draw them roughly. Then write one sentence explaining what a negative dot product would be saying if these were two sentences rather than two arrows.

22. The lab measured the dot product of the stock market sentence and the photosynthesis sentence at -0.016586 (lab/out/we5_embeddings.json, row 4, column 5). Write a short paragraph on what that number does and does not license you to claim. Address in particular whether it means the model believes the two sentences contradict each other.

23. A classmate says: “384 dimensions means the model has worked out 384 different facts about each sentence.” Write a reply of three or four sentences. Say what is wrong with the claim and what would be a more accurate way to describe the 384 numbers.

24. Suppose an embedding model did not normalise its output, so that longer sentences came back as longer vectors. Explain, using the $\mathbf{a}$ and $\mathbf{d}$ example from this chapter, what would go wrong if you ranked search results by raw dot product. Then state what you would need to divide by to repair it. This is the argument Chapter 9 makes in full.

25. The embedding model in this chapter holds 22,713,216 parameters (lab/out/appendix_python_reference_checks.json, key embed). The retrieval work in Chapter 10 also used bge-small-en-v1.5, which holds 33,360,000 (_research/00-lab-verified-findings.md, section 11). Compute the ratio of the larger to the smaller, to two decimal places. Then, in two sentences, say why a course whose subject is access and resource cost cares about a ratio like that.

26. Qwen2.5-0.5B-Instruct holds 494,032,768 parameters (lab/out/we3_params_quant.json) and takes 953 megabytes of disk, while the embedding model in this chapter holds 22,713,216 parameters and takes 87 megabytes (both disk figures from lab/out/appendix_python_reference_checks.json, key disk_usage). Compute both ratios, parameters and megabytes, to one decimal place. The two ratios are not equal. Suggest one reason why the disk figures might not scale exactly with the parameter counts. Chapter 6 has the answer if you want to check yourself.

What is next¶

You now have a number that says how much two arrows agree, and one defect in it: the number grows when an arrow grows, whether or not the direction improved. That defect is fatal for comparing sentences, because a longer sentence is not a better answer.

Chapter 9 repairs it in one step, by dividing by both lengths. The repair has a name, cosine similarity, and the measured case that proves it is needed is already in this chapter: $\mathbf{a} = (3,4)$ and $\mathbf{d} = (6,8)$ point in precisely the same direction, and their raw dot product is 50 while $\mathbf{a}$ with itself is only 25. After the division, both come out at exactly 1.0000.