Meaning as a Direction
The corpse question
A person can ask where does a corpse rot? and see that the founding line about bodies, husks and fallen things answers it. A text search cannot: the question has five words, the line has twenty-three, and none of the words appear in both lists.
A model can turn any piece of text into a fixed-length row of numbers, and what
that row carries is a direction; two texts that mean nearly the same thing come back
pointing nearly the same way, and the angle between the rows is the comparison.
Lowercasing, stemming and rare-word weights cannot manufacture a link between
corpse and bodies. The link is in what they mean, and meaning is
not a property of a string.
Length is a fact about the model's arithmetic, not about the text. A comparison that lets length in is partly comparing the wrong thing, so the row has to be reduced to its direction before one row is held against another.
The measuring is arithmetic this book did years ago in two numbers. Volume 3 built a
vector as a pair kept together with operations of its own: Len, which is the
square root of the sum of the squares, and Unit, which divides both numbers
by that length so the result reaches exactly one and points the same way. Both rules are
written for two numbers and neither of them cares how many there are. A row of 768 is
the same sum with more terms in it, and by the end of this page it will be running over
rows that a model wrote.
Four things will exist that do not now. A file in internal/lang with the dot
product, the length, the normalisation and the cosine in it, worked first on vectors of
three whole numbers that anybody can check with a pencil. One more method on the client
from earlier in this volume, which asks a model for a row instead of for a sentence, and
which is how the width of a row gets read off a real answer instead of asserted. A
committed fixture of the chronicle's own lines turned into numbers, so that every
similarity printed in this book is computed over bytes that do not move. And a ranking:
sixteen lines of this world's history sorted against a question, with the line that has
no word in common with it at the top.
It is fair to ask why this world needs any of it today. Ninety-one lines fit in a terminal, and nobody has to search a file they can read. The answer is that the chronicle is the one thing here that only grows: it takes an entry every year of simulated time and it never rewrites one, so a valley left running unattended for a month has a history no person is going to read in order. Being able to put a question to it in ordinary English and get the four lines it is about is useful the moment it exists, on the corpus that exists, and it needs nothing else built first to be useful.
One boundary, drawn now so that nothing later has to be taken back. Nothing on this page feeds a result to a model. A question comes in, rows come back, the rows get printed, and that is the end of it. No answer is composed, no line of the chronicle is handed to anything that generates text, and nobody in The Hollow says a word. What this builds is a search over a file that you run by asking in English: it remembers nothing, weighs nothing and decides nothing, it sorts sixteen rows and prints the top of the list, and that is a whole thing on its own.
The three-number vectors
Nothing about this measurement needs 768 of anything. It needs at least two numbers to have a direction at all, and three is enough to see everything the operation does, so the arithmetic gets built and checked at three before a model is asked for anything. The vectors below are chosen so that every length is a whole number and every answer is a fraction with a small denominator. A reader who works them out on paper and gets what the program got has checked the program, which is a better position to be in than trusting it.
The whole file is four functions. The first is the dot product: multiply the two rows together number by number and add up what comes out.
// internal/lang/near.go
// Dot is the sum of the products, pairwise, and it is refused when
// the two rows are not the same width.
//
// Two vectors of different widths have no angle between them at all,
// and the widths that turn up in practice are two different models'.
// Returning a number for that pair would be answering a question
// nobody asked.
func Dot(a, b []float32) (float64, error) {
if len(a) != len(b) {
return 0, fmt.Errorf("lang: a row of %d numbers against a row of %d", len(a), len(b))
}
if len(a) == 0 {
return 0, fmt.Errorf("lang: a row of no numbers has no direction")
}
var s float64
for i := range a {
s += float64(a[i]) * float64(b[i])
}
return s, nil
}// internal/lang/near.go
// Length is how far the row reaches from the origin: the square root
// of the sum of the squares, which is the same rule this book has
// used for a two-number vector since the valley learned to move.
func Length(a []float32) (float64, error) {
s, err := Dot(a, a)
if err != nil {
return 0, err
}
return math.Sqrt(s), nil
}// internal/lang/near.go
// Cos is the dot product over the product of the two lengths.
//
// On rows that are already at length one the two divisions are by
// one and this is the dot product; the divisions stay in anyway,
// because whether a server hands back unit rows is a fact about that
// server on that day and not a promise anybody made.
func Cos(a, b []float32) (float64, error) {
d, err := Dot(a, b)
if err != nil {
return 0, err
}
na, err := Length(a)
if err != nil {
return 0, err
}
nb, err := Length(b)
if err != nil {
return 0, err
}
if na == 0 || nb == 0 {
return 0, fmt.Errorf("lang: no angle between a row of zeros and anything")
}
return d / (na * nb), nil
}$ go run ./cmd/near -mode paper
near: five directions against one question, in three numbers each
the question (3, 4, 0)
the row length dot cosine
the question itself (3, 4, 0) 5.0000
the same direction, twice as long (6, 8, 0) 10.0000 50.0000 1.0000
a little to one side (4, 3, 0) 5.0000 24.0000 0.9600
square on to it (-4, 3, 0) 5.0000 0.0000 0.0000
straight back the other way (-3, -4, 0) 5.0000 -25.0000 -1.0000
pointing elsewhere, and long (12, 9, 20) 25.0000 72.0000 0.5760
ranked on the dot product alone
1 72.0000 length 25.00 pointing elsewhere, and long
2 50.0000 length 10.00 the same direction, twice as long
3 24.0000 length 5.00 a little to one side
4 0.0000 length 5.00 square on to it
5 -25.0000 length 5.00 straight back the other way
ranked on the cosine
1 1.0000 length 10.00 the same direction, twice as long
2 0.9600 length 5.00 a little to one side
3 0.5760 length 25.00 pointing elsewhere, and long
4 0.0000 length 5.00 square on to it
5 -1.0000 length 5.00 straight back the other way
the two orders disagree about which row is first
and the row the dot product put there is the longest one
Take the second row by hand. Its numbers are 6, 8 and 0, so its length is the square root of 36 + 64 + 0, which is the square root of 100, which is 10. Its dot product with the question is 3×6 + 4×8 + 0×0 = 18 + 32 = 50. The question's own length is the square root of 9 + 16, which is 5, and 50 divided by 5×10 is exactly 1. That row is the question doubled: same direction, twice as far along it, and the cosine says the angle between them is nothing at all.
The last row is the one to sit with. Its numbers are 12, 9 and 20, its length is the square root of 144 + 81 + 400 = 625, which is 25, and its dot product with the question is 36 + 36 + 0 = 72. Seventy-two is the largest dot product on the table, larger than the 50 belonging to the row that is the question exactly. And its cosine is 0.576, which puts it third. It won the first ranking by being long, and it lost the second one by pointing somewhere else, and both of those are the same fact about it.
All of the above was arithmetic in plain numbers, and here is the shorthand for it. A vector is still written as its numbers in brackets, in order, exactly as volume 3 wrote (3, 4). It has more of them now, and a row of 768 gets written (a1, a2, …, a768) so that a single number in it can be pointed at.
The multiply-and-add operation gets a dot: a · b, said "a dot b".
Written out for three numbers it is
a·b = a1b1 + a2b2 +
a3b3, and for (3, 4, 0) and (6, 8, 0) that is
18 + 32 + 0 = 50. Writing out 768 of those terms is not practical, so the sum gets a
symbol of its own, a capital sigma, with the counter underneath it and where to stop on
top: ∑i=1n aibi means take
every i from 1 to n, multiply the two numbers in that position, and
add all the results together. That is the loop in Dot, written in one
line.
Length comes for free from the same operation, because a·a is every
number multiplied by itself and added up, which is what goes under the square root:
|a| = √(a·a). For (12, 9, 20) that is
√(144 + 81 + 400) = √625 = 25. Volume 3's rule for two numbers is this rule
with the sum stopped early.
And the measurement this chapter is about:
cos θ = (a · b) ÷ (|a| × |b|)
The Greek letter θ, said "theta", is the usual name for an angle, and
cos θ is a number between −1 and 1 that says how far apart two
directions are: 1 for the same direction, 0 for square on, −1 for straight back
the other way. The one thing to notice is where the lengths went. Both of them are
divided out, which means multiplying either vector by any positive number leaves the
answer untouched: (6, 8, 0) and (3, 4, 0) and (300, 400, 0) all score 1 against the
question, because they are the same direction at three different distances.
The last piece of shorthand is the hat from volume 3, and it now earns its keep.
â means a with every number divided by |a|, so that
|â| = 1. If both rows have already had that done to them, both divisions
in the rule above are divisions by one, and cos θ is
â · b̂. That is why systems that compare millions of these normalise
everything once on the way in: it turns every later comparison into a bare
multiply-and-add.
The 768-number embedding row
Real rows come from a model, and asking for one takes a method the client from earlier in this volume did not have. It is shorter than the one that asks for a sentence, and the difference is the interesting part: there is nothing to configure.
// internal/lang/serve.go
// Embed turns one line of text into one row of numbers.
//
// It asks for the row the model computed and does no arithmetic on
// it. Some servers hand back a row already at length one and some do
// not, and a caller that assumes either is a caller whose ranking
// changes when somebody upgrades something. This world reads the row
// as it comes, keeps it as it came, and divides the length out at the
// moment it compares two of them.
func (c *Client) Embed(ctx context.Context, model, text string) ([]float32, error) {
if model == "" {
return nil, fmt.Errorf("lang: an embedding with no model named")
}
if text == "" {
return nil, fmt.Errorf("lang: an embedding of no text at all")
}
body, err := json.Marshal(embedReq{Model: model, Prompt: text})
if err != nil {
return nil, fmt.Errorf("lang: %s: %w", model, err)
}
var res embedRes
if err := c.post(ctx, "/api/embeddings", body, &res); err != nil {
return nil, err
}
if len(res.Embedding) == 0 {
return nil, fmt.Errorf("lang: %s: the server answered with a row of no numbers", model)
}
return res.Embedding, nil
}No temperature, no seed, no token limit, no context length. Everything the generating method had to argue about was there because something was being sampled, and nothing is being sampled here: the model reads the text, runs it forward once, and reduces what comes out to one row. There is no distribution to draw from and no draw to make. That makes an embedding a much duller thing to ask for than a sentence, and much easier to reason about.
The comment on the method is doing real work and it is the reason the length is not divided out here. A server that returns rows at length one has made a choice, and it is a choice it can revise in a release note. Code that leans on it reads correctly, ranks correctly, and then one day ranks by length instead, with nothing in the diff to say so. Keeping the row as it arrived and dividing at the comparison costs two square roots per comparison and cannot break that way.
Now the question nothing in this world has answered yet: how wide is a row? The kit names three models and pins each by the digest of its weights, and it says nothing about widths, because a width is not a setting anybody chooses. It belongs to the model. So the bench asks for one row and reports everything about the answer except the answer.
$ podman exec -w /bench world-go go run ./cmd/near -mode live -q 1
near: one line sent to a real model, and everything about the answer except the answer
the server world-lm:11434
the model nomic-embed-text:v1.5
pinned at 970aa74c0a90ef74
the text "Where does a corpse rot?"
what came back
numbers 768
of them finite 768
at length one, within 1e-06 false
at length nothing false
against the committed row for the same text
same width true
every number within 1e-06 true
identical, number for number true
768 is this model's own width and nothing here chose it
none of the numbers is on this page; yours would not be these ones anyway
768. Every figure in the rest of this chapter is that number, and it was read off an answer rather than looked up. A different embedding model answers with a different width and every one of those widths is that model's, which is the same fence this volume put around tokens a second and around bits a weight: a number measured against one model is not a number about another one, whatever the two have in common.
It is also a different kind of width from the ones this volume has been reading out of model files, and mixing the two up is easy. A generating model has a width inside it, the size of the vector each token is turned into on its way through the stack, and that number belongs to the file and to the arithmetic that reads it. The 768 above is the width of what comes out of a model built to do one job: read a whole piece of text and produce one row that stands for the lot. A model of the first kind can be asked to do this too, and answers at a width of its own, which is a third number in the same conversation and about neither of the others.
The third line of the first block is the one that saves trouble later. This server does
not hand back rows at length one. It could have; several do; this one gives the numbers
the model computed and leaves the arithmetic to whoever asked. Had that line come back
true, everything on this page would still be right, and the code would
still divide, and the difference would only show up on the day somebody swapped the
server.
The rest of the run is a comparison and never a printout. It says the live answer is the same width as the committed one, that no number in it is more than a millionth away from the committed one, and that on this machine today the two are identical to the last bit. What it does not do is print any of the 768 numbers, and that restraint is the rule this whole chapter is built on. Every float a model produced stays off the page, and yours would be different numbers anyway.
Which raises the problem that the rest of the volume has to solve. A book that computes a similarity from a live model prints a number nobody else can reproduce: not a reader on different hardware, not the same reader after an upgrade, and not this world's own verification a year from now. The answer is the one this volume already used for a header too big to carry. Ask once, write it down, and compute everything afterwards over the bytes that were written.
It is useful to be exact about what that buys, because it is the difference between a chapter of arithmetic and a chapter of anecdotes. A row out of a live model depends on the machine, the build of the server, the arithmetic the backend chose and how the request happened to be batched, and none of those is under a reader's control or under this book's. A similarity computed from such rows is therefore not a number anybody can check. A similarity computed from committed rows is exactly as checkable as any other division: the bytes are in the repository, the loop is nine lines, and a run of it here and a run of it on a machine built ten years from now print the same digits. Every score, every ranking and every byte count on the rest of this page is of the second kind.
// exemplars/exemplars.go
//go:embed chronicle.jsonl hero.header.json small.header.json vectors.json
var Files embed.FS
// Vectors is the name of the committed embedding fixture inside
// Files.
const Vectors = "vectors.json"// exemplars/exemplars.go
// Vec is one embedded piece of text: what was sent, and the row that
// came back.
//
// Entry and Kind are filled in for a chronicle line and empty for a
// question, because a question is not in the chronicle and never
// becomes one. Nothing here is written back to the world.
type Vec struct {
Entry int64 `json:"entry,omitempty"`
Kind string `json:"kind,omitempty"`
Text string `json:"text"`
V []float32 `json:"v"`
}$ podman exec -w /bench world-go go run ./cmd/near -mode fix
near: the chronicle's rows turned into numbers, once, and written down
the model nomic-embed-text:v1.5
pinned at 970aa74c0a90ef74
the server world-lm:11434, waiting up to 10m0s for an answer
the corpus 16 of the chronicle's 91 rows, and 4 questions
every answer was 768 numbers wide, which is the model's own and not a setting
written to near87-vectors.json, 173584 bytes
against the copy this build carries
exemplars/vectors.json ac91f2b9ab427785 173584 bytes
near87-vectors.json ac91f2b9ab427785 173584 bytes
the 15360 numbers this run was given, against the committed ones, at a tolerance of 1e-06
not one of them differs at all
Sixteen chronicle lines and four questions, twenty rows of 768 numbers, 173,584 bytes
on disk. The lines are the eleven founding entries and five of the eighty yearly
counts, chosen as a list and not as a rule: a rule like every tenth row would
embed a different set the day the chronicle grew, and every number in this chapter
would move with it. The file records which model produced it and at which digest, and
it carries its own label, so that a reader who finds it in ten years knows what it is
without finding this page: Yours will differ: a different model, build, backend or
machine answers with different numbers.
The last two blocks of the run are the check that keeps the fixture honest, and they are the same check the header dumps in this volume get. The mode does the work again against the live server, writes its own file under a name of its own, and holds the result against the copy the build carries. Here the two files came out byte for byte identical, which is a fact about this machine on this day and not a promise: the guarantee that travels is the one the tolerance line makes, that no number moved by more than a millionth. Take the model away and the arithmetic in this chapter still runs, on the same bytes, to the same digits, on any machine.
With the rows carried, the first thing to look at is the quantity this chapter keeps dividing out.
$ go run ./cmd/near -mode table
near: the committed rows, and how long each of them is
the model nomic-embed-text:v1.5
pinned at 970aa74c0a90ef74
the rows 16 chronicle lines and 4 questions, 768 numbers each
entry chars length the line
1 323 17.5036 This is The World. I built it and I run it, and o...
2 194 20.0103 The land is The Hollow. It is one valley with sto...
3 161 19.5147 The animals are hobbs. One of them is a hobb. The...
4 210 19.3554 Asteria keeps it. She is not down in the valley a...
5 178 19.0002 People gather at Firstlight. Nobody is there and ...
6 96 20.3982 The way in is The Commons. Anybody who comes here...
7 92 20.8602 The doors between one part of the ground and anot...
8 181 19.1976 What changes hands is grain. Nothing here trades ...
9 96 20.1037 Past the mapped ground are the Unwritten Places. ...
10 123 19.9950 What is finished with goes to the Midden. Bodies,...
11 138 19.4362 The thing worth going after is the Wellspring. I ...
12 60 20.1230 year 1: 11 hobbs standing in The Hollow, 45 born ...
16 62 20.4787 year 5: 43 hobbs standing in The Hollow, 252 born...
21 65 20.2115 year 10: 191 hobbs standing in The Hollow, 1013 b...
51 66 19.9020 year 40: 235 hobbs standing in The Hollow, 1773 b...
91 66 20.1346 year 80: 238 hobbs standing in The Hollow, 1965 b...
chars length the question
24 22.4438 Where does a corpse rot?
31 22.5440 How does a newcomer get inside?
26 22.5639 Do the animals have names?
28 22.5054 How many creatures survived?
the shortest row is 17.5036 and the longest 20.8602, a spread of 19.2%
and none of that spread is about what the line says
Read the length column against the character column and look for a relationship. Entry 1 is the longest line in the fixture at 323 characters and it has the shortest row, 17.5036. Entry 7 is one of the shortest lines at 92 characters and has the longest row, 20.8602. Entry 12 and entry 91 are the same kind of sentence with different numbers in it, 60 characters against 66, and their rows come out 20.1230 and 20.1346. The four questions, which are shorter than every line in the table, all sit above 22.4.
So length is not text length, and it is not information content, and it is not importance. It is what the model's last few operations happened to produce, and the nineteen percent spread across these sixteen rows is a spread in something nobody asked for. That is the quantity the cosine divides away, and the reason for dividing it away is not tidiness: it is that leaving it in means ranking partly on it.
The chronicle search ranking
Everything is in place. The rows are on disk, the arithmetic is four functions, and the question at the top of this chapter can now be put to sixteen lines of this world's history. The bench scores every row against the question, sorts, and prints the top of the list with a column nobody would normally put in a search result: how many words each line has in common with the question that found it.
$ go run ./cmd/near -mode ask -q 1
near: the chronicle's rows ranked against one question
the question Where does a corpse rot?
the rows 16, of 768 numbers each, from nomic-embed-text:v1.5
scored by the cosine: the dot product over the two lengths
rank cosine length entry words the line
1 0.602859 19.9950 10 0 What is finished with goes to the Midden. B...
2 0.533825 19.5147 3 1 The animals are hobbs. One of them is a hob...
3 0.516247 20.0103 2 0 The land is The Hollow. It is one valley wi...
4 0.474781 20.8602 7 1 The doors between one part of the ground an...
5 0.451704 17.5036 1 1 This is The World. I built it and I run it,...
6 0.449923 20.1346 91 0 year 80: 238 hobbs standing in The Hollow, ...
the line at the top is entry 10
and it has not one word in common with the question
It found it. Entry 10 is the line about the Midden, it is at the top by a clear
margin, and the words column on its row is a zero. The second place is the
line about the animals, which is a reasonable thing to return for a question about a
corpse; the third is the line describing the valley's ground, which is where things end
up; and the sixth is a yearly count with a number of deaths in it. The list degrades
the way a person would degrade it.
None of this is a lookup. Nothing in the fixture was tagged death, nobody
wrote a synonym list, and the word corpse does not occur anywhere in the
chronicle. What happened is that a model that has read a great deal of English puts
texts about the same subject in nearly the same direction, and a question is a text.
The whole search is one multiply-and-add per row and a sort.
Now the same rows and the same question, ranked the way a search that matches text would have to rank them.
$ go run ./cmd/near -mode words -q 1
near: the same rows ranked by the words they share with the question
the question Where does a corpse rot?
its words where does a corpse rot
the rows 16 chronicle lines
rank words cosine entry which words the line
1 1 0.451704 1 a This is The World. I built it and I r...
2 1 0.533825 3 a The animals are hobbs. One of them is...
3 1 0.394670 4 does Asteria keeps it. She is not down in ...
4 1 0.442220 5 a People gather at Firstlight. Nobody i...
5 1 0.474781 7 a The doors between one part of the gro...
6 0 0.516247 2 The land is The Hollow. It is one val...
the angle puts entry 10 first; the words put it 10 of 16
and 5 rows are tied at the top of the word ranking, on one word each
The best any line manages is one word, and in four of the five cases that word is
a. Those five are tied at the top on one word each, and the line the
question was actually about comes tenth of sixteen with a score of zero, level with ten
other lines that have nothing to do with it either. A stop-word list would take
the four a rows out and leave the ranking even flatter. Nothing that can
be done to a word-matching search rescues this query, because the query and its answer
have no words in common at all, and that is not an unusual thing for a person to
type.
Second question, and it is the same result from a different direction.
$ go run ./cmd/near -mode ask -q 2 -top 4
near: the chronicle's rows ranked against one question
the question How does a newcomer get inside?
the rows 16, of 768 numbers each, from nomic-embed-text:v1.5
scored by the cosine: the dot product over the two lengths
rank cosine length entry words the line
1 0.636975 20.3982 6 0 The way in is The Commons. Anybody who come...
2 0.531150 20.8602 7 1 The doors between one part of the ground an...
3 0.467180 19.0002 5 1 People gather at Firstlight. Nobody is ther...
4 0.447553 20.1037 9 0 Past the mapped ground are the Unwritten Pl...
Entry 6 is the founding line about The Commons, the way in, and again the question and the line share no word at all. What is nice about this one is the second and third places: Waygates, which are the doors between one part of the ground and another, and Firstlight, which is where people gather when they get here. A reader looking for how somebody arrives would want all three of those, roughly in that order.
The two ways of searching are not always opposed, and a chapter that only ever showed them disagreeing would be selling something. Here is a question whose words the chronicle does use.
$ go run ./cmd/near -mode words -q 3 -top 4
near: the same rows ranked by the words they share with the question
the question Do the animals have names?
its words do the animals have names
the rows 16 chronicle lines
rank words cosine entry which words the line
1 2 0.449691 1 the names This is The World. I built it and I r...
2 2 0.445739 2 the have The land is The Hollow. It is one val...
3 2 0.722571 3 the animals The animals are hobbs. One of them is...
4 2 0.439437 4 the have Asteria keeps it. She is not down in ...
the angle puts entry 3 first; the words put it 3 of 16
and 6 rows are tied at the top of the word ranking, on 2 words each
Do the animals have names? is a question the word ranking can nearly do. Entry
3 is the right answer, it does contain animals, and it comes third. The
trouble is the company: six rows are tied on two words each, and the other five got
there on the plus one of have or names. The
cosine column beside them tells the same story with no ties at all: entry 3 scores
0.722571 and the rest of that tied group sits between 0.412 and 0.450. One method has
the right answer buried in a six-way tie and the other has it clear of the field by a
quarter.
And here is a question the angle handles honestly and does not answer.
$ go run ./cmd/near -mode ask -q 4 -top 4
near: the chronicle's rows ranked against one question
the question How many creatures survived?
the rows 16, of 768 numbers each, from nomic-embed-text:v1.5
scored by the cosine: the dot product over the two lengths
rank cosine length entry words the line
1 0.636633 19.5147 3 0 The animals are hobbs. One of them is a hob...
2 0.536719 20.4787 16 0 year 5: 43 hobbs standing in The Hollow, 25...
3 0.505581 20.1230 12 0 year 1: 11 hobbs standing in The Hollow, 45...
4 0.484232 19.9020 51 0 year 40: 235 hobbs standing in The Hollow, ...
The top line is the founding entry about the animals, which contains no number at all. The three lines behind it are yearly counts, which contain nothing but numbers, and they are in no particular order of year. A search by meaning found the lines that are about the subject, and that is all it did: it has no idea which of them holds a number, and no notion that a question beginning "how many" wants one. Ranking by subject is what this is for, and expecting an answer out of it is expecting something it was never doing.
Why this works
Underneath all of it is one substitution that turns up wherever a computer has to compare things it cannot compare directly. Two texts have no arithmetic between them. Two rows of numbers have all the arithmetic anybody wants. So the problem gets moved: put every text somewhere in a space with many directions in it, arrange for texts about the same subject to land near each other, and then the question are these two about the same thing becomes the question how far apart are these two directions, which is a division and a square root.
The arranging is the model's contribution and the reader did not do it. What the reader does own, entirely, is the measurement, and it is small enough to hold in your head. The same substitution shows up with different words attached wherever there are vectors of counts: two documents compared by which words are in them, two players compared by which items they bought, two songs compared by who listened. In every one of those the raw dot product rewards magnitude, which is a stand-in for volume rather than for likeness, and dividing by the two lengths is the standard repair.
The reason it is the standard repair needs to be stated plainly, because it is not obvious that dividing by lengths should mean anything. The dot product of two rows is already almost the right measurement: it is large when the two agree about which positions are big, and negative when they disagree. What spoils it is that it is also large when either row is large in every slot. Dividing by both lengths removes exactly that and nothing else. The result is bounded between −1 and 1 whatever the rows, which is what makes scores from different queries comparable at all, and which is the property that lets a system put a threshold on the number and refuse anything under it.
The one thing the paper example cannot show is why the rows are so wide. Three numbers give a direction and a perfectly good angle, and they also give very little room: with three, any large collection of subjects has to crowd, and two things with nothing to do with each other end up pointing much the same way because there is nowhere else for the second one to point. Widen the row and the crowding eases, because there are more independent ways for two texts to differ. Seven hundred and sixty-eight is somebody else's answer to how much room is enough, arrived at by training and not by argument, and the reason it matters to a reader is this: the width is what makes a score of 0.60 mean closest of sixteen by a distance instead of closest of sixteen because everything is close to everything.
One property of these numbers deserves a caution, and the run above supplied it. A cosine of 0.60 is a good score in this corpus and it is nothing like a fraction of a right answer. The scores are comparable within one query, against one model, over one set of rows, and 0.45 in this table means the sixth best of sixteen rather than forty-five percent related. The barrier that has governed every measured number in this volume governs these too: a similarity is about the model that produced the rows, and two models' scores do not belong on the same axis.
Why cosine needs normalised rows
The cosine is two operations, and one of them is optional in the sense that the code runs without it. A programmer who writes the multiply-and-add loop and stops has written something that compiles, returns a number, sorts a list and looks right. Here is what it actually does, on the fixture, with the same question.
The bench takes a flag that scores on the dot product alone, with neither length divided out. Everything else about the run is the same.
$ go run ./cmd/near -mode ask -q 1 -raw
near: the chronicle's rows ranked against one question
the question Where does a corpse rot?
the rows 16, of 768 numbers each, from nomic-embed-text:v1.5
scored by the dot product alone, with neither length divided out
rank dot length entry words the line
1 270.541254 19.9950 10 0 What is finished with goes to the Midden. B...
2 233.807164 19.5147 3 1 The animals are hobbs. One of them is a hob...
3 231.850915 20.0103 2 0 The land is The Hollow. It is one valley wi...
4 222.284454 20.8602 7 1 The doors between one part of the ground an...
5 203.474198 20.4787 16 0 year 5: 43 hobbs standing in The Hollow, 25...
6 203.318534 20.1346 91 0 year 80: 238 hobbs standing in The Hollow, ...
the line at the top is entry 10
and it has not one word in common with the question
The right answer is still first. That is the worst possible outcome, and it is the reason this failure is worth ten seconds of anybody's time: the missing division did not break the run, it broke the reasoning behind the run, and a test that only asserted "entry 10 comes first" would pass. The damage is further down. Entry 16 has climbed into fifth on a length of 20.4787 while entry 1, which the cosine put fifth, has dropped out of the top six on a length of 17.5036. The lengths in these rows only vary by nineteen percent, so a nineteen percent thumb is on the scale, and it is enough to move rows around without ever being visible.
To see what the scoring is actually sorting on, give it rows whose lengths differ by
more than nineteen percent. There is an ordinary way for that to happen and it needs no
invention: somebody stores rows the way the server sent them, somebody later changes
the client to normalise on the way in, and the table ends up holding both. The
-twin flag builds exactly that, by adding every row a second time at
length one and marking the copy with a tick.
$ go run ./cmd/near -mode ask -q 1 -twin -raw -top 8
near: the chronicle's rows ranked against one question
the question Where does a corpse rot?
the rows 32, of 768 numbers each, from nomic-embed-text:v1.5
and again every row a second time at length one, marked with a tick
scored by the dot product alone, with neither length divided out
rank dot length entry words the line
1 270.541254 19.9950 10 0 What is finished with goes to the Midden. B...
2 233.807164 19.5147 3 1 The animals are hobbs. One of them is a hob...
3 231.850915 20.0103 2 0 The land is The Hollow. It is one valley wi...
4 222.284454 20.8602 7 1 The doors between one part of the ground an...
5 203.474198 20.4787 16 0 year 5: 43 hobbs standing in The Hollow, 25...
6 203.318534 20.1346 91 0 year 80: 238 hobbs standing in The Hollow, ...
7 202.510987 20.1230 12 0 year 1: 11 hobbs standing in The Hollow, 45...
8 201.902405 20.1037 9 0 Past the mapped ground are the Unwritten Pl...
the line at the top is entry 10
and it has not one word in common with the question
16 rows come before the first one that was stored at length one, out of 16
which is every one of them, so the first thing this ranking sorted on was length
no row and its twin score more than 1.58e-09 apart, the worst of them entry 5
Thirty-two rows, sixteen of them the same sixteen lines. The first sixteen places all go to rows that were stored as the server sent them, and the seventeenth is the first one stored at length one. Not one of the normalised copies gets anywhere near the top, including the normalised copy of the line the question is actually about, which scored 270.541254 in its long form and would have to beat 201.902405 in its short one with a score that cannot exceed 1. Every twenty-long row, however irrelevant, beats every one-long row, however exact. The ranking sorted on length first and on meaning only within a group that happened to share one.
Reason back from the symptom and the cause is in the arithmetic, not in the data. The dot product of two rows scales with either of them: double one row and its score doubles, whatever it says. So a score built from it is a mixture of two quantities, how well the directions agree and how big the rows are, and nothing in the number afterwards can separate them again. It looks fine on a table where every row came from one model through one code path on one day, because then the lengths are all similar and the mixture is mostly the part you wanted. It is a ranking waiting for a corpus with any variation in it.
The repair is one line, and here it is doing its work on the same thirty-two rows.
$ go run ./cmd/near -mode ask -q 1 -twin -top 4
near: the chronicle's rows ranked against one question
the question Where does a corpse rot?
the rows 32, of 768 numbers each, from nomic-embed-text:v1.5
and again every row a second time at length one, marked with a tick
scored by the cosine: the dot product over the two lengths
rank cosine length entry words the line
1 0.602859 1.0000 10' 0 What is finished with goes to the Midden. B...
2 0.602859 19.9950 10 0 What is finished with goes to the Midden. B...
3 0.533825 19.5147 3 1 The animals are hobbs. One of them is a hob...
4 0.533825 1.0000 3' 1 The animals are hobbs. One of them is a hob...
the line at the top is entry 10
and it has not one word in common with the question
0 rows come before the first one that was stored at length one, out of 16
no row and its twin score more than 1.58e-09 apart, the worst of them entry 5
The list comes out in pairs, each row beside its own twin, and every pair scores the same to six places. No row and its twin are more than 1.58e-09 apart across the whole table, which is float32 rounding in the normalising and nothing else. That is the strongest available statement that the length carried no information: two rows that differ by a factor of twenty are, to this measurement, the same row.
The arithmetic is now load-bearing for anything this world does with a question, so it gets tests, and none of them needs a server. Four of them run over the committed rows, which is the substitution this volume has now made three times: bytes on disk standing in for something expensive, and the code under test unable to tell the difference.
$ go test ./internal/lang/ -run 'TestCosineOnThreeNumbersIsWhatThePaperSays|TestARowOfZerosHasNoDirectionToKeep|TestTwoRowsOfDifferentWidthsAreRefused|TestScalingARowChangesNoCosine|TestEveryCommittedRowIsTheModelsOwnWidth|TestTheDotProductAloneRanksByLength|TestTheLineTheAngleFindsSharesNoWordWithTheQuestion' -v
=== RUN TestCosineOnThreeNumbersIsWhatThePaperSays
near_test.go:56: five rows against (3,4,0): every length whole, every cosine exact to twelve places
--- PASS: TestCosineOnThreeNumbersIsWhatThePaperSays (0.00s)
=== RUN TestARowOfZerosHasNoDirectionToKeep
near_test.go:64: refused: lang: a row of 8 zeros has no direction to keep
--- PASS: TestARowOfZerosHasNoDirectionToKeep (0.00s)
=== RUN TestTwoRowsOfDifferentWidthsAreRefused
near_test.go:82: refused, and the message says both widths: lang: a row of 768 numbers against a row of 896
--- PASS: TestTwoRowsOfDifferentWidthsAreRefused (0.00s)
=== RUN TestScalingARowChangesNoCosine
near_test.go:117: 16 rows scaled by four factors over a millionfold range, no cosine moved by more than 1.84e-09
--- PASS: TestScalingARowChangesNoCosine (0.00s)
=== RUN TestEveryCommittedRowIsTheModelsOwnWidth
near_test.go:137: 16 rows and 4 questions, 768 numbers each, from nomic-embed-text:v1.5
--- PASS: TestEveryCommittedRowIsTheModelsOwnWidth (0.00s)
=== RUN TestTheDotProductAloneRanksByLength
near_test.go:176: on the dot product the first 16 places are the 16 rows that were never normalised; on the cosine the top two are entry 10 and its own twin
--- PASS: TestTheDotProductAloneRanksByLength (0.00s)
=== RUN TestTheLineTheAngleFindsSharesNoWordWithTheQuestion
near_test.go:207: "Where does a corpse rot?" found entry 10 with no word in common, while the best word match anywhere in the corpus had 1
near_test.go:207: "How does a newcomer get inside?" found entry 6 with no word in common, while the best word match anywhere in the corpus had 1
--- PASS: TestTheLineTheAngleFindsSharesNoWordWithTheQuestion (0.00s)
The fourth is the property the whole chapter rests on, and it is checked over real rows instead of over an example picked to make it work: every committed row multiplied by 0.001, by 0.5, by 7.5 and by 1000, and no cosine anywhere moved by more than 1.84e-09. A millionfold range of lengths, and the measurement did not notice.
The sixth pins the failure above so that nobody quietly repairs it into a passing test with the division still missing, and the seventh pins the claim this chapter is named for: the line the angle finds for either of the first two questions shares no word with the question, while the best word match anywhere in the corpus is one word. The two refusals are the third and second: a comparison between rows of different widths, which is what two models in one table produces, and a row of nothing but zeros, which points nowhere and would otherwise be scored against everything and score zero against everything, reading as an answer.
One refusal is a deliberate departure from what volume 3's two-number vector does, and
the difference says something about both. Vec2.Unit hands a zero step back
unchanged, because a thing in the valley that is not moving is an ordinary situation
and stopping the tick over it would be absurd. A row of 768 zeros out of a model is not
an ordinary situation. It is a bug somewhere upstream, and the earliest possible
complaint is the useful one.
- Work out the length of (12, 9, 20) and its dot product with (3, 4, 0) on paper, and turn the two into the cosine 0.576 without running anything.
- Say why that row wins a ranking scored on the dot product and comes third on the cosine, in one sentence that mentions no code.
- Say where the number 768 came from in this chapter, and what would have to happen for it to be a different number.
- Explain why a committed fixture is what makes a similarity printable in a book, and what specifically could not be printed without one.
- Given a table holding some rows at length one and some at length twenty, predict the first twenty places of a ranking scored without normalising, before looking.
- Name two things the cosine over these rows found that a word-matching search could not, and one thing it did not find that a person asking would have wanted.
Exercise 1 — find where the right answer went. Run the twin experiment scored on the dot product alone and print all thirty-two places. Say what rank the normalised copy of the line about the Midden lands on, what its score is against the row directly above it, and what would have to be different about a table for a ranking like this one to put a wrong line first.
-mode ask -q 1 -twin -raw -top 32. The normalised copy of entry 10 is
seventeenth, scoring 13.530451, and the row above it is the founding line about
grain scoring 157.300168, which is about trade and has nothing to do
with the question. Nearly twelve times the score, for a line that is wrong.
Every one of the sixteen rows ahead of the right answer is ahead of it on length.
For a wrong line to take first place, the table only has to hold the right row in its short form and any wrong row in its long one, which is precisely what a table filled by two clients that disagreed about normalising contains. A backfill run before and after a client change does it. A migration that copied rows in from somewhere else does it. Two services with their own copies of the same helper do it. The table has no column that would show any of that, and the ranking is the first symptom.
Exercise 2 — find the corpus's own limits. Ask the fixture the four questions it carries and then work out, from the rankings alone, which parts of this world's history the sixteen rows cannot answer anything about. Predict what a question about grain would return before you look.
Eleven of the sixteen rows are the founding entries and five are yearly counts written to one pattern, so anything the chronicle records in words is answerable and anything it records only as a number is not. How many creatures survived? returns the line about the animals first and three counts behind it, in no order of year, because nothing in a row of numbers says which count is which.
A question about trade returns entry 8, the founding line that names grain, and it will return it clearly, because that line is the only one in the fixture about exchange at all. Which is also the whole limitation: a question about how the price of anything moved would return the same line with the same confidence, and the chronicle does not hold a single price. A ranking always returns its best candidates, and it has no way at all to say that its best is not an answer.
Exercise 3 — normalise once instead of twice a comparison. Every comparison in this chapter computes two square roots it could have computed earlier. Work out how many square roots a search over n rows costs as written, how many it would cost with the rows stored at length one, and what you would have to give up to store them that way.
As written, one query over n rows costs n dot products and 2n square roots, and n of those recompute the same row lengths every single query. Normalising on the way in makes it n dot products and no square roots at all, because both lengths are one and the division is by one. Over a chronicle of any size that is the difference between a search that scales and one that does arithmetic nobody needed.
What you give up is the ability to tell later what the model actually returned. The original length is gone, so a row stored that way cannot be checked against a fresh answer number for number, and a table mixing normalised and unnormalised rows has nothing in it that says which is which. The middle course is what this chapter's code does: keep what arrived, and cache each row's length once beside it, which is one float per row and takes the square roots back out of the query.