Every Word Looks at Every Other
The bare digit token
One line of this world's history encodes to seven numbers and decodes back to the same
sixty-six bytes. That is a good encoding; it is still not reading. The gap shows up in
token 2, 0, because by itself it says nothing useful about the year
40 it helps spell.
A head keeps two kinds of weight: three stored matrices fixed before any row arrives, and attention weights worked out from the row itself every time one token reads another. Volume 5's network cannot do that second job. Its weights are numbers in a slice, identical for every row the creature ever sees, so they can say that one input counts double but not that this particular token needs the year beside it.
The first four tokens make the problem concrete. Token 1 is year 4, token 2
is 0, token 3 is thirty-eight bytes running from a colon to the middle of a
four-digit number, and token 4 is 7. Two are single digits, but neither is a
number a reader can use alone: the 0 finishes a year, and the 7
sits inside a birth count.
A volume 5 neuron multiplies each input by a stored weight, adds the products with a bias and returns the total. A layer is a row of those neurons, and a creature controller is three hundred and seventy-eight numbers in fixed loops. It can drive an animal around a valley for eighty years because the numbers stay stable enough for evolution to tune.
That sentence is the whole of what is new here, and everything after it is arithmetic
small enough to check with a pencil. By the end of the page there is a Head
in internal/lang: three matrices, a dot product, a divide, a softmax and a
weighted sum, run over four tokens with every intermediate number printed and every one of
them worked by hand first. Then the same head runs at four, eight, sixteen and thirty-two
tokens with its multiply-adds counted as it goes, and the reason nobody offers a machine
that reads an unlimited amount of text at once stops being a rule somebody imposed and
becomes a column in a table.
Nothing on this page starts a container, dials anything or loads a model. The valley is untouched again: not one simulation package gains a line, and the sealed-import test the last chapter wrote reads the same six packages and finds the same nothing.
Three matrices over four tokens
A token arrives as an integer, and arithmetic on an integer standing for a run of bytes is meaningless: token 383 is not seven times token 55 in any sense anybody can use, and nothing sensible happens if you average them. So every token in the vocabulary gets a short row of numbers instead, and the integer's only remaining job is to say which row. In a trained model that table of rows is learned along with everything else. Here it is typed, four numbers a token, and this page says so plainly instead of leaving it to be inferred: nothing on this page works out what these ought to be, and the arithmetic below does not care where they came from.
The four slots were chosen so that the arithmetic has something to find. Slot 1 is 1 when the token is a digit standing on its own. Slot 2 is 1 when it carries a year. Slot 3 is 1 when it carries a count of animals. Slot 4 says roughly how much of a sentence the token stands for. On these four tokens that gives token 1 a year, token 3 a count and most of a sentence, and tokens 2 and 4 nothing at all except the first slot.
$ go run ./cmd/attend -mode row
attend: the row this head reads, by a table of 512
the first 4 tokens of one chronicle entry
year 40: 235 hobbs standing in The Hollow, 1773 born and 1567 gone
n token bytes stands for
1 331 6 "year 4"
2 48 1 "0"
3 383 38 ": 235 hobbs standing in The Hollow, 17"
4 55 1 "7"
the four numbers each of them arrives with, typed by hand
n a bare digit carries a year carries a count how much it says
1 "year 4" 0.00 1.00 0.00 0.50
2 "0" 1.00 0.00 0.00 0.00
3 ": 235 hobbs standing... 0.00 0.00 1.00 1.00
4 "7" 1.00 0.00 0.00 0.00
Four numbers is a ludicrously small description of a token and it is the right size for a page where every product gets printed. A model you can download works at a few thousand numbers a token, and the arithmetic does not change at all between four and four thousand: the loops are the same loops with a different bound, and the only thing that changes is what fits on a page.
Now the three matrices. Each one reads a token's four numbers and produces four new ones, and the three have names that hold for the rest of this book. The query is what a token is asking about. The key is what a token offers to anything that asks. The value is what a token contributes once something has decided to listen to it. One token, four numbers in, three sets of four numbers out, and no interaction between tokens anywhere in this step.
// internal/lang/attend.go
// Head is one attention head: three projection matrices and the rule
// for using them.
//
// D is how many numbers a token arrives with. Dk is how wide a query,
// a key and a value are. Wq, Wk and Wv hold D*Dk numbers each, flat,
// one run of D per output slot, which is the layout internal/mind uses
// for a row of neurons and is here for the same reason: a single run
// of numbers is a thing you can copy, write down and compare, and a
// struct of structs is not.
//
// The three matrices are the stored weights, fixed before any row of
// tokens arrives and the same for every row. They are not the weights
// that decide how much one token counts toward another. Those are
// worked out from the row itself, every time, and they do not exist
// until it does.
type Head struct {
D, Dk int
Wq, Wk, Wv []float64
// Scale divides every score by the square root of Dk. It is a
// field and not a constant so that leaving it off can be run
// rather than described.
Scale bool
}$ go run ./cmd/attend -mode weights
attend: the three matrices, 16 numbers each, one run of 4 per output slot
Wq slot 1 0.0000 0.0000 2.0000 0.0000
slot 2 3.0000 0.0000 0.0000 0.0000
slot 3 0.5000 0.0000 0.0000 0.0000
slot 4 0.0000 0.5000 0.0000 0.0000
Wk slot 1 1.0000 0.0000 0.0000 0.0000
slot 2 0.0000 1.0000 0.0000 0.0000
slot 3 0.0000 0.0000 1.0000 0.5000
slot 4 0.0000 0.0000 0.0000 1.0000
Wv slot 1 1.0000 0.0000 0.0000 0.0000
slot 2 0.0000 1.0000 0.0000 0.0000
slot 3 0.0000 0.0000 1.0000 0.0000
slot 4 0.5000 0.0000 0.0000 1.0000
Read Wq a row at a time and it says what a token asks for. Its second row is
3.0000 against slot 1 and zero everywhere else, so a bare digit ends up
with a 3 in query slot 2 and every other token ends up with a 0 there. Wk's second row
is a 1 against slot 2, so a token carrying a year ends up with a 1 in key slot 2. Query
slot 2 and key slot 2 get multiplied together shortly, and that pairing is the entire
mechanism in miniature: a bare digit is asking a question that only a year-carrying
token can answer loudly.
Forty-eight numbers, in three runs of sixteen, flat. That layout is the one volume 5 settled on for a controller and it is here for the same reasons and not out of habit: a flat run of numbers can be copied with one call, written to a file with no format beyond its own length, compared against another one term by term, and sliced into rows by multiplication. Anything built out of nested structs needs a page of code for each of those.
Four and four are not arbitrary widths either. Four tokens is few enough to print every product of, and four is the key width because a step two pages from here divides by the square root of it, and the square root of four is 2.
Do token 2 by hand before running anything. Its four numbers are 1, 0, 0 and 0. Wq's first
row is (0.00, 0.00, 2.00, 0.00), so query slot 1 is 0×1 plus three products of zero,
which is 0. The second row is (3.00, 0.00, 0.00, 0.00), so query slot 2 is 3×1 = 3.
The third gives 0.5×1 = 0.5, and the fourth gives 0. The query for 0 is
(0, 3, 0.5, 0), and it took sixteen multiplications and twelve additions to get there.
The same sixteen through Wk give its key. Row by row: 1×1 = 1, then three rows whose
first entry is zero, so the key for 0 is (1, 0, 0, 0). Token 1 needs the same
pencil check, since its four numbers are (0, 1, 0, 0.5) and two of them are non-zero. Key slot 1 is
0. Key slot 2 is 1×1 = 1. Key slot 3 is 1×0 + 0.5×0.5 = 0.25. Key slot 4
is 1×0.5 = 0.5. So year 4 offers the key (0, 1, 0.25, 0.5).
// internal/lang/attend.go, inside Attend
project := func(m []float64) [][]float64 {
out := make([][]float64, n)
for t := range x {
out[t] = make([]float64, h.Dk)
for j := 0; j < h.Dk; j++ {
row := m[j*h.D : (j+1)*h.D]
sum := 0.0
for i, v := range x[t] {
sum += row[i] * v
w.Cost.Project++
}
out[t][j] = sum
}
}
return out
}
w.X = x
w.Q = project(h.Wq)
w.K = project(h.Wk)
w.V = project(h.Wv)$ go run ./cmd/attend -mode project
attend: what each token turns into
1 "year 4" query 0.0000 0.0000 0.0000 0.5000
key 0.0000 1.0000 0.2500 0.5000
value 0.0000 1.0000 0.0000 0.5000
2 "0" query 0.0000 3.0000 0.5000 0.0000
key 1.0000 0.0000 0.0000 0.0000
value 1.0000 0.0000 0.0000 0.5000
3 ": 235 hobbs standing... query 2.0000 0.0000 0.0000 0.0000
key 0.0000 0.0000 1.5000 1.0000
value 0.0000 0.0000 1.0000 1.0000
4 "7" query 0.0000 3.0000 0.5000 0.0000
key 1.0000 0.0000 0.0000 0.0000
value 1.0000 0.0000 0.0000 0.5000
the query for token 2, twice
lang.Head, projecting 0.0000 3.0000 0.5000 0.0000
mind.Layer, biases at zero 0.0000 3.0000 0.5000 0.0000
the same numbers true
The pencil said (0, 3, 0.5, 0) for token 2's query and (0, 1, 0.25, 0.5) for token 1's
key, and both are on the run. The last three lines are the point of the stage.
mind.Layer is volume 5's row of neurons, untouched by this volume and
imported by the bench exactly as it stands: hand it the token's four numbers, hand it Wq
as the weight slice, hand it four zeros where the biases go, and out come the identical
four numbers. The same loop, over the same slice, with the same index arithmetic
deciding which run of it belongs to which output. Three matrices per token, and every
one of them is a row of neurons under a different name.
The zeros where the biases go are a decision and not a convenience. Volume 5's neuron needed its bias badly: every weight in the worked example there was negative and every reading was zero or positive, so without a number added on top the total could never climb above zero and the creature would never go anywhere. A projection has no such problem, because nothing downstream compares a query against a fixed threshold. A query only ever gets multiplied by a key, and adding the same constant to every query would add the same amount to every score in the row, which the next step cancels out exactly.
So the three matrices are the genome half of the design. They were fixed before this entry existed, they will be the same for the next entry and for a line of English that has nothing to do with this world, and if a creature carried them they would be forty-eight numbers in the same flat slice everything else in this book gets written down as. Nothing up to here is new. What comes next is.
The attention score square
Twelve numbers now exist per token and there are four tokens, and none of them have said a word to each other. The step that makes them talk is one multiplication per slot: take token i's query, take token j's key, multiply them slot by slot, add the four products up. One number comes out. It is large when the two rows lean the same way, near zero when they have nothing in common, and negative when they lean against each other.
Work the row for token 2, since its query is (0, 3, 0.5, 0) and all four keys are on the page above. Against token 1's key (0, 1, 0.25, 0.5): 0×0 = 0, 3×1 = 3, 0.5×0.25 = 0.125, 0×0.5 = 0, which adds to 3.125. Against token 2's own key (1, 0, 0, 0): 0×1 and three products of zero, so 0. Against token 3's key (0, 0, 1.5, 1): 0.5×1.5 = 0.75. Against token 4's key, which is token 2's key over again: 0. The row is 3.125, 0, 0.75, 0. Do that four times, once per token, and the result is a square of sixteen numbers.
Then divide every one of them by 2, the square root of the key width. Why that divide happens has an answer, the answer is a measurement and not a matter of taste, and it is the next section. For now it is a step.
$ go run ./cmd/attend -mode square
attend: every query against every key, 4 tokens, keys 4 wide
the divide by 2.0000, the square root of 4
every query dotted with every key
query \ key 1 2 3 4
"year 4" 0.2500 0.0000 0.5000 0.0000
"0" 3.1250 0.0000 0.7500 0.0000
": 235 hobbs standing... 0.0000 2.0000 0.0000 2.0000
"7" 3.1250 0.0000 0.7500 0.0000
the same, divided by 2.0000
query \ key 1 2 3 4
"year 4" 0.1250 0.0000 0.2500 0.0000
"0" 1.5625 0.0000 0.3750 0.0000
": 235 hobbs standing... 0.0000 1.0000 0.0000 1.0000
"7" 1.5625 0.0000 0.3750 0.0000
Sixteen numbers, and the four the pencil produced are the second row of the first block. The square is not symmetric, and no rule says it has to be: the number in row 2, column 1 is token 2's question against token 1's offer, and the number in row 1, column 2 is a different question against a different offer. Reading down a column tells you how much one token is being asked for; reading across a row tells you what one token is asking.
Four scores are still four scores. What is wanted is four shares: fractions that add up to one, so a token's answer can be a mixture of the four values in known proportions. The operation that turns one into the other is the softmax, and it is two lines of arithmetic.
Three ideas, every one of them already done above in full with numbers. This is the shorthand, and it earns its place because a real head runs this over thousands of tokens and writing all of them out is not an option.
First the dot product. Two rows of the same length, multiplied slot by slot and added up, written with a dot between them. For the row above: (0, 3, 0.5, 0) · (0, 1, 0.25, 0.5) = 0 + 3 + 0.125 + 0 = 3.125. That single number is the only thing a head ever works out about a pair of tokens.
Now the whole score, for the token in position i against the token in position j. Write qi for the first one's query, kj for the second one's key, and dk for how wide those rows are, which is 4 here.
si,j = qi · kj ÷ √dk
Check it against the run: 3.125 ÷ √4 = 3.125 ÷ 2 = 1.5625, which is the first entry of the second block. With n tokens there is one score for every pair, so there are n×n of them, and at four tokens that is the sixteen printed above.
Then the softmax, which takes one row of that square and turns it into shares. Raise e to each number in the row, then divide each result by the total of all of them. e is 2.718282 and a bit, and the three powers this row needs are e1.5625 = 4.770733, e0 = 1 and e0.375 = 1.454991.
ai,j = esi,j ÷ ∑m=1..n esi,m
The total for token 2's row is 4.770733 + 1 + 1.454991 + 1 = 8.225725, and dividing each of the four by it gives 0.5800, 0.1216, 0.1769 and 0.1216. Two properties fall out and both are the reason this operation gets used instead of something cheaper. Everything raised to a power of e comes out above zero, so no share can be negative and no token can contribute backwards. And every entry is divided by the same total, so the row adds to one exactly, whatever the scores were.
Last, the answer. Token i's answer is every token's value multiplied by that token's share, added up one slot at a time:
oi = ∑j=1..n ai,j vj
Volume 5 called that a weighted sum and so does this. The difference is where the weights came from. There they were read out of a slice a genome had filled in before the creature was born. Here ai,j was computed four lines ago out of the tokens themselves, and the next row of tokens will produce a different one.
// internal/lang/attend.go, the rest of Attend
for i := 0; i < n; i++ {
w.Raw[i] = make([]float64, n)
w.Scaled[i] = make([]float64, n)
for j := 0; j < n; j++ {
sum := 0.0
for c := 0; c < h.Dk; c++ {
sum += w.Q[i][c] * w.K[j][c]
w.Cost.Score++
}
w.Raw[i][j] = sum
w.Scaled[i][j] = sum / w.Div
}
w.Attn[i] = Softmax(w.Scaled[i])
}
for i := 0; i < n; i++ {
w.Out[i] = make([]float64, h.Dk)
for j := 0; j < n; j++ {
share := w.Attn[i][j]
for c := 0; c < h.Dk; c++ {
w.Out[i][c] += share * w.V[j][c]
w.Cost.Mix++
}
}
}$ go run ./cmd/attend -mode scores
attend: four rows of shares, and the values mixed in those proportions
the divide by 2.0000, the square root of 4
each row as shares of one
query \ key 1 2 3 4
"year 4" 0.2565 0.2264 0.2907 0.2264
"0" 0.5800 0.1216 0.1769 0.1216
": 235 hobbs standing... 0.1345 0.3655 0.1345 0.3655
"7" 0.5800 0.1216 0.1769 0.1216
what each row came to
row 1 adds to 1.0000 most of it, 0.2907, on token 3
row 2 adds to 1.0000 most of it, 0.5800, on token 1
row 3 adds to 1.0000 most of it, 0.3655, on token 2
row 4 adds to 1.0000 most of it, 0.5800, on token 1
the values, mixed in those proportions
query \ key 1 2 3 4
"year 4" 0.4528 0.2565 0.2907 0.6453
"0" 0.2431 0.5800 0.1769 0.5884
": 235 hobbs standing... 0.7311 0.1345 0.1345 0.5672
"7" 0.2431 0.5800 0.1769 0.5884
what token 2 arrived with and what it leaves with
in 1.0000 0.0000 0.0000 0.0000
out 0.2431 0.5800 0.1769 0.5884
the multiply-adds it took
making queries, keys and values 192
every query against every key 64
mixing the values 64
all of it 320
Row 2 of the share square is 0.5800, 0.1216, 0.1769, 0.1216, which is the row the Interlude worked by hand, to the last digit printed. The four numbers as printed add to 1.0001 while the row itself adds to 1.0000: each was rounded to four places for its column, and the program did the sum before rounding rather than after.
The last two lines are what the whole page was for. The token 0 arrived
carrying (1, 0, 0, 0): a bare digit, no year, no count, no sentence. It leaves carrying
(0.2431, 0.5800, 0.1769, 0.5884), and the one to read is slot 2, the year slot, which
arrived at exactly nothing and leaves at 0.5800. The token did not change and the table it was
looked up in did not change. What happened is that the head spent 58% of the answer on
the one token in the row that had a year in it, so a description of 0 now
has a year in it as well.
Read the other rows for what they say too. Row 1 is 0.2565, 0.2264, 0.2907,
0.2264: nearly flat, because year 4's query is (0, 0, 0, 0.5) and a query
that is nearly nothing produces scores that are nearly equal, and equal scores come out
of a softmax as equal shares. A head with no question spreads its answer evenly across
the row, and that is correct behaviour and not a defect. Row 3 splits 0.3655 and 0.3655
across tokens 2 and 4, which are the two bare digits, because the long token's query is
2 in slot 1 and slot 1 of a key is exactly the bare-digit marker.
Rows 2 and 4 are identical, to the last decimal place, and they deserve a paragraph rather
than a glance. Token 2 is 0 and token 4 is 7. They arrive with
the same four numbers, because both are bare digits and the typed table has no entry for
which digit, and they sit in different places in the row. The head cannot tell them apart.
Nothing it computes has any notion of which token came first, how far apart two of them
are, or whether one is next to another: a query dotted with a key gives the same number
whichever end of the row each came from.
The repair has a name. It is called positional encoding, and it works by adding something to each token's numbers before any of this arithmetic starts, chosen so that it depends on where the token sits. Two identical tokens in different places stop being identical, and everything downstream carries on unchanged. This chapter names it and does not build it, and the reason to name it right here is that this page has measured the hole it fills: two rows the same, in a real run, on purpose.
The name is doing some work it has not earned and it helps to know which part. What the arithmetic above does is a weighted average with the weights read off a similarity, and "attention" is a reasonable label for that: the row of shares says where the answer came from, and the largest share does mark the token that mattered most. Reading a share square is a real way to find out what a model leaned on.
What it does not have is anything the ordinary word implies about noticing. There is no selection here, in the sense of some tokens being read and others skipped. Every token is projected, every pair is scored, every value is mixed into every answer, and a share of 0.0372 is still a multiplication that happened. Nothing is saved by paying less attention to something. That is why the next section's cost is what it is, and it is also why a share square is easier to interpret than most things inside a model: it hides nothing, because it skipped nothing.
The square-root divide
The step taken on trust so far is dividing every score by the square root of the key width. At the width on this page that is a division by 2, and skipping it does not look like much of a change. The head carries a flag for exactly this, so instead of arguing about it, run it.
Dropping the divide is one field. Every score comes out exactly twice as large, which sounds harmless, because everything downstream is a ratio and a doubling ought to cancel.
$ go run ./cmd/attend -mode scores -noscale | tail -29 | head -12
each row as shares of one
query \ key 1 2 3 4
"year 4" 0.2603 0.2027 0.3342 0.2027
"0" 0.8468 0.0372 0.0788 0.0372
": 235 hobbs standing... 0.0596 0.4404 0.0596 0.4404
"7" 0.8468 0.0372 0.0788 0.0372
what each row came to
row 1 adds to 1.0000 most of it, 0.3342, on token 3
row 2 adds to 1.0000 most of it, 0.8468, on token 1
row 3 adds to 1.0000 most of it, 0.4404, on token 2
row 4 adds to 1.0000 most of it, 0.8468, on token 1
It does not cancel. Row 2 went from 0.5800 on token 1 to 0.8468, and the other three tokens fell from about a sixth of the row each to about a twenty-seventh. Work back to why: the softmax raises e to each score, and doubling a score squares its power of e. The 4.770733 that came out of 1.5625 becomes 22.759895, while the ones that were 1 stay 1. A gap of 1.5625 between two scores was worth a ratio of about 4.8 to 1 in the shares; a gap of 3.125 is worth 23 to 1. A softmax does not care about the size of a score, only about the gaps between scores in a row, and multiplying every score by two doubles every gap.
At four numbers a key that is a difference of emphasis and nothing worse, and there is the trap: a head built without the divide runs, prints plausible rows, and survives every test anybody writes at a width they can read on a page. The damage is a function of the key width, so the way to find it is to measure at several widths instead of arguing at one. Four tokens each time, every query and key drawn one number at a time off a seeded generator, a thousand rows at each width.
$ go run ./cmd/attend -mode scale
attend: what the divide is for, 1000 rows of four tokens at each width
every query and every key drawn off seed 82, one number at a time
--- divided by the root --- --- the divide left off ---
keys spread biggest rows at spread biggest rows at
wide of a row share 0.99 of a row share 0.99
4 1.9694 0.4993 0 3.9389 0.6570 20
16 2.0020 0.5056 0 8.0078 0.8156 160
64 2.0555 0.5123 0 16.4441 0.9071 477
256 2.0686 0.5222 0 33.0969 0.9605 746
Now the cause is on the page. A dot product of two rows of dk numbers is a sum of dk products, and a sum of more independent terms spreads further from its middle: four times as many terms spreads about twice as wide. The left half shows the divide holding that still, with the spread of a row sitting near 2 whether a key is four numbers wide or two hundred and fifty-six. The right half shows what happens without it: 3.9, then 8.0, then 16.4, then 33.1, doubling every time the width quadruples, which is precisely what dividing by the square root undoes.
The last column is the damage. Of a thousand rows at 256 numbers a key, 746 put more than 99% of the row onto a single token. A head doing that has stopped mixing: it picks the best-matching token, discards the other three, and hands back a copy of one value. It does that because the scores got large, not because the tokens deserved it, and the same head on the same four tokens with the divide in place is still spreading its answer across all four. Down the divided half of the table, at every width, the count of pinned rows is zero.
So the square root is not a fudge factor somebody found by experiment. It is the number that makes a score mean the same thing at any key width, which is what lets one design be used at four numbers a token and at four thousand without the behaviour changing underneath.
Softmax has one more property to have in hand before this page leaves it, and the code carries it in three lines. The largest number in a row is subtracted from every entry before e is raised to anything. That cancels exactly, since dividing the top and the bottom of a fraction by the same amount changes neither, and it means the biggest exponent is always 0 and the biggest power of e is always 1. Without it a score of 800 becomes an infinity in a float64, and every share in that row becomes an infinity divided by an infinity, which is not a number at all. Scores of 800 do not turn up in a four-token example. They turn up the first time somebody runs a head on real weights.
Longer rows cost squares
Everything so far ran on four tokens because four tokens can be checked with a pencil. Real text is longer, and what happens to the work as the row grows is already visible in the loops that have been written. The projections run once per token: three matrices, dk output slots each, D multiply-adds a slot. Add a token, add a fixed amount of work, and the total goes up in a straight line. The scores do not behave that way. Every query is dotted with every key, so a row of n tokens holds n×n scores, and the mixing at the end walks the same square again.
That is an argument, and an argument is not a measurement. The Cost counters
sit inside the loops above and increment where the multiply-add actually happens, so what
comes back is what the machine did and not what a formula predicts it would have done.
// internal/lang/attend.go
// Cost is the multiply-adds one run of a head did, counted where they
// happened rather than worked out afterwards.
//
// Project is linear in the number of tokens: three matrices per token
// and nothing about the row. Score and Mix are quadratic: every token
// against every other, twice over.
type Cost struct {
Tokens int
Project int
Score int
Mix int
}
// Total is all of them.
func (c Cost) Total() int { return c.Project + c.Score + c.Mix }
// Quadratic is the part that grows with the square of the row.
func (c Cost) Quadratic() int { return c.Score + c.Mix }$ go run ./cmd/attend -mode cost
attend: the same head, four row lengths, multiply-adds counted where they happen
4 numbers a token, keys 4 wide, the row taken off the front of the chronicle
tokens q, k and v the pairs all of it x row above pair share
4 192 128 320 40.0%
8 384 512 896 2.80 57.1%
16 768 2048 2816 3.14 72.7%
32 1536 8192 9728 3.45 84.2%
the counting against the arithmetic, at 32 tokens
3 x 32 x 4 x 4, the projections 1536
2 x 32 x 32 x 4, the pairs 8192
which comes to 9728
and the head counted 9728
the same number true
where the pairs overtake the projections
2 x n x n x Dk passes 3 x n x D x Dk at n = 7
The second column doubles when the row doubles: 192, 384, 768, 1536. The third column quadruples: 128, 512, 2048, 8192. Those are the two halves of the design doing exactly what their loops say they will, and the fourth column is the sum a machine would pay. It goes up by 2.80, then 3.14, then 3.45, climbing toward 4 and never arriving, because the linear half never quite goes away.
The last column is where the argument lands. At four tokens the pairwise work is 40% of the run and the projections are the bigger half. At thirty-two it is 84%. The crossover on this head is at seven tokens, and it is not a property of these particular matrices: it is 2n2dk passing 3nDdk, which happens as soon as n is past one and a half times the width of a token. A model with four thousand numbers a token crosses over at around six thousand tokens, and every prompt shorter than that spends most of its arithmetic on the projections. Beyond it, the pairs are the bill.
Which settles a question that otherwise looks like an arbitrary limit. Every system that reads text has a maximum number of tokens it will take at once, and that maximum is called the context window. It is not there because somebody was being stingy, and it is not a licensing decision dressed up as a technical one. Doubling the row roughly quadruples the work of one head, and a real model runs several heads side by side, once per layer, over dozens of layers. The window is where somebody drew a line across a curve that climbs as the square, having decided the far side of the line did not earn its cost.
The same arithmetic says what a longer window costs, in a way that is easy to get backwards. Going from a thousand tokens to two thousand costs close to four times as much, not twice, on every head, in every layer, and the bill is paid again for every token the machine produces. A reader holding that can look at any published context length and make a decent guess at what was given up to get it.
$ go test ./internal/lang/ -run 'AProjectionIsAWeightedSumAndNothingElse|EveryScoreIsAQueryTimesAKeyDividedByTwo|EveryAttentionRowIsSharesOfOne|TwoTokensWithTheSameNumbersGetTheSameRow|TheCostIsThreeProjectionsAndTwoPasses|SoftmaxSurvivesAScoreNothingCanRaiseETo|AMatrixOfTheWrongSizeIsRefused|ATokenOfTheWrongWidthIsRefused' -v
=== RUN TestAProjectionIsAWeightedSumAndNothingElse
--- PASS: TestAProjectionIsAWeightedSumAndNothingElse (0.00s)
=== RUN TestEveryScoreIsAQueryTimesAKeyDividedByTwo
--- PASS: TestEveryScoreIsAQueryTimesAKeyDividedByTwo (0.00s)
=== RUN TestEveryAttentionRowIsSharesOfOne
--- PASS: TestEveryAttentionRowIsSharesOfOne (0.00s)
=== RUN TestTwoTokensWithTheSameNumbersGetTheSameRow
attend_test.go:124: tokens 2 and 4 are the same four numbers in different places and the head answers identically
--- PASS: TestTwoTokensWithTheSameNumbersGetTheSameRow (0.00s)
=== RUN TestTheCostIsThreeProjectionsAndTwoPasses
--- PASS: TestTheCostIsThreeProjectionsAndTwoPasses (0.00s)
=== RUN TestSoftmaxSurvivesAScoreNothingCanRaiseETo
--- PASS: TestSoftmaxSurvivesAScoreNothingCanRaiseETo (0.00s)
=== RUN TestAMatrixOfTheWrongSizeIsRefused
attend_test.go:172: lang: Wv holds 3 numbers and a 2 by 2 matrix wants 4
--- PASS: TestAMatrixOfTheWrongSizeIsRefused (0.00s)
=== RUN TestATokenOfTheWrongWidthIsRefused
attend_test.go:185: lang: token 2 of 4 arrives with 2 numbers and the head reads 4
--- PASS: TestATokenOfTheWrongWidthIsRefused (0.00s)
PASS
ok theworld/internal/lang 0.003s
Eight tests, and every one of them is a sentence off this page turned into something that can fail. The pencil's query and key for token 2. The pencil's score row and the divide by two. Every share above zero and every row adding to one. Two tokens with the same numbers getting the same row, which turns the positional hole from a remark into an assertion the module has to keep. The counted multiply-adds agreeing with the formula at four row lengths, checked at both ends so a miscount in either would show. A softmax handed a score no float64 can raise e to. And two refusals: a matrix with the wrong number of entries, and a token of the wrong width, both of which a flat slice would otherwise read straight past into the next row and answer confidently about.
Why attention weights are data
Take the names off and one attention head is a weighted sum, which is the operation this book has been running since a creature first decided whether to walk toward a plant. What changed is where the weights live. In a row of neurons they live in a slice: fixed at birth, carried for a lifetime, changed only by something outside the network. In a head they are worked out from the input, in the middle of the run, from numbers the input itself produced a few lines earlier. The three matrices are still fixed, and they are the less interesting half. They are the machinery that turns tokens into questions and offers. The questions and the offers belong to the row.
That is why it carries past language and past this world. Any time there is a set of items
and each one's answer should depend on which of the others happen to be present, this is
the arithmetic that does it, and it needs no rule about what the items are. It does not
know that token 383 contains a place name. It does not know that 0 is a
digit. It knows that one row of four numbers lines up with another row of four numbers,
and the alignment is the whole of the judgement. Everything a head appears to understand
got into those rows by way of the three matrices, and the three matrices got their numbers
from somewhere this chapter has deliberately not gone near.
The price is on the page as well, and it is not small. Every pair of tokens costs work, so a row of n tokens costs n×n pieces of it, which means the length of a row is a number somebody has to choose rather than a number that can be left open. That trade is the one thing to carry out of this chapter if nothing else survives: the mechanism gets its reach from letting everything see everything, and the bill for that arrives in the same sentence.
What this page did not build, listed so nothing later has to guess. There is one head here; a real model runs several side by side over the same row, each with its own three matrices, and joins their answers together end to end. Nothing here knows where a token sits, and the fix has a name on this page and no code. There is no second stage after the mixing, no normalising step between one head's output and the next thing that reads it, and no stack: heads live inside a block that repeats, and that arrangement is something to read out of a real file rather than draw from memory. And there is no gradient anywhere, no backward pass, no loss and no learning rate. Every number above was either typed or counted. Nothing on this page learned anything.
- Which of a head's weights are fixed before the row arrives and which are worked out
from it, and which of the two kinds
mind.Layerhas. - Why token 2 leaves this head with 0.5800 in a slot it arrived at zero in, and which token that 0.5800 came out of.
- Why rows 2 and 4 of the share square are identical to the last decimal place, and what would have to be added to the tokens to separate them.
- Why doubling every score does not cancel out of a softmax, and what the 746 rows out of 1000 at a key width of 256 are counting.
- Why the projection column of the cost table doubles when the row doubles while the pair column quadruples, and what that makes the total's ratio approach.
- What a context window is a decision about, given a curve that climbs as the square of the row.
Exercise 1 — make a token look somewhere else. Row 2 puts most of its share on token 1. Change one number in one matrix so it puts most of its share on token 3 instead, and say which number and why before you run it.
Token 2's query is built by Wq's four rows reading (1, 0, 0, 0), so only Wq's first
column can affect it at all. Slot 2 of that column is the 3.0000 producing the large
score against token 1's key, and slot 3 is the 0.5000 producing the 0.75 against
token 3's key. Raise the second one and the balance tips. In
cmd/attend/show.go, change Wq's third row from
0.50, 0.00, 0.00, 0.00 to 2.50, 0.00, 0.00, 0.00, then run
go run ./cmd/attend -mode square and
go run ./cmd/attend -mode scores. The raw row becomes 3.625 against
token 1 and 3.75 against token 3, and the shares follow. The score against token 1
moved too, which is the half nobody predicts: token 1's key carries 0.25 in the slot the
raised number reads, so raising the query slot from 0.5 to 2.5 buys token 1 another
0.5 as well. Three things to notice on the
way out. Rows 2 and 4 move together, because their tokens are still identical. And
token 2's answer now carries the count slot where it used to carry the year slot,
which is the same head reading the same row and reporting something different about
it because one stored weight moved.
Exercise 2 — find the width where the missing divide breaks a real row. The scale measurement runs at 4, 16, 64 and 256. Find the key width at which more than half the undivided rows are pinned above 0.99, and check whether the divided ones ever get there.
The mode takes its widths on a flag, so no code is needed:
go run ./cmd/attend -mode scale -widths 64,96,128,160,192
It crosses five hundred out of a thousand somewhere between 64 and 128. The divided
column stays at zero pinned rows the whole way and keeps its spread near 2 whatever
you pass it. Push the widths out to 1024 and 4096 and the undivided column keeps
climbing while the divided one does not move at all, which is the clearest statement
of what the square root does: it makes a score mean the same thing at any width. A
second run to try is -trials 10, where the columns wobble enough to
mislead you, which is why the mode averages a thousand.
Exercise 3 — price a window. Work out on paper how many multiply-adds this head would do on a row of 512 tokens, then get the program to agree with you or disagree.
The two terms are 3nDdk for the projections and 2n2dk for the pairs, with D and dk both 4. That is 3 × 512 × 16 = 24,576 for the first and 2 × 512 × 512 × 4 = 2,097,152 for the second, giving 2,121,728 in total, of which the pairs are 98.8%. Then:
go run ./cmd/attend -mode cost -rows 128,256,512,1024The bench counts as it runs and prints its own check of the formula on the last row it was given, so a disagreement is either your arithmetic or a real defect in the counting. Notice the ratio column sitting at 3.91, 3.95 and 3.98 instead of the 2.80 it showed at eight tokens. The linear term has stopped mattering, and doubling the row now costs almost exactly four times as much.