Four Bits and a Scale
Sixteen bytes a block
The header priced every tensor and left one constant unexplained. The type most of the model uses packs 256 weights into 144 bytes, but 256 weights at four bits each would take 128 bytes. The other sixteen bytes were taken on trust.
A quantized weight is a small whole number with no unit on it; a shared scale and offset turn it back into a weight, and those block fields are part of the price. This file holds 23,804,928 blocks of that type, and sixteen bytes each is 380,878,848 bytes. That is more than a third of a gigabyte, about eight percent of the model on disk, and the reason a reader who multiplies parameters by four bits gets a reproducibly wrong answer.
The rule here is the one the last chapter finished on. A tensor's size on disk is its weight count divided by the weights in one block of its element type, multiplied by what that block costs; add every tensor, the header and the padding, and the total is the file. Offsets are not consulted, which makes the sum a check instead of a restatement.
By the end of this page there is a second file in internal/lang that takes
the size table apart field by field, so that every byte count in it is a thing the code
can explain instead of assert; a run of the estimate everybody makes first, held against
the file and missing by a number this page then accounts for in full; the bits a weight
each of the three types in this model actually costs; the names of the tensors this file
keeps at a wider type, which blocks of the stack got them and what fraction of the bytes
they are; the same accounting run against the other model in the kit, which turns out not
to be written the way its name suggests; and 256 real numbers put through a block of this
format and read back out, with the error printed and the values the format has no room for
named.
One practical note, because it changes how much of a day this costs. Nothing on this page
opens a model. There is no container, no mount, no socket and no four-and-a-half gigabyte
file anywhere in it. Every figure about somebody else's model is computed from the header
this world dumped once and now carries in exemplars/, forty-five kilobytes
standing in for the whole thing, and every figure about real numbers is computed from
weights this world draws for itself. So the bench runs on a bare workstation in about the
time it takes to compile, and the numbers it prints are the same on any machine, because
there is nothing in them that a machine could change.
The flat four-bit estimate
Start with the estimate, because it is the one everybody reaches for and because being precise about why it fails is most of what this chapter has to teach. The model has 7,615,616,512 weights. It says four bits on the tin. Seven and a half billion times four bits is a little under 3.81 gigabytes, so a machine with four gigabytes free ought to hold it, and a page that stopped there would have said something false out of two true things.
The accounting from the last chapter is already written and it takes one flag to make it
lie. -flat tells the sizer to ignore what each tensor is actually written in
and charge every weight four bits, which is exactly the arithmetic the estimate does, only
performed by machinery careful enough to show the damage.
Here is the same accounting the file passed cleanly, with every tensor charged four bits a weight whatever the header says it holds.
$ go run ./cmd/bits -mode sum -flat
bits: every tensor sized at a flat four bits a weight
the hero model, qwen2.5:7b-instruct-q4_K_M, 339 tensors, 4683073952 bytes on disk
type tensors weights bytes bits a weight share
f32 141 333312 166656 4.0000 0.00%
q4_K 169 6094061568 3047030784 4.0000 80.02%
q6_K 29 1521221632 760610816 4.0000 19.98%
all 339 7615616512 3807808256 4.0000 100.00%
header and padding 5953952
every tensor's bytes 3807808256
the two added together 3813762208
the file on disk 4683073952
left over 869311744
4.0063 bits a weight once the header is counted too
The last chapter's version of that block ended with a zero, and a zero was the whole of its claim. This one ends with 869,311,744, which is not a small discrepancy or a rounding or an alignment gap. It is a fifth of the file unexplained, and the only thing the run got wrong was one assumption about what a weight costs.
Now the same mistake stated the way a person actually makes it, with the header left out of the reckoning as well, and then taken apart.
$ go run ./cmd/bits -mode miss
bits: parameters times four bits, and what it does not cover
the hero model, qwen2.5:7b-instruct-q4_K_M, 339 tensors, 4683073952 bytes on disk
the estimate
weights 7615616512
at four bits a weight 3807808256
the file on disk 4683073952
short by 875265696
where the bytes it does not cover are
f32 333312 weights at 28.0000 bits a weight over four 1166592
q4_K 23804928 blocks of 256 at 0.5000 bits a weight over four 380878848
q6_K 5942272 blocks of 256 at 2.5625 bits a weight over four 487266304
the header and its padding 5953952
in all 875265696
and the tensors cost 4677120000, which is 4.9132 bits a weight and not four
875,265,696 bytes short, and the reason the number looks arbitrary is that it is three unrelated mistakes added together with a fourth on top. The estimate charged four bits to 333,312 numbers the file keeps at full width, which is 1,166,592 bytes. It charged four bits to a billion and a half weights the file deliberately keeps at a wider type, which is 487,266,304. It charged four bits to the weights that really are four-bit coded, and forgot that every block of them carries sixteen bytes of its own, which is 380,878,848. And it forgot the header, which on this file is 5,953,952 and on a smaller file would matter a great deal more.
Follow that back from the symptom and the lesson is about where the number came from, not about arithmetic. Four is a name. It appears in the tag on the file, in the way people talk about a model, and in the type's own label, and in none of those places is it a measurement of anything. The bits a weight a file actually spends is a fact about its blocks, and a block is a thing with fields in it. So the fix is not a correction factor. It is to go and look at what a block holds.
The block fields
A four-bit code is a number from 0 to 15. On its own it is not a weight and it is not anything: there is no unit on it and no way to read it. To turn it back into a number somebody can multiply by, two more things are needed, and both of them have to be in the file. One is how far apart two neighbouring codes are. The other is where the run of codes starts. Call those the step and the offset, and the sixteen values a code can name are then the sixteen numbers you get by taking that many steps up from the offset.
If every weight carried its own step and offset the format would be pointless, because two full-width numbers to save half of one is a loss. So a step and an offset are shared, and how widely they are shared is the entire design question. Share them across too few weights and the bookkeeping costs more than the codes save. Share them across too many and a single unusual weight drags the step wide for everybody. The type this model is mostly written in answers that question twice over, at two different widths, and that is the thing the reader is about to see in code.
// internal/lang/quant.go
// Part is one field of a block: what it holds, what it costs, and
// whether it is weight data or the bookkeeping that gives weight data
// a value.
//
// Kind is the column the whole chapter turns on. Add up the parts
// marked codes and divide by the weights in the block and you get the
// number the type's name says. Add up the rest and you get the
// difference between that number and what the file actually spends.
type Part struct {
Name string
Bytes int64
Kind string // codes, scales, or raw for a type that is not quantized at all
}// internal/lang/quant.go — one entry of the layout table.
12: { // q4_K
{"256 four-bit codes, two to a byte", 128, "codes"},
{"8 sub-block scales and 8 offsets, six bits each", 12, "scales"},
{"the block's scale, f16", 2, "scales"},
{"the block's offset, f16", 2, "scales"},
},$ go run ./cmd/bits -mode ladder
bits: what one block holds, for every element type in the file
the hero model, qwen2.5:7b-instruct-q4_K_M, 339 tensors, 4683073952 bytes on disk
f32 1 weight in 4 bytes
4 raw the number itself
4 32.0000 bits a weight: 32.0000 of codes and 0.0000 of scales
q4_K 256 weights in 144 bytes
128 codes 256 four-bit codes, two to a byte
12 scales 8 sub-block scales and 8 offsets, six bits each
2 scales the block's scale, f16
2 scales the block's offset, f16
144 4.5000 bits a weight: 4.0000 of codes and 0.5000 of scales
q6_K 256 weights in 210 bytes
128 codes 256 codes, the low four bits
64 codes the top two bits of all 256
16 scales 16 sub-block scales, eight bits each
2 scales the block's scale, f16
210 6.5625 bits a weight: 6.0000 of codes and 0.5625 of scales
and every one of those fields adds up to the byte count the size table asserts
There are the sixteen bytes. Twelve of them hold eight steps and eight offsets, six bits each, one pair for every run of thirty-two weights inside the block. The other four are two numbers in sixteen-bit floating point that belong to the whole block: one that the eight six-bit steps are measured in, and one that the eight six-bit offsets are measured in. Two levels of scale, the cheap one repeated eight times and the expensive one written once, and that arrangement is what the K on the end of the type's name refers to.
The last line of the run is the check that makes the rest of it something other than a story. These field lists are written by hand, from the format, and a hand-written description of somebody else's binary layout is a hypothesis. So the code adds its own fields up and holds the total against the byte count the sizer has been using since the last chapter, for every type in the table and not only the ones this file happens to use. A layout that does not come to its own row is caught before it can be printed.
Read the q6_K rows beside the q4_K ones and the second design shows through. Its codes are six bits, and six bits do not divide a byte, so the file writes the low four bits of all 256 codes in one run of 128 bytes and the top two bits of all 256 in a second run of 64. Splitting a number across two arrays looks perverse until both halves are byte-aligned and a program can unpack the whole block with shifts and masks and no division at all.
Three types is not a table. The reader's own code can size fifteen of them, so print the lot and see what the range of the choice actually is.
$ go run ./cmd/bits -mode ladder -all
bits: every element type this reader can size, and what a weight costs in each
type weights bytes bits a weight of codes of scales
f32 1 4 32.0000 32.0000 0.0000
f16 1 2 16.0000 16.0000 0.0000
q4_0 32 18 4.5000 4.0000 0.5000
q4_1 32 20 5.0000 4.0000 1.0000
q5_0 32 22 5.5000 5.0000 0.5000
q5_1 32 24 6.0000 5.0000 1.0000
q8_0 32 34 8.5000 8.0000 0.5000
q8_1 32 36 9.0000 8.0000 1.0000
q2_K 256 84 2.6250 2.0000 0.6250
q3_K 256 110 3.4375 3.0000 0.4375
q4_K 256 144 4.5000 4.0000 0.5000
q5_K 256 176 5.5000 5.0000 0.5000
q6_K 256 210 6.5625 6.0000 0.5625
q8_K 256 292 9.1250 8.0000 1.1250
bf16 1 2 16.0000 16.0000 0.0000
not one of the quantized ones costs what its name says
Every quantized row costs more than its name. The cheapest overhead on the table is 0.4375 bits a weight and the dearest is 1.125, and there is no arrangement anywhere in it under which a four-bit type spends four bits. Notice also that the two families pay for their scales in different ways. The plain types at the top share one number across thirty-two weights and stop there. The k-quants at the bottom share across 256 and then spend part of what they saved on a second, finer level of scale inside the block, which is how q4_K arrives at the same 4.5 bits as q4_0 while dividing its block into eight separately scaled runs instead of one.
All of it is one division. A block of this type holds w = 256 weights in c = 144 bytes. A byte is 8 bits, so the block is 144 × 8 = 1,152 bits, and 1,152 ÷ 256 = 4.5. That is the number, and the only trick in it is remembering that the divisor is the weights and not the fields.
Split the same sum in two to see where the half comes from. The codes are k = 128 bytes: 128 × 8 = 1,024 bits, and 1,024 ÷ 256 = 4 exactly, which is the number on the tin and is true of the codes alone. Everything else is s = c − k = 16 bytes: 16 × 8 = 128 bits, and 128 ÷ 256 = 0.5. So b = 8c ÷ w = 4.5, made of 4 that buys sixteen values a weight and 0.5 that buys the right to know what those values are.
Run it on the wider type and the split falls differently. q6_K is w = 256 in c = 210, so b = 210 × 8 ÷ 256 = 6.5625. Its codes are 128 + 64 = 192 bytes, giving 192 × 8 ÷ 256 = 6, and its scales are 18 bytes, giving 18 × 8 ÷ 256 = 0.5625. Two extra bits of code a weight, and a sixteenth of a bit more overhead.
One tensor's bytes then come out of the same three numbers. A tensor of n weights takes n ÷ w blocks, each costing c, so its size is n ÷ w × c bytes, which is the same thing as n × b ÷ 8. The output table of this model is 3,584 × 152,064 = 544,997,376 weights at q6_K: 544,997,376 ÷ 256 = 2,128,896 blocks, 2,128,896 × 210 = 447,068,160 bytes. The same figure the other way round is 544,997,376 × 6.5625 ÷ 8 = 447,068,160. Both routes have to agree or the layout is wrong, and one of the tests below is nothing but that comparison run over every tensor in both committed headers.
The one thing that expression will not do is take a fractional block. n ÷ w has to be a whole number, and a controller in The Hollow owns 378 weights, which is one block of 256 and 122 left over. A 378-number tensor cannot be written in this type at all without being padded first, and the reader from the last chapter refuses one instead of rounding, because a byte count derived from a part block is a plausible number and a wrong one.
Four and nine tenths bits a weight
With every type in the file explained, the accounting can be done honestly, and this is the sum the whole chapter exists to produce. Take each tensor's shape, look up what its element type costs a block, work out how many blocks it needs, multiply. Consult no offset, open no file, believe nothing the header says about where anything lives. Then add the header and its padding on and see whether the answer is the file.
$ go run ./cmd/bits -mode sum
bits: every tensor sized from its shape and its element type alone
the hero model, qwen2.5:7b-instruct-q4_K_M, 339 tensors, 4683073952 bytes on disk
type tensors weights bytes bits a weight share
f32 141 333312 1333248 32.0000 0.03%
q4_K 169 6094061568 3427909632 4.5000 73.29%
q6_K 29 1521221632 1247877120 6.5625 26.68%
all 339 7615616512 4677120000 4.9132 100.00%
header and padding 5953952
every tensor's bytes 4677120000
the two added together 4683073952
the file on disk 4683073952
left over 0
4.9194 bits a weight once the header is counted too
Zero left over, from arithmetic that never looked at an offset. That number was already true when the block sizes were taken from a table, and it is a different claim now that every one of those sizes has been broken into fields and each field named: the file adds up, and the reason it adds up is understood rather than observed.
The column to read is the fourth one. Nothing in this file costs four bits a weight. The bulk of it costs 4.5, a quarter of it costs 6.5625, a thin slice costs 32, and the weighted answer over all 7,615,616,512 weights is 4.9132. Counting the header as well takes it to 4.9194, which on a file this size is a difference in the fourth decimal place and on a small file is not.
4.9132 against 4 is twenty-three percent, and twenty-three percent of the thing that dominates the memory a model needs is the difference between a configuration that starts and one that does not. It is also, usefully, a number a reader can carry: for a file tagged this way, a weight costs about five bits, and parameters times five bits over eight is a first estimate that is wrong by less than two percent instead of by nearly a fifth.
Written at more than one width
The per-type table has an oddity in it that the last chapter noticed and did not explain. The file is tagged for a four-bit quantization, and 29 of its 339 tensors are written in a six-bit one. Not by accident, and not as a fallback: whoever produced this file chose, tensor by tensor, to spend more bytes on some of them than on the rest. The choice is recorded nowhere except in the element type of each info record, so the only way to find out which tensors got it is to go through them.
$ go run ./cmd/bits -mode mixed
bits: which tensors are kept wider than the rest
the hero model, qwen2.5:7b-instruct-q4_K_M, 339 tensors, 4683073952 bytes on disk
3 element types in the file, the narrowest quantized one q4_K at 4.5000 bits a weight
and the widest q6_K at 6.5625
name at q6_K at q4_K wider bytes
blk.N.ffn_down.weight 14 14 779735040
blk.N.attn_v.weight 14 14 21073920
output.weight 1 0 447068160
in all 29 1247877120
the weights in those tensors 1521221632, which is 19.98% of the file's weights
what they cost at q6_K 1247877120, which is 26.68% of the tensors' bytes
what they would cost at q4_K 855687168
so the decision costs 392189952, which is 8.37% of the file
and the blocks of the stack that got the wider type
ffn_down.weight 0 1 2 5 7 12 14 17 20 23 24 25 26 27
attn_v.weight 0 1 2 5 9 11 14 17 20 23 24 25 26 27
Three names, and one of them is not in the stack at all. output.weight is
the table at the back of the model that turns a vector into a score for every token in
the vocabulary, and every byte of it is at the wider type. The other two are inside the
blocks: the value projection, which is the third of the three matrices an attention head
uses, and the down projection, which is the wide-to-narrow half of the feed-forward
layer. Exactly fourteen of the twenty-eight blocks got the wider version of each, and
the other fourteen did not.
Read the two lists at the bottom side by side. Blocks 0, 1 and 2 are on both, and so are 24, 25, 26 and 27. The first three blocks of the stack and the last four are held wider in both tensors, and the remaining seven picks are scattered through the middle at roughly every third block. Whoever wrote this file spent its extra bytes at the two ends of the stack and then thinned out towards the centre, and while this page cannot say why, it can say that the pattern is deliberate: a rule that picked fourteen blocks at random would not put both ends of the stack on both lists.
Then the three figures underneath, because they answer three different questions and people mix them up. Those 29 tensors hold 19.98% of the model's weights. They take 26.68% of its bytes, which is higher for the obvious reason. And the decision to keep them wide, instead of writing them at the same 4.5 bits as everything else, cost 392,189,952 bytes, or 8.37% of the file. That last one is the number an operator actually cares about, and it is the only one of the three that is about the decision instead of about the outcome.
A tag on a file is a claim about the file, and a claim is a thing to check. The kit holds a second model with the same tag on it, four hundred megabytes against four and a half gigabytes, and the accounting takes one flag to point somewhere else.
$ go run ./cmd/bits -mode sum -role small
bits: every tensor sized from its shape and its element type alone
the small model, qwen2.5:0.5b-instruct-q4_K_M, 290 tensors, 397807936 bytes on disk
type tensors weights bytes bits a weight share
f32 121 71552 286208 32.0000 0.07%
q5_0 132 251854848 173150208 5.5000 44.19%
q8_0 13 137510912 146105344 8.5000 37.29%
q4_K 12 52297728 29417472 4.5000 7.51%
q6_K 12 52297728 42900480 6.5625 10.95%
all 290 494032768 391859712 6.3455 100.00%
header and padding 5948224
every tensor's bytes 391859712
the two added together 397807936
the file on disk 397807936
left over 0
6.4418 bits a weight once the header is counted too
Five element types, and the two that dominate it are not k-quants at all. Nearly half of this file is written at 5.5 bits a weight and better than a third at 8.5, and the type the tag names accounts for 7.51% of its bytes. The weighted answer is 6.3455 bits a weight against the larger file's 4.9132, which is twenty-nine percent more per weight for two files carrying the same three letters and the same trailing M.
There is a good reason and it is not the point. A model this small has tensors whose dimensions are 128 and 896, and a k-quant needs its weight count to divide by 256, so the writer falls back to the thirty-two weight types wherever it must; and a half billion parameter model has so little redundancy in it that four-bit codes hurt more. The point is that the name did not say any of that, and a reader who had compared these two files by their tags would have compared two things that are not the same. Every number on this page belongs to the file it was read from, and a bits-a-weight figure taken off one model is not a figure about another one even when the labels match.
The residue is zero here too, on a file with five types in it instead of three, and that is the accounting rule doing the only job it has: 391,859,712 bytes of tensors plus 5,948,224 of header and padding is 397,807,936, and the file is 397,807,936.
The 256-weight block round trip
Everything so far has been about what the format costs. What it takes is a separate question and it needs different numbers: not a header, but weights, put through the format and read out again with the difference printed. The weights this world has to hand are its own. A controller in The Hollow is 24 inputs, a middle row of 12 and 6 outputs, its 378 weights drawn in flat order from stream 14 on the world seed, every one of them between minus one and one and then divided by the square root of the fan its neuron adds up. Nobody chose them to make a point, which is the only property this page needs from them.
The quantizer is written the way the format writes one, in two levels. Every run of thirty-two weights gets sixteen rungs stretched between its own lowest and highest number. Those eight steps and eight offsets are then themselves rounded to six bits each, against two numbers held in sixteen-bit floating point for the whole block. Reading a weight back out is one line: the block's step times the sub-block's step code times the weight's own code, less the block's offset times the sub-block's offset code.
// internal/lang/quant.go
// The eight ladders, in full precision, before anything is
// rounded. step is the distance between two rungs and off is how
// far below zero the bottom rung sits.
var step, off [Subs]float64
for j := range Subs {
run := vals[j*Sub : (j+1)*Sub]
lo, hi := run[0], run[0]
for k, v := range run {
if math.IsNaN(v) || math.IsInf(v, 0) {
return Block{}, fmt.Errorf("lang: weight %d of the block is %v, which no ladder has a rung for", j*Sub+k, v)
}
lo = math.Min(lo, v)
hi = math.Max(hi, v)
}
if lo > 0 {
lo = 0
b.Flat++
}
step[j] = (hi - lo) / 15
off[j] = -lo
}// internal/lang/quant.go
// Back reads the block out again: 256 numbers, every one of them a
// rung on its own sub-block's ladder.
func (b Block) Back() []float64 {
out := make([]float64, Super)
for j := range Subs {
s := b.D * float64(b.Sc[j])
o := b.Dmin * float64(b.Mn[j])
for i := range Sub {
at := j*Sub + i
out[at] = s*float64(b.Q[at]) - o
}
}
return out
}$ go run ./cmd/bits -mode round
bits: 256 weights of a 24-12-6 controller through one q4_K block and out again
drawn on world seed 5, stream 14, 378 weights in the row, block starts at 0
what the block holds
the block's scale 0.0004220008850097656
the block's offset 0.003238677978515625
8 sub-block scales [62 62 57 63 61 63 62 60]
8 sub-block offsets [58 60 50 61 60 61 63 55]
256 codes, 0 to 15 [12 13 13 7 1 9 10 1] ... [1 5 7 4]
written down 144 bytes
ladders starting at 0 0
the first sub-block's ladder: the only sixteen numbers its 32 weights can come back as
-0.187843322754 -0.161679267883 -0.135515213013 -0.109351158142
-0.083187103271 -0.057023048401 -0.030858993530 -0.004694938660
0.021469116211 0.047633171082 0.073797225952 0.099961280823
0.126125335693 0.152289390564 0.178453445435 0.204617500305
the first 8 weights
n went in code came back off by
0 0.113737677987 12 0.126125335693 0.012387657706
1 0.159530218718 13 0.152289390564 -0.007240828154
2 0.165056277822 13 0.152289390564 -0.012766887258
3 -0.001856715926 7 -0.004694938660 -0.002838222734
4 -0.164724912571 1 -0.161679267883 0.003045644687
5 0.054667885212 9 0.047633171082 -0.007034714130
6 0.084729732706 10 0.073797225952 -0.010932506754
7 -0.169342568901 1 -0.161679267883 0.007663301018
over all 256
distinct numbers in 256
distinct numbers out 96, and 8 ladders of 16 rungs is 128 at the most
largest error 0.012939653851, at weight 27 of sub-block 1
mean error 0.006154076932
how much of a weight survives
within 1 percent of itself 30 of 256
within 10 percent 175 of 256
the lowest of the first 32 -0.187516522994, back as -0.187843322754
the highest of the first 32 0.203254216671, back as 0.204617500305
sub-block by sub-block
n rung to rung largest error mean error
0 0.026164054871 0.012766887258 0.005920995071
1 0.026164054871 0.012939653851 0.006586259048
2 0.024054050446 0.011471153610 0.005253766755
3 0.026586055756 0.012559609770 0.007199957263
4 0.025742053986 0.012181908549 0.004864096139
5 0.026586055756 0.012501260866 0.007070502301
6 0.026164054871 0.012570653638 0.006202100168
7 0.025320053101 0.012649403936 0.006134938710
256 of these numbers in memory is 2048 bytes and this block is 144
The two lines in the middle are the plainest statement of what quantization is. Two hundred and fifty-six different numbers went in and ninety-six different numbers came out, and ninety-six was not the target of anything: it is what happens when eight runs of thirty-two are each squeezed onto sixteen rungs and some of the rungs go unused. The ceiling is 128 and it cannot be raised without spending more bits, because sixteen is all a four-bit code can name.
The sixteen rungs above them are the sixteen values the first thirty-two weights are allowed to come back as, and it repays a moment's staring, because it is the whole answer to which values the format cannot hold. It cannot hold 0.05. It cannot hold anything between 0.047633171082 and 0.073797225952, which is a whole rung wide and holds nothing at all. Every weight in that run lands on the nearest rung and the error is its distance to that rung, so the largest error possible is half a step, and the measured largest over all 256 is 0.012939653851 against a step of 0.026164054871. That bound is not a hope. One of the tests below checks it on every weight.
Which makes the loss easy to reason about and easy to get backwards. It is not the big weights that suffer. A sub-block's ladder is pinned at both ends by its own lowest and highest number, and the run shows both surviving to four decimal places. It is everything between the rungs, and the width of a rung is set by the two values furthest apart in the run. The 30 of 256 that came back within one percent of themselves are the ones that both landed near a rung and were big enough for half a step to be a small fraction of them. The ones that lost most of themselves in proportion are the small ones: weight 3 of the block went in at -0.001856715926 and came back as -0.004694938660, which is an absolute error smaller than most on the page and more than half again the number it was. The error is bounded uniformly and the harm it does is not.
One honesty about those errors before they get quoted anywhere. The quantizer above takes the obvious ladder for each run of thirty-two: bottom rung at the lowest number, top rung at the highest, fourteen evenly spaced rungs in between. A production writer does not stop there. It tries several ladders for each run and keeps whichever one leaves the smallest total error, which usually means giving up a little at the ends to gain in the crowded middle. So the errors on this page are the ones the simplest possible writer of this format would make, and a real file's are somewhat smaller. The byte counts are unaffected: a search changes which sixteen numbers a run comes back as, never how many bytes it takes to say so.
That is the well-behaved case, where all thirty-two numbers in a run sit in the same band because they were drawn from the same distribution. Real tensors are not always so obliging, and the format's exposure to one badly-placed number is best seen by putting one there. The flag replaces the first weight of the block with a number the rest of the row would never produce, and changes nothing else.
$ go run ./cmd/bits -mode round -spike 1 | tail -18
how much of a weight survives
within 1 percent of itself 25 of 256
within 10 percent 164 of 256
the lowest of the first 32 -0.187516522994, back as -0.187843322754
the highest of the first 32 1.000000000000, back as 0.999967575073
sub-block by sub-block
n rung to rung largest error mean error
0 0.079187393188 0.038701550040 0.021983894822
1 0.026395797729 0.013094026759 0.006621249165
2 0.023881912231 0.011403626003 0.005022536288
3 0.026395797729 0.013145881201 0.006948999730
4 0.026395797729 0.012209682308 0.006000525761
5 0.026395797729 0.013097403701 0.007445098863
6 0.026395797729 0.011275972643 0.006094645115
7 0.025138854980 0.011967908930 0.006009596647
256 of these numbers in memory is 2048 bytes and this block is 144
Sub-block 0's rungs were 0.026164054871 apart and are now 0.079187393188 apart, three times as coarse, because its sixteen rungs have to span from its lowest number all the way up to a value the other thirty-one are nowhere near. Its largest error went from 0.012766887258 to 0.038701550040 and its mean from 0.005920995071 to 0.021983894822. The weight that caused all of it came back as 0.999967575073, off by three hundred-thousandths, because it is the top rung.
The other seven runs pay too, and the mechanism is the second level of scale. Their
step codes were [62 62 57 63 61 63 62 60] and are now
[63 21 19 21 21 21 21 20]: the block's shared step had to grow to describe
sub-block 0's ladder, so every other ladder is now named with about a third as many of
the sixty-four available codes, and their rung spacings come out slightly wrong in the
fourth decimal place. Twenty-five of 256 within one percent instead of thirty, and 164
within ten percent instead of 175. One number in the wrong place taxed all 256.
What none of that says is how much any of it costs an answer. This page has measured the error a format introduces into a set of numbers, and the error is real and it is arithmetic. What it does to the quality of a model's output is a different quantity entirely, it is measured by running models against test sets on hardware this volume does not have, and no run on this page went anywhere near it. A number for that is not being withheld here. There is not one, and quoting one would mean inventing it.
Why this works
The move underneath the whole format is older than models and it turns up wherever a program has to store a lot of numbers that are similar to each other. Store each number independently and you pay full price for every one of them, and most of what you have paid for is the part they have in common. Store the common part once and give each number a short code that says where it sits relative to it, and the cost per number falls to whatever the code costs plus a share of the common part. The saving is entirely a function of how many numbers share, and the loss is entirely a function of how different they are.
Which is why the choice of how much to share is the whole design, and why q4_K makes it twice. One block of 256 amortises the two expensive numbers over enough weights that they disappear: four bytes over 256 weights is an eighth of a bit each. But 256 weights are too many to share one ladder without a straggler spoiling it, so the block subdivides into eight runs of thirty-two, each with its own cheap step and offset. Twelve bytes for eight pairs is three eighths of a bit a weight, which is what buys back the resolution the big block gave up. Half a bit a weight, spent in two places for two different reasons.
The second habit here is one this book has now used on the same file twice. When a program depends on a constant it did not derive, the constant is a liability until something checks it. The last chapter took 144 bytes a block from a table and produced a sum that came out exactly, which is evidence but not proof: a wrong block size would have produced a wrong total, and nothing in that run would have said which of the fifteen table entries was to blame. Writing the fields down and making the code add them up turns one global check into fifteen local ones, and the fifteen tell you where the mistake is. A number you can only test in aggregate is a number you cannot debug.
Both habits generalise past this format. Any time a size table, a wire layout or a struct alignment is written down from a specification, the specification's own arithmetic is sitting there unused: fields add up to records, records add up to files. Making the code perform that addition costs a dozen lines and catches the class of mistake that is otherwise invisible until something a long way downstream produces a plausible wrong answer.
Why block bookkeeping prices weights
The arithmetic on this page is now load-bearing for anything that wants to know what a model costs, so it gets tests, and none of them needs a model. Two of them run against the committed headers, which is the same substitution the last chapter made: forty-five kilobytes standing in for four and a half gigabytes, and the code under test cannot tell the difference.
$ go test ./internal/lang/ -run 'TestEveryBlockLayoutAddsUpToItsOwnSizeInTheTable|TestNoQuantizedTypeCostsWhatItsNameSays|TestABlockOfRealNumbersIsWrittenAsOneHundredAndFortyFourBytes|TestEveryWeightComesBackOnItsOwnSubBlocksLadder|TestNoWeightIsMoreThanHalfARungFromWhereItStarted|TestABlockThatIsNotTwoHundredAndFiftySixWeightsIsRefused|TestASixteenBitScaleIsWhatTheBlockActuallyHolds|TestTheCommittedHeadersCostWhatTheirTypesSayTheyDo' -v
=== RUN TestEveryBlockLayoutAddsUpToItsOwnSizeInTheTable
quant_test.go:39: all 15 element types taken apart, and every one of them adds up
--- PASS: TestEveryBlockLayoutAddsUpToItsOwnSizeInTheTable (0.00s)
=== RUN TestNoQuantizedTypeCostsWhatItsNameSays
quant_test.go:71: q4_K costs 4.5 bits a weight and q6_K 6.5625, block overhead included
--- PASS: TestNoQuantizedTypeCostsWhatItsNameSays (0.00s)
=== RUN TestABlockOfRealNumbersIsWrittenAsOneHundredAndFortyFourBytes
quant_test.go:86: 256 weights written down as 144 bytes, which is what the table asserts
--- PASS: TestABlockOfRealNumbersIsWrittenAsOneHundredAndFortyFourBytes (0.00s)
=== RUN TestEveryWeightComesBackOnItsOwnSubBlocksLadder
quant_test.go:111: all 256 weights landed on one of the 16 rungs their own sub-block has
--- PASS: TestEveryWeightComesBackOnItsOwnSubBlocksLadder (0.00s)
=== RUN TestNoWeightIsMoreThanHalfARungFromWhereItStarted
quant_test.go:131: every error inside half a rung, which is the whole of what four bits promises
--- PASS: TestNoWeightIsMoreThanHalfARungFromWhereItStarted (0.00s)
=== RUN TestABlockThatIsNotTwoHundredAndFiftySixWeightsIsRefused
quant_test.go:138: refused, and the message says both numbers: lang: a q4_K block holds 256 weights and 255 were handed over
quant_test.go:145: refused: lang: weight 100 of the block is +Inf, which no ladder has a rung for
--- PASS: TestABlockThatIsNotTwoHundredAndFiftySixWeightsIsRefused (0.00s)
=== RUN TestASixteenBitScaleIsWhatTheBlockActuallyHolds
quant_test.go:159: 0.1 is held as 0.0999755859375, and a scale that has been through sixteen bits stays there
--- PASS: TestASixteenBitScaleIsWhatTheBlockActuallyHolds (0.00s)
=== RUN TestTheCommittedHeadersCostWhatTheirTypesSayTheyDo
quant_test.go:193: hero: 7615616512 weights, 4677120000 bytes of tensors, 4.913188569965641 bits a weight, 0 left over
quant_test.go:193: small: 494032768 weights, 391859712 bytes of tensors, 6.345485358574433 bits a weight, 0 left over
--- PASS: TestTheCommittedHeadersCostWhatTheirTypesSayTheyDo (0.00s)
The first is the one that matters most and it is the cheapest: fifteen hand-written field lists, each added up and held against the byte count the sizer uses. The eighth is the accounting rule in its strongest form, run over both committed headers, checking not only that the residue is zero but that every single tensor's byte size equals its weight count times its type's bits a weight divided by eight. Two ways of computing the same thing, over 629 tensors, agreeing every time.
The fifth pins the promise the format makes. Sixteen rungs between a run's lowest and highest number means no weight can be more than half a step from where it started, and a quantizer with a rounding bug in it would break that quietly, producing numbers that are merely a bit worse rather than obviously wrong. The sixth pins the two refusals: a block that is not 256 weights, and a weight that is not a number at all. Neither is a case a real file would produce, and both are cases where a program that carried on would carry on with numbers nobody could account for.
- Name the four fields of a q4_K block and their byte costs, and produce 4.5 bits a weight from them without looking anything up.
- Say which of those fields the 4 in the type's name accounts for, and what the other half a bit is buying.
- Take 7,615,616,512 weights, charge them four bits each, and account for all 875,265,696 bytes by which that misses the file, in the four separate pieces it is made of.
- Given a tensor of 3,584 by 512 at q4_K, produce its byte size two ways: through blocks of 256, and through bits a weight.
- Say why 378 weights cannot be written as a q4_K tensor at all, and what a reader that rounded the block count instead of refusing would produce.
- Explain why one weight far from the rest of its run of thirty-two makes the other thirty-one worse, and why the offending weight itself comes back nearly exact.
Exercise 1 — price a model you have not got. Somebody tells you a 13-billion-parameter model is available at q4_K_M and asks whether it fits in eight gigabytes of memory. Answer with the arithmetic on this page, and say what you cannot answer.
The naive figure is 13,000,000,000 × 4 ÷ 8 = 6.5 gigabytes, which leaves a gigabyte and a half of room and is the wrong number. The hero model on this page came out at 4.9132 bits a weight over its tensors, so a first estimate on the same mix of types is 13,000,000,000 × 4.9132 ÷ 8, or about 7.98 gigabytes. The room is gone. The weights alone now fill the machine, and the weights are not the only thing a loaded model holds.
What you cannot answer is anything exact, and it is important to be clear about
why. The fraction of tensors kept at the wider type is a decision the person who
wrote that particular file made, and the two files on this page differ so much that
one came out at 4.9132 bits a weight and the other at 6.3455 with the same tag. So
the honest answer is an estimate with a stated basis and a note that the real figure
needs the file, and the run that produces the real figure is -mode sum
against that file's own header.
Exercise 2 — find the cheapest honest quantization. Using the fifteen-row table, work out which type gives the lowest bits a weight, what its overhead fraction is, and why that is the one nobody uses by default.
-mode ladder -all. q2_K is the cheapest at 2.625 bits a weight, and its
split is the most lopsided on the table: 2 bits of codes and 0.625 of scales, so
nearly a quarter of what it spends is bookkeeping. Compare that with q8_0, where 0.5
bits of overhead sits on 8 bits of code and is under six percent.
The reason overhead grows as a share is that it is nearly fixed. A block's scales cost about the same whatever the codes are, so the narrower the codes the larger the fraction they are. Two bits a weight also means four values a run of thirty-two, and the round-trip run on this page shows what sixteen values does to a set of weights that had 256 distinct values in it. Four would do considerably more. The arithmetic says a q2_K block is 42% smaller than a q4_K one; what that costs an answer is the question this page has already said it does not measure.
Exercise 3 — make the second level of scale earn its keep. The block spends 12 of its 16 overhead bytes on eight sub-block ladders instead of one ladder for all 256. Design a run of numbers where the extra seven ladders buy nothing, and one where they buy a great deal, and predict the errors before running either.
Buys nothing: 256 numbers drawn from one narrow band, where all eight runs of
thirty-two have nearly the same lowest and highest values. Their step codes come out
nearly equal, the eight ladders are nearly the same ladder, and a single ladder for
the whole block would have given almost identical errors. That is close to what the
plain run on this page shows, where the eight step codes are
[62 62 57 63 61 63 62 60] and the eight rung spacings all fall between 0.0240 and 0.0266.
Buys a great deal: one run of thirty-two spread far wider than the other seven. Run
-mode round -spike 1 and read the sub-block table: sub-block 0's rungs
open to 0.079187393188 while the other seven stay near 0.026, and their errors stay
near where they were. With one ladder for all 256, every weight in the block would
have been on sub-block 0's coarse ladder and every error would have tripled. Twelve
bytes bought the containment. The same experiment is also the argument for the
third level nobody has built here: eight is not a magic number, it is a point on a
curve between bookkeeping and resolution, and the format picked it.