The Same Block, Over and Over
Four and a half gigabytes on disk
Four and a half gigabytes of model bytes sit on disk, and the hash says they are the bytes this world meant to fetch. That guarantee proves identity, not structure. To price the file, the reader has to open its header.
A model file is a fixed head of four numbers, a run of typed key-value pairs, one record for every tensor giving its name, dimensions, element type and byte offset, padding up to the declared alignment, and then the weights. The format is ordinary binary: little-endian numbers, strings as length plus bytes, arrays as type plus count plus elements, and no delimiters.
Nothing in it is compressed, encrypted or dumped out of memory. The claim that the file is a black box fails as soon as the header is read, because the header names every tensor, where its bytes begin and what type its elements use.
By the end of this page there is a reader in internal/lang that walks all of
that; a small file of this world's own, written by the same code and read back to prove
the reader is a reader and not a set of guesses that happen to work; a listing of every
tensor in the pulled model, which turns out to be a handful of names repeated once a
layer; the model's vocabulary size, its embedding width, its head counts and the context
it was trained on, read out of the file and not off a web page; a dumped copy of that
header committed under exemplars/, so the rest of this volume can do
arithmetic about seven billion parameters without seven billion parameters; and a total
that comes to the size of the file on disk exactly.
This book has written binary before, and both times the file was small. A generator's state went to disk as twenty bytes, four and then sixteen, so that a saved world could carry on drawing the same numbers it would have drawn. A creature's brain went the other way and was written as text, one JSON object a line, at about two and a half times what the numbers occupy in memory, because a person had to be able to open the file. Four and a half gigabytes is where that trade stops being a trade. Nobody is reading this file in an editor, so the format spends nothing on being readable and everything on being walkable in one pass.
One practical note before any of it, because it changes how this page fits into a day's work. Nothing here loads a model. Nothing here opens a socket, sends a prompt or waits on an answer. The reading modes below open a file, read about six megabytes off the front of it and stop, and the arithmetic modes do not even do that: they read a copy of the header the build carries. So the order this runs in does not matter, no model has to be resident, and the whole chapter costs about as much as listing a directory.
The 408-byte sketch file
A parser written against somebody else's four-gigabyte file has a bad failure mode. It reads plausible numbers, prints them, and there is nothing to check them against except the parser that produced them. Write a file first and the problem goes away: the bytes are yours, the numbers that went in are known, and a reader that gets them back is a reader.
Start with the list of types the format uses to say what a value is. There are thirteen
of them and their numbers go on the wire, so the order of the block is the format's and not
this program's to tidy. That is exactly the case iota is for, and the comment above it
says so, because the next person to look at it will want to sort the names alphabetically.
// internal/lang/gguf.go
// Kind is the type of one metadata value, and the numbers are the
// format's own: they go on the wire, so the order of this block is
// not ours to tidy.
type Kind uint32
const (
KindU8 Kind = iota
KindI8
KindU16
KindI16
KindU32
KindI32
KindF32
KindBool
KindString
KindArray
KindU64
KindI64
KindF64
kindCount
)// internal/lang/gguf.go
// Tensor is one info record: what the tensor is called, how big it
// is, what its elements are, and where in the data region its bytes
// begin.
//
// Offset is counted from the start of the tensor data and not from
// the start of the file, which is the single most useful thing to
// know about it and the reason the alignment matters at all.
type Tensor struct {
Name string `json:"name"`
Dims []uint64 `json:"dims"`
Type Elem `json:"type"`
Offset uint64 `json:"offset"`
}// internal/lang/sketch.go — the writer's one arithmetic decision.
// The order is the format's and there is no room in it for taste:
// the magic, the version, the two counts, the metadata, the info
// record, the padding, the weights.
info := e.at
data := roundUp(info, int64(align))
mark(fmt.Sprintf("padding up to the declared alignment of %d", align))
e.put(make([]byte, data-info))
mark(fmt.Sprintf("tensor data: %d f32 weights", len(s.Vals)))
for _, v := range s.Vals {
e.f32(v)
}$ go run ./cmd/block -mode write
block: a gguf file written by this program and read back by the same reader
what went in
metadata entries 5
tensors 1
the tensor blk.0.counts.weight, 3 x 2, f32
alignment declared 64
where every field went
at bytes field
0 4 magic: the four bytes GGUF
4 4 version, a u32
8 8 how many tensors, a u64
16 8 how many metadata entries, a u64
24 46 metadata: general.architecture (string)
70 55 metadata: general.name (string)
125 33 metadata: general.alignment (u32)
158 34 metadata: counts.block_count (u32)
192 76 metadata: counts.tokens (array)
268 59 info record: blk.0.counts.weight
327 57 padding up to the declared alignment of 64
384 24 tensor data: 6 f32 weights
408 bytes written to block85-write.gguf, sha256 989abb48b613e7a6
what the reader found
version 3
tensors 1
metadata entries 5
alignment 64, read out of the metadata
architecture counts, 1 block, 3 tokens named
last info record ends at 327
padding 57
tensor data starts at 384
blk.0.counts.weight 6 weights, 24 bytes, at offset 0
327 + 57 + 24 = 408, and the file is 408
the numbers, read at the declared alignment
written [0.5 1.5 2.5 3.5 4.5 5.5]
read [0.5 1.5 2.5 3.5 4.5 5.5]
6 of 6 came back the number that went in
The writer hands back a span for every field it wrote, which is where that middle table comes from. Nobody counted those offsets. They are what the writer's own byte counter said as it went past, and printing them turns a format description into a thing you can point at.
Two of the choices in that file are deliberate and both are about the padding. The alignment is sixty-four, which is not what a real model declares, and the file says so out loud in a metadata key. A writer and a reader that quietly agreed on the same default would produce a file that works and teaches nothing; a number written down in the file is a number the reader has to go and get. And the metadata was sized so that the last info record ends at 327, which is not a multiple of sixty-four, so there are fifty-seven bytes of padding instead of none. A file that happened to land on a boundary would let a broken reader pass.
The last three lines are the whole reason this stage exists. Six numbers went in and the same six came out, and neither the writer nor the reader was told what the other was doing. Everything after this points that reader at a file it has never seen.
Look at those four hundred and eight bytes as bytes once, because a format nobody has seen in hexadecimal stays a diagram in the reader's head.
$ go run ./cmd/block -mode bytes
block: block85-bytes.gguf, every byte of it, with a name against every field
0 47 47 55 46 magic: the four bytes GGUF
4 03 00 00 00 version, a u32
8 01 00 00 00 00 00 00 00 how many tensors, a u64
16 05 00 00 00 00 00 00 00 how many metadata entries, a u64
24 14 00 00 00 00 00 00 00 67 65 6e 65 ... metadata: general.architecture (string)
70 0c 00 00 00 00 00 00 00 67 65 6e 65 ... metadata: general.name (string)
125 11 00 00 00 00 00 00 00 67 65 6e 65 ... metadata: general.alignment (u32)
158 12 00 00 00 00 00 00 00 63 6f 75 6e ... metadata: counts.block_count (u32)
192 0d 00 00 00 00 00 00 00 63 6f 75 6e ... metadata: counts.tokens (array)
268 13 00 00 00 00 00 00 00 62 6c 6b 2e ... info record: blk.0.counts.weight
327 00 00 00 00 00 00 00 00 00 00 00 00 ... padding up to the declared alignment of 64
384 00 00 00 3f 00 00 c0 3f 00 00 20 40 ... tensor data: 6 f32 weights
408 bytes in the file, and not one of them is a delimiter
Read the first two rows and the whole format is there. 47 47 55 46 is
GGUF in ASCII. 03 00 00 00 is the number three, with the
least significant byte first, which is the same convention the framebuffer packs a
colour into a thirty-two bit word with and the same one every number in this file uses.
Then 14 00 00 00 00 00 00 00 at offset 24 is twenty, and the twenty bytes
after it spell general.architecture. Nothing announces the end of that
string. The length did.
That is why the padding row matters and why it is not decoration. In a format with no delimiters, position is meaning. Every field is found by having read the field before it, and the only place the chain breaks is the one gap that is not a field at all.
Pointed at four and a half gigabytes
The file to open is the one the server downloaded, and finding it is already done. The
store keeps every model's weights as a blob whose filename is the sha256 of the blob,
and Weights walks the manifest to that path. The toolchain container has the
model volume mounted read-only, so the reader below opens the very file the digest was
taken over. No copy is made anywhere, which matters more than it sounds: a copy is a
second thing that can differ from the first.
// internal/lang/gguf.go
// ReadHeader walks a GGUF file's header and stops at the first byte
// of weights.
//
// size is the file on disk, which the caller has and this function
// cannot get from an io.Reader. Everything else comes off the stream
// in one pass, forwards, with no seeking: the head, the metadata, the
// info records, and then the arithmetic that turns the end of the
// last record into the start of the data.
func ReadHeader(r io.Reader, size int64) (Header, error) {
d := &reader{r: r}
var m [4]byte
if !d.fill(m[:]) {
return Header{}, fmt.Errorf("lang: reading the magic: %w", d.err)
}
if m != magic {
return Header{}, fmt.Errorf("lang: this file opens %q and a GGUF file opens %q", m, magic)
}$ podman exec -w /bench world-go go run ./cmd/block -mode head
block: the header of a model this world pulled, read where the server left it
the file
role hero, qwen2.5:7b-instruct-q4_K_M
blob sha256-2bada8a7450677000f678be90653b85d364de7db25eb5ea54136ada5f3933730
pinned at 2bada8a745067700
bytes 4683073952
the four numbers at the front
magic GGUF
version 3
tensors 339
metadata entries 34
what the metadata says the model is
architecture qwen2
name Qwen2.5 7B Instruct
blocks in the stack 28
embedding width 3584
attention heads 28
key and value heads 4
feed-forward width 18944
trained context 32768
vocabulary 152064 tokens
merges 151387
and what falls out of those two lines
one head is 128 wide, because 3584 divided by 28 is 128
where the header stops
alignment 32, which the file does not declare and so is the format's own
metadata ends at 5934631
last info record ends at 5953935
padding 17
tensor data starts at 5953952
Every one of those lines is a field, and none of them is an opinion. The reader that
produced them is under two hundred lines of encoding/binary, it goes
forwards and never seeks, and it stops the moment it reaches a byte of weights.
Read that middle block slowly, because it is the first time in this volume that a model's insides have been numbers instead of a description. There are 152,064 tokens in the vocabulary and 151,387 merges, which is the same machinery counted off a corpus earlier in this volume with the merge table built by hand, only far larger and trained by somebody else on text nobody here has seen. There are twenty-eight attention heads and the embedding is 3,584 wide, so a head is 128 wide, and 128 is the number whose square root divides every score in a head before the shares are worked out.
Then a line that would look like a typo to anybody who had not met attention: four key and value heads against twenty-eight query heads. The queries get twenty-eight sets of numbers and the keys and values get four, shared out among them. That is deliberate, and it is not free: the design gives up some of the independence between heads and buys back a great deal of what has to be kept while a conversation is running. The tensor listing later on this page confirms it in the only way that counts: the query projection there is 3,584 wide and the key and value projections are 512, which is 4 times 128.
The trained context is 32,768 tokens. What this world asks for is 4,096, written down in the operator's kit instead of left to a default. Both numbers are true and they answer different questions: one is what the model was trained to handle, the other is what this world has decided to pay for. A page that printed the larger one as though it described a running server would be saying something false out of two true things.
Now the whole metadata block, all thirty-four entries, in the order the file holds them.
$ podman exec -w /bench world-go go run ./cmd/block -mode meta
block: every metadata entry of qwen2.5:7b-instruct-q4_K_M, in the order the file holds them
n key type value
0 general.architecture string "qwen2"
1 general.type string "model"
2 general.name string "Qwen2.5 7B Instruct"
3 general.finetune string "Instruct"
4 general.basename string "Qwen2.5"
5 general.size_label string "7B"
6 general.license string "apache-2.0"
7 general.license.link string "https://huggingface.co/Qwen/Qwen2.5-7B-Instr" ... and 21 more bytes
8 general.base_model.count u32 1
9 general.base_model.0.name string "Qwen2.5 7B"
10 general.base_model.0.organization string "Qwen"
11 general.base_model.0.repo_url string "https://huggingface.co/Qwen/Qwen2.5-7B"
12 general.tags array 2 of string
13 general.languages array 1 of string
14 qwen2.block_count u32 28
15 qwen2.context_length u32 32768
16 qwen2.embedding_length u32 3584
17 qwen2.feed_forward_length u32 18944
18 qwen2.attention.head_count u32 28
19 qwen2.attention.head_count_kv u32 4
20 qwen2.rope.freq_base f32 1e+06
21 qwen2.attention.layer_norm_rms_epsilon f32 9.999999974752427e-07
22 general.file_type u32 15
23 tokenizer.ggml.model string "gpt2"
24 tokenizer.ggml.pre string "qwen2"
25 tokenizer.ggml.tokens array 152064 of string
26 tokenizer.ggml.token_type array 152064 of i32
27 tokenizer.ggml.merges array 151387 of string
28 tokenizer.ggml.eos_token_id u32 151645
29 tokenizer.ggml.padding_token_id u32 151643
30 tokenizer.ggml.bos_token_id u32 151643
31 tokenizer.ggml.add_bos_token bool 0
32 tokenizer.chat_template string "{%- if tools %}\n {{- '<|im_start|>system\\" ... and 2465 more bytes
33 general.quantization_version u32 2
Entry zero earns its position. A file keys its own architecture and then prefixes its
architecture keys with that name, so the block count of this model lives at
qwen2.block_count and not at any fixed key. A reader that hard-coded the
prefix would work on exactly one family of models and fail silently on every other one,
reporting no block count at all instead of a wrong one. So the reader reads entry zero
first and builds the rest of its key names out of it.
Three of those entries are arrays and between them they are most of the header. The reader walks their elements, counts them, and keeps the count. It does not keep the elements, and that is a decision rather than an omission: 152,064 token strings are about two megabytes, nothing in this volume's arithmetic touches one of them, and the only fact anybody needs from that array is how many there are. It is the same question this world already answered about a genome in a database column, and the same answer: what nothing is going to filter on gets stored whole or not at all, never picked apart for the pleasure of having it in pieces.
Notice how much of that table is not about the model at all. A licence, a link to a
licence, a base model with a name and an organisation and a repository, a list of tags,
a list of languages, a two-and-a-half-kilobyte template describing how a conversation
should be laid out before it is encoded. None of it changes a weight. It is provenance
and packaging, and it sits in the same key-value block as the block count because the
format has exactly one place to put anything. Eight of the thirty-four entries carry
the architecture's own name as a prefix and give the model's dimensions. Ten more begin
with tokenizer, the template among them. The remaining sixteen are
general: what the thing is called, who made it, what it descends from, what licence it
carries, and which quantization it was written by. Having all of that inside the file
instead of beside it is the reason a model is one artefact and not a directory somebody
has to keep together.
Two of the entries are floating point and there are only two, in a file with seven billion floats in it: a base frequency for the positional scheme, and the small number added under a square root so that a norm cannot divide by zero. Everything else in the header is a count, a name or a flag. Three more are token identifiers, in the same sense the merge table built earlier in this volume produces them: the number a piece of text was turned into. Here they name the token that means an answer has ended, the one that means a sequence has begun, and the one used to pad. The boolean beside them says whether the beginning-of-sequence token should be put in front of an encoded prompt automatically. A client that got that flag wrong would send every prompt one token longer or shorter than the model expects, which is the kind of mistake that degrades answers without ever producing an error.
Seventeen bytes
The last info record of that file ends at 5,953,935. The first weight does not start there. It starts at 5,953,952, and the seventeen bytes in between are zeros that belong to nothing.
The reason is that a program running the model wants to map the weights and hand the pointer straight to arithmetic, and that arithmetic is much happier when a block of weights begins on a round address. So the format lets the file declare an alignment, and the tensor data begins at the first multiple of it at or after the end of the header. A file that declares nothing is written at thirty-two, which is what this model does. Every tensor's offset is then counted from that aligned start and not from the front of the file, so getting the start wrong moves every tensor in the file by the same amount.
Rounding up to a multiple is one line of arithmetic and it is the only arithmetic in the format. The header ends at i = 5,953,935 and the alignment is a = 32. Divide: 5,953,935 ÷ 32 = 186,060.46875. That is not a whole number, so the header ends in the middle of a thirty-two byte slot, and the next slot starts at 186,061 × 32 = 5,953,952. The padding is the difference, 5,953,952 − 5,953,935 = 17. In one expression, d = ⌈i ÷ a⌉ × a, and written the way a program actually does it, with integer division throwing the remainder away, d = (i + a − 1) ÷ a × a. Both give 5,953,952. When the header already ends on a multiple the padding is zero and the two expressions still agree, which is the case a hand-rolled version usually gets wrong.
Now the sum this whole page is aimed at. The file is S = 4,683,073,952 bytes. The header takes 24 bytes of fixed head, then 5,934,607 bytes of metadata, then 19,304 bytes of info records: 24 + 5,934,607 + 19,304 = 5,953,935, which is i, and 17 more of padding makes d = 5,953,952. So the weights have 4,683,073,952 − 5,953,952 = 4,677,120,000 bytes to live in. Add up what the 339 tensors cost, each one from its dimensions and its element type alone with no offset consulted, and the total T is 4,677,120,000. The residue S − d − T is 0.
Not nearly zero, and not zero to a rounding. Zero. A tensor of w weights of a type whose block holds b weights in c bytes takes w ÷ b × c bytes, and for a plain 32-bit float b is 1 and c is 4, so the norm of 3,584 numbers costs 3,584 × 4 = 14,336. For the four-bit type that most of this file is written in, b is 256 and c is 144: the embedding table is 3,584 × 152,064 = 544,997,376 weights, which is 544,997,376 ÷ 256 = 2,128,896 blocks, which is 2,128,896 × 144 = 306,561,024 bytes. Why 256 weights cost 144 bytes rather than 128 is a question this page takes as given, and the answer is not in the header.
Here is the mistake, and it is the one every hand-written reader of this format makes once. The last info record ends. The next byte is right there. Read the weights from it. The offsets in the info records are relative to the start of the tensor data, and the tensor data obviously starts where the header stopped.
Do it on the small file this page wrote and the answer arrives immediately, because that file's padding is fifty-seven bytes of zeros and the six weights are the other side of them.
$ go run ./cmd/block -mode write -noalign | tail -5
the numbers, read at the end of the info records, ignoring the declared alignment
written [0.5 1.5 2.5 3.5 4.5 5.5]
read [0 0 0 0 0 0]
0 of 6 came back the number that went in
No error. No panic. No short read. Six perfectly good 32-bit floats, all of them zero, and zero is an entirely ordinary number for a weight to be. If those six had gone into a matrix multiply the multiply would have run.
Now the same mistake on the real file, where the padding is seventeen bytes instead of fifty-seven and, far worse, is not a multiple of four. The tensor to read is the output norm, which is 3,584 plain floats and the one place in this model where a reader can look at raw numbers without dequantizing anything. Correctly first.
$ podman exec -w /bench world-go go run ./cmd/block -mode floats
block: output_norm.weight of qwen2.5:7b-instruct-q4_K_M, read as f32
tensor data at 5953952, the alignment applied
tensor offset 4677105664
read from 4683059616
numbers 3584
the first 8 of them
3.671875 3.703125 3.6875 3.75 3.765625 3.640625 3.703125 3.65625
smallest -0.173828125
largest 10.75
mean 3.8387838091169084
not a number, or infinite 0
smaller than a hundred 3584 of 3584
Those look like what they are. A norm's gains sit in a narrow band around a small number, every one of the 3,584 is smaller than a hundred, and the whole vector could be printed without a single entry in it giving anybody pause. Now move the base seventeen bytes and change nothing else.
$ podman exec -w /bench world-go go run ./cmd/block -mode floats -noalign
block: output_norm.weight of qwen2.5:7b-instruct-q4_K_M, read as f32
tensor data at 5953935, where the info records ended
tensor offset 4677105664
read from 4683059599
numbers 3584
the first 8 of them
2.194136e+11 6.509904e-13 1.3750541e-06 0.010565084 1.547449e+26 2.475899e+27 6.1897474e+26 1.5845753e+29
smallest -1.0633905095917742e+37
largest 1.7014248153468387e+38
mean 5.108488675451043e+35
not a number, or infinite 0
smaller than a hundred 408 of 3584
Read the last two lines together and the trap is visible. Not one of the 3,584 numbers is a not-a-number and not one is infinite, so every check a careful program might think to apply passes. And 408 of them are smaller than a hundred, so a run that sampled a few values and eyeballed them has a real chance of seeing four plausible numbers in a row and moving on.
Follow the mechanism back from the symptom. Seventeen is not a multiple of
four, so every float being read is built out of the last byte or three of one real float
and the first bytes of the next. A 32-bit float keeps its exponent in the top nine bits
and its mantissa in the twenty-three below, and shifting the window by one, two or
three bytes slides mantissa bits up into the exponent. Mantissa bits are as good as
random, so the exponent becomes as good as random, and a random exponent covers the
whole range a float32 can express, from about 10-38 to about
1038. Nothing in that is invalid. Every bit pattern a float32
can hold is a number, and a program asking whether these are numbers is asking a
question the format cannot answer for it.
Which is the shortest statement of why this failure is the bad kind. A crash names its
own line. A wrong number does not, and this particular wrong number does not even look
wrong. The defence is one line of arithmetic in the reader and the discipline of never
writing the offset down twice: the header carries a single field for where the data
starts, it is computed once at the end of the walk, and no caller is allowed the choice.
The -noalign flag exists so that the wrong version can be run instead of
described, and it is the only way to reach it.
The repeated tensor records
Three hundred and thirty-nine info records is more than anybody reads. It is also, once
you take one thing out of the names, a very short list. Every tensor belonging to a layer
is called blk.n.something, and the number is the only
part of that name which is not a name. Take it out and what is left is the architecture.
Before the listing, the header wants to stop being four and a half gigabytes away. The reading modes above need the model volume mounted, which means a container, which means the rest of this volume could not do arithmetic about this model on a bare workstation. So the header is dumped to JSON and committed, the way an eighty-year run of the valley's own history was dumped and committed earlier in this volume: written once, checked by the run that would have written it again, and then carried.
// exemplars/exemplars.go
//go:embed chronicle.jsonl hero.header.json small.header.json
var Files embed.FS
// Headers is the dumped model headers inside Files, one to a role of
// the kit. A role with no header here is a role nothing in this world
// can compute about, and the map is the check: a caller asks for a
// role and is told when there is not one, instead of building a
// filename out of a string and getting a missing file.
var Headers = map[string]string{
"hero": "hero.header.json",
"small": "small.header.json",
}$ podman exec -w /bench world-go go run ./cmd/block -mode dump
block: the header of qwen2.5:7b-instruct-q4_K_M, written out as json
the model 4683073952 bytes
its header 45241 bytes
34 metadata entries and 339 info records, and no weights at all
written to block85-hero.header.json
sha256 15f7511a8aa48368
against the copy this build carries
exemplars/hero.header.json 15f7511a8aa48368
and the run just made 15f7511a8aa48368
the same bytes
Forty-five kilobytes stands in for four and a half gigabytes, at a ratio of about a hundred thousand to one, and every field in it came off a real file. What that file was is an operator's decision and yours will differ: point the same mode at whatever model you pulled and you get your own header, your own tensor names and your own totals, and every mode below will then be talking about your file instead of this one. What does not differ is the arithmetic, and the last three lines are how a reader checks that the committed copy is the file rather than a story about it.
With the header carried, the listing is a program that opens nothing. Here is the whole model with the block numbers taken out.
$ go run ./cmd/block -mode stack
block: every tensor of the hero model, with the block number taken out of the name
architecture qwen2, 28 blocks, 339 tensors in the file
count name shape type bytes
1 token_embd.weight 3584 x 152064 q4_K 306561024
28 blk.N.attn_norm.weight 3584 f32 401408
28 blk.N.ffn_down.weight 18944 x 3584 q4_K x14 q6_K x14 1314410496
28 blk.N.ffn_gate.weight 3584 x 18944 q4_K 1069350912
28 blk.N.ffn_up.weight 3584 x 18944 q4_K 1069350912
28 blk.N.ffn_norm.weight 3584 f32 401408
28 blk.N.attn_k.bias 512 f32 57344
28 blk.N.attn_k.weight 3584 x 512 q4_K 28901376
28 blk.N.attn_output.weight 3584 x 3584 q4_K 202309632
28 blk.N.attn_q.bias 3584 f32 401408
28 blk.N.attn_q.weight 3584 x 3584 q4_K 202309632
28 blk.N.attn_v.bias 512 f32 57344
28 blk.N.attn_v.weight 3584 x 512 q4_K x14 q6_K x14 35524608
1 output.weight 3584 x 152064 q6_K 447068160
1 output_norm.weight 3584 f32 14336
15 names, and 12 of them are the ones that repeat
in front of the stack 306561024 bytes
the stack itself 3923476480 bytes, over 28 blocks
behind it 447082496 bytes
the three together 4677120000 bytes
the smallest block 131135488 bytes
the largest block 149112832 bytes
so the blocks are the same twelve names and not the same number of bytes
That is the architecture, and it arrived as a directory listing. One table in front, of 152,064 rows and 3,584 columns, which is every token in the vocabulary given a position in the same space the model thinks in. Twelve names in the middle, each appearing twenty-eight times. Two things behind: a final norm, and an output table the same size as the embedding table that turns a vector back into a score for every token in the vocabulary.
The twelve are the ones to read carefully, because the arithmetic they name is
arithmetic this book has already done by hand. attn_q,
attn_k and attn_v are the three projections that turn a token
into a query, a key and a value. attn_output is what mixes the heads back
together afterwards. attn_norm and ffn_norm steady the numbers
going into each half. And ffn_gate, ffn_up and
ffn_down are a plain feed-forward layer, wider in the middle than at either
end: 3,584 in, 18,944 across, 3,584 out.
Look at what the widths confirm. The query projection is 3,584 by 3,584, which is twenty-eight heads of 128. The key and value projections are 3,584 by 512, which is four heads of 128, which is the head count the metadata gave. And the biggest lines in the table are feed-forward, not attention. The three feed-forward rows come to 3,453,112,320 bytes of the stack's 3,923,476,480, which is close to nine tenths of it, and the four attention weight rows come to 469,045,248, which is under an eighth. Attention is where the interesting behaviour is; it is not where the weights are.
The repetition is the architecture, and that is not a disappointment. A network in this
book has always been layers run in a fixed order, each one's outputs becoming the next
one's inputs, with the topology living in the program and only the numbers living in
the file. This is the same arrangement at a scale where the numbers are four and a half
gigabytes: a program that knows what one block does, and a file that hands it
twenty-eight sets of weights to do it with. There is no wiring in the file. There is a
count called block_count, a naming convention, and a loop in whatever runs
the model.
Which puts the block count above every other number in the header. Get the embedding width wrong and every matrix is the wrong size and nothing runs at all. Get the block count wrong and a program finds every tensor it goes looking for, runs without complaint, and is running a model that is not the one on the disk.
A row saying twenty-eight is a claim about twenty-eight things. Take one block apart and hold the rest against it.
$ go run ./cmd/block -mode layer -layer 0
block: block 0 of the hero model, tensor by tensor
name shape type offset bytes
attn_k.bias 512 f32 438667264 2048
attn_k.weight 3584 x 512 q4_K 438669312 1032192
attn_norm.weight 3584 f32 306561024 14336
attn_output.weight 3584 x 3584 q4_K 439701504 7225344
attn_q.bias 3584 f32 446926848 14336
attn_q.weight 3584 x 3584 q4_K 446941184 7225344
attn_v.bias 512 f32 454166528 2048
attn_v.weight 3584 x 512 q6_K 454168576 1505280
ffn_down.weight 18944 x 3584 q6_K 306575360 55695360
ffn_gate.weight 3584 x 18944 q4_K 362270720 38191104
ffn_norm.weight 3584 f32 438652928 14336
ffn_up.weight 3584 x 18944 q4_K 400461824 38191104
12 tensors, 149112832 bytes
against the other 27 blocks
same names and shapes 27
and the same element types too 11
same shapes, some type elsewhere 16
Twelve tensors: two norms, three biases, four attention weights and three feed-forward weights. All twenty-seven other blocks carry the same twelve names at the same twelve sizes, and the comparison is a run rather than a sentence, because a claim that twenty-eight things are the same is exactly the claim nobody checks.
Eleven of them are also written in the same element types, and sixteen are not, which is the one place the repetition is not perfect. Two of the twelve names are held at a wider type in some blocks and a narrower one in others, and that shows up in the last two lines of the collapsed listing: the smallest block is 131,135,488 bytes and the largest is 149,112,832. The file is written at more than one precision. Where the wider ones fall is the file's own business, and this page reports it without explaining it.
Which leaves the claim this page opened with. Add it all up.
$ go run ./cmd/block -mode account
block: the hero model added up out of its own header
the header, in the four pieces it is made of
the four numbers at the front 24
34 metadata entries 5934607
339 info records 19304
padding to alignment 32 17
tensor data starts at 5953952
the tensors, gathered by element type
type tensors weights bytes
f32 141 333312 1333248
q4_K 169 6094061568 3427909632
q6_K 29 1521221632 1247877120
all 339 7615616512 4677120000
header and padding 5953952
every tensor's bytes 4677120000
the two added together 4683073952
the file on disk 4683073952
left over 0
339 tensors start where the one before them ended, and 0 do not
Zero left over. The last line is the second half of the same check and a stronger one: every tensor begins at the byte where the tensor before it ended, all 339 of them, so the offsets the file wrote down agree with the sizes computed from dimensions and types alone. Two independent accounts of where four and a half gigabytes went, agreeing to the byte.
The weights column is the number people quote when they name a model. 7,615,616,512 of them, which is where the seven in the name comes from, and 141 of the 339 tensors hold 333,312 of those weights between them. The norms and the biases are almost nothing. Nearly all of a model is three matrices a block and two tables at the ends.
The contiguity deserves a sentence of its own, because it explains why the info records carry an offset at all. If every tensor follows the one before it, a program could work out where each begins by adding up sizes, and the offsets would be redundant. They are written down anyway, and the payoff is that a program wanting one tensor out of 339 can map the file and go straight to it without having read a byte of the other 338. Sizes and offsets are two accounts of one layout, and a file where they disagree is a file whose reader should stop. That is what the last line checks, and on this file it has 339 chances to find a disagreement and finds none.
The listing and the accounting are the reader's, not the model's, so point them at the
other file in the kit. Nothing below is a measurement of anything: these are two file
layouts read off two disks, the four-and-a-half gigabyte
qwen2.5:7b-instruct-q4_K_M above and the four hundred megabyte
qwen2.5:0.5b-instruct-q4_K_M here, and no rate, load time or answer from
either of them is anywhere near this page.
$ go run ./cmd/block -mode stack -role small
block: every tensor of the small model, with the block number taken out of the name
architecture qwen2, 24 blocks, 290 tensors in the file
count name shape type bytes
1 token_embd.weight 896 x 151936 q8_0 144643072
24 blk.N.attn_norm.weight 896 f32 86016
24 blk.N.ffn_down.weight 4864 x 896 q4_K x12 q6_K x12 72317952
24 blk.N.ffn_gate.weight 896 x 4864 q5_0 71909376
24 blk.N.ffn_up.weight 896 x 4864 q5_0 71909376
24 blk.N.ffn_norm.weight 896 f32 86016
24 blk.N.attn_k.bias 128 f32 12288
24 blk.N.attn_k.weight 896 x 128 q5_0 1892352
24 blk.N.attn_output.weight 896 x 896 q5_0 13246464
24 blk.N.attn_q.bias 896 f32 86016
24 blk.N.attn_q.weight 896 x 896 q5_0 13246464
24 blk.N.attn_v.bias 128 f32 12288
24 blk.N.attn_v.weight 896 x 128 q5_0 x12 q8_0 x12 2408448
1 output_norm.weight 896 f32 3584
14 names, and 12 of them are the ones that repeat
in front of the stack 144643072 bytes
the stack itself 247213056 bytes, over 24 blocks
behind it 3584 bytes
the three together 391859712 bytes
the smallest block 9717248 bytes
the largest block 10883840 bytes
so the blocks are the same twelve names and not the same number of bytes
$ go run ./cmd/block -mode account -role small | tail -8
header and padding 5948224
every tensor's bytes 391859712
the two added together 397807936
the file on disk 397807936
left over 0
290 tensors start where the one before them ended, and 0 do not
Twenty-four blocks instead of twenty-eight, an embedding 896 wide instead of 3,584, and 290 info records instead of 339. The twelve repeating names are the same twelve, in the same order, doing the same jobs. And the sum comes out the same way: 391,859,712 bytes of tensors plus 5,948,224 of header and padding is 397,807,936, which is the file, with nothing left over.
Two details in that listing carry the warning. There are fourteen names here and
fifteen in the larger file, and the missing one is output.weight: this
model has no separate output table, so the 3,584 bytes behind the stack are its final
norm and nothing else, which is 896 floats at four bytes apiece. And the header is
5,948,224 bytes against the larger file's 5,953,952, which is very nearly the same
number for a file roughly a twelfth the size. A header is mostly a vocabulary, these
two carry 151,936 and 152,064 tokens, and the part of a model that describes it does
not grow with the model. So the header is one and a half percent of this file and a
little over a tenth of a percent of the other one.
Why this works
The move on this page is older than models and it comes up any time a program has to read something it did not write. There are two ways to lay out a binary file. One is to put markers in it, so a reader can hunt for the next thing and recover when it gets lost. The other is to write lengths in front of values and nothing else, so a reader knows where it is only because it knows where it has been. This format takes the second, and every format that has to be read in one pass over a stream takes the second, because searching for a marker in four and a half gigabytes is not a thing anybody wants to do.
The price of that choice is the whole of what makes the alignment dangerous. A length-prefixed format has no way to notice that a reader is one byte out. It cannot, because any byte is a legal byte at any position. So the reader carries the burden that the format gave up: it computes each position once, from the position before it, and it never lets a caller supply one. In this reader the tensor data offset is a field on the header, worked out at the end of the walk, and the only way to get a wrong one is a flag that exists to demonstrate it.
The second move generalises further. A file you can add up is a file you understand well enough to argue with. It is not the same as understanding the model, and this page has learned nothing at all about what the weights mean. But an object whose every field has been parsed and whose every byte has been assigned to something is not a black box, whatever else it is, and the difference between a total that comes out exactly and a total that comes out about right is the difference between having read the format and having read a description of it.
There is a third habit on this page and it is the cheapest of the three. When a program has to read something it did not write, write one first. The file this chapter opened with is four hundred and eight bytes and took an afternoon, and it is the only reason anybody can say the reader works instead of saying the reader agrees with itself. The move carries to a wire protocol, an archive format, a save file some other program wrote, the framing on a message queue: build the smallest instance the format allows, put known values in it, and make the reader hand them back. A parser with no writer beside it is a hypothesis.
Why headers price model files
Everything on this page rests on a reader, so the reader gets tests, and none of them needs a model. Four hundred and eight bytes of this world's own standing in for four and a half gigabytes is the substitution this volume's pinning tests already made with forty-nine, and it works for the same reason: the code under test cannot tell how big the file is.
$ go test ./internal/lang/ -run 'ASketchWrittenAndReadIsTheSameNumbers|TheAlignmentComesOutOfTheFilesOwnMetadata|ATensorReadWhereTheInfoRecordsEndedIsNotTheTensor|AFileThatDoesNotOpenGGUFIsRefused|ATruncatedHeaderNamesTheFieldItDiedOn|ATensorsBytesComeFromItsShapeAndTypeAlone|TheCommittedHeadersAccountForTheirFiles|AHeaderSurvivesBeingWrittenDown' -v
=== RUN TestASketchWrittenAndReadIsTheSameNumbers
gguf_test.go:71: 408 bytes: 327 of header, 57 of padding, 24 of weights, and the counts and the array survived
--- PASS: TestASketchWrittenAndReadIsTheSameNumbers (0.00s)
=== RUN TestTheAlignmentComesOutOfTheFilesOwnMetadata
gguf_test.go:103: 16, 64 and 4096 all read back, and a file that declares nothing reads back at 32
--- PASS: TestTheAlignmentComesOutOfTheFilesOwnMetadata (0.00s)
=== RUN TestATensorReadWhereTheInfoRecordsEndedIsNotTheTensor
gguf_test.go:138: 57 bytes of padding ignored: all 6 weights read, none of them a number that was written, and no error anywhere
--- PASS: TestATensorReadWhereTheInfoRecordsEndedIsNotTheTensor (0.00s)
=== RUN TestAFileThatDoesNotOpenGGUFIsRefused
gguf_test.go:150: refused, and the message names both what was there and what should have been: lang: this file opens "GGML" and a GGUF file opens "GGUF"
--- PASS: TestAFileThatDoesNotOpenGGUFIsRefused (0.00s)
=== RUN TestATruncatedHeaderNamesTheFieldItDiedOn
gguf_test.go:162: a header cut at 200 of 408 bytes: lang: metadata entry 4 of 5: EOF
--- PASS: TestATruncatedHeaderNamesTheFieldItDiedOn (0.00s)
=== RUN TestATensorsBytesComeFromItsShapeAndTypeAlone
gguf_test.go:194: f32, q4_K and q6_K sized from shape alone; a part block and an unknown type both refused
--- PASS: TestATensorsBytesComeFromItsShapeAndTypeAlone (0.00s)
=== RUN TestTheCommittedHeadersAccountForTheirFiles
gguf_test.go:216: hero: 5953935 bytes of header, 17 of padding, 339 tensors, 0 bytes left over
gguf_test.go:216: small: 5948196 bytes of header, 28 of padding, 290 tensors, 0 bytes left over
--- PASS: TestTheCommittedHeadersAccountForTheirFiles (0.00s)
=== RUN TestAHeaderSurvivesBeingWrittenDown
gguf_test.go:240: 38775 bytes of committed header read and written back byte for byte, and an unknown field is refused
--- PASS: TestAHeaderSurvivesBeingWrittenDown (0.00s)
The third one is the failure above, pinned so it stays a failure: it asserts that ignoring the padding produces numbers, that none of them is a number that was written, and that nothing anywhere returns an error. A test that only checked for an error would pass on a reader that had been fixed by accident and on one that had never been broken.
The sixth is the accounting rule reduced to arithmetic, and it also checks the two refusals that keep it honest. A tensor whose weight count is not a whole number of blocks and a tensor of an element type this reader has never heard of are both errors, because a size worked out from a type nobody checked is a number that looks exactly like a right one. That is the same instinct the whole package is written with: say what is missing, name the thing it was missing from, and never carry on with a plausible zero.
- Walk the four parts of the file in order and say what has to have been read before each one can be found, and why the format needs no delimiter anywhere.
- Given a header ending at 5,953,935 and an alignment of 32, produce 5,953,952 and 17 on paper, and say where each of the 339 tensor offsets is measured from.
- Say why the alignment cannot be known until the metadata has been read, and what a file that declares no alignment is written at.
- Explain why a tensor's byte size is computed from its dimensions and element type instead of from the offset of the tensor after it, and what the second calculation is good for.
- Name the twelve tensors of one block, group them into the two halves they belong to, and say which of the twelve are the ones the file spends most of its bytes on.
- Say what the reader has to hard-code to find
qwen2.block_count, and what it would break if it hard-coded the prefix instead.
Exercise 1 — declare a bigger alignment. Write this page's own four-hundred-byte file at an alignment of 4096 instead of 64 and predict, before running it, what changes and what does not.
-mode write -align 4096. The header is the same length it was, because
the alignment key holds a number and the number takes four bytes at any value, so
the last info record still ends at 327. The padding grows from 57 to 3,769, the
tensor data starts at 4,096, and the file grows to 4,120 bytes for the same six
weights. The tensor's own offset stays 0: an offset is measured from the start of
the data and knows nothing about where that is.
The point of running it is the ratio. On a four-hundred-byte file an alignment of 4,096 is ninety percent padding, and on a four-and-a-half-gigabyte file it would be too small to see. Alignment costs a bounded number of bytes once, which is why nobody thinks about it and why a reader that gets it wrong is wrong everywhere at once.
Exercise 2 — find the wider tensors. The listing says two of the twelve names are held at a wider element type in some blocks. Work out which blocks, and what fraction of the file's bytes they are.
-mode layer -layer 1 and a few others will show you individual blocks,
but the quick way is to notice that the collapsed listing already gives the counts,
fourteen and fourteen, and that the totals are there too. The two mixed rows come to
1,314,410,496 and 35,524,608 bytes together, which is 1,349,935,104 against the
4,677,120,000 the tensors occupy, or a little under twenty-nine percent. That is the
bytes of both types in those rows, not the bytes of the wider one alone.
Separating those two is a different calculation and needs the per-tensor sizes
and not the row totals, which is exactly the sort of thing the committed header
is for. Run -mode layer on block 1 and on block 3 and compare the
element type column: the pattern of which blocks get the wider type is a decision
whoever produced the file made, and it is written down in the file rather than
anywhere else.
Exercise 3 — break the file on purpose. Corrupt one byte of a length in the file this page wrote and predict what the reader says.
Write the file, then change the byte at offset 24, which is the length of the first key, and open it again. Raising it from 20 to 21 makes the key swallow one byte of what follows; the four bytes after that are then read as a value type and come out as 100,663,296, so the reader stops at offset 57 saying that is not a type it knows. Raising it to 64 sends the reader far enough that the next thing it takes for a string length is 7,449,358,543,090,511,616, which the reader refuses at offset 101 on the grounds that no string is that long. Push it to 255 and the error prints a key name made of the whole rest of the header, which is its own diagnostic: a message quoting a key that is obviously not one says exactly where the reader lost the thread.
Which is what the two caps in the reader are for. A string length above sixty-four megabytes and a tensor count above sixteen million are both refused, and neither is a real limit on a real file. They are there so that a file which has gone wrong produces an error near the mistake instead of an allocation the machine cannot serve, and they are the cheapest defence available to any reader of a format where every length is a number somebody else wrote.