Eight Gigabytes
Whether it fits is a sum
Ask whether a model runs on an eight-gigabyte machine and the answer comes back with a shrug in it. Somebody ran one like it once. Somebody else says you need a graphics card. A forum post says it swaps. None of that is a measurement and none of it is about your machine, and the reason the question attracts opinions is that most people asking it have never had the two files in front of them.
What a loaded model costs to hold is three sums: the weights, the cache and the overhead. Two of them are arithmetic over numbers that are already in the file. The third is measured once. This world has had both files for five chapters. The header of the big one was walked field by field, its three hundred and thirty-nine tensors were sized from their shapes alone, and the sizes added up to the file on disk with nothing left over: 4,683,073,952 bytes at 4.9132 bits a weight once the block overhead is counted. The small one came to 397,807,936 bytes at 6.3455, and chapter 86 already made the point that two files carrying the same four-bit tag are not two files priced alike.
Everything below is built on those committed headers. The offline calculator also reads committed allocation fixtures, so its arithmetic needs no model server. The live weighing and pacing modes do. Hardware changes the rate: how fast the tokens come, and whether the thing sits in a graphics card's memory or spills into the processor's. That moves the rate by an order of magnitude and moves none of the arithmetic, so this page prints both kinds of number and keeps them apart.
A sum is not the same claim as a reading off a running process, and the page keeps that distinction too. A real allocation moves with the backend, the cache format, the batch size, the parallelism and whether the weights sit on a card or in main memory; file bytes need not be resident bytes, because mappings and conversions land somewhere. What the arithmetic settles is what the file and the settings demand, which is the part that is the same everywhere.
By the end of it there is a program that takes a header, a context length and a ceiling and says yes or no with the working shown; a table of what a server actually holds beside a model, weighed at nine context lengths; one configuration that fits and one that does not, with the knob between them named; and the daemon from the last volume with one job added to it, running between two ticks rather than inside one, with the bill for having a model beside this world measured in the world's own units.
The new code goes in internal/lang/room.go. It is bytes from end to end and
it opens nothing:
// Cache is the four numbers one model's key-value cache is a product
// of, read out of its own header.
//
// The four fields are the whole of it. Layers is how many attention
// blocks the file stacks; Heads is how many key-and-value heads each
// block keeps, which on this architecture is fewer than it has query
// heads; Width is one head's numbers, which is the embedding width
// divided by the query head count; and Elem is how many bytes one
// number of cache costs, which is a setting of the server and not a
// fact about the file.
type Cache struct {
Layers int64
Heads int64
Width int64
Elem int64
}
// ...
// Token is the raw KV storage per token slot, in bytes. A backend may
// reserve all context slots at load time, before tokens occupy them.
func (c Cache) Token() int64 {
return KeysAndValues * c.Layers * c.Heads * c.Width * c.Elem
}
// At is the cache at a context length.
func (c Cache) At(ctx int64) int64 { return c.Token() * ctx }
and cmd/fits is the bench that drives it. Its first mode reads the committed
header of one model and prints the three sums with everything they were made of:
$ go run ./cmd/fits -mode sums
fits: what one model costs to hold, in three sums
the model qwen2.5:7b-instruct-q4_K_M, in the hero role
pinned at 2bada8a745067700
the context 4096 tokens, and this model was trained at 32768
a cache number 2 bytes, assumed by this calculator
conversations 1 in flight
the weights, which are the file
the file on disk 4,683,073,952
tensors in it 339
bits a weight, block counted 4.9132
the cache, which is arithmetic
attention blocks 28
key and value heads apiece 4
numbers in one head 128
bytes one number costs 2
keys, and values as well 2
so, one token of conversation 57,344
and 4096 tokens of it 234,881,024
for every conversation held 234,881,024
the overhead, which is measured
what the server holds besides 163,753,568
weighed against ollama 0.12.3, at 4096 tokens of context
all three 5,081,708,544
the cache is 4.6% of this estimate; one extra cache alone adds 234,881,024
fixture arithmetic, not measured process/host/GPU memory or concurrency
Five gigabytes and eighty-one megabytes, on a machine that has eight thousand million bytes to give. Every figure above the total came from the header, the kit or a number somebody weighed, and the banner says which each one is: the context is the operator's setting, printed off the kit and not typed into the program; the two bytes a cache number costs is this calculator's assumption, not a queried server setting; the 4,096 sits against the 32,768 the file itself was trained at, because those two numbers are different questions and the file only answers the second one.
$ go run ./cmd/fits -mode sums -role small
fits: what one model costs to hold, in three sums
the model qwen2.5:0.5b-instruct-q4_K_M, in the small role
pinned at c5396e06af294bd1
the context 4096 tokens, and this model was trained at 32768
a cache number 2 bytes, assumed by this calculator
conversations 1 in flight
the weights, which are the file
the file on disk 397,807,936
tensors in it 290
bits a weight, block counted 6.3455
the cache, which is arithmetic
attention blocks 24
key and value heads apiece 2
numbers in one head 64
bytes one number costs 2
keys, and values as well 2
so, one token of conversation 12,288
and 4096 tokens of it 50,331,648
for every conversation held 50,331,648
the overhead, which is measured
what the server holds besides 320,032,704
weighed against ollama 0.12.3, at 4096 tokens of context
all three 768,172,288
the cache is 6.6% of this estimate; one extra cache alone adds 50,331,648
fixture arithmetic, not measured process/host/GPU memory or concurrency
Two details in that run set the boundary. The small model's weights are a twelfth of the big one's and its overhead is nearly twice as large, which is already a warning that the third term is not a percentage of the first. And the model that is a twelfth of the size costs a sixth as much to hold, because two of its three terms have almost nothing to do with how many parameters it has.
The term that grows while you are talking
The weights are a file and the file does not move. Load the model, ask it nothing, ask it for a novel: 4,683,073,952 bytes either way. The second term behaves in the opposite way, and confusing the two is where nearly every wrong estimate of what a model costs comes from.
Chapter 82 built one attention head by hand on four tokens and left a fact behind that is about to become a line item. Every token in a row is turned into a query, a key and a value; every query is dotted against every key. A model reading a conversation does that again for every new token, and it would be doing the same arithmetic over the same earlier tokens each time, so it keeps the keys and the values it has already worked out and reuses them. That store is the cache, and it holds one key and one value for every token, in every block, for every head that has them.
So the size of it is a product of five numbers, four of which are in the header. The file says twenty-eight attention blocks, four key-and-value heads apiece, and an embedding width of 3,584 across twenty-eight query heads, which makes one head 128 numbers wide. The fifth depends on the server's cache format: this calculator assumes two bytes per number. Multiply those out and double the answer, because there are keys and there are values.
Numbers first. One block of the big model keeps, for one token, four keys of 128 numbers each: 4 × 128 = 512 numbers. It keeps four values the same size, another 512. At two bytes a number that block spends 2,048 bytes on one token. There are twenty-eight blocks, so one token of conversation costs 28 × 2,048 = 57,344 bytes.
Fifty-seven kilobytes for one token, and there is nothing exotic in the arithmetic: it is a count of the boxes multiplied by the size of a box. A context of 4,096 tokens is 4,096 × 57,344 = 234,881,024 bytes. The trained context of 32,768 is eight times that, 1,879,048,192 bytes, which is two fifths of the file the weights came in.
Written out, with K and V standing for the two arrays and nothing else standing for anything:
cache = 2 × L × H × W × n × b
Four of the six are read out of the header and never chosen. n is chosen, and it is the only one on the list that anybody turns a dial on, which is what makes the whole of this chapter's second half an argument about one number.
Two details in that product earn a sentence each. The head width is a division rather than a field: the format writes down the embedding width and the query head count and leaves the head width to whoever wants it, so a reader that guessed it would be guessing. And the key-and-value head count is four while the query head count is twenty-eight, which is a design decision made by the people who trained this model and is the single largest saving in the whole term. Twenty-eight key-and-value heads would have made a token cost 401,408 bytes instead of 57,344, and the same conversation seven times dearer.
The number the file does not have
Two sums down and the third has to be got a different way, because nothing in the header says it. A server holding a model holds working room as well: somewhere to put the arithmetic of one batch of tokens while it is being done. How much is a decision the server's own code makes out of the model's width and the batch it works in, and a program that guessed at it would be publishing a guess in the same column as two exact figures.
So it gets weighed. The server is asked to put a model in memory at a named context and
then asked how much it needs to hold it; the two terms that can be computed are subtracted;
what remains is the residual this page calls overhead. Ollama's /api/ps reports
size and size_vram; subtracting file bytes and raw cache bytes
from size defines this residual, without establishing a process-RSS or
whole-machine measurement. A match in one row follows from the subtraction itself.
Other contexts test how well that estimate carries.
The committed capture used Ollama 0.12.3 on an eight-core Ryzen 7 3700X with 30 GB
of RAM, CPU-only. The digest-pinned image appears in the volume's run scripts.
They explicitly set OLLAMA_MAX_LOADED_MODELS=1 and
OLLAMA_HOST=0.0.0.0:11434. Requests name their context length; the
calculator assumes two-byte, f16 KV elements and one cache slot by default.
The capture does not record the effective cache type, num_batch,
OLLAMA_NUM_PARALLEL or Flash Attention setting. Those remain unknown,
not measured defaults. -elem and -held change arithmetic,
not server configuration.
Record those effective settings, model digest, backend/device placement and memory counters before making a new capacity claim. Keep the old capture as evidence of that run; a dependency upgrade needs a new comparison, not replacement numbers guessed from the file.
The client needs one method it has not had. Asking a server for a completion loads the model as a side effect, and timing that request would be timing a disk read with a sentence on the end of it. A request with no prompt in it loads and answers nothing:
// Load puts a model in memory at a context and asks it for nothing.
//
// Loading and generating are two requests here rather than one, and
// keeping them apart is the difference between a rate about a model and
// a rate about a disk. The first answer out of a model that is not
// resident waits for several gigabytes to be read; the second does not.
// A page that timed one request and called the answer tokens a second
// would be quoting the disk.
//
// The server answers this with an empty response and its own word for
// why it stopped, which is the word load and not the word stop.
func (c *Client) Load(ctx context.Context, model string, want int) error {
if model == "" {
return fmt.Errorf("lang: a load with no model named")
}
body, err := json.Marshal(loadReq{Model: model, Options: ctxOnly{NumCtx: want}})
if err != nil {
return fmt.Errorf("lang: %s: %w", model, err)
}
var res generateRes
if err := c.post(ctx, "/api/generate", body, &res); err != nil {
return err
}
if !res.Done {
return fmt.Errorf("lang: %s: the server sent a load it had not finished", model)
}
if res.Response != "" || res.Eval != 0 {
return fmt.Errorf("lang: %s: a load answered with %d tokens, which makes it a generation",
model, res.Eval)
}
return nil
}The last check is the one that matters. If a server ever answers this with a token in it, the request stopped being a load and the run says so instead of quietly counting a completion nobody meant to spend. Beside it goes a reader for what the server is holding, which takes four fields off the answer and drops the rest: a name, the bytes, the context it was loaded at, and how many of those bytes are on a graphics card. There is no timestamp on it, and the relative-time column the server also offers is not read, for the reason every listing in this volume gives: a field never read is a field never printed by accident.
This runs inside the toolchain container on the closed bridge, and it uses the small model because the small model is the one this world is allowed to load as often as it likes. Nine loads, no generations, and one question after each:
$ podman exec -w /bench world-go go run ./cmd/fits -mode weigh -role small -rungs 1024,2048,4096,6144,8192,12288,16384,24576,32768
fits: what the server holds beside the weights and the cache
the server world-lm:11434, waiting up to 10m0s for an answer
the model qwen2.5:0.5b-instruct-q4_K_M, in the small role
pinned at c5396e06af294bd1
the file 397,807,936 bytes
a cache number 2 bytes, so 12,288 bytes a token
each rung is one load and no generation at all
context server says the cache the residue server's step cache's step
1024 728,850,688 12,582,912 318,459,840 - -
2048 741,957,888 25,165,824 318,984,128 13,107,200 12,582,912
4096 768,172,288 50,331,648 320,032,704 26,214,400 25,165,824
6144 794,386,688 75,497,472 321,081,280 26,214,400 25,165,824
8192 820,601,088 100,663,296 322,129,856 26,214,400 25,165,824
12288 941,189,376 150,994,944 392,386,496 120,588,288 50,331,648
16384 1,119,447,296 201,326,592 520,312,768 178,257,920 50,331,648
24576 1,475,963,136 301,989,888 776,165,312 356,515,840 100,663,296
32768 1,832,478,976 402,653,184 1,032,017,856 356,515,840 100,663,296
the residue column is what this world writes down as the overhead, and it is
the only figure in this chapter's arithmetic that came off a socket
what the committed table says, against what this run just measured
context committed measured here difference
1024 318,459,840 318,459,840 0
2048 318,984,128 318,984,128 0
4096 320,032,704 320,032,704 0
6144 321,081,280 321,081,280 0
8192 322,129,856 322,129,856 0
12288 392,386,496 392,386,496 0
16384 520,312,768 520,312,768 0
24576 776,165,312 776,165,312 0
32768 1,032,017,856 1,032,017,856 0
Read the last two columns first, because between them they check that the cache arithmetic is right at all. From 4,096 tokens to 8,192, in two steps of 2,048, the server's own figure goes up by 26,214,400 bytes each time while the cache term goes up by 25,165,824. Near enough that the arithmetic is plainly what is moving; far enough apart to show that something else is moving with it, at 1,048,576 bytes a step, or half a kilobyte a token. The residual grows with it; these counters alone do not identify which internal buffer supplied that difference.
Then read the residue column top to bottom. It sits between 318 and 322 megabytes for the first five rows, over a stretch where the cache goes up eightfold, and then at 12,288 tokens it stops sitting still: 392 megabytes, then 520, then 776, then 1,032. By the last row the working room is more than three times what it was on the flat, and is bigger than the cache and bigger than the file. This capture reports 1,832,478,976 bytes for the model at that context, about four and a half times its file size.
That is the finding of the mode and it is why the overhead is written down as a table rather than as a constant. Each row carries the context it was taken at, and the type that holds one marks reuse at a longer context:
// Weighed is an overhead figure and where it came from.
//
// The two fields that are not the number are the point of the type. A
// figure measured against one server is not a figure about another,
// and a figure measured at one context is not a figure about another
// either, so neither can be dropped on the way to a page. Floor says
// the figure was taken at a shorter context than the one it is being
// used at. Treating that extrapolation as a lower bound requires an
// assumption of monotonic overhead on the same backend and settings.
type Weighed struct {
Bytes int64
Ctx int64
Server string
Floor bool
}
The last table in the run is the fixture check. The nine numbers this run measured are
the nine numbers committed in room.go, to the byte, which is what makes
the offline budgets reproducible without a container. A fresh server's report can
differ; reproducing arithmetic over a fixture does not reproduce the original hardware
conditions.
The big model is weighed at one context and no others, because loading it costs most of the memory this world is allowed and this volume loads it once. Asking for its overhead at 32,768 tokens therefore gets the 4,096-token figure back with a flag on it saying so.
The Floor flag keeps its original API name, but represents an assumption:
overhead stays constant or grows on the same backend and settings. The small model's
measured rows support that pattern over those contexts; they do not prove it for the
big model. Certain() checks only whether fixture contexts match. Neither
method certifies physical capacity. A reused residual is an extrapolation that needs
measurement at the intended configuration.
Two models against eight billion bytes
A budget wants a ceiling, and the ceiling here is a flag with eight thousand million bytes in it. That is smaller than the 8,589,934,592 bytes a machine sold with eight gigabytes of memory actually carries, and the gap is deliberate. Nothing gets a whole machine. An operating system wants some of it, this world's database wants some, and the daemon that runs The Hollow wants some, so a model server handed every byte the hardware has would have been priced against a machine nobody owns. This ceiling is smaller, but the gap does not establish enough headroom for those services. Measure and reserve their peak use separately; the ceiling is a planning input, adjustable by flag.
The sum below explores a hypothetical configuration: the hero and small model resident
together, each with one cache slot. The reference server cannot hold this pair because
OLLAMA_MAX_LOADED_MODELS=1 limits it to one resident model. Asking for the
embedding model replaces the currently loaded model; it does not join the other two.
The two-model sum neither describes that server's observed residency nor measures the
unpriced embedding model. A real multi-model deployment needs its own allocation capture.
$ go run ./cmd/fits -mode room
fits: estimated two-model residency, against a ceiling
the ceiling 8,000,000,000 bytes
the context 4096 tokens for both
conversations 1 of each in flight
a cache number 2 bytes
the cache term counted
model weights cache overhead all three
hero 4,683,073,952 234,881,024 163,753,568 5,081,708,544
small 397,807,936 50,331,648 320,032,704 768,172,288
hero qwen2.5:7b-instruct-q4_K_M
small qwen2.5:0.5b-instruct-q4_K_M
what the configuration costs 5,849,880,832
the ceiling 8,000,000,000
left over 2,150,119,168
the estimate is under the ceiling; actual fit is unverified, and each overhead fixture matches this context, not necessarily this backend or concurrency
hypothetical residency; the reference server loads at most one model
reserve host/process memory separately; measure the actual backend and concurrency
2,150,119,168 allows at most 9 extra hero cache(s) in this estimate, at 234,881,024 apiece; concurrency is unverified
The estimate leaves 2,150,119,168 bytes. Dividing by 234,881,024 gives nine extra cache slots if every other term stays fixed. That is a cache-only upper bound within this estimate, not nine proven conversations. Parallel requests share weights, but can change batching, work buffers, scheduling and latency. Set the desired server parallelism and measure the workload before admitting it.
The ladder prices the same pair across context lengths. Most rows reuse a shorter-context hero residual, which the output marks explicitly:
$ go run ./cmd/fits -mode ladder
fits: estimated two-model residency across context lengths
the ceiling 8,000,000,000 bytes
conversations 1 of each in flight
a cache number 2 bytes
context both caches both overheads all of it left under cap
4096 285,212,672 483,786,272 5,849,880,832 2,150,119,168 yes 9
6144 427,819,008 484,834,848 5,993,535,744 2,006,464,256 yes* 5
8192 570,425,344 485,883,424 6,137,190,656 1,862,809,344 yes* 3
12288 855,638,016 556,140,064 6,492,659,968 1,507,340,032 yes* 2
16384 1,140,850,688 684,066,336 6,905,798,912 1,094,201,088 yes* 1
24576 1,711,276,032 939,918,880 7,732,076,800 267,923,200 yes* 0
32768 2,281,701,376 1,195,771,424 8,558,354,688 -558,354,688 NO 0
* a shorter-context overhead is reused; calling it a floor assumes monotonic growth.
under compares only the estimate to the ceiling; no row proves actual fit.
The reference server loads at most one model, not this hypothetical pair.
cap is a cache-only upper bound within the estimate, not measured concurrency.
Reserve host/process memory and remeasure overhead at the intended concurrency.
At 4,096 tokens the pair's estimate is 5,849,880,832 bytes; at 32,768 it is 8,558,354,688, exceeding the ceiling by 558,354,688. The raw cache reservation grows with the selected context, while the table supplies or extrapolates the residual.
Read the cap column as a cache-only limit. At 24,576 tokens
the pair is inside the ceiling by 267,923,200 bytes, and one more conversation of the
big model would want 1,407,647,744. Longer contexts leave less estimated room for
parallel slots even before their other costs enter the budget.
Only the 4,096 row has matching-context fixtures for both models. A star marks an under-ceiling estimate that reuses the hero's 4,096-token residual. The last row also uses that extrapolation; its NO rejects the estimate, without proving how a different backend would allocate memory.
The mistake is easy to make and hard to see, so the bench can make it on purpose. The
-nocache flag leaves the second term out and prices the weights and the
overhead alone, which is how almost everybody estimates this:
$ go run ./cmd/fits -mode room -ctx 32768 -nocache
fits: estimated two-model residency, against a ceiling
the ceiling 8,000,000,000 bytes
the context 32768 tokens for both
conversations 1 of each in flight
a cache number 2 bytes
the cache term left out, which is what this run is for
model weights cache overhead all three
hero 4,683,073,952 0 163,753,568 4,846,827,520
small 397,807,936 0 1,032,017,856 1,429,825,792
hero qwen2.5:7b-instruct-q4_K_M
small qwen2.5:0.5b-instruct-q4_K_M
what the configuration costs 6,276,653,312
the ceiling 8,000,000,000
left over 1,723,346,688
the estimate is under the ceiling; actual fit is unverified, and a shorter-context overhead is reused; a floor requires a monotonic-overhead assumption
hypothetical residency; the reference server loads at most one model
reserve host/process memory separately; measure the actual backend and concurrency
a budget with no cache term in it has no answer about conversations
Six gigabytes and a quarter against a ceiling of eight, and 1,723,346,688 bytes of room to spare. Both files are on the disk, both numbers came off them, and the answer is wrong. Here is the same configuration with the term put back:
$ go run ./cmd/fits -mode room -ctx 32768
fits: estimated two-model residency, against a ceiling
the ceiling 8,000,000,000 bytes
the context 32768 tokens for both
conversations 1 of each in flight
a cache number 2 bytes
the cache term counted
model weights cache overhead all three
hero 4,683,073,952 1,879,048,192 163,753,568 6,725,875,712
small 397,807,936 402,653,184 1,032,017,856 1,832,478,976
hero qwen2.5:7b-instruct-q4_K_M
small qwen2.5:0.5b-instruct-q4_K_M
what the configuration costs 8,558,354,688
the ceiling 8,000,000,000
over by 558,354,688
the estimate exceeds the ceiling, and a shorter-context overhead is reused; a floor requires a monotonic-overhead assumption
hypothetical residency; the reference server loads at most one model
reserve host/process memory separately; measure the actual backend and concurrency
no estimated room remains; one extra hero cache alone would require 1,879,048,192
The difference between the two runs is 2,281,701,376 bytes and it is entirely the cache. This is a worked arithmetic failure, not a captured out-of-memory event. The omitted bytes may matter at load time: this chapter's weighing mode loads with no prompt and already observes context-dependent allocation. A backend can reserve KV storage for the configured context before live tokens occupy it. Token occupancy then grows inside that reservation, without necessarily increasing process memory per token. A dynamically growing cache can behave differently. Either way, budget the intended capacity instead of assuming a short first answer proves the longest request will fit.
And the memory is only half of what a long conversation costs. Here is the same instruction sent to the small model with more and more of the chronicle in front of it, with the same context, the same token limit and the same seed every time, and with not a word of any answer read or printed:
$ podman exec -w /bench world-go go run ./cmd/fits -mode long -role small -limit 16
fits: the same question with more and more conversation in front of it
the server world-lm:11434
the model qwen2.5:0.5b-instruct-q4_K_M, in the small role
pinned at c5396e06af294bd1
the context 4096 tokens
at most 16 tokens back, every time
temperature 0.00, seed 89
nothing any of these answers said is read here or printed here
lines prompt answer seconds tokens a sec against the first
1 115 16 0.609 215.12 1.00x
8 378 16 0.996 395.58 1.64x
32 1051 16 2.271 469.83 3.73x
64 2011 16 3.572 567.50 5.87x
measured here on the processor, with no graphics card in it; yours will differ.
What does not differ is which column grows and which one does not: the answer
is the same length every time and the prompt is what got dearer
The answer is sixteen tokens in all four rows and the wall clock runs from 0.609 seconds to 3.572, on an eight-core Ryzen 7 3700X, on the processor, with no graphics card in it. Those four are the machine's and yours will be four other numbers; the two columns beside them count the prompt and the answer. The fixed prompt and tokenizer determine the input count; sixteen is the requested output limit, not a guarantee that another generation cannot stop earlier. Every token of prompt has to be read before a single token of answer can be written, and reading it is the work chapter 82 counted by hand: each token dotted against every other. At two thousand tokens that cost is still rising about in step with the prompt, which is what chapter 82's own crossover arithmetic predicted for a model of this width; the term that grows with the square of the row is waiting further up. A budget with no cache term in it has no way of seeing any of this coming, because the number it left out is the only number in the sum that knows how long the conversation is.
Loaded once and timed twice
Everything so far has been done to the big model without loading it. Its header was dumped to a file five chapters ago and every figure since has come off those bytes, which is why a chapter about seven billion parameters has been running on a workstation. That was the cheap half of the bargain. Now the expensive half: put the thing in memory once, weigh it where it stands, and ask it one question.
The load and the question are two requests, and separating them is the only way the rate at the end of this run means anything. A first answer out of a model that is not resident spends most of its time reading four and a half gigabytes off a disk. Time that and divide by the tokens and you have measured a disk.
Which is also why the server is emptied first, and why that is a command on this page instead of an assumption behind it. The run above left the small model in memory, and this server holds one model at a time, so the load below would be a load that evicted something — and the mode refuses to start at all if the file it means to time is already resident. Killing the container and starting it again is the shortest way to a server holding nothing; the weights are in a volume and are not touched by it.
$ podman kill world-lm && podman start world-lm
world-lm
world-lm
$ podman exec -w /bench world-go go run ./cmd/fits -mode hero
fits: the model the budget is about, loaded once and asked one question
the server world-lm:11434, waiting up to 10m0s for an answer
the model qwen2.5:7b-instruct-q4_K_M
pinned at 2bada8a745067700
the context 4096 tokens
at most 96 tokens back
temperature 0.00, seed 89
the prompt 261 bytes, 3 chronicle lines and one instruction
the server is holding 0 model(s), and this is not one of them
what it costs to hold, at this context
the weights, which are the file 4,683,073,952
the cache, 57,344 a token 234,881,024
the residue, which is the overhead 163,753,568
the three of them 5,081,708,544
and what the server says it needs 5,081,708,544
of which, on a graphics card 0
what the exchange cost
tokens of prompt, its count 120
tokens of prompt, our own table 69
tokens of answer 27
tokens in the whole decision 147
it stopped because stop
and what the clock said, which belongs to this machine and not to the file
putting the weights in memory 3.107s
the whole exchange after that 7.465s
tokens a second over the decision 19.69
measured here on the processor, with no graphics card in it; yours will differ
what the model wrote, and yours will differ:
These lines describe the number of hobbs present and the total born and gone over three consecutive years in a place called The Hollow.
Take the memory block first, because it is the last brick in the calculator. The server says it needs 5,081,708,544 bytes to hold this model at 4,096 tokens. The file is 4,683,073,952 of them and the cache arithmetic accounts for 234,881,024 more, so the residue is 163,753,568, and that is the figure the whole ladder above was built on. It is the one number in this chapter that came off a socket, and the run says so on the line that prints it.
Then the token block, which is chapter 81 arriving with a bill. The prompt is 261 bytes of plain text and it costs 120 tokens by the model's own table. The merge table this volume trained on the chronicle alone makes the same 261 bytes 69 tokens, because that table learned this exact corpus and nothing else, and 69 is the sort of number it can only produce about text that looks like The Hollow's own history. Two tables, one unit, and the unit is what everything in this volume has been priced in.
Last, the clock, and every figure on it belongs to one machine. Putting the weights in memory took 3.107 seconds and the exchange after that took 7.465, for 147 tokens, which is 19.69 tokens a second. Measured on an eight-core Ryzen 7 3700X, on the processor, with no graphics card involved anywhere. GPU performance and allocation depend on the device and backend; this run establishes neither a speedup factor nor GPU memory use.
The sentence at the bottom is an exemplar. A fresh run can change its words, token count and timings, as well as the server's allocation report.
One decision took 7.465 seconds. That is an interesting number for about four seconds, and then the useful question arrives: what does a day of that buy? The arithmetic is a division and it takes the rate as a flag, because the rate is the reader's and not this book's:
$ go run ./cmd/fits -mode hour -in 120 -out 27 -rate 19.69
fits: what a measured rate buys, in decisions
a decision is 120 tokens of prompt and 27 of answer
which is 147 tokens
measured at 19.69 tokens a second
seconds a decision 7.47
decisions an hour 482.2
decisions a day 11572.9
a decision 2 times the size: 14.93 seconds, 241.1 an hour
a decision 4 times the size: 29.86 seconds, 120.6 an hour
a decision 8 times the size: 59.73 seconds, 60.3 an hour
the rate came off a machine and the arithmetic did not. It is also a rate
about one size of decision: a model reads a prompt faster than it writes an
answer, so the four rows above are a plan and not a measurement
Four hundred and eighty-two decisions an hour, eleven and a half thousand a day, on a processor, from a seven-billion-parameter model at four bits a weight and a context of 4,096. Every one of those qualifiers has to travel with the number. A rate taken from the small model would be a different number about a different file, and setting the two side by side without naming the model, the quantization, the context and the machine for both is the one mistake this volume has been most careful not to make.
The three rows underneath are a plan and they say so. Doubling the size of a decision halves the number of them per hour if the rate holds, and the rate does not quite hold, because a model reads a prompt faster than it writes an answer. The run prints that caveat itself instead of leaving it to a footnote, and the honest way to get the number for a decision of a different size is to measure a decision of that size.
Between two ticks
There is one job in this world that wants a model, and the last chapter left it running by hand: read the chronicle lines that have no vector yet, ask for a row apiece, write them back. Twenty-six lines of loop. The question that chapter did not answer is where it goes, and the answer has been fixed since the last volume: not inside a tick.
The rule arrived as a rule about databases. A tick that can open a transaction is a tick that can wait on a disk, so nothing in the simulation reaches the store. Put a model server on the same bridge and the same rule comes back an order of magnitude harder. A tick is a hundred milliseconds at the rate this world was built to run at. The average embedding cost on this machine is 51 milliseconds; the decision the big model answered above took seven and a half seconds. Waiting for either call delays the next tick, whether the call sits inside a tick or immediately before it.
So the job goes where the save goes: on the boundary between two ticks, where no
phase is part way through, no view is open and no creature is half stepped. That
protects state consistency, not response time. This teaching example calls
onePass synchronously; the world clock stops advancing until it returns.
The call changes wall-clock pace without changing the ordered tick arithmetic.
for w.tick() < ticks {
// The boundary. Everything in this loop that is not a tick
// happens here, because here is the one place where no phase is
// part way through and nothing is holding anything.
if w.tick()%terra.Year == 0 && w.tick() > 0 {
text := fmt.Sprintf("year %d: %d %s standing in %s, %d born and %d gone",
w.tick()/terra.Year, len(w.r.Live), doc.Many("living"), doc.One("ground"),
w.p.Made, w.r.Gone)
if err := db.Tell(ctx, w.tick(), "count", text); err != nil {
return err
}
}
if c.fill && w.tick()%c.every == 0 {
at := time.Now()
n, sent, err := onePass(ctx, db, cl, m, c.batch)
inJob += time.Since(at)
if err != nil {
return err
}
if sent > 0 {
passes++
}
asked += sent
wrote += n
}
w.v.Tick()
w.fold()
}
One pass avoids an unbounded drain of the whole history.
onePass takes up to a batch of waiting lines, embeds them, writes them and
returns. A job that ran until there was nothing left would have the whole history as its
worst case, and the history is the one thing about this world that only grows. Everything
it holds it holds inside that call: by the time it returns, every socket it wanted is
done with, and the next tick starts from where the last one ended. The batch bounds
the number of rows, not an interactive latency guarantee. Each network request still
has its own timeout; several slow requests can stall this serial loop for a long time.
The flag that turns it off is what makes the next two runs a measurement. Both found the same world from the same seed, tell the chronicle the same count once a year, and run for the same twenty valley-years. One of them talks to a model server and the other does not.
$ podman exec -w /bench world-go go run ./cmd/fits -mode pace -years 20
fits: the valley, ticking, with the eighth table being filled between ticks
the world seed 5, 16x12 cells, 21 browsers and 8 hunters standing
the run 20 valley-years, which is 72000 ticks
the chronicle one count a year, told between ticks
the job off
what the world did, which the flag does not touch
ticks run 72000
animals walking at the end 235
born 1645
gone 1439
plants standing 132
the digest of every tick a6b27c9079249f5c
what the chronicle holds
lines 30
rows in the eighth table 0
lines with no row from this model 30
what the job did between ticks
passes that had work 0
lines sent to the model 0
rows the copies reported 0
spent inside the job 0s
72000 ticks in 31.169s, 2310 ticks a second (measured here; yours will differ)
Thirty lines in the chronicle: eleven written at the founding and one a year for the nineteen year-boundaries the run crossed. None of them has a vector, because nothing has asked for one.
$ podman exec -w /bench world-go go run ./cmd/fits -mode pace -years 20 -fill
fits: the valley, ticking, with the eighth table being filled between ticks
the world seed 5, 16x12 cells, 21 browsers and 8 hunters standing
the run 20 valley-years, which is 72000 ticks
the chronicle one count a year, told between ticks
the job on, every 3600 ticks, 32 waiting lines a pass
the model nomic-embed-text:v1.5, at world-lm:11434
pinned at 970aa74c0a90ef74
what the world did, which the flag does not touch
ticks run 72000
animals walking at the end 235
born 1645
gone 1439
plants standing 132
the digest of every tick a6b27c9079249f5c
what the chronicle holds
lines 30
rows in the eighth table 30
lines with no row from this model 0
what the job did between ticks
passes that had work 20
lines sent to the model 30
rows the copies reported 30
spent inside the job 1.524s
for one line of chronicle 51ms
ticks this world runs in that 112
72000 ticks in 32.703s, 2202 ticks a second (measured here; yours will differ)
Hold the two runs against each other a block at a time. Two hundred and thirty-five
animals walking, 1,645 born, 1,439 gone, 132 plants standing, and the same sixteen
characters folded over all seventy-two thousand ticks:
a6b27c9079249f5c in both. Putting a language model beside this world changed
not one number the world produced. That is the design working, and it is checkable rather
than asserted, because the digest covers every plant, every creature, the ground, the
litter and four counters, once a tick, for the whole run.
What did change is the clock. 2,310 ticks a second became 2,202: the world's own pace is down by just under five per cent, and 1.524 seconds of the 32.7 went into the job. Thirty lines of chronicle at 51 milliseconds apiece, and the run puts that in the world's own units: one round trip to the embedding model is 112 ticks of this valley. Both figures were measured on an eight-core Ryzen 7 3700X with the database and the model server in containers beside the daemon, on the processor, and yours will differ. The counts and the digest agree under the same reference simulation environment.
The 112-tick figure uses the accelerated bench rate, about 2,202 ticks per second. It does not mean 112 scheduled 100 ms intervals: 51 ms is roughly half of one such interval. An average also hides the slowest call and the initial backlog. This serial result establishes a throughput cost, not that interactive tick deadlines hold.
An interactive implementation needs a bounded queue and bounded workers, request deadlines and cancellation, plus a stated overload policy. Workers consume immutable requests and return results; only the simulation owner admits state changes at a tick boundary. Merely adding a goroutine lets completion timing choose which tick receives an answer, breaking replay if that answer changes the world.
Name each request and its intended admission rule. Record accepted outcomes and their admission ticks, including rejection, timeout and fallback decisions, so replay reads the recorded inputs instead of calling a live model again. Deterministic request order alone does not make wall-clock completion deterministic. Those are prerequisites for the planned village and multiplayer work, not features this serial bench implements.
$ go test ./internal/lang/ -run 'TestOneTokenOfCacheIsBothArraysAndNotOne|TestAHeadWidthThatIsNotWholeIsRefused|TestTheCacheGrowsWithTheContextAndTheWeightsDoNot|TestConversationsInFlightBuyCachesAndNotWeights|TestTheSamePairFitsAtOneContextAndNotAtAnother|TestAFloorIsTheLeastAnOverheadCanBe|TestABudgetWithNoCacheTermAnswersTheSameAtEveryContext|TestNobodyHasWeighedTheEmbeddingModel|TestADecisionIsPricedInTokensAndPaidForInSeconds' -v
=== RUN TestOneTokenOfCacheIsBothArraysAndNotOne
room_test.go:65: hero: 28 layers x 4 heads x 128 wide x 2 bytes, doubled, is 57344 bytes a token
room_test.go:65: small: 24 layers x 2 heads x 64 wide x 2 bytes, doubled, is 12288 bytes a token
--- PASS: TestOneTokenOfCacheIsBothArraysAndNotOne (0.00s)
=== RUN TestAHeadWidthThatIsNotWholeIsRefused
room_test.go:80: lang: 3584 numbers a token over 27 heads is not a whole head width
--- PASS: TestAHeadWidthThatIsNotWholeIsRefused (0.00s)
=== RUN TestTheCacheGrowsWithTheContextAndTheWeightsDoNot
room_test.go:100: the weights are 4683073952 bytes at every context and the cache runs to 1879048192
--- PASS: TestTheCacheGrowsWithTheContextAndTheWeightsDoNot (0.00s)
=== RUN TestConversationsInFlightBuyCachesAndNotWeights
room_test.go:110: estimated one slot is 768172288 bytes and four are 919167232, holding the other terms fixed
--- PASS: TestConversationsInFlightBuyCachesAndNotWeights (0.00s)
=== RUN TestTheSamePairFitsAtOneContextAndNotAtAnother
room_test.go:136: the estimate is under at 4096 by 2150119168 bytes and over at 32768 by 558354688, with extrapolated overhead
--- PASS: TestTheSamePairFitsAtOneContextAndNotAtAnother (0.00s)
=== RUN TestAFloorIsTheLeastAnOverheadCanBe
room_test.go:158: lang: the hero model has been weighed at 4096 tokens and nothing shorter than 1024
room_test.go:170: 9 weighings of the small model, none of them smaller than the one below it
--- PASS: TestAFloorIsTheLeastAnOverheadCanBe (0.00s)
=== RUN TestABudgetWithNoCacheTermAnswersTheSameAtEveryContext
room_test.go:191: with the cache left out the weights are 397807936 bytes whatever the context is set to
--- PASS: TestABudgetWithNoCacheTermAnswersTheSameAtEveryContext (0.00s)
=== RUN TestNobodyHasWeighedTheEmbeddingModel
room_test.go:198: lang: nobody has weighed the embed model
room_test.go:210: small: 9 weighing(s), all of them against ollama 0.12.3
room_test.go:210: hero: 1 weighing(s), all of them against ollama 0.12.3
--- PASS: TestNobodyHasWeighedTheEmbeddingModel (0.00s)
=== RUN TestADecisionIsPricedInTokensAndPaidForInSeconds
room_test.go:237: a decision of 240 tokens at 20 a second is 12 seconds and 300 an hour
--- PASS: TestADecisionIsPricedInTokensAndPaidForInSeconds (0.00s)
PASS
ok theworld/internal/lang 0.023s
Three of those are about arithmetic and six are about what the calculator refuses. A header whose embedding width is not a whole number of heads is turned down instead of rounded, because a rounded head width comes back as a cache figure wrong by a head and nothing would ever say so. An overhead is refused for a context nothing was weighed at or below. And a budget with the cache term left out declines to answer the question about conversations at all, which is correct: the number it was asked for is a division by the term it was told to ignore.
Why this works
The useful idea here is older than models and applies to anything that has to be held in a finite machine. Split a cost into the part that is fixed when you choose the artefact, the part that grows with the work, and the part nobody can derive; compute the first two; measure the third and write down the conditions you measured it under. What comes out is not a benchmark, it is a function, and a function can be asked about a configuration nobody has run.
KV storage is absent from the weights file, but the selected context can make it visible in allocation counters at load time. Reserved slots and occupied slots answer different questions. The calculator prices reserved capacity; it does not watch live token occupancy.
Keeping the budgets and the rates in separate columns costs one habit and buys the only thing a reader on a different machine actually needs: a way to check. Every byte figure on this page came out of a committed file or a committed weighing, so a reader who runs the same modes gets the same bytes and can tell immediately if their file is not the file this book priced. Every second and every tokens-a-second on this page names a processor. Neither kind of number is more honest than the other; mixing them is what makes both useless.
You should be able to do all of this without opening a browser.
- Work out, from a header alone, what one token of conversation costs a given model: blocks times key-and-value heads times head width times bytes an element, doubled, and the head width is a division and not a field.
- Say which of the three sums moves when you change the context length, which moves when you add a second conversation, and which moves when you change neither.
- Explain why the small model on this page costs 320,032,704 bytes of overhead at 4,096 tokens and 1,032,017,856 at 32,768, and why writing either of those down without the context beside it makes it useless.
- Turn estimated room into a cache-slot upper bound, then name the measurements needed before treating that result as concurrent capacity.
- Point at any figure on this page and say whether it would be the same on a machine with a graphics card in it.
- Explain why a synchronous call between ticks still stalls the clock, and what bounded workers and recorded admission would need to change before interactive play.
Exercise 1 — price a model with the other head arrangement. Somebody offers you a seven-billion-parameter model with 28 key-and-value heads instead of 4, everything else the same. Work out its cache at 4,096 tokens and at 32,768, and say what happens to the ladder.
One token becomes 2 × 28 × 28 × 128 × 2 = 401,408 bytes, seven times the 57,344 on this page. At 4,096 tokens that is 1,644,167,168 bytes of cache for one conversation, against 234,881,024 here. This paper exercise holds file bytes fixed to isolate KV arithmetic; changing a real model's head arrangement can also change its projection weights and needs a new header and allocation measurement.
Reusing the old residual only as an assumption, the estimate stays below eight at 4,096: roughly 5,080,881,888 of weights, 483,786,272 of measured overhead and about 1,694,498,816 of cache for the two models comes to 7.26 gigabytes, inside eight with about three quarters of a gigabyte to spare. There is no estimated room for a second hero cache, where the original estimate allowed nine extra caches. Neither estimate measures concurrency. And at 32,768 tokens the big model's cache alone is 13,153,337,344 bytes, so the configuration is over the ceiling before the weights are counted.
That is the whole argument for the arrangement these files use, made in bytes: sharing one key-and-value head between seven query heads costs the model some of its ability to attend in different ways at once, and reduces the raw KV storage by a factor of seven.
Exercise 2 — find the last context with room for a second conversation.
Read the cap column of the ladder and find the largest context at which
the estimate leaves room for one extra cache of the big model. Predict whether
it is above or below 8,192 tokens before you look, and say what you would have to do to
answer for a context that is not on the ladder.
The column reads 9, 5, 3, 2, 1, 0, 0 down the seven rows, so the answer is 16,384 tokens. Below it the estimate leaves room for an extra hero cache; at 24,576 it leaves less than one cache's worth, without establishing physical fit.
Predicting it needs one observation: doubling the context makes each conversation twice as dear and eats the room that would have paid for it, so the count falls faster than the context rises. Nine at 4,096 becomes three at 8,192, roughly a quarter, so another doubling to 16,384 lands somewhere near one and a further one lands at none. The estimated ladder gives that cache-only limit.
A context that is not on the ladder can still be asked about, and the answer comes
back marked. -mode ladder -rungs 5120,7168 prices both rows on the
nearest available shorter-context residuals, says 7 and 4 extra caches, and puts a
star on each: the totals use extrapolated overheads. Weighing the two rungs with
-mode weigh -role small -rungs 5120,7168 against a running server
improves the small-model estimate. The hero residual and concurrent workload still
need measurement.
Exercise 3 — change how often the job looks. The between-ticks job cost this valley just under five per cent of its pace. Run the pace mode with the job every 360 ticks instead of every 3,600 and predict what happens to that percentage, and to the number of lines embedded, before you look.
Neither moves much, and the reason is the point. The cost of the job is the cost of the lines it embeds, and how often it looks does not change how many lines there are: thirty lines is thirty round trips whether they are collected in twenty passes or in two hundred. What ten times as many passes buys is ten times as many queries against an empty table, which are cheap, and a vector that exists sooner after its line is written.
The percentage is set by the ratio of two things, and neither is the cadence: seconds of embedding against seconds of ticking. This world writes one chronicle line a valley-year and runs a valley-year in about a second and a half, so it generates 51 milliseconds of work for every 1,500 milliseconds of ticking. Three per cent, in the steady state, plus whatever backlog the founding lines left.
Fewer lines reduce inference work. A separate machine can reduce local CPU contention, but this synchronous client would still wait on its replies. Bounded asynchronous workers can stop those waits from blocking ticks; they cannot make computation or network traffic free.
What the language machinery does not touch
This volume put a language beside The World and changed nothing inside it. There is a merge table trained on The Hollow's own history, one attention head worked out by hand on four tokens, a table of counts that writes fluent sentences about years that never happened, a model nobody here trained pinned by the hash of its own weights, and a sum that assigns every one of its 4,683,073,952 bytes to a tensor with nothing left over.
What it cannot do is a list that has not got shorter. Nobody in The Hollow says a word. There is no villager, no character, and nothing in the valley has a thought about anything: the hobbs still act on a row of twenty-four numbers through a wiring they were born with, exactly as they did before any of this arrived. No tick calls a model inside its simulation phase; the serial boundary job still delays subsequent ticks. Nothing that comes back from a search is fed to a model, nothing here is a memory, and the eighth table is thirty rows of numbers about thirty sentences a program wrote about a population count. The model on the other side of that socket does not know this world exists.
The sharpest limit is inside the result rather than beside it. Every honest thing on this page is honest because it was counted: the bytes came off files, the ticks came off a clock, the digest came off seventy-two thousand ticks of arithmetic that a second run reproduced exactly. The one thing on the page that none of that touches is the sentence the model wrote at the bottom of stage 7. Its bytes and token count can be checked, and its factual claims can be compared with the source rows. What these checks cannot promise is identical text from another live run. Keep that exemplar separate from the arithmetic fixtures and from any recorded decision the simulation must replay.