Chapter 84 — A Model You Did Not Build

Five gigabytes of model files

Three chapters of this volume have been ordinary Go over small numbers: a merge table, an attention head and a count model with a seeded picker. Their code and corpus are in the tree, so every number they print can be produced again on any machine.

Every file this world runs on is named by the hash of its own contents, fetched once while the network is still open, checked against that hash, and used only after the network has been shut behind it. The model weights on this page are different: five gigabytes across three files, trained elsewhere on data and hardware this book does not have. They cannot sit in the repository the way the pinned Postgres dependency does, so the pin has to be the digest.

The last volume solved the smaller version by vendoring a seven and a half megabyte Go dependency. That reasoning still holds. A thing fetched at run time is a thing the world cannot promise, so a thing too large to carry has to prove its bytes before the network closes.

By the end of this page there is a third container on the bridge, and it holds a model server; three model files in a volume of their own, each one checked against a digest this world committed before it ever ran; a bridge with no road off it, proved by the same run that proved it last time with one more line in it; internal/lang grown by a client, which is the only thing in this module that will ever open a socket to a model server; and one prompt sent twice, which comes back as two different answers and is the fact everything after this page is built on.

One thing this page does not do is train anything. Counting pairs to learn a merge table is counting. Counting which token follows which is counting. Nothing in this volume computes a gradient, and the weights that arrive here arrive finished.

The three model roles

There are three models and they are not three sizes of the same decision. Each one is here for a different job, and keeping the jobs apart is what stops this volume from quoting a number about one file as though it were a number about another.

The first is small: half a billion parameters, squeezed to four bits a weight. It loads in a moment and answers on a processor, and it is what this world generates from everywhere. The second is the one the budget is about: seven billion parameters at the same four bits, which comes to most of the memory an eight-gigabyte machine has to spend. It is pulled here and it is read as a file far more often than it is loaded, because reading a header costs nothing. The third answers with numbers instead of words, and how many numbers is the model's business and not this file's.

The kit goes in configs/, beside the founding document, and for the same reason: it is what an operator settles before anything runs.

▣ Build · stage 1 — the kit, and a digest beside every name

// configs/models.json, with the notes cut for length
{
  "server": {
    "host": "world-lm",
    "port": 11434,
    "wait": "10m",
    "context": 4096,
    "note": "..."
  },
  "models": {
    "small": {
      "name": "qwen2.5:0.5b-instruct-q4_K_M",
      "digest": "c5396e06af294bd101b30dce59131a76d2b773e76950acc870eda801d3ab0515",
      "note": "..."
    },
    "hero": {
      "name": "qwen2.5:7b-instruct-q4_K_M",
      "digest": "2bada8a7450677000f678be90653b85d364de7db25eb5ea54136ada5f3933730",
      "note": "..."
    },
    "embed": {
      "name": "nomic-embed-text:v1.5",
      "digest": "970aa74c0a90ef7482477cf803618e776e173c007bf957f635f1015bfcfef0e6",
      "note": "..."
    }
  }
}

// internal/lang/kit.go — the roles, and what a role is for
type Role string

const (
	// Small is the model this world generates from. It is the cheap
	// one, and cheap is the whole of its qualification.
	Small Role = "small"
	// Hero is the model the budget is about.
	Hero Role = "hero"
	// Embed is the model that answers with numbers instead of words.
	Embed Role = "embed"
)

// Roles is the three in the order this book prints them, which is
// smallest first because that is the order they cost.
var Roles = []Role{Small, Hero, Embed}

$ go run ./cmd/asks -mode kit
asks: the kit this world carries, read out of the file the build embeds

  the server
    peer                   world-lm:11434
    one request may take   10m0s
    context asked for      4096 tokens

  the three models, and the digest each is pinned at
    small  qwen2.5:0.5b-instruct-q4_K_M   c5396e06af294bd1
    hero   qwen2.5:7b-instruct-q4_K_M     2bada8a745067700
    embed  nomic-embed-text:v1.5          970aa74c0a90ef74

  no socket was opened to print any of that, and no prompt is in the file

That mode runs on a bare workstation with no containers anywhere, because the file is embedded in the binary and nothing in the mode opens anything. Two of its lines are doing work the rest of the chapter leans on. 10m0s is how long one request may take, and it is enormous on purpose: this server has no graphics card under it, and a deadline sized for the fast path would make the slow path look broken. 4096 tokens is how much context this world asks for, written down here instead of left to whatever the server would have chosen, because a setting nobody wrote down is a setting nobody can argue with later.

And no prompt is in the file. What to say to a model belongs to the program saying it and changes whenever the program does. Which model, at which digest, belongs to whoever operates this world and changes when they decide it does. Putting the two in one file makes every prompt edit look like an infrastructure change.

The fence around those three entries matters more than any of them, and it is this volume's version of the rule the last one lived by. A number measured against one model, at one quantization, at one context length, is not a number about any other one. Both halves of a bad comparison are usually true, which is what makes it easy to write: a tokens-a-second figure taken off the small model is a real measurement, and printing it anywhere near the seven-billion file's memory total says something false out of two true things. Any page setting two of these beside each other names all four of model, quantization, context and machine, for both.

⌥ Tool — ollama

ollama downloads model files, keeps them in a store on disk, loads one into memory on demand and answers JSON over HTTP. ollama pull <name> fetches one, ollama list shows what the store holds, and ollama --version is the health-check line. Its own command line appears on this page exactly where using it is the thing being taught, which is pulling the three files; everything after that goes through the client written below. It runs in podman here and never in docker.

Pulling before the bridge closes

Here is the awkward part, said out loud and not glossed over. A download inside a network with no route out is impossible. The last volume's contract could start on a machine with no network at all, because everything it needed was in the tree. This one cannot, and that is a real thing to have lost.

So the run comes in two halves and the order is the whole design. First, with the network still open, three images are pulled by digest and three models are pulled into a named volume, and every one of them is checked against a digest committed here. Only then is the bridge created, and from that moment nothing reaches out. The closure is proved after the prefetch and not before it, and no run may claim otherwise.

▣ Build · stage 2 — the prefetch, outside the closure

# Everything in this block happens with the network open. Nothing
# after it does.
podman pull docker.io/library/golang@sha256:277b40a9f20e4346f3b3386104f2a6c11caf2318a55c3a13ee89a920264bd717
podman pull docker.io/pgvector/pgvector@sha256:ccc6e83d6e35e931dc7c5def2022729d5a6c370318d099181995567ff1fb4d6b
podman pull docker.io/ollama/ollama@sha256:36ccf1b4c161179a198a19e897aa0a1acd54cd5eeb6ab99c80f2ef50ba7fa106

# The models land in a volume of their own, so they survive the
# container being removed and rebuilt. Pulling them needs a server
# running, and this is the only moment in the run when that server has
# anywhere to reach.
podman volume create world-models
podman run -d --name world-lm -v world-models:/root/.ollama \
	docker.io/ollama/ollama@sha256:36ccf1b4c161179a198a19e897aa0a1acd54cd5eeb6ab99c80f2ef50ba7fa106
for m in qwen2.5:0.5b-instruct-q4_K_M qwen2.5:7b-instruct-q4_K_M nomic-embed-text:v1.5; do
	podman exec world-lm ollama pull "$m"
done
podman rm -f world-lm

# From here on there is no road out. The bridge has no gateway and
# nothing has a published port.
podman network create --internal --subnet 10.89.77.0/24 world-net

# autovacuum is off in this world's database, and it is off for one
# reason: a query plan printed on a page five chapters from here must
# not depend on whether a background worker happened to gather
# statistics between two runs. Everything else about this server is
# the default.
podman run -d --name world-db --network world-net --ip 10.89.77.10 \
	--tmpfs /var/lib/postgresql/data:rw,size=512m \
	-e POSTGRES_USER=world -e POSTGRES_PASSWORD=world -e POSTGRES_DB=world \
	-e PGDATA=/var/lib/postgresql/data/pgdata \
	docker.io/pgvector/pgvector@sha256:ccc6e83d6e35e931dc7c5def2022729d5a6c370318d099181995567ff1fb4d6b \
	-c autovacuum=off

podman run -d --name world-lm --network world-net --ip 10.89.77.12 \
	-v world-models:/root/.ollama \
	-e OLLAMA_HOST=0.0.0.0:11434 -e OLLAMA_MAX_LOADED_MODELS=1 \
	docker.io/ollama/ollama@sha256:36ccf1b4c161179a198a19e897aa0a1acd54cd5eeb6ab99c80f2ef50ba7fa106

# The toolchain container gets the model volume read-only, so the
# reader's own code can look at the files the server downloaded.
podman run -d --name world-go --network world-net --ip 10.89.77.11 \
	-e WORLD_DB_HOST=world-db -e WORLD_DB_USER=world \
	-e WORLD_DB_PASSWORD=world -e WORLD_DB_NAME=world \
	-e GOFLAGS=-mod=vendor -e GOPROXY=off -e GOTOOLCHAIN=local -e LC_ALL=C \
	-v "$PWD/src":/src:ro,Z -v world-models:/ollama:ro \
	docker.io/library/golang@sha256:277b40a9f20e4346f3b3386104f2a6c11caf2318a55c3a13ee89a920264bd717 sleep infinity
podman exec world-go cp -r /src /bench

$ podman network inspect world-net --format '{{.Internal}} {{range .Subnets}}{{.Subnet}}{{end}}'
true 10.89.77.0/24

The named volume is what makes the two halves fit together. It belongs to no container: podman rm -f world-lm takes the server away and leaves five gigabytes of weights exactly where they were, so the server that comes back up on the closed bridge finds them already there and has no reason to ask anybody for anything. The database image also changed, because the plain one does not carry an extension this world is going to want out of it, and it is pinned by digest for the same reason the last one was.

The closure is a claim, so it has to be a run. The bench dials four addresses from inside the toolchain container: the two peers by the names podman's own resolver gives them, the Go module proxy by name, and an address on the public internet by number so that no name resolution happens at all.

$ podman exec -w /bench world-go go run ./cmd/asks -mode probe
asks: everything this container can and cannot open a socket to

  the database           world-db:5432          connected
  the model server       world-lm:11434         connected
  the module proxy       proxy.golang.org:443   dial tcp: lookup proxy.golang.org on 10.89.77.1:53: no such host
  an address, by number  1.1.1.1:443            dial tcp 1.1.1.1:443: connect: network is unreachable

  2 of 4 answered, and both of them are containers on this bridge

Four addresses, three different kinds of answer. The two peers resolve because podman runs a small DNS server on the bridge that knows the containers attached to it. The proxy fails at that same resolver, which forwards nothing, so the name does not exist as far as anything on this bridge is concerned. The last line skips names entirely and the kernel answers before a packet leaves: there is no route in the container's table that could carry it. Nothing timed out and nobody refused anything.

The summary line counts the answers instead of announcing a verdict. A bench that prints closed whatever happened is a bench that will one day print it about an open bridge, and the person reading the page will have no way to tell.

Figure 84.1 — the prefetch is the only part of this run with a road out, and the check at the end of it is what lets everything below the line stop trusting the network it no longer has. The volume is the hinge: it is filled in the upper band and read in the lower one, by a server that never has to ask for it again.

The content-addressed blob

Pinning is easy to assert and the assertion proves nothing. What makes it checkable here is a convenience of the server's: it stores every layer of every model as a file whose name is the sha256 of that file's contents. A model is a small JSON manifest listing its layers, and one of those layers is the weights.

So there are four separate claims about one file, and reading all four is the point, because any one of them alone looks fine.

▣ Build · stage 3 — four claims, and what each of them is worth

// internal/lang/kept.go
//
// Four of these fields are digests or sizes and they are four
// different claims. Pinned is what this world wrote down before it
// ever ran. Layer is what the manifest the server downloaded says the
// weights are. Blob is the name of the file on disk, which the server
// makes out of the digest. Sum is what the bytes in that file
// actually hash to. A pin is only worth anything when all four agree,
// and the point of reading all four is that any one of them alone
// would have looked fine.
type Kept struct {
	Name   string // the name a person types: qwen2.5:0.5b-instruct-q4_K_M
	Pinned string // the digest this world committed
	Layer  string // the digest the manifest gives the weights layer
	Blob   string // the path of the file holding them
	Bytes  int64  // its size on disk
	Says   int64  // the size the manifest claims
	Sum    string // sha256 of the bytes, once Verify has read them
}

// internal/lang/kept.go — one layer is the weights. Not none, and
// not two: a manifest with two would be a model this reader has no
// opinion about, and picking the first would be an opinion held
// quietly.
	if found != 1 {
		return Kept{}, fmt.Errorf("lang: the manifest for %s names %d weight layers, and one is the only number that means anything", name, found)
	}
	k.Blob = filepath.Join(root, "blobs", "sha256-"+k.Layer)
	fi, err := os.Stat(k.Blob)
	if err != nil {
		return Kept{}, fmt.Errorf("lang: the weights of %s: %w", name, err)
	}
	k.Bytes = fi.Size()
	return k, nil

$ podman exec -w /bench world-go go run ./cmd/asks -mode blobs
asks: what /ollama/models holds, checked against the names on disk, and not the bytes behind them

  model                               bytes  pinned at        on disk          agrees
  qwen2.5:0.5b-instruct-q4_K_M    397807936  c5396e06af294bd1 c5396e06af294bd1 yes
  qwen2.5:7b-instruct-q4_K_M     4683073952  2bada8a745067700 2bada8a745067700 yes
  nomic-embed-text:v1.5           274290656  970aa74c0a90ef74 970aa74c0a90ef74 yes

  5355172544 bytes of weights across 3 models
  the digests above were read, not computed: -sum reads the bytes

That is the cheap check and it costs three small reads and three calls to stat. It catches a model that was never downloaded, a manifest naming something the pin does not, and a truncated file, because a short download keeps its name and loses its length. What it cannot catch is a file whose bytes moved while its name stayed put, and the last line of the run says so, instead of letting the yes column imply more than it knows.

$ podman exec -w /bench world-go go run ./cmd/asks -mode blobs -sum
asks: what /ollama/models holds, checked against every byte of every blob, read and hashed

  model                               bytes  pinned at        on disk          agrees
  qwen2.5:0.5b-instruct-q4_K_M    397807936  c5396e06af294bd1 c5396e06af294bd1 yes
  qwen2.5:7b-instruct-q4_K_M     4683073952  2bada8a745067700 2bada8a745067700 yes
  nomic-embed-text:v1.5           274290656  970aa74c0a90ef74 970aa74c0a90ef74 yes

  5355172544 bytes of weights across 3 models
  every digest above was computed here, from the bytes on this disk

Same table, different meaning. Every digest in the third column was computed here, out of five and a third gigabytes read off this disk, and the fact that it matches the second column is the strongest thing this chapter can say about a file it did not make. The reader now has a model whose bytes they can account for, which is a smaller claim than understanding it and a much larger one than trusting it.

There is a second digest in play and confusing the two is easy, so the bench prints it separately and labels it. Ask the server what it holds, through the client written below, and the identifier it gives back is the hash of the manifest, not of any weights.

$ podman exec -w /bench world-go go run ./cmd/asks -mode holds
asks: what world-lm:11434 says it has, asked over the closed bridge

  model                                 bytes  the server's own id
  nomic-embed-text:v1.5             274302450  0a109f422b47e3a3
  qwen2.5:0.5b-instruct-q4_K_M      397821319  a8b0c51577010a27
  qwen2.5:7b-instruct-q4_K_M       4683087332  845dbda0ea48ed74

  3 models, and not one of them was fetched by this request
  the id above names a manifest and is not the digest of any model's weights

Every byte count there is larger than the matching one in the table above, and by a different amount each time, because the server is adding up all of a model's layers and the earlier table counted only the weights. Neither number is wrong. They are answers to two different questions, and a page that printed one of them under the other's heading would have said something false without writing anything untrue. The listing is sorted by name here because the server hands it back in the order it last wrote each file, which is a clock, and this book does not order anything by a clock.

▣ Build · stage 4 — the pin, as five things that can fail

$ go test ./internal/lang/ -run 'APinnedModelAgreesWithItselfFourWays|ABlobThatDoesNotHashToItsNameIsCaught|AManifestWithTwoWeightLayersIsRefused|ASizeTheManifestGotWrongIsCaughtWithoutReadingTheFile|TheKitPinsThreeModelsAndHoldsNoPrompt' -v
=== RUN   TestAPinnedModelAgreesWithItselfFourWays
    kept_test.go:86: 49 bytes, and the four claims about them are the same claim
--- PASS: TestAPinnedModelAgreesWithItselfFourWays (0.00s)
=== RUN   TestABlobThatDoesNotHashToItsNameIsCaught
    kept_test.go:121: one bit changed, the size unmoved, and only reading the bytes found it
--- PASS: TestABlobThatDoesNotHashToItsNameIsCaught (0.00s)
=== RUN   TestAManifestWithTwoWeightLayersIsRefused
    kept_test.go:139: two sets of weights and a name with no tag are both refused
--- PASS: TestAManifestWithTwoWeightLayersIsRefused (0.00s)
=== RUN   TestASizeTheManifestGotWrongIsCaughtWithoutReadingTheFile
    kept_test.go:166: a short file is caught by a stat, and a wrong one needs the whole read
--- PASS: TestASizeTheManifestGotWrongIsCaughtWithoutReadingTheFile (0.00s)
=== RUN   TestTheKitPinsThreeModelsAndHoldsNoPrompt
    kept_test.go:201: 3 models, 3 roles, 3 distinct digests, and one server at world-lm:11434
--- PASS: TestTheKitPinsThreeModelsAndHoldsNoPrompt (0.00s)
PASS
ok  	theworld/internal/lang	0.005s

Forty-nine bytes stand in for four gigabytes in four of those, and nothing in the code under test can tell the difference, which is the whole reason a pinning rule can be tested without a model. The one that matters most flips a single bit in the middle of a file and leaves everything else alone: the name still agrees with the manifest, the size is still right, the cheap check still passes, and only reading the bytes finds it. That is the failure the expensive mode exists for, and it takes three lines to cause on purpose and a very bad afternoon to notice by accident.

The Ollama client deadline

The server ships with a command line that will happily generate text, and this world does not use it. Two reasons, and the second one is the load-bearing one. A program that shells out to a tool gets whatever that tool decided to print, in whatever order it decided to print it, and has to parse a person's output back into numbers. And internal/lang is the only place in this module allowed to reach a model server at all, which is a rule the package has carried since it was created and which a test enforces by reading the import lines of every file in the simulation packages and in the store. A world that talks to a model by running a subprocess has moved that rule somewhere no test can see it.

So the world writes its own, and it is net/http and encoding/json and nothing else. No dependency goes into go.mod for this, and vendor/ is untouched: the protocol is JSON over HTTP and the standard library already speaks both.

▣ Build · stage 5 — a request that carries every setting it was given

// internal/lang/serve.go
type Ask struct {
	Model  string
	Prompt string

	// Limit is the most tokens the server may produce. It is not
	// advice. Canon caps what this book spends on inference, and an
	// unbounded generation is a bill nobody wrote down.
	Limit int

	// Context is how much the server is asked to hold. Zero means
	// the server's own default, which is the one setting this world
	// declines to leave to a default.
	Context int

	// Temp is the temperature chapter 83 built by hand: the number
	// every score is divided by before the shares are worked out.
	Temp float64

	// Seed is the server's own generator, and a negative value asks
	// it to draw its own instead.
	//
	// Negative, rather than zero, and the difference is measured
	// rather than assumed: a seed of nought is a seed, and a server
	// handed one produces the same answer every time it is asked. A
	// field left out of a request is not a field set to zero either,
	// which is the second half of the same trap, and the reason this
	// one is always sent.
	Seed int
}

// internal/lang/serve.go — what comes back, with the fields this
// book argues about and no others.
//
// There is no timestamp here and no duration. The server reports
// both and this world does not read them, because a struct with a
// wall clock in it is a struct that ends up printed.
type Said struct {
	Model string
	Text  string

	// In and Out are tokens: what the prompt came to once the server
	// had encoded it, and how many the server produced. They are the
	// unit chapter 81 settled, counted by somebody else's table.
	In, Out int

	// Why is the server's own word for why it stopped: because the
	// model produced an end token, or because Limit was reached.
	Why string
}

$ go test ./internal/lang/ -run 'AGenerationWithNoTokenLimitIsRefused|TheClientSendsEverySettingItWasGiven|ASeedOfNoughtIsASeed|AClientWithADeadlineGivesUp|AnUnfinishedAnswerIsAnError|AServerThatIsNotOneIsNamedInTheError' -v
=== RUN   TestAGenerationWithNoTokenLimitIsRefused
    serve_test.go:77: both refused, and no socket was opened to refuse them
--- PASS: TestAGenerationWithNoTokenLimitIsRefused (0.00s)
=== RUN   TestTheClientSendsEverySettingItWasGiven
    serve_test.go:123: seven settings went in and the same seven went out on the wire
--- PASS: TestTheClientSendsEverySettingItWasGiven (0.00s)
=== RUN   TestASeedOfNoughtIsASeed
    serve_test.go:150: nought, minus one and eighty-four all arrive as themselves, and the field is always sent
--- PASS: TestASeedOfNoughtIsASeed (0.00s)
=== RUN   TestAClientWithADeadlineGivesUp
    serve_test.go:174: the deadline is the whole request and not the connect, which is why the slow answer needed a long one
--- PASS: TestAClientWithADeadlineGivesUp (0.35s)
=== RUN   TestAnUnfinishedAnswerIsAnError
    serve_test.go:187: an answer the server had not finished is refused rather than trimmed
--- PASS: TestAnUnfinishedAnswerIsAnError (0.00s)
=== RUN   TestAServerThatIsNotOneIsNamedInTheError
    serve_test.go:211: the status, the path and what the peer said are all in the one line
--- PASS: TestAServerThatIsNotOneIsNamedInTheError (0.00s)

Six tests of a model client and not one of them has a model behind it. They run against forty lines of HTTP handler that records what it was sent and answers with whatever the test told it to, which is the same move the readiness wait used against a database: the thing being checked is arithmetic over a struct, and it needs four gigabytes of weights exactly as much as that one needed Postgres.

The third of them is the one to read twice. Nought is a number. A caller who meant draw your own and left the seed field at its zero value has silently pinned every generation the program will ever make, and everything will look fine until somebody wonders why the answers never change. This client takes a negative value for draw your own and always sends the field, so there are no defaults left anywhere for either side to have an opinion about.

⚠ Worked failure — the peer that answers the socket and nothing else

A container that has been stopped refuses connections in a microsecond, and every layer of the program finds out at once. The interesting failure is the other one: a peer whose listening socket is still there and whose process is not running. The kernel completes the handshake, the request goes out, and nothing comes back. Ever. podman pause produces exactly that, and it is one command.

$ podman pause world-lm
world-lm
$ timeout 25 podman exec -w /bench world-go go run ./cmd/asks -mode hang -set-wait -wait 0s; echo "exit=$?"
asks: one request to world-lm:11434, 676 bytes of prompt, no deadline at all on the client
asks: nothing below this line until an answer arrives or the client
      gives up, and one of those is not guaranteed to happen
exit=124

Nothing after the banner, and the only reason that run ever ended is the twenty-five second guard wrapped around it from outside. Exit code 124 is timeout saying it killed the thing rather than the thing finishing. A daemon meant to run for weeks unattended, with one goroutine sitting in that call, has one fewer worker and no log line anywhere saying so, and the day somebody investigates it will look like a slow model.

Now the same request, same paused server, same second, with the client's own deadline put back.

$ podman exec -w /bench world-go go run ./cmd/asks -mode hang -set-wait -wait 10s
asks: one request to world-lm:11434, 676 bytes of prompt, 10s on the client
asks: nothing below this line until an answer arrives or the client
      gives up, and one of those is not guaranteed to happen
asks: lang: Post "http://world-lm:11434/api/generate": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
exit status 1

Read the error backwards. While awaiting headers says the connection was made and the request was sent, so this is not a name that failed to resolve and not a port with nothing behind it. Client.Timeout exceeded says the client deadline fired rather than the peer's. And the URL names the peer, which is the one thing somebody holding a log line at three in the morning actually needs.

The cure is a number somebody had to choose, and choosing it is where this gets interesting. Ten seconds is what the run above used and it is far too short for this world: on a processor, a cold model load and a first answer take longer than that on purpose, so a ten-second deadline would turn every cold start into an error. The kit says ten minutes. That is not caution, it is the slow path measured and then written down, and it is why the deadline lives in a file an operator edits instead of in a constant somebody guessed. Put the server back with podman unpause world-lm.

The repeated prompt run

Everything up to here has been about making the input to this thing knowable down to the bit. The bytes are pinned, the network is shut, the settings are in a file, and the client sends every one of them. So here is the first generation, with all of that printed beside it.

The prompt is the operator's and it is addressed to the model as a piece of equipment: three entries out of the committed chronicle, and an instruction to say in one sentence what they have in common. Nobody in The Hollow is speaking, nobody is being asked anything, and nothing that comes back is anybody's voice.

$ podman exec -w /bench world-go go run ./cmd/asks -mode ask
asks: one prompt, sent by the client this world wrote

  server                 world-lm:11434
  one request may take   10m0s
  model                  qwen2.5:0.5b-instruct-q4_K_M
  pinned at              c5396e06af294bd1
  context asked for      4096 tokens
  at most                64 tokens back
  temperature            0.80
  seed                   none: the server draws its own
  prompt                 676 bytes, chronicle entries 3,4,5

  the three entries the prompt carries, out of the committed chronicle:
    entry 3, tick 0, genesis, 161 bytes
    entry 4, tick 0, genesis, 210 bytes
    entry 5, tick 0, genesis, 178 bytes

  189 tokens in, 31 tokens out, stopped because: stop

  what the model wrote, and yours will differ:
    | All three entries describe a situation involving multiple entities in one place (a valley) that are not named, each with their own unique characteristics or actions.

Six hundred and seventy-six bytes of prompt came to a hundred and eighty-nine tokens, which is the unit this volume opened by settling, counted this time by a table nobody here built. Thirty-one tokens came back and the server stopped because the model produced an end token rather than because the limit of sixty-four was reached. Every one of those numbers replays. The sentence at the bottom does not, and it carries the label that says so: it was measured on an eight-core Ryzen 7 3700X, on the processor, with no graphics card involved anywhere, and it will not be your sentence.

Now send it again, and change nothing at all.

$ podman exec -w /bench world-go go run ./cmd/asks -mode twice
asks: the same prompt, twice, with nothing changed in between

  server                 world-lm:11434
  one request may take   10m0s
  model                  qwen2.5:0.5b-instruct-q4_K_M
  pinned at              c5396e06af294bd1
  context asked for      4096 tokens
  at most                64 tokens back
  temperature            0.80
  seed                   none: the server draws its own
  prompt                 676 bytes, chronicle entries 3,4,5
  prompt sha256          62c9d3cb9c3787bd

  both requests carried the same 676 bytes and the same 6 settings

  answer 1: 189 tokens in, 33 tokens out, stopped because: stop, sha256 52337e6c3e26a73e
  answer 2: 189 tokens in, 28 tokens out, stopped because: stop, sha256 ebb73ab8fa75f475

  the two answers are not the same bytes: 159 and 153 of them

  answer 1, and yours will differ:
    | All three entries describe a narrative or account of an unnamed subject, indicating that they are all about the same entity: a non-human creature named "hobb."

  answer 2, and yours will differ:
    | All three entries describe an archive of animal names in a mysterious system, but none are named themselves or provided explicit names for their members.

One prompt, one hash over it, two requests carrying identical bytes and identical settings, and two different answers. The token counts differ. The lengths differ. The digests differ, which is the part that matters, because two answers that disagree in one word look the same at a glance and a digest never does.

Nothing has gone wrong. The chapter before this one built the mechanism by hand: a model does not answer a question, it produces a distribution over what comes next, and something else draws from it. That chapter's sampler drew off a seeded generator, so its runs reproduced. This server was asked to draw its own, so they do not. The mechanism arriving here is the one the reader already wrote, running on somebody else's weights.

Which raises the obvious question, and the answer is a run and not an opinion. Pin the seed and send it twice more.

$ podman exec -w /bench world-go go run ./cmd/asks -mode twice -seed 84 -quiet
asks: the same prompt, twice, with nothing changed in between

  server                 world-lm:11434
  one request may take   10m0s
  model                  qwen2.5:0.5b-instruct-q4_K_M
  pinned at              c5396e06af294bd1
  context asked for      4096 tokens
  at most                64 tokens back
  temperature            0.80
  seed                   84
  prompt                 676 bytes, chronicle entries 3,4,5
  prompt sha256          62c9d3cb9c3787bd

  both requests carried the same 676 bytes and the same 6 settings

  answer 1: 189 tokens in, 30 tokens out, stopped because: stop, sha256 18156b3dcdeb782f
  answer 2: 189 tokens in, 30 tokens out, stopped because: stop, sha256 18156b3dcdeb782f

  the two answers are the same bytes

  neither answer is printed here: what is being compared is whether
  they are the same bytes, and two digests say that better than two
  paragraphs do

Same digest twice, so the seed does what a seed does. That is a much weaker guarantee than it looks, and the weakness is the reason no page in this volume quotes a model's words as though they were a fact about anything. It holds for this server, running this build, with this file loaded, on this processor. Move the same digest onto a machine with a graphics card, or let the server batch the request differently because something else asked it something at the same moment, and the arithmetic underneath is done in a different order over floating-point numbers, where order changes answers. The seed pins the draw. It does not pin the numbers being drawn from.

Why this works

The move on this page generalises past models and containers, so it needs a portable form. When a system contains something you cannot reproduce, the useful question is where exactly the boundary around it runs, and how much of the system is left outside it. Making the thing itself reproducible is usually not on offer.

Here the boundary is drawn tightly and everything outside it is checked. The bytes of the file are pinned and hashed. The network is closed and the closure is demonstrated by walking to the edge. The settings are in a file, sent explicitly, and tested for having been sent. The prompt is hashed. The token counts, the byte counts and the digests all replay. What is left inside the boundary is one thing: the words the model chose. That is genuinely irreducible, and everything this volume does afterwards is arranged so that the irreducible part never gets to move a number the book quotes.

The alternative, and it is what most projects do, is to leave the boundary undrawn. Pull a model by tag, let the settings default, read the answer, and print a measurement. Every part of that is reproducible-looking and none of it is reproducible, and the failure mode is not an error at all: it is a number that was true in March.

✓ Checkpoint — the model, pinned and served

Name the four separate claims Kept holds about one weights file, and say which one a truncated download breaks and which one a single flipped bit breaks.
Explain why the prefetch has to happen before podman network create --internal, and say what a machine with no network can no longer do that it could in the last volume.
Say what the server's own id for a model is the hash of, and why it is a larger number of bytes than the weights digest's file.
Give the reason a seed of nought is a trap in this client's request, and what this client sends instead when the caller wants the server to draw its own.
Describe the failure a stopped container cannot show you and a paused one can, and name the one field that is the whole defence against it.
State what changes and what does not when the same prompt goes to the same pinned model twice.

⚡ Exercises — try first, then reveal

Exercise 1 — break a pin on purpose. Point the bench at a model store where one blob has been renamed, and predict which of the two checks catches it and what it says.

The volume is mounted read-only into the toolchain container, so the tampering has to happen where the store is writable. Copy the store somewhere else first, rename one blob, and point -root at the copy. The cheap check catches it and never reads a byte: Weights builds the blob's path out of the digest in the manifest, so a renamed file is a path that does not exist, and the error names the model and the missing file. Renaming it back and instead flipping a byte inside it gets the other answer, and only under -sum: the path resolves, the size is right, the cheap check says yes, and the hash disagrees.

The general lesson is in which check found which fault. A path built out of a digest gives you integrity of the naming for free, every time anything opens the file. Integrity of the contents costs a full read and is therefore something you do deliberately, at a moment you chose. Both guarantees matter, and they are not the same guarantee.

Exercise 2 — find the temperature where the answers stop differing. Run the same prompt twice at falling temperatures and watch the two digests converge.

-mode twice -temp 0.1 -quiet and then -temp 0, with -quiet so that the comparison is two hashes and no text. At a temperature near zero the distribution collapses onto whichever token was already ahead, so both runs take the same path and the digests agree. Chapter 83 built exactly this by hand and showed what it costs: the run stops being able to leave a loop it has entered, because nothing is left that could pick the second-place token.

The thing to notice is that the two digests agreeing at temperature zero is not the same guarantee as the two digests agreeing under a fixed seed, even though the output looks identical. One is a draw made the same way twice. The other is no draw at all. And neither survives being moved to a machine that does the arithmetic in a different order.

Exercise 3 — prove the negative control. Give the toolchain container a second bridge that was created without --internal, and run the probe again.

A closure claimed and never tested against an open bridge is a claim about the bench and not about the network. Create a second network with no --internal on it, attach the container that is already running to that one as well with podman network connect, and run -mode probe. Attaching rather than starting a fresh container is what makes this a measurement of one thing: the two peers are still on the closed bridge and still resolve, so the only lines that can move are the two that failed. Both of them connect — the name resolves, because a bridge that is not internal forwards the queries its own resolver cannot answer, and the numeric address has a gateway to hand its packets to — and the summary line reads everything answered: this bridge has a road off it instead of counting two out of four.

That last line is why the bench works out its summary from the answers rather than printing a verdict it was born with. A run that reports closure on an open bridge is worse than no run, because it is a green light attached to nothing, and it is the same failure as a walk that finds no files and passes cheerfully. Disconnect the container and remove the extra network afterwards; a leftover bridge with a road out is the next probe's quiet lie.