A Second Container
The store boundary
The long run at the end of the last chapter wrote an archive of two centuries: every birth and every death in them, one whole genome to a line, in a file opened for appending and never for anything else. The valley that produced it is the best thing The Hollow has been so far. The file it produced is the worst.
Nothing in internal/sim, internal/terra,
internal/beast, internal/gene or internal/mind
imports internal/store, and nothing inside a tick calls into it. The arrows
run one way. A tick that can open a transaction is a tick that can block on a socket, and a
tick that can block on a socket has no budget at all.
Ask the file what the mean generation time was and there is exactly one way to find out: open it at byte zero and read to the end, parsing every line and holding a running total. Ask it the same question again and you do the same work again.
Ask which lineage a particular creature came from and you read the whole thing looking for one record, then read it again for that record's parent, and again for its parent, until you reach a founder. Six questions about ancestry is six full passes over a file that was sixteen megabytes after two hundred years and will be a great deal more after two thousand.
The other half of the problem is worse and quieter. That file is a log of what happened. It is not the world. Kill the process that is writing it and the valley is gone: the bodies, the stores, the roster, the seed bank, the generators halfway through their streams.
The archive can tell you what the world did and it cannot put the world back. A world that is supposed to run for weeks unattended has to survive the machine under it being rebooted, and no amount of appending to a file gets you that.
So this world needs a database: not as a place to put logs, but as the thing that holds what the world is and what it did, in a form that can be asked questions and read back into a running process. Getting one means depending on somebody else's code, because Postgres speaks a binary protocol over a socket and nothing in the standard library speaks it.
It also means a second container. Everything so far has run in one, with the network switched off, and that switch was the proof that the module reached nothing.
That is the same move volume six made when it put every random draw in one package and gave each stream a number. The reason it worked there is the reason it works here: the interesting property of a system is usually a property of what a piece of it is allowed to reach, and the cheapest way to keep such a property is to make reaching impossible instead of making it rare. Every valley bench in this book ends on a line saying how many ticks a second the machine managed. That line means something because a tick is arithmetic over slices with a cost you can predict. Let one of them wait on somebody else's disk and the number stops being about the world.
By the end of this page there is one new dependency, pinned to a version and copied into
the tree so the build never fetches it; one new package, internal/store, which
owns every connection this world makes and is the only place in it allowed to say SQL; two containers on a bridge
with no route off it, proved by a run and not asserted; a readiness wait that is a
bounded loop and not a sleep; and cmd/worldd, opened again, and the first
binary in this book meant to be left running. There is no schema. Nothing on this page
creates a table, and the two questions it asks the server are what version it is and what
it holds, to which the honest answer is nothing.
The pgx dependency
Postgres does not answer HTTP and it does not take SQL over a pipe. A client opens a TCP socket, sends a startup message, negotiates authentication, and from then on exchanges length-prefixed binary messages with a one-byte tag on the front of each: parse, bind, describe, execute, sync, and about forty others coming back. Every value in a row arrives either as text or in the server's own binary encoding, and which one depends on what the client asked for. Writing that is a book. It is not this book.
So the world takes github.com/jackc/pgx/v5, which speaks that protocol and
brings a connection pool with it, and one line goes into go.mod. Two things
have to be true about that line for the contract to survive it. The version has to be
exact, because "whatever was latest on the day you built it" is not a thing anybody can
reproduce. And the code has to be in the tree, because a build that downloads is a build
that can fail differently on Tuesday.
# in the module root
go get github.com/jackc/pgx/v5@v5.10.0
go mod tidy
go mod vendor// go.mod, after all three
module theworld
go 1.26
require github.com/jackc/pgx/v5 v5.10.0
require (
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
golang.org/x/sync v0.17.0 // indirect
golang.org/x/text v0.29.0 // indirect
)$ podman exec -w /bench world-go grep '^# ' vendor/modules.txt
# github.com/jackc/pgpassfile v1.0.0
# github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761
# github.com/jackc/pgx/v5 v5.10.0
# github.com/jackc/puddle/v2 v2.2.2
# golang.org/x/sync v0.17.0
# golang.org/x/text v0.29.0
One require line was typed and six modules landed on disk. Two of them are
small files that read Postgres's own client configuration; one is a connection pool
that pgx uses and does not hide; two are pieces of the extended standard library, a
semaphore and the Unicode tables that normalise a password before it is hashed. Twenty
nine packages, two hundred and forty nine Go files, seven and a half megabytes of
them. The module's own Go code, everything seven volumes of this book have written,
comes to one and seven tenths of a megabyte counted the same way. The dependency is
four and a half times the size of the world.
go mod vendor is what puts them in the tree. It reads the module graph,
works out which packages are actually reached by something in the module, and copies
exactly those into vendor/, with vendor/modules.txt recording
which module each one came from and at what version. From then on the toolchain builds
against that directory instead of the module cache, and the effect is simple to
state: go build stops being an operation that can involve the network.
go.sum is a different thing and it is worth not confusing them.
go mod tidy left twenty six lines in it covering twelve modules, which is
six more than are vendored. The extra six are things pgx's own tests need and nothing
in this build reaches. go.sum records a hash for every module version the
graph mentions, so that if you ever do download one you can tell whether you got the
bytes somebody else got. It is a statement about the module graph.
vendor/ is a statement about the build. The first tells you that what you
fetched was right; the second means you never fetch.
The container the module builds in sets three environment variables of its own, which turn all of that from an intention into a rule. They show up below beside the two the image sets for itself.
$ podman exec world-go sh -c 'env | grep "^GO" | sort'
GOFLAGS=-mod=vendor
GOLANG_VERSION=1.26.7
GOPATH=/go
GOPROXY=off
GOTOOLCHAIN=local
-mod=vendor says build from the directory. GOPROXY=off says
there is no module proxy, so any attempt to fetch is an error and not a wait.
GOTOOLCHAIN=local says use the Go in this image and never go looking for
another one, which the module's go 1.26 line could otherwise trigger. The
three together mean a build either succeeds out of what is on disk or fails loudly, and
there is no third outcome where it quietly works because the machine happened to have
something cached.
$ podman exec -w /bench world-go sh -c 'go build ./... && echo "the build says nothing, which is what a build that reaches nothing says"'
the build says nothing, which is what a build that reaches nothing says
Silence is the claim, so here is the claim tested. Turn vendor mode off in the same container, with the same files, and ask for the same binary.
$ podman exec -e GOFLAGS=-mod=mod -w /bench world-go go build ./cmd/worldd
go: downloading github.com/jackc/pgx/v5 v5.10.0
internal/store/kin.go:10:2: module lookup disabled by GOPROXY=off
internal/store/kin.go:11:2: module lookup disabled by GOPROXY=off
internal/store/store.go:28:2: module lookup disabled by GOPROXY=off
There it is, and it is the whole argument in four lines. Without
-mod=vendor the toolchain goes looking for pgx in the module cache, does not
find it, tries to download it, and is stopped by GOPROXY=off. The build
genuinely wants the network; vendoring is what makes it stop wanting. That matters more
here than in most projects, because the thing this book has been proving with
--network none for five volumes is exactly this property, and the next
section takes --network none away.
// internal/store/store.go — the package comment is the rule
//
// Package store is the only place in this world that speaks SQL.
//
// The rule this package exists to make checkable is one sentence long:
// nothing in internal/sim, internal/terra, internal/beast, internal/gene
// or internal/mind imports internal/store, and nothing inside a tick
// calls into it. The arrows run one way. This package is allowed to
// know everything about the simulation; the simulation is not allowed
// to know that a database exists.
package store
// Config is everything worldd needs in order to reach a database, and
// nothing at all about what it will do when it gets there.
type Config struct {
Host string // the name of the peer, resolved on the container network
Port int
User string
Pass string
Name string // the database, not the world
MaxConns int32
MinConns int32
Life time.Duration // how long any one connection may live
Jitter time.Duration // spread on Life, so a pool does not retire all at once
Idle time.Duration // how long an unused connection may sit before it is closed
Check time.Duration // how often the pool looks over its own connections
// The wait. Budget is the whole of it and Step is the gap between
// attempts; there is no attempt count anywhere, because a count is
// a guess about how slow a machine is and a budget is not.
Budget time.Duration
Step time.Duration
}// internal/store/store.go — one connection string, printed two ways
// URL is the connection string, password and all. It is handed to the
// driver and to nothing else.
func (c Config) URL() string { return c.url(c.Pass) }
// Safe is URL with the password taken out. Everything that prints a
// connection string prints this one, including the errors, because a
// log line is a thing that gets pasted into a bug report.
func (c Config) Safe() string { return c.url("xxxxx") }
func (c Config) url(pass string) string {
u := url.URL{
Scheme: "postgres",
User: url.UserPassword(c.User, pass),
Host: c.Peer(),
Path: "/" + c.Name,
}
// sslmode=disable, and it is a statement about this network rather
// than a shortcut: the socket runs between two containers on a
// bridge with no route off it and no published port, so there is
// no wire for anybody to listen on. A database reachable from
// anywhere else gets verify-full and a certificate to check.
u.RawQuery = url.Values{"sslmode": {"disable"}}.Encode()
return u.String()
}$ go run ./cmd/reach -mode dsn
reach: the database this world would look for, and what it would do with it
peer world-db:5432
connection string postgres://world:xxxxx@world-db:5432/world?sslmode=disable
the pool
connections, at most 4
connections, at least 1
a connection lives 30m0s, give or take 5m0s
an idle one lives 5m0s
the pool looks over itself every 30s
the wait
budget 30s
step 250ms
attempts 120 at most, and no number anywhere set it
no socket was opened to print any of that
That is the one mode of the chapter's bench that runs anywhere, including on a bare
workstation with no containers at all, because it opens nothing. It exists to make the
configuration a thing you can look at instead of a thing you deduce from four
constructor calls. Every number in it is argued for further down this page, and the
password is the one value that never appears: URL goes to the driver,
Safe goes everywhere else, and the two are the same function called with
different arguments so there is no way for one to drift from the other.
The architectural rule at the top of that package is a comment, and a comment is a promise no test checks. Volume six's lesson about that was to make the invariant a run. The same trick works on an import graph, and it is nine lines of the standard library: read every Go file in the five sealed packages, parse the import block only, and look for one path.
// internal/store/rule_test.go
var sealed = []string{"sim", "terra", "beast", "gene", "mind"}
func TestNoSimulationPackageImportsTheStore(t *testing.T) {
const forbidden = "theworld/internal/store"
fset := token.NewFileSet()
checked := 0
for _, pkg := range sealed {
dir := filepath.Join("..", pkg)
ents, err := os.ReadDir(dir)
if err != nil {
t.Fatalf("reading %s: %v", dir, err)
}
for _, e := range ents {
if e.IsDir() || !strings.HasSuffix(e.Name(), ".go") {
continue
}
path := filepath.Join(dir, e.Name())
f, err := parser.ParseFile(fset, path, nil, parser.ImportsOnly)
if err != nil {
t.Fatalf("parsing %s: %v", path, err)
}
checked++
for _, imp := range f.Imports {
p, err := strconv.Unquote(imp.Path.Value)
if err != nil {
t.Fatalf("%s: unreadable import %s", path, imp.Path.Value)
}
if p == forbidden {
t.Errorf("%s imports %s: a tick that can open a transaction has no budget", path, forbidden)
}
}
}
}
if checked < 40 {
t.Fatalf("only %d files read across %v: the walk found less than the tree holds", checked, sealed)
}
t.Log("every file in every sealed package read, and not one of them names it")
}$ go test ./internal/store/ -run TestNoSimulationPackageImportsTheStore -v
=== RUN TestNoSimulationPackageImportsTheStore
rule_test.go:61: every file in every sealed package read, and not one of them names it
--- PASS: TestNoSimulationPackageImportsTheStore (0.00s)
PASS
ok theworld/internal/store 0.00s
The floor of forty files is there because the dangerous failure of a test like this is not a wrong answer, it is a walk that finds nothing and passes cheerfully. A test that reports success on an empty directory is worse than no test, because it is a green light attached to nothing. The count only ever grows as the tree does, so the floor costs nothing and catches the one way the check can quietly stop checking.
The podman bridge
Every contract in this book since the second volume's has been one
golang:1.26 container run with --network none. That flag gave the container a loopback interface and
nothing else, and it was doing two jobs at once. It proved the build fetched nothing, and
it proved the run talked to no external host. Vendoring took over the first job in the section
above. The second one has to be rebuilt, because the run now has somebody to talk to.
A podman network created with --internal is a bridge that containers can
reach each other on and that has no gateway, so there is no route off it. Nothing on it
can reach the internet and nothing on the internet can reach it, and with no
-p anywhere there is no published port either: the socket exists only
between the two containers.
# the bridge. --internal is the whole point; the pinned subnet is so that
# the error messages this book quotes are the same on every machine.
podman network create --internal --subnet 10.89.77.0/24 world-net
# the database. Pinned by digest and not by tag, so the version string it
# reports is a fact and not a moving target. Its data directory is a
# tmpfs, so it starts empty on every run and takes nothing away with it.
podman run -d --name world-db --network world-net --ip 10.89.77.10 \
--tmpfs /var/lib/postgresql/data:rw,size=512m \
-e POSTGRES_USER=world -e POSTGRES_PASSWORD=world -e POSTGRES_DB=world \
-e PGDATA=/var/lib/postgresql/data/pgdata \
docker.io/library/postgres@sha256:485935f94cc7165afa896978809c37b592dc07f0a37d2c8f645f12412d0212c8
# the toolchain, on the same bridge, with the module mounted read-only and
# the world's own settings in its environment.
podman run -d --name world-go --network world-net --ip 10.89.77.11 \
-e WORLD_DB_HOST=world-db -e WORLD_DB_USER=world \
-e WORLD_DB_PASSWORD=world -e WORLD_DB_NAME=world \
-e GOFLAGS=-mod=vendor -e GOPROXY=off -e GOTOOLCHAIN=local -e LC_ALL=C \
-v "$PWD/src":/src:ro,Z docker.io/library/golang:1.26 sleep infinity
podman exec world-go cp -r /src /bench$ podman network create --internal --subnet 10.89.77.0/24 world-net
world-net
$ podman network inspect world-net --format '{{.Internal}} {{range .Subnets}}{{.Subnet}}{{end}}'
true 10.89.77.0/24
Three of those choices are worth the sentence each. The digest instead
of the tag: postgres:16 is a name that points at different bytes in
different months, and the one line this book quotes from the far side of the socket is
the server's own description of itself. A tag would have made that line the only thing
in the contract that could change without anybody changing anything. The
tmpfs: the data directory lives in memory, so it is gone when the
container is, and every run starts against a database with nothing in it. That costs a
second of start-up and buys a guarantee that nothing here is quietly depending on state
left behind by the last run. The pinned subnet: a connection error from
Go names the address it dialled, and if podman picked a different subnet on every
machine then a page quoting one of those errors would be unquotable. Ten dots
eighty-nine dots seventy-seven is an arbitrary choice made once so that the error text
is arithmetic and not weather.
podman runs and connects containers without a daemon and without root.
podman network create --help lists what a network can be, and
--internal is the flag this chapter turns on;
podman network inspect prints what one actually is, which is the only way
to be sure. Everything here works the same under a rootful podman and the same again
under any other engine that speaks the same flags. This book uses podman and never
docker.
Now the closure, which is a claim and therefore has to be a run. The chapter's bench has a mode that dials three addresses from inside the toolchain container: the peer, by the name podman's own resolver gives it; the Go module proxy, by name; and an address on the public internet, by number, so that no name resolution is involved at all.
$ podman exec -w /bench world-go go run ./cmd/reach -mode probe
reach: everything this container can and cannot open a socket to
world-db:5432 connected
proxy.golang.org:443 dial tcp: lookup proxy.golang.org on 10.89.77.1:53: no such host
1.1.1.1:443 dial tcp 1.1.1.1:443: connect: network is unreachable
one peer, reached by name, and no road off the bridge it is on
Three lines and three different answers, and the first of them is the one that
surprises people: world-db is in no hosts file anybody wrote.
podman runs a small DNS server on the bridge that answers with the addresses of the
containers attached to it, which is why the connection string can say a name instead of a
number and keep saying it when the addresses change. The second line is that same
resolver being asked about something not on the bridge and saying it has never heard of
it: an internal network's DNS forwards nothing. The third skips DNS entirely and goes
straight at an address, and the kernel answers before a packet is sent, because there is
no route in the container's table that could carry it. Nothing timed out and no server
refused anything; the kernel had nowhere to send the packet, so it never sent one.
That is what replaces --network none, and it is a stronger statement than
the flag was. --network none said the container had no network. This says the
container has a network, with exactly one thing on it, and here is the run that walked to
the edge and found nothing there.
psql is Postgres's own terminal client, and the only copy of it on this
bridge is inside the database container, because nothing was installed in the other
one. podman exec world-db psql -U world -d world opens a session;
-c runs one statement and exits, -tAc does the same with the
column headers and the alignment stripped, and \? lists the backslash
commands, of which \dt and \d+ are the two to learn
first.
$ podman exec world-db psql -U world -d world -c '\dt'
Did not find any relations.
That is the state this page leaves the database in, and it is deliberate. The plumbing works and there is nothing in it. Getting a socket to exist and deciding what a world looks like as tables are two separate problems with two separate ways of being wrong, and doing them in one sitting is how you end up with a schema decided by whatever the connection code happened to make easy.
Volumes four, five and six each had a contract that was one command. You ran
./run.sh, one container came up, everything inside it happened in order,
and the container went away. That is over. Volume seven's contract creates a network,
starts a database, waits for it, starts a second container, runs the module in it, and
then has to take all three down again, including when something in the middle failed.
A run interrupted with a keystroke leaves a container and a bridge behind, and the next
run will fail on the name already being in use until somebody removes them.
There is no version of this that does not cost that. A test that involves a database
involves a database, and pretending otherwise means either a fake that proves nothing
about SQL or a shared server whose state is somebody else's problem. What can be done
is to make the teardown unconditional instead of hopeful, which in a shell script
means a trap on exit and not a line at the bottom, and to say plainly
that an interrupted run needs podman rm -f world-db world-go and
podman network rm -f world-net before the next one.
-mod=vendor away and the arrow on the
right starts working. The lower band is kept shut by a bridge with no gateway, and the
two crossed arrows are the two runs on this page that walked into them and came back
with an error.
Readiness from pg_isready
A container that has been started is not a database that is listening. Postgres on a fresh data directory has to create it first: it lays down the system catalogues, starts a temporary server on a unix socket to run its own setup, shuts that down, and only then starts the real one on the TCP port. With the data directory on a tmpfs, that happens on every single run. So there is a window of a second or two in which the container is up, the name resolves, and the port is closed.
The wrong fix is a sleep, and it is wrong in both directions at once. Long enough to be safe on the slowest machine anybody will run this on is time wasted on every machine that is faster. Short enough to feel quick is a race that passes on the desk it was written at and fails on a loaded runner. And a sleep that turns out to be too short does not fail as a sleep; it fails four lines later, as a connection error, in a program that has no idea it was started too early.
The right fix is to ask. In the shell that brings the containers up, that is
pg_isready, which is Postgres's own one-line health check, in a loop with a
ceiling on it.
ready=
for _ in $(seq 1 60); do
if podman exec world-db pg_isready -h world-db -U world -d world -q; then ready=yes; break; fi
sleep 1
done
[ -n "$ready" ] || { echo "world-db: never started listening"; exit 1; }
podman exec world-db pg_isready -h world-db -U world -d world$ podman exec world-db pg_isready -h world-db -U world -d world
world-db:5432 - accepting connections
The -h world-db is not decoration and it is the part most people leave off.
pg_isready with no -h connects over the unix socket in
/var/run/postgresql. During initialisation the official image runs its
temporary server on that socket and nowhere else, precisely so that nothing outside can
connect while the database is being set up. So there is a stretch of time when
pg_isready over the socket says accepting connections and the TCP
port is still refusing every connection offered to it. A loop waiting on the socket
form is not waiting for the thing the Go program is about to need.
Measured on this desk, an eight-core Ryzen 7 3700X, and different on your machine:
over ten restarts of the container, the socket said yes while the port was still
refusing on eight of them. That is not a small window and it is not a rare one, and
being wrong eight times in ten is the mild version of the problem: a check wrong once
in ten would be worse, because the nine right answers teach you to trust it and the
tenth looks like a bug somewhere else. -h world-db asks over the same
transport the world will use, and there is no version of this argument that does not
end with "test the thing you are going to do".
The shell's loop gets the containers into the right order. The program still needs its own, because a daemon meant to be left running will be restarted by things that are not shell scripts: an init system after a reboot, an orchestrator moving it to another machine, a person. In every one of those cases it may come up before its database does, and it has to survive that without a wrapper.
// internal/store/wait.go
// Pinger is the whole of what Wait needs. Taking an interface here
// rather than a *DB is not decoration: it is what lets the readiness
// arithmetic be tested by a program that never opens a socket, and the
// tests below this file do exactly that.
type Pinger interface {
Ping(context.Context) error
}
// Wait asks a database whether it is there, over and over, until it
// says yes or the budget runs out. It returns the number of attempts
// it made.
//
// ...
//
// note, if it is given, is called once for every attempt that failed,
// with the attempt number and the driver's own error. A caller that
// wants a quiet start passes nil.
func Wait(ctx context.Context, p Pinger, budget, step time.Duration, note func(int, error)) (int, error) {
if step <= 0 {
step = 250 * time.Millisecond
}
deadline := time.Now().Add(budget)
for n := 1; ; n++ {
err := p.Ping(ctx)
if err == nil {
return n, nil
}
if note != nil {
note(n, err)
}
if ctx.Err() != nil {
return n, ctx.Err()
}
// The test is whether another whole step fits inside what is
// left, and not whether the deadline has passed. Sleeping past
// a deadline to make one more attempt would turn a budget of
// 30s into a budget of 30s plus a step, and a budget nobody
// can rely on is a number in a config file.
if !time.Now().Add(step).Before(deadline) {
return n, fmt.Errorf("no answer inside a budget of %s, after %s: %w", budget, attempts(n), err)
}
time.Sleep(step)
}
}// internal/store/store_test.go — a database that is not there, and one
// that turns up on the third try. Neither opens a socket.
type stub struct {
answersOn int // the attempt on which it starts saying yes; 0 means never
asked int
}
func (s *stub) Ping(context.Context) error {
s.asked++
if s.answersOn > 0 && s.asked >= s.answersOn {
return nil
}
return errors.New("dial error: connection refused")
}
func TestAWaitWithNoBudgetTriesOnce(t *testing.T) {
s := &stub{}
n, err := Wait(context.Background(), s, 0, 50*time.Millisecond, nil)
if err == nil {
t.Fatal("a database that never answers must not come back as ready")
}
if n != 1 {
t.Fatalf("attempts = %d, want 1: a budget of nothing still buys one try", n)
}
if s.asked != 1 {
t.Fatalf("pings = %d, want 1", s.asked)
}
if !strings.Contains(err.Error(), "after 1 attempt") {
t.Fatalf("error does not say how many tries it made: %v", err)
}
}$ go test ./internal/store/ -run 'AWaitWithNoBudget|AWaitStopsOnTheFirst|AWaitSpendsItsBudget|AWaitCountsEveryFailure' -v
=== RUN TestAWaitWithNoBudgetTriesOnce
--- PASS: TestAWaitWithNoBudgetTriesOnce (0.00s)
=== RUN TestAWaitStopsOnTheFirstAnswer
--- PASS: TestAWaitStopsOnTheFirstAnswer (0.00s)
=== RUN TestAWaitSpendsItsBudgetAndNoMore
--- PASS: TestAWaitSpendsItsBudgetAndNoMore (0.00s)
=== RUN TestAWaitCountsEveryFailureExactlyOnce
--- PASS: TestAWaitCountsEveryFailureExactlyOnce (0.00s)
PASS
ok theworld/internal/store 0.00s
Four tests of a readiness wait, and none of them touches a network. That is what the one-method interface buys. A stub that counts how many times it was asked and starts saying yes on a chosen attempt makes every branch of the loop reachable without a container, without a port, and without the timing of somebody else's process: the no-budget case is exact, the third-try case is exact, and the case where nothing ever answers is checked against the clock and not against a count, so it holds whatever the machine is doing. The one test that does take real time takes a fifth of a second, which is its whole budget.
Take the two numbers this world ships with. The budget is thirty seconds and the step is a quarter of a second. If the database never answers at all, the loop pings, waits a step, pings again, and stops when another whole step would not fit: thirty divided by a quarter is a hundred and twenty, so a hundred and twenty attempts and then an error, at thirty seconds and not a moment more. A budget of nothing is the one edge, and it still buys a single attempt, because the loop asks before it looks at the clock.
Now the case that actually happens. Measured from inside the container next to it, on an eight-core Ryzen 7 3700X and different on your machine, this database's port opens about three quarters of a second after it is started. The loop is checking every quarter second, so it finds out somewhere between three quarters of a second and a whole one: it notices on the first check taken after the port opened, and the checks are a step apart. The time thrown away is whatever was left of that step, and it is under a quarter of a second however long the database took. That is the whole of the cost.
Against that, a sleep. A two-second sleep in front of a database that took three quarters of one throws away one and a quarter seconds, every run, forever. The same sleep in front of a database that takes three, because the machine is loaded or the data directory is cold, does not wait a moment longer; it hands a closed port to the next line of the program. The budget is better on both counts, and the reason is that it measures the thing it cares about instead of guessing a number that stands in for it.
The same three lines in symbols.
N = whole part of (B ÷ T), or 1 if that comes to nothing
time to notice a database that opened at R = T × (R ÷ T rounded up)
time wasted = that, minus R, which is always less than T
// cmd/worldd/main.go
func main() {
once := flag.Bool("once", false, "report what the database says and exit, instead of holding the pool open")
loud := flag.Bool("loud", false, "print every failed attempt while waiting, not just the last")
flag.Parse()
cfg, err := store.FromEnv()
if err != nil {
fmt.Fprintln(os.Stderr, "worldd:", err)
os.Exit(1)
}
fmt.Printf("worldd: postgres at %s, database %s, as %s\n", cfg.Peer(), cfg.Name, cfg.User)
ctx := context.Background()
db, err := store.Open(ctx, cfg)
if err != nil {
fmt.Fprintln(os.Stderr, "worldd:", err)
os.Exit(1)
}
defer db.Close()
var note func(int, error)
if *loud {
note = func(n int, err error) { fmt.Fprintf(os.Stderr, "worldd: attempt %d: %v\n", n, err) }
}
if _, err := store.Wait(ctx, db, cfg.Budget, cfg.Step, note); err != nil {
fmt.Fprintf(os.Stderr, "worldd: the database at %s did not answer: %v\n", cfg.Peer(), err)
os.Exit(1)
}
v, err := db.Version(ctx)
if err != nil {
fmt.Fprintln(os.Stderr, "worldd:", err)
os.Exit(1)
}
fmt.Printf("worldd: %s\n", v)
fmt.Printf("worldd: pool of at most %d connections, each retired after %s, give or take %s\n",
cfg.MaxConns, cfg.Life, cfg.Jitter)
fmt.Println("worldd: no schema here yet, so nothing to write and nothing to read")
if !*once {
stop := make(chan os.Signal, 1)
signal.Notify(stop, os.Interrupt, syscall.SIGTERM)
fmt.Println("worldd: holding the pool open; interrupt to stop")
<-stop
fmt.Println("worldd: asked to stop")
}
db.Close()
fmt.Println("worldd: stopped, and every connection handed back")
}$ podman exec -w /bench world-go go run ./cmd/worldd -once
worldd: postgres at world-db:5432, database world, as world
worldd: PostgreSQL 16.15 (Debian 16.15-1.pgdg13+2) on x86_64-pc-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit
worldd: pool of at most 4 connections, each retired after 30m0s, give or take 5m0s
worldd: no schema here yet, so nothing to write and nothing to read
worldd: stopped, and every connection handed back
Everything before this in the book was a bench: a program with an end written into it,
so many ticks or years or generations, which prints what it found and stops. worldd has
no end in it. Without -once it blocks on a channel that nothing sends to
except the signal handler, and the last two lines are what a program does when somebody
finally asks it to stop: it says so, gives the connections back, and says that too. The
defer and the explicit Close at the bottom are not
duplication being sloppy. The defer covers the error paths above; the explicit call is
the one that runs on the ordinary path, before the last line is printed, so that the
line is true when it is printed and not shortly afterwards.
That second line is the only thing in this volume quoted from the far side of the socket, and it is why the image is pinned by its digest.
Set the wait to nothing and start the daemon in the same breath as the database, which is what happens by default to anything started by a compose file, a unit file or an orchestrator that brings two things up at once. The database is killed and started again here to put it back in the state it is in for the first second or two of every run.
$ podman kill world-db && podman start world-db world-db world-db $ podman exec -e WORLD_DB_WAIT=0 -w /bench world-go go run ./cmd/worldd worldd: postgres at world-db:5432, database world, as world worldd: the database at world-db:5432 did not answer: no answer inside a budget of 0s, after 1 attempt: failed to connect to `user=world database=world`: 10.89.77.10:5432 (world-db): dial error: dial tcp 10.89.77.10:5432: connect: connection refused exit status 1
Read that error from the inside out, because every layer of it is telling you something. The innermost part is the kernel's: connect: connection refused. That is not a timeout and not a name that failed to resolve. A machine answered, at 10.89.77.10, and said there is nothing listening on port 5432. The name worked, the route worked, the container is running. Postgres is inside it laying down a data directory and has not opened the port yet.
The next layer out is pgx's, and it names the address it dialled and the connection parameters it dialled with, which is exactly the pair you need when the answer turns out to be that you were talking to the right port on the wrong host. The outermost layer is the daemon's own: one attempt, inside a budget of zero. And that is the diagnosis. The program did not fail because the database is broken; it failed because it asked once, at the only moment in the next month when the answer was no.
The cure is the loop, and here is the same instant with the budget the daemon ships with instead of zero. Same command, same restart, same second.
$ podman exec -w /bench world-go go run ./cmd/worldd -once
worldd: postgres at world-db:5432, database world, as world
worldd: PostgreSQL 16.15 (Debian 16.15-1.pgdg13+2) on x86_64-pc-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit
worldd: pool of at most 4 connections, each retired after 30m0s, give or take 5m0s
worldd: no schema here yet, so nothing to write and nothing to read
worldd: stopped, and every connection handed back
The reason to walk through this instead of writing the loop and moving on is that the failure is invisible until it happens and then it looks like something else. On a developer's machine the database is already up, so a program with no wait in it works every time it is run by hand and fails only when something starts it automatically. The report that comes back says "it crashes on the server". It does not crash on the server. It starts too early, everywhere, and the server is the only place anything starts it early enough to notice.
The connection pool lifetime
Opening a connection to Postgres is not cheap. The server forks a process for it, and the client and server exchange a startup message, an authentication round trip and a handful of parameter settings before a single query can be sent. Doing that per query would be absurd for a daemon that runs for weeks. So there is a pool: a small set of connections kept open, handed out on request and given back afterwards.
The whole difficulty of a pool is that a connection in it is doing nothing most of the time, and things that do nothing get quietly killed. The server has its own idea of how long an idle session may sit. A restart of the database takes every one of them with it. A firewall between two machines keeps a table of the connections it has seen and drops entries that have been silent too long, after which the two ends are each convinced they are still connected and neither ever hears from the other again. None of those tell the client anything. The client finds out at the worst moment: the next time it tries to use the connection, which is the middle of doing something it cared about.
MaxConnLifetime is the answer to that, and the idea generalises well past
databases. Retire the connection on your own schedule, while it still works, rather than
discovering on somebody else's schedule that it stopped working. A connection with a
thirty-minute life is one that never gets old enough for any of the above to have
happened to it, and the retirement lands at a moment when no query was in the middle of
anything.
// internal/store/store.go
func Open(ctx context.Context, c Config) (*DB, error) {
pc, err := pgxpool.ParseConfig(c.URL())
if err != nil {
return nil, fmt.Errorf("store: %s: %w", c.Safe(), err)
}
pc.MaxConns = c.MaxConns
pc.MinConns = c.MinConns
pc.MaxConnLifetime = c.Life
pc.MaxConnLifetimeJitter = c.Jitter
pc.MaxConnIdleTime = c.Idle
pc.HealthCheckPeriod = c.Check
p, err := pgxpool.NewWithConfig(ctx, pc)
if err != nil {
return nil, fmt.Errorf("store: %s: %w", c.Safe(), err)
}
return &DB{pool: p, cfg: c}, nil
}// internal/store/store.go — the defaults, and every one of them argued
func Default() Config {
return Config{
Host: "world-db",
Port: 5432,
User: "world",
Name: "world",
MaxConns: 4,
MinConns: 1,
Life: 30 * time.Minute,
Jitter: 5 * time.Minute,
Idle: 5 * time.Minute,
Check: 30 * time.Second,
Budget: 30 * time.Second,
Step: 250 * time.Millisecond,
}
}$ podman exec -w /bench world-go go run ./cmd/reach -mode life
reach: a connection retired by the clock rather than by a broken query
one connection at a time, retired after 2s, the pool looking every 250ms
three queries: two together, then one after 3s of nobody asking
queries 1 and 2, taken together, ran on one backend true
queries 1 and 3, taken a lifetime apart, ran on one backend false
no process id is printed above: the number a server gives a backend is
the database's own, and nothing in this book quotes one
The bench asks the server which of its own processes served each query and compares the answers rather than printing them. Two queries run back to back get the same one, which is the pool doing its job. A third query taken after the connection has outlived its two seconds gets a different one, which is the pool doing the other half of its job: the health check found a connection older than its lifetime, closed it while no caller was holding it, and opened a fresh one when the next query asked. Nothing broke, nothing was retried, and no query anywhere saw an error.
Five numbers, and each one is a decision.
Life, thirty minutes, and up to five more. Short enough that nothing on the list above has had time to happen, long enough that a connection is reused thousands of times before it goes. The five minutes are the jitter: when a connection is made, a random slice of up to five minutes is added to its half hour, so its age is fixed the day it is born and no two are the same. Without that, a pool that opened four connections in the same second would retire all four in the same second, and four reconnections at once is a small stampede on a predictable timetable.
Idle, five minutes. A connection no caller has used in five minutes is closed even if it is young, because holding a backend open on the server for a caller that has gone quiet is rent paid for nothing.
Check, thirty seconds. Neither of the two rules above enforces itself. Something has to walk the pool and act on them, and it does so twice a minute. That is the granularity of both: a connection is closed somewhere in the thirty seconds after its own age comes up, not on the second.
MaxConns, four. This is the number the architectural rule at the top of the chapter pays for. Nothing inside a tick asks this package for anything, so the only callers are the handful of jobs that run between ticks, and there are not four of those happening at once. A pool sized for imaginary concurrency is not free: every connection in it is a process on the server, with its own memory, whether or not anybody is using it. The general form is that a pool should be sized by what actually contends for it, and the reason this world can answer that question at all is that it decided, on purpose, that the busiest loop in the program is not a caller.
MinConns, one. One connection is kept open so the first query after a quiet stretch does not pay for a handshake, and one is enough because there was never going to be a burst.
$ go test ./internal/store/ -run 'APasswordNever|EveryConnectionIsRetired|AConfigItCannotRead' -v
=== RUN TestAPasswordNeverReachesALogLine
--- PASS: TestAPasswordNeverReachesALogLine (0.00s)
=== RUN TestEveryConnectionIsRetiredWhileItStillWorks
--- PASS: TestEveryConnectionIsRetiredWhileItStillWorks (0.00s)
=== RUN TestAConfigItCannotReadIsAnErrorAndNeverADefault
--- PASS: TestAConfigItCannotReadIsAnErrorAndNeverADefault (0.00s)
PASS
ok theworld/internal/store 0.00s
The middle one of those is the odd test and the useful one. It does not check that the pool works; it checks that the five numbers stand in the right relation to each other: that every one of them is set at all, that the jitter is smaller than the life it spreads, that an idle connection goes before an old one does, that the health check runs often enough to act on either, and that the pool is small. Those are the constraints the paragraphs above argue for, and a constraint written only in a paragraph is one somebody will edit past in a year. The third test is the same idea aimed at the environment: a port that is not a number and a wait that is not a length of time both stop the daemon rather than falling back to a default, because a daemon that silently connected to the default host after a typo would be writing one world's history into another world's database and looking healthy while it did it.
Finally, the daemon doing the thing it is for: started, left alone, and interrupted.
$ podman exec -w /bench world-go sh -c 'go build -o /tmp/worldd ./cmd/worldd && (/tmp/worldd & p=$!; sleep 2; kill -INT $p; wait $p)'
worldd: postgres at world-db:5432, database world, as world
worldd: PostgreSQL 16.15 (Debian 16.15-1.pgdg13+2) on x86_64-pc-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit
worldd: pool of at most 4 connections, each retired after 30m0s, give or take 5m0s
worldd: no schema here yet, so nothing to write and nothing to read
worldd: holding the pool open; interrupt to stop
worldd: asked to stop
worldd: stopped, and every connection handed back
Two seconds there and it would have been two weeks with no change to the program. The interesting line is the last one, and it pays to be exact about what it costs to leave it out. A process killed without a handler exits with its connections still open as far as it is concerned; the server finds out when the sockets close, tears the backends down, and rolls back anything they were in the middle of. Nothing is corrupted. But "nothing is corrupted" is a different sentence from "everything was finished", and a daemon that will one day be holding a transaction when somebody stops it should be in the habit of being asked rather than shot.
Two closures with different reach
The pattern here leaves the valley entirely, and it is about what "this program does not reach the network" actually claims when it is said about a real build.
There are two of them, and they are kept by different machinery at different times. The first is about the build: the set of source files that go into the binary is fixed and on disk, so building it twice a year apart gives the same bytes. The second is about the run: the set of hosts the program will talk to is fixed, so running it cannot become a conversation with somebody you did not plan on. For five volumes this book kept both with one flag, and the flag was doing the work of two different guarantees without anybody noticing they were separable. The moment the run needed one peer, they had to come apart, and each one needed its own mechanism: a directory and two environment variables for the build, a bridge with no gateway for the run.
The second thing is about proving a negative, and it is the same lesson volume six drew
from a phase step that was supposed to be invisible. "This container cannot reach the
internet" is a sentence, and a sentence in a comment counts for nothing next to a run
that tried it and printed what came back. Both of the crossed arrows in the figure are runs.
The build one is a real go build that really went looking for pgx and was
really stopped. The run one is a real socket call that came back unreachable before a
packet left the machine. Neither costs more than a second, and both turn an assumption
into a line of output that changes the day somebody breaks it.
The third is about where waiting is allowed to live, and it is the whole of the architectural rule. Every piece of this chapter can block: a DNS lookup, a connect, a handshake, a query, a pool with all four connections busy. The world's tick loop cannot afford any of them, not because blocking is bad but because a tick with an unpredictable cost stops being a unit you can count. The way to keep that is not discipline, because discipline is a thing that lasts until somebody is in a hurry. It is to make the simulation packages unable to name the package that owns the socket, and then to write the check that says they still cannot.
And the fourth is smaller and comes up everywhere: pin the thing whose output you quote. The image is pinned by digest because the page prints the server's version string. The subnet is pinned because the page prints an error containing an address. The dependency is pinned to a version because the page quotes error messages that the driver itself writes. None of those three is a security measure and none of them is about reproducible builds in the abstract. Each one is there because a specific line of output would otherwise be a line no run could check.
- Say what
go mod vendorputs in the tree thatgo.sumdoes not, and why this book's contract needs the first even though it already had the second. - Given the three lines the probe printed, say which of them involved the bridge's DNS server, which one skipped it, and what the difference between no such host and network is unreachable tells you about where each one stopped.
- Work out how long the readiness loop takes to give up if the database never comes up at all, and say why the answer is not thirty seconds plus one step.
- The worked failure printed an error with three layers in it. Name what each layer tells you, and say which layer would have changed if the container name had been wrong instead of the timing.
- Explain why the pool retires a connection that is working perfectly well, using at least two of the things that can kill an idle connection without telling the client.
- Given the rule that nothing inside a tick calls
internal/store, say how you would justify aMaxConnsof four to somebody who thinks it looks far too small.
Exercise 1 — break the closure on purpose. Bring the two containers up on an ordinary podman network instead of an internal one, and predict which lines of the probe change before you run it.
Drop --internal from the podman network create line and
leave everything else alone. The first line does not move: the peer was always
reachable and reaching it never needed a route off the bridge. The other two change
for two different reasons. The numeric address now has a gateway to hand its
packets to, so network is unreachable becomes a connection on any machine
with a way out. The name is the more interesting one: a bridge that is not internal
forwards the queries its own DNS server cannot answer, so
proxy.golang.org resolves and the dial after it succeeds.
$ podman exec -w /bench world-go go run ./cmd/reach -mode probe (on a bridge created without --internal)
reach: everything this container can and cannot open a socket to
world-db:5432 connected
proxy.golang.org:443 connected
1.1.1.1:443 connected
this bridge is open: something answered that a closed one could not
The last line changed too, and it is the reason the bench reads its summary off the three answers instead of printing a sentence about closure regardless. Now do the other half: build in that container with vendor mode off and a real module proxy.
$ podman exec -e GOFLAGS=-mod=mod -e GOPROXY=https://proxy.golang.org,direct -w /bench world-go go build ./cmd/worldd
go: downloading github.com/jackc/pgx/v5 v5.10.0
go: downloading github.com/jackc/puddle/v2 v2.2.2
go: downloading github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761
go: downloading github.com/jackc/pgpassfile v1.0.0
go: downloading golang.org/x/text v0.29.0
go: downloading golang.org/x/sync v0.17.0
Six downloads and a binary, exit status zero. The order the six come in is the
toolchain's business and will not be the same twice. That is the whole point of
doing this: with the network open, a build that has quietly stopped using
vendor/ still succeeds, and the only visible difference between it and
a hermetic build is six lines no reader sees. A closure is not something you notice
losing.
Exercise 2 — count the attempts. Work out how many attempts a budget of one second buys, then point the daemon at a port with nothing behind it and check the answer against the run.
The step is a quarter of a second, so the interlude's arithmetic gives the whole part of one divided by a quarter: four. Nothing has to be starting up to check that. A port with nothing behind it refuses every attempt in the same words, which is what makes it a clean way to count, and the database container is listening on 5432 and on nothing else.
$ podman exec -e WORLD_DB_PORT=5433 -e WORLD_DB_WAIT=1s -w /bench world-go go run ./cmd/worldd -loud
worldd: postgres at world-db:5433, database world, as world
worldd: attempt 1: failed to connect to `user=world database=world`: 10.89.77.10:5433 (world-db): dial error: dial tcp 10.89.77.10:5433: connect: connection refused
worldd: attempt 2: failed to connect to `user=world database=world`: 10.89.77.10:5433 (world-db): dial error: dial tcp 10.89.77.10:5433: connect: connection refused
worldd: attempt 3: failed to connect to `user=world database=world`: 10.89.77.10:5433 (world-db): dial error: dial tcp 10.89.77.10:5433: connect: connection refused
worldd: attempt 4: failed to connect to `user=world database=world`: 10.89.77.10:5433 (world-db): dial error: dial tcp 10.89.77.10:5433: connect: connection refused
worldd: the database at world-db:5433 did not answer: no answer inside a budget of 1s, after 4 attempts: failed to connect to `user=world database=world`: 10.89.77.10:5433 (world-db): dial error: dial tcp 10.89.77.10:5433: connect: connection refused
exit status 1
Four, as predicted, and no line of the program says four anywhere: change the step and it moves, change the budget and it moves. The final line carries the same driver error as the last note, because the error a loop gives up with is the last one it saw.
Now try the same budget against a database that really is starting, and something turns up that a dead port cannot show you. There is a second window, after the port opens and before the server will answer anything, in which the connection is accepted and the reply to the startup message is FATAL: the database system is starting up. On the same eight-core Ryzen 7 3700X, and different on your machine, that window was a few hundredths of a second on a restart and long enough on a first start to be caught by a loop checking four times a second. So "too early" is two states and not one, and a readiness check that only opened a socket would have called the second of them ready.
Exercise 3 — watch the jitter do its job. The lifetime demo runs with the jitter set to zero. Predict what a pool of four connections with a thirty second life and no jitter does at the thirty second mark, and what changes when the jitter comes back.
With no jitter and four connections opened within a few milliseconds of each other, all four cross their lifetime inside the same health check. The check closes all four, and the next four queries each pay for a fresh connection: a process fork on the server, a startup message, an authentication round trip. Then thirty seconds later it happens again, at the same offset, forever. The pool has turned itself into a metronome, and every beat is four handshakes at once.
The jitter is a random slice of up to five minutes added to each connection's lifetime at the moment the connection is made, so four connections opened together are retired at four different moments spread across that window. Nothing about the average changes. What changes is that the work is spread instead of arriving in a lump, and the pattern that a busy server would otherwise see, four reconnections every thirty seconds on the dot, is gone. It is the same reasoning as spreading the start times of a hundred machines' nightly jobs, and the same one line of configuration solves it.
There is a database on the bridge, the world can reach it, and it has nothing in it. The questions the archive file could not answer are questions about ancestry, and answering them means deciding what a creature is when it is written down: which of the numbers in a genome are things you will ever filter or join on, and which are things you will only ever read back in one piece. A genome is thousands of bytes of weights that nothing will ever compare, and a parent is one integer that everything will.