The World Vol 7 · The Living Valley
ch 76 / 105
Chapter 76

The World as Tables

Columns for asked questions

There is a database on the bridge and nothing in it. What goes in it has to be decided before a single row is written, because a table that holds two centuries of history is a table you are stuck with.

A column exists because something filters, joins, groups or climbs on it. Everything else is one blob beside those columns, read back whole and never looked inside. The test is a question somebody actually asks, and not a feeling about which numbers are important.

A creature written to the archive is one JSON object to a line, appended and never rewritten, a line for every birth and a line for every death. A born line holds the creature's number, the tick it stood up on, the species the register sorted it into, its two parents, and then every number it inherited.

The inherited part is the ten body factors, the controller, the three temperament genes and the four look genes. The controller is the bulk of it. A grown network is written as its nodes and links, every link carrying the innovation number it was minted under, and a creature with a hundred and forty-four links in it is a line most of the way to nine kilobytes.

The questions asked after a creature is dead are smaller: who its parents were, what tick it was born on, which species it belonged to, how many generations it stood from a founder, and which of the two rows it stood on. Every one of those is an integer, and all of them together come to forty-four bytes. No question has mentioned the weight on link two hundred and eleven.

That gap is the whole design. One false schema treats every number as equally likely to be asked about, gives each one a column, and produces a table nine thousand columns wide that is slow at everything and good at nothing.

The other false schema treats the whole creature as one lump of bytes, which is what the archive already does, and produces something you cannot ask a question of at all.

Applied to a creature it gives seven integer columns and one jsonb column, with an index on five of the seven and nothing anywhere pointing into the eighth. Applied to the rest of the world it gives six more tables: the world itself, the chronicle it keeps, the deaths, the species register, and the two that a saved world takes. Seven in all, and the argument for each column in each of them is the same argument in a different setting.

By the end of this page there is a migrations/ directory holding the schema as numbered files, a runner that applies them in order inside a transaction apiece, a ledger table that records what has been applied, and seven tables in a database that had none. And running the whole thing a second time does nothing whatsoever, which takes more thought than it sounds like it should.

Three things are not on this page and it helps to name them now. Nothing is written into any of the seven tables: there is not a row in this world yet, and putting one in is a separate problem with a separate way of being wrong. No snapshot is taken, though the two tables one would go into are made here, because they are part of the schema and a schema is made all at once or it is made in pieces that do not fit. And no question is asked of the creature table beyond counting its rows, which comes to nought.

The queried columns

The rule says to measure before choosing, so here is the measurement. The chapter's bench runs a short valley of both rows, keeps an archive of it exactly as the evolution volume did, and then reads that archive back to work out what a creature costs and which parts of the cost are the parts anything asks about. Twelve years is enough: long enough that genomes have drifted away from their founders and lines are thirty generations deep, short enough that the run is forty-two thousand ticks and about sixteen seconds on an eight-core Ryzen 7 3700X. That last number is this desk talking and yours will differ; every other number below is the same on any machine.

▣ Build · stage 1 — a creature with its genome taken out
$ go run ./cmd/tables -mode weigh | tail -72 | head -18
tables: what a creature weighs, and which of its numbers anything ever asks about

  a 16x12 valley, 21 browsers off stream 12 and 8 hunters off stream 22, 12 years
  every birth and every death appended to tables.jsonl, and nothing in the valley reads it

  the archive at the end of 12 years
    born records                                         1303
    of those, on the browsers' row                       1287
    and on the hunters'                                    16
    died records                                         1096
    browsers still walking                                207
    hunters still walking                                   0
    bytes                                            11913330
    steps from the deepest creature to a founder           33

  the last born record of each row, with its genome taken out
{"r":"born","id":1303,"tick":43644,"kind":1,"of":1228,"with":700}
{"r":"born","id":54,"tick":905,"kind":1,"of":28,"row":1}

Those two lines are six of the seven columns, written out, and the seventh is argued into existence further down this page. The first is a browser born in the twelfth year to two parents; the second is one of sixteen hunters this valley ever had, born in the first year to a single parent, on a row that is written down because it is not the default. Sixty-odd bytes each. Everything else on the line the archive really wrote is the genome, and the run above threw it away to print these.

The hunters are gone, which the block above says in its own count of them still walking, and that is the valley telling the truth about itself and not a fault in the run. Their sixteen born records are in the archive all the same, which is the point of an archive: an animal that lasted two years is as permanently written down as one that lasted a century.

▣ Build · stage 2 — the split, in bytes
$ go run ./cmd/tables -mode weigh | tail -53 | head -17
  one born record, split the way this schema splits it
                                                        bytes    share
    the genome, as one column would hold it            9041.6    99.3%
    everything else on the line                          60.3     0.7%
    the whole line                                     9101.8   100.0%

  the seven columns, and what asks about each
    column      type     bytes   asked by
    id          bigint       8   every join, every climb, every death
    row         int          4   which of the two rows this animal stood on
    species     int          4   how many stood in a species, and when it opened
    tick        bigint       8   everything about when: births a year, a lifetime, a lag
    parent      bigint       8   one step up a pedigree, and the whole climb is this column
    mate        bigint       8   the other step up, and who bred with whom
    generation  int          4   how far a creature stands from a founder
                            44   the whole of what is queryable
    genome      jsonb     9042   read back whole, joined on never

Ninety-nine point three percent against nought point seven. The right-hand column of the lower table is the test the rule asks for, filled in for each of the seven, and every one of them passes it with a question somebody has already asked in this book. id is what a death is recorded against and what a parent is a reference to. tick is every question about when. species is the register's own key. parent and mate are the two ways up a pedigree. row says whether an animal browsed or hunted, and it is a column because the entire trophic argument is a question about counting the two rows separately.

generation is the one that has to be argued differently, because it is not a fact about a creature at all: it is the length of the walk from that creature to a founder, and it can always be worked out again from parent. Storing a thing you can recompute is usually a mistake. It is four bytes on a row of nine thousand, and the recomputation is a climb of thirty-three steps done once for every creature in every table of drift anybody ever prints. Four bytes to turn a walk into a read is not a close call.

The other half of the argument is why the genome cannot be columns even if somebody wanted it to be, and it has nothing to do with size.

▣ Build · stage 3 — a genome is not a fixed number of numbers
$ go run ./cmd/tables -mode weigh | tail -28 | head -11
  how many numbers a genome holds, over the 1303 born records
     numbers   creatures   browsers   hunters
         168        1099       1099         0
         169          25         25         0
         170          80         80         0
         171          80         80         0
         172           2          2         0
         173           1          1         0
         192          16          0        16
    7 widths in 12 years, and a column apiece would need 192 columns
    Postgres stops a table at 1600, and a genome is not 192 numbers twice running

Seven different widths in twelve years, in one small valley. A founding browser carries a hundred and sixty-eight numbers and a founding hunter a hundred and ninety-two, because the two rows are wired to different sets of outputs; and structural mutation adds a link or a node to a lineage whenever it fires, so a browser thirty generations down carries a hundred and seventy-three. There is no width that is the width of a genome. There is a width per creature, and it changes as the population evolves, which is the entire point of the population.

A table of columns has one width for every row in it. To hold this valley you would need a hundred and ninety-two columns, of which most rows would use a hundred and sixty-eight and leave the rest null; and the first creature to grow past that would need a migration, on a live database, in the middle of year forty. Postgres would let you go to sixteen hundred columns and then stop, and a network with five hundred links in it is well past that. The variable width is what settles it, and the ninety-nine percent is how much it costs to be wrong.

▣ Build · stage 4 — the row itself, and what an index on it costs
$ go run ./cmd/tables -mode weigh | tail -35 | head -6
  the same creature as one row of the creature table
    the seven integer columns                              44
    the row's own header, whatever is in it                23
    the genome column                                  9041.6
    a row                                              9108.6
    what a query can filter on, of a row                0.48%
$ go run ./cmd/tables -mode weigh | tail -16 | head -14
  finding one creature by its number, in a history of 1303 births
    records read walking the archive                     2399
    bytes parsed doing it                            11913330
    comparisons in an ordered index, log2 n                11

  climbing the deepest pedigree in it, 33 steps to a founder
    records read, a pass over the file a step           79167
    comparisons, one descent a step                       363
    the first over the second                             218

  and what keeping one column in order costs
    keys in an index on parent, at 8 bytes each         10424
    the table those keys point into                  11867724
    the keys, as a share of the table                   0.09%

The first block is the row Postgres stores: twenty-three bytes of its own bookkeeping, forty-four bytes of integers, and the genome. The second is the reason anybody bothers. The archive answers "who was creature 1303's parent" by reading eleven point nine megabytes from the beginning; a column kept in order answers it in eleven comparisons, because that is how many halvings it takes to get from thirteen hundred to one. Climb the deepest line in the valley and the file is read thirty-three times over, seventy-nine thousand records, against three hundred and sixty-three comparisons. Two hundred and eighteen to one, and the ratio grows as the history does, because one side of it is a count and the other is a logarithm.

The last block is the part that surprises people who have been taught to be careful with indexes. An index on parent holds one eight-byte key for every row, and the rows are nine kilobytes each, so the keys come to nine hundredths of one percent of the table. The tree around them is bigger than the keys alone, and it is still nothing. That is why the creature table can afford four indexes without anybody arguing about it: an index costs what it costs relative to the row it points into, and a table with a blob in it is the cheapest table in the world to index.

∑ Math Interlude — a row, an index, and a climb

Three sums, all of them arithmetic on numbers that came out of the run above or out of the definition of a type. Start with the row. Postgres puts twenty-three bytes in front of every row it stores, whatever is in it. A bigint is eight bytes and an int is four, and those are the types' own widths and not measurements. Four bigints and three ints come to thirty-two plus twelve, which is forty-four. The genome measured nine thousand and forty-one point six. So a row is 23 + 44 + 9,041.6 = 9,108.6 bytes, and the part of it a WHERE clause can mention is 44 of those, which is nought point four eight percent.

Second sum, the index. An index on one bigint column holds one key of eight bytes for every row. Against a row of 9,108.6 that is 8 ÷ 9,108.6, which is nought point zero nine percent, and the number of rows never enters it, because the ratio is per row. The four kept beside the primary key come to nought point three five percent between them. Now do the same sum on a different table. Two bigints and a header is 23 + 16 = 39 bytes a row, so one index on one of them is 8 ÷ 39, better than a fifth of the table, and four of them come to more than four fifths of it. Same eight bytes, same arithmetic, and the answers are three orders of magnitude apart. What decides whether an index is expensive is the row beside it.

Third sum, the climb, and it has two different counts in it. The archive holds a line for every birth and a line for every death, so finding one creature in it means reading all R of them: 2,399 here, against N = 1,303 births. An ordered column holds one key a birth, and finding one of those means halving the range until one is left, which takes log2 N comparisons rounded up: 11 at 1,303, and 17 at a hundred thousand. Climbing a pedigree G steps deep is G × R records one way and G × log2 N comparisons the other, so the G cancels and the saving is R ÷ log2 N. Here that is 2,399 ÷ 11, which is the 218 the run printed. Grow the history to a hundred thousand births and R grows with it while the logarithm crawls from 11 to 17, so the factor goes to something near twelve thousand. That is the whole reason for putting a history in a database instead of a file, stated as a division.

Bthe genome as one value: 9,041.6 bytes on this run, and a different number on yours
Kthe seven integer columns together: 44 bytes, from the types and not from a measurement
Hthe 23 bytes Postgres puts in front of every row, whatever the row holds
Nhow many rows the creature table holds: 1,303 here, one a birth
Rhow many records the archive file holds: 2,399 here, one a birth and one a death
Ghow many steps a pedigree climb takes: 33 for the deepest line in this valley
log2 Nhow many halvings it takes to get from N to one: 11 at 1,303, 17 at 100,000
a ÷ ba divided by b
One row of the creature table, drawn to scale A wide bar standing for one row of the creature table, 9,108.6 bytes long. A sliver at its left end is the row header and the seven integer columns; the whole of the rest is the genome column, jsonb, 9,041.6 bytes, 99.3 percent of the row. An arrow leads down from the sliver to an enlarged band showing the seven integer columns side by side: id, eight bytes, indexed; row, four bytes, not indexed; species, four bytes, indexed; tick, eight bytes, indexed; parent, eight bytes, indexed; mate, eight bytes, indexed; generation, four bytes, not indexed. Below them a single wide box for the genome column, marked read back whole and joined on never. ONE ROW OF THE CREATURE TABLE, TO SCALE 9,108.6 bytes: what the database stores for one birth genome, as jsonb: 9,041.6 bytes, 99.3% THE 67 BYTES AT THAT END, ENLARGED id 8 bytes indexed row 4 bytes no index species 4 bytes indexed tick 8 bytes indexed parent 8 bytes indexed mate 8 bytes indexed generation 4 bytes no index 44 bytes, plus 23 of header, and every question in this book is about one of them AND THE COLUMN NOTHING EVER ASKS ABOUT genome jsonb fetched by id, handed back whole an index on one of the seven is 8 bytes of key against a 9,108-byte row
Figure 76.1 — the row drawn at its real proportions. The five highlighted columns are the ones an index is kept on; row holds two distinct values in the whole table and generation is read in whole bands at a time, so neither buys anything by being ordered. Every byte outside the sliver at the left is fetched by id and handed back without the server looking inside it.
◆ Note — the column the archive did not have

Working the seven columns out found a hole in the file they are meant to come out of. The archive wrote a creature's number, tick, species and parents, and it did not write which row the animal stood on, because until this volume there was only one row. A genome read back off such a line comes back on the browsers' row with every one of its numbers correct, and a factor of 1.20 on a browser's Bulk is a different weight from the same 1.20 on a hunter's, so the animal you get back is not the animal that lived.

So the record gained a field, written only when it is not the default, and the two lines in the first block above are what that looks like. This is what a schema is for besides holding rows: it is a list of what somebody has to write down, and the exercise of making the list finds the missing write sites.

$ go test ./internal/gene/ -run ARecordCarriesTheRowItStoodOn -v
=== RUN   TestARecordCarriesTheRowItStoodOn
    roll_test.go:281: both rows written down, both read back, and a browser's line unchanged
--- PASS: TestARecordCarriesTheRowItStoodOn (0.00s)
PASS
ok  	theworld/internal/gene	0.00s

The migration files

A schema is a set of files under migrations/, numbered, applied in the order of their numbers, and never edited once they have been applied to anything anywhere. Six files make the seven tables, because the two a snapshot takes are made together and one change is one file.

The world comes first, since everything else refers to it.

▣ Build · stage 5 — the world, and the chronicle it keeps
-- The world itself: one row, written once at tick zero, never updated.
-- Everything else in this schema hangs off it.
--
-- one is always true and can be nothing else, so a unique constraint
-- on it is the database refusing a second world.
CREATE TABLE world (
    world   text    NOT NULL,
    seed    bigint  NOT NULL,
    opened  bigint  NOT NULL,
    genesis jsonb   NOT NULL,
    one     boolean NOT NULL DEFAULT true,

    CONSTRAINT world_key         PRIMARY KEY (world),
    CONSTRAINT world_is_one_row  UNIQUE (one),
    CONSTRAINT world_one_is_true CHECK (one),
    CONSTRAINT world_opened_on_a_real_tick CHECK (opened >= 0)
);

Four columns an operator chose and a fifth that is a trick. one is true and a check constraint forbids it being anything else, so a unique constraint on it means the table can hold exactly one row. Two worlds in one database is not something this book has an answer for: the snapshot tables name a world and the creature table does not, so a second world would put two creatures numbered 1 in the same table with nothing to tell them apart. The database can be made to refuse that, in three lines, so it is.

genesis is a blob for the same reason the genome is. It is prose an operator wrote, read once and printed thereafter, and nothing will ever be selected by a line of it.

-- What happened, in the world's own words, appended and never
-- rewritten. entry is here because tick is not a key: a tick in which
-- four things happen is four rows, and the database hands out the
-- order rather than letting anything write its own.
CREATE TABLE chronicle (
    entry bigint GENERATED ALWAYS AS IDENTITY,
    tick  bigint NOT NULL,
    kind  text   NOT NULL,
    text  text   NOT NULL,

    CONSTRAINT chronicle_key PRIMARY KEY (entry),
    CONSTRAINT chronicle_happened_on_a_real_tick CHECK (tick >= 0)
);

-- Every question asked of a chronicle is about a stretch of time, and
-- entry on the end returns one tick's rows in the order they were written.
CREATE INDEX chronicle_by_tick ON chronicle (tick, entry);

entry is the column no query asked for and the one that has to be argued hardest. The chronicle records what happened, and the obvious key is the tick it happened on, until you notice that four things can happen on one tick and that a founding document is a stretch of lines all landing at once. A table whose rows have no order cannot answer "and then what", so the database hands out a number in the order the rows arrive.

GENERATED ALWAYS AS IDENTITY rather than a plain default is the part to copy. It means an insert that tries to supply its own entry is refused outright, so there is no road by which a program can put a row in the middle of a history. The index puts entry on the end of tick so that the rows of one tick come back in the order they were written without anybody sorting them.

▣ Build · stage 6 — a birth, and a death somewhere else
-- One row a birth: seven numbers and a genome. The seven are every
-- number anything will ever filter, join, group or climb on. The
-- genome is read back whole or not at all, so it is one column.
--
-- parent and mate are null on a founder, because nought is a creature
-- id and null is not. Nothing here is ever updated.
CREATE TABLE creature (
    id         bigint NOT NULL,
    row        int    NOT NULL,
    species    int    NOT NULL,
    tick       bigint NOT NULL,
    parent     bigint,
    mate       bigint,
    generation int    NOT NULL,
    genome     jsonb  NOT NULL,

    CONSTRAINT creature_key    PRIMARY KEY (id),
    CONSTRAINT creature_parent FOREIGN KEY (parent) REFERENCES creature (id),
    CONSTRAINT creature_mate   FOREIGN KEY (mate)   REFERENCES creature (id),

    -- A creature that is its own parent is a climb that never ends.
    CONSTRAINT nothing_is_its_own_parent CHECK (id <> parent AND id <> mate),
    -- Every birth is paid for by one parent, which may or may not have
    -- had a mate. A second parent with no first one is a row somebody
    -- wrote wrong.
    CONSTRAINT no_second_parent_without_a_first CHECK (parent IS NOT NULL OR mate IS NULL),

    CONSTRAINT creature_born_on_a_real_tick  CHECK (tick >= 0),
    CONSTRAINT creature_generation_counts_up CHECK (generation >= 0)
);

-- The two columns a pedigree is climbed by, and the two a history is
-- grouped and windowed by. row is not indexed and never will be: it
-- holds two distinct values in the whole table.
CREATE INDEX creature_by_parent  ON creature (parent);
CREATE INDEX creature_by_mate    ON creature (mate);
CREATE INDEX creature_by_species ON creature (species);
CREATE INDEX creature_by_tick    ON creature (tick);

Everything above the constraints was argued in bytes two sections ago. The constraints are the part that is free. parent and mate are foreign keys back into the same table, so a child naming a parent no row ever wrote down is impossible; the archive is written in birth order, and a parent always exists before its child does, so the constraint costs one lookup on a key that is already indexed. nothing_is_its_own_parent makes a climb that never ends impossible, in a check the server evaluates once per insert and never again.

no_second_parent_without_a_first is a rule about this world and not about databases: every birth here is paid for by one parent, which may or may not have found a mate. A row with a mate and no parent describes neither a founder nor a child; it is a mistake, and a schema that can name a mistake should.

Nulls are load-bearing in two of those columns. A founder has no parents, and the temptation is to write nought, because nought is what the archive writes and what a Go zero value is. Nought is a creature number. Creature nought does not exist today and a schema that depends on that is a schema with a bomb in it, so a founder's parents are null, which is SQL's own word for a value that is not there.

-- One row a death. A table and not two more columns on creature,
-- because a creature row is written once and never touched again:
-- columns for a death would mean coming back years later to update a
-- row nine kilobytes wide.
--
-- id is the key and not a column beside one, which is the database
-- holding a fact about this world: a creature dies once.
CREATE TABLE death (
    id    bigint NOT NULL,
    tick  bigint NOT NULL,
    cause text   NOT NULL,

    CONSTRAINT death_key           PRIMARY KEY (id),
    CONSTRAINT death_of_a_creature FOREIGN KEY (id) REFERENCES creature (id),
    CONSTRAINT death_on_a_real_tick CHECK (tick >= 0)
);

CREATE INDEX death_by_tick ON death (tick);

The interesting decision is that this is a table. Two more columns on creature, a nullable died_on and a nullable cause, would hold exactly the same facts and would look tidier in a listing. What they would cost is that every birth writes two nulls it has to come back and fill in later, on a row nine kilobytes wide, which means the commonest write in the whole system becomes an update of a row that is ninety-nine percent genome no statement changed.

Split into two tables, every statement this world runs against these two is an insert. A creature row is true the moment it is written and stays true. A death row is written once. Nothing is ever rewritten, which is the same property the archive file had and the reason it was trustworthy, kept while gaining the ability to be asked a question. And id is the primary key and not an ordinary column beside one, which is the database holding one more fact about this world: a creature dies once.

▣ Build · stage 7 — the register, and the two a save takes
-- The species register, and the only table here whose rows change
-- after they are written: closed is filled in on the tick the last
-- member dies, and holder changes hands. number and opened never move.
--
-- closed is null and not nought. In memory the register uses nought
-- for a species that is still standing and gets away with it because
-- nothing ever closed on tick nought; in a database nought is a tick
-- and null is the absence of one.
CREATE TABLE species (
    number int    NOT NULL,
    opened bigint NOT NULL,
    closed bigint,
    holder bigint,

    CONSTRAINT species_key    PRIMARY KEY (number),
    CONSTRAINT species_holder FOREIGN KEY (holder) REFERENCES creature (id),
    CONSTRAINT species_opened_on_a_real_tick CHECK (opened >= 0),
    CONSTRAINT species_closes_after_it_opens CHECK (closed IS NULL OR closed >= opened)
);

This is the one table in the schema whose rows change after they are written, and saying so plainly is better than pretending otherwise. closed is filled in on the tick a species loses its last member, and holder changes hands whenever the register decides it should. number and opened never move.

closed being null and not nought is the small thing on this page most likely to be got wrong by somebody translating the program into a schema. In memory the register uses nought to mean a species that is still standing, and it gets away with it because nothing has ever closed on tick nought. Write that into a column and the world acquires a species that closed at the beginning of time as soon as anything closes on the first tick, which the founding document is about to make a busy one. Null is not a tidier nought. It is the answer to a different question, and "has this species closed" is that question.

-- A saved world, in two tables: one row saying a save happened, and
-- one row for each part of what was saved.
--
-- taken_at is the only wall-clock column in this schema, and it is for
-- whoever is looking at the database. A world's own clock is tick.
CREATE TABLE snapshot (
    world    text        NOT NULL,
    tick     bigint      NOT NULL,
    taken_at timestamptz NOT NULL DEFAULT now(),
    parts    int         NOT NULL,

    CONSTRAINT snapshot_key   PRIMARY KEY (world, tick),
    CONSTRAINT snapshot_world FOREIGN KEY (world) REFERENCES world (world),
    CONSTRAINT snapshot_on_a_real_tick CHECK (tick >= 0),
    CONSTRAINT snapshot_has_parts CHECK (parts > 0)
);

-- One row a component. bytea and not jsonb: a part is whatever the
-- thing that owns it marshalled itself into, and this table hands back
-- exactly the bytes it was given.
CREATE TABLE snapshot_part (
    world text   NOT NULL,
    tick  bigint NOT NULL,
    part  text   NOT NULL,
    bytes bytea  NOT NULL,

    CONSTRAINT snapshot_part_key PRIMARY KEY (world, tick, part),
    CONSTRAINT snapshot_part_of  FOREIGN KEY (world, tick)
        REFERENCES snapshot (world, tick) ON DELETE CASCADE
);

Two tables and one save. The parent row says a save happened at a tick and how many parts it had; the child rows are the parts. ON DELETE CASCADE means that removing an old save takes its parts with it, so there is no way to end up with parts belonging to a save that is not there.

bytes is bytea and not jsonb, which is the column-or-blob rule pushed one step further. A part is whatever the thing that owns it turned itself into, and this table's whole job is to hand back the same bytes it was given. Parsing them as JSON on the way in and rebuilding them on the way out would be two chances to change something in a value whose only requirement is being unchanged.

taken_at is the only wall clock in this schema and the only column in it no run in this book prints. A world's clock is tick, and everything the world reasons about is in ticks; the timestamp is for a person looking at the database wondering when they last saved. It carries now() as a default, which makes it the server's opinion of the time rather than the program's, and that is the correct owner for a fact about the database instead of about the world.

$ go run ./cmd/tables -mode plan
tables: the schema this build holds, and not one statement of it run

  n    file                     sha256, first 16    bytes   stmts   what it makes
  0001 0001_world.sql           55b759d43aa929d4      641       1   world
  0002 0002_chronicle.sql       58878101d9384245      735       2   chronicle
  0003 0003_creature.sql        5f95f200d36d9d2e     1740       5   creature
  0004 0004_death.sql           43845ea86c947233      696       2   death
  0005 0005_species.sql         54029ed637c5b65d      858       1   species
  0006 0006_snapshot.sql        e80d9a5289d0f66c     1183       2   snapshot, snapshot_part

  6 files, 5853 bytes, applied in the order of their numbers and no other
  the ledger, "migration", is not one of them: it has to exist before any of them runs

The whole schema is not quite six kilobytes of text, which is less than one creature. That mode opens no socket: the files are compiled into the binary with go:embed, so a program can say what schema it holds without a database being anywhere near it, and a build whose migrations are numbered wrong fails before it dials anything. The checksums are the other half of that and they come back further down.

Applying migrations twice

Six files in a directory are not a schema until something applies them, and the whole difficulty of applying them is the second time. The first run is easy: connect, send the statements in order, stop. The second run has to look at a database that already holds everything and do nothing at all, without being told which run it is.

The instinct is to make the SQL survive being run twice by putting IF NOT EXISTS on everything, and it fails in a way that takes a year to show up. The worked failure at the end of this section runs it. The answer this world uses is that the database keeps a list of what has been applied to it, and every run consults the list before it sends anything.

▣ Build · stage 8 — a ledger, and the decision it drives
// internal/store/migrate.go
// Ledger is the one table in this database that is not part of the
// world. It is the record of which migrations have been applied, and
// it is the whole of what makes running them twice safe: the second
// run does nothing because the ledger says the first one already did
// it, and not because the SQL was written to survive being run again.
const Ledger = "migration"

// ledger is the only statement in this system that has to be safe to
// run against a database that may or may not already have had it run.
// Everything else is protected by the table it creates; this one is
// the bootstrap, and IF NOT EXISTS is exactly the right tool for a
// statement with nothing to look itself up in.
const ledger = `
CREATE TABLE IF NOT EXISTS ` + Ledger + ` (
    n           int         NOT NULL,
    name        text        NOT NULL,
    sha256      text        NOT NULL,
    applied_at  timestamptz NOT NULL DEFAULT now(),

    CONSTRAINT migration_key PRIMARY KEY (n)
)`

That is an eighth table and it is not part of the world. It belongs to the migration system, and it holds a row for every file that has been applied: the number, the name, the sha256 of the bytes that were applied, and when. It is also the one statement in the system carrying IF NOT EXISTS, and that is consistent instead of an exception. Every other statement is protected by this table. This table has nothing to look itself up in, so it has to be written to survive its own second run.

// internal/store/migrate.go
// Pending is the whole of the decision Migrate makes, with no database
// anywhere in it: given the migrations this build holds and the rows
// the ledger already has, which ones still have to run.
//
// It is a function and not four lines inside Migrate because it is the
// part that can be wrong in a way nobody notices, and a check that
// needs a container to run is a check that gets run less often than
// one that does not.
func Pending(steps []Step, done []Applied) (Report, error) {
	var rep Report
	by := map[int]Applied{}
	for _, a := range done {
		by[a.N] = a
	}
	held := map[int]bool{}
	for _, s := range steps {
		held[s.N] = true
	}
	for _, a := range done {
		if !held[a.N] {
			return rep, fmt.Errorf(
				"store: the database has had %s applied and this build does not hold it: "+
					"a schema built by something newer than the program opening it", a.Name)
		}
	}
	for _, s := range steps {
		if a, ok := by[s.N]; ok {
			if a.Sum != s.Sum {
				return rep, fmt.Errorf(
					"store: %s has changed since it was applied: the ledger holds %s and the file is %s. "+
						"A migration that has run anywhere is finished; a change goes in a new one",
					s.Name, short(a.Sum), s.Short())
			}
			rep.Skipped = append(rep.Skipped, s)
			continue
		}
		rep.Ran = append(rep.Ran, s)
	}
	return rep, nil
}

Four cases, and the checksum is the one that earns its keep. A migration already in the ledger with matching bytes is skipped. One not in the ledger is run. One in the ledger whose file has changed since stops everything, because the number is still 0003, the ledger still says 0003 ran, and the database now holds something no file in the tree describes. Nothing else can see that mistake. The fourth is a ledger row this build has never heard of, which means an older program has been pointed at a newer database, and starting up and writing into it would be the worst thing available.

Pending takes two slices and returns a plan, with no database in its signature anywhere. That is deliberate: the part of a migration runner that goes subtly wrong is the deciding and not the executing, and a check that needs a container to run is a check somebody skips.

$ go test ./internal/store/ -run 'TheMigrationsAreNumbered|TheSchemaIsSevenTables|ASecondRunHasNothing|AHalfMigrated|AMigrationEdited|ASchemaBuiltBySomethingNewer' -v
=== RUN   TestTheMigrationsAreNumberedFromOneWithNoGaps
    migrate_test.go:37: 6 migrations, numbered 1 to 6, every one of them read
--- PASS: TestTheMigrationsAreNumberedFromOneWithNoGaps (0.00s)
=== RUN   TestTheSchemaIsSevenTablesAndNoOthers
    migrate_test.go:73: seven tables, in six files, and not one of them is the ledger
--- PASS: TestTheSchemaIsSevenTablesAndNoOthers (0.00s)
=== RUN   TestASecondRunHasNothingLeftToDo
--- PASS: TestASecondRunHasNothingLeftToDo (0.00s)
=== RUN   TestAHalfMigratedDatabaseRunsOnlyWhatIsLeft
--- PASS: TestAHalfMigratedDatabaseRunsOnlyWhatIsLeft (0.00s)
=== RUN   TestAMigrationEditedAfterItRanIsRefused
--- PASS: TestAMigrationEditedAfterItRanIsRefused (0.00s)
=== RUN   TestASchemaBuiltBySomethingNewerIsRefused
--- PASS: TestASchemaBuiltBySomethingNewerIsRefused (0.00s)
PASS
ok  	theworld/internal/store	0.00s

Six tests of a migration runner and not one of them opens a socket. The second is the odd one: it reads the SQL out of the embedded files, pulls every CREATE TABLE out of it, and fails unless the set is exactly the seven this world has. A schema that quietly grows an eighth table is the drift that stays hidden until somebody asks what a table is for and gets no answer.

▣ Build · stage 9 — one transaction a migration
// internal/store/migrate.go
// Migrate brings a database up to the schema this build holds.
//
// Each migration runs inside its own transaction together with the
// ledger row that records it, so the two cannot come apart: Postgres
// rolls back DDL like anything else, and a migration that fails in its
// fourth statement leaves the database exactly as it was before its
// first, with nothing written in the ledger to say otherwise. A run
// that stops in the middle stops on a migration boundary, which is the
// only place a schema is ever in a state somebody can reason about.
//
// One transaction for the whole run would be tidier and is wrong: a
// long migration on a live database holds locks for as long as the
// slowest thing in the set, and a failure at the end throws away work
// that had nothing wrong with it.
func (d *DB) Migrate(ctx context.Context, steps []Step) (Report, error) {
	var rep Report
	if _, err := d.pool.Exec(ctx, ledger); err != nil {
		return rep, fmt.Errorf("store: the ledger: %w", err)
	}
	done, err := d.Applied(ctx)
	if err != nil {
		return rep, err
	}
	plan, err := Pending(steps, done)
	if err != nil {
		return rep, err
	}
	rep.Skipped = plan.Skipped
	for _, s := range plan.Ran {
		if err := d.apply(ctx, s); err != nil {
			return rep, err
		}
		rep.Ran = append(rep.Ran, s)
	}
	return rep, nil
}

// apply runs one migration and writes its ledger row, both or neither.
func (d *DB) apply(ctx context.Context, s Step) error {
	tx, err := d.pool.Begin(ctx)
	if err != nil {
		return fmt.Errorf("store: %s: %w", s.Name, err)
	}
	defer tx.Rollback(ctx)

	if _, err := tx.Exec(ctx, s.SQL); err != nil {
		return fmt.Errorf("store: %s: %w", s.Name, err)
	}
	const row = `INSERT INTO ` + Ledger + ` (n, name, sha256) VALUES ($1, $2, $3)`
	if _, err := tx.Exec(ctx, row, s.N, s.Name, s.Sum); err != nil {
		return fmt.Errorf("store: %s: recording it: %w", s.Name, err)
	}
	if err := tx.Commit(ctx); err != nil {
		return fmt.Errorf("store: %s: %w", s.Name, err)
	}
	return nil
}

The line to look at is tx.Exec(ctx, s.SQL) sitting in the same transaction as the insert that records it. Postgres rolls back CREATE TABLE the way it rolls back anything else, which is not true of every database and is why this can be done at all. A migration that fails in its fourth statement leaves the database exactly as it was before its first, with no ledger row claiming otherwise, and a run that stops in the middle stops on a migration boundary.

One transaction around the whole run would look tidier and would be worse. A long migration against a live database holds its locks for as long as the slowest statement in the set, and a failure in the last file throws away five files with nothing wrong with them. One transaction a migration is the unit that matches what a migration is.

$ podman exec -w /bench world-go go run ./cmd/tables -mode up -twice
tables: bringing a database up to the schema this build holds

  the first run
    0001 0001_world.sql           applied
    0002 0002_chronicle.sql       applied
    0003 0003_creature.sql        applied
    0004 0004_death.sql           applied
    0005 0005_species.sql         applied
    0006 0006_snapshot.sql        applied
    6 applied, 0 already there, and this run changed the database
    the ledger holds 6 rows, numbered 0001 0002 0003 0004 0005 0006

  the second run
    0001 0001_world.sql           already applied, and its bytes still agree
    0002 0002_chronicle.sql       already applied, and its bytes still agree
    0003 0003_creature.sql        already applied, and its bytes still agree
    0004 0004_death.sql           already applied, and its bytes still agree
    0005 0005_species.sql         already applied, and its bytes still agree
    0006 0006_snapshot.sql        already applied, and its bytes still agree
    0 applied, 6 already there, and this run changed nothing at all
    the ledger holds 6 rows, numbered 0001 0002 0003 0004 0005 0006

That is the idempotence claim in one run instead of two, so there is no question of the second block coming from a different afternoon. Six applied and a ledger holding six rows; then six skipped, none applied, and the same six rows. The second block says the bytes still agree, which is the checksum being checked and not merely stored.

Notice what the second run did not do. It did not ask the server whether the tables exist. It did not send a statement that would have been harmless. It read six rows and decided, and six statements never sent are six statements that cannot go wrong.

⌥ Tool — psql

psql is Postgres's own terminal client, and the only copy of it on this bridge is inside the database container. -c runs one statement and exits; -tAc does the same with the headers and alignment stripped. The backslash commands are psql's own and not SQL: \dt lists tables, \d with a table name prints its columns, indexes and constraints, \di lists indexes, and \? lists the rest. Read a schema back through psql instead of through your own program, because your own program will tell you what it meant to make.

$ podman exec world-db psql -U world -d world -c '\dt'
           List of relations
 Schema |     Name      | Type  | Owner 
--------+---------------+-------+-------
 public | chronicle     | table | world
 public | creature      | table | world
 public | death         | table | world
 public | migration     | table | world
 public | snapshot      | table | world
 public | snapshot_part | table | world
 public | species       | table | world
 public | world         | table | world
(8 rows)

Eight relations for seven tables, and the eighth is the ledger. The page says so out front, because a reader counting tables against the design would otherwise spend a minute working out which one is not in the plan. The ledger belongs to the machinery that made the other seven, and the mode that reads the schema back leaves it out for exactly that reason.

$ podman exec world-db psql -U world -d world -c 'SELECT n, name, left(sha256, 16) AS sha256 FROM migration ORDER BY n;'
 n |        name        |      sha256      
---+--------------------+------------------
 1 | 0001_world.sql     | 55b759d43aa929d4
 2 | 0002_chronicle.sql | 58878101d9384245
 3 | 0003_creature.sql  | 5f95f200d36d9d2e
 4 | 0004_death.sql     | 43845ea86c947233
 5 | 0005_species.sql   | 54029ed637c5b65d
 6 | 0006_snapshot.sql  | e80d9a5289d0f66c
(6 rows)

The same sixteen characters the plan printed off the files. The ledger holds what was applied and the plan reads what is on disk today, and the two agreeing is a fact rather than an assumption. That query carries an ORDER BY and so does every query in this book. A SELECT without one has no order at all: the server may hand the rows over in whatever way was cheapest that morning, and a page quoting one is quoting a decision the planner made. The applied_at column is not selected here and is not selected anywhere, because it holds a wall-clock time and no two runs of this would agree about it.

$ podman exec world-db psql -U world -d world -c '\d creature'
                Table "public.creature"
   Column   |  Type   | Collation | Nullable | Default 
------------+---------+-----------+----------+---------
 id         | bigint  |           | not null | 
 row        | integer |           | not null | 
 species    | integer |           | not null | 
 tick       | bigint  |           | not null | 
 parent     | bigint  |           |          | 
 mate       | bigint  |           |          | 
 generation | integer |           | not null | 
 genome     | jsonb   |           | not null | 
Indexes:
    "creature_key" PRIMARY KEY, btree (id)
    "creature_by_mate" btree (mate)
    "creature_by_parent" btree (parent)
    "creature_by_species" btree (species)
    "creature_by_tick" btree (tick)
Check constraints:
    "creature_born_on_a_real_tick" CHECK (tick >= 0)
    "creature_generation_counts_up" CHECK (generation >= 0)
    "no_second_parent_without_a_first" CHECK (parent IS NOT NULL OR mate IS NULL)
    "nothing_is_its_own_parent" CHECK (id <> parent AND id <> mate)
Foreign-key constraints:
    "creature_mate" FOREIGN KEY (mate) REFERENCES creature(id)
    "creature_parent" FOREIGN KEY (parent) REFERENCES creature(id)
Referenced by:
    TABLE "creature" CONSTRAINT "creature_mate" FOREIGN KEY (mate) REFERENCES creature(id)
    TABLE "creature" CONSTRAINT "creature_parent" FOREIGN KEY (parent) REFERENCES creature(id)
    TABLE "death" CONSTRAINT "death_of_a_creature" FOREIGN KEY (id) REFERENCES creature(id)
    TABLE "species" CONSTRAINT "species_holder" FOREIGN KEY (holder) REFERENCES creature(id)

Eight columns, five indexes, four checks and two self-referencing foreign keys, and then the same table being named by everything else: its own two keys, the death table, and the register's holder. That block is this chapter's design printed by the server instead of claimed by the author, and reading it is how you find out whether a constraint you wrote in a file is a constraint the database is actually keeping.

$ podman exec -w /bench world-go go run ./cmd/tables -mode show
tables: the world, as the server describes it back

  table          column      type                      null  indexes
  chronicle      entry       bigint                    no          2
                 tick        bigint                    no          1
                 kind        text                      no          0
                 text        text                      no          0
  creature       id          bigint                    no          1
                 row         integer                   no          0
                 species     integer                   no          1
                 tick        bigint                    no          1
                 parent      bigint                    yes         1
                 mate        bigint                    yes         1
                 generation  integer                   no          0
                 genome      jsonb                     no          0
  death          id          bigint                    no          1
                 tick        bigint                    no          1
                 cause       text                      no          0
  snapshot       world       text                      no          1
                 tick        bigint                    no          1
                 taken_at    timestamp with time zone  no          0
                 parts       integer                   no          0
  snapshot_part  world       text                      no          1
                 tick        bigint                    no          1
                 part        text                      no          1
                 bytes       bytea                     no          0
  species        number      integer                   no          1
                 opened      bigint                    no          0
                 closed      bigint                    yes         0
                 holder      bigint                    yes         0
  world          world       text                      no          1
                 seed        bigint                    no          0
                 opened      bigint                    no          0
                 genesis     jsonb                     no          0
                 one         boolean                   no          1

  7 tables, 32 columns, and one of the columns is a genome

  rows in each of them
    chronicle               0
    creature                0
    death                   0
    snapshot                0
    snapshot_part           0
    species                 0
    world                   0

  seven tables and no rows: a schema is made before there is a history to put in it

The same schema read out of the catalogue by the program that made it, with the ledger left out and one column added: how many indexes name each column. Five of the creature table's eight are indexed and three are not, and the three are row, generation and genome. The last is this chapter's whole argument, and the first two are the same argument at a smaller size. row holds two distinct values in the entire table, so an index on it can never rule out enough of the table to be used. generation is read in whole bands at a time and never looked up singly.

Then the counts, all seven of them nought, because nothing has happened in this world yet. The last thing to try is three rows the schema ought to refuse.

$ podman exec -w /bench world-go go run ./cmd/tables -mode refuse
tables: three rows the creature table will not take

  a creature that is its own parent
    tried     id 1, parent 1, mate null
    SQLSTATE  23514
    said      new row for relation "creature" violates check constraint "nothing_is_its_own_parent"
    detail    Failing row contains (1, 0, 1, 900, 1, null, 0, {}).

  a creature with a mate and no parent
    tried     id 2, parent null, mate 1
    SQLSTATE  23514
    said      new row for relation "creature" violates check constraint "no_second_parent_without_a_first"
    detail    Failing row contains (2, 0, 1, 900, null, 1, 0, {}).

  a creature descended from one that does not exist
    tried     id 3, parent 99, mate null
    SQLSTATE  23503
    said      insert or update on table "creature" violates foreign key constraint "creature_parent"
    detail    Key (parent)=(99) is not present in table "creature".

  creature holds 0 rows: three statements sent and nothing written

Two of them are refused by check constraints and the third by a foreign key, which is the difference between 23514 and 23503: the first two are rules about one row and the third is a rule about a row against the rest of the table. Every one of the three would otherwise have been a wrong answer years later, in a query no reader thought to doubt, and preventing all three cost twelve lines in a file. The count at the bottom is the part that lets this appear on a page about a schema with no data in it: three statements were sent, all three were refused, and the table is as empty as it was.

⚠ Worked failure — migrations made safe the tempting way

Take the ledger away and send the six files straight at the server, which is what everybody writes on the afternoon the database is empty and running them twice has not come up yet. The database is killed and started first so that it is properly empty: its data directory is a tmpfs, so a restart is a fresh server with nothing in it.

$ podman kill world-db && podman start world-db
world-db
world-db
$ podman exec -w /bench world-go go run ./cmd/tables -mode bare
tables: the same six files, with nothing keeping track of them

  the first run applied all 6 of them and complained about none of them
  the second run stopped on migration 0001
  SQLSTATE 42P07
tables: store: 0001_world.sql: ERROR: relation "world" already exists (SQLSTATE 42P07)
exit status 1

The first run works perfectly and that is the trap. Nothing about the six files is wrong: they make the seven tables and complain about nothing. The second run dies on the first statement it sends, carrying the server's own code for it, 42P07, an object that already exists. Every layer of that error is telling the truth. The server refused to make a table it already has. The runner named the file it was in the middle of. The exit status is one, so a deploy script stops here.

Stopping loudly is the good case. The tempting repair is to make every statement harmless by putting IF NOT EXISTS in front of it, at which point the second run stops complaining and everybody moves on. Here is that repair, against a database somebody has already been at: one where creature was made by an earlier draft of 0003, before the row column was argued into existence.

$ podman kill world-db && podman start world-db
world-db
world-db
$ podman exec -w /bench world-go go run ./cmd/tables -mode bare -loose
tables: the same six files with IF NOT EXISTS on every one of them

  a database somebody has already been at: creature made by an older 0003
    creature holds 7 columns and the schema calls for 8
    the ones it has not got: row
  the first run applied all 6 of them and complained about none of them
  the second run applied all 6 of them and complained about none of them

  and this is the creature table it left behind
    creature holds 7 columns and the schema calls for 8
    the ones it has not got: row

  26 statements, every one of them a success, and the creature table is
  still one column short of the schema that just ran twice

Twenty-six statements, every one a success, exit status nought, and the table is one column short of the schema in the tree. That is the failure. The keyword made each statement safe to run and made the database unknowable at the same time: a statement that succeeded and a statement that was skipped look identical from outside, so the run cannot tell you what it is looking at.

The reasoning runs backwards through what each approach knows. IF NOT EXISTS asks the server one question a statement, "is there something by this name", and a name matching is a long way from a table being right. The ledger asks a different question, "has this file been applied to this database", and the answer is a row somebody wrote at the moment it was true. That is also why the checksum is in the ledger. A name can be right while the bytes behind it are somebody else's; a checksum cannot.

The general form travels well past databases. Idempotence is a property you get by recording that an operation happened and consulting the record, and not one you get by making an operation harmless to repeat. The difference shows up on the day the operation is repeated against something that changed underneath it.

Why questions choose columns

There is one idea under all of this and it generalises past valleys, past Postgres, and past anything with rows in it. When you are deciding how to store something, the useful question is which numbers something will look a record up by, and never which numbers matter most.

A genome is the most important thing about a creature by any reading. It decides what the animal weighs, how far it sees, what a mouthful costs it, whether it strikes. It is also the part no query will ever mention, because no question anybody asks starts with "find me the creatures whose eleventh link weighs more than nought point four". The questions all start with a parent, a tick, a species or a number. Importance and queryability came apart completely, and a schema built on importance would have spread nine thousand numbers across nine thousand columns to make the important thing prominent, and made everything slower to answer a question no page asks.

The second idea is that the cost of an index is a ratio and not an amount. Eight bytes a row is a fixed number; whether it is expensive depends entirely on the row it sits beside. On the creature table it is nine hundredths of a percent, and the four kept beside the key are an afterthought. On a table of two bigints the same index is a fifth of the table, and four of them come to four fifths of it. Both sentences are about the same eight bytes. Anybody who has learned to be sparing with indexes learned it on narrow tables, and the rule they learned does not survive a table with a blob in it.

The third is about nulls, and it came up twice on this page for the same reason. A founder's parent is null and not nought; a standing species has a null closing tick and not nought. Both of those are the program's in-memory habit meeting a database, and the habit is a Go zero value doing double duty as "there isn't one". That works while nothing real ever holds the zero value. Creature nought does not exist and nothing closed on tick nought, so both habits are safe today, and both would become wrong silently rather than loudly. SQL has a word for a value that is not there, and using it costs nothing.

And the fourth is the one the failure box is about. Making an operation harmless to repeat and making it idempotent are different projects. The first is a property of each statement and it can be had with a keyword; the second is a property of the system, and it needs a record of what has happened and something that reads the record before acting. The difference is invisible while nothing has changed underneath, which is to say it is invisible for exactly as long as it takes to matter.

✓ Checkpoint — the seven tables and the ledger
  • Given the measured numbers on this page, work out what fraction of the creature table a fifth index on a bigint column would add, and say why the same index on a table of two bigints is a different decision.
  • Name the four columns of creature that carry an index beside the primary key, and the two integer columns that deliberately do not, with the reason each of the second pair was left out.
  • Say why a death is a row in its own table instead of two nullable columns on creature, in terms of what the commonest write in the system becomes under each design.
  • species.closed is null while a species stands. Describe the bug that appears the first time something closes on tick nought if the column had held nought for "still open" instead.
  • The second run of the migrations sent no statements at all. Say what it read instead, and what the checksum in that record catches that the file name does not.
  • Explain why chronicle.entry is GENERATED ALWAYS rather than a default the caller could override, using the founding document as the example.
⚡ Exercises — try first, then reveal
Exercise 1 — add an eighth table. Write a seventh migration that makes a table of your own, and predict which of the six tests fails and what it prints before you run them.

Put anything you like in migrations/0007_note.sql. The numbering test passes, because 0007 follows 0006 with no gap. The four about the ledger pass, because none of them cares what the migrations make. The one that fails is the one that reads the SQL and counts what it creates.

$ printf 'CREATE TABLE note (id bigint GENERATED ALWAYS AS IDENTITY, text text NOT NULL, CONSTRAINT note_key PRIMARY KEY (id));\n' > migrations/0007_note.sql && go test ./internal/store/ -run TheSchemaIsSevenTablesAndNoOthers -v
=== RUN   TestTheSchemaIsSevenTablesAndNoOthers
    migrate_test.go:54: the migrations make 8 tables, not 7: [world chronicle creature death species snapshot snapshot_part note]
--- FAIL: TestTheSchemaIsSevenTablesAndNoOthers (0.00s)
FAIL
FAIL	theworld/internal/store	0.003s
FAIL

Take the file out again afterwards, or every run of the tests from here on says the same thing. It prints the whole list in migration order, which is the useful thing about failing that way instead of on a count: the reader of the failure can see at a glance which table is the unintended one. A test like this is not there to stop the schema ever growing. It is there so that growing it is a decision somebody makes on purpose, in the same commit that changes the list.

Exercise 2 — edit a migration that has already run. Bring a database up, add a blank line and a comment to 0003_creature.sql, and run the migrations again. Nothing about the schema changed. Predict what happens.

It stops, and it stops even though the change was a comment, because the runner compares the bytes and has no way of knowing that these particular bytes did not matter. That is the right trade. A checker that tried to decide whether an edit was meaningful would have to understand SQL, and would eventually wave through the one edit that was.

$ podman exec -w /bench world-go sh -c 'printf "\n-- one line added to a migration that has already run\n" >> migrations/0003_creature.sql'
$ podman exec -w /bench world-go go run ./cmd/tables -mode up
tables: bringing a database up to the schema this build holds

tables: store: 0003_creature.sql has changed since it was applied: the ledger holds 5f95f200d36d9d2e and the file is 61ee8c586ed16f89. A migration that has run anywhere is finished; a change goes in a new one
exit status 1

Both checksums are in the message, which is what makes it actionable: the first is what the database was built from and the second is what is in your tree, so you can tell at once whether you are looking at your own edit or at somebody else's branch. The cure is never to put the file back; it is to leave the applied file alone and write the change as 0007.

Exercise 3 — find the wall. The widest genome in the twelve-year run held 192 numbers. Make a table with a column for each of them, then make one with 1,601 columns, and find out where Postgres stops.

One hundred and ninety-two columns is fine, which is the uncomfortable part: the design that cannot work is a design the database will let you build.

$ podman exec world-db psql -U world -d world -c "CREATE TABLE narrow ($(seq 0 191 | sed 's/.*/g& float8/' | paste -sd,))"
CREATE TABLE
$ podman exec world-db psql -U world -d world -c "CREATE TABLE wide ($(seq 0 1600 | sed 's/.*/g& float8/' | paste -sd,))"
ERROR:  tables can have at most 1600 columns

Sixteen hundred is the wall and it is a hard one. A controller in this world may grow to five hundred and twelve links, and a link is a weight, a source, a target and an innovation number, so a fully grown genome is a couple of thousand numbers on its own. The table-of-columns design does not get slow at that point. It stops being possible, in the middle of a run that has been going for weeks, and the only warning is a lineage that happened to grow.

Now do the arithmetic the other way. If the genome were columns, every row would be as wide as the widest creature the table has ever held, so a valley of browsers that produces one deep hunter pays for that hunter on every row it has. The blob pays per creature. Work out what that costs at the widths in the table on this page: 1,099 creatures at 168 numbers each against 1,303 rows at 192.

There are seven tables and not a row in any of them. What goes in first is not a creature, because the thing a running world would lose if the machine under it were rebooted is not its history: it is The Hollow itself, the bodies and the stores and the seed bank and every generator part way through its stream. Writing that down means deciding when a world is in a state that can be written down at all, and a tick caught half way through has a view open, a roster part stepped and a generator that has already spent numbers on a creature no snapshot can see yet.