⌂ The World Vol 5 · Bodies and Brains
ch 53 / 105
Chapter 53

Choosing the Squash

The function between layers

The controller has twenty-four inputs, twelve hidden sums and six output sums. Without a function between the rows, the hidden row costs 228 extra numbers and buys the same answers one row could have produced.

One function of one number goes between the hidden row and the output row, and it is judged on two facts: whether it bends, and where its floor and ceiling sit. A straight middle collapses. A bounded middle changes how the row above must be priced.

The far end has a different fault. A step decision can say 1 or 0, so a creature turning on that value asks for the largest turn every time it asks for any turn. The mannerisms work made turning take time; a hard yes throws that away.

Four candidates go through the same test: step, sigmoid, hyperbolic tangent and rectifier. The plots come from their own arithmetic, the two-row gate proves why a bend matters, and the controller keeps tanh in the hidden row while leaving the six output scores bare.

Four candidates, eleven sums

Take one hidden neuron and two ticks before taking anything else. A sensor row holds twenty-four numbers; pick two of them and a neuron that reads them with weights of 1.50 and −1.00 against a bias of −0.20. On a tick when the first sense reads 0.80 and the second −0.50, the sum is 0.80 × 1.50 = 1.20, plus −0.50 × −1.00 = 0.50, less the bias of 0.20, which comes to 1.50. On a tick when the same two senses read −0.40 and 0.60, the sum is −0.60 − 0.60 − 0.20 = −1.40. Two ticks, and 2.90 between them.

That gap is the thing the layer above needs to know about, and each candidate passes a different amount of it upward. The step reports 1 on the first tick and 0 on the second, so all 2.90 of the difference arrives as 1.00 and the same 1.00 would have arrived if the sums had been 0.01 and −0.01. The rectifier reports 1.50 and 0.00: the whole of the first tick's sum survives intact and the whole of the second is erased, which is 1.50 of the 2.90. The two built on e need more than a calculator at this stage, and the program has them.

Each of the four takes a number and returns a number, so all four fit the same one-line signature and none of them needs to know anything about layers, weights or creatures. That signature gets a name, because a network is about to hold one as a field.

▣ Build · stage 1 — the four, and the do-nothing pass
// internal/mind/act.go
package mind

import "math"

// Act is what goes between one row of neurons and the next: one number
// in, one number out, run over every middle total before the row above
// multiplies it. Step already fits this signature. The three below are
// the ones that can answer with something other than yes or no.
type Act func(float64) float64

// Bare hands the total back untouched. It is the pass as it ran before
// this file existed, named so that a table can print it beside the
// others and so the collapse can be run instead of asserted.
func Bare(z float64) float64 { return z }

// Sigmoid squeezes the whole number line into the range 0 to 1,
// crossing 0.5 at zero and never quite reaching either end.
func Sigmoid(z float64) float64 { return 1 / (1 + math.Exp(-z)) }

// Tanh is that same curve stretched to run from -1 to 1 and centred on
// zero, so a neuron with nothing to say says nothing, and one that
// disagrees can push the other way instead of only going quiet.
func Tanh(z float64) float64 { return math.Tanh(z) }

// ReLU passes a positive total through unchanged and flattens
// everything below zero. It has a floor and no ceiling.
func ReLU(z float64) float64 {
	if z < 0 {
		return 0
	}
	return z
}
// cmd/squash/main.go

// squash is one candidate, named for the tables.
type squash struct {
	Name string
	F    mind.Act
}

// The five settings every table below runs. Bare is first because it is
// what the forward pass already had.
var squashes = []squash{
	{"bare", mind.Bare},
	{"step", mind.Step},
	{"sigmoid", mind.Sigmoid},
	{"tanh", mind.Tanh},
	{"relu", mind.ReLU},
}

// four is the four candidates without the do-nothing pass.
var four = squashes[1:]

// head prints one row of column names over the four candidates.
func head(first string) {
	fmt.Printf("  %9s", first)
	for _, s := range four {
		fmt.Printf(" %11s", s.Name)
	}
	fmt.Println()
}

// hand is the two ticks the chapter works with a calculator before any
// of this becomes a table: one hidden neuron with two weights and a
// bias, on a tick with a plant ahead of the creature and a tick with
// the plant behind it.
func hand() {
	w := [2]float64{1.5, -1.0}
	bias := -0.20
	ticks := [2][2]float64{{0.80, -0.50}, {-0.40, 0.60}}

	fmt.Printf("squash: one hidden neuron, weights %.2f and %.2f, bias %.2f\n", w[0], w[1], bias)
	fmt.Printf("  %8s %8s %9s", "sense a", "sense b", "sum")
	for _, s := range four {
		fmt.Printf(" %11s", s.Name)
	}
	fmt.Println()

	var got [2][]float64
	var sum [2]float64
	for t, in := range ticks {
		sum[t] = in[0]*w[0] + in[1]*w[1] + bias
		fmt.Printf("  %8.2f %8.2f %9.4f", in[0], in[1], sum[t])
		for _, s := range four {
			v := s.F(sum[t])
			got[t] = append(got[t], v)
			fmt.Printf(" %11.6f", v)
		}
		fmt.Println()
	}

	fmt.Printf("\n  the two sums differ by %.4f; what each one passes upward:\n", sum[0]-sum[1])
	for i, s := range four {
		fmt.Printf("  %9s %11.6f\n", s.Name, got[0][i]-got[1][i])
	}
}
$ go run ./cmd/squash -mode hand
squash: one hidden neuron, weights 1.50 and -1.00, bias -0.20
   sense a  sense b       sum        step     sigmoid        tanh        relu
      0.80    -0.50    1.5000    1.000000    0.817574    0.905148    1.500000
     -0.40     0.60   -1.4000    0.000000    0.197816   -0.885352    0.000000

  the two sums differ by 2.9000; what each one passes upward:
       step    1.000000
    sigmoid    0.619758
       tanh    1.790500
       relu    1.500000

The two hand sums come back as 1.5000 and −1.4000, and the last block is the part to stare at. Of the 2.9000 that separated the two ticks, the step passes 1.000000, the rectifier 1.500000, the sigmoid 0.619758 and tanh 1.790500. Tanh passes the most because it uses both sides of zero, and the sigmoid passes the least because it spends its whole range between 0 and 1 and both ticks land well inside it. None of the four passes the difference on unchanged, and none of them is meant to: a function that did would be the straight middle the last section ruled out. Widen the sums past a couple of units and the picture changes again.

One mode walks eleven sums, then two far enough out that the bending is over, and finishes by handing each function an infinity, which is the cheapest way there is to ask one where it is allowed to go. 

// cmd/squash/main.go

// curve prints what each candidate answers at eleven sums, then at two
// sums far enough out that three of the four have stopped moving.
func curve() {
	fmt.Println("squash: one number in, one number out")
	head("sum")
	for _, x := range []float64{-4, -2, -1, -0.5, -0.1, 0, 0.1, 0.5, 1, 2, 4} {
		fmt.Printf("  %9.2f", x)
		for _, s := range four {
			fmt.Printf(" %11.6f", s.F(x))
		}
		fmt.Println()
	}
	fmt.Println()
	head("far out")
	for _, x := range []float64{-20, 20} {
		fmt.Printf("  %9.2f", x)
		for _, s := range four {
			fmt.Printf(" %11.6f", s.F(x))
		}
		fmt.Println()
	}
	fmt.Printf("  the floor and the ceiling each one is walled in by:\n")
	for _, s := range four {
		lo, hi := s.F(math.Inf(-1)), s.F(math.Inf(1))
		fmt.Printf("  %9s %11.4f %11.4f\n", s.Name, lo, hi)
	}
}
$ go run ./cmd/squash -mode curve
squash: one number in, one number out
        sum        step     sigmoid        tanh        relu
      -4.00    0.000000    0.017986   -0.999329    0.000000
      -2.00    0.000000    0.119203   -0.964028    0.000000
      -1.00    0.000000    0.268941   -0.761594    0.000000
      -0.50    0.000000    0.377541   -0.462117    0.000000
      -0.10    0.000000    0.475021   -0.099668    0.000000
       0.00    0.000000    0.500000    0.000000    0.000000
       0.10    1.000000    0.524979    0.099668    0.100000
       0.50    1.000000    0.622459    0.462117    0.500000
       1.00    1.000000    0.731059    0.761594    1.000000
       2.00    1.000000    0.880797    0.964028    2.000000
       4.00    1.000000    0.982014    0.999329    4.000000

    far out        step     sigmoid        tanh        relu
     -20.00    0.000000    0.000000   -1.000000    0.000000
      20.00    1.000000    1.000000    1.000000   20.000000
  the floor and the ceiling each one is walled in by:
       step      0.0000      1.0000
    sigmoid      0.0000      1.0000
       tanh     -1.0000      1.0000
       relu      0.0000        +Inf

Read down the columns and the differences are already plain. The step column holds two values and no others, and it takes the jump between them just above zero, so the 0.00 row reads 0.000000 and the 0.10 row reads 1.000000. The sigmoid column moves on every row and never leaves the range 0 to 1, but it is centred on 0.5, so a neuron with a sum of zero still hands 0.5 up to the layer above. Tanh moves on every row too and answers 0.000000 at a sum of zero. The rectifier is two straight pieces hinged at the origin, and its column is the only one whose bottom row and top row differ by more than 1.

The last block is the important one. Handing each function an infinity asks it where it can go, and three of the four answer with a pair of ordinary numbers. The rectifier answers +Inf. Every other line in this chapter follows from that table.

∑ Math Interlude — the four in symbols, after the numbers

Everything above was arithmetic on eleven sums. Here is the same arithmetic written the way it appears everywhere else, which is useful only because you will meet these four written this way for the rest of your life. Call the number arriving from the weighted sum x.

step(x) = 1 if x > 0, else 0

Zero is not above zero, so step(0) is 0. That is the 0.00 row of the table above, and the 0.00 bearing of the table further down.

One new symbol does the rest of the work. e is a fixed number, 2.718281828..., and e−x means that number multiplied by itself −x times, which for a negative exponent means divided instead. At x = 0 it is 1, because anything to the power 0 is 1. At x = 1 it is 1 ÷ 2.71828 = 0.36788. At x = −1 it is 2.71828 itself. It falls toward 0 as x grows and climbs without limit as x goes negative, and in Go it is math.Exp(-x).

σ(x) = 1 ÷ (1 + e−x)

Check it at zero: 1 ÷ (1 + 1) = 0.5, the middle row of the table. Check it at 2: e−2 = 0.13534, and 1 ÷ 1.13534 = 0.880797, which is what the sigmoid column printed. The denominator is always larger than 1 and always finite, so the answer is always between 0 and 1 and never either.

tanh(x) = (ex − e−x) ÷ (ex + e−x)

The top and the bottom differ by one sign. At x = 0 they are 1 − 1 = 0 over 1 + 1 = 2, so tanh(0) = 0. As x grows, e−x shrinks toward nothing and the fraction approaches ex ÷ ex = 1. Going the other way the same argument gives −1. It is the sigmoid doubled and slid down: tanh(x) = 2σ(2x) − 1, which you can check at x = 1, where 2 × 0.880797 − 1 = 0.761594.

relu(x) = max(0, x)

max(a, b) is whichever of the two is larger, and it is one if in code. The name is short for rectified linear unit, which describes a straight line with the negative half cut off.

xthe number arriving from a neuron's weighted sum, any size, either sign
step(x)1 when x is above zero, 0 when it is not; zero itself counts as not above
ethe fixed number 2.718281828..., the base these two curves are built on
e−xe to the power −x: 1 at x = 0, 0.36788 at x = 1, 2.71828 at x = −1; math.Exp(-x)
σ(x)said "sigma of x", the sigmoid; runs 0 to 1 and crosses 0.5 at zero
tanh(x)said "tanch of x", the hyperbolic tangent; runs −1 to 1 and crosses 0 at zero
max(a, b)whichever of the two numbers is larger
relu(x)max(0, x): zero below the origin, the number itself above it
◆ Note — the name, and the ones left off the list

These four are called activation functions, from the days when a neuron was described as firing or not firing; the older word for the same slot is the transfer function. There are plenty more. A leaky rectifier returns 0.01x below the origin instead of 0, so a neuron that has gone negative can still be moved. Smoother relatives of the rectifier turn up in very large models, where the gentler bend matters to the training. And softmax is different in kind: it takes a whole row at once and returns numbers between 0 and 1 that add up to exactly 1, for when a row is meant to be read as a set of probabilities. Nothing in this book reads a row that way, so softmax has no work here. The four on the list are the ones whose behaviour can be predicted from their arithmetic before a network is built, and prediction is the whole exercise: no creature in this world is ever trained, so every property has to be right by construction.

Eleven rows of a table is a thin way to meet a curve. The renderer already draws straight lines between whole pixels and hashes what it drew, so the four can be plotted from the same functions the table called.

▣ Build · stage 2 — one panel per candidate, drawn by arithmetic
// cmd/squash/plot.go

// The picture: four square panels in a row, each one plotting a
// candidate across the same sums and against the same zero line, with
// its name under it. Every measurement is odd or even so that zero
// lands on a whole pixel instead of between two.
const (
	Panel = 65 // one panel, square, with a middle column and a middle row
	Half  = Panel / 2
	Gap   = 4  // the margin round the strip and the gutter between panels
	Lab   = 8  // the row of glyphs under each panel
	PerX  = 8  // pixels across per unit of sum, so the panel runs -4 to 4
	PerY  = 25 // pixels down per unit of answer, so the panel runs -1.28 to 1.28
)

// The four palette entries the picture is drawn in.
const (
	ground render.Color = 0xFF181A29 // the panel a curve is drawn on
	rule   render.Color = 0xFF323847 // the two axes
	ink    render.Color = 0xFFF0BE35 // the curve itself
	tag    render.Color = 0xFF8A979E // the name under the panel
)

// col is the pixel column a sum lands in, and row the pixel row an
// answer lands in. Both are measured from the panel's own corner.
func col(x float64) int { return Half + int(math.Round(x*PerX)) }
func row(y float64) int { return Half - int(math.Round(y*PerY)) }

// panel draws one candidate: the ground, the two axes, then a line
// from every column to the next. The clip is the panel, so a curve
// that leaves the top is cut at the edge instead of scribbling on its
// neighbour.
func panel(b *render.Buffer, ox, oy int, s squash) (int, int) {
	b.SetClip(render.Rect{X0: ox, Y0: oy, X1: ox + Panel, Y1: oy + Panel})
	b.FillRect(render.Rect{X0: ox, Y0: oy, X1: ox + Panel, Y1: oy + Panel}, ground)
	b.Line(ox, oy+row(0), ox+Panel-1, oy+row(0), rule)
	b.Line(ox+col(0), oy, ox+col(0), oy+Panel-1, rule)

	inside, off := 0, 0
	px, py := 0, 0
	for c := range Panel {
		x := float64(c-Half) / PerX
		r := row(s.F(x))
		if r >= 0 && r < Panel {
			inside++
		} else {
			off++
		}
		if c > 0 {
			b.Line(ox+px, oy+py, ox+c, oy+r, ink)
		}
		px, py = c, r
	}
	b.SetClip(b.Bounds())
	return inside, off
}
$ go run ./cmd/squash -mode plot -shot assets/frames/squash-four.png
squash: four panels of 65 by 65, 280 by 83 altogether
  each panel runs -4.00 to 4.00 across at 8 pixels a unit,
  and -1.28 to 1.28 down at 25 pixels a unit
             plotted  off top  its own panel
       step       65        0  f610be2be6be9bff
    sigmoid       65        0  815dbad914ebb81d
       tanh       65        0  0feb7ea8172f9bbf
       relu       43       22  8a02dee3c856f042
  the strip ea19ca6ca2d15f964dd9ad3af99683c708c3c520d27e5b272bd14eb463780d90
  wrote assets/frames/squash-four.png at 3 times, 840 by 249
squash-four.png (PNG, 840 × 249; four 3× plots)

A panel is 65 pixels square so that there is a middle column and a middle row for the two axes to sit on: 32 pixels either side of zero, 8 pixels for every unit across and 25 for every unit down. Sixty-five columns are sampled, one per pixel, and each is joined to the last with the same line routine the plants are drawn with. The count beside each name is how many of those sixty-five landed inside the panel. Three of the four scored 65. The rectifier scored 43, with 22 of its columns off the top, and it did not have to be told to stop: the clip was set to the panel, so the twenty-two columns whose answer landed above the frame were cut by the same check that keeps a sprite off its neighbour.

Four dark square panels in a row on a near-black strip, each with a faint horizontal axis across its middle and a faint vertical axis down its centre, and a gold curve drawn on it. The first, labelled STEP, is flat along the axis on the left then jumps straight up and runs flat along the top half. The second, SIGMOID, rises in a smooth S from the axis line on the left to near the top on the right, never dipping below that line. The third, TANH, is a steeper S centred on the crossing point, running from low in the panel to high in it. The fourth, RELU, is flat along the axis on the left and then rises as a straight diagonal that leaves the top edge of the panel about two thirds of the way across.
Figure 53.1 — assets/frames/squash-four.png: the four candidates over the same sums, from −4 on the left to 4 on the right, each on axes crossing at the origin. Sigmoid never dips below its panel's centre line, and tanh crosses that line where the two axes meet: the 0.500000 and 0.000000 entries of the table's middle row, drawn. The rectifier's diagonal running off the top is the picture of +Inf.

The amount passed upward

The forward pass has to be told which of these to use, and there is exactly one place in it where the question arises. The pass gains a field and one call.

▣ Build · stage 3 — the field the pass gains, and the loop that uses it
// internal/mind/net.go — the two edits this chapter makes to the pass

type Net struct {
	In     int // numbers the network is handed
	Hid    int // neurons in the middle row
	Out    int // numbers it hands back
	Squash Act // run over every middle total before the output row reads it
	W      []float64
}

// Forward runs in through both rows and writes the answer into out. hid
// is the caller's scratch for the middle row. Nothing here allocates:
// whoever owns the network makes its two buffers once and keeps them.
//
// Between the two rows sits Squash, run over every middle total in
// place. A network never handed one runs the way this pass ran before
// there was anything to hand it: the middle totals go straight on.
func (n Net) Forward(in, hid, out []float64) {
	Layer(in, n.W[n.HidW():n.HidB()], n.W[n.HidB():n.OutW()], hid)
	if n.Squash != nil {
		for j, z := range hid {
			hid[j] = n.Squash(z)
		}
	}
	Layer(hid, n.W[n.OutW():n.OutB()], n.W[n.OutB():], out)
}

One field and one loop, and nothing else in the package moved. Layer is untouched, so the arithmetic that was checked against a pencil is the same arithmetic. The four offset methods are untouched, so the weight layout is the same layout and Weights(24, 12, 6) is still 378. The second Layer call is untouched, with nothing at all between its totals and out. And the nil check earns its two lines: every network built before this chapter still runs, and it still produces the numbers it produced, because a Net with no Squash skips the loop entirely. A network handed mind.Bare takes the loop and comes out at exactly the same numbers, which is what lets the collapse be run beside the alternatives instead of argued about.

Now put one neuron in front of a real question. A creature's rays report where the nearest plant sits off its heading, scaled to −1 for hard left and 1 for hard right the way every entry in the sensor row is scaled. One neuron reads that number with a weight of 4 and no bias, and its answer is read as how much of a turn the creature commits to. Walk the bearing across its whole range and watch what each candidate is capable of saying.

▣ Build · stage 4 — the same neuron, four ways
// cmd/squash/main.go

// bearing is one neuron reading one number: where the nearest plant
// sits off the creature's heading, scaled to -1 hard left and 1 hard
// right, weighted by gain and squashed. The answer is read as how much
// of a turn the creature commits to.
func bearing(gain float64, steps int) {
	fmt.Printf("squash: one neuron, weight %.1f, bias 0, reading a bearing from -1 to 1\n", gain)
	head("bearing")
	for i := -4; i <= 4; i++ {
		b := float64(i) / 4
		fmt.Printf("  %9.2f", b)
		for _, s := range four {
			fmt.Printf(" %11.6f", s.F(gain*b))
		}
		fmt.Println()
	}
	fmt.Printf("\n  different answers across %d bearings evenly spaced from -1 to 1:\n", steps+1)
	for _, s := range four {
		seen := map[float64]bool{}
		for i := 0; i <= steps; i++ {
			seen[s.F(gain*(-1+2*float64(i)/float64(steps)))] = true
		}
		fmt.Printf("  %9s %6d\n", s.Name, len(seen))
	}
}
$ go run ./cmd/squash -mode bearing
squash: one neuron, weight 4.0, bias 0, reading a bearing from -1 to 1
    bearing        step     sigmoid        tanh        relu
      -1.00    0.000000    0.017986   -0.999329    0.000000
      -0.75    0.000000    0.047426   -0.995055    0.000000
      -0.50    0.000000    0.119203   -0.964028    0.000000
      -0.25    0.000000    0.268941   -0.761594    0.000000
       0.00    0.000000    0.500000    0.000000    0.000000
       0.25    1.000000    0.731059    0.761594    1.000000
       0.50    1.000000    0.880797    0.964028    2.000000
       0.75    1.000000    0.952574    0.995055    3.000000
       1.00    1.000000    0.982014    0.999329    4.000000

  different answers across 201 bearings evenly spaced from -1 to 1:
       step      2
    sigmoid    201
       tanh    201
       relu    101

Two hundred and one bearings, and the step neuron produces two answers. Every plant anywhere to the left gets the same 0.000000 and every plant anywhere to the right gets the same 1.000000, so a creature wired this way turns as hard for a plant one degree off its nose as for one at the edge of its vision, and then turns exactly as hard again on the next tick. Sigmoid and tanh give a different answer for every one of the two hundred and one. The rectifier gives 101, because it answers 0.000000 for every bearing on the left and a different number for each on the right: half a creature's world flattened into one reply.

Tanh's column is the one to look at closely. It reads 0.000000 at a bearing of zero, ±0.761594 a quarter of the way out, and ±0.999329 at the edges. A plant straight ahead buys no turn at all, a plant slightly off buys a small one, a plant far round the side buys nearly the whole turn on offer. That is a controller saying how much. Sigmoid says how much too, but it says 0.500000 for the plant straight ahead, so a creature reading that column as a turn spends half of every tick's turning on a plant it is already pointed at.

Where the one function sits in a single forward pass Five boxes in a row, joined left to right by gold arrows. The first holds the sensor row, 24 numbers, ranged minus 1 to 1. The second holds the hidden sums, 12 numbers, ranged anything. The third, highlighted in gold, holds the squash, tanh, and is annotated below as the only one in the pass. The fourth holds the hidden row, 12 numbers, ranged minus 1 to 1. The fifth holds the output sums, 6 numbers, ranged anything, and is annotated below as left bare. A footer says the picture covers one tick of one creature. WHERE THE ONE FUNCTION SITS IN ONE PASS sensor row 24 numbers -1 to 1 hidden sums 12 numbers anything the squash tanh 12 calls hidden row 12 numbers -1 to 1 output sums 6 numbers anything the only one in the pass left bare across: one tick of one creature
Figure 53.2 — the 288 hidden weights turn the first box into the second and the 72 output weights turn the fourth into the fifth; the one function is the middle box. Both boxes marked anything hold weighted sums, which can come to any size at all, and the box between them is what puts a range back.

The two-input gate

Bending the middle was supposed to make the second row earn the 228 numbers it costs. Testing that needs a question one row of weights provably cannot answer, and the smallest one there is takes two numbers, each either 0 or 1, and asks whether exactly one of them is on. Two inputs, four cases, one answer.

Wire it by hand. One hidden neuron adds the two inputs, multiplies by a gain of 10 and subtracts 5, so its sum is above zero when at least one input is on. The other does the same with 15 subtracted, so its sum is above zero only when both are. The output neuron wants the first without the second, and it is priced twice, for a reason the run is about to make obvious.

▣ Build · stage 5 — nine weights, two prices
// cmd/squash/main.go

// The question: is exactly one of these two numbers on. The four cases,
// and what the answer should be.
var (
	cases = [4][2]float64{{0, 0}, {0, 1}, {1, 0}, {1, 1}}
	want  = [4]bool{false, true, true, false}
)

// hiddenRow is the row both weight sets share: one neuron whose sum is
// above zero when at least one input is on, one whose sum is above zero
// only when both are. The gain g is how hard both of them are driven,
// and the two biases are half of it and one and a half of it, so the
// question stays the same question at any gain.
func hiddenRow(g float64) []float64 {
	return []float64{g, g, g, g, -g / 2, -1.5 * g}
}

// The output row is priced twice, because the hidden row's off state is
// not the same number in both. Under step, sigmoid and relu an idle
// neuron hands up 0 and the price of the second one is -3. Under tanh
// an idle neuron hands up -1, and -3 of that is +3.
func offZero(g float64) []float64  { return append(hiddenRow(g), 1, -3, -0.5) }
func offMinus(g float64) []float64 { return append(hiddenRow(g), 1, -1, -1) }

// gate runs the two-layer net over the four cases under one candidate
// and reports how many of the four it got right.
func gate(s squash, w []float64) int {
	n := mind.Net{In: 2, Hid: 2, Out: 1, Squash: s.F, W: w}
	hid, out := n.Buffers()
	fmt.Printf("  %-8s %9s %9s %9s %9s %9s   %s\n",
		s.Name, "in x", "in y", "hidden 1", "hidden 2", "out", "says")
	right := 0
	for i, c := range cases {
		in := []float64{c[0], c[1]}
		n.Forward(in, hid, out)
		yes, said, mark := out[0] > 0, "no", "wrong"
		if yes {
			said = "yes"
		}
		if yes == want[i] {
			right++
			mark = "right"
		}
		fmt.Printf("  %-8s %9.2f %9.2f %9.4f %9.4f %9.4f   %-3s %s\n",
			"", c[0], c[1], hid[0], hid[1], out[0], said, mark)
	}
	fmt.Printf("  %-8s %d of 4\n\n", "", right)
	return right
}
$ go run ./cmd/squash -mode gate
squash: is exactly one of the two on? a 2-2-1 net, 9 weights
  hidden weights [10 10 10 10], hidden biases [-5 -15]
  output row priced for an off state of 0: weights [1 -3], bias [-0.5]
  bare          in x      in y  hidden 1  hidden 2       out   says
                0.00      0.00   -5.0000  -15.0000   39.5000   yes wrong
                0.00      1.00    5.0000   -5.0000   19.5000   yes right
                1.00      0.00    5.0000   -5.0000   19.5000   yes right
                1.00      1.00   15.0000    5.0000   -0.5000   no  right
           3 of 4

  step          in x      in y  hidden 1  hidden 2       out   says
                0.00      0.00    0.0000    0.0000   -0.5000   no  right
                0.00      1.00    1.0000    0.0000    0.5000   yes right
                1.00      0.00    1.0000    0.0000    0.5000   yes right
                1.00      1.00    1.0000    1.0000   -2.5000   no  right
           4 of 4

  sigmoid       in x      in y  hidden 1  hidden 2       out   says
                0.00      0.00    0.0067    0.0000   -0.4933   no  right
                0.00      1.00    0.9933    0.0067    0.4732   yes right
                1.00      0.00    0.9933    0.0067    0.4732   yes right
                1.00      1.00    1.0000    0.9933   -2.4799   no  right
           4 of 4

  relu          in x      in y  hidden 1  hidden 2       out   says
                0.00      0.00    0.0000    0.0000   -0.5000   no  right
                0.00      1.00    5.0000    0.0000    4.5000   yes right
                1.00      0.00    5.0000    0.0000    4.5000   yes right
                1.00      1.00   15.0000    5.0000   -0.5000   no  right
           4 of 4

  the same row under tanh, whose off state is -1 instead of 0
  tanh          in x      in y  hidden 1  hidden 2       out   says
                0.00      0.00   -0.9999   -1.0000    1.5001   yes wrong
                0.00      1.00    0.9999   -0.9999    3.4996   yes right
                1.00      0.00    0.9999   -0.9999    3.4996   yes right
                1.00      1.00    1.0000    0.9999   -2.4997   no  right
           3 of 4

  output row re-priced for an off state of -1: weights [1 -1], bias [-1]
  tanh          in x      in y  hidden 1  hidden 2       out   says
                0.00      0.00   -0.9999   -1.0000   -0.9999   no  right
                0.00      1.00    0.9999   -0.9999    0.9998   yes right
                1.00      0.00    0.9999   -0.9999    0.9998   yes right
                1.00      1.00    1.0000    0.9999   -0.9999   no  right
           4 of 4

Start with the bare block, because it fails in a way that generalizes. Its out column reads 39.5000, 19.5000, 19.5000, −0.5000. Work the algebra and there is no mystery: hidden 1 is 10(x+y) − 5, hidden 2 is 10(x+y) − 15, and the output is hidden 1 minus three times hidden 2 minus a half, which comes to −20(x+y) + 39.5. The whole network is a straight line in one quantity, x+y, and a straight line crosses zero once. It can answer "at least one" or "both". Asking it to say yes at x+y = 1 and no on either side of that is asking a line to bend, and no choice of the nine weights will do it: three of four is the ceiling, and it stays the ceiling if you add nine more layers of the same kind.

Step, sigmoid and the rectifier all get four of four out of the identical weights. The hidden pair now answers on the two sides of its own threshold instead of running away with the magnitude, so the output row sees 0 and 0, then 1 and 0, then 1 and 1, and a straight line through those three points is easy. The three of them differ in how they get there. Step hands up exact zeros and ones. Sigmoid hands up 0.0067 and 0.9933, close enough that the same output weights work with the margin barely moved: 0.4732 where step gave 0.5000. The rectifier hands up 5.0000 and 15.0000, the raw magnitudes, and the output row happens to be priced to cancel them.

Then tanh, on the same nine weights, drops back to three of four, and its wrong row is the first one: two inputs off, both hidden neurons idle, and the answer comes out 1.5001. The reason is in the hidden columns. Under the other three an idle neuron hands up 0 and contributes nothing at all to the sum above it. Under tanh an idle neuron hands up −1, which is as loud as an active one and points the other way, so the output weight of −3 that was subtracting an active neuron is now adding an idle one: −3 × −1 = +3. Re-price that row for the off state it actually has, weights 1 and −1 against a bias of −1, and tanh answers four of four with a comfortable margin either side.

That is the second half of the rule, in numbers. A squash decides not only whether the layer above it can express something, but what the layer above it costs to set: the same nine weights mean different things under two functions whose graphs look almost the same.

The idle hidden row

One question is left. If a function in the middle is this useful, why not put one over the output row too? The obvious first wiring uses one candidate everywhere, which is fewer decisions and one constant to change. Six output sums come off a creature's controller on an ordinary tick, and the highest of the six is the action it takes.

⚠ Worked failure — one function everywhere, and a valley that sat still
// cmd/squash/main.go

// The six numbers one creature's output row came out with on one tick.
// Every one is negative, which is an ordinary tick: nothing in reach is
// worth much, and the least bad thing to do is walk.
var actions = []string{"rest", "walk", "sprint", "turn left", "turn right", "bite"}

// best is the action a creature takes: the highest score, and the
// earliest in table order when two of them are equal.
func best(s []float64) int {
	at := 0
	for i, v := range s {
		if v > s[at] {
			at = i
		}
	}
	return at
}

// pick prints the six scores as each candidate would leave them, and
// the action each choice ends in.
func pick(scores []float64) {
	fmt.Println("squash: six scores off a bare output row, and the action each squash picks")
	fmt.Printf("  %-11s", "action")
	for _, s := range squashes {
		fmt.Printf(" %11s", s.Name)
	}
	fmt.Println()
	cols := make([][]float64, len(squashes))
	for j, s := range squashes {
		cols[j] = make([]float64, len(scores))
		for i, v := range scores {
			cols[j][i] = s.F(v)
		}
	}
	for i, name := range actions {
		fmt.Printf("  %-11s", name)
		for j := range squashes {
			fmt.Printf(" %11.6f", cols[j][i])
		}
		fmt.Println()
	}
	fmt.Printf("  %-11s", "it does")
	for j := range squashes {
		fmt.Printf(" %11s", actions[best(cols[j])])
	}
	fmt.Println()
}
$ go run ./cmd/squash -mode pick
squash: six scores off a bare output row, and the action each squash picks
  action             bare        step     sigmoid        tanh        relu
  rest          -1.420000    0.000000    0.194662   -0.889599    0.000000
  walk          -0.350000    0.000000    0.413382   -0.336376    0.000000
  sprint        -2.100000    0.000000    0.109097   -0.970452    0.000000
  turn left     -0.880000    0.000000    0.293178   -0.706419    0.000000
  turn right    -0.610000    0.000000    0.352059   -0.544127    0.000000
  bite          -3.050000    0.000000    0.045217   -0.995524    0.000000
  it does            walk        rest        walk        walk        rest

The symptom in the running valley was that creatures with the rectifier over both rows did nothing. Not sometimes: every creature, every tick, resting until its store ran out and it fell over. The scores were printed to find out which action was scoring so high, and every score was 0.000000, which is a strange thing for twelve weighted sums to agree on.

The bare column says why. All six sums are negative, which is an ordinary tick rather than a broken one: nothing in reach is much use, and the least bad of six poor options is to walk. The rectifier flattens every negative number to exactly zero, so the six sums that were ranked −0.35, −0.61, −0.88, −1.42, −2.10, −3.05 all become 0.000000 and the ranking is gone. best is left with six equal numbers and returns the first, which is rest. Step does the same thing for the same reason, from the other side: it flattens everything negative to 0 as well.

Sigmoid and tanh keep the ordering and pick walk, the same action the bare row picks, and that is the argument against them as well. Both are increasing functions, so applying either to all six leaves them in the same order they were already in, and the answer cannot change. Six calls to math.Exp a creature a tick, bought to produce a result already sitting in the sums. There is a way for them to change the answer, and it is worse than doing nothing: tanh returns exactly 1.0 for every input above about 19, so two large sums that were ranked can come back tied, and the tie goes to whichever action the table lists first.

The rule underneath is small. A monotonic function applied to every candidate in a comparison either changes nothing or destroys information, and never adds any. The six numbers are only ever compared with each other, never added to something, never handed to another layer, never read as a probability. So the output row is left bare, and the only function in the whole pass is the one on the hidden row.

Why the bend belongs in the middle

Strip the creature out and two independent properties have been under test all chapter. One is whether the function bends. Any bend at all, anywhere, is enough to stop two layers collapsing into one, and that is the entire job of the middle of a network: the step, the two S-curves and the hinge all did it, using four quite different bends. The other is where the function's floor and ceiling sit. That decides what the weights above it have to be set to, whether an idle neuron is silent or shouting, and whether one loud neuron can drown out the others.

The pattern reaches past networks. Chain any number of stages that each scale their input by a constant, and the chain has one gain overall: adding stages buys nothing, and the only cure is a stage that does something other than scale. Put a limiter in the chain and two things happen at once. The stages downstream can now do work no arrangement of the upstream ones could, and the limiter's floor and ceiling become facts every downstream stage has to be designed around. That second half is the one people forget, and it is the same lesson the plants taught with their ceiling: a bound nobody typed still ends up in every number computed after it.

Those two are why the valley's controller runs tanh through its hidden row. It bends. Its floor and ceiling are −1 and 1, so twelve hidden neurons all contribute on the same scale and none of them can dominate the eleven others whatever its sum came to. It is centred on zero, so a neuron with nothing to say contributes nothing, and a neuron can disagree by going negative instead of only by going quiet. And it answers a different number for every input, so a controller reading it can say a little left.

The rectifier is the interesting one to reject, because in most of this field it is the default. Its case is speed and its case is good: a comparison instead of an exponential. What it lacks is the ceiling, and this book never trains a network, so nothing will ever come along and scale a runaway weight back down. Ask what one hidden neuron can hand up in the worst case and the arithmetic is short, and the mode that prints it totals the pass on the way past, with a wrapper that tallies the calls as they happen.

▣ Build · stage 6 — what a pass costs, counted rather than guessed
// cmd/squash/main.go

// counted wraps a candidate in a tally, so what a pass costs is read
// off a run instead of counted on paper and hoped for.
func counted(f mind.Act, n *int) mind.Act {
	return func(x float64) float64 {
		*n++
		return f(x)
	}
}

// cost prints what one forward pass of the valley's controller comes
// to, and what a valley of them comes to.
func cost(creatures int) {
	calls := 0
	n := mind.Net{In: 24, Hid: 12, Out: 6, Squash: counted(mind.Tanh, &calls)}
	size := mind.Weights(n.In, n.Hid, n.Out)
	n.W = make([]float64, size)
	in := make([]float64, n.In)
	hid, out := n.Buffers()
	n.Forward(in, hid, out)

	muls := n.In*n.Hid + n.Hid*n.Out
	adds := muls + n.Hid + n.Out
	fmt.Printf("squash: one pass of a %d-%d-%d net, and %d of them\n", n.In, n.Hid, n.Out, creatures)
	fmt.Printf("  %-22s %10d %14d\n", "weights", size, size*creatures)
	fmt.Printf("  %-22s %10d %14d\n", "multiplies", muls, muls*creatures)
	fmt.Printf("  %-22s %10d %14d\n", "additions", adds, adds*creatures)
	fmt.Printf("  %-22s %10d %14d\n", "squash calls", calls, calls*creatures)
	fmt.Printf("  one squash call for every %.1f multiplies\n", float64(muls)/float64(calls))

	worst := float64(n.In) + 1
	fmt.Printf("\n  every input at 1 and every weight at 1 puts %.1f into one hidden neuron:\n", worst)
	for _, s := range four {
		fmt.Printf("  %9s %11.4f\n", s.Name, s.F(worst))
	}
}
$ go run ./cmd/squash -mode cost
squash: one pass of a 24-12-6 net, and 3000 of them
  weights                       378        1134000
  multiplies                    360        1080000
  additions                     378        1134000
  squash calls                   12          36000
  one squash call for every 30.0 multiplies

  every input at 1 and every weight at 1 puts 25.0 into one hidden neuron:
       step      1.0000
    sigmoid      1.0000
       tanh      1.0000
       relu     25.0000

Twenty-four inputs at their maximum of 1, twenty-four weights at 1, plus a bias of 1, comes to 25.0 arriving at one hidden neuron. Three of the four hand up 1.0000 and the rectifier hands up 25.0000, twenty-five times what a neuron of ordinary opinion contributes, on weights nobody chose deliberately. The output row would then be reading one number and eleven rounding errors.

The cost of the choice is on the lines above it. A pass of the valley's controller is 360 multiplies, 378 additions and 12 calls to tanh: one call for every thirty multiplies. Three thousand creatures put that at 1,080,000 multiplies and 36,000 calls in a tick. Half a tick at ten ticks a second is fifty milliseconds, and those are the numbers the second half of that budget has to be argued from.

✓ Checkpoint — four functions, and where each belongs
  • Given any sum, produce all four answers to six decimal places with a calculator, including the two that need e to a power, and check them against the -mode curve table.
  • Say which of the four have a ceiling and which have a floor, and get the program to tell me by handing each one an infinity.
  • Shown that a bare hidden row reduces this gate to −20(x+y) + 39.5, explain why no output row over it reaches four of four, and why adding layers of the same kind does not help.
  • Handed an output row priced for a hidden row that idles at 0, re-price it for one that idles at −1, and predict the new numbers before running it.
  • Given six output sums, say which squashes over them would change the chosen action and which cannot, without running anything.
  • For a net of any three sizes, state its weight count, its multiplies, its additions and its squash calls per pass.
⚡ Exercises — try first, then reveal
Exercise 1 — turn the neuron's weight down. Run go run ./cmd/squash -mode bearing -gain 0.5. Predict what happens to the four counts at the bottom, and to the largest turn tanh ever asks for.

The counts do not move: 2, 201, 201, 101. A weight only stretches the input across the curve, and stretching cannot merge two answers that were different or separate two that were the same. What changes is how much of the curve gets used. Tanh's extremes fall from ±0.999329 to ±0.462117, so a creature reading that column as a turn never commits more than 46% of a turn even for a plant right at the edge of its vision, and a plant at a quarter bearing buys 0.124353 where it used to buy 0.761594. The weight on a sense is the gain on a control loop, and setting it too low leaves the creature unable to react hard to anything.

Exercise 2 — drive the gate's hidden pair ten times more gently. Run go run ./cmd/squash -mode gate -hidden 1, which uses hidden weights of 1 and biases of −0.5 and −1.5. Predict which candidates still answer four of four.

Only step does. Its answers depend on the sign of the sum and nothing else, and the signs did not move. Sigmoid drops to two of four: its hidden pair now hands up 0.6225 and 0.3775 for one input on, and the output row priced for a separation of nearly a whole unit sees 0.6225 − 3 × 0.3775 − 0.5 = −1.0102 and says no. The rectifier also drops to two of four, and its failing rows are worth a look: they read exactly 0.0000 in the out column, where 5.0000 − 0.5 used to sit, so the verdict turns on the strict comparison in out[0] > 0. Tanh re-priced drops to two of four as well. Only the function that ignores magnitude survives having the magnitude taken away, which is the same fact as its two answers in the bearing table.

Exercise 3 — give the creature something to do. Run go run ./cmd/squash -mode pick -scores -1.42,-0.35,-2.10,-0.88,-0.61,0.40, which is the same tick with the bite sum lifted from −3.05 to 0.40. Predict which columns change their pick.

All five agree on bite. The rectifier still flattens the five negatives to zero, but 0.40 survives as 0.400000 and beats them; step turns the same six into five zeros and a one. The failure needed every score to be negative, which makes it the kind of bug a short demo run never finds and a valley left alone overnight finds immediately: it strikes exactly on the ticks when there is nothing much to do, and then the creature never does anything again, because resting does not change anything the senses read.

The network is now the right size, wired in the right order, with the right function in the middle and nothing over the end, and it still cannot do a thing. Every weight quoted in this chapter was typed by hand: nine for the gate, twenty-five for the worst case, twelve columns of a table. A creature needs 378, and no hand is going to type 378 numbers for every creature that is ever born in The Hollow.

← PreviousRows of Neurons, Run in Order Next →Where the Weights Come From