A Broker Between Every Device
Every new device edits her
She answers over the network now: chapter 89 put her voice and her mind behind HTTP routes, so anything on the LAN can ask her a question. The house is a different problem. A motion sensor from one vendor, a doorbell from another, a plug that speaks its own dialect over a socket nobody documented, a window contact that was only ever meant to talk to a phone app. The obvious plan is to teach her each of them, one integration at a time.
Count what that costs first. Five devices and two things that care (her, and a log you keep on a spare Pi) comes to ten connections, each of them code somebody has to write, authenticate, reconnect and keep working; buy a sixth device and you write two more. All ten live inside her, so her source changes every time the hardware in your house does. The timing is worse than the arithmetic: a sensor that fires while she is restarting has nowhere to put the message, because the only listener it knows about is the one that just went away.
A broker inverts the direction of knowledge. Put one small server in the middle and nothing addresses anything: the sensor publishes "motion happened" under a name, she asks for every name of that kind, and neither one holds a reference to the other. A topic is a mailbox, not a phone number. The topic string is the entire interface: publishers and subscribers agree on names, never on addresses. The device you buy next year becomes one more publisher of a name she already subscribes to, and her code does not move.
By the end of this chapter Mosquitto is running on the same box she is,
labs/mqtt_bridge.py turns every topic and payload that arrives into a
lookup against the automation registry from chapter 17, and adding a device to the
house costs one row of data plus one handler. You will also meet the two ways this
arrangement bites: a stored message that outlives its meaning, and one process that the
whole house quietly depends on.
sudo apt install mosquitto mosquitto-clients installs the broker and the
two command-line tools you will use all chapter, then
sudo systemctl enable --now mosquitto starts it and brings it back after
a reboot. Since version 2.0 the default configuration listens only on 127.0.0.1 and
refuses anonymous clients, so the broker is useless to the rest of the house until you
add a config file under /etc/mosquitto/conf.d/ with a
listener 1883 line and a password_file. Do that when the
first real device arrives, not before. On her side,
uv add paho-mqtt pulls in the 2.x client used below.
A namespace you have to design
The broker has no opinion about your names. It stores strings and matches strings, which leaves the naming entirely to you, and this is the one decision in the chapter you cannot revise cheaply. Every sensor you ever flash carries the names you picked, and renaming a level means reflashing hardware that may be screwed to a wall.
Four levels, in this order: home/<area>/<device>/<kind>.
The area comes before the device because subscriptions are read left to right, so the
earlier a level sits the more useful it is as a filter:
home/kitchen/# hands you one room, every device in it, forever, and no such
question is askable if the device name comes first. The kind goes last because it is the
level she filters on, and there are exactly three of them.
stateis what a device currently is:open,OFF,21.5. Published by the device, and stored by the broker so a late arrival can catch up.eventis what just happened: a button press, a motion trip. It is true for an instant and meaningless afterwards, so nothing stores it.setis somebody asking a device to change. Published by her, consumed by the device, and never stored, for a reason the failure box below makes concrete.
One name lives outside the tree: glados/status, which is her bridge saying
whether it is running. It is not part of the house, so it does not sit under
home/, and putting it there would mean her own subscriptions had to
carefully avoid herself.
# labs/mqtt_bridge.py
def topic_matches(subscription: str, topic: str) -> bool:
"""True when a broker would deliver `topic` to a subscriber of `subscription`."""
levels = subscription.split("/")
parts = topic.split("/")
if parts[0].startswith("$") and levels[0] in ("+", "#"):
return False
for i, level in enumerate(levels):
if level == "#":
return True
if i >= len(parts):
return False
if level != "+" and level != parts[i]:
return False
return len(levels) == len(parts)
CASES = [
("home/+/+/event", "home/kitchen/motion/event"),
("home/+/+/event", "home/front/doorbell/event"),
("home/+/+/event", "home/kitchen/motion/state"),
("home/+/+/event", "home/kitchen/counter/motion/event"),
("home/kitchen/#", "home/kitchen/motion/event"),
("home/kitchen/#", "home/kitchen"),
("home/#", "home/hall/lights/set"),
("#", "$SYS/broker/uptime"),
]
if __name__ == "__main__":
for subscription, topic in CASES:
verdict = "deliver" if topic_matches(subscription, topic) else "drop "
print(f"{verdict} {subscription:<16} <- {topic}")
$ uv run python -m labs.mqtt_bridge
deliver home/+/+/event <- home/kitchen/motion/event
deliver home/+/+/event <- home/front/doorbell/event
drop home/+/+/event <- home/kitchen/motion/state
drop home/+/+/event <- home/kitchen/counter/motion/event
deliver home/kitchen/# <- home/kitchen/motion/event
deliver home/kitchen/# <- home/kitchen
deliver home/# <- home/hall/lights/set
drop # <- $SYS/broker/uptime
No broker was involved in that run, and that is the point of writing the function at all: your naming scheme becomes something you can test in one second, before any hardware exists to be renamed. The fourth case is the one people get wrong from memory. A plus sign is not a wildcard for "anything below here"; it fills exactly one level, so a sensor published under an extra level of nesting goes silently undelivered no matter how reasonable its name looks.
Two smaller rules hide in the last three cases. A hash matches the parent level too,
so home/kitchen/# also delivers a message published on
home/kitchen itself. And a leading wildcard skips the
$SYS tree, where brokers publish their own statistics; without that rule
a client subscribed to # for debugging would be flooded with the broker
reporting on itself. Stage 4 reuses this same function on the receiving side, so her
dispatch rows and her subscriptions are decided by one implementation of one rule.
Connected, and what the broker keeps
import paho.mqtt.client as mqtt
BROKER_HOST = "127.0.0.1"
BROKER_PORT = 1883
CLIENT_ID = "glados-bridge"
STATUS_TOPIC = "glados/status"
SUBSCRIPTIONS = [("home/+/+/event", 1), ("home/+/+/state", 1)]
def on_connect(client, userdata, flags, reason_code, properties) -> None:
if reason_code != 0:
print(f"[MQTT] refused: {reason_code}")
return
print(f"[MQTT] connected to {BROKER_HOST}:{BROKER_PORT} "
f"(session_present={flags.session_present})")
client.subscribe(SUBSCRIPTIONS)
client.publish(STATUS_TOPIC, "online", qos=1, retain=True)
for topic, qos in SUBSCRIPTIONS:
print(f"[MQTT] subscribed {topic} qos {qos}")
def on_message(client, userdata, msg) -> None:
payload = msg.payload.decode("utf-8", errors="replace")
print(f"[MQTT] {msg.topic} = {payload!r} (qos {msg.qos}, retain {int(msg.retain)})")
def build_client() -> mqtt.Client:
client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2,
client_id=CLIENT_ID, clean_session=False)
client.will_set(STATUS_TOPIC, "offline", qos=1, retain=True)
client.on_connect = on_connect
client.on_message = on_message
return client
if __name__ == "__main__":
client = build_client()
client.connect(BROKER_HOST, BROKER_PORT, keepalive=60)
client.loop_forever()
$ uv run python -m labs.mqtt_bridge # terminal 1, then two publishes from terminal 2
[MQTT] connected to 127.0.0.1:1883 (session_present=False)
[MQTT] subscribed home/+/+/event qos 1
[MQTT] subscribed home/+/+/state qos 1
[MQTT] home/kitchen/motion/event = 'detected' (qos 1, retain 0)
[MQTT] home/office/window/state = 'open' (qos 0, retain 0)
$ mosquitto_pub -t 'home/kitchen/motion/event' -m detected -q 1 # terminal 2 $ mosquitto_pub -t 'home/office/window/state' -m open -r
Four details in build_client are doing real work.
CallbackAPIVersion.VERSION2 is required by paho 2.x, which refuses to
guess which callback signature you wrote against; the version 2 signature is the one
with reason_code and properties on it. A fixed
client_id plus clean_session=False asks the broker to
remember this client between connections, which is what makes
flags.session_present meaningful and what stage 6 leans on. And
will_set registers a last will: a message the broker itself publishes if
this client vanishes without saying goodbye, so anything watching
glados/status learns her bridge died within a keepalive or two.
Look closely at the second publish. It went out with -r, the retain flag,
and it arrived with retain 0. That is correct and it surprises everybody:
the flag on a delivered message does not mean "this was published with retain", it
means "the broker is replaying a stored message to you now". A live delivery to a
client that was already subscribed always reads zero.
$ uv run python -m labs.mqtt_bridge # started again; nothing happened in the house meanwhile
[MQTT] connected to 127.0.0.1:1883 (session_present=True)
[MQTT] subscribed home/+/+/event qos 1
[MQTT] subscribed home/+/+/state qos 1
[MQTT] home/office/window/state = 'open' (qos 0, retain 1)
$ mosquitto_sub -t 'home/#' -v --retained-only -W 1
home/office/window/state open
$ mosquitto_pub -t 'home/office/window/state' -r -n $ mosquitto_sub -t 'home/#' -v --retained-only -W 1
She learned the window was open without anybody publishing anything, because the
broker keeps exactly one retained message per topic and hands it to every new
subscriber the moment the subscription is accepted. That is the difference between
state and event stated as a storage policy: a device that publishes its state
retained answers the question "what is it now?" for a subscriber that was not born
yet, while an event published without the flag is gone the instant the last connected
subscriber has it. It is also why the window row in the next section watches a
state topic: she wants the current truth at startup, not a history of
hinge movements she missed.
The third command deletes it. -n publishes a zero-length payload and
-r makes it retained, and a retained empty message is the one thing a
broker treats as "forget this topic", which is why the second listing prints nothing
at all. There is no other way to remove one. A retained message with persistence
enabled outlives the process, the reboot, and your memory of having sent it.
The hall lamp is a smart plug that drops off the network during firmware updates. You
publish OFF to it while it is down, nothing happens, and the fix looks
obvious: retain the command, so the plug gets it whenever it comes back.
$ mosquitto_pub -t 'home/hall/lights/set' -m OFF -q 1 -r
It works. The plug reconnects an hour later, receives OFF, obeys. Two
weeks after that the complaints start: the hall lamp turns itself off a second after
anybody switches it on, but only sometimes. Then a power cut at 2 a.m. brings the
whole house back at once and every lamp in the hall snaps off again on its own. The
house has developed a personality, and not one of hers.
$ mosquitto_sub -t 'home/hall/lights/#' -v --retained-only -W 1
home/hall/lights/state ON
home/hall/lights/set OFF
Two retained messages, and only the first belongs there. The broker is doing precisely
what it was told: a retained message is delivered to every subscriber at every
subscribe, and the plug subscribes to its own set topic every time it
boots or its Wi-Fi flaps. The stored OFF from two weeks ago is now a
standing order, replayed forever, and nothing in MQTT expires it.
mosquitto_pub -t 'home/hall/lights/set' -r -n clears it and the symptom
stops immediately. The rule underneath is the reason the three kinds in the namespace
are separated at all. State is a fact that stays true until the device says otherwise,
so retaining it is honest. A command was true once, at the moment somebody meant it,
and retaining one means instructing every future version of that device to obey a
decision made in the past. This is also why publish_set in the next stage
spells out retain=False even though that is already the default: the
argument is there to be read by the person who is about to think retaining commands is
a clever trick.
The dispatch she already has
Nothing above dispatched anything. That part of the problem was solved in chapter 17,
where rules became data, handlers became code, and a registry joined them by name; and
in chapter 35, where execute_action gained a try around the
handler so one broken action could not take the caller down with it. The bridge needs no
new dispatcher. It needs a new kind of trigger row, because the thing arriving is no
longer a sentence somebody spoke, it is a topic and a payload.
from glados.core import GladOSCore
from labs.automation import register_action
from labs.command_loop import execute_action
CLIENT: mqtt.Client | None = None
CORE: GladOSCore | None = None
TRIGGERS: list[dict] = [
{"filter": "home/+/motion/event", "payload": "detected", "action": "greet_arrival"},
{"filter": "home/front/doorbell/event", "payload": "*", "action": "announce_visitor"},
{"filter": "home/+/window/state", "payload": "open", "action": "note_open_window"},
]
def match_triggers(topic: str, payload: str) -> list[dict]:
return [t for t in TRIGGERS
if topic_matches(t["filter"], topic) and t["payload"] in ("*", payload)]
def split_event(text: str) -> tuple[str, str, str]:
"""'home/kitchen/motion/event detected' -> ('kitchen', 'motion', 'detected')"""
topic, _, payload = text.partition(" ")
levels = topic.split("/")
return levels[1], levels[2], payload
def publish_set(area: str, device: str, value: str) -> None:
topic = f"home/{area}/{device}/set"
if CLIENT is None:
print(f"[DRY] {topic} = {value}")
return
CLIENT.publish(topic, value, qos=1, retain=False)
print(f"[PUB] {topic} = {value}")
def say(line: str) -> None:
if CORE is None:
print(f"[SAY] {line}")
return
CORE.speak(line)
@register_action("greet_arrival")
def greet_arrival(text: str) -> None:
area, _, _ = split_event(text)
publish_set(area, "lights", "ON")
@register_action("announce_visitor")
def announce_visitor(text: str) -> None:
say("Someone is at the front door. I have decided not to be excited about it.")
@register_action("note_open_window")
def note_open_window(text: str) -> None:
area, device, value = split_event(text)
say(f"The {area} {device} is {value}. The heating is, of course, still running.")
def dispatch(topic: str, payload: str) -> int:
fired = 0
for rule in match_triggers(topic, payload):
if execute_action(rule["action"], f"{topic} {payload}"):
fired += 1
if fired == 0:
print(f"[SKIP] {topic} = {payload}")
return fired
if __name__ == "__main__":
for topic, payload in [
("home/kitchen/motion/event", "detected"),
("home/front/doorbell/event", "pressed"),
("home/office/window/state", "open"),
("home/office/window/state", "closed"),
("home/kitchen/lights/set", "ON"),
]:
dispatch(topic, payload)
$ uv run python -m labs.mqtt_bridge # no broker, no microphone, no speaker
[DRY] home/kitchen/lights/set = ON
[SAY] Someone is at the front door. I have decided not to be excited about it.
[SAY] The office window is open. The heating is, of course, still running.
[SKIP] home/office/window/state = closed
[SKIP] home/kitchen/lights/set = ON
A trigger row carries a filter, not a topic, so one row covers every room you will
ever have: home/+/motion/event is written once and matches the sensor you
mount in the garage next spring. The payload field is a second, cruder test, because
a topic tells you which device spoke and the payload tells you what it said, and
open and closed arrive on the same topic. A payload of
"*" means the row does not care, which suits a doorbell: there is only
one thing a doorbell has to say.
The last two lines of output are the interesting ones. closed matches the
window filter but not its payload, so nothing fires. And her own command,
home/kitchen/lights/set, matches no row at all, because no row ends in
set. That is the feedback loop everybody builds once by accident:
subscribe to home/# to catch everything, publish a command, receive your
own command, act on it, publish again. The namespace makes it unrepresentable instead
of catching it with a flag, and the level that does the work is the last one.
Two compromises deserve naming. Chapter 17 fixed the handler signature at one string,
so the bridge packs the topic and payload into one and every handler unpacks it with
split_event; a closure per handler would read better and would cost the
registry its ability to be called from anywhere else. And CLIENT and
CORE are module globals with printed fallbacks, in the pattern chapter 32
set for a missing voice, so the dispatch layer stays runnable with no broker and no
audio hardware attached. That is why the output above reproduces on your machine.
from labs.system_config import build_default_config
from labs.wire_core import build_core
def on_message(client, userdata, msg) -> None:
payload = msg.payload.decode("utf-8", errors="replace")
print(f"[MQTT] {msg.topic} = {payload!r} (qos {msg.qos}, retain {int(msg.retain)})")
dispatch(msg.topic, payload)
def main() -> None:
global CLIENT, CORE
CORE = build_core(build_default_config())
CLIENT = build_client()
CLIENT.connect(BROKER_HOST, BROKER_PORT, keepalive=60)
try:
CLIENT.loop_forever()
except KeyboardInterrupt:
CLIENT.publish(STATUS_TOPIC, "offline", qos=1, retain=True).wait_for_publish()
CLIENT.disconnect()
print("\n[MQTT] bridge down")
if __name__ == "__main__":
main()
$ uv run python -m labs.mqtt_bridge # her voice comes out of the speaker, not the terminal
Loading whisper base on cpu...
Loading F5-TTS...
[MQTT] connected to 127.0.0.1:1883 (session_present=True)
[MQTT] subscribed home/+/+/event qos 1
[MQTT] subscribed home/+/+/state qos 1
[MQTT] home/kitchen/motion/event = 'detected' (qos 1, retain 0)
[PUB] home/kitchen/lights/set = ON
[MQTT] home/front/doorbell/event = 'pressed' (qos 1, retain 0)
^C
[MQTT] bridge down
$ mosquitto_sub -t 'home/+/lights/set' -v -W 20 # terminal 3, watching what she publishes
home/kitchen/lights/set ON
There is no lamp in this house. Terminal three is standing in for one, and her code
neither knows nor cares: she published an intention under a name, the broker matched
it against whoever asked, and the day a real plug subscribes to that topic the only
thing that changes is who receives the message. Nothing prints between the doorbell
line and the ^C because the doorbell handler speaks, and speaking happens
through the speaker.
Count the diff from stage 4 honestly. One added line in on_message, one
main that wires the two globals, and the entire house is now connected to
an automation engine written back in volume 2. Adding a smoke alarm tomorrow is one
dict in TRIGGERS and one decorated function; the connection code, the
subscriptions and the callbacks stay untouched, because they never knew what a smoke
alarm was in the first place.
paho calls on_message from its network loop, the same thread that reads
sockets and answers the broker's keepalive pings. Her say blocks for as
long as the sentence takes to synthesize and play, and every message that arrives
during it waits in the socket buffer. Block for longer than the keepalive interval and
the broker decides she is gone, publishes her will, and disconnects her. For a house
with three sensors this never bites; for one that also runs a model on every event, put
dispatch behind a queue.Queue and a worker thread, which is
the third exercise.
Why this works: the broker holds the filters
A subscription is a string the broker files away next to a connection. When any client publishes, the broker walks its table of filters, applies the rule stage 1 implemented, and copies the message to each match. That is the entire mechanism, and every property of the design falls out of it. Publishers cannot know their audience, because the audience is not consulted at publish time. Adding a second consumer costs nothing anywhere else, because subscribing does not touch the publisher. And the count that started this chapter collapses: five devices and two consumers is seven connections to one broker, and the eighth device adds one.
Chapter 15 built this inside one process, where publishers and subscribers shared an event name and nothing else. A broker is that idea with a network in the middle and a little memory of its own, and the memory is what buys the second kind of decoupling: not just who from whom, but when from when. A retained message reaches a subscriber that did not exist when it was sent. Kafka, NATS and Redis publish and subscribe with different durability guarantees and different ordering promises, but the pattern you just built is the one they are all variations of.
Which leaves the question the failure box only half answered. A command is published, and the device it names is unplugged. Where does it go?
$ mosquitto_sub -i hall-lights -c -q 1 -t 'home/hall/lights/set' -v # the plug, connecting once
^C
$ mosquitto_pub -t 'home/hall/lights/set' -m OFF -q 1 # published while the plug is down
$ mosquitto_sub -i hall-lights -c -q 1 -t 'home/hall/lights/set' -v # the plug, coming back
home/hall/lights/set OFF
The message was held for a client that was not connected, and delivered the moment it
returned. Three things had to be true at once for that: the subscriber used a stable
-i client id, it asked for a session that survives disconnection with
-c, and the publish used -q 1. Drop any one and the broker
has nowhere to put the message. At QoS 0 it is discarded on the spot, and
mosquitto_pub exits 0 without complaint, because from its point of view
handing the message to the broker was the job.
A device that has never connected has no session either, so nothing can be queued
against it, and a command sent to a plug you have not set up yet evaporates in total
silence. That gives three possible fates for a command aimed at an absent device: lost
instantly, delivered late (possibly hours late, which for "turn the lights on" is often
worse than losing it), or retained and repeated forever. Pick per topic, deliberately.
The way to know whether the hall lamp is on is to read
home/hall/lights/state, which the lamp itself publishes retained, and
never to assume that having published a set made something true.
One more piece of honesty, because this design has a cost you should agree to on purpose: every device, every rule and every command in this chapter runs through a single process on a single box. Mosquitto is small and it does not crash often, but it is one systemd unit on one SD card, and it stops for all the boring reasons software stops. When it does, publishers get a connection error or queue locally and forget, devices retry on their own schedules, and nothing anywhere raises an alarm. The house does not break. It goes quiet.
At 2 a.m. that means the hall sensor trips into nothing, the lamp never receives its
ON, and the only symptom is a light that did not come on while you were
carrying a glass of water down the stairs. She will still answer you in the morning,
cheerfully, about a house she has been out of contact with for six hours, because her
voice loop never needed the broker. Four mitigations, in order of how much they actually
buy:
- Run the broker under systemd with
Restart=always, and turn onpersistence trueso retained messages and queued sessions survive the restart instead of vanishing with the process. - Let something outside MQTT notice. A watchdog that reports its findings by publishing to the broker reports nothing at all when the broker is the casualty; hers can check the connection every minute and say so out loud through the speaker.
- Understand what the last will covers.
will_setannounces her bridge dying, published by the broker on her behalf. If the broker is what died, there is nobody left to publish anyone's will. - Keep the physical path. Wall switches stay wired, and nothing you would need in an emergency (door locks, the heating, a smoke alarm's own siren) depends on a hobby broker being awake.
Checkpoint, and a house that talks back
- I can say where
+and#are each legal in a filter, and whyhome/kitchen/#also delivers a message published onhome/kitchen. - I can name the level of the topic that stops her acting on her own commands, and show that it does by pointing at one line of stage 4's output.
- I can read the retain flag on a received message and say whether the broker was replaying storage or forwarding a live publish.
- I can delete a retained message, and explain why a retained
setturned into a standing order the plug obeyed at every boot. - I can list the three fates of a command published to an offline device, and the exact conditions that produce each one.
- I can add a sensor to her house without opening the file that talks to the broker, and name what a dead broker looks like from inside the house.
Exercise 1 — build the lamp she is talking to. Write
labs/fake_lamp.py: subscribe to home/hall/lights/set, print
each command, and publish the new value retained to
home/hall/lights/state. Then trip the hall sensor and watch the round
trip.
It is about twenty lines and reuses build_client almost unchanged: a
different client id, one subscription, and an on_message that calls
client.publish(f"home/hall/lights/state", payload, qos=1, retain=True).
Add a hall row to TRIGGERS, run all three processes, and publish one
motion event. You will see her [PUB] line, the lamp printing what it
received, and then her bridge logging the lamp's new state arriving on the
state topic she was already subscribed to. Kill the lamp and restart it: her bridge sees the retained state
again before anything happens. Two programs, both directions, neither one importing
the other.
Exercise 2 — a rule that depends on something else being true. Make the front door trigger announce a visitor only when the house is empty, using the state messages she is already receiving.
Keep a dict LAST_STATE: dict[str, str] and write every
state topic into it inside on_message, before dispatch.
Then add an optional key to the trigger row, something like
{"requires": ("home/hall/presence/state", "away")}, and have
match_triggers skip any row whose requirement does not match
LAST_STATE. Test it without a broker by filling
LAST_STATE by hand and calling dispatch twice with the
same doorbell event: once with presence home, once with
away. Notice what you did not have to do, which is change any handler.
The rows got richer and the engine stayed still.
Exercise 3 — get off the network thread. Put
dispatch behind a queue.Queue and one worker thread, then
prove the network loop is free by timing the log lines.
on_message becomes two lines: log, then QUEUE.put((topic,
payload)). A daemon thread runs while True: dispatch(*QUEUE.get()).
To see the difference, register a handler that calls time.sleep(3) and
publish two events half a second apart. Before the change, the second
[MQTT] line appears three seconds late; after it, both log lines appear
immediately and the actions finish in order behind them. Print time.strftime("%H:%M:%S") at the front of every line so the fix is
visible instead of theoretical.
The house can reach her and she can reach the house. That exposes something the single prompt behind her has been hiding: she is now asked about the state of a building, the contents of her own memory, and the servos in her neck, and one set of instructions pulled in three directions is mediocre at all three. Chapter 91 splits her into specialists with a coordinator deciding who answers.