Stage 2 of 6 · Theory · about 35 minutes

Publish & subscribe

One architectural move, routing every message through a broker addressed by topic, and the problems of the last page dissolve. This page is the theory of that move and of MQTT, the protocol that made it the lingua franca of the Internet of Things.

The pattern: a middleman changes everything

In the publish/subscribe pattern, no producer of information ever addresses a consumer. Producers publish messages to named channels, topics, on a broker; consumers subscribe to the topics they care about; the broker delivers each message to every current subscriber of its topic. The design buys decoupling along two axes that matter enormously in distributed systems:

Decoupling in space. The node does not know the Pi exists; the Pi does not know the node's address; each knows only the broker and the topic names. Consequences cascade: a new consumer (your laptop, a logger, Week 11's dashboard) subscribes and instantly receives, with zero change at the source; a replaced sensor node publishes to the same topic and every consumer carries on, none the wiser. One-to-many distribution, the thing request/response does worst, is the broker's native gait.
Decoupling in time, partially. Publisher and subscriber need not act in the same instant: a retained message (below) lets a subscriber that arrives late still learn the latest state, and QoS 1 lets the broker hold deliveries for a briefly disconnected subscriber. MQTT is not a message queue hoarding full histories, but it meaningfully loosens the demand that everyone be awake at once.

One publish, any number of listeners. The node sends each reading once; the broker fans it out to every subscriber of the topic, today's two and next week's third, and the node's code never changes as the audience grows. That asymmetry is the whole economy of pub/sub.

The pattern also settles the efficiency complaint. Nothing polls: data moves only when published, subscribers receive within milliseconds of the event, and the arithmetic flips, ten nodes reporting once a second cost ten small messages a second total, however many consumers listen, instead of consumers × nodes requests that mostly return "no change".

MQTT: history and shape

MQTT (historically Message Queuing Telemetry Transport) was designed in 1999 by Andy Stanford-Clark of IBM and Arlen Nipper, to carry oil-pipeline telemetry over expensive, unreliable satellite links, a birth certificate that explains its character: minimal bytes, tolerance for flaky networks, and almost no demands on the device. It became an open OASIS standard and, by the 2010s, the de facto protocol of the IoT; the specification and ecosystem live at mqtt.org. The shape, in one paragraph: MQTT runs over TCP (the internet's reliable byte-stream transport, the same one SSH rides), standard port 1883 (8883 for the TLS-encrypted variant). Every participant is a client holding one connection to the broker; a client may publish, subscribe, or both over that single connection. A message is a topic plus a payload of arbitrary bytes, and the broker's one job is routing: deliver each message to the current subscribers whose filters match its topic.

Topics: a namespace you design

A topic is a UTF-8 string with / as its hierarchy separator: bench/s07/light. The broker imposes almost no rules, which means the quality of a topic scheme is entirely a design act, Week 8's message-format rung at system scale. The conventions that age well:

General to specific, left to right: site, then station, then quantity. Growth then lands naturally: bench/s07/temp beside bench/s07/light, bench/s12/light beside both.
Lowercase, no spaces, no leading slash, and case-sensitivity respected: bench/s07/Light is a different topic, and the resulting bug is silent.
One quantity per topic for simple values; a structured payload (JSON, in the ladder) when values truly travel together.
Reserve the branches you will need: this course's namespace is bench/<station>/light for telemetry, bench/<station>/status for liveness, bench/<station>/cmd for commands, documented in docs/topics.md, because an undocumented topic scheme is tribal knowledge, and A2's brief may define its own, which then governs.

Wildcards

Subscriptions (never publications) may use two wildcards, and they are the payoff of a clean hierarchy:

Filter	Matches	Reads as
`bench/s07/light`	Exactly that topic	This station's light
`bench/+/light`	`bench/s07/light`, `bench/s12/light`, ...	Every station's light: `+` fills exactly one level
`bench/s07/#`	Everything under `bench/s07/`, any depth	Everything about this station: `#` ends the filter and swallows the rest
`#`	Everything on the broker	The firehose; superb for debugging, rude in production

Quality of service: three promises

Each message is published, and each subscription made, at a QoS level: a per-message contract about delivery effort, priced in extra round trips.

QoS	Promise	Mechanism	Risk accepted	Right for
0	At most once	Fire and forget: one packet, no acknowledgement	A message may be lost	Periodic telemetry: the next reading arrives in a second anyway; the course default
1	At least once	Stored and re-sent until acknowledged	A message may arrive twice	Commands and alerts, where loss hurts more than a duplicate, if handling is idempotent
2	Exactly once	A four-step handshake per message	Highest cost and latency	The rare case where a duplicate is as bad as a loss; not used in this course

The engineering habit: choose the cheapest level the message's meaning tolerates, per message, not a blanket "best". Note for the bench: the node's MicroPython client (umqtt.simple) implements QoS 0 and 1 only, a deliberate smallness that fits the device.

Retained messages

Normally a subscriber hears only what is published after it subscribes. Publish with the retain flag and the broker additionally stores that message as the topic's sticky note: every future subscriber receives it immediately on subscribing. One retained message per topic, replaced by the next retained publish. The canonical use is last-known-state: a dashboard opening at 15:00 should not wait for the next heartbeat to learn a node is online; the retained status message tells it instantly. The canonical gotcha: retained values outlive their publishers, so a long-gone test message can greet every new subscriber for weeks ("ghost messages"); clearing one means publishing an empty retained message to the topic, and the troubleshoot table has the command.

The last will

Crashed nodes do not announce their crash; that is what makes them crashed. MQTT's elegant answer is the last will and testament: at connect time, while healthy, a client lodges a message with the broker, "if I vanish without a clean disconnect, publish this on my behalf". The broker detects the vanishing through the connection's keep-alive (below) and executes the will. Pair the will with a retained status topic and liveness monitoring falls out for free: connect, lodge will offline (retained) on bench/s07/status, then publish online (retained) to the same topic; whichever happens to the node, the topic tells the truth, and any subscriber, present or future, knows. The ladder builds exactly this.

Connections and keep-alive

An MQTT client holds one long-lived TCP connection, over which everything flows. The client declares a keep-alive interval at connect; if the broker hears nothing, not even a heartbeat ping, for one and a half intervals, it declares the client dead, closes the connection, and fires any last will. Two bench consequences: a publisher that sits idle longer than its keep-alive must ping (the node's one-second publish rhythm makes this moot; the parameter still exists), and the will's trigger latency is governed by this interval, a dead node is discovered in roughly a keep-alive and a half, not instantly. The default of "no keep-alive" in some clients means the dead-client detection is weak; the ladder sets it explicitly.

Why "lightweight" is literal

The claim is not marketing. An MQTT packet's fixed header is two bytes; publishing 42.3 on bench/s07/light costs the header, the topic string and four payload bytes, a few dozen bytes on the wire, where a minimal HTTP request-response for the same number runs to hundreds, headers both ways, connection ceremony included. Multiply by one message a second, by months of battery, by thousands of nodes, by satellite pricing circa 1999, and the design's birthplace explains every choice: topics instead of URLs, a standing connection instead of per-request handshakes, binary framing instead of text headers. Constraint is the mother of elegance here, and your 520-kilobyte node is exactly the citizen it was built for.

Security, honestly

The broker you configure next page accepts anonymous, unencrypted connections on the bench network: no credentials, no TLS. That is a deliberate, scoped engineering decision, right for a classroom LAN and wrong nearly everywhere else, and Week 3's discipline requires saying both halves out loud. What the configuration opens: anyone on the bench network can read every topic and publish to any topic, commands included. Why it is acceptable here: the network is local and supervised, the data is a light level, and the lesson's focus is the protocol. What production does instead: authentication (Mosquitto supports users and passwords, and certificates), TLS on port 8883 so traffic cannot be read or forged in transit, and topic-level access control so a client can touch only its own branches. The habit to keep from Week 3 stands: know exactly what you opened, write it down, and never promote a bench configuration to the real world by copy-paste.

Checklist for this stage

I can explain the broker's decoupling in space and (partially) in time, with one consequence each
I can write this course's three topic branches and say what + and # match
I can state the three QoS promises, their costs, and the course's default with its reason
I can explain retained + last will as a liveness pattern, including who publishes what, when
I can say what our bench broker leaves open and what production adds

Check yourself

A logger, a dashboard and a classmate's laptop all want the light feed. What changes on the node?

Nothing. Each new consumer subscribes at the broker; the node keeps publishing once to its topic and the broker fans out. Space decoupling means the audience can grow or churn with zero change at the source.

Ten nodes, once-a-second readings, four consumers. Count the messages per second under polling and under pub/sub.

Polling: each consumer asks each node, 4 × 10 = 40 request/response exchanges a second, mostly reporting no change. Pub/sub: 10 publishes a second, full stop, plus broker fan-out on the local link; and consumers learn of changes in milliseconds instead of up to a full polling period late.

Which topics does bench/+/light match, and why is bench/#/light not a legal filter?

Every topic with exactly one level between bench and light: all stations' light feeds. # may only appear as the filter's final element, swallowing everything below that point, so nothing can follow it.

Why is QoS 0 the right choice for the light feed but a poor one for a "stop the pump" command?

A lost light reading costs one second of staleness; the next reading heals it, so the cheapest promise fits. A lost stop command has no successor coming to heal it; QoS 1's at-least-once promise fits, with the duplicate risk handled by making the command idempotent (stopping twice is stopping).

A dashboard opened at 15:00 instantly shows "s07: online" although the node last published at 14:58. Explain the mechanism.

The status message was published retained: the broker kept it as the topic's last-known value and delivered it the moment the dashboard subscribed. Without retain, the dashboard would show nothing until the next status publish.

The node's power is cut mid-run. Walk through how, and roughly when, subscribers learn.

No disconnect packet is sent; the broker waits out the keep-alive window (about one and a half intervals), declares the client dead, and publishes its lodged last will, offline, retained, on the status topic. Current subscribers hear it then; future ones get the retained copy immediately. The will was lodged at connect time, which is the only time a crashing client could have arranged its own obituary.