How I Gave a Public Message Channel an Identity

Introduction

A. A channel anyone could use

My laptop's GPU has 4 GB of video memory, which is not enough for the heaviest part of several of my projects. That part — generating images and narration with large models — runs on a Google Colab runtime with a T4 and about 15 GB of VRAM. The laptop keeps the rest, including a browser interface from which I start work, redo items, and watch progress.

A Colab runtime cannot be called from outside. It has no stable address and no port that can be opened to the world, so the interface cannot simply send it a request. What joins the two is a message broker speaking MQTT: a small server that both sides connect to. The interface publishes commands to named topics and subscribes to status; the worker in Colab does the opposite. Neither needs to be reachable, because both dial out. Two MQTT features matter below. A retained message is kept by the broker and handed to anyone who subscribes later. A Last Will is a message the broker publishes on a client's behalf if that client disappears without saying goodbye.

The first broker was a public one — free, already running, nothing to install. For finding out whether the design worked at all, that was exactly right, and it did work. But a public broker has no idea who anyone is. Anyone who guesses a topic name can read what is published there, and anyone can publish to it, including commands to my worker.

The challenge is to give that public message channel an identity — replace it with one that knows who is connecting, allows each client only what it should, and encrypts what crosses the internet — without breaking anything that already used it.

Easy to state, awkward to build. Almost none of the difficulty was in the broker itself, which I already had running for other work. It was in the two clients, and in the safeguards those clients had grown for the public broker — some of which became pointless once the broker had identities, and some of which became actively dangerous.

B. Why it had to change, and what "identity" means here

Four reasons arrived together. The first was exposure: topics and message contents on a public broker are open to anyone who can guess the names. The second was reliability: the public broker announced that a client could have 32 messages in flight at once, and measured, it accepted four before refusing with "quota exceeded". The third was that the interface needed to be opened from the internet, behind a login — and a login means little if the channel underneath it lets anyone through. The fourth made the rest affordable: I already ran a broker of my own, with access control, for other projects.

"Identity" here means three concrete things. Authentication: every connection proves who it is — the worker with a username and password, the browser with a token from a login. Authorisation: an access-control list (ACL) on the broker decides which topics each identity may read and write. Encryption: everything travelling across the internet goes over TLS, so a password or a token cannot be read on the way.

There are two very different clients to bring across. The worker is a Python program in a notebook, which can connect over plain TCP and set any option it likes. The interface is a web page, which can only reach the broker over a WebSocket — a connection a web page opens to a server and keeps open, so messages can flow both ways — and is limited to what a browser allows. Most obstacles below belong to one or the other, and several turn out to be places where the browser simply cannot do what the obvious design assumes.

C. Eight obstacles in the way

Four changes at once make every failure ambiguous. A new broker, a login, expiring tokens, and mandatory encryption, all switched on together, would leave any failure without an obvious cause.

A safeguard that only made sense on a public broker. Topic names had been derived from a hash so that nobody could guess them — computed in four separate places that all had to agree.

Access control fails without a sound. A connection refused outright is easy to diagnose. A connection that succeeds and then silently receives nothing is not, and that is what an ACL mismatch looks like.

A worker that is refused keeps running. On the public broker, nothing was ever refused, so the worker never checked. On a broker with passwords, ignoring a refusal means hours of apparently normal operation that achieve nothing.

A browser cannot add headers to a WebSocket handshake. The usual way to present a token — an Authorization header — is not available to a web page opening this kind of connection.

Tokens expire underneath a live connection. The login issues short-lived tokens, and the broker disconnects a client whose token has expired. The interface must renew its connection without dropping messages in the gap.

Shared subscriptions do not deliver retained messages. The feature that stops the interface receiving each message twice during a renewal turned out to hide exactly the messages that say whether the worker is alive.

An encrypted port does not refuse an unencrypted client — it hangs. The symptom looks like a wrong password, which sends the investigation in the wrong direction.

The order follows the migration itself. The first obstacle is about how to change things at all; the next three about the worker and the broker's rules; the next three about the browser; and the last about encryption, which came last because it was the one change that would break every client at once.

Method

A. Four stages, and a probe before any of them

Here is the answer to the first obstacle: change one thing at a time, and make each step something that can be tested and undone on its own.

# stage 1  own broker, fixed topic prefix, worker credentials
# stage 2  browser login (PKCE), token presented to the broker
# stage 3  encryption only: plain port closed, TLS port required

Each stage is a separate change that leaves the system working. If stage two breaks something, stage one is still a known-good state to return to, and the list of suspects is short.

Before any of the interface work, a small script was written to prove the broker, the login provider, and the access rules worked on their own. Without it, every failure in the interface would have carried two possible causes: a bug in the interface, or a broker configured differently than expected. With it, a failure in the interface was a failure in the interface.

That script needed to perform a real login, and there was already working login code in another project of mine. It imports that code rather than copying it:

sys.path.insert(0, str(APIGA_SCRIPTS))
from login import fetch_code

code, verifier, _nonce = fetch_code(...)

The reason is maintenance, not convenience. Two implementations of the same login flow that must keep agreeing will eventually disagree, and the one that drifts is always the one nobody is looking at.

B. The hash that lost its reason to exist

This answers the second obstacle. On the public broker, the topic prefix was derived from a hash of the storage access key, so that it could not be guessed:

export async function topicHash(accessKey: string): Promise<string> {
  const bytes = new TextEncoder().encode(accessKey)
  const digest = await crypto.subtle.digest('SHA-256', bytes)
  return Array.from(new Uint8Array(digest))
    .map((b) => b.toString(16).padStart(2, '0'))
    .join('')
    .slice(0, 8)
}

That calculation existed in four copies — the interface, the image worker, the audio worker, and a fake worker used for testing — and all four had to produce exactly the same eight characters. If one drifted, the interface and the worker would listen on different topics, and each would sit silently waiting for the other.

The hash existed because the broker was public. Unguessable names were the only isolation available. Once the broker authenticated every client and an ACL decided who could touch which topic, isolation came from identity, and the hash protected nothing. So it was removed, and the prefix became a fixed value:

PREFIX = MQTT_TOPIC_ROOT   # a fixed prefix, e.g. "app/studio"

A contract test now reads all the sources directly and fails if any of them disagree, so the fixed prefix cannot drift the way the hash could.

C. The rejection that looks like silence

This answers the third obstacle, and removing the hash turned out to close two silent failures at once.

The first was that the old hashed prefix could never have matched the ACL on the new broker, which allows a fixed prefix and nothing else. A client using the old prefix would connect successfully — and then every subscription would be refused. MQTT reports that refusal inside the subscription acknowledgement, as a code of 128, and a client that does not look at that code sees only a connection that works and receives no messages. So the interface now checks every acknowledgement:

c.subscribe(f, o, (err, granted) => {
  if (err) return reject(err)
  // SUBACK 128 = refused by the ACL; the connection itself "succeeded"
  const bad = (granted ?? []).find((g) => g.qos >= 128)
  if (bad) return reject(new Error(...))
  resolve()
})

The second was in the hash itself. crypto.subtle exists only in a secure context — a page served over HTTPS or from localhost. Open the interface over plain HTTP and it throws, the prefix is never computed, and the user is stuck on the credentials screen with no error at all. Deleting the function deleted the trap.

D. A worker that notices it was turned away

This answers the fourth obstacle. On the public broker, the worker's connection callback never checked whether it had been accepted, because nothing was ever refused. On a broker with passwords, a wrong password produces a refusal, and a worker that ignores it carries on — printing its usual log lines, generating nothing anyone will see, for hours until someone wonders why the queue is not moving.

def _on_connect(client, userdata, flags, reason_code, properties=None):
    if reason_code != 0:
        # refused: say so now, in the first lines of the log
        print(...)

The credentials themselves come from Colab's secret store, with environment variables as a fallback so the notebook can still be simulated on the laptop:

try:
    from google.colab import userdata
    user = userdata.get("MQTT_USERNAME")
    pw = userdata.get("MQTT_PASSWORD")
except Exception:
    pass
user = user or os.environ.get("MQTT_USERNAME")
pw = pw or os.environ.get("MQTT_PASSWORD")

What that rules out matters more than what it does. The notebook is committed to a git repository, so a password written into it would stay in the history forever, even after it was deleted from the file. There is also an ordering rule: the MQTT library sends the credentials inside the very first packet of the connection, so they must be set before connecting. Setting them afterwards changes nothing and produces no error.

E. A token where the password goes

This answers the fifth obstacle. The browser logs in through OAuth2 with PKCE — the flow designed for applications that cannot keep a secret, because anything shipped to a browser can be read. It receives a short-lived JWT, a signed token stating who the user is. The broker needs to see that token. The standard place would be an Authorization header on the connection, and a web page cannot add headers to a WebSocket handshake.

So the token travels as the MQTT password, which the browser can set:

clientId: `slideshow-ui-${Math.random().toString(16).slice(2, 10)}`,
username: auth?.username,
password: token,
reconnectPeriod: 0,

Two details of that configuration are rules imposed by the broker, found by testing rather than by reading. The MQTT username must be exactly the preferred_username claim inside the token; anything else is refused with reason code 134. And the client ID must be new on every connection, because a second connection with the same ID ejects the first — which matters in the next section.

The login flow itself carries two checks that are easy to leave out because leaving them out still works:

if (q.get('state') !== flow.state) throw new Error(...)

const c = decodeClaims(tok.id_token) as Claims & { nonce?: string }
if (c.nonce && c.nonce !== flow.nonce) throw new Error(...)

Without the state check, an attacker can trick a victim into completing a login with the attacker's authorisation code, so that the victim ends up signed in as the attacker. The nonce ties the returned identity token to this particular login attempt. The tokens are kept in session storage rather than local storage, so they disappear when the tab is closed, and the PKCE implementation was tested against the official test vectors from its specification rather than against values of my own.

F. Renewing a connection without a gap

This answers the sixth obstacle. Tokens live for 300 seconds, and the broker disconnects a client when its token expires. The interface therefore has to replace its connection every few minutes, indefinitely, without missing messages.

The obvious approach is to let the MQTT library reconnect automatically. That is why reconnectPeriod is zero in the configuration above: the library's automatic reconnect would reconnect using the token it already has — the expired one — and be refused again, forever.

Instead, renewal is make-before-break. At 80% of the token's lifetime a second connection is opened, and the first is closed only once the second is fully ready:

const tok = await auth.refresh()
const next = makeClient(tok.access_token)
attach(next)
await onceConnected(next)
await subscribeAll(next)
const old = client
client = next
old.end(true)   // only now that the new one is truly ready

"Fully ready" means subscribed, not merely connected. A connection that exists but has not finished subscribing receives nothing, so closing the old one at that moment would drop whatever arrived in between. And if renewal fails — the login provider is briefly unavailable, say — the old connection is kept, still serving until its own token actually expires, and renewal is retried a few seconds later.

G. The subscription that hides the worker's heartbeat

This answers the seventh obstacle. During a renewal, two connections are briefly open, and both are subscribed to the same topics — so every message would arrive twice. MQTT's shared subscriptions, written with a $share/group/ prefix, fix that: the broker delivers each message to only one member of the group.

Testing against this broker showed a side effect the documentation did not warn about. A shared subscription does not receive retained messages. And retained messages are exactly how the interface learns, on opening, whether the worker is alive — through the worker's status and its Last Will. So the interface holds two subscriptions:

return [
  [`$share/${GROUP}/${prefix}/#`, { qos: 1 }],
  [`${prefix}/worker/status/#`, { qos: 1 }],
]

A shared one for everything, and an ordinary one for the worker's status alone. The status messages may now arrive twice during a renewal, and that is harmless: each carries its own timestamp, and the interface judges freshness by that, not by how many copies it received.

H. The port that hangs

This answers the eighth obstacle. Once the broker was reachable from the internet, the plain MQTT port was closed and the TLS port became the only way in. The browser was unaffected — its WebSocket already went over HTTPS. The worker was not.

A client speaking plain MQTT to a TLS port is not refused. The server waits for a TLS handshake that never arrives, and the client waits for a reply that never comes, until a timeout. From the outside that looks like a credentials problem, so the time goes into checking the password. On Colab it meant a cell sitting silently for minutes. The worker's diagnostic now distinguishes the two: a refusal with reason code 134 or 135 means credentials, and a hang with no response at all means encryption.

c.tls_set(cert_reqs=ssl.CERT_REQUIRED)   # before connect()
c.username_pw_set(_user, _pw)            # before connect()
c.connect(MQTT_HOST, MQTT_PORT, ...)

The order is not a style choice. The library uses the TLS settings when it builds the socket, so calling tls_set after connect has no effect — the connection proceeds, unencrypted, without any complaint. Of every mistake tried deliberately to check that the tests would catch it, that one was the most subtle.

Two things were deliberately left out. There is no option to turn certificate verification off, because an emergency switch like that tends to stay on and quietly become the default — and what it would expose here is a password crossing the internet. And the broker test script lost a line that had disabled verification back when the broker still used a local, untrusted certificate. Leaving it would have meant the test kept passing even if the certificate chain broke, hiding the very failure the test existed to catch.

Results and Discussion

A. What the eight decisions add up to

The worker and the interface have run through the private broker for weeks since the move: the browser logs in and renews its connection every few minutes without losing messages, and the Colab worker connects over TLS with credentials that never appear in the repository.

B. Safeguards belong to their environment

Most of this work was not adding security. It was re-examining safeguards that had been correct for the old environment and finding out what each one became in the new one.

Some became pointless. The hashed topic name was the right defence on a broker where anyone could listen, and on a broker with identities it only added four copies of a calculation that could drift apart. Some became dangerous. An automatic reconnect is a sensible default until tokens expire, and then it becomes a loop of refused connections. A flag that skips certificate checks is a reasonable convenience for a local test broker, and a hole once the broker is on the internet.

And some protections simply did not exist before, because the old environment never produced the failure they guard against. Nothing refused the worker on a public broker, so it never learned to listen for refusal. Nothing checked subscriptions, because nothing denied them. Nothing had to tell a TLS hang apart from a bad password, because there was no TLS.

The general version is worth stating. When the environment under a system changes, the safeguards cannot be carried across unexamined. Each one was an answer to a question the old environment asked. The work is to ask which questions the new environment asks instead — and to expect that several of the new failures will be silent, because the old environment never needed them to make a sound.

C. What this does not cover

This is one person's system. The access rules separate the interface from the worker, not one team member from another, and a multi-user setup would need identities and rules per person. The 300-second token lifetime is the login provider's setting; the 80% renewal point follows from it, and a different lifetime would change the timing but not the design.

The two clients are also not treated alike. The browser's credential expires every five minutes and renews itself; the worker's password lasts until someone changes it, and changing it is a manual step in Colab's secret store. That asymmetry is acceptable for a worker that only one person starts, but it is the first thing to revisit if more workers, or more people, ever connect.

Conclusion

A 4 GB GPU put the heavy work on a Colab runtime, a public broker connected it to the laptop, and that broker worked. Giving it an identity — authentication, access control, encryption — replaced the broker in an afternoon. Bringing the two clients across took the rest: removing a hash that no longer protected anything, noticing refusals that arrived silently, carrying a token where a browser has no header to put it, renewing connections before they expire, subscribing twice because one kind of subscription hides the most important messages, and telling a hang apart from a rejection.

Doing it in four separate stages, with the broker proven on its own first, is what kept each surprise small. Most of those surprises were safeguards that had been right for the old environment and wrong for the new one.

What carries over is a habit for any migration: list every safeguard the old setup relied on, and for each one ask what it protects against now — and what new failure the new setup can produce that the old one never could.

Terms

WebSocket

A long-lived, two-way connection that a web page opens to a server. It is the only way a browser can talk to a message broker, and it limits what the page can attach when connecting.

ACL

Access-control list. A set of rules on the broker deciding which identities may publish or subscribe to which topics.

SUBACK

The broker's acknowledgement of a subscription request. It carries a code per topic; a code of 128 or above means the subscription was refused.

JWT

JSON Web Token. A signed, short-lived token stating who the holder is, which a server can verify without contacting the issuer.

PKCE

Proof Key for Code Exchange. An OAuth2 extension that lets applications unable to keep a secret, such as web pages, complete a login safely.

make-before-break

Replacing a connection by opening and fully preparing the new one before closing the old, so there is never a moment with neither.

secure context

A web page served over HTTPS or from localhost. Some browser features, including the cryptography API used for hashing, exist only there.