01The shape of a broadcast
Three ideas do all the work: contiguous ranges over contact-id space, one coordinator holding a cursor per range, and page workers that do the actual sending and report back. Everything else in this guide follows from those three.
The coordinator holds 32 cursors and nothing else. It never enumerates recipients, so its write rate — roughly six a second — is the same for a thousand contacts as for half a million.
The constraint that forces this shape is unglamorous. A Durable Object handles roughly a thousand requests a second, so a 500,000-contact broadcast running at 50,000 an hour cannot route every individual send through one object. The usual answer is to shard the coordinator, which trades a simple problem for a hard one: who owns which contact, and what happens when a shard dies halfway through a page.
Nothing in that loop grows with your list. The expensive work happens in page workers, which are horizontally cheap and individually disposable.
Progress is a cursor, so a crash and a pause are indistinguishable from the coordinator’s point of view: in both cases the cursor did not advance, and the next dispatch re-issues that exact page. A paused broadcast’s pages get parked and retried rather than dropped. Nothing is lost and nothing repeats — and pausing costs almost nothing, because the alarm re-arms itself on a thirty-second cycle instead of a five-second one.
02Why nothing counts your list first
The most common request for a system like this is a progress bar with a denominator. The reason there is not one in the send loop is not laziness; it is that the denominator is a lie that gets more expensive to tell as your list grows.
The system knows where it is, not how far it has to go. That is a weaker guarantee than a percentage and a much stronger one than a percentage that is quietly wrong.
| The problem with a count | What it costs |
|---|---|
| A count is a full scan | Counting the members of an audience or a segment means visiting every matching row. No index stores the answer, because the answer changes on every insert, unsubscribe and segment recomputation — so the number gets slower to compute exactly as it gets more expensive to be wrong about. |
| It is stale on arrival | By the time a count of 480,000 has been computed and rendered, someone has unsubscribed, an import has finished, and a segment’s hourly sweep has moved a few thousand people across the boundary. |
| The drift gets blamed on the wrong thing | A progress bar built on that number will drift, and the drift will be blamed on the sending rather than on the denominator. |
| It forces serialisation | A design that has to maintain an accurate total has to serialise something somewhere — which is what makes every other property in this guide impossible. |
- One indexed query, three numbers: COUNT(*), MIN(id), MAX(id)
- The segment case reads segment_members rather than re-evaluating the expression — membership is maintained continuously, and re-running the predicate here would make accepting a send O(audience)
- Filters unsubscribed = 0 as of that instant
- The total it returns is a snapshot at acceptance, for display
- Takes the two ids and splits the space between them into 32 ranges
- Ranges are boundaries in ULID space, not row counts, so it can compute them without knowing how many rows fall inside each one
- Stores the total as state and never consults it again
- Nothing in the send loop reads it: who actually receives a message is decided page by page, against the audience as it is then
03Splitting the audience into ranges
Thirty-two ranges, fixed at preparation, never rebalanced. The obvious alternative — hand out work dynamically so fast workers pick up more — is better on paper and worse in every failure mode that actually happens.
A range is owned by its index, permanently, so there is no claim to expire and nothing to reconcile between what was handed out and what came back.
- Needs a shared claim: a global record of which unit of work is owned by whom
- Written on every hand-out, and has to be correct under contention
- A worker that dies mid-page needs its claim to expire, which means a lease with a timeout
- The timeout must be longer than the slowest legitimate page and shorter than your patience
- Wrong in either direction is a duplicate send or a stall
- A range is owned by its index, permanently
- A page job is a statement about where a range’s cursor is; any worker can execute it
- A dead worker means the cursor simply did not advance
- The next tick dispatches the same page again
- No global bookkeeping, no lease, no reconciliation
# what the coordinator stores, per range { start: "01J8Q0000...", end: "01J8Q3FFF...", cursor: "01J8Q1M2K...", done: false, dispatched: 4200 } # a page job is derived from it, and carries no identity of its own { rangeIndex: 7, after: cursor || start, until: end, limit: 200 }
| Safeguard | What it prevents |
|---|---|
| The cursor is monotonic | An advance to an id lower than the current one is ignored, so a duplicated or delayed report cannot rewind a range. |
| A claim in broadcast_sends before the send is accepted | A range that genuinely is re-enumerated finds the claims already there and sends nothing twice. |
| The reconciler | A crash between the claim and the acceptance leaves an unsent claim, which is exactly what it looks for. |
The cost of fixed ranges is skew. If your contact ids are unevenly distributed, some ranges finish long before others and the tail is served by fewer workers than the start. In practice ULIDs are time-ordered and a real audience accumulates steadily, so the tail is short — a slightly ragged tail costs minutes, while a lease you got wrong costs duplicate mail to real people.
04Pacing against a limit nobody published
Every receiving provider enforces a quota it does not publish, and ramps it with your reputation. The only honest way to learn that number is to observe where sends start being refused — so the send path does exactly that, and treats the answer as expensive to get wrong in one direction and cheap in the other.
One actor per sending domain owns the three things that have to be serialised somewhere: a send-rate governor, a learned daily ceiling, and a per-provider circuit breaker. A broadcast consults it before releasing tokens.
| Signal | What happens |
|---|---|
| A domain with no history | Starts at a daily ceiling of 5,000 — low enough not to trip a new ramp. |
| A provider rejects a send for exceeding its quota | The ceiling is set to half of what was sent today, with a floor of 100, and the provider is held open for five minutes. |
| A day ends without reaching 90% of the ceiling | The ceiling is allowed to double, capped at 5,000,000. At most double: providers ramp gradually and a sudden jump looks like an attack. |
| Five failures inside five minutes | The circuit breaker opens for thirty seconds, then lets a single attempt through on a clean counter. |
| The daily ceiling is reached | Sends are refused with a retry-after that points at the next UTC midnight, rather than being retried into the same wall. |
| Your broadcast throttle exceeds what the domain sustains | Both buckets have to grant before a page goes out, so the domain governor is what you will actually observe. |
Underneath the daily ceiling sits an ordinary token bucket for instantaneous rate; the broadcast coordinator has its own, sized from the throttle you set.
The asymmetry is the point. Halving on a rejection is deliberately aggressive and doubling on a clean day is deliberately the fastest growth allowed, because overshooting a ramp costs reputation and reputation is far more expensive to recover than throughput. Losing an hour of sending is an inconvenience; getting a domain’s reputation knocked down is weeks of careful behaviour.
A warm-up ramp and a learned ceiling are the same mechanism seen from two angles. If you are starting on a new domain, the ceiling is already doing the conservative thing — see warming up a sending domain for the part the mechanism cannot do for you, which is deciding who to send the first few thousand messages to.
05Watching a broadcast go out
With no total, “progress” has to mean something else. It means: how much of the id space has been walked, and how many messages have actually been dispatched. Both are exact, neither is a percentage of anything, and together they tell you more than a progress bar would.
A broadcast is complete when the last range is marked done — which happens when a page worker walks a range and finds nothing left in it. Not when a total is reached; there is no total to reach.
| What status returns | How to read it |
|---|---|
| The broadcast state | Running, paused, complete. Pausing and crashing look the same from here. |
| How many ranges are done | Your coarse position, in thirty-seconds. |
| How far each unfinished cursor has moved between its start and its end | Your fine position — and because ids are time-ordered it is a reasonable proxy for proportion. |
| The sum of what each range has dispatched | The only number here that is a count of real messages. |
- A low open rate in the first hour — opens arrive over days and are heavily distorted by machine fetches
- The first hour’s figure is dominated by whoever happens to be at their desk
- A broadcast sitting at the same position for several minutes: usually the domain governor refusing capacity because the learned daily ceiling has been reached
- In that case the retry points at the next UTC midnight and the broadcast continues tomorrow — it is working, it is just declining to do the thing that would hurt you
- Complaint rate above roughly 0.1% of delivered
- Hard-bounce rate climbing past a couple of percent
- Both visible in the first few thousand messages
- Pausing is instant and resuming cannot re-send
If you are making a decision on opens at all, read what an open actually means first — the headline number and the number you can act on are not the same number.
What just happened
A broadcast is thirty-two cursors and a token bucket. Nothing in the send loop counts your list, nothing grows with your audience, and pausing is the same operation the system already performs after a crash — which is why it is instant and why resuming cannot re-send. The thing most likely to bite you later is treating the progress bar as a deadline: pacing is deliberate, the daily ceiling is learned by being refused, and a broadcast that looks stalled at 40% is usually a domain that has hit a quota you cannot see and should not fight.