Webhooks end to end: signing, retries and replay

01How a delivery is signed

Every delivery is a POST with a JSON body and three headers of ours. Only one of them is cryptographic, but the other two are the difference between a debuggable integration and a guessing game, so it is worth knowing all three before you write any code.

The signature is an HMAC-SHA256, hex-encoded, over a string built from two parts joined by a literal dot: the timestamp, then the exact body bytes we sent. Written out, the whole construction is one line.

signed_payload = t + "." + raw_request_body
v1             = hex( HMAC_SHA256(endpoint_secret, signed_payload) )
header         = "t=" + t + ",v1=" + v1

# MailySend-Signature: t=1767225600,v1=6f1c…9ab2

Why the timestamp is inside the signed string. If the timestamp travelled alongside the signature rather than underneath it, an attacker who captured one delivery could resend it forever with a fresh timestamp and it would keep verifying. Because t is part of the message being signed, changing it invalidates the signature, and the only way to produce a signature for a new timestamp is to hold the secret. That is what makes the tolerance window meaningful rather than decorative.

The default tolerance is 300 seconds. Outside that window the signature is still perfectly valid mathematics; the delivery is rejected anyway, because a request that has been sitting in someone's proxy log for an hour has no business being accepted.

02Verify a signature, live

Compute a signature yourself, right here, with the same construction the delivery path uses. Change the body by one character and watch the whole hex string change — that is the property the next section is about.

t=1767225600,v1=<computing…>

Now the same thing in the language your handler is actually written in. Every sample below does four things in the same order: parse the header, reject a stale timestamp, recompute the HMAC over t + "." + body, and compare in constant time. There is no SDK requirement anywhere — this is standard-library work in every one of them.

import { createHmac, timingSafeEqual } from 'node:crypto'

export function verify(secret: string, header: string, raw: Buffer) {
  const parts = Object.fromEntries(header.split(',').map((p) => p.split('=')))
  const t = Number(parts.t)
  if (!Number.isFinite(t) || Math.abs(Date.now() / 1000 - t) > 300) return false

  // raw is the body as bytes, not a re-serialised object.
  const expected = createHmac('sha256', secret)
    .update(Buffer.concat([Buffer.from(t + '.'), raw]))
    .digest('hex')
  const got = Buffer.from(parts.v1 ?? '')
  return got.length === expected.length && timingSafeEqual(got, Buffer.from(expected))
}

import hashlib, hmac, time

def verify(secret: str, header: str, raw: bytes) -> bool:
    parts = dict(p.split("=", 1) for p in header.split(","))
    try:
        t = int(parts["t"])
    except (KeyError, ValueError):
        return False
    if abs(time.time() - t) > 300:
        return False

    expected = hmac.new(
        secret.encode(), f"{t}.".encode() + raw, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, parts.get("v1", ""))

func Verify(secret, header string, raw []byte) bool {
	parts := map[string]string{}
	for _, p := range strings.Split(header, ",") {
		if kv := strings.SplitN(p, "=", 2); len(kv) == 2 {
			parts[kv[0]] = kv[1]
		}
	}
	t, err := strconv.ParseInt(parts["t"], 10, 64)
	if err != nil || math.Abs(float64(time.Now().Unix()-t)) > 300 {
		return false
	}

	mac := hmac.New(sha256.New, []byte(secret))
	mac.Write([]byte(strconv.FormatInt(t, 10) + "."))
	mac.Write(raw)
	expected := hex.EncodeToString(mac.Sum(nil))
	return hmac.Equal([]byte(expected), []byte(parts["v1"]))
}

require 'openssl'

def verify(secret, header, raw)
  parts = header.split(',').map { |p| p.split('=', 2) }.to_h
  t = Integer(parts['t'], exception: false)
  return false if t.nil? || (Time.now.to_i - t).abs > 300

  expected = OpenSSL::HMAC.hexdigest('SHA256', secret, "#{t}.#{raw}")
  OpenSSL.secure_compare(expected, parts['v1'].to_s)
end

function verify(string $secret, string $header, string $raw): bool {
  $parts = [];
  foreach (explode(',', $header) as $p) {
    [$k, $v] = array_pad(explode('=', $p, 2), 2, '');
    $parts[trim($k)] = trim($v);
  }
  $t = (int)($parts['t'] ?? 0);
  if ($t === 0 || abs(time() - $t) > 300) return false;

  $expected = hash_hmac('sha256', $t . '.' . $raw, $secret);
  return hash_equals($expected, $parts['v1'] ?? '');
}

boolean verify(String secret, String header, byte[] raw) throws Exception {
  Map<String, String> parts = new HashMap<>();
  for (String p : header.split(",")) {
    String[] kv = p.split("=", 2);
    if (kv.length == 2) parts.put(kv[0].trim(), kv[1].trim());
  }
  long t = Long.parseLong(parts.getOrDefault("t", "0"));
  if (Math.abs(Instant.now().getEpochSecond() - t) > 300) return false;

  Mac mac = Mac.getInstance("HmacSHA256");
  mac.init(new SecretKeySpec(secret.getBytes(UTF_8), "HmacSHA256"));
  mac.update((t + ".").getBytes(UTF_8));
  String expected = HexFormat.of().formatHex(mac.doFinal(raw));
  return MessageDigest.isEqual(
    expected.getBytes(UTF_8), parts.getOrDefault("v1", "").getBytes(UTF_8));
}

Notice what none of them do: none parse the JSON before verifying, and none reconstruct the body from a parsed object. Every one takes the bytes it was handed. That is not stylistic.

03The mistake everyone makes once

This is the one that costs everybody an afternoon exactly once. The signature covers the bytes we transmitted. Your framework, helpfully, has already turned those bytes into an object by the time your handler runs — and turning that object back into a string does not give you the bytes back.

JSON serialisation is not canonical. Key order can change, a space after a colon can appear or vanish, a non-ASCII character may come back \u00e9 instead of é, a float may be reprinted with a different number of digits, and a trailing newline may be dropped. Every one of those produces a byte string that is a perfectly valid encoding of the same data and a completely different HMAC. Not a nearly-matching one — HMAC has no notion of nearly. One flipped byte anywhere and the hex output is unrecognisable.

# what we signed and sent
{"type":"email.delivered","data":{"email_id":"em_7Kq2xR"}}

# what JSON.stringify(req.body) hands back
{"type":"email.delivered","data":{"email_id":"em_7Kq2xR"}}

# identical here — and not identical the day someone adds an accent,
# a number with a fractional part, or a differently ordered key.

That is the cruel part: a round-trip through your JSON parser usually matches, which is why the bug ships. It fails later, in production, on the one event type that carries a customer's name with a diaeresis in it, and it fails as a signature error rather than as an encoding error, so you spend the afternoon looking at your secret.

// Express — the raw parser is scoped to this one route.
app.post(
  '/hooks/mailysend',
  express.raw({ type: 'application/json' }),
  (req, res) => {
    const header = req.get('mailysend-signature') ?? ''
    if (!verify(process.env.WEBHOOK_SECRET, header, req.body)) {
      return res.status(400).send('bad signature')
    }
    const event = JSON.parse(req.body.toString('utf8'))  // now it is safe
    …
  },
)

While you are here: make the handler idempotent on the event id. Delivery is at-least-once, which is the honest guarantee — the alternative, exactly once, would require us to know that your handler committed, and a response that never arrives is indistinguishable from a response that was never sent. So the same event can reach you twice: your handler succeeded but the response timed out on the way back, a retry was already in flight, or somebody pressed replay.

Every event carries a stable MailySend-Event-Id, and it is stable across all of those cases — retries reuse it, and a replay deliberately reuses it too, taking a fresh delivery id instead. So the contract is simple: store the event id with a unique constraint and let the insert conflict tell you it is a duplicate. Do that before the side effect, not after.

INSERT INTO processed_events (event_id, seen_at)
VALUES (?, ?)
ON CONFLICT (event_id) DO NOTHING;
-- zero rows affected → you have already handled this one. Return 200.

Return 2xx as soon as you have durably recorded the event, and do the slow work afterwards. The delivery request is abandoned after ten seconds; a handler that renders a PDF inline will be marked failed, retried, and will render the PDF again.

04The retry ladder

"We retry with backoff" tells you nothing about whether your endpoint being down over lunch loses data. Here is the actual shape, with the actual numbers, so you can answer that for yourself.

There are two mechanisms because no single one fits. The queue ladder exists to absorb a deploy, a container restart, or one momentary 502, and it should be invisible to you. A queue cannot hold a message for a day, so anything still failing is handed to a durable actor that owns the long tail. An alarm is the wrong tool for a delivery that will succeed on the second attempt, and a queue is the wrong tool for one that needs to wait until tomorrow.

$ curl -s $BASE/v1/webhooks/wh_3Qd/deliveries -H "Authorization: Bearer $KEY"
{ "data": [
  { "attempt": 3, "status": "failed",    "response_status": 502, "duration_ms": 118 },
  { "attempt": 4, "status": "delivered", "response_status": 200, "duration_ms": 94 }
] }

A successful delivery resets the consecutive-failure counter to zero. This matters more than it sounds: an endpoint that fails nineteen times over three weeks and succeeds in between is never disabled, because it is flaky rather than gone. The counter measures a run, not a total.

05Disabling and replaying

Twenty consecutive failures — no success anywhere in between — and the endpoint is disabled. Deliveries stop, the reason and the time are recorded, and nothing is lost that you cannot get back.

Twenty in a row is not a flaky handler. It is a handler that has been deleted, a domain that has expired, or a certificate that stopped renewing three weeks ago. Continuing to POST at it for days would be pure waste on both sides — ours in queue time, yours in log noise and, if the hostname has since been re-registered by somebody else, in sending your delivery events to a stranger. Disabling is the safe failure, and it is loud: the endpoint shows as disabled with the timestamp it happened, so it is a state you can see rather than a silence you have to notice.

Re-enabling clears the counter, so a fixed endpoint starts from zero rather than one failure away from being switched off again.

Replay takes a delivery and tries it again. The important detail is what it keeps and what it changes: the event_id is the same — it is the same event, and a consumer that deduplicates on the event id must see the replay as something it already knows about rather than as a second incident — while the delivery id is new, so the two attempts stay distinguishable in your logs and ours.

POST /v1/webhooks/wh_3Qd/deliveries/del_8Xz/replay

{
  "id": "del_9Fb",          // new attempt
  "event_id": "ev_2Ln",    // unchanged — dedupe on this
  "replay_of": "del_8Xz",
  "status": "pending"
}

One honest limit. Replay rebuilds the body from the stored event, so it needs that event to still be in the detail store. Past that retention it cannot be replayed and the API says so explicitly rather than delivering you a plausible-looking reconstruction — the error names /v1/exports as where to retrieve it from the archive instead. A webhook body invented after the fact would be worse than an error, because you would have no way to tell it apart from a real one.

If you are wiring up the credential side of this as well, the API keys guide covers rotation, and endpoint secrets rotate on the same principle: overlap, cut over, revoke.

Common questions