Signed Webhooks Between Microservices: Treat the Payload as a Doorbell

When Hoadhaa Estate, the property-management app I build for landlords and tenants in the Maldives, needed subscription billing, I didn't put invoices inside it. I built a separate billing service, Tenant Manager. It owns plans, invoices, late fines, dunning reminders and payments, including bank-transfer slips that a human reviews. Hoadhaa owns properties, leases and maintenance, and never stores a price, an invoice or a card number. That split left one question to answer properly: how does Hoadhaa know, right now, whether a landlord is paid up?

The answer was signed webhooks, but with a design choice that changed how much they need to be trusted. The receiver doesn't act on what the webhook says. It treats it as a doorbell.

The shape: ask, cache, get notified

The two services share exactly one identifier. A landlord organisation in Hoadhaa is an Owner row with a ULID primary key, and that ULID is sent to Tenant Manager as the customer's external_ref. There is no mapping table to keep in sync.

Hoadhaa asks Tenant Manager for a landlord's entitlement through a small gate object that resolves in four tiers:

1. Per-request memo. Asked ten times in one request, it calls out once.

2. Cache for ten minutes. The normal path.

3. Stale cache for 24 hours. Served, with a logged warning, when Tenant Manager can't be reached.

4. Fail open. Nothing cached and the service is down: access granted.

The last tier is deliberate. A billing outage must never lock landlords out of their own data. The worst case of failing open is that someone gets a few hours of free service. The worst case of failing closed is every landlord locked out at once.

Ten minutes is too slow when someone has just paid and wants their buttons back, though. That's the webhook's job: when anything about a customer changes, Tenant Manager tells Hoadhaa straight away.

Signing: what goes into the HMAC

Every delivery carries one header:

X-TM-Signature: t=<unix>,v1=<hmac_sha256(secret, "{t}.{rawBody}")>

Each detail is there for a reason:

• •The timestamp is inside the signed string. If you sign only the body, a captured request stays valid forever. With t in the signed string, the receiver can reject anything older than five minutes, and an attacker can't change t without breaking the signature.
• •Sign the raw body, verify the raw body. The receiver hashes $request->getContent(), the exact bytes it received. It never re-encodes parsed JSON. Re-serialising changes key order and escaping, and then valid signatures start failing for no visible reason.
• •Compare in constant time. hash_equals, never ===.
• •The header is versioned. v1= leaves room to rotate the scheme later without breaking old receivers.

The sending side signs with the same logic it publishes for integrators, so the documented verification and the real one can't drift apart.

The receiver acts on nothing in the body

This is the decision I'm happiest with. After the signature checks out, the receiver uses only two things from the payload: the event_id and the customer's external_ref. The new status in the body is ignored.

if (! Cache::add('tm:webhook:'.$eventId, true, 86400)) {
    return $this->ok(['received' => true, 'duplicate' => true]);
}

// Drop the cache rather than trusting the payload's view of the status:
// the next read asks Tenant Manager directly.
$gate->forget($ownerId);
$gate->forOwner($ownerId);

Cache::add is atomic. It only writes if the key doesn't exist yet, so it doubles as a deduplication lock. Retries of the same event within 24 hours get a cheap 200 and don't run any side effects twice.

Then, instead of writing "suspended" or "active" from the payload, the receiver drops the cached entitlement and re-reads it from Tenant Manager's API, the authoritative source. The result: even if a webhook were forged or replayed, it couldn't grant or revoke anyone's access. The most it can do is trigger one extra lookup. The signature still matters, because it keeps noise and fake in-app notifications out. But the security model doesn't depend on the signature alone.

There's also a practical benefit. Webhook ordering stops mattering. If invoice.payment_succeeded arrives after subscription.reactivated, it doesn't matter, because each one only says "go look", and looking always returns the current truth.

Delivery: record first, retry for a day

On the sending side, publishing a domain event does two things in order:

1. Record the event, always, even when the client app has no endpoint configured yet. That history is what lets you replay an event, or just explain what happened.

2. Queue one delivery job per active endpoint.

The delivery job uses a 15-second timeout and a backoff ladder of 1 minute, 5 minutes, 30 minutes, 2 hours, 6 hours, then 24 hours. The ladder is long on purpose. The realistic failure is a client app being down for an afternoon, not a permanent misconfiguration, and giving up after a few minutes would lose the notification for good. Every attempt writes a delivery row with the response code, a truncated response body, the latency and the next retry time. When the ladder runs out, the row is marked exhausted and a warning is logged. When a landlord says "I paid but I'm still blocked", that table answers it in one query.

Deliveries run on a named webhooks queue. One operational trap: if a worker isn't told to listen on that queue, events pile up silently in the jobs table and nothing errors. The setup docs now spell out the exact --queue list for both services.

Idempotency keys: where they help and where they hurt

Webhooks handle one direction. For calls from Hoadhaa to Tenant Manager, I used idempotency keys, and learned where not to use them.

Customer creation sends Idempotency-Key: customer:{ownerUlid}. Creating a customer really is a once-only operation, so a retried registration job can't create duplicates, and the backfill command for existing landlords is safe to run again.

I first put a fixed key on subscription creation too. That was a mistake. Tenant Manager stores the response for a key and replays it forever, so when a landlord cancelled and later subscribed again, they got back the old, cancelled subscription. Duplicate protection for subscriptions now comes from a "do you already have one?" pre-check plus the server's own 409 subscription_exists. The rule I took from it: only use an idempotency key when the operation should happen once per entity, not once per attempt.

Enforcement that can't hit the wrong people

When a landlord's subscription lapses, Hoadhaa answers 402 on writes. The middleware only steps in when all of these are true: billing is enabled and enforcement is on, the request is a write, the user is acting as a landlord, the path isn't in an always_allow list (viewing the subscription, paying, switching role, auth), and the entitlement really denies access. Reads always work, the tenant side of the app is never touched, and a blocked user can always find the way out.

There's also an observation mode. With write blocking turned off, statuses are still fetched, cached and shown, but nothing is refused. I want a week of that before enforcement goes live. And the billing_status column on the owner row is only a mirror for reporting. Nothing ever gates on it.

The deadlock I didn't expect

In local development, both apps sometimes froze for about 15 seconds and then failed with a cURL timeout. php artisan serve runs one worker by default. Hoadhaa was waiting on Tenant Manager for an entitlement while Tenant Manager was posting a webhook back to Hoadhaa's only worker. Each was waiting for the other, and only the curl timeout broke it. Setting PHP_CLI_SERVER_WORKERS fixed development. The real lesson is to assume two services that call each other synchronously will deadlock under load, and to keep timeouts short and fallbacks cheap.

Takeaway

Sign webhooks with a timestamp inside the HMAC, verify the raw bytes, and compare in constant time. More importantly, design the receiver so it doesn't need to trust the payload. Deduplicate on the event id, treat the message as a signal to re-read from the source of truth, and the system is robust to forgery, replays and out-of-order delivery all at once. On the sending side, record every event before delivering it, retry for longer than you think, and log every attempt. And make billing fail open: an outage in a billing service shouldn't lock customers out of the product.