Operate reliably

shape-only — retry, rate limiter, window, outbox worker, and metrics are MockTransport/locally tested · applies to @wats/graph, @wats/persistence, @wats/service · reviewed 2026-07-05

This page wires the reliability surfaces that ship across packages. None of them are on by default — you opt into each one.

Reliable transport with retry and rate limiting

import { GraphClient, createFetchTransport, createReliableTransport, createTokenBucketRateLimiter } from "@wats/graph";

const transport = createReliableTransport(
  createFetchTransport(),
  {
    retries: 3,
    baseDelayMs: 500,
    maxDelayMs: 8_000,
    retryPosts: "never",
    rateLimiter: createTokenBucketRateLimiter({ capacity: 20, refillPerSecond: 5 })
  }
);

const client = new GraphClient({ accessToken, apiVersion, baseUrl, transport });

Keep retryPosts: "never" for sends unless you have verified the endpoint's server-side idempotency contract. A lost response does not mean the server did nothing: it may have accepted the message before the connection failed.

"network-only" permits a retry after a network error when the request carries an Idempotency-Key header. The header alone does not make the operation idempotent. "always" also permits POST retries on 5xx. Both modes require verified server-side duplicate prevention; neither promises exactly-once delivery. The existing 429 backoff policy applies in every mode.

The token bucket smooths your outbound rate so a burst does not trip Meta's 429. It is not Meta quota management — Meta has no published per-token send budget. Tune capacity and refill from your own telemetry; never assume a bucket rate equals Meta's limit.

Idempotency-Key header on sends

await fetch(`${origin}/api/messages/text`, {
  method: "POST",
  headers: {
    authorization: `Bearer ${bearer}`,
    "content-type": "application/json",
    "idempotency-key": crypto.randomUUID()
  },
  body: JSON.stringify({ to, text })
});

When a PersistenceStore is injected, the service stores the request hash keyed by Idempotency-Key. A retry with the same key and body replays the stored response; the same key with a different body returns 409 idempotency_conflict. Pair this with retryPosts: "network-only" so a retried POST carries the key.

24-hour window gate before free-form sends

import { canSendFreeForm, getConversationWindowState } from "@wats/persistence";

const open = await canSendFreeForm(store, { phone, now: new Date().toISOString() });
if (!open) {
  // send an approved template instead — free-form outside the window is rejected by Meta
}

The helper only sees inbound rows projected into wats_messages. The service projects both outbound sends and inbound messages automatically when a persistence store is configured. With no inbound rows for a phone the helper returns closed — the safe default (send a template). See the Persistence Reference for the window state shape.

Outbox worker with metrics

import { startOutboxWorker } from "@wats/persistence";
import { createMetricsRegistry, createOutboxMetricsReporter, createWatsServiceApp } from "@wats/service";

const metrics = createMetricsRegistry();
const app = createWatsServiceApp({ profile, secrets, persistence: store, metricsRegistry: metrics });
const reporter = createOutboxMetricsReporter(metrics, { adapter: store.backend });

const worker = startOutboxWorker(store, {
  handler: async (item) => {
    // Reconstruct and send from application-owned state keyed by item.id.
  },
  intervalMs: 5_000,
  ...reporter
});

The worker is caller-owned. The service send routes stay synchronous; enqueuing into the outbox is the caller's job. Its handler reads application-owned send state; WATS stores only the payload hash, not message content. See the Persistence Reference for the lease, stop(), and onReport shape.

Plan failure recovery before running the worker unattended. The current loop retries failed items without a terminal attempt limit. You own dead-letter handling and retention of completed work. Do not enqueue an uncertain send for automatic retry: first reconcile whether the remote endpoint accepted it.

What to alert on

• graph_operations_total{outcome="error"} rate — outbound Graph failures. Spike means Meta is rejecting sends or the network is degraded.
• outbox_depth{adapter, state="pending"} growth — pending items accumulating faster than the worker drains them. Sustained growth means the handler is failing or the worker interval is too coarse.
• webhook_normalization_total{outcome="error"} — handler failures inside webhook dispatch. Non-zero means your listener threw; the webhook is still acknowledged so the failure is silent without this signal.
• Webhook auth failures — a spike in 401/404 on profile.webhook.path POST means someone (or something) is hitting the webhook without a valid signature.

The depth gauge is registry-only: it is observable through /metrics and is not forwarded to a user-owned TelemetrySink. The counter outbox_processed_total{outcome} does flow through the sink seam.

Related docs

• Service Reference
• Persistence Reference
• Telemetry for operators
• Transport and Testing