Webhooks are the backbone of asynchronous payment integrations. When a charge succeeds, a subscription renews, or a dispute is opened, your systems need to know immediately. At PaymentHooks, we deliver over 50 million webhook events per day across tens of thousands of endpoints, and our delivery guarantee is simple: every event will be delivered at least once, even if your server is temporarily unreachable. Achieving this at scale required solving several non-trivial distributed systems challenges.
Our delivery pipeline begins when a domain event is committed to our event store. A dedicated consumer picks up the event, resolves all matching webhook subscriptions, and enqueues a delivery job for each. The delivery worker makes an HTTP POST to the endpoint with a JSON payload, a timestamp, and an HMAC-SHA256 signature. If the endpoint responds with a 2xx status within 30 seconds, the delivery is marked successful. Any other response, including timeouts and connection errors, triggers our retry logic.
The retry schedule follows a modified exponential backoff: 1 minute, 5 minutes, 30 minutes, 2 hours, 8 hours, and 24 hours. After six failed attempts spanning roughly 34 hours, the event is moved to a dead letter queue and the endpoint is flagged as unhealthy. Merchants receive an email notification after the third consecutive failure and can inspect failed deliveries in the dashboard. The dead letter queue retains events for 30 days, and merchants can replay any event with a single click.
To prevent a single slow endpoint from consuming all delivery capacity, we implemented per-endpoint concurrency limits and circuit breakers. If an endpoint accumulates five consecutive failures within a ten-minute window, the circuit opens and all pending deliveries for that endpoint are paused for five minutes. This protects both our infrastructure and the merchant's server from thundering-herd scenarios during outages. Merchants can monitor endpoint health, delivery latency percentiles, and failure rates in real time through the Webhooks dashboard.