When PaymentHooks processed its first $10 million in monthly volume three years ago, our architecture was a straightforward monolith running on three application servers behind a load balancer. That design served us well through our first $100 million, but the path to $1 billion required fundamental rethinking of how we handle concurrency, data partitioning, and failure isolation. This post chronicles the key engineering decisions that enabled that hundred-fold growth.
The most impactful change was decomposing our transaction processing pipeline into an event-driven architecture built on Apache Kafka. Every payment intent, authorization attempt, capture, and refund is now published as an immutable event. Downstream consumers handle webhook delivery, analytics aggregation, reconciliation, and fraud scoring independently. This decoupling eliminated the cascade failures we experienced during peak traffic, where a slow webhook endpoint could back-pressure the entire authorization flow.
On the data tier, we migrated from a single PostgreSQL instance to a horizontally sharded cluster using Citus. Transactions are sharded by merchant ID, which aligns with our access patterns and ensures that a high-volume merchant cannot create hot-spot contention for others. Read replicas serve the dashboard and reporting queries, keeping analytical workloads isolated from the latency-critical payment path. Our P99 database query latency dropped from 45ms to 8ms after the migration.
Finally, we invested heavily in observability. Every service emits structured traces via OpenTelemetry, and we built a custom anomaly detection layer that monitors authorization rate deviations per gateway, per BIN range, per five-minute window. When the system detects a statistically significant drop, it automatically shifts traffic to alternate gateways before our on-call engineers even receive an alert. This capability alone has prevented an estimated 14 potential incidents over the past six months.