Design a high-throughput publish-subscribe event system

Design an event ingestion and subscription system. Some clients send events fire-and-forget, up to 1,000,000 events per second in total. Other clients register subscription rules and a webhook, and must be notified of every event that matches. Define the rule expression language, the registration APIs, webhook delivery with retries and backoff, ordering and filtering guarantees, back-pressure, sharding and scaling, and address multi-tenant isolation, rate limiting and security.

Requirements

• Functional: POST /events accepts events (single or batched); POST /subscriptions registers a rule and a webhook URL; matching events are delivered to the webhook; subscriptions can be paused, updated and deleted; delivery status is visible.
• Non-functional: 1M events/s ingest with low latency to acknowledge; at-least-once delivery to webhooks; one slow or failing subscriber never slows others; tenants are isolated from each other.

Clarifying questions

• How large are events, and what do they look like (typed JSON with a schema)?
• How many subscriptions, and how many match a typical event?
• Is ordering required, and per what key (per tenant, per entity)?
• How long must undelivered events be kept for a subscriber that is down?
• Is "fire-and-forget" allowed to lose events, or must every accepted event be durable?

What a strong answer covers

• Ingestion: stateless front ends behind a load balancer, batching into a partitioned durable log (Kafka-like), partitioned by tenant and an ordering key; acknowledge after the log write.
• Rule language: a small, safe expression language over event fields (equality, ranges, in, prefix, and/or), compiled once at registration; no arbitrary code. Indexing rules (for example by event type and exact-match fields) so each event is checked against few candidates.
• Matching: consumer workers per partition evaluate candidate rules and write matches into per-subscription delivery queues.
• Delivery: per-subscriber queues and worker pools; retries with exponential backoff and jitter; a dead-letter queue after a limit; signed payloads (HMAC) with timestamps; idempotency keys so receivers can deduplicate.
• Ordering: guaranteed only per key within a subscription, at the cost of one in-flight delivery per key; otherwise unordered for throughput.
• Back-pressure and isolation: per-subscriber concurrency limits and circuit breakers, per-tenant ingest quotas and rate limits, separate queues so a slow endpoint only delays itself.
• Scale and failure: partition counts sized for peak, consumer groups that rebalance, replication of the log, and metrics on lag per subscription.
• Security: authenticated producers, verified webhook ownership, egress controls that block internal addresses (SSRF), and secrets rotation.

Follow-ups

• One event matches 100,000 subscriptions. How does fan-out work?
• A subscriber was down for a day and comes back. Replay or drop?
• Offer exactly-once delivery to subscribers. Is it possible, and what would you promise instead?