Design an offline-first IoT log collection platform

Millions of devices that are often offline write success and failure logs locally and upload them in batches when they reconnect. Operators can also ask a set of devices to start capturing a particular log type for a while.

Scale: about ten million devices, one million connected at a time, 200,000 records a second normally and a reconnect surge of a million a second for fifteen minutes. Records may arrive up to a day late.

Cover:

• an upload API where retries never store a record twice, with a result per record;
• delivering capture requests to devices that may be offline, and telling "delivered" apart from "the device actually started";
• success and failure counts per log type and time window, provisional within minutes and revised as late data lands;
• absorbing the reconnect storm, and storage tiers for hot and archived logs.