Design a reliable job scheduler

Design a service that runs jobs at scheduled times at large scale, such as payment jobs that must run on time and exactly once in effect. Cover storing schedules, finding jobs that are due, handing them to workers, retries and backoff, idempotency so a retried payment is not charged twice, worker crashes, monitoring missed runs, and scaling the scheduler itself.