← Questions

Design a batch job orchestrator over a rate-limited LLM API

0
Scale AISystem designLast asked May 2025

Operators submit jobs of up to 5,000 tasks, stored as JSON documents in MongoDB. Each task is finished by an outside LLM provider: a blocking, rate-limited API that takes no more than 10 tasks per call. Design the job service and its workers: batching tasks into requests, admission and rate limits, idempotency, retries and partial failures, per-task progress that operators can watch, and a final job result. Note that a timed-out request may already have run at the provider.

Your canvas
Draw your answer hereBoxes, arrows and notes, saved for you to come back to.Sign in to start drawing