Work queue with a dead-letter queue and dependency cycle detection
I was asked this
No reports yet. Sign in to say where you were asked it.
Build an in-memory work queue for pipeline tasks that depend on each other (download, then tokenize, then train). Workers call next_ready() to take a task and report back with complete or fail.
Part 1: dependencies and retries. submit(task, deps=()) returns False for a duplicate id. A dependency may name a task not submitted yet; the task waits for it. A task becomes runnable once every dependency has succeeded; runnable tasks are handed out first-in, first-out. fail(task, error) puts the task at the back of the line while it has used fewer than max_attempts attempts (return "READY"), otherwise it is dead (return "DEAD"). complete and fail on a task that is not running return False / None. status(task) returns PENDING, READY, RUNNING, SUCCEEDED, DEAD or BLOCKED (or None).
Part 2: cycles. A submit that would create a dependency cycle (including through tasks submitted earlier with not-yet-known dependencies, and a task depending on itself) raises ValueError and changes nothing.
Part 3: dead letters. A dead task goes into the dead-letter queue with the reason "failed after N attempts: <error>". Every task downstream of it can never run: mark it BLOCKED and add it to the dead-letter queue with the reason "blocked by <dead task>", naming the dead task at the root. A task submitted onto a dead or blocked dependency is blocked at once. dead_letters() lists (task, reason) in the order they arrived. redrive(task) gives a dead task a fresh set of attempts and unblocks downstream tasks that have nothing else dead upstream.
class WorkQueue:
def __init__(self, max_attempts: int): ...
def submit(self, task: str, deps=()) -> bool: ...
def next_ready(self) -> str | None: ...
def complete(self, task: str) -> bool: ...
def fail(self, task: str, error: str = "") -> str | None: ...
def status(self, task: str) -> str | None: ...
def dead_letters(self) -> list: ...
def redrive(self, task: str) -> bool: ...
Sign in to see the worked answer
The full answer, chapter by chapter, with what interviewers expect at each level. Free with an account: one full answer per company.
Sign in to see the worked answer
The full answer, chapter by chapter, with what interviewers expect at each level. Free with an account: one full answer per company.
Sign in to submit your code; every attempt is kept here.
