Design LLM-assisted grading of contributor work

Contributors submit finished task work, and each submission is graded against a versioned rubric by a rate-limited third-party LLM that can give different answers for the same input. Every submission belongs to one tenant, and only that tenant's authorized users may see the work, the evidence or the result. Design the system from submission to final grade: accept work, schedule grading in the background, catch unreliable judgments and send them to human reviewers, and serve one durable, auditable outcome per submission. Writing tasks, training the model and paying contributors are out of scope.

Requirements

• Functional: submit one result against an immutable task and rubric version and get a submission ID back quickly; grade it asynchronously within provider and tenant limits; flag low-confidence or conflicting judgments and route them to human review; show progress and one final outcome with its rubric, model and prompt version, evidence and any human review.
• Non-functional: no submission graded twice into two outcomes; tenant isolation on every read; graceful slowdown when the provider throttles or fails; a full audit trail for disputes.

Clarifying questions

• How many submissions per day, and how fast must a grade arrive?
• What does a rubric look like: criteria with scores, pass/fail, or free text?
• What makes a judgment unreliable enough to need a human, and how many reviewers are there?
• Can a rubric change while submissions are in flight, and should old ones be regraded?
• Who may dispute a grade, and what happens then?

What a strong answer covers

• Data model: submissions pinned to task and rubric versions, grading attempts (model, prompt version, raw output, parsed scores), review tasks and one final outcome per submission written once.
• Pipeline: an accept path that only stores and enqueues, a scheduler with per-tenant quotas and a provider-wide token bucket, idempotent workers with leases, and retries that don't create a second outcome.
• Quality control: structured output with per-criterion evidence, several samples or judges and agreement thresholds, self-reported confidence treated with care, gold submissions with known grades to calibrate and detect drift, and spot checks of auto-accepted grades.
• Human review: queue, assignment, the reviewer seeing the model's evidence, and how their decision overrides and is recorded.
• Tenant isolation: tenant ID on every row and key, checks in the serving path, and no cross-tenant prompt or cache sharing.
• Trade-offs named: cost of multiple samples against accuracy, how much to send to humans, latency against batching.

Follow-ups

• A new model version grades 5% harsher: how do you notice and roll it out safely?
• A contributor disputes a grade weeks later: reproduce exactly what happened.
• The provider is down for an hour.
• Prompt injection inside the submitted work.