Design a RAG system end to end

Design a retrieval-augmented generation (RAG) system that answers employees' questions over a company's internal text: wikis, documents, tickets and similar sources. Cover the path from raw documents to a grounded answer with citations, and how you keep it fresh, fast, private and measurably good in production.

Requirements

• Functional:
  • Ingest documents from several sources: parse, chunk, embed and index them.
  • Retrieve with vector search, keyword search or both, then rerank.
  • Build the prompt, generate an answer that cites its sources, and say so when the documents do not support an answer.
  • Respect access control: a user retrieves and sees only what they may read.
• Non-functional: answers in a few seconds, streamed; new or changed documents searchable within minutes; sensitive data protected; cost per query known.

Clarifying questions

• How many documents and users, and how often do documents change?
• What kinds of questions: lookups of a fact, or summaries across many documents?
• Are there per-document permissions, and are some sources regulated?
• What latency and cost per query are acceptable?
• Is there a set of real questions with known answers to evaluate against?

What a strong answer covers

• Ingestion: connectors, text extraction (including tables and scans), chunking with overlap and section titles, metadata and permissions carried on every chunk, versioning and deletes.
• Retrieval: hybrid lexical and vector search, filtering by permission before ranking, a cross-encoder reranker, and how many chunks reach the prompt.
• Generation: a prompt that asks for citations, checking that each citation points at a retrieved chunk, and abstaining when the evidence is thin.
• Freshness: incremental re-indexing on change events and removing deleted content promptly.
• Evaluation: retrieval recall and precision on a labelled set, answer faithfulness and correctness (human review plus model-graded checks), and online signals such as thumbs and follow-up questions.
• Scale, cost and monitoring: caching, model choice per query type, index sharding, latency per stage, and drift in the questions people ask.

Follow-ups

• Users report confident wrong answers. How do you find the cause: retrieval, the prompt or the model?
• A document is deleted for legal reasons. How fast can you guarantee it stops appearing?
• How would you support questions that need data from a database, not only text?