Skip to main content
Machine learning, neural networks, LLMs, retrieval-augmented generation, model serving, and the systems engineering behind production AI.

Artificial Intelligence

Machine learning, neural networks, LLMs, retrieval-augmented generation, model serving, and the systems engineering behind production AI.

Retrieval-Augmented Generation

Trace a RAG answer from docs to citation

Follow a question through chunking, embedding, retrieval, ranking, and a grounded, cited generation.

Stage

Chunks
Query
Artifact
Why it matters

    Retrieval-Augmented Generation

    Intermediate (3/5) ~3–4 hours Chunking Embeddings Vector Database Retrieval Re-ranking Grounding Citations Hallucination Prereqs: LLMs & the Transformer

    Why RAG Exists

    An LLM’s knowledge freezes at training time and lives inside its weights. Retrieval-augmented generation (RAG) solves both problems at once: before answering, the system retrieves the relevant passages from your documents and feeds them into the prompt as context. The model answers from that context, not from memory.

    ConcernWithout RAGWith RAG
    Fresh informationModel doesn’t know itRetrieved live
    Private dataNever in trainingIndexed and retrieved
    HallucinationModel guessesGrounded in citations
    UpdatesRetrainingRe-index the docs

    The Pipeline

    1. Ingest — split source documents into small, self-contained chunks.
    2. Embed — convert each chunk into a dense vector.
    3. Index — store vectors in a vector database for fast search.
    4. Retrieve — embed the user’s question and fetch the top-k nearest chunks.
    5. Rank — optionally re-score the candidates with a precise cross-encoder.
    6. Generate — prompt the LLM with the retrieved context and demand citations.

    The RAG visualizer on this page steps through this exact flow.

    Chunking Trade-offs

    Chunk sizeProsCons
    Small (~200–400 tokens)Precise retrievalLoses surrounding context
    Large (~800–1000 tokens)Self-contained contextDiluted relevance

    Overlap neighboring chunks so a passage split across a boundary isn’t lost. Chunk boundaries should respect section structure where possible — splitting mid-sentence hurts retrieval.

    Retrieval and Ranking

    Similarity is measured with cosine similarity on normalized vectors: nearest neighbors in embedding space, not keyword matches. “Can I return this?” finds the refund policy even though the words differ.

    A cross-encoder re-ranker scores the small top-k set against the exact query and is far more accurate than the embedding score — at the cost of speed. Use it when precision matters more than latency.

    Grounded Generation

    The prompt tells the model: “Answer using only the provided context, cite the chunk ids you used, and say you don’t know if the context doesn’t contain the answer.” This converts hallucination from a probability into a detectable, auditable event — and the citations give users a path to verify.

    RAG vs Fine-tuning

    ApproachBest forCost
    RAGFacts, freshness, private knowledgeIndexing + retrieval infra
    Fine-tuningStyle, tone, format, domain fluencyTraining + eval + serving

    They’re complementary, not either/or. RAG changes what the model knows; fine-tuning changes how it behaves. Most production systems use both: fine-tune the behavior, RAG the facts.

    Practice Trajectory

    1. Design chunking for a 40-page product manual and justify the size.
    2. Explain why a user’s “return” question can match a “refund” chunk.
    3. List two failure modes of retrieval and how re-ranking fixes them.
    4. Write a system prompt that forces cited, grounded answers.
    5. Decide whether to fine-tune or RAG for a support chatbot over an internal wiki.