RAG retrieval: get the right chunk into the top k

The model in a RAG (retrieval-augmented generation) feature sees the chunks its search returns, and an answer cannot use a chunk that never arrived. Build retrieval in steps: clean and chunk the documents, filter by permission, search by keyword and vector together, rerank, and cite the chunk used. Then test retrieval on its own, by checking whether the right chunk comes back in the top k for real questions.

The techniques below come from each vendor’s official docs and Anthropic’s contextual retrieval post, last verified on 1 November 2026. None of it was tested for this post.

This is part 29 of the series. Part 26 built an eval suite for a whole LLM feature. This part looks at the retrieval step inside it, which can fail quietly while the answer still sounds confident.

The retrieval pipeline, from raw document to cited chunk

A retrieval pipeline has seven stages, and each one can lose the right chunk in its own way. Anthropic’s contextual retrieval post describes the usual setup. It splits the corpus into chunks “usually no more than a few hundred tokens”, embeds them, and stores them in a vector database.

Every stage after the first adds settings: chunk size, overlap, k, fusion weights, the reranker’s cut-off. None of them has a right value in general. The retrieval test at the end of this post is where you choose them.

Clean and chunk documents so each chunk stands alone

Clean the text before you split it

Ingestion is where bad text enters the index, and every later stage inherits it. Strip navigation, cookie banners, footers and repeated boilerplate, so those words stop matching every query. Keep the headings, because they tell a chunk what it belongs to.

Hidden text can do more than add noise. OWASP’s LLM08 Vector and Embedding Weaknesses describes a résumé with white-on-white instructions that a RAG screening system later follows. Its fix calls for “text extraction tools that ignore formatting and detect hidden content”. It also asks for every document to be validated before it joins the knowledge base.

Old copies cause a quieter failure. The same OWASP page warns that “Data federation knowledge conflict errors can occur when data from multiple sources contradict each other”. If last year’s returns policy is still in the index, search can rank it above this year’s. Index one current version of each document, and delete the chunks of a version you replace.

Record the metadata at ingestion, while you still know the source. An example chunk record, with made-up names and values:

{
  "chunk_id": "returns-policy@v7#damaged-items-2",
  "passage_id": "returns-policy#damaged-items",
  "doc_version": 7,
  "title": "Returns policy",
  "section": "Refunds > Damaged items",
  "source_url": "https://help.example.com/returns#damaged-items",
  "allowed_groups": ["support-staff", "customers-eu"],
  "language": "en",
  "updated_at": "2026-09-14",
  "embedding_model": "<model name and version>",
  "text": "Returns policy > Refunds > Damaged items: ..."
}

The passage_id stays the same when you re-chunk, so test labels survive a change of chunk size. The allowed_groups field comes from the source system’s own permissions. Copy it at ingestion, and update it when those permissions change.

Chunk size is a trade-off you measure

Chunking decides what one search result can carry. Small chunks match a question closely, but a small chunk can lose the context that makes it useful. Large chunks keep that context, but one vector then has to stand for several ideas, and each chunk costs more tokens in the prompt.

Anthropic’s contextual retrieval post shows the context problem with one line from a financial filing, “The company’s revenue grew by 3% over the previous quarter”. That chunk “doesn’t specify which company it’s referring to or the relevant time period”. Its fix is contextual retrieval, which prepends “chunk-specific explanatory context to each chunk before embedding” and before building the keyword index. The post generates that context with a model call per chunk, so it adds an ingestion cost.

Microsoft’s chunking guide for Azure AI Search lists the common methods:

• Fixed-size chunks with some overlap.
• Variable-size chunks, split at sentence ends or at headings in HTML or Markdown.
• Semantic chunks that keep related sentences together.
• Custom combinations. One example is to “append the document title to chunks from the middle of the document to prevent context loss”.

Starting points differ between vendors. OpenAI’s retrieval guide says vector store files are split into 800-token chunks with 400 tokens of overlap by default. The Azure chunking guide recommends starting with 512 tokens and 25 percent overlap. Two vendors, two starting points, and neither has seen your documents.

Anthropic’s contextual retrieval post notes that “The choice of chunk size, chunk boundary, and chunk overlap can affect retrieval performance”. Pick two or three candidate settings, index the same documents with each, and run the retrieval test on all of them.

Embed queries and documents with the same model and settings

An embedding is a list of numbers that stands for the meaning of a piece of text, so similar meanings land close together. Retrieval compares the query’s embedding with each chunk’s. That comparison works when both came from the same model, with the settings the provider documents for queries and for documents.

Each row comes from the linked docs, last verified on 1 November 2026; none was tested for this post. The Voyage AI details come from Anthropic’s page, not from Voyage’s own docs.

Changing the embedding model means embedding the whole corpus again. Google’s migration note says “The embedding spaces between gemini-embedding-001 and gemini-embedding-2 are incompatible”. An upgrade means “you must re-embed all of your existing data”. Store the model name on every chunk, as in the record above, and refuse to search an index that mixes two.

Shorter vectors are cheaper to store and search. Google’s embeddings docs say a smaller output dimensionality can “save storage space and increase computational efficiency”, “while sacrificing little in terms of quality”. Check “little” on your own test set before you shrink them. Anthropic’s embeddings page says the same about choosing a provider: “you should assess a variety of embeddings vendors to find the best fit for your specific use case.”

Filter by metadata, including who may see each chunk

Metadata filters narrow a search before ranking decides anything. Language, product, region, document version and date are common fields. Permission is the field that matters most, because a chunk the user may not see should never reach the prompt.

Take the user’s groups from the signed-in session, and pass them as a filter in the search request. The model never chooses them. Microsoft’s security filter pattern for Azure AI Search stores group IDs in a filterable field and queries with group_ids/any(g:search.in(g, 'group_id1, group_id2')). The same page notes that the principal “is just a string”. Your code decides which groups go into it, so that code needs its own tests.

Other APIs take the same shape. OpenAI’s retrieval guide filters on file attributes “before performing semantic search”. Gemini’s File Search takes a metadata_filter string such as author="Robert Graves".

Where the filter runs changes what comes back. Microsoft’s vector filter docs describe three modes:

• preFilter applies the filter during the search. It “guarantees that k results are returned if they exist in the index”, and costs more when few documents match.
• postFilter filters each shard’s top k. It “can create false negatives for highly selective filters or small k values.”
• strictPostFilter (preview) filters the global top k, and “can return zero results (even if matches exist)”.

A permission filter is often highly selective. A user in one small team can see a sliver of the index. Filter after the top k, and that user’s right chunk can fall outside it, so they get fewer results or none.

Hybrid search and reranking find the chunk vectors miss

Keyword and vector search miss different things

Vector search finds chunks that mean the same as the question, even with no shared words. Keyword search, usually scored with BM25, finds exact words. Anthropic’s contextual retrieval post gives an example. For a query about “Error code TS-999”, an embedding model “might find content about error codes in general, but could miss the exact “TS-999” match.”

Microsoft’s hybrid search overview names more cases: “product codes, highly specialized jargon, dates, and people’s names”. These “perform better with keyword search because it can identify exact matches”. Hybrid search runs both searches and merges the two ranked lists.

Reciprocal rank fusion (RRF) is the usual way to merge them. Elastic’s RRF docs give the formula: each list adds 1 / (k + rank) to a document’s score. This k is a ranking constant, rank_constant, 60 by default, and has nothing to do with the top k. RRF works on ranks, so a BM25 score and a cosine similarity never have to be compared.

Elastic’s RRF docs say “RRF requires no tuning”. Products still expose weights. OpenAI’s retrieval guide lets you set rrf_embedding_weight and rrf_text_weight. Elastic’s rank_window_size sets how deep each list goes, and “A higher value will improve result relevance at the cost of performance”. Leave them at the defaults until your test set shows a reason to move them.

Rerank a wider first pass

A reranker reads the query and each candidate together, and orders the candidates by relevance. Cohere’s Rerank overview puts it as: “Given a query and a list of documents, Rerank indexes the documents from most to least semantically relevant to the query”. The common pattern is to fetch a wide candidate set with hybrid search and keep the top few after reranking.

Reranking costs time on every request. Anthropic’s contextual retrieval post says it “inevitably adds a small amount of latency”. It describes “an inherent trade-off between reranking more chunks for better performance vs. reranking fewer for lower latency and cost.” Microsoft’s hybrid search docs suggest setting k to 50 for the vector query when you use its semantic ranker.

Reranker scores are for ordering. Cohere’s best practices warn that “you can’t assume that a document with a relevance score of 0.9109375 is twice as relevant as one with a relevance score of 0.04421997”. To set a cut-off, the same page suggests 30 to 50 representative queries, each paired with a borderline-relevant document. Use their average score as a reference.

Citations that point to the retrieved chunk

A citation lets a reader check the answer against the source, and it lets you check that the answer came from retrieval. Send each chunk to the model with its ID and its source, so the citation can point back to it.

Anthropic, OpenAI and Gemini each return citations in a structured form:

• Anthropic’s search result content blocks carry your own retrieved chunks, and “each citation carries the source and title you provided”. Its citations docs say citations “are guaranteed to contain valid pointers to the provided documents”. The same page says citations cannot be used together with structured outputs.
• OpenAI’s file search adds file_citation annotations with a file_id and filename. The tool “will not return search results by default”; ask for them with include: ["file_search_call.results"].
• Gemini’s File Search returns file_citation annotations, and a page_number for documents with pages.

A valid pointer says where a quote came from. Whether that quote supports the sentence beside it is a question about the answer, and it belongs in the answer eval from Part 26.

Retrieval has citation checks of its own. Every cited chunk ID should be in the set retrieval returned for that request. Every chunk in that set should also pass the user’s permission filter. Record the retrieved chunk IDs on each request’s trace, so both checks can run on production traffic too. Part 21 lists which retrieval fields a trace records by default and which are opt-in.

Test retrieval on its own, before you judge the answer

A retrieval test asks one question per case: did the right passage come back in the top k? It needs no model call and no judge, so it runs fast and gives the same result every time. When an answer is wrong, it also tells you whether search or the model is to blame.

Elastic’s ranking evaluation docs define two useful metrics:

• Recall at k is “the fraction of relevant documents in those first k results relative to all possible relevant results”. The page notes it “does not take into account the position of the relevant documents within the top k results”.
• Mean reciprocal rank (MRR) takes “the reciprocal of the rank of the first relevant document”, averaged over all queries. A first hit at position 3 scores 1/3.

Anthropic’s contextual retrieval experiments use the same idea from the other side. They report “1 minus recall@20”, “the percentage of relevant documents that fail to be retrieved within the top 20 chunks.”

Build the test set from real questions, such as search logs and support tickets. Label each one with the passages that answer it. Add cases for exact codes, cases with no answer in the index, and cases where the user’s groups should hide the answer. An example test set, with made-up questions and IDs:

{"id": "q-001", "question": "Can I return a damaged kettle after 30 days?", "relevant": ["returns-policy#damaged-items"], "groups": ["customers-eu"], "kind": "paraphrase"}
{"id": "q-002", "question": "error E-4012 at checkout", "relevant": ["payment-errors#e-4012"], "groups": ["customers-eu"], "kind": "exact-code"}
{"id": "q-003", "question": "What is the staff discount?", "relevant": [], "forbidden": ["staff-handbook#discounts"], "groups": ["customers-eu"], "kind": "permission"}

An example script that scores those cases, given your own retrieve function:

import json

def evaluate(path, retrieve, k=10):
    recalls, ranks, leaks = [], [], []
    for line in open(path, encoding="utf-8"):
        case = json.loads(line)
        hits = retrieve(case["question"], groups=case["groups"], k=k)
        found = [hit["passage_id"] for hit in hits]
        if set(found) & set(case.get("forbidden", [])):
            leaks.append(case["id"])  # a permission failure, whatever the score
        relevant = set(case["relevant"])
        if not relevant:
            continue
        recalls.append(len(relevant & set(found)) / len(relevant))
        first = next((i + 1 for i, p in enumerate(found) if p in relevant), None)
        ranks.append(1 / first if first else 0.0)
    return {
        "recall_at_k": sum(recalls) / len(recalls),
        "mrr": sum(ranks) / len(ranks),
        "permission_leaks": leaks,
    }

Run it for each setting you are choosing between: chunk size, vector only against hybrid, with and without the reranker, a few values of k. Change one thing per run. Split the results by kind, because hybrid search can help exact codes and do nothing for paraphrases. A permission leak fails the run whatever the scores say.

Keep this test in the same suite as your answer evals from Part 26. Run it whenever the documents, the chunking or the embedding model change.

Two prompts for auditing and testing RAG retrieval

Both prompts work in any agent that can read your repository, with no tool-specific features.

Prompt 1: audit a RAG retrieval pipeline

An example prompt for mapping an existing retrieval pipeline before you change anything.

Task: audit the retrieval pipeline in <the service or path>.

Do not change code. Do not call model, embedding or search APIs.
For each stage, find the code and answer the questions:
1. Ingest: how is text extracted? Is boilerplate or hidden text
   removed? How are old versions of a document removed?
2. Chunk: which method, size and overlap? Do chunks keep their
   title and section?
3. Embed: which model? Is the same model used for queries and
   documents? Is the model name stored with each chunk?
4. Filter: which metadata fields exist? Where do the user's groups
   come from? Does the filter run before or after the top k?
5. Search: keyword, vector or both? How are the lists merged?
6. Rerank: which reranker, how many candidates in, how many out?
7. Cite: are chunk IDs and sources sent to the model? Are cited IDs
   checked against the retrieved set?
8. Test: is there a retrieval test separate from the answer evals?

Cite the file and line for each answer. Mark anything you could not
confirm as unverified.
Never print, expose or commit secret values.
Do not commit, push, deploy, run destructive commands or make external
changes unless I explicitly authorise it.
If two reasonable attempts fail, stop and report the failure rather
than repeatedly guessing.

End with the three gaps most likely to return the wrong chunk.

Check its answer to question 4 yourself. That one decides who sees what.

Prompt 2: build a retrieval test set

An example prompt for drafting test cases from your own questions and documents. A person still reviews every label.

Task: draft a retrieval test set for <the RAG feature>.
Questions come from: <search logs, tickets or a file of questions>
Documents are in: <path or index name>

1. Pick questions that cover the main topics. Keep the users' own
   wording.
2. For each, find the passages that answer it, and record their
   passage IDs. If none answers it, record an empty list.
3. Add cases for exact codes, product names and dates.
4. Add cases where a user group must not see the answer, with the
   forbidden passage IDs.
5. Write the cases as JSON Lines with id, question, relevant,
   forbidden, groups and kind.
6. Mark every label you are unsure of for human review.

Remove personal data from questions before writing them.
Never print, expose or commit secret values.
Do not commit, push, deploy, run destructive commands or make external
changes unless I explicitly authorise it.
If two reasonable attempts fail, stop and report the failure rather
than repeatedly guessing.

The draft labels are guesses until someone who knows the documents checks them.

Quick answers

What is hybrid search in RAG?

Hybrid search in RAG (retrieval-augmented generation) runs a keyword search, often scored with BM25, and a vector search on the same query. It merges the two ranked lists, usually with reciprocal rank fusion, so exact codes and paraphrased questions can both find their chunk.

What chunk size should a RAG system use?

A RAG (retrieval-augmented generation) system has no chunk size that suits every corpus. Small chunks match closely but can lose context, and large chunks keep context but blur the match. Index your documents with a few settings and compare recall at k.

What does a reranker do in a RAG pipeline?

A reranker scores each candidate chunk against the query and reorders them, so the most relevant chunks reach the prompt. It runs on every request and adds latency, so teams pass it a wider candidate set and keep the top few.

How do you evaluate RAG retrieval separately from the answer?

RAG (retrieval-augmented generation) retrieval is evaluated with a set of real questions, each labelled with the passages that answer it. Recall at k checks whether those passages appear in the top k, and mean reciprocal rank checks how high the first one ranks.

Should permission filters run before or after vector search?

Permission filters in a vector search should run during the search, before the top k is chosen. A filter applied after the top k can drop every allowed chunk for a user with narrow access, leaving fewer results or none.

More on AI coding is in the AI category.