Darkmind / Issue 036

Make Each Retrieval Slot Earn a Different Job

Diversify a broad candidate set before it reaches the model, so a fixed context budget covers distinct evidence instead of five near-duplicate chunks.

Nov 12, 20264 min readRetrieval design

A retrieval system can return five excellent matches and still give an AI a weak evidence set. This happens when the matches are neighboring chunks from one manual, repeated wording from one policy, or several versions of the same answer. The model receives five ways to repeat one claim, while the exception, definition, date, or conflicting source that would make the answer safe never reaches the prompt.

The practical move is to diversify after you retrieve candidates and before you assemble context. First retrieve more candidates than you intend to send to the model. Then select a small set that remains relevant to the question while making each new item sufficiently different from the items already chosen. Maximal Marginal Relevance (MMR) is one established way to make that trade-off: Qdrant describes it as iterative selection that balances similarity to the query with separation from already selected points.[1] Elasticsearch uses MMR diversification to prevent similar documents from dominating top results and explicitly calls out RAG as a use case for reducing prompt redundancy.[2]

This is not a replacement for filters, source authority, or answerability checks. Use those first. A diverse set of sources from the wrong tenant, wrong product version, or expired policy is still the wrong set. Diversification is the next question: once the eligible candidates are relevant, are you spending your limited context on distinct evidence?

Start with a larger candidate pool

Do not run MMR over every item in a corpus. Start with a normal, scoped retrieval pass: apply permissions, tenant, date, product, and version boundaries; then retrieve a modest pool of candidates using your normal keyword, vector, or hybrid path. That pool is where relevance has already done its job.

Next decide what counts as “too similar” for the task. For chunked documentation, it may be embedding similarity between chunks. For policy research, it may be chunks from the same document section or source revision. For incident search, it could be the same service, incident ID, or log signature. Keep an origin field—such as document_id, section_id, version, and chunk_id—with every candidate. A vector score alone cannot tell you that three chunks are merely adjacent paragraphs from the same page.

Qdrant exposes both a candidate limit and a diversity setting for MMR, where the setting controls the relevance-versus-diversity balance.[1] Elasticsearch’s diversify retriever likewise takes results from an inner retriever, applies MMR to the top-N window, and exposes a lambda parameter for the trade-off.[2] You do not need either product to use the workflow. The useful design is retrieve broadly, select deliberately, then send the selected evidence with its source metadata.

Copy this context-diversity card

Use this after eligibility filters and initial retrieval, but before the answer model sees the passages.

QUESTION: [exact user question]
REQUIRED ANSWER PARTS: [object, scope, fact, exception, date]

1. RETRIEVE CANDIDATES
   - Apply access, tenant, product, version, and time boundaries first.
   - Retrieve [20–50] candidates with normal keyword/vector/hybrid ranking.
   - Preserve: candidate_id, source_id, document_id, section_id, version,
     locator, relevance_score, excerpt.

2. SELECT A DIVERSE EVIDENCE SET
   - Select at most [K] items, one at a time.
   - First item: highest eligible relevance.
   - Each later item must both:
       a. support a named unanswered part of QUESTION, and
       b. add information not already represented by the selected set.
   - Penalize near-duplicates by embedding similarity and by shared origin
     (same document/section/version), unless corroboration is required.
   - Keep a candidate only if it has a distinct role:
       governing rule | definition | exception | date/version | corroboration.

3. RETURN A SELECTION RECEIPT
   {selected: [{candidate_id, source_id, locator, role, supports}],
    excluded_as_redundant: [{candidate_id, similar_to}],
    uncovered_requirements: [...],
    next_safe_action: answer | retrieve_for_named_gap | needs_verification}

4. DRAFTING RULE
   - Cite the selected source beside each supported claim.
   - Do not treat diversity as proof; inspect the selected excerpts.
   - If a required answer part remains uncovered, state that gap or retrieve
     specifically for it.

The role field prevents diversity from becoming a cosmetic score tweak. Suppose the question asks whether a retention rule applies to backups in a named plan. One chunk may state the general duration, another may define the plan, and a third may state the backup exception. Three near-identical chunks about the duration should not displace the exception. Conversely, if three independent sources genuinely corroborate a disputed fact, preserve them intentionally and label them as corroboration rather than pretending they are three distinct answer parts.

Tune the balance with missed evidence, not aesthetics

More diversity is not automatically better. Qdrant notes that its diversity control spans relevance at one end and diversity at the other; Elastic documents that higher lambda weights query similarity more heavily, while lower values weight diversity more heavily.[1][2] An overly diverse selection can replace the governing passage with loosely related material. A nearly relevance-only selection can fill the prompt with duplicates.

Test with real questions that have multiple required parts. For each, mark the minimum evidence set a reviewer needs: perhaps the rule, the scope qualifier, and an exception. Compare ordinary top-K with diversified top-K. Review not only answer quality, but whether the selection receipt exposes the missing part and whether the selected context includes redundant sources. Tune candidate-pool size, selected-K, and the diversity balance separately; changing all three at once makes a result hard to explain.

Failure mode: confusing variety with coverage

MMR measures difference, not truth, authority, freshness, or full answer coverage. It can select a novel but irrelevant chunk, and it can wrongly suppress repeated text that is actually needed to resolve a cross-reference. Never use it to bypass source filters, version checks, or an answerability gate. Keep the original ranking and the selection receipt so a reviewer can see what was excluded and why.

The habit to keep

do not let your context window become an echo chamber. Retrieve eligible candidates first, then make every slot contribute a different piece of evidence the answer actually needs.

Sources

  1. Qdrant, “Search Relevance”
  2. Elasticsearch, “Diversify retriever”