Darkmind / Issue 033

Partition a Semantic Cache Before You Trust It

Treat a semantic-cache hit as a candidate answer that must match the caller’s policy partition, source version, and output contract before it can bypass a fresh model call.

Nov 9, 20264 min readCost-aware AI

A normal cache reuses an answer only when a key matches exactly. A semantic cache takes a riskier shortcut: it embeds a new request, finds an earlier request with similar meaning, and may return the earlier answer without calling the model. That can cut latency and repeated inference work for FAQ-shaped traffic, but it also creates a new kind of wrong answer. “How do I return an item?” and “What is the return policy for this enterprise contract in France?” may be close in vector space while needing different evidence, permissions, or policy terms.

The practical move is to partition a semantic cache before you let it answer. Similarity should search only inside a hard compatibility boundary: tenant, audience or entitlement, locale, knowledge/source version, workflow and output contract, model or policy version, and safety state. Then use a thresholded nearest-match result as a candidate, not proof. A hit that is semantically close but outside any required boundary is a miss.

Microsoft’s semantic-cache policy explicitly warns that similarity-based reuse can surface responses that are incorrect, outdated, or unsafe for the current request.[1] Its policy exposes a vary-by partition and recommends using user or user-group identifiers to control cross-user access; it also says a lower threshold is appropriate for sensitive use cases.[1] Redis describes the same design in data terms: combine soft similarity with hard metadata boundaries such as tenant, locale, model version, and safety flags, and expire entries with a TTL.[3] The important distinction is that an embedding score is a relevance signal, not authorization to replay an answer.

Decide what must match exactly

Start from the properties that could make a prior answer inapplicable even when the question means nearly the same thing. A support bot might require an exact match on tenant_id, plan, locale, knowledge_version, response_schema_version, and safety_policy_version. A developer assistant might instead partition by repository, branch, dependency lockfile digest, tool-surface fingerprint, and permission scope.

Do not put these values only in a prompt that asks the model to be careful. Make them structured metadata on both the request and the stored entry, and apply them inside the cache lookup. A cache shared across users must not rely on a later application check that can be skipped or implemented differently by another caller.

Copy this semantic-cache admission card

Put this in front of a read-only answer route. Keep writes, approvals, personalized recommendations, and time-sensitive decisions on a fresh path unless you have separately designed their cache semantics.

INCOMING REQUEST
- normalized_question: [text after deterministic normalization]
- partition: {
    tenant_id, audience_or_entitlement, locale,
    workflow_id, output_contract_version,
    knowledge_version, model_or_policy_version, safety_class
  }
- as_of: [request time or explicit effective date]

LOOKUP (trusted cache service)
1. Require exact equality for every partition field.
2. Search only eligible entries for the nearest question embedding.
3. Require score >= [validated threshold for this workflow].
4. Reject entries that are expired, invalidated, unverified, or carry a
   different safety class or output-contract version.
5. On a hit, return the stored answer plus cache_entry_id, source/version
   receipt, score, created_at, and expires_at.
6. On a miss, run the normal retrieval/model path; store only an answer that
   passed its normal evidence and safety checks.

CALLER RULES
- A cache hit may answer only this read-only workflow and partition.
- Never treat a hit as permission to call a tool, send, publish, or change data.
- Log hit | miss | rejected_partition | rejected_threshold with reasons.

Notice that knowledge_version belongs in the key boundary, not only in the cache’s expiration setting. A short TTL limits how long an answer can survive, but it cannot know that a policy was just corrected or a product flag changed. Invalidate the affected partition or advance its version when the underlying corpus, policy, model behavior, or output schema changes. Microsoft’s setup guide separates lookup and store policies and recommends tracing test calls to confirm that cache reuse actually happens.[2] Extend that test: prove that a changed partition does not reuse the old entry.

Tune with false hits, not only hit rate

A high hit rate can mean the threshold is useful—or that the cache is serving broad near-matches. Build a small test set containing paraphrases that should reuse, lookalike questions that must miss, cross-tenant requests, different locales, changed policy versions, and questions whose answer is time-sensitive. Review the top candidate and the rejected candidates, not merely the dashboard total.

Set the threshold per workflow. A low-consequence internal FAQ can tolerate more reuse than a benefits, billing, security, or compliance answer. Microsoft cautions that thresholds above 0.2 may cause mismatches in its policy and specifically calls for lower values in sensitive uses; treat that as a starting point for evaluation, not a universal number.[1] If an answer needs fresh retrieval to establish its date, scope, or evidence, semantic similarity should not bypass that work.

Failure mode: caching a polished mistake

A semantic cache can amplify an answer that was fluent but unverified when it was first stored. It can also freeze a response after its source changes. Only admit answers that already passed the normal evidence, policy, and output validation for their workflow. Store their source or corpus version and validation receipt beside the text; cache entries are disposable accelerators, not a second source of truth.

The habit to keep

let similarity find a possible reuse, but make exact partitions, freshness, and validation decide whether that old answer is eligible for this caller now.

Sources

  1. Microsoft Learn, “llm-semantic-cache-lookup policy”
  2. Microsoft Learn, “Enable semantic caching for LLM APIs”
  3. Redis, “Semantic cache”