Darkmind / Issue 042

Warm the Cache Before You Fan Out

Turn the first request for a reusable AI context into a verified warm-up step, then release parallel work only after the cache is ready—or fall back deliberately when it is not.

Nov 18, 20264 min readEfficient agent systems

A shared prompt prefix does not automatically mean a shared cache hit. This matters when an agent coordinator is about to launch a batch of parallel jobs—say, fifty document classifications that all need the same policy, tool definitions, examples, and reference packet. If every worker starts at once, they can all become cache-writing requests. You pay fresh-input or cache-write costs repeatedly, and the slowest cold start may still set the batch’s visible latency.

The practical move is to make cache warm-up an explicit stage before the fan-out. Send one bounded request containing the exact reusable prefix, wait until the provider reports that the cache is available, then release the workers that share it. Treat the warm-up result as a small receipt, not a hopeful delay: it should identify the prefix version, cache partition or routing key, response usage, expiry, and the observed cache-read result from a second request.

This is a coordination technique, not a new way to arrange a prompt. A stable prefix remains necessary, but it is not sufficient when many requests arrive together. Anthropic says a cache entry becomes available only after the first response starts; for concurrent requests that need hits, it recommends waiting for the first response before sending subsequent requests.[1] OpenAI similarly describes cache reuse as dependent on reaching a machine that holds an unexpired matching entry, with routing influenced by organization, region, model, request load, and the initial prompt content.[2]

Make the first request a deliberate warm-up

Choose a lightweight representative suffix: a harmless classification, extraction, or “ready” task that exercises the same model, tools, developer instructions, cache settings, and reusable context as the eventual workers. Do not warm the cache with a different prompt and assume it transfers. Cache matching is about the eligible rendered prefix, so a changed tool definition, reordered instruction, tenant-specific addition, or different processing region can turn a warm-up into an unrelated entry.[1][2]

The warm-up request needs an ordinary useful result only if your product has one. Its main job is to create the reusable state and return provider usage data. Then send a single probe with the same prefix plus a different tiny suffix. Release the batch only if the probe shows the expected cached-input or cache-read evidence for your provider.

The probe catches the expensive false positive: a successful warm-up that does not create a reusable entry.

Copy this cache warm-up gate

Put this gate in the coordinator immediately before a batch whose requests intentionally share context.

BATCH CACHE CONTRACT
- prefix_version: [hash/version of reusable instructions, tools, and corpus]
- model + region: [exact deployment]
- cache partition/routing key: [tenant-safe stable key, if supported]
- cache settings: [TTL, explicit/implicit breakpoints, retention]
- expected minimum cached tokens: [provider/model-specific threshold]
- batch size and deadline: [N jobs, start-by time]

WARM-UP
1. Build request = exact shared prefix + harmless warm-up suffix.
2. Send one request; record request ID, prefix_version, and usage fields.
3. Wait for the provider’s documented cache-ready point.
4. Send one probe = exact shared prefix + distinct tiny suffix.
5. Accept only if probe reports cache-read evidence at or above the expected threshold.

RELEASE RULE
- cache_ready: enqueue workers with the identical prefix contract.
- cache_not_ready but deadline allows: retry one warm-up with bounded backoff.
- cache_not_ready and deadline does not allow: run a deliberately limited cold batch,
  record the miss, and do not claim cache savings.

RECEIPT
{prefix_version, model, region, partition, warmed_at, expires_at,
 warm_usage, probe_usage, decision: cache_ready|cold_fallback|blocked}

Version the inputs that affect reuse: tool schemas, developer instructions, reference version, cache settings, and model. Partition tenant-specific policies before warming; a cache key is not a shortcut around data or billing boundaries. OpenAI says caches are not shared across organizations or regional processing boundaries; Anthropic describes workspace-level isolation.[1][2]

Release workers with a cache contract, not a vague hope

After the probe passes, preserve the prefix contract: job builders must not inject a timestamp, reorder tools, or add instructions before the cache boundary. Put job-specific text after it. Log cache-read evidence per worker and calculate hit rate by prefix version and partition.

A cache is an optimization, not durable state. Record its expiry estimate and re-run the gate after it passes. A miss may cost more or take longer, but must not change the answer or cross a tenant boundary.[1][2]

Failure mode: warming work that cannot be reused

Do not add a warm-up stage to a two-job batch with a small prompt, or to workers whose prefixes differ materially. The coordination overhead can exceed the saving. It can also create a thundering herd if every queue consumer independently decides to warm the same key.

Use one owner per {prefix_version, model, region, partition} and make other consumers await its short-lived receipt. If the warm-up fails, let one bounded retry happen; then choose cold fallback, a smaller batch, or a later run. The goal is not to force a hit. It is to make a cache miss visible before parallel work multiplies it.

The habit to keep

before a large agent fan-out, prove that the shared context is reusable in this provider, this partition, and this moment. Warm once, probe once, then release the batch with a receipt that can explain the cost.

Sources

  1. Anthropic, “Prompt caching”
  2. OpenAI, “Prompt caching”