Darkmind / Issue 042

Warm the Cache Before You Fan Out

Turn the first request for a reusable AI context into a verified warm-up step, then release parallel work only after the cache is ready—or fall back deliberately when it is not.

Nov 18, 20264 min readEfficient agent systems

A shared prompt prefix does not automatically mean a shared cache hit. This matters when an agent coordinator is about to launch a batch of parallel jobs—say, fifty document classifications that all need the same policy, tool definitions, examples, and reference packet. If every worker starts at once, they can all become cache-writing requests. You pay fresh-input or cache-write costs repeatedly, and the slowest cold start may still set the batch’s visible latency.

The practical move is to make cache warm-up an explicit stage before the fan-out. Send one bounded request containing the exact reusable prefix, wait until the provider reports that the cache is available, then release the workers that share it. Treat the warm-up result as a small receipt, not a hopeful delay: it should identify the prefix version, cache partition or routing key, response usage, expiry, and the observed cache-read result from a second request.

This is a coordination technique, not a new way to arrange a prompt. A stable prefix remains necessary, but it is not sufficient when many requests arrive together. Anthropic says a cache entry becomes available only after the first response starts; for concurrent requests that need hits, it recommends waiting for the first response before sending subsequent requests.[1] OpenAI similarly describes cache reuse as dependent on reaching a machine that holds an unexpired matching entry, with routing influenced by organization, region, model, request load, and the initial prompt content.[2]

Make the first request a deliberate warm-up

Choose a lightweight representative suffix: a harmless classification, extraction, or “ready” task that exercises the same model, tools, developer instructions, cache settings, and reusable context as the eventual workers. Do not warm the cache with a different prompt and assume it transfers. Cache matching is about the eligible rendered prefix, so a changed tool definition, reordered instruction, tenant-specific addition, or different processing region can turn a warm-up into an unrelated entry.[1][2]

The warm-up request needs an ordinary useful result only if your product has one. Its main job is to create the reusable state and return provider usage data. Then send a single probe with the same prefix plus a different tiny suffix. Release the batch only if the probe shows the expected cached-input or cache-read evidence for your provider.

The probe catches the expensive false positive: a successful warm-up that does not create a reusable entry.

Copy this cache warm-up gate

Put this gate in the coordinator immediately before a batch whose requests intentionally share context.

BATCH CACHE CONTRACT
- prefix_version: [hash/version of reusable instructions, tools, and corpus]
- model + region: [exact deployment]
- cache partition/routing key: [tenant-safe stable key, if supported]
- cache settings: [TTL, explicit/implicit breakpoints, retention]
- expected minimum cached tokens: [provider/model-specific threshold]
- batch size and deadline: [N jobs, start-by time]

WARM-UP
1. Build request = exact shared prefix + harmless warm-up suffix.
2. Send one request; record request ID, prefix_version, and usage fields.
3. Wait for the provider’s documented cache-ready point.
4. Send one probe = exact shared prefix + distinct tiny suffix.
5. Accept only if probe reports cache-read evidence at or above the expected threshold.

RELEASE RULE
- cache_ready: enqueue workers with the identical prefix contract.
- cache_not_ready but deadline allows: retry one warm-up with bounded backoff.
- cache_not_ready and deadline does not allow: run a deliberately limited cold batch,
  record the miss, and do not claim cache savings.

RECEIPT
{prefix_version, model, region, partition, warmed_at, expires_at,
 warm_usage, probe_usage, decision: cache_ready|cold_fallback|blocked}

Version the inputs that affect reuse: tool schemas, developer instructions, reference version, cache settings, and model. Partition tenant-specific policies before warming; a cache key is not a shortcut around data or billing boundaries. OpenAI says caches are not shared across organizations or regional processing boundaries; Anthropic describes workspace-level isolation.[1][2]

Release workers with a cache contract, not a vague hope

After the probe passes, preserve the prefix contract: job builders must not inject a timestamp, reorder tools, or add instructions before the cache boundary. Put job-specific text after it. Log cache-read evidence per worker and calculate hit rate by prefix version and partition.

A cache is an optimization, not durable state. Record its expiry estimate and re-run the gate after it passes. A miss may cost more or take longer, but must not change the answer or cross a tenant boundary.[1][2]

Failure mode: warming work that cannot be reused

Do not add a warm-up stage to a two-job batch with a small prompt, or to workers whose prefixes differ materially. The coordination overhead can exceed the saving. It can also create a thundering herd if every queue consumer independently decides to warm the same key.

Use one owner per {prefix_version, model, region, partition} and make other consumers await its short-lived receipt. If the warm-up fails, let one bounded retry happen; then choose cold fallback, a smaller batch, or a later run. The goal is not to force a hit. It is to make a cache miss visible before parallel work multiplies it.

The habit to keep

before a large agent fan-out, prove that the shared context is reusable in this provider, this partition, and this moment. Warm once, probe once, then release the batch with a receipt that can explain the cost.

Sources

  1. Anthropic, “Prompt caching”
  2. OpenAI, “Prompt caching”

Previous issues

  • Issue 041 · Nov 17, 2026 · Agent reliability
    Make Every Tool Argument Point Back to Evidence

    Require an agent to attach an allowed evidence reference to every consequential tool argument, then have code construct the final payload from those references instead of trusting reconstructed values.

  • Issue 040 · Nov 16, 2026 · Agent design
    Tell the Agent When Not to Use a Tool

    Put exclusion rules, safe alternatives, and output limits in each tool definition so an agent can reject a tempting near-match before it produces a bad call.

  • Issue 039 · Nov 15, 2026 · Research workflows
    Name the Unknowns Before You Start AI Research

    Turn a broad request into a short decision frame that separates fixed requirements from open variables, so an AI researches the question you actually need answered instead of a plausible substitute.

  • Issue 038 · Nov 14, 2026 · Agent design
    Draw the Dependency Line Before You Parallelize Agents

    Split only the work that can finish from the same stable input, then give each parallel agent a bounded deliverable and a single synthesis rule.

  • Issue 037 · Nov 13, 2026 · Prompt operations
    Put Every Production Prompt in a Versioned Helper

    Treat a production prompt as a typed, reviewed, testable code artifact, so a prompt edit has an owner, a diff, a fixture, and a rollback path.

  • Issue 036 · Nov 12, 2026 · Retrieval design
    Make Each Retrieval Slot Earn a Different Job

    Diversify a broad candidate set before it reaches the model, so a fixed context budget covers distinct evidence instead of five near-duplicate chunks.

  • Issue 035 · Nov 11, 2026 · Document workflows
    Locate Before You Read the PDF

    Turn a large PDF question into a page-finding pass and a visual-evidence pass, so the model spends attention on the pages that can actually support the answer.

  • Issue 034 · Nov 10, 2026 · Agent reliability
    Validate the Tool Result Before It Enters the Agent Loop

    Put a typed result gate between every tool response and the model, so malformed, partial, stale, or out-of-scope data becomes a safe receipt instead of unexamined context.

  • Issue 033 · Nov 9, 2026 · Cost-aware AI
    Partition a Semantic Cache Before You Trust It

    Treat a semantic-cache hit as a candidate answer that must match the caller’s policy partition, source version, and output contract before it can bypass a fresh model call.

  • Issue 032 · Nov 8, 2026 · Agent security
    Put an Egress Gate Before an Agent Calls an External Tool

    Check every model-to-tool payload against a destination-specific data policy before it leaves the application, so an agent cannot turn useful context into an accidental disclosure.

Browse all 42 issues →