Prompt caching in action

Prompt Caching Study Notes

**I. Core Concepts & Threshold**

• Purpose: Prompt caching stores the results of analyzing long, static inputs (like system prompts and tool schemas) to speed up processing and reduce costs.
• Minimum Threshold: Caching is typically triggered when the total input (prompt + schemas) exceeds 1,024 tokens.
• Key Metrics: The response object includes a usage field, which reports cache_read_input_tokens and cache_write_input_tokens to track caching activity.

**II. Implementation Steps**

• Caching the System Prompt:
• Add a cache_point part of type default to the list of prompt parts.
• This ensures the system prompt is cached consistently across requests.
• Caching Tool Schemas:
• Create a dedicated variable (e. g. , tools_with_cache) that concatenates the list of tools with a cache_point part of type default.
• Pass this cached list of tools into the request.

**III. Caching Behavior**

• Cache Write: Occurs during the first request when the system analyzes and stores the input (e. g. , the initial system prompt).
• Cache Read: Occurs on subsequent requests when the system retrieves the pre-analyzed result from the cache instead of re-processing the input.
• Impact of Changes:
• Changing the user's text prompt (the user message) does not invalidate the system prompt cache; a cache read continues.
• Changing the system prompt (even slightly) invalidates the cache, resulting in a new cache write.
• Cache Expiration: Cache entries are automatically cleared if no request is made within a five-minute window.

**IV. Benefits**

• Efficiency: Significantly reduces the computational load and generation time for requests involving long, static inputs.
• Cost Savings: Reduces the overall cost of text generation by avoiding redundant processing of large system prompts and tool schemas.

Takeaways

• Prompt caching speeds up processing and reduces costs by storing results of long, static inputs like system prompts and tool schemas.
• Caching is typically triggered when the total input (prompt plus schemas) exceeds 1,024 tokens.
• Implementation requires adding a cache_point part of type default to the system prompt and tool schemas.
• A cache write occurs during the first request; a cache read occurs on subsequent requests.
• Changing the system prompt invalidates the cache, but changing the user message does not.