Rules of prompt caching

Prompt Caching Key Concepts

• Core Function: Prompt caching stores the processing work done by CLOD on an initial request in a temporary cache. Subsequent requests with identical content retrieve this saved work, saving processing effort.
• Activation: Caching is not enabled by default; it must be manually activated by adding a "cache breakpoint" to a block within a message.
• Cache Duration: The cached work is temporary and only persists for five minutes.
• Use Case: Most effective when the same content is repeatedly sent to CLOD.
• Implementation Requirement: To add a cache control field, the text block must be written in "long form" (Content field → List → Dictionary with Type "text" and the actual text).
• Caching Scope: A cache breakpoint caches all work performed up to and including that specific breakpoint.
• Cache Invalidation: For a follow-up request to utilize the cache, the content must be identical up to the breakpoint. Any change invalidates the cached work for that section.
• Applicable Locations for Breakpoints:
• Message blocks (User, Assistant, etc. ).
• Tool schemas.
• System prompts.
• Other blocks (e. g. , image blocks, tool use, tool results).
• Processing Order: CLOD processes inputs in a specific order: Tools → System Prompt → Messages. Breakpoints are applied to this sequence.
• Multiple Breakpoints: Up to four cache breakpoints can be added to strategically cache different parts of the request (e. g. , caching the tools and system prompt while allowing the user message to change).
• Minimum Content Length: There is a minimum requirement of 1024 tokens for content to be written to the cache.

Takeaways

• Prompt caching saves processing effort by storing the work done on identical requests.
• Caching is not automatic and must be manually activated by adding a "cache breakpoint. "
• The cached content must be identical up to the breakpoint; any change invalidates the cached work for that section.
• There is a minimum content length requirement of 1024 tokens for content to be written to the cache.