Rules of prompt caching

Prompt Caching Key Concepts

Definition and Function

• Core Idea: Prompt caching saves the processing work performed by the model (e. g. , CLOD) on an initial request.
• Mechanism: If an identical request is made later, the system retrieves the saved work from the cache instead of reprocessing the input.
• Use Case: Most beneficial when repeatedly sending the same content to the model.
• Process: It is a two-phase process:
• Initial request (to write data into the cache).
• Follow-up requests (to read and reuse the cached data).

Enabling and Controlling Caching

• Enabling: Prompt caching is disabled by default and must be explicitly enabled using a new message part called a cache point.
• Cache Point Function: When included in a message, a cache point instructs the system to cache all work done on the content before that point.
• Scope: The caching system only works if the content preceding the cache point is identical across requests.
• Cache Limit: Cached data is not permanent; it is only stored for five minutes.

Rules and Limitations

• Minimum Content: There must be at least 1024 tokens of content before a cache point for the system to actually cache the work.
• Content Scope: Cache points are not limited to single text parts; they can cache work done across multiple messages and different parts.

Application Areas

• Message Parts: Caching standard text and image parts.
• Tool Definitions: Caching the JSON schemas within tool definition lists.
• System Prompts: Caching the content of the system prompt.
• Common Practice: Caching tools and system prompts is a common implementation strategy because these elements rarely change between requests.

Takeaways

• Prompt caching saves processing work by retrieving saved results for identical requests instead of reprocessing the input.
• Caching is disabled by default and must be explicitly enabled using a cache point within a message.
• For caching to work, the content preceding the cache point must be identical across requests and must contain at least 1024 tokens.
• Cached data is temporary, stored only for a maximum of five minutes.
• Common use cases include caching system prompts, tool definitions, and standard text/image parts.