Prompt caching in action

Prompt Caching Study Notes

**I. Core Concepts**

• Purpose: Prompt caching is used to improve efficiency by storing and reusing identical content (like system prompts or tool schemas) across multiple requests.
• Key Components:
• System Prompt: A large block of text (e. g. , 6k tokens) defining the AI's behavior.
• Tool Schemas: Definitions of available functions/tools (e. g. , 1. 7k tokens).
• Caching Mechanism: When content is cached, the model performs a "cache write" (storing the data) and subsequent requests perform a "cache read" (retrieving the stored data).

**II. Implementation Strategy**

• Goal: Modify the chat function to automatically enable caching for tools and the system prompt by default.
• Caching Tool Schemas:
• The cache control field must be added to the tool schema definition.
• The recommended practice is to create a copy of the entire tools list, clone the last tool schema within that copy, add the cache control field (set to ephemeral), and then replace the original last tool with the cloned version.
• Benefit: This robust method prevents issues if the order of tools is changed later in the application.
• Caching System Prompts:
• The system prompt must be wrapped in a specific text block (dictionary) that includes:
• type: text
• text: [The system prompt content]
• cache control: ephemeral

**III. Caching Behavior and Rules**

• Caching Order: The model processes and caches inputs in a specific order: Tools → System Prompt → User Message.
• Cache Read: If the input content (tools, system prompt, or message) is identical to a previous request, the model reads the data from the cache, reducing input token usage.
• Cache Write: A cache write occurs when the model processes new, unique content and stores it for future use.
• Cache Invalidation (When the Cache is Lost):
• Changing Tools: Any modification to the tool schemas invalidates the tool cache, forcing a new cache write.
• Changing System Prompt: Any modification to the system prompt invalidates the system prompt cache, forcing a new cache write for that prompt.
• Partial Caching: If the tools remain identical but the system prompt is changed, the model will perform a partial cache read (for the tools) and a cache write (for the new system prompt).

**IV. Summary**

• Prompt caching is highly effective when dealing with repetitive content, such as the same list of messages, the same tool schemas, or the same system prompt.

Takeaways

• Prompt caching improves efficiency by storing and reusing identical content, such as system prompts and tool schemas, across multiple requests.
• To enable tool caching, the cache control: ephemeral field must be added to the tool schema definition.
• System prompts must be wrapped in a specific text block that includes the cache control: ephemeral setting.
• Changing the tools or the system prompt invalidates the respective cache, forcing a new cache write.
• Partial caching allows the model to read unchanged content (e. g. , tools) while writing new content (e. g. , a modified system prompt).