Streaming

Study Notes: Implementing Streaming with Bedrock

💡 Problem with Standard Request/Response

• Latency Issue: Traditional request/response models wait until the entire response is generated before sending it back.
• User Experience: This waiting time (e. g. , 3 to 30 seconds) often exceeds user expectations for immediate feedback.

🚀 Solution: Converse Stream

• Function: Use the converse_stream function to enable response streaming.
• Mechanism: Instead of waiting for the full response, the model sends back generated text chunks as they are produced.
• Goal: Allow the user to start reading the response immediately, chunk by chunk.

⚙️ How Converse Stream Works

• Initial Request: The server sends the user's message to Bedrock using converse_stream.
• Initial Response: Immediately receive an initial response object (acknowledgment that the request was received; no generated text yet).
• Event Stream: The response object contains a stream key, which is a generator. Iterating over this generator yields a sequence of events.
• Chunk Delivery: Each event contains a piece of the generated response. The application sends this chunk to the browser/mobile app for display.
• Completion: The process repeats until the full message is generated and the stream concludes.

🧩 Key Event Types

• Event Generator: The stream object is an iterable generator that yields various event types.
• Content Block Delta (The Focus): This is the critical event. It contains the actual piece of text generated by the model.
• Text Location: The generated text is found at event. content_block_delta. delta. text.
• Typical Event Order (Text Generation):
• message_start
• Multiple content_block_delta events (the text chunks)
• content_block_stop
• message_stop
• metadata
• Note on Tool Use: The content_block_start event is relevant for tool use but is not typically seen when only generating text.

💻 Implementation Tips

• Display Chunks: When printing chunks in Python, use print(chunk, end="") to prevent the automatic addition of newlines, ensuring the text streams continuously.
• Data Collection: Maintain a separate variable (e. g. , text = "") and append every received chunk to it to store the complete, final message.

Takeaways

• converse_stream enables response streaming by sending generated text chunks immediately, reducing latency compared to standard request/response models.
• The response object contains a stream key, which is an iterable generator that yields a sequence of events as the model generates text.
• The actual piece of generated text is found within the content_block_delta event at the path event. content_block_delta. delta. text.
• To ensure text streams continuously without line breaks, use print(chunk, end="") when displaying chunks in the application.