Generating test datasets

Prompt Evaluation Workflow: Key Concepts

**I. Workflow Goal and Constraints**

• Objective: To build a custom workflow that evaluates the performance of a prompt.
• Prompt Purpose: To assist users in writing code specifically for AWS use cases.
• Input: A user-defined task.
• Required Output Types: The model must respond with one of three specific formats:
• Python code
• JSON configuration
• Raw regular expression
• Output Constraint: The response must contain only the requested output, with no extraneous explanations, headers, or footers.

**II. Prompt Design and Data Set Creation**

• Draft Prompt (V1): A simple instruction, such as "Please provide a solution to the following task. "
• Data Set Definition: A collection of inputs that are fed into the prompt for testing.
• Data Set Structure: An array of JSON objects, where each object contains a task property.
• Data Set Generation: Can be created manually or automatically by prompting a powerful model (Claude).
• Optimization: When generating the data set, use a faster model (e. g. , Haiku) to save time and resources.

**III. Implementation Steps (Code)**

• Setup: Initialize a client and update the model ID to utilize an optimized inference profile (e. g. , Haiku).
• Helper Function (GenerateDataSet): This function uses a detailed prompt to instruct Claude to generate the required evaluation test cases.
• Forcing JSON Output: To reliably extract JSON from the model's text response, use:
• A pre-filled assistant message (e. g. , "`json").
• A stop sequence (e. g. , three backticks) to terminate the model's generation.
• Post-Processing: The raw text response must be parsed using a JSON module (e. g. , json. loads) to convert it into a usable data structure.
• Final Step: Save the generated data set to a JSON file to prevent the need for constant regeneration.

Takeaways

• The workflow evaluates prompt performance for AWS code generation, requiring the model to respond strictly in one of three formats: Python code, JSON configuration, or raw regular expression.
• The evaluation data set must be structured as an array of JSON objects, where each object contains a defined task property.
• Optimization involves using powerful models (e. g. , Claude) for generating the data set, while utilizing faster models (e. g. , Haiku) for inference and setup to save resources.
• Reliable JSON extraction requires using a pre-filled assistant message and a stop sequence, followed by post-processing using a JSON module to parse the raw text response.