Generating test datasets

Prompt Design Goals

• Create a prompt to help users write AWS-specific code
• Accept user tasks as input
• Output must be one of three types: Python code, JSON configuration, or regular expression
• Output should contain only the solution with no explanations, headers, or footers

Evaluation Workflow Steps

Step 1: Draft the Prompt

• Start with a simple version (V1) that asks for a solution to a given task
• Refine iteratively based on evaluation results

Step 2: Assemble a Dataset

• Create a collection of test inputs to evaluate the prompt
• Dataset format: array of JSON objects, each with a "task" property
• Can be assembled manually or generated automatically using Claude
• Use faster models like Haiku for dataset generation to save costs

Step 3: Generate Test Cases

• Use Claude to automatically generate realistic test cases
• Request multiple examples covering different output types (Python, JSON, regex)
• Parse Claude's response as JSON using pre-filling and stop sequences

Implementation Details

• Use pre-filling technique: add assistant message with opening code fence (`json) before calling Claude
• Use stop sequences to control output (e. g. , stop at closing backticks)
• Parse returned text with JSON. loads() to extract structured data
• Save dataset to a JSON file for reuse during evaluation

Dataset Structure

• Each test case is a JSON object with a "task" property
• Start with 3 test cases to validate the workflow
• Include diverse task types to test all three output formats

Takeaways

• Create a prompt that accepts AWS-related tasks and outputs only the solution (Python code, JSON, or regex) with no explanations
• Assemble a dataset of test inputs as JSON objects with "task" properties, using Claude to generate realistic test cases across all three output types
• Use pre-filling with opening code fences and stop sequences to reliably extract structured JSON responses from Claude
• Iteratively refine the prompt (V1, V2, etc. ) based on evaluation results against your test dataset
• Start with 3 diverse test cases covering different output formats to validate the workflow before scaling