Generating test datasets

Prompt Evaluation Workflow: Key Concepts

**I. Prompt Goal and Constraints**

• Objective: Create a prompt that helps users write code specifically for AWS use cases.
• Input: A user-defined task.
• Required Output: The response must be one of three specific formats:
• Python code
• JSON configuration
• Raw regular expression
• Strict Constraint: The output must contain only the requested code/format, with no headers, footers, or explanatory text.

**II. Dataset Assembly**

• Definition: A dataset is an array of inputs (tasks) fed into the prompt.
• Structure: The dataset is an array of JSON objects, where each object contains a task property.
• Process: Run the prompt against every task in the dataset to generate evaluation data.
• Generation Methods: Datasets can be assembled manually or automatically using an LLM (e. g. , Claude).
• Efficiency Tip: Use faster models (like Haiku) when generating large test datasets.

**III. Implementation Steps (Code Workflow)**

• Dataset Generation Prompt: A specialized prompt is used to instruct the LLM to generate the required test cases (the array of JSON objects).
• API Interaction:
• The request is structured using a list of messages (User message + Assistant message).
• A specific stop sequence (e. g. , backtick, backtick, backtick) is used to control the LLM's output generation.
• The raw text response from the LLM must be parsed using json. loads.
• Final Output: The generated dataset is saved to a file (e. g. , dataset. json) using json. dump with an indent of two for readability.

Takeaways

• The prompt must be strictly constrained to output only specific formats (Python code, JSON configuration, or raw regular expression) for AWS tasks.
• A dataset is structured as an array of JSON objects, where each object contains a user-defined task.
• Dataset generation requires a specialized prompt and API interaction using a list of messages and a specific stop sequence.
• The raw text response from the LLM must be parsed using json. loads to create the final dataset.
• Use faster models when generating large test datasets to improve efficiency.