A typical eval workflow

Overview

• No single standardized methodology for assembling evaluation workflows across the industry
• Many open-source and paid tools available, but you can build custom workflows from scratch
• Starting small with a custom implementation helps you understand how workflows function before scaling up

Step 1: Initial Prompt Draft

• Write a basic prompt that you want to improve
• Example: "Please answer the user's question" with interpolated user input

Step 2: Create Evaluation Dataset

• Assemble a collection of possible inputs to test your prompt
• Can range from 3 test cases to hundreds or thousands in real-world scenarios
• Can be created manually or generated using Claude

Step 3: Generate Prompts & Get Responses

• Combine each dataset input with your prompt template to create fully fleshed-out prompts
• Feed each complete prompt into Claude
• Collect all responses from Claude

Step 4: Grade the Responses

• Pair each question with its corresponding Claude response
• Use a grader to score each response (e. g. , 1-10 scale)
• Higher scores indicate better quality; lower scores indicate room for improvement
• Grading methodology has hidden complexity (covered in detail later)

Step 5: Calculate Average Score

• Average all individual scores together
• This provides an objective metric for prompt performance

Step 6: Iterate & Compare

• Modify the prompt based on results
• Run the entire pipeline again with the updated prompt
• Compare scores between versions to determine which performs better
• Higher score indicates the improved version

Takeaways

• Build custom evaluation workflows by starting small to understand how they function before scaling up
• Create an evaluation dataset of test inputs (3 to thousands of cases) and run them through your prompt to collect responses
• Grade each response on a consistent scale (e. g. , 1-10) and calculate the average score as an objective performance metric
• Iterate by modifying your prompt and re-running the full pipeline to compare scores between versions and identify improvements