Running the eval

Overview

• Process each test case by merging it with a prompt, sending to Claude, and grading the output
• Three main functions form the backbone of an eval pipeline: run_prompt, run_test_case, and run_eval

run_prompt Function

• Takes a single test case as input
• Merges the task with the test case data
• Sends the combined prompt to Claude via the chat API
• Returns the raw output from Claude
• Currently produces unformatted output (no constraints on response format)

run_test_case Function

• Calls run_prompt to get Claude's output
• Grades the output (currently placeholder with hardcoded score of 10)
• Returns a dictionary containing:
• The output from Claude
• The original test case
• The score

run_eval Function

• Loads the dataset
• Iterates through each test case
• Calls run_test_case for every test case
• Collects all results into a list
• Returns the complete results

Current Limitations

• No formatting instructions in prompts → Claude returns excessive output
• Grading is not yet implemented (hardcoded scores only)
• Runtime is slow (approximately 31 seconds for full dataset with Haiku model)
• Next step: implement graders to evaluate Claude's outputs against expected results

Takeaways

• An evaluation pipeline has three core functions: run_prompt sends merged task+test data to Claude, run_test_case calls run_prompt and grades the output, and run_eval orchestrates the full workflow across all test cases
• Claude's responses are currently unformatted and excessive because prompts lack formatting instructions
• Grading is not yet implemented—scores are hardcoded at 10, so actual evaluation against expected results must be added next