Running the eval

Evaluation Pipeline Concepts

Overall Workflow:

• The evaluation process involves iterating through a dataset of records (test cases).
• Each test case is merged with a prompt, fed into Claude, and the resulting output is then processed by a grader.
• Flow: Test Case → Merge with Prompt → Claude → Output → Grader.

Key Functions:

• run_prompt
• Purpose: To generate a response from the LLM (Claude) for a single test case.
• Process: Takes a test case (JSON object), merges it with a predefined prompt (e. g. , "Please solve the following task. "), and calls the chat function.
• Output: The raw text output received from Claude.
• Note: Currently lacks specific formatting instructions (e. g. , forcing JSON or Python output).

• run_test_case
• Purpose: To execute the full evaluation cycle for one individual test case.
• Process:
• Calls run_prompt using the test case.
• Performs grading on the resulting output (currently hardcoded).
• Returns a summary dictionary.
• Output: A dictionary containing the Claude output, the original test case, and the score.

• run_eval
• Purpose: To orchestrate the entire evaluation across the entire dataset.
• Process:
• Loads the complete dataset.
• Loops through every test case in the dataset.
• Calls run_test_case for each individual record.
• Assembles all individual results into a final list.
• Output: A list of all results from the evaluation.

Summary of Pipeline Components:

• The three functions (run_prompt, run_test_case, run_eval) constitute the vast majority of the evaluation pipeline structure.
• The primary remaining task is the full implementation of the grader.

Takeaways

• The evaluation workflow iterates through test cases: Test Case → Merge with Prompt → Claude → Output → Grader.
• The pipeline is structured around three core functions: run_prompt, run_test_case, and run_eval.
• run_prompt is responsible for generating the raw text response from Claude for a single test case.
• run_test_case executes the full evaluation cycle for one record, including calling run_prompt and performing grading.
• run_eval orchestrates the entire process by looping through the dataset and calling run_test_case for every record.