Exercise on prompt evals

Overview

• Goal: enhance model grader by providing context about what constitutes a good solution
• Process requires only two steps

Step 1: Generate Solution Criteria in Dataset

• Modify the prompt in the generate_dataset function
• Request that each test case include a solution criteria key
• Solution criteria should describe characteristics of a good solution
• Example: "A good solution would include X, Y, and Z characteristics"
• Regenerate the dataset to produce updated test cases with solution criteria included

Step 2: Inject Solution Criteria into Grading Prompt

• Locate the grade_by_model function and its prompt
• Add the newly generated solution criteria to the grading prompt
• Insert it after the original task and generated output
• Use tags to structure the criteria (important for prompt engineering)
• Interpolate the solution_criteria key from the test case object into the prompt
• This gives the model grader explicit guidance on evaluation standards

Expected Outcome

• Model grader receives clearer context about solution quality expectations
• Reasoning section in grader output becomes more detailed and informed
• Scores reflect better-calibrated evaluation based on defined criteria

Takeaways

• Generate solution criteria for each test case in the dataset by modifying the generate_dataset function to include a solution criteria key describing characteristics of a good solution
• Inject the solution criteria into the grading prompt within the grade_by_model function using structured tags to provide explicit evaluation guidance
• Model grader will produce more detailed reasoning and better-calibrated scores when given clear context about solution quality expectations