Exercise on prompt evals

Improving Model Graders via Context Injection

• Goal: Enhance the performance and reasoning of a model grader by providing it with explicit context regarding what constitutes a "good solution. "
• Method: The improvement is achieved through a two-step process:

• Step 1: Augment Data Generation (Creating the Context)
• Modify the prompt used to generate the dataset.
• Instruct the data generation prompt to include specific "solution criteria" within every test case.
• Result: The test case objects now contain a dedicated key (e. g. , "solution criteria") detailing the desired characteristics of a correct answer.

• Step 2: Inject Context into Grading (Applying the Context)
• Modify the prompt used by the model grader.
• Insert the newly generated "solution criteria" into the grader prompt.
• Placement: The criteria should be placed immediately after the solution being evaluated by the model grader.
• Outcome: This provides the grader with a clear benchmark, leading to more detailed and accurate evaluation reasoning.

Takeaways

• The goal is to enhance model grader performance by providing explicit context regarding what constitutes a "good solution. "
• Data generation must be augmented to include specific "solution criteria" within every test case object.
• The model grader prompt must be modified to inject these "solution criteria" immediately following the solution being evaluated.
• This context injection provides the grader with a clear benchmark, leading to more detailed and accurate evaluation reasoning.