Exercise on prompt evals

Improving Model Grader Context via Two-Step Prompt Engineering

The goal is to enhance a model grader by providing it with explicit context regarding what constitutes a "good solution. " This is achieved through a two-step process:

Step 1: Augmenting the Dataset (Data Generation)

• Action: Modify the initial prompt used to generate the dataset.
• Requirement: Instruct the data generation prompt to include specific "solution criteria" within every test case.
• Result: The resulting test cases now contain an additional key (e. g. , "solution criteria") that describes the characteristics of an ideal solution.

Step 2: Injecting Context into the Grader (Evaluation)

• Action: Modify the prompt used by the model grader.
• Implementation: Insert the newly generated "solution criteria" into the grader prompt.
• Placement: The criteria should be placed immediately after the original solution being evaluated.
• Outcome: Providing this explicit context allows the model grader to better understand the expected quality, leading to more detailed and "fleshed out" reasoning in its evaluation.

Takeaways

• The goal is to enhance model grader performance by providing explicit context on what constitutes a "good solution. "
• Step 1 involves augmenting the dataset by instructing the data generation prompt to include specific "solution criteria" in every test case.
• Step 2 requires injecting these generated "solution criteria" into the model grader prompt.
• The criteria must be placed immediately after the original solution being evaluated.
• This two-step process enables the grader to provide more detailed and accurate evaluation reasoning.