Prompt engineering

Module Structure

• Series of videos building on a single prompt iteratively
• Start with a poorly-written initial prompt, then improve it step-by-step using different techniques
• Run evaluations after each improvement to measure progress
• Goal: demonstrate how each technique increases output quality and reliability

Project Goal

• Create a prompt that generates a one-day meal plan for an athlete
• Inputs: height, weight, physical goal, dietary restrictions
• Output: structured meal plan with specific foods, portions, timing, caloric totals, and macro breakdowns

Evaluation Pipeline Setup

• Use updated flexible evaluation pipeline (notebook: 001_prompting)
• Pipeline wraps dataset generation, model grading, and evaluation in a PromptEvaluator class
• Supports concurrency via max_concurrent_tasks parameter
• Higher concurrency speeds up evals but risks rate limit errors
• Recommendation: start with 3, reduce to 1 if rate limits occur

Dataset Generation

• Define prompt purpose and required inputs as a dictionary
• Specify number of test cases (recommendation: 2-3 for quick iteration; larger numbers for real evaluations)
• Generated dataset saved as dataset. json with structured test cases

Prompt Development Workflow

• Write prompt in run_prompt() function
• Function receives test case inputs as a dictionary
• Interpolate input values into prompt template
• Use extra_criteria parameter to specify grading requirements (e. g. , "include daily caloric total, macro breakdown, meals with exact foods")

Evaluation Output

• Evaluation score indicates prompt quality
• output. html file generated after each eval run
• Dashboard shows per-test-case results, scores, reasoning, and actual outputs
• Use dashboard to identify improvement areas

Takeaways

• Iteratively improve a single prompt using techniques and evaluate progress after each change to measure quality gains
• Set up an evaluation pipeline using PromptEvaluator class with concurrency control (start at 3, reduce to 1 if rate limits occur)
• Define test cases in a dataset with specific inputs (height, weight, physical goal, dietary restrictions) and use extra_criteria to specify grading requirements
• Review the output. html dashboard after each evaluation to identify which test cases failed and where the prompt needs improvement
• Write prompts in a run_prompt() function that interpolates input values into the template and returns structured outputs