Prompt engineering

Prompt Engineering Fundamentals

• Definition: Prompt engineering is the process of refining an initial prompt to achieve more reliable and higher quality outputs from a language model.
• Module Workflow:
• Write an initial, often poor, prompt.
• Evaluate the prompt and observe a low score.
• Apply prompt engineering techniques step-by-step.
• Re-evaluate to confirm performance improvement with each change.

Goal of the Initial Prompt

• The prompt's objective is to generate a one-day meal plan for an athlete.
• Required Inputs: The prompt must accept and utilize the athlete's:
• Height
• Weight
• Physical Goal
• Dietary Restrictions

Evaluation Pipeline Components

• Prompt Evaluator: A class that manages the entire evaluation process, including dataset generation and model grading.
• Concurrency: The evaluator supports running multiple API calls simultaneously to speed up the process.
• Caution: High concurrency can lead to rate limit errors; reduce the concurrency value if errors occur.
• Dataset Generation: This step defines the prompt's overall purpose and lists all required input properties (e. g. , height in CM, weight in kilograms).
• Prompt Function: This function is called for every generated test case and is responsible for interpolating the specific input data (from the test case dictionary) into the prompt template.

Evaluation Process

• Initial Prompt: The first version of the prompt is typically simple and naive, resulting in a very poor evaluation score.
• Extra Criteria: The evaluation function accepts an optional extra criteria argument. This allows developers to specify detailed grading requirements for the model (e. g. , the output must include a daily caloric total, macro nutrient breakdown, and exact food portions/timing).
• Reporting: After running an evaluation, an output. html file is generated, providing a formatted report on every test case, including scores, reasoning, and solution criteria.

Takeaways

• Prompt engineering is the process of refining initial prompts to achieve reliable and higher quality outputs from a language model.
• The workflow is iterative: write a prompt, evaluate its performance, apply engineering techniques, and re-evaluate.
• Initial prompts must be designed to accept and utilize specific required inputs, such as height, weight, physical goal, and dietary restrictions.
• Evaluation is managed by a Prompt Evaluator, which uses Dataset Generation and a Prompt Function to test cases.
• Detailed grading requirements can be specified during evaluation using an optional extra criteria argument.