Exercise on prompting

Exercise Overview

• Goal: Improve an existing prompt to extract topics from scholarly article passages
• Input: A content property containing one paragraph of text
• Output: JSON array of strings containing all topics mentioned in the passage
• Success metric: Achieve evaluation score of 7 or higher

Key Prompt Engineering Techniques Applied

1. Be Simple and Direct

• Replace vague instructions with clear, explicit language
• Example: "Extract key topics mentioned from a passage of text from a scholarly journal into a JSON array of strings"
• This single change improved the score dramatically (to ~9. 5)

2. Use XML Tags for Structure

• Wrap interpolated content with descriptive XML tags (e. g. , <text>... </text>)
• Creates clearer connection between prompt instructions and provided data
• Improves model's understanding of input boundaries

3. Provide Step-by-Step Instructions

• Break down the task into explicit sequential steps:
• Closely examine the provided text
• Identify each topic mentioned
• Add each topic to a JSON array
• Respond with only the JSON array
• Include explicit constraint: "Do not provide any other text or commentary"

4. Evaluation and Iteration

• Run evaluation cell to generate output. html report
• Review report to identify common failure patterns
• Iterate on prompt based on specific feedback
• Test improvements by re-running evaluation

Common Issues to Avoid

• Not returning proper JSON format
• Including extra commentary or text beyond the required output
• Vague or indirect instructions that leave room for interpretation

Takeaways

• Be simple and direct: replace vague instructions with clear, explicit language for dramatic score improvements
• Use XML tags to structure interpolated content and create clearer boundaries between instructions and data
• Provide step-by-step instructions broken into explicit sequential steps with clear output constraints
• Iterate based on evaluation feedback: run evaluations, identify failure patterns, and test improvements systematically
• Avoid common failures: improper JSON format, extra commentary, and ambiguous instructions