Code based grading

Overview

• Code grader validates model output to ensure it returns only plain Python, JSON, or regular expressions without explanations
• Must verify valid syntax for the code type received

Validation Approach

• Create three helper validator functions: validateJSON, validatePython, validateRegEx
• Each function attempts to parse/compile the output in its respective format
• Return score of 10 if parsing succeeds; return 0 if parsing fails
• Use a general gradeSyntax function to determine which validator to call based on expected format

Dataset Updates

• Add a format key to each test case specifying expected output type (JSON, Python, or regex)
• Update the dataset generation prompt to include format specifications
• Ensures grader knows which validation function to apply

Prompt Improvements

• Clarify in the prompt that output should be only Python, JSON, or plain regex
• Explicitly state: no comments, commentary, or explanations
• Use pre-filled assistant message with opening code fence (triple backticks with "code" label)
• Add closing backticks as stop sequence to constrain model output format

Score Calculation

• Rename initial model grader score to modelScore for clarity
• Calculate syntax score using the code grader
• Merge scores by taking average: (modelScore + syntaxScore) / 2
• Final score indicates overall quality combining both model logic and syntax validity

Testing & Iteration

• Run evaluation to get baseline score
• Adjust prompt and re-evaluate to measure improvements
• Iterative process needed to determine if changes enhance performance

Takeaways

• Create three validator functions (validateJSON, validatePython, validateRegEx) that parse/compile output and return 10 for success or 0 for failure
• Add a format key to each test case specifying expected output type so the grader knows which validator to apply
• Clarify in the prompt that output must be only code/JSON/regex with no explanations, and use pre-filled assistant message with code fence and stop sequence to constrain model output
• Calculate final score as the average of modelScore and syntaxScore to combine logic quality with syntax validity
• Run iterative evaluation cycles to measure whether prompt and validation improvements enhance performance