Code based grading

Code Grader Implementation Notes

**I. Code Grader Functionality**

• Purpose: The code grader verifies that the model's output is in a required format (Python, JSON, or Regular Expression) and that the syntax is valid.
• Validation Process:
• Three helper functions (validateJSON, validatePython, validateRegEx) are defined.
• Each function attempts to parse or compile the model output.
• Success: If parsing/compilation is successful, the score is 10.
• Failure: If an error occurs during parsing, the syntax check fails, and the score is 0.

**II. Implementation Steps**

• Data Requirement: The test case dataset must include a "format key" (e. g. , "Python," "JSON," "RegEx") to tell the grader which validator to run.
• Step 1: Add Validation Functions: Implement the three validator functions and a general grade syntax function that checks the test case format and calls the appropriate validator.
• Step 2: Update Dataset: Modify the dataset generation process to include the required "format key" for each task.
• Step 3: Update Prompt Template: Refine the prompt to strictly instruct the model to respond only with the required format (Python, JSON, or RegEx) and to omit all commentary or explanation.
• Step 4: Merge Scores: Combine the scores from the model grader and the syntax grader.
• Final Score = (Model Score + Syntax Score) / 2

**III. Prompt Engineering Techniques**

• Enforcing Output Format: To ensure the model provides raw content without explanation, use a pre-filled assistant message and a closing stop sequence (e. g. , three backticks).
• Goal: The prompt must guide the model to produce only the desired code or data structure.

Takeaways

• The code grader verifies that model output adheres to a required format (Python, JSON, or Regular Expression) and that the syntax is valid.
• Validation is performed by helper functions that assign a score of 10 upon successful parsing or compilation, and 0 upon failure.
• The test case dataset must include a "format key" to instruct the grader which specific validator to execute.
• The final score is determined by averaging the model score and the syntax score: Final Score = (Model Score + Syntax Score) / 2.
• Prompt engineering techniques, such as using stop sequences, are essential to ensure the model provides only the required raw output without commentary.