Why This Matters

Scoring is the bottleneck of evals. Right rubric per task makes scoring reliable and automatable.

Concept Explained

Three rubric types: pass/fail (binary), Likert (1–5 scale), LLM-as-judge (AI scores AI). Choose per task type.

How It Works

Pass/fail for code that runs. Likert for subjective quality. LLM-as-judge for scale (automate). Measure inter-rater reliability for human scoring.

Step-by-Step Tutorial

1. Choose rubric per task

2. Apply pass/fail

3. Apply Likert

4. Use LLM-as-judge

5. Measure inter-rater reliability

Real-World Example

A team used LLM-as-judge for their 25-task eval. Calibrated against human scoring (agreement 85%). Automated monthly eval runs. Caught regressions without manual scoring burden.

Example Prompts / Commands / Code

"""Task type                  Rubric           Why
-------------------------------------------------------
Code that runs            Pass/fail        Objective
Code quality              Likert (1-5)    Subjective
Refactor correctness      Pass/fail        Tests pass?
Refactor quality          Likert          Subjective
Explanation accuracy      LLM-as-judge    Hard to score manually
Test coverage             Pass/fail        Coverage %
Test quality              Likert          Subjective

For automation:
- Pass/fail: scriptable
- Likert: human or LLM-as-judge
- LLM-as-judge: fully automated, calibrated
"""

"""Score this Copilot output on a scale of 1-5 for:
- Correctness: does it work?
- Completeness: covers edge cases?
- Idiomaticity: matches conventions?

Task: [task description]
Copilot output: [output]

For each dimension, give a score (1-5) and one-sentence justification.

Output as JSON:
{
  "correctness": {"score": N, "reason": "..."},
  "completeness": {"score": N, "reason": "..."},
  "idiomaticity": {"score": N, "reason": "..."}
}

Calibrate against human scoring: run on 10 tasks scored by humans. Compare. Adjust prompt if disagreement >20%.
"""

Common Mistakes

• Using Likert for code that runs — should be pass/fail.
• LLM-as-judge without calibration — unreliable.
• Single human scorer — no inter-rater reliability.
• Not documenting rubric — inconsistent scoring.

Best Practices

• Choose rubric per task type.
• Pass/fail for code that runs.
• Likert for subjective quality.
• LLM-as-judge for scale; calibrate against humans.
• Measure inter-rater reliability (>80% agreement).

Troubleshooting

Practical Exercise

Professional Challenge

Frequently Asked Questions

Further Reading

• LLM-as-judge research

Official References

• LLM-as-judge paper

Related lessons: PR-29, PR-31