Why This Matters

This capstone applies PR-29 to PR-35 into one deliverable: an eval harness that continuously tests the coding agent.

Concept Explained

An agent eval harness: define tasks (issues with acceptance criteria), delegate to agent, score the PR, track over time.

How It Works

Define 10–20 agent-eligible tasks. Each has acceptance criteria. Delegate to agent. Score PR: criteria met? Tests pass? Code quality? Track scores over time.

Step-by-Step Tutorial

1. Define tasks

2. Delegate to agent

3. Score PRs

4. Track

5. Automate

Real-World Example

A team built an agent eval harness. 15 tasks. Baseline: 80% pass. After model upgrade: 85%. After prompt template change: 70% — regression caught. Rolled back template.

Example Prompts / Commands / Code

"""[Task definitions (10-20)]
       |
       v
[Delegation script] -> [Coding agent] -> [PR opened]
       |
       v
[Scoring script]
   - Criteria met? (automated check)
   - Tests pass? (run CI)
   - Code quality? (LLM-as-judge)
       |
       v
[Score report] -> [Dashboard]
       |
       v
[Alerts on regression]

Run: weekly, or before/after model upgrades
"""

"""Task: Add null check to parseDate

Issue:
- Function: src/date.ts:parseDate
- Bug: throws TypeError on null
- Fix: add null check, throw descriptive error

Acceptance criteria:
- [ ] parseDate(null) throws TypeError('input is required')
- [ ] parseDate('2024-01-15') returns Date (unchanged)
- [ ] parseDate('invalid') returns null (unchanged)
- [ ] Test added in tests/date.test.ts

Scoring:
- Criteria met (0-4): automated check
- Tests pass (0/1): run CI
- Code quality (0-5): LLM-as-judge
- Total: 0-10

Pass: total >= 8
Fail: total < 8
"""

Common Mistakes

• Tasks not agent-eligible — agent flails.
• No acceptance criteria — can't score.
• Not automating scoring — manual burden.
• Not tracking over time — regressions slip in.

Best Practices

• Define 10–20 agent-eligible tasks.
• Each with specific acceptance criteria.
• Automate scoring (criteria + tests + LLM-as-judge).
• Track scores over time.
• Alert on regressions; run before/after upgrades.

Troubleshooting

Practical Exercise

Professional Challenge

Frequently Asked Questions

Further Reading

• Coding agent

Official References

• Coding agent

Related lessons: PR-29, PR-36