Why This Matters

This capstone builds a real eval harness that continuously tests the coding agent. Combines eval (PR-29 to PR-36), coding agent (PR-15 to PR-20).

Concept Explained

Harness: define tasks, delegate to agent, score PRs, track over time, alert on regressions.

How It Works

Define 15–25 agent-eligible tasks. Each with acceptance criteria. Delegate to agent. Score: criteria met? Tests pass? Quality? Track. Alert on regressions.

Step-by-Step Tutorial

1. Define tasks

2. Delegate

3. Score

4. Track

5. Alert

6. Automate

Real-World Example

A team built this harness. Ran weekly. Caught regression when model upgraded — score dropped from 80% to 70%. Rolled back. Harness saved them from shipping regression.

Example Prompts / Commands / Code

"""[Task definitions (15-25)]
       |
       v
[Delegation script] -> [Agent] -> [PRs]
       |
       v
[Scoring]
   - Criteria met? (scripted check)
   - Tests pass? (CI)
   - Quality? (LLM-as-judge)
       |
       v
[Report] -> [Dashboard] -> [Alerts]
       |
       v
[Weekly run; before/after upgrades]
"""

"""Eval run: 2026-08-30
Model: claude
Tasks: 20

Results:
- Pass: 16 (80%)
- Fail: 4 (20%)

Failures:
- Task 3: missed edge case (null input)
- Task 7: scope creep (refactored unrelated code)
- Task 12: hallucinated API
- Task 18: tests didn't pass

Comparison to last week: 80% (no change)
Comparison to baseline: 78% (+2%)

Status: No regression. Proceed.
"""

Common Mistakes

• Too few tasks — low signal.
• No automation — manual burden.
• No tracking — can't detect regressions.
• Not running before upgrades — surprises.

Best Practices

• 15–25 agent-eligible tasks with criteria.
• Automate scoring (criteria + tests + LLM-as-judge).
• Track scores over time.
• Run weekly and before/after upgrades.
• Alert on regressions (>5% drop).

Troubleshooting

Practical Exercise

Professional Challenge

Frequently Asked Questions

Further Reading

• Coding agent

Official References

• Coding agent

Related lessons: PR-36, PR-48