Why This Matters

Model upgrades can introduce regressions. Without tracking, you ship regressions to production. This lesson builds the regression safety net.

Concept Explained

Regression tracking: run eval set before and after model upgrades. Canary: test on small subset first. Rollback if regression detected.

How It Works

Before upgrade: baseline eval. After: re-run eval. If score drops >5%: rollback. Canary: roll out to 10% of users first; monitor for issues.

Step-by-Step Tutorial

1. Baseline

2. Upgrade

3. Re-run eval

4. Canary

5. Rollback if needed

Real-World Example

A team upgraded GPT-4 to GPT-5. Baseline eval: 78%. After upgrade: 82%. Good — rolled out to canary (10% of users). After 48 hours, no issues. Rolled out to 100%. Eval-based upgrade prevented surprises.

Example Prompts / Commands / Code

"""1. Baseline eval (before upgrade)
   Score: 78%

2. Apply upgrade (e.g., switch model from GPT-4 to GPT-5)

3. Re-run eval
   Score: 82% (+4%) → proceed

4. Canary: roll out to 10% of users
   Monitor for 24-48 hours:
   - Error rates
   - User feedback
   - Support tickets

5. If canary clean: roll out to 100%
   If issues: rollback. Investigate.

6. Document upgrade:
   - Date
   - Old model → new model
   - Eval scores (before/after)
   - Canary results
   - Any issues
"""

"""Rollback immediately if:
- Eval score drops >5%
- Error rate increases >2x
- User complaints spike
- Security issue detected

Investigate (don't rollback) if:
- Eval score drops <5%
- Minor quality concerns
- Edge case regressions

Document all upgrades and regressions in a changelog.
"""

Common Mistakes

• Upgrading without baseline eval — can't detect regressions.
• 100% rollout without canary — surprises affect everyone.
• No rollback criteria — indecision during incidents.
• Not documenting upgrades — repeat mistakes.

Best Practices

• Baseline eval before upgrade.
• Re-run eval after upgrade.
• Rollback if >5% drop.
• Canary to 10% first; monitor 24–48 hours.
• Document all upgrades and regressions.

Troubleshooting

Practical Exercise

Professional Challenge

Frequently Asked Questions

Further Reading

• Copilot changelog

Official References

• Changelog

Related lessons: PR-29, PR-32