.png){kind=small}
Designing a Golden Task Set
Curate a high-signal set — task selection, balance, contamination.
What You Will Learn
- Curate a high-signal golden set.
- Balance task types.
- Avoid contamination.
- Vary difficulty.
- Document task design.
Why This Matters
A golden set is only as good as its tasks. Bad tasks produce misleading scores. This lesson builds the curation skill.
Concept Explained
High-signal tasks: representative of real work, varied in type and difficulty, not in training data (contamination), with clear pass/fail.
How It Works
Pick tasks across categories (refactor, test, debug, explain, generate). Vary difficulty (easy, medium, hard). Verify not in public training data. Define clear pass/fail criteria.
Step-by-Step Tutorial
1. Pick categories
Refactor, test, debug, explain, generate, convert.2. Vary difficulty
Easy (one-liner), medium (function), hard (module).3. Avoid contamination
Don't use public LeetCode/HackerRank problems.4. Define pass/fail
Specific, testable criteria per task.5. Balance
Equal representation across categories and difficulty.Real-World Example
A team built a golden set with 5 refactor, 5 test, 5 debug, 5 explain tasks. Balanced easy/medium/hard. Verified no contamination. Set was high-signal: detected real regressions when prompts changed.
Example Prompts / Commands / Code
"""Categories (5 tasks each):
1. Refactor — extract function, simplify conditional, rename, async conversion, pattern application
2. Test — unit test, integration test, edge case, mock, parametrize
3. Debug — null pointer, race condition, off-by-one, type error, performance
4. Explain — regex, SQL, shell pipeline, complex function, architecture
5. Generate — CLI script, API endpoint, type definition, config file, schema
Difficulty per category:
- 1 easy (one-liner or simple function)
- 2 medium (multi-function or moderate complexity)
- 2 hard (module-level or complex reasoning)
Total: 25 tasks
Pass/fail criteria per task:
- Specific (not subjective)
- Testable (run code, check output)
- Documented (so any reviewer can score)
"""Common Mistakes
- All tasks same type — eval is narrow.
- All tasks same difficulty — doesn't catch all regressions.
- Contaminated tasks (public problems) — model may have memorized.
- Vague pass/fail — subjective scoring.
Best Practices
- Balance across 5+ categories.
- Vary difficulty (easy/medium/hard).
- Avoid contamination (no public problems).
- Define specific, testable pass/fail criteria.
- Document task design.
Troubleshooting
| Problem | How to Fix |
|---|---|
| Scores are inconsistent | Refine pass/fail criteria. Or use multiple reviewers. |
| All tasks pass | Set is too easy. Add harder tasks. |
Practical Exercise
Your Turn
Design a 25-task golden set. Balance categories and difficulty. Define pass/fail. Verify no contamination.
Professional Challenge
Run the set on 2 models. Compare scores. Identify which model wins per category.
Key Takeaways
- High-signal golden set: balanced, varied, uncontaminated.
- 5+ categories: refactor, test, debug, explain, generate.
- Vary difficulty (easy/medium/hard).
- Avoid contamination (no public problems).
- Define specific, testable pass/fail.
Frequently Asked Questions
How many tasks?
How often to refresh?
Further Reading
Official References
SEO Metadata
SEO title: Designing a Golden Task Set
Meta description: Curate a high-signal set — task selection, balance, contamination.
Primary keyword: designing a golden task set
Secondary keywords: designing a golden task set
Search intent: Informational
URL slug: /designing-golden-task-set-high-signal
Categories: AI Tools, GitHub Copilot
Tags: GitHub Copilot, Professional, Evaluation, Golden Set, Task Design, IMCSEIAN, Tutorial, IMCSEIAN
Featured image concept: IMCSEIAN lesson card for Designing a Golden Task Set
Comments
Comments
Post a Comment