Individual skill
eval-design
Use when the user needs to design evaluation datasets, create test cases, stratify samples, generate adversarial examples, extract eval dimensions from traces/specs, or build a labeled evaluation set. Also use when the user mentions test data design, eval coverage, difficulty stratification, synthetic data generation for eval, or "how to create good evaluation data." Outputs datasets in OpenJudge-compatible format.
skills/eval_pipeline/01-eval-design/SKILL.md
Source commit eb34c3b94c908d875eda025e7863bc765c44ad76
What was checked
- Matches the observed repository HEAD
- Yes
- Scanner rules are current
- Yes
- Repository scan complete
- No
- Skill scan complete
- No
- Observed behavior
- Network or broad shell
- Last checked
- Nov 10, 2026
These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.
Install this skill
Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.
An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.
Repository context
- Repository stars (not skill ratings)
- 1,090
- Repository last updated
- Nov 10, 2026
- License
- Apache-2.0
Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.
- alignment Repository topic
- reward Repository topic
- reward-model Repository topic
- rlhf Repository topic
- agent Repository topic
- evaluation Repository topic
- grader Repository topic
- llm Repository topic
- agent-skills Repository topic
- ai-agent Repository topic
- design Derived category
- data Derived category
- testing Derived category
Files and excerpt
File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.
Browse files
SKILL.mdcore · 11,173 bytesSKILL.mdcore · 11,173 bytesSKILL.mdcore · 11,173 bytesscripts/coverage_check.pyruns · 5,670 bytes
Read excerpt
# Eval Design
Design high-quality evaluation datasets that measure what actually matters for your
application. You extract evaluation dimensions from business context, structure them
into stratified test cases, and output datasets ready for OpenJudge `GradingRunner`.
## When to Activate
- User has agent traces / production logs and wants to build an eval set from them
- User has evaluation principles but needs properly stratified test data
- User wants to generate adversarial examples that stress-test their system
- User needs coverage analysis — are they testing all the right things?
- User wants a labeling guide for human annotators
