Menu

Individual skill

agentsop-domain-eval-set

0upvotes

Build and govern a 50-200 example domain-specific held-out benchmark sampled from real traffic. Distinct from public benchmarks (MMLU/HumanEval/GSM8K via lm-evaluation-harness) which measure GENERAL capability. Only a held-out domain set predicts whether THIS system works on YOUR data. Collect real examples, label, hold out (never train/prompt on it), size 50-200, version it, refresh on drift.

skills/agentsop-domain-eval-set/SKILL.md

Read pinned source

Source commit cf4376869e5f23f82404c3021a2a1315dfca318d

What was checked

Matches the observed repository HEAD
Yes
Scanner rules are current
No
Repository scan complete
No
Skill scan complete
No
Observed behavior
Not checked yet
Last checked
Nov 6, 2026

These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.

Install this skill

Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.

An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.

Repository context

Repository stars (not skill ratings)
469
Repository last updated
Nov 8, 2026
License
MIT

Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.

Files and excerpt

File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.

Browse files
Read excerpt

# domain-eval-set — Your Held-Out Domain Benchmark

> "Compiled program beats baseline on a *held-out* test set (not the val set used in optimization)."
> — DSPy SOP exit criterion [dspy.ai/learn/optimization/overview/]

> "Build the eval loop **before** optimizing anything. Every subsequent change must be gated on these numbers."
> — LlamaIndex SOP Stage 2

This is an **ENHANCE overlay** skill. It produces one artifact — a versioned,
sealed, human-labeled set of 50–200 examples drawn from *your* domain — that
other skills consume: `[[agentsop-regression-gate]]` enforces it on every PR,
`[[agentsop-metric-design]]` defines the scoring function applied to each example, and
`[[lm-evaluation-harness]]` runs the *complementary* public-capability axis. The
core claim: **public benchmarks tell you the model is smart in general; only a