Individual skill
llm-evaluation
Builds evaluation harnesses for LLM products - golden datasets, deterministic checks, calibrated LLM-as-judge rubrics, and CI regression gates that turn "seems good" into a tracked number. Use when someone asks "how do I know this prompt change didn't break anything", "set up evals for my RAG pipeline", "is LLM-as-judge reliable", "why did quality drop after the model swap", or is shipping an LLM feature with no quality measurement. Do NOT use for classical ML model reporting (precision/recall, ROC curves, confusion matrices on trained classifiers) - use model-evaluation-report instead; do NOT use for analyzing online A/B experiments - use ab-test-analyzer instead.
skills/llm-evaluation/SKILL.md
Source commit 97a13607d669157a4a16d5ba79b77f208319f7b9
What was checked
- Matches the observed repository HEAD
- Yes
- Scanner rules are current
- No
- Repository scan complete
- No
- Skill scan complete
- Yes
- Observed behavior
- Not checked yet
- Last checked
- Nov 6, 2026
These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.
Install this skill
Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.
An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.
Repository context
- Repository stars (not skill ratings)
- 24
- Repository last updated
- Oct 13, 2026
- License
- MIT
Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.
- claude Repository topic
- claude-skills Repository topic
- skill-me Repository topic
- testing Derived category
Files and excerpt
File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.
Browse files
SKILL.mdcore · 8,057 bytes
Read excerpt
# LLM Evaluation
A prompt or model change that looks fine on the three examples you tried will break inputs you did not try, and without an eval you find out from users. The costly mistake this skill prevents is shipping on vibes: an uncalibrated judge or a ten-example "test" gives you a number that moves randomly, so real regressions hide inside noise. Build the dataset first, calibrate the judge before trusting it, and gate changes in CI.
## Operating procedure
Follow the order: dataset before metrics, calibration before judging, pipeline before iteration. A judge calibrated against nothing is a random-number generator with confidence.
### Step 1: Gather inputs
Collect these before building anything. Where the user cannot answer, use the default and label the value a guess.
1. Task definition - what the system produces, in one sentence.
2. What "good" means - 3-5 quality dimensions (correctness, groundedness, tone, format). Default: correctness plus format validity.