Menu

Individual skill

metric-validation-harness

0upvotes

Empirically validates a software metric before trusting or optimizing it — point it at any candidate metric (a command that takes a path and prints one number) plus a corpus, and it runs experiments that try to falsify each property a good metric must have. Checks determinism (same input, same number across runs and hash seeds), invariance to cosmetic edits (also an anti-gaming probe), monotonicity under construct-increasing edits, discrimination, robustness on edge inputs, near-linear tractability, and construct validity (convergent, discriminant vs LOC, predictive AUC, lift over a baseline). Trigger whenever someone proposes, reviews, tunes, or ships a metric, score, or index, asks "is this metric any good", suspects a score tracks LOC or jumps between runs, or builds a deterministic optimization target. It is the empirical companion to the deterministic-metric-design skill and is read-only.

skills/.experimental/metric-validation-harness/SKILL.md

Read pinned source

Source commit cf93c57cac89d6fc3e4194686000411567f5caf3

What was checked

Matches the observed repository HEAD
Yes
Scanner rules are current
No
Repository scan complete
No
Skill scan complete
No
Observed behavior
Not checked yet
Last checked
Nov 5, 2026

These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.

Install this skill

Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.

An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.

Repository context

Repository stars (not skill ratings)
222
Repository last updated
Aug 15, 2026
License
MIT

Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.

Files and excerpt

File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.

Browse files
Read excerpt
# Metric Validation Harness

Point this harness at a candidate metric and a corpus, and it runs experiments that try to **falsify** each property a trustworthy, optimizable metric must have. It is the empirical companion to `deterministic-metric-design`: that skill tells you to *prove* monotonicity, invariance, determinism, and construct validity; this skill *runs the experiment* and reports PASS/FAIL, each result mapped to the design-skill category it checks.

**Read-only.** It computes and reports; it never modifies your metric, the corpus, or any external state. Safe to run unsupervised.

## When to Apply

- Someone proposes, reviews, tunes, or ships a metric / score / index and you need evidence it is sound
- A score "feels off" — you suspect it tracks LOC, jumps between runs, or saturates
- You are about to let an agent **optimize** a metric and need to know it can't be gamed by cosmetic edits
- You built a candidate per `deterministic-metric-design` and want to empirically confirm the properties you argued for
- You are choosing between two metrics and need to know which actually predicts the outcome (and beats a trivial baseline)

## Workflow Overview
Back to top