Menu

Individual skill

jev-eval

0upvotes

Build and run a labelled eval set for a System One model (Jev, Von, or any typed-decision config), then sweep criteria wordings and thresholds against it. Use when a Jev/Von classification is wrong or unreliable, when choosing between the hosted API and a local open model, when tuning noul thresholds, or before shipping any typed-decision feature. Produces an accuracy-by-wording matrix and a calibrated threshold.

jev-eval/SKILL.md

Read pinned source

Source commit b2267d9171af4f56848d82992381b20b89b44056

What was checked

Matches the observed repository HEAD
Yes
Scanner rules are current
Yes
Repository scan complete
No
Skill scan complete
No
Observed behavior
Network or broad shell
Last checked
Nov 10, 2026

These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.

Install this skill

Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.

An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.

Repository context

Repository stars (not skill ratings)
425
Repository last updated
Oct 20, 2026
License
MIT

Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.

Files and excerpt

File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.

Browse files
Read excerpt

# Evaluating a typed-decision config

**The eval set is the product.** A System One model's accuracy is dominated by how the
question was written, and the failure mode is silent — it returns a confident, type-valid,
wrong answer. Without labels you cannot tell a bad question from a bad model.

Measured: rewriting the criteria moved an open model from **4/15 to 14/15** on 15 records.
No model change. Then the same comparison at 150 records put that model at **61%** overall
against Jev's **97%** — the 15-record read was an artifact of a small, easy set. Both facts
are the point: wording swings results, and small sets lie about which way.

## Run it

```bash
Back to top