Menu

Individual skill

Assay

0upvotes

Assertion-first eval framework: code asserts, calibrated judges, pass@k, A/B comparison, agentic evals, judgment-layer question systems and code-extract-decide-write pipelines. USE WHEN eval, evaluate, benchmark, judge, judge prompt, judge panel, regression test, assert, llm-rubric, calibrate judge, judged residue, pass@k, pass^k, grade output, compare prompts or models, A/B test, which model did we like, model retro, ablation, with/without, test agent, agentic eval, noul/choice/score question, glance row, judged miss, proof this, check the output, did it come out as asked, design judgment questions, question system, design a pipeline, extract values, Clairic, LLM-only baseline, judgment layer status/report/tune, is this caller ready to enforce. NOT FOR scientific-method framing (Science), property/mutation testing (Hardening), live UI verification (Interceptor), non-eval prompts (Prompting), TypeSafe apps outside LifeOS (typesafe-ai).

LifeOS/install/skills/Assay/SKILL.md

Read pinned source

Source commit f6ab5e0228bfb356c82edbf52b69344dbd585486

What was checked

Matches the observed repository HEAD
Yes
Scanner rules are current
Yes
Repository scan complete
No
Skill scan complete
No
Observed behavior
Network or broad shell
Last checked
Nov 10, 2026

These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.

Install this skill

Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.

Standard SKILL.md format has not been attested at this commit. No installation command is available; inspect the source.

Repository context

Repository stars (not skill ratings)
19,782
Repository last updated
Nov 5, 2026
License
MIT

Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.

Files and excerpt

File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.

Browse files
Read excerpt

# Assay — Assertion-First AI Evaluation

**Contents:** Ideal State · Intent Routing · The taxonomy · Choosing a check: tier first, then shape · Selection, acceptance, and unavailable evidence · The canonical path · Doctrine (from Anthropic; encode, don't restate) · The image lane (visual evals) · Harness integration · Judgment-layer question systems · Data & Config · Gotchas · Examples

An eval gives an AI an input, then applies **assertions** to its output to measure success (Anthropic's definition). A case is `{id, prompt, assert:[...]}`. Each assertion is either deterministic (code, free) or model-graded (a calibrated judge). Cases run multiple trials and report **pass^k** (every trial passes; the reliability metric and the regression gate) and **pass@k** (any trial passes). Proof applies the same tiers to one run's output (the film, image, page, email or document it made) against what was asked, before the principal sees it: `Intents/Proof.md`, with the whole system in `LIFEOS/DOCUMENTATION/Assay/AssaySystem.md`. CreateQuestions designs the fast tier itself: it turns a fuzzy decision into a measured judgment-layer question system (`LIFEOS/DOCUMENTATION/Assay/AssaySystem.md` § Judgment layer).

Sources: [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) (2026-01), [A Statistical Approach to Model Evaluations](https://www.anthropic.com/research/statistical-approach-to-model-evals) (2024-11), [develop tests](https://platform.claude.com/docs/en/docs/build-with-claude/develop-tests), the eval-awareness contamination post (2026-03), and the live `claude plugin eval` surface.

**Freshness contract:** "aligned to Anthropic's doctrine" is a live claim. When designing a new suite class or touching `## Doctrine`, re-fetch the demystifying-evals post and flag where it has moved. Report divergence, never auto-adopt; an unreachable URL never blocks a run.

## Ideal State

- Every check sits at the lowest tier that can honestly answer it (code, glance, judge, human), and only code runs on a blocking path.
- Every score ships with pass^k, SEM and a 95% CI from the trials that completed, with the requested and completed trial counts beside it; fewer completed trials than requested never claims pass^k.
Back to top