Individual skill
opik-online-eval
Take a judge live on production traffic — create an Opik online evaluation rule (LLM-as-judge or Python metric) on a project with sampling, filters, variable mapping, and a cost cap, then confirm new traces are being scored. Works over the SDK's REST client; reads rules and score names via the MCP when connected. Returns the rule, the score name it emits, and how to watch it. Use for "score production traces", "monitor hallucinations in prod", "take this judge live", "set up an online evaluation rule", "alert me when quality drops". Not for offline experiments (use evaluate or compare) or for finding what is already broken (use diagnose).
plugins/opik/skills/opik-online-eval/SKILL.md
Source commit 9f66464c5fb5f58f70f277f48cbccdb93025bfdd
What was checked
- Matches the observed repository HEAD
- Yes
- Scanner rules are current
- No
- Repository scan complete
- No
- Skill scan complete
- No
- Observed behavior
- Not checked yet
- Last checked
- Nov 6, 2026
These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.
Install this skill
Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.
An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.
Repository context
- Repository stars (not skill ratings)
- 226
- Repository last updated
- Nov 6, 2026
- License
- Apache-2.0
Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.
- mcp-server Repository topic
- claude-code Repository topic
- llm-observability Repository topic
- mcp Repository topic
- model-context-protocol Repository topic
- opik Repository topic
- python Repository topic
- generative-ai Repository topic
- engineering Derived category
Files and excerpt
File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.
Browse files
Read excerpt
# Online Eval — Take a Judge Live
**Definition of done:** an **online evaluation rule exists on the project, is enabled, and has scored at least one new trace** — confirmed by reading a fresh trace's feedback score, not by the create call returning. The rule scores one failure mode, samples at a rate the project's volume can afford, carries a cost cap, and maps its variables to the fields the traces actually have. If traffic hasn't arrived yet, the done state is "rule live, unverified — watch this filter". If the rule can't be created, stop at the **first** genuine blocker and return **exactly one** next step.
Operate: **one failure mode per rule, sample before you scale, cap the spend, verify on a real trace — and change no application code.** This skill writes to Opik only.
## Inputs
The entry point is `/opik-online-eval <what to score>` ("hallucinations", "the refund-window answer", "refusals"), `/opik-online-eval` right after `/opik-evaluate` validated a judge (take that judge live), or `/opik-online-eval <project>`. Infer the rest; treat these as **optional overrides**:
- project (default: configured) · scope (default: **trace**; `span` for one model call, `thread` for whole conversations) · sampling rate (default: 1.0 under ~1k traces/day, else 0.1–0.2) · filters (default: none; typical: an `environment` or tag filter) · judge model · cost cap (default: set one) · variable mapping (default: `input → input`, `output → output`).
Ask only at a genuine, non-inferable blocker (see **Blockers**).
