Menu

Individual skill

opik-compare

Run a candidate against the baseline over an Opik test suite and read the numbers back — which cases broke, which got fixed, the per-metric deltas, worst rows, and whether the two runs are comparable — with the Opik compare-view link. Runs via the SDK; reads results via the MCP when connected. Does not issue a ship/no-ship verdict. Use for "did my fix work", "compare against the baseline", "run the regression suite", "why did quality drop", "which cases regressed", "compare these two experiments". Not for the ship/hold decision itself (use verify), live production triage (use diagnose), building an evaluation from scratch (use evaluate), or capturing a single case (use test).

plugins/opik/skills/opik-compare/SKILL.md

Read pinned source

Source commit 9f66464c5fb5f58f70f277f48cbccdb93025bfdd

What was checked

Matches the observed repository HEAD
Yes
Scanner rules are current
No
Repository scan complete
No
Skill scan complete
No
Observed behavior
Not checked yet
Last checked
Nov 6, 2026

These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.

Install this skill

Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.

An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.

Repository context

Repository stars (not skill ratings)
226
Repository last updated
Nov 6, 2026
License
Apache-2.0

Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.

Files and excerpt

File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.

Browse files
Read excerpt

# Compare — Candidate vs Baseline Over a Suite

**Definition of done:** two experiments on the same suite — a **baseline** and a **candidate** — read back item by item, with the **per-metric deltas**, the cases that went **pass → fail** (regressions) and **fail → pass** (fixes), the worst rows, a note on whether the runs are comparable, and the **Opik compare-view link**. If only one run exists, the done state is "baseline created — rerun after the change". If the suite can't be run or read, stop at the **first** genuine blocker and return **exactly one** next step. Aggregate scores alone are not a comparison; a verdict is not this skill's job.

Operate: **run the candidate the same way the baseline was run, read both back from Opik rather than from the run's console output, name the specific cases that changed — and change no application code.** The only file this skill writes is a throwaway runner outside the repo.

## Inputs

The entry point is `/opik-compare <suite>` (run the suite now as the candidate, compare against the latest prior run), `/opik-compare <experiment-A> <experiment-B>` (read two existing runs, no new run), or `/opik-compare` right after `/opik-test` (that suite). Infer the rest; treat these as **optional overrides**:

- suite (default: `<project>-regressions`, or the one `/opik-test` just wrote to) · baseline (default: the most recent earlier experiment on the suite) · candidate name (default: `candidate-<short git sha>`) · judge model for assertions (default: the suite's / the baseline's) · runs per item (default: the suite's policy).

Ask only at a genuine, non-inferable blocker (see **Blockers**).