comet-ml/opik-mcp

# Opik SDK Reference

Opik is an open-source LLM observability platform. This skill is a **reference**
for the SDK. To instrument a codebase step by step (detect frameworks, add
config, emit and verify a trace), use the task-shaped `opik-instrument` skill.

## Core concepts

A trace is one execution path (one request → one response). Spans are the
operations inside it and form a hierarchy.

### Span types — the ONLY valid values

| Type | Use for |

# Annotate — A Review Queue Annotators Can Read

**Definition of done:** the **queue exists with the view saved, and the user knows where to check it** — the write returned without a `warning`, and the answer carries the queue's link. A view is built from the project's **real** traces or threads, never from a guessed shape. If the queue or the view can't be saved, stop at the **first** genuine blocker and return **exactly one** next step.

Operate: **look at real items first, cover every shape you saw, show text not JSON, keep scoring in Opik's own panel.** This skill writes to Opik only and changes no application code.

## Inputs

`/opik-annotate <what annotators judge>` ("the drafted reply", "whether the agent solved the ticket"), `/opik-annotate <queue name or id>` (change that queue's view), or a request to remove a view. Infer the rest; optional overrides: project (default: configured) · scope (default: `trace`; `thread` for whole conversations) · feedback definitions (default: the queue's own).

## Activation — the only in-scope work

### 1. Resolve the project and the queue
With the Opik MCP connected: `list('annotation_queue', project_name=…)` shows the project's queues with their scope and item count; `read('annotation_queue', id_or_name)` returns one with its instructions, feedback definitions and view summary. The view's html is stated by size; `read('annotation_queue', id, fields=['annotator_view.html'])` returns it whole — read it before changing a view, and change only what was asked.

# Compare — Candidate vs Baseline Over a Suite

**Definition of done:** two experiments on the same suite — a **baseline** and a **candidate** — read back item by item, with the **per-metric deltas**, the cases that went **pass → fail** (regressions) and **fail → pass** (fixes), the worst rows, a note on whether the runs are comparable, and the **Opik compare-view link**. If only one run exists, the done state is "baseline created — rerun after the change". If the suite can't be run or read, stop at the **first** genuine blocker and return **exactly one** next step. Aggregate scores alone are not a comparison; a verdict is not this skill's job.

Operate: **run the candidate the same way the baseline was run, read both back from Opik rather than from the run's console output, name the specific cases that changed — and change no application code.** The only file this skill writes is a throwaway runner outside the repo.

## Inputs

The entry point is `/opik-compare <suite>` (run the suite now as the candidate, compare against the latest prior run), `/opik-compare <experiment-A> <experiment-B>` (read two existing runs, no new run), or `/opik-compare` right after `/opik-test` (that suite). Infer the rest; treat these as **optional overrides**:

- suite (default: `<project>-regressions`, or the one `/opik-test` just wrote to) · baseline (default: the most recent earlier experiment on the suite) · candidate name (default: `candidate-<short git sha>`) · judge model for assertions (default: the suite's / the baseline's) · runs per item (default: the suite's policy).

Ask only at a genuine, non-inferable blocker (see **Blockers**).

# Diagnose — Surface the Traces Worth Attention

**Definition of done:** a **ranked shortlist** of the online/production traces (and Diagnostics issues) worth attention, each carrying the **signal that flagged it** and its **trace id**, scoped to a project and a recent window, and ready to hand to `/opik-explain`. "Worth attention" means errored, slow, regressed, or low online-eval score — not a dump of every trace, and never offline experiment results. If the project can't be read, stop at the first genuine blocker and return one next step.

Operate: **rank by real signal over live data, surface the few things worth a look, hand the top one to `/opik-explain` — and change no code.** This skill is read-only by design.

## Inputs

The entry point is `/opik-diagnose` (the current project), `/opik-diagnose <project>`, or `/opik-diagnose <what to look for>` (e.g. "slow traces", "errors today"). Infer the rest; treat these as **optional overrides**:

- project (default: inferred from config/repo) · window (default: recent) · signal focus (default: all — errors, latency, regressions, low scores) · shortlist size (default: a handful).

Ask only at a genuine, non-inferable blocker (see **Blockers**).

# Evaluate — Build an Evaluation and Run It

**Definition of done:** an **experiment in Opik with scores**, its **link**, and a **score summary** the user can act on — produced by running the user's app (or prompt) over a set of cases with scoring that targets real failure modes. If the evaluation can't be built or run, stop at the **first** genuine blocker and return **exactly one** next step. A dataset with no run, a judge prompt with no scores, or a plan in prose is not success.

Operate: **ground the cases in what actually goes wrong, score with code before judges, judge one failure mode at a time, run once end to end, read the scores back from Opik — and change no application code.** The only file this skill writes is a runner outside the repo.

## Inputs

The entry point is `/opik-evaluate` (evaluate the app in this repo), `/opik-evaluate <what>` ("the RAG answers", "the refund flow"), or `/opik-evaluate <dataset-or-suite>` (run against existing cases). Infer the rest; treat these as **optional overrides**:

- cases source (default: existing suite/dataset → recent traces → synthetic) · scoring (default: heuristics where an expected output exists, else one judge per failure mode) · approach (default: **test suite** for agents; **dataset + `evaluate()`** when you need built-in metrics or exact matching) · judge model · sample size (default: 20–50 items).

Ask only at a genuine, non-inferable blocker (see **Blockers**).

# Explain — Root-Cause an Opik Trace and Ground It in the Code

**Definition of done:** a grounded root cause for the requested trace (or pattern), tied to **specific evidence spans** and paired with **exactly one** suggested next step. "Grounded" means the explanation names the failing/anomalous span and connects it to the code or data that produced it — not a restatement of the trace. If the target can't be fetched or read, stop at the **first** genuine blocker and return one concrete next step. A trace dump is not an explanation.

Operate: **investigate over the real trace data, reason against the repo, commit to a single most-likely root cause with its evidence — and change no code.** This skill is read-only by design.

## Inputs

The entry point is `/opik-explain <trace-id>` (one trace) or `/opik-explain <describe the behavior>` (a pattern to find and explain). Infer the rest; treat these as **optional overrides**:

- project name (default: inferred from config/repo) · time window for a pattern (default: recent) · a known-good trace to compare against.

Ask only at a genuine, non-inferable blocker (see **Blockers**).

# Instrument — Add Opik Tracing and Verify a Real Trace

**Definition of done:** a representative, safely-executed path produces a trace that is confirmed in Opik and a direct trace link is returned. If verification can't be completed safely or autonomously, stop at the **first** genuine blocker and return **exactly one** concrete next step. Code edits alone are not success.

Operate: **opinionated in execution, conservative in code changes, automatic in routine decisions, uncompromising about verifying value — but never by running something unsafe.**

## Inputs

The entry point is just `/opik-instrument` (optionally `/opik-instrument <path>`). Infer everything else; treat these only as **optional overrides** the user may pass, never as required setup:

- target path (default: project root) · project name (default: inferred from the repo) · run command (default: an inferred safe path) · `migrate_prompts` (default: **false**).

Never turn inference into a questionnaire. Ask only when you hit a genuine, non-inferable blocker (see **Blockers**).

# Online Eval — Take a Judge Live

**Definition of done:** an **online evaluation rule exists on the project, is enabled, and has scored at least one new trace** — confirmed by reading a fresh trace's feedback score, not by the create call returning. The rule scores one failure mode, samples at a rate the project's volume can afford, carries a cost cap, and maps its variables to the fields the traces actually have. If traffic hasn't arrived yet, the done state is "rule live, unverified — watch this filter". If the rule can't be created, stop at the **first** genuine blocker and return **exactly one** next step.

Operate: **one failure mode per rule, sample before you scale, cap the spend, verify on a real trace — and change no application code.** This skill writes to Opik only.

## Inputs

The entry point is `/opik-online-eval <what to score>` ("hallucinations", "the refund-window answer", "refusals"), `/opik-online-eval` right after `/opik-evaluate` validated a judge (take that judge live), or `/opik-online-eval <project>`. Infer the rest; treat these as **optional overrides**:

- project (default: configured) · scope (default: **trace**; `span` for one model call, `thread` for whole conversations) · sampling rate (default: 1.0 under ~1k traces/day, else 0.1–0.2) · filters (default: none; typical: an `environment` or tag filter) · judge model · cost cap (default: set one) · variable mapping (default: `input → input`, `output → output`).

Ask only at a genuine, non-inferable blocker (see **Blockers**).

# Optimize — Improve a Prompt Against Data

**Definition of done:** an **optimized prompt** whose score on the metric **beats the baseline on data it was not tuned on**, the **optimization run link** in Opik, the cost of getting there, and the winner **saved as a new prompt version** (when the prompt lives in the library) — with the swap into code left as the next step. If the optimization can't run within a stated budget, stop at the **first** genuine blocker and return **exactly one** next step. A prompt that scores higher only on its own training items is not an improvement.

Operate: **measure the baseline first, state the budget before spending it, hold data out, pick the algorithm for the failure you see, save the winner where it can be versioned — and change no application code.** The only file this skill writes is a runner outside the repo; the prompt is saved to Opik, not into the codebase.

## Inputs

The entry point is `/opik-optimize <prompt-name>` (a prompt-library prompt), `/opik-optimize <path or function>` (a prompt in code), or `/opik-optimize` (find the system prompt in this repo). Infer the rest; treat these as **optional overrides**:

- dataset (default: existing dataset for the project → export the regression suite → build from traces) · metric (default: heuristic on `expected_output` if present, else one binary judge) · algorithm (default: `MetaPromptOptimizer`) · budget (default: `n_samples=50`, `max_trials=10`) · model · validation split (default: hold out 20%).

Ask only at a genuine, non-inferable blocker (see **Blockers**).

# Test — Capture a Failing Case as a Repeatable Check

**Definition of done:** one test-suite item exists in Opik that reproduces the failure — its `data` carries the failing input (and the expected output when one is known), and it has **one or two binary assertions** that would have failed on the bad trace and pass on a correct one. The item is in a named suite scoped to the project, is confirmed by reading it back, and is ready for `/opik-compare` to run. If the case can't be captured, stop at the **first** genuine blocker and return **exactly one** next step. Writing a pytest file, or describing a test in prose, is not success.

Operate: **extract the case from real trace data, write assertions a judge can decide with a yes or no, store them where `/opik-compare` will find them — and change no application code.** This skill writes to Opik, never to the repo.

## Inputs

The entry point is `/opik-test <trace-id>` (capture that trace), `/opik-test <describe the failure>` (find the trace, or capture from the description alone), or `/opik-test` right after `/opik-explain` (capture the trace it just explained). Infer the rest; treat these as **optional overrides**:

- suite name (default: `<project>-regressions`) · project (default: the trace's project, else the configured one) · expected output (default: none — assertions carry the expectation) · execution policy (default: the suite's; `runs_per_item: 3, pass_threshold: 2` only for an intermittent failure).

Ask only at a genuine, non-inferable blocker (see **Blockers**).

# Verify — Ship or Hold, Against a Policy You Can Read

**Definition of done:** one verdict — **`ship`**, **`hold`**, **`needs_review`**, or **`insufficient_evidence`** — computed from a **declared policy** over the baseline-vs-candidate numbers, with **every criterion listed with its threshold, the observed value, and pass/fail**, the cases behind any failure named, and the compare-view link. The policy is either the repo's `opik-release-policy.yaml` or the documented defaults, and the report says which. If the two runs can't be read or aren't comparable, stop at the **first** genuine blocker and return **exactly one** next step. "Looks good to me" is not a verdict; a verdict without its criteria is not one either.

Operate: **apply the policy mechanically, show your arithmetic, refuse to ship on a judge nobody validated, and change no application code.** The only file this skill may write is the policy file, and only when the user says so. It never deploys.

## Inputs

The entry point is `/opik-verify` right after `/opik-compare` (its baseline and candidate), `/opik-verify <suite>` (the two most recent runs on the suite), or `/opik-verify <baseline-id> <candidate-id>`. Infer the rest; treat these as **optional overrides**:

- policy (default: `opik-release-policy.yaml` at the repo root or under `.opik/`, else the defaults below) · which experiments (default: as above) · `--record` (default: off — write the verdict into the candidate experiment's config).

Ask only at a genuine, non-inferable blocker (see **Blockers**).