Menu

Individual skill

opik-evaluate

0upvotes

Build an LLM evaluation and run it against the app, returning an Opik experiment with scores and its link. Picks a test suite with judge assertions or a dataset with metrics, sources cases from traces or synthetic data, scores heuristics-first then one-failure-mode judges, runs client-side via the SDK or server-side for prompt-only targets, and reads the scores back. Covers RAG evaluation, error analysis, writing and validating LLM judges against human labels, and auditing an existing eval pipeline. Use for "evaluate my agent", "measure quality", "build an eval", "write an LLM judge for hallucinations", "audit our evaluation pipeline", "how good is my RAG", "set up evals for this". Not for before/after on an existing suite (use compare), one regression case (use test), scoring production traffic (use online-eval), or the ship/hold decision (use verify).

plugins/opik/skills/opik-evaluate/SKILL.md

Read pinned source

Source commit 9f66464c5fb5f58f70f277f48cbccdb93025bfdd

What was checked

Matches the observed repository HEAD
Yes
Scanner rules are current
No
Repository scan complete
No
Skill scan complete
Yes
Observed behavior
Not checked yet
Last checked
Nov 6, 2026

These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.

Install this skill

Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.

An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.

Repository context

Repository stars (not skill ratings)
226
Repository last updated
Nov 6, 2026
License
Apache-2.0

Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.

Files and excerpt

File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.

Browse files
Read excerpt

# Evaluate — Build an Evaluation and Run It

**Definition of done:** an **experiment in Opik with scores**, its **link**, and a **score summary** the user can act on — produced by running the user's app (or prompt) over a set of cases with scoring that targets real failure modes. If the evaluation can't be built or run, stop at the **first** genuine blocker and return **exactly one** next step. A dataset with no run, a judge prompt with no scores, or a plan in prose is not success.

Operate: **ground the cases in what actually goes wrong, score with code before judges, judge one failure mode at a time, run once end to end, read the scores back from Opik — and change no application code.** The only file this skill writes is a runner outside the repo.

## Inputs

The entry point is `/opik-evaluate` (evaluate the app in this repo), `/opik-evaluate <what>` ("the RAG answers", "the refund flow"), or `/opik-evaluate <dataset-or-suite>` (run against existing cases). Infer the rest; treat these as **optional overrides**:

- cases source (default: existing suite/dataset → recent traces → synthetic) · scoring (default: heuristics where an expected output exists, else one judge per failure mode) · approach (default: **test suite** for agents; **dataset + `evaluate()`** when you need built-in metrics or exact matching) · judge model · sample size (default: 20–50 items).

Ask only at a genuine, non-inferable blocker (see **Blockers**).
Back to top