ai-evals-course/evals-skills

# Build a Custom Annotation Interface

## Overview

Build an HTML page that loads traces from a data source (JSON/CSV file), displays one trace at a time with Pass/Fail buttons, a free-text notes field, and Next/Previous navigation. Save labels to a local file (CSV/SQLite/JSON). Then customize to the domain using the guidelines below.

## Data Display

Format all data in the most human-readable representation for the domain. Emails should look like emails. Code should have syntax highlighting. Markdown should be rendered. Tables should be tables. JSON should be pretty-printed and collapsible.

- **Collapse repetitive elements.** If every trace shares the same system prompt, put it in a `<details>` toggle.
- **Extract and surface key metadata.** If traces contain a property name, client type, or session ID buried in the data, extract it and display it prominently as a header or badge.
- **Color-code by role or status.** Use left-border colors to distinguish user messages, assistant messages, tool calls, and system prompts at a glance.
- **Group related elements visually.** Tool calls and their responses should be visually linked (indentation, shared border).

# Error Discovery Skill

You are running an interactive error analysis session. The user has a dataset (JSONL, CSV, JSON, etc.) of LLM outputs or traces and wants to discover failure modes by reviewing samples.

This skill is meant for interactive sessions only.

This skill has two parts:
- This file covers phases 1 through 4. You read the data, design the UI, build it, and select samples.
- [review-loop.md](review-loop.md) covers phase 5. You run the interactive review session.

Read this file first to build everything. Once the app is running and the human starts reviewing, follow [review-loop.md](review-loop.md).

## Progress updates

# Eval Audit

Inspect an LLM eval pipeline and produce a prioritized list of problems with concrete next steps.

## Overview

1. Gather eval artifacts: traces, evaluator configs, judge prompts, labeled data, metrics dashboards
2. Run diagnostic checks across six areas
3. Produce a findings report ordered by impact, with each finding linking to a fix

## Prerequisites

Access to eval artifacts (traces, evaluator configs, judge prompts, labeled data) via an observability MCP server or local files. If none exist, skip to "No Eval Infrastructure."

# Evals Start

This plugin splits eval work into targeted skills. Your job here is small: find the row below that matches the user's situation, tell the user which skill you are loading and why, then load that skill and follow its workflow from start to finish instead of improvising your own version of it.

| Situation | Skill to load |
|---|---|
| Has traces, wants to find failure modes, no established taxonomy yet | `error-discovery` |
| Has an existing eval pipeline and wants to know if it can be trusted | `eval-audit` |
| Has a known failure mode that code can check (e.g., format, schema, regex, execution) | `write-code-eval` |
| Has a known failure mode and wants an LLM judge for it | `write-judge-prompt` |
| Has an LLM judge or evaluator and wants to check its quality | `validate-evaluator` |
| Has no traces to review yet | `generate-synthetic-data`, then `error-discovery` |
| Wants a custom annotation interface for some other labeling task | `build-review-interface` |
| Wants to evaluate a RAG pipeline | `evaluate-rag` |

# Evaluate RAG

## Overview

1. Do error analysis on end-to-end traces first. Determine whether failures come from retrieval, generation, or both.
2. Build a retrieval evaluation dataset: queries paired with relevant document chunks.
3. Measure retrieval quality with Recall@k (most important for first-pass retrieval).
4. Evaluate generation separately: faithfulness (grounded in context?) and relevance (answers the query?).
5. If retrieval is the bottleneck, optimize chunking via grid search before tuning generation.

## Prerequisites

Complete error analysis on RAG pipeline traces before selecting metrics. Inspect what was retrieved vs. what the model needed. Determine whether the problem is retrieval, generation, or both. Fix retrieval first.

# Generate Synthetic Data

Generate diverse, realistic test inputs that cover the failure space of an LLM pipeline.

## Prerequisites

Before generating synthetic data, identify where the pipeline is likely to fail. Ask the user about known failure-prone areas, review existing user feedback, or form hypotheses from available traces. Dimensions (Step 1) must target anticipated failures, not arbitrary variation.

## Core Process

### Step 1: Define Dimensions

Dimensions are axes of variation specific to your application. Choose dimensions based on where you expect failures.

# Validate Evaluator

Calibrate an LLM judge against human judgment.

## Overview

1. Split human-labeled data into train (10-20%), dev (40-45%), test (40-45%)
2. Run judge on dev set and measure TPR/TNR
3. Iterate on the judge until TPR and TNR > 90% on dev set
4. Run once on held-out test set for final TPR/TNR
5. Apply bias correction formula to production data

## Prerequisites

# Write a code evaluator

Start with a failure mode found through error analysis. Write one check for that failure mode, much like a unit test that asserts what should hold for each trace.

1. State the rule and identify the trace fields or reference data the check needs. If the rule requires interpretation, use `write-judge-prompt`.
2. Implement the check in the project's language and eval framework. Return a result and a reason in the format that framework expects.
3. Test known passes and failures, including borderline cases. Run the check on available traces and inspect mistakes. If the rule uses a proxy for human judgment, compare its results with human labels.

## Examples

| Failure mode | Possible check |
|---|---|
| Invalid output structure | Parse the output and check required fields |
| Missing or forbidden text | Match a string or pattern |

# Write LLM-as-Judge Prompt

Design a binary Pass/Fail LLM-as-Judge evaluator for one specific failure mode. Each judge checks exactly one thing.

## Prerequisites

- Error analysis is complete. The failure mode is identified.
- You have human-labeled traces for this failure mode (at least 20 Pass and 20 Fail examples).
- A code-based evaluator cannot check this failure mode. Exhaust code-based options before reaching for a judge — many failure modes that seem subjective reduce to keyword checks, regex, or API calls when you understand the domain. Example: detecting whether an AI interviewing coach suggests "general" questions (asking about typical behavior instead of a specific past event) seems to require semantic understanding, but in practice a keyword check for words like "usually," "typical," and "normally" could work quite well.

## The Four Components

Every judge prompt requires exactly four components: