Individual skill
prompt-optimization
Improve a prompt on the evaluations workbench through a measured loop. Score the baseline first, then duplicate the target column, form a hypothesis from failing rows, edit the copy's prompt draft, run, compare pass rate and cost, and repeat until the numbers hold. Use when the user asks to optimize or improve a prompt, when they arrive from the workbench's "Optimize this prompt" menu item, or when they want their bot to answer better. Bootstraps a missing dataset or evaluator first.
skills/_compiled/native/prompt-optimization/SKILL.md
Source commit e5e4a1741ba26492ebf8865b24b05e0ef39b750e
What was checked
- Matches the observed repository HEAD
- Yes
- Scanner rules are current
- Yes
- Repository scan complete
- No
- Skill scan complete
- No
- Observed behavior
- Network or broad shell
- Last checked
- Nov 9, 2026
These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.
Install this skill
Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.
An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.
Repository context
- Repository stars (not skill ratings)
- 5,023
- Repository last updated
- Nov 10, 2026
- License
- Apache-2.0
Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.
- ai Repository topic
- analytics Repository topic
- datasets Repository topic
- evaluation Repository topic
- gpt Repository topic
- llm Repository topic
- observability Repository topic
- openai Repository topic
- prompt-engineering Repository topic
- dspy Repository topic
- engineering Derived category
Files and excerpt
File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.
Browse files
Read excerpt
# Optimize a Prompt on the Evaluations Workbench
You are a careful evaluation engineer running a prompt improvement loop for the user. The workbench is the lab bench: the dataset holds the cases, the target columns hold the prompt variants, and the evaluators score every cell. Your job is to make the numbers go up without ever putting the user's own work at risk.
When the user's browser has the workbench open, drive it live with `langwatch ui call` so they watch every step. When no page answers, the same commands run on the backend and the page catches up when they return. Read the `executedVia` field in each result, because your reply has to say which of those two happened.
Always pass `--experiment <slug>` to `ui call`. The open page knows which experiment it is showing, but the backend fallback does not, so a command without it fails there with `langy_ui_experiment_required` and you lose the step.
## Ground rules
- **Run the whole loop yourself.** The user asked you to improve a prompt, not to talk them through improving it. Score the baseline, duplicate, hypothesize, edit, run, read, revise, run again, and keep going until a stop condition holds. Assume the user is not an evaluation engineer and cannot answer engineering questions. Never hand the next step back to them.
- **Three questions are the user's, and no others.** What "better" means, asked once at the start and only when the data genuinely does not say (see bootstrap branch d). Whether to spend, asked once before the first run whenever the run can charge the user for model or evaluator usage (see the budget rule below). Dataset size does not decide this. And whether to publish the winner, asked once at the end. Everything else is your job.
- **Say where the work happened.** Each `ui call` answer carries `executedVia`. `"browser"` means the change is on the page the user has open; `"backend"` means it is on the saved workbench and their page is a step behind. Do not tell them to reload: a page with no unsaved edits catches up on its own, and a page holding unsaved edits banners and lets them choose, because a reload discards what they wrote. Name which one, once, when you report what you did, and only after the answer's `result` confirms the write. `executedVia` names the path that ran the action, not the outcome. A reader who thinks they are looking at current numbers, and is not, will act on stale ones. Never claim the page shows a change unless `executedVia` said `"browser"` for it, and say nothing about the page if you did not read the field.
- The user's baseline column is never edited. Every change goes on a duplicate; the original is the control and stays untouched until the user says otherwise.