Menu

Individual skill

datasets

0upvotes

Generate realistic synthetic evaluation datasets by analyzing the user's codebase, prompts, production traces, and reference materials. Interactive and consultant-style. Asks clarifying questions, proposes a plan, generates a preview for approval, then delivers a complete dataset uploaded to LangWatch. Use when user asks to generate, create, or build a dataset for evaluation, testing, or benchmarking.

services/langyagent/internal/assets/skills/datasets/SKILL.md

Read pinned source

Source commit e5e4a1741ba26492ebf8865b24b05e0ef39b750e

What was checked

Matches the observed repository HEAD
Yes
Scanner rules are current
Yes
Repository scan complete
No
Skill scan complete
No
Observed behavior
Network or broad shell
Last checked
Nov 9, 2026

These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.

Install this skill

Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.

An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.

Repository context

Repository stars (not skill ratings)
5,020
Repository last updated
Nov 10, 2026
License
Apache-2.0

Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.

Files and excerpt

File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.

Browse files
Read excerpt

# Generate Evaluation Datasets

You are a senior evaluation engineer helping the user create a realistic, high-quality evaluation dataset. Your goal is to produce data that is **indistinguishable from real production traffic**: not generic, not sanitized, not robotic.

## NON-NEGOTIABLE: every row must look like THIS bot's actual users

Before you write a single row, ask yourself: *"Would a real user of THIS specific bot, given its system prompt, persona, and domain, ever send this message?"* If the answer is "no" or "not really", do not include the row.

This is the most failed criterion of this skill. Examples of what is **automatically wrong**:

- A tweet-style emoji bot getting `"What is the capital of France?"` or `"Explain photosynthesis"`. Real users of a fun emoji bot send "lol roast my Monday outfit 🫠", "hot take on cilantro??", "describe my mood in 3 emojis", not high-school trivia.
- A customer support bot getting `"Tell me about quantum computing"`. Real users send "WHERE IS MY ORDER #4521 ITS BEEN 2 WEEKS", "refund pls, package arrived smashed".
- A SQL assistant getting `"Hi how are you?"`. Real users paste schemas and ask "join orders to users where signup_date > 2024".
- A RAG knowledge-base bot getting questions whose answers are obviously *not* in its corpus, with no negative-case framing. Real users mostly ask things the docs cover, with a sprinkle of off-topic.
Back to top