Individual skill
nlp
Use when choosing how to tokenize text or which transformer type fits an NLP task, when a tokenizer over-fragments non-English text or inflates token cost, when picking a language metric, or when classification, NER or summarization output looks wrong and it is unclear whether the tokenizer, the architecture or the metric is at fault. Covers subword tokenizers, encoder versus decoder versus encoder-decoder choice, sentence embeddings, and the metric families. NOT retrieval or vector search (that is `embeddings-search`), NOT the RAG loop (that is `rag`), NOT prompt wording (that is `prompt-engineering`), NOT training the network (that is `finetuning`).
skills/nlp/SKILL.md
Source commit d9d6bb338f66cc1e23998585e580a0ea45bbcb82
What was checked
- Matches the observed repository HEAD
- Yes
- Scanner rules are current
- No
- Repository scan complete
- No
- Skill scan complete
- No
- Observed behavior
- Not checked yet
- Last checked
- Nov 6, 2026
These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.
Install this skill
Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.
An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.
Repository context
- Repository stars (not skill ratings)
- 566
- Repository last updated
- Nov 4, 2026
- License
- MIT
Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.
- agent-skills Repository topic
- agentic-ai Repository topic
- ai-agents Repository topic
- ai-coding Repository topic
- claude-code Repository topic
- cli Repository topic
- cline Repository topic
- codex Repository topic
- coding-assistant Repository topic
- cursor Repository topic
- nlp Declared skill tag
- tokenization Declared skill tag
- bpe Declared skill tag
- wordpiece Declared skill tag
- ner Declared skill tag
- text-classification Declared skill tag
- bleu Declared skill tag
- rouge Declared skill tag
- perplexity Declared skill tag
- transformers Declared skill tag
- engineering Derived category
Files and excerpt
File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.
Browse files
SKILL.mdcore · 14,269 bytesSKILL.mdcore · 14,269 bytesSKILL.mdcore · 14,269 bytesSKILL.mdcore · 14,269 bytesSKILL.mdcore · 14,269 bytesSKILL.mdcore · 14,269 bytesevals/README.mddoc · 774 bytesevals/cases.yamlasset · 4,526 bytesreferences/evaluation.mddoc · 5,024 bytesreferences/tokenization.mddoc · 4,812 bytes
Read excerpt
# nlp — tokenize the text, pick the model type, pick the metric
You own the **language-modeling discipline**: how raw text becomes tokens, which transformer
architecture fits a task, and which metric actually tells you whether it worked. When the
question is "which tokenizer," "BERT or GPT or T5 for this," "why does my Catalan text cost 3×
the tokens," or "is this BLEU score meaningful," this is the skill. You stop at retrieval, the
RAG loop, prompt wording, and the training step itself — those route out (below).
## Route out first (loud — do not duplicate these)
- **Retrieval embeddings + vector search** (which embedding model, chunking, recall@k, rerank)
→ [`../embeddings-search/SKILL.md`](../embeddings-search/SKILL.md). Sentence embeddings *live
here* as a task; using them to *retrieve* is theirs.
- **The retrieve → prompt → generate → answer loop** and groundedness → [`../rag/SKILL.md`](../rag/SKILL.md).