Individual skill
kafka-pipelines
Designs Kafka streaming pipelines end to end - partition-count sizing math, key choice, consumer-group sizing, offset strategy, delivery semantics, poll tuning, and dead-letter handling - with production configs. Use when someone asks "how many partitions should this topic have", "my consumer group keeps rebalancing", "consumer lag keeps growing", "how do I handle poison messages", or is designing a new topic or event-driven pipeline. Do NOT use for Spark batch or PySpark job tuning - use spark-jobs instead; do NOT use for HTTP webhook ingestion reliability - use webhook-receiver-hardener instead.
skills/kafka-pipelines/SKILL.md
Source commit 97a13607d669157a4a16d5ba79b77f208319f7b9
What was checked
- Matches the observed repository HEAD
- Yes
- Scanner rules are current
- No
- Repository scan complete
- No
- Skill scan complete
- No
- Observed behavior
- Not checked yet
- Last checked
- Nov 6, 2026
These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.
Install this skill
Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.
An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.
Repository context
- Repository stars (not skill ratings)
- 24
- Repository last updated
- Oct 13, 2026
- License
- MIT
Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.
- claude Repository topic
- claude-skills Repository topic
- skill-me Repository topic
- engineering Derived category
Files and excerpt
File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.
Read excerpt
# Kafka Pipelines
Partition count is the one Kafka decision that is nearly permanent: it caps consumer parallelism forever, and increasing it later rehashes keys and breaks per-key ordering for every keyed consumer. The two costly mistakes this skill prevents are under-partitioning a topic that later gets hot, and committing offsets before processing - which silently loses records on the first crash.
## Operating procedure
### Step 1: Gather inputs
Collect these before choosing any number. Label estimates as guesses and revisit after load testing.
1. Peak produce throughput (MB/s and messages/s) - peak, not average. Default guess: 3x the average.
2. Average message size.
3. Per-record consumer processing time, including downstream calls (DB writes, HTTP). This usually dominates.
4. Ordering requirement: which records must stay ordered relative to each other? That defines the key.