Menu

Individual skill

io-bound-data-processing

0upvotes

Processing, transforming, or moving datasets that may exceed RAM on a single low-compute box — covers memory discipline (streaming, generators, dtype shrinkage), I/O access patterns (sequential vs random, mmap, async), data formats (Parquet vs CSV vs JSON, predicate pushdown), chunking & batching, spill-to-disk (external merge sort, DuckDB/Polars), pipelining (bounded queues, backpressure, checkpointing), codec selection (zstd/lz4/gzip), concurrency for I/O-bound workloads (asyncio, threads, prefetch), and observability (iowait vs CPU%, rows/sec, py-spy/strace). Trigger on "process a large file", "stream this", "out-of-core", "OOM kill", "this is slow", or code with `pd.read_csv` of multi-GB files, `requests.get(...).content` on big bodies, `BytesIO` on unbounded inputs, per-row INSERTs, sequential `requests.get` loops, falling `tqdm` rates — even if I/O or memory isn't mentioned. Complement to computer-science-algorithms.

skills/.experimental/io-bound-data-processing/SKILL.md

Read pinned source

Source commit cf93c57cac89d6fc3e4194686000411567f5caf3

What was checked

Matches the observed repository HEAD
Yes
Scanner rules are current
No
Repository scan complete
No
Skill scan complete
No
Observed behavior
Not checked yet
Last checked
Nov 5, 2026

These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.

Install this skill

Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.

An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.

Repository context

Repository stars (not skill ratings)
222
Repository last updated
Aug 15, 2026
License
MIT

Observed agent integration: codex

Files and excerpt

File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.

Browse files
Read excerpt
# Community I/O-bound data processing on constrained resources Best Practices

A reference for engineers processing datasets larger than RAM on a single low-compute box. Organized by execution-lifecycle impact: rules near the top of the table govern *whether the job runs at all*; rules near the bottom shave the last 10 %. Optimize from the top of the waterfall.

**Scope:** the patterns that show up in real ETL / data-engineering / batch work on a laptop, a 2-vCPU container, or a Raspberry Pi-class node — streaming, formats, chunking, spill, backpressure, codecs, and the concurrency model that actually matches an I/O-bound bottleneck. Out of scope (covered elsewhere): the algorithmic primitives themselves (see [`computer-science-algorithms`](../computer-science-algorithms/)), distributed compute beyond a single box (use Spark/Dask), and database-engine internals (see official docs).

Distilled from [Apache Arrow / Parquet docs](https://arrow.apache.org/docs/), [Polars User Guide](https://docs.pola.rs/), [DuckDB docs](https://duckdb.org/docs/), [pandas — Scaling to large datasets](https://pandas.pydata.org/docs/user_guide/scale.html), Linux man pages ([mmap(2)](https://man7.org/linux/man-pages/man2/mmap.2.html), [sendfile(2)](https://man7.org/linux/man-pages/man2/sendfile.2.html), [posix_fadvise(2)](https://man7.org/linux/man-pages/man2/posix_fadvise.2.html)), Brendan Gregg's [USE method](https://www.brendangregg.com/usemethod.html) and *Systems Performance*, Kleppmann's [*Designing Data-Intensive Applications*](https://dataintensive.net/), and the [zstd](https://github.com/facebook/zstd) / [lz4](https://github.com/lz4/lz4) reference benchmarks.

## When to Apply

Reach for these rules when:

- A job OOM-kills, swaps, or runs much slower than expected on a small box
- Input is *larger* than RAM and you need to scan, filter, aggregate, sort, or join it
- A pipeline has unbounded buffers between stages, or memory grows linearly during a "streaming" job