stas00/the-art-of-debugging

Executable file

find . -name "*html" -exec rm {} \;

6. **[Debugging Machine Learning Projects](https://github.com/stas00/ml-engineering/tree/master/debug)** (external)

I announce any significant updates on my twitter channel [https://twitter.com/StasBekman](https://twitter.com/StasBekman).

You can download various ebook formats of this book: * [PDF](https://huggingface.co/stas/the-art-of-debugging-book/resolve/main/Stas%20Bekman%20-%20The%20Art%20of%20Debugging.pdf?download=true)

* [PDF](https://huggingface.co/stas/the-art-of-debugging-book/resolve/main/Stas%20Bekman%20-%20The%20Art%20of%20Debugging.pdf?download=true)

* [EPUB](https://huggingface.co/stas/the-art-of-debugging-book/resolve/main/Stas%20Bekman%20-%20The%20Art%20of%20Debugging.epub?download=true)

See also the companion skills: [Machine Learning Engineering](https://github.com/stas00/ml-engineering/blob/master/skills/ml-engineering/SKILL.md) and [Stas' Python Cookbook](https://github.com/stas00/python-cookbook/blob/master/SKILL.md).

- [Machine Learning Engineering Open Book](https://github.com/stas00/ml-engineering) — training and operating large-scale ML models.

- [Stas' Python Cookbook](https://github.com/stas00/python-cookbook) — everyday Python and standard-library recipes.

If you found a bug, typo or would like to propose an improvement please don't hesitate to open an [Issue](https://github.com/stas00/the-art-of-debugging/issues) or contribute a PR.

✔ **Books:**  [Machine Learning Engineering](https://github.com/stas00/ml-engineering) |

[Machine Learning Engineering](https://github.com/stas00/ml-engineering) |

[The Art of Debugging](https://github.com/stas00/the-art-of-debugging) |

[Stas' Python Cookbook](https://github.com/stas00/python-cookbook)

✔ **Applications:**  [ipyexperiments](https://github.com/stas00/ipyexperiments)

[ipyexperiments](https://github.com/stas00/ipyexperiments)

✔ **Tools and Cheatsheets:**  [bash](https://github.com/stas00/bash-tools) |

[bash](https://github.com/stas00/bash-tools) |

[conda](https://github.com/stas00/conda-tools) |

[git](https://github.com/stas00/git-tools) |

[jupyter-notebook](https://github.com/stas00/jupyter-notebook-tools) |

[make](https://github.com/stas00/make-tools) |

[python](https://github.com/stas00/python-tools) |

[tensorboard](https://github.com/stas00/tensorboard-tools) |

[unix](https://github.com/stas00/unix-tools)

✔ **Other Machine Learning:**  [ML ways](https://github.com/stas00/ml-ways) |

[ML ways](https://github.com/stas00/ml-ways) |

[Porting](https://github.com/stas00/porting)

numbers, runs too slowly, or when the user mentions gdb, strace, py-spy,   core files, CUDA_LAUNCH_BLOCKING, ldd/nm/LD_PRELOAD, cProfile, or distributed   training hangs. Distilled from "The Art of Debugging", the latest version of which ca

training hangs. Distilled from "The Art of Debugging", the latest version of which can be found at https://github.com/stas00/the-art-of-debugging

The latest SKILL.md version can be found at https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md

> Distilled from **The Art of Debugging Open Book** by Stas Bekman - source: https://github.com/stas00/the-art-of-debugging (CC BY-SA 4.0). This skill is a condensed index; each section links back to the full chapter for depth.

Actionable methodology + copy-paste recipes for debugging Unix / Python / PyTorch programs. Apply the general loop first; then jump to the domain cheatsheet for the failure at hand. For scaling this up to large-model training/inference on r

1. **Reproduce reliably.** Get one command that triggers the bug every time. If it's flaky, pin the nondeterminism (seeds, ordering, timing, network, uninitialized memory) first - you can't debug what you can't repeat. 2. **Shrink the paylo

2. **Shrink the payload.** Make the repro *fast*: fewer layers, tiny model/data, one process, one CPU/GPU, one node. A 2-second repro beats a 2-minute one - you'll run it hundreds of times. See [methodology](https://github.com/stas00/the-ar

- **Atomic debug cycles.** Make each iteration a single self-contained, repeatable command (setup + run in one shot) so you never hand-redo multi-step state. See [atomic debug cycles](https://github.com/stas00/the-art-of-debugging/blob/mast

- **Automate diagnostics, minimize typing.** Alias the repro and your most-used commands; one keystroke to re-run. See [alias frequently used commands](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#alias-f

- **One-liner programs.** `perl`/`awk`/`python -c` to slice logs, extract fields, transform data on the fly instead of writing throwaway scripts. See [the power of one-liner programs](https://github.com/stas00/the-art-of-debugging/blob/mast

- **Juggle configs cleanly** when running many debug experiments so results don't get confused. See [juggling multiple sets of configs](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#juggling-multiple-sets-

`traceback.print_stack()` shows callers without stopping (useful when the same function is reached via many paths). See [am I editing the right file and the right class?](https://github.com/stas00/the-art-of-debugging/blob/master/methodolog

- **Bisect a regression.** `git bisect start / bad / good <rev>` walks commits automatically to the one that broke things - script the test for `git bisect run`. See [finding a breaking commit by bisecting](https://github.com/stas00/the-art

- **Small/synthetic payload first.** Use tiny or synthetic inputs; switch to real data only when the bug is data-dependent. See [real vs random vs synthetic data](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README

- **Race conditions.** Reordering/timing bugs hide under async; forcing synchronous execution can expose (or mask) them - note which. See [avoiding race conditions](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/READ

- **Cap resources on purpose** to test failure paths: emulate a nearly-full disk, limited CPU RAM, or limited GPU memory. See [running out of resources](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#runnin

- **Watch resources live.** `watch -n1 nvidia-smi` / `free -h` / `df -h` in a second visible terminal to correlate a hang/OOM with what the machine is doing. See [watching and reproducing resource issues](https://github.com/stas00/the-art-o

- **Inject `sleep`** to freeze a program at the interesting moment so you can attach a debugger or snapshot state. See [uses for sleep](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#uses-for-sleep).

- **HPC/SLURM:** keep the allocation and re-run with `srun` instead of re-`sbatch`-ing to cut per-iteration overhead. See [SLURM salloc and srun fast debug combo](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README

Full chapter: [Unix Tools for Debugging](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md).

Combine as `set -euo pipefail`. See [controlling script execution](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#controlling-script-execution).

Classic use: a process at 100% CPU with no output, or hung on I/O/network. See [strace](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#strace).

See [nohup](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#nohup).

- **`make`** - after editing compiled sources, rebuild before re-testing, or you'll debug a stale binary. See [make](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#make).

- **Terminal ergonomics:** search long scrollback and copy multi-line commands cleanly; keep an **informative prompt** (host, path, git branch, last exit code) so you always know where/what ran. See [shell environment](https://github.com/st

Full chapter: [Debugging Compiled Programs](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md). Compile with `-g` for debug symbols.

sudo sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t  # control where cores go

See [segmentation fault, core files and gdb](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#segmentation-fault-core-files-and-gdb).

See [run the program under gdb](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#run-the-program-under-gdb).

sudo gdb --pid=PID    # attach; then: thread apply all bt

See [get the backtrace from the still running process](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#get-the-backtrace-from-the-still-running-process).

See [debugging shared libraries and symbol resolution](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#debugging-shared-libraries-and-symbol-resolution) ([ldd](https://github.com/stas00/the-art-of-debu

Full chapter: [Debugging Python Programs](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md).

- **Print effectively** instead of scattering bare `print`:   - **auto-print name+value** so you never mislabel output. See [auto-print what's being observed](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#auto-

- **auto-print name+value** so you never mislabel output. See [auto-print what's being observed](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#auto-print-whats-being-observed).

- **dump all attributes** of an object to see its real state. See [printing object variables](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#printing-object-variables).

- **trace calls/returns** with `q` (writes to `/tmp/q`, doesn't pollute stdout). See [q](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#q).

See [ensuring the Python package you edit is the one that is run](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#ensuring-the-python-package-you-edit-is-the-one-that-is-run) and [make tests use the git repo's pa

- **Who called this?** `traceback.print_stack()` or the `die` trick to reveal the caller in complex codebases. See [who is calling?](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#who-is-calling).

No sudo? `echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope`. **The first line of each dump is where it's stuck.** See [py-spy](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#py-spy).

No sudo? `echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope`. **The first line of each dump is where it's stuck.** See [py-spy](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#py-spy).

For sub-ms functions, bump `pstats` precision (e.g. `pstats.f8 = lambda x: f"{x:6.3f}"`) so timings aren't all `0.000`. See [profilers](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#profilers) and [cProfile](ht

Full chapter: [Debugging PyTorch Programs](https://github.com/stas00/the-art-of-debugging/blob/master/pytorch/README.md).

Shrink the model, not the problem - make a full run finish in seconds: - **Fewer layers** via (1) a local clone with config edits, (2) editing the config object on the fly, or (3) hacking the modeling code. See [reducing the number of layer

- **Fewer layers** via (1) a local clone with config edits, (2) editing the config object on the fly, or (3) hacking the modeling code. See [reducing the number of layers](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#r

- **Tiny random model + tiny tokenizer + tiny dataset** for near-instant iterations; reproduce at full scale only for scale-only bugs. See [making a tiny model](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#making-a-tin

See [dealing with async CUDA bugs](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#dealing-with-async-cuda-bugs).

- Distinguish OOM in **`forward`** (activations - batch/seq len) vs **`backward`** (gradients/optimizer state). See [debugging CUDA OOM in forward](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#debugging-cuda-oom-in-for

- **Fragmentation** (free memory exists but not contiguous): tune `PYTORCH_ALLOC_CONF` (e.g. `expandable_segments:True`, `max_split_size_mb:...`). See [overcoming CUDA OOM due to memory fragmentation](https://github.com/stas00/the-art-of-de

- **See who allocated what** with the memory profiler / allocation tracing; probe the ceiling with the allocatable-GBs test. See [PyTorch memory profiler](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#pytorch-memory-pro

- **CPU OOM / peak RAM:** see [CPU memory](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#cpu-memory).

Find where bad values first appear; watch fp underflow/overflow (especially fp16/bf16); expect small, benign cross-device numeric differences. Inspect tensors compactly (shape/device/dtype/stats) and use `lovely-tensors` for one-line summar

See [segfaults and getting a backtrace from a core file](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#segfaults-and-getting-a-backtrace-from-a-core-file).

1. **Verify comms first** with a minimal all-reduce test (`torch-distributed-gpu-test.py`); rule out network/NCCL before app code. See [getting nodes to talk to each other](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#

2. **Dump every rank's stack at once** with `py-spy` (recipes for python/deepspeed/accelerate, across nodes via `srun`/`pdsh`). Ranks stuck at *different* lines reveal the desync (a mismatched collective). See [diagnosing crashes, hangs and

3. **Make distributed output legible:** prefix every log line with `node:rank`, and target `pdb` at one rank. See [prefixing logs](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#prefixing-logs-with-noderank-interleaved-a

4. **Narrow further:** check for a [network-level hang](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#network-level-hanging), [isolate a bad GPU](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#isolat

For the cluster-level context around these bugs (verifying node connectivity, NCCL/InfiniBand tuning, network benchmarking, checkpointing/fault tolerance), see [Machine Learning Engineering](https://github.com/stas00/ml-engineering/blob/mas

See [measuring durations](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#measuring-durations).

- **Profile ops** with `torch.profiler` (CPU+GPU, op-level, small overhead); when it's not enough, drop to `cProfile` for pure-Python hot spots. See [performance and profiling](https://github.com/stas00/the-art-of-debugging/tree/master/pyto

- **Read the linked chapter section** before applying an unfamiliar recipe - each has worked examples, caveats, and copy-paste scripts. - Prefer built-in, low-overhead tools (`py-spy`, `strace`, `gdb`, env vars) that need no source changes 

- For large-scale ML training/inference engineering (bottleneck analysis, throughput/memory, distributed hangs at cluster scale, fault tolerance), use the companion skill: [Machine Learning Engineering](https://github.com/stas00/ml-engineer

2. Download the free version of [Prince XML](https://www.princexml.com/download/). It's used to build the pdf version of this book.

1. In-book links use the file form `pytorch/README.md#anchor`, never the bare-directory form `pytorch#anchor`. The compiled HTML/PDF/epub build resolves the file form to a real anchor, but a bare directory has no anchor target in the built 

2. GitHub-web caveat for large files: GitHub's blob (file) view does not serve heading-anchor ids for large Markdown files, so an absolute deep link such as `.../blob/master/pytorch/README.md#anchor` does not scroll to the section on a cold

3. Remedy for that case only: for an *absolute* cross-repo deep link into a large folder-README, use the folder (`tree`) form `https://github.com/OWNER/REPO/tree/master/pytorch#anchor` instead of `.../blob/master/pytorch/README.md#anchor`. 

7. A link can resolve perfectly and still be broken. After moving or rewriting any block that other places point at, check the referrers, not the thing you just edited. Grep for every inbound reference to the thing you moved, and for each o

9. This book's one oversized deep-link target is `pytorch/README.md`, and it *is* a folder README - so the item 3 remedy applies: absolute/cross-repo links into it take the tree form `https://github.com/stas00/the-art-of-debugging/tree/mast

wget --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36" URL

8. A `000` result is bot mitigation, not death. Confirm with the browser user-agent recipe in `Sources and citations` item 9 before calling anything dead. 9. Watch the URL extractor itself. Markdown wraps URLs in ways that corrupt a naive r

9. Watch the URL extractor itself. Markdown wraps URLs in ways that corrupt a naive regex: a closing paren ends the match, and a URL inside backticks picks the backtick up as its tail. 10. Two moves are invisible to an HTTP check, and `buil

10. Two moves are invisible to an HTTP check, and `build/check-redirects.py` reports both. A **`<meta http-equiv="refresh">`** is markup rather than a status, so `curl -L` does not follow it and the stale URL answers `200` forever. Chains a

#!/usr/bin/env python3

# network. But one defect is detectable locally and cheaply, so it is caught on every run: a URL # wrapped into another URL. An archive citation legitimately carries two schemes # (web.archive.org/web/<ts>/https://original), so three or mor

# wrapped into another URL. An archive citation legitimately carries two schemes # (web.archive.org/web/<ts>/https://original), so three or more means a replacement was applied to a # URL that was a substring of a longer one. Concrete near-

# (web.archive.org/web/<ts>/https://original), so three or more means a replacement was applied to a

#!/usr/bin/env bash

for url in "$@"; do     case "$url" in         http://*|https://*) ;;

case "$url" in         http://*|https://*) ;;         *)

http://*|https://*) ;;

if result=$(curl "${curl_args[@]}" "$url" 2>&1); then

printf 'LIVE %s -> %s\n' "$url" "$result"     elif result=$(curl --http1.1 "${curl_args[@]}" "$url" 2>&1); then         printf 'LIVE %s -> %s\n' "$url" "$result"

elif result=$(curl --http1.1 "${curl_args[@]}" "$url" 2>&1); then

#!/usr/bin/env bash

#!/usr/bin/env python3

Two mechanisms are checked. HTTP redirects, which curl follows for us, and <meta refresh>,

# A <meta http-equiv="refresh"> is a redirect that curl does not follow, because it is markup

u = re.sub(r'^(https://github\.com/[^/]+/[^/]+)\.git$', r'\1', u)

try:         r = subprocess.run(['curl', '-sL', '--max-time', '20', '-A', BROWSER_UA,

r = subprocess.run(['curl', '-sL', '--max-time', '20', '-A', BROWSER_UA,

"""curl never sends the #fragment, so `final` always lacks it. Carry it over, or a naive

#!/usr/bin/env python3

#!/usr/bin/env python3

("|[" + "n" * 20 + "](https://example.com/" + "n" * 80 + ")|\n|---|\n|x|", False),

#!/usr/bin/env python3

return (         '<?xml version="1.0" encoding="UTF-8" standalone="no"?>\n'         '<svg xmlns="http://www.w3.org/2000/svg" '

'<?xml version="1.0" encoding="UTF-8" standalone="no"?>\n'         '<svg xmlns="http://www.w3.org/2000/svg" '         'xmlns:xlink="http://www.w3.org/1999/xlink" '

'<svg xmlns="http://www.w3.org/2000/svg" '

'xmlns:xlink="http://www.w3.org/1999/xlink" '

'xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" '

#!/usr/bin/env python

# rtfm https://linkchecker.github.io/linkchecker/man/linkcheckerrc.html

Executable file

my_repo_url = "https://github.com/stas00/the-art-of-debugging/blob/master"

with open(cover_md_file, "w") as f:         f.write(f"""         PDF Conversion was done with https://www.princexml.com

f.write(f"""         PDF Conversion was done with https://www.princexml.com <div class="page_break"></div>

PDF Conversion was done with https://www.princexml.com

This is an ebook version of [The Art of Debugging Open Book by Stas Bekman](https://github.com/stas00/the-art-of-debugging/) generated on {date}.

As this book is constantly being updated, if you downloaded it as a pdf or an epub file and the date isn't recent, chances are that it's already outdated - make sure to check the latest version at [https://github.com/stas00/the-art-of-debug

Executable file

#!/usr/bin/env python3

Executable file

Executable file

# - empty links (i.e. just local anchor to the same doc)     # - skip explicit .md links     # - external links like https://...

# - external links like https://...

# skip:     # - empty links (i.e. just local anchor to the same doc)     # - external links like https://...

# - external links like https://...

# # run to test some of these utils     # para = 'bb [Markdown text](foo.md#tar) aaa bb [Markdown text2](foo/#bar) aaa [Markdown text3](http://ex.com/foo/#bar)'

# para = 'bb [Markdown text](foo.md#tar) aaa bb [Markdown text2](foo/#bar) aaa [Markdown text3](http://ex.com/foo/#bar)'

/*   CSS style sheet for prince html2pdf system (http://www.princexml.com/)

CSS style sheet for prince html2pdf system (http://www.princexml.com/)

prince --no-author-style -s prince_style.css http://en.wikipedia.org/wiki/Winter_war -o foo.pdf

@import url(http://www.princexml.com/fonts/gentium/index.css);

git clone https://github.com/stas00/the-art-of-debugging

sudo sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t

sudo sysctl -w kernel.core_pattern=core-%e.%p.%h.%t

case study: [this issue](https://github.com/pytorch/pytorch/issues/59384#issuecomment-854953165) shows a huge traceback from PyTorch crashing.

sudo gdb --pid=107903

case study: see this [Issue](https://github.com/pytorch/pytorch/issues/60158#issuecomment-865142029) for an example of how this approach was used to diagnose a deadlock in PyTorch. This is also a good example of seeing a backtrace for multi

case study: [this issue](https://github.com/pytorch/pytorch/issues/46807#issuecomment-718452462) shows how gdb was used to get a backtrace on a crashing test.

footnote: more useful recipes for python + gdb can be found [here](https://wiki.python.org/moin/DebuggingWithGdb).

footnote: this chapter assumes you only have the binary. In the less common case where you do have the source and can recompile it, building with the compiler sanitizers - `-fsanitize=address` for memory errors (out-of-bounds, use-after-fre

sudo ldconfig

case study: in other situations a symbol could be missing because the program wasn't linked properly at build time. Here is a bug report [undefined symbol curandCreateGenerator for torch extensions](https://github.com/pytorch/pytorch/issues

Primary reference: [NVIDIA CUDA installation guide for Linux](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/).

sudo apt-get install -y ltrace

<?xml version="1.0" encoding="UTF-8" standalone="no"?> <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" width="1536" height="1988" viewBox="0 0 1

<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" width="1536" height="1988" viewBox="0 0 1536 1988">

If you are into Machine Learning and want to go into a detailed process please read: [Faster debug and development with tiny models, tokenizers and datasets](https://github.com/stas00/ml-engineering/blob/eb81cd46a3b508d7ef6bd3f528134f9b7721

footnote: I was able to get a [PCIe version of A100 working in my desktop](https://stasosphere.com/entrepreneur-being/262-getting-nvidia-a100-80gb-pcie-to-work-on-a-consumer-motherboard-with-custom-water-cooling/). I was hoping that I could

rm -r data

rm -r data; ./launch.sh

This is also why notebook technologies like [Jupyter Notebook](https://jupyter.org/), [IPython](https://en.wikipedia.org/wiki/IPython) and alike, which allow you to go back and forth between different lines of code and re-execute them selec

rm -r data; echo start; ./launch.sh

rm -r data && echo start && ./launch.sh

# reset branch's HEAD to a given commit hash: # find the last commit that was supposed to be the HEAD, e.g.: # https://github.com/fastai/fastai/commit/1c63e868d3d11e73d9f51f58cbd271e67a0fe983

# find the last commit that was supposed to be the HEAD, e.g.: # https://github.com/fastai/fastai/commit/1c63e868d3d11e73d9f51f58cbd271e67a0fe983 # and now reset the branch's HEAD to it

# https://github.com/fastai/fastai/commit/1c63e868d3d11e73d9f51f58cbd271e67a0fe983

# use the PR version of transformers till it's merged git clone https://github.com/huggingface/transformers -b add-model-idefics

git clone https://github.com/huggingface/transformers -b add-model-idefics

# reshard this model - replace only this to do another model git clone https://huggingface.co/HuggingFaceM4/idefics-9b

git clone https://huggingface.co/HuggingFaceM4/idefics-9b

git rm pytorch_model-*.bin

git rm model*safetensors

Here are some of my cheatsheets if you need a starter: [ [python](https://github.com/stas00/python-tools/blob/main/python.txt) |

[ [python](https://github.com/stas00/python-tools/blob/main/python.txt) | [git](https://github.com/stas00/git-tools/blob/master/git.txt) |

[python](https://github.com/stas00/python-tools/blob/main/python.txt) |

[git](https://github.com/stas00/git-tools/blob/master/git.txt) |

[bash](https://github.com/stas00/bash-tools/blob/main/bash.txt) |

[make](https://github.com/stas00/make-tools/blob/main/make.txt) |

[conda](https://github.com/stas00/conda-tools/blob/master/conda.txt)

Sometimes it's even possible to completely automate the reporting w/o requiring any typing at all. When I try to debug memory leaks I use tools that report memory usage or deltas automatically. e.g. I developed [ipyexperiments](https://gith

gdb supports the same idea via `~/.gdbinit`, where you can add your own [user-defined commands](https://sourceware.org/gdb/current/onlinedocs/gdb.html/Define.html) and `alias`es. And for ready-made productivity boosters there are full enhan

Learn your shell's line-editing and history shortcuts - they save a lot of debug time. These are covered in the Unix chapter under [Handy shell shortcuts](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#handy-shell

footnote: stepping aside from the discussion of debugging I use one-liners for refactoring/renaming - since I can then share these with others who are impacted by my changes and they can just copy-n-paste my commands and immediately update 

As you warm up to the joy and profit of using one-liners make sure to empower yourself with [Handy shell shortcuts](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#handy-shell-shortcuts) to navigate the command lin

python -c "import torch; exec('with torch.cuda.device(0):\n  x = torch.ones(1,1)')"

sudo mount -t tmpfs -o size=1G,user,mode=1777 tmpfs ~/ramdisk

Just beware that `tmpfs` uses volatile virtual memory. When you unmount this partition or reboot your system all files created in this partition will disappear. You can read about tmpfs [here](https://www.kernel.org/doc/html/latest/filesyst

sudo umount ~/ramdisk

You can read [A Deep Investigation into MMAP Not Leaking Memory](https://stasosphere.com/entrepreneur-being/301-mmap-memory-leak-investigation/) to see how I used this technique to figure out whether MMAP leaks memory or not.

For the detailed manpage see [systemd-run](https://www.freedesktop.org/software/systemd/man/systemd-run.html) and for additional properties that can be set [this section](https://www.freedesktop.org/software/systemd/man/systemd.resource-con

If you have a recent NVIDIA GPU (A100, H100, and higher) you could also reshape your GPU using Multi-Instance GPU (MIG) as explained [here](https://www.nvidia.com/en-us/technologies/multi-instance-gpu/). So if you're trying to ensure your s

git clone https://github.com/huggingface/transformers

So for example if you know that `transformers==4.33.0` was good and `transformers@main` (`HEAD`) is bad, find which commit is corresponding to the tag `4.33.0` by visiting [the releases page](https://github.com/huggingface/transformers/rele

You can then go to `https://github.com/huggingface/transformers/commit/` and append the commit SHA to that url which will take you to the commit, (e.g. `https://github.com/huggingface/transformers/commit/57f44dc4288a3521bd700405ad41e90a4687

This package, that can be installed from https://github.com/zestyping/q or `pip install q`, is designed for a quick code and function tracing. By default it sends its output into the `/tmp/q` file, so in one console start watching this file

You can control what attributes are dumped via various args to [`rich.inspect()`](https://rich.readthedocs.io/en/stable/reference/init.html#rich.inspect).

The closest Python built-in way I found is `inspect.getmembers()`, but alas it won't let me filter the variable attributes (but it does let you filter by other types of attributes via a [`predicate` argument](https://docs.python.org/3.13/li

git clone https://github.com/huggingface/transformers/

git clone https://github.com/huggingface/transformers/

we used `src` because that's where source files of HF `transformers` package reside, that is they are under `[src/](https://github.com/huggingface/transformers/tree/main/src/). Other git repos could use a different path where they place the

There is an easy solution to that. You can see how I did it in [ipyexperiments](https://github.com/stas00/ipyexperiments). I created [tests/conftest.py](https://github.com/stas00/ipyexperiments/blob/39c9b454e89e53b74c74dcb579c12ecf3d2161b9/

If your project uses a sub-directory like HF `transformers`' `src`, simply add it to the `git_repo_path`, e.g. you can see how this is done [here](https://github.com/huggingface/transformers/blob/6cbc1369a330860c128a1ba365f246751382c9e5/con

The sections above are terminal-driven - you type commands at a prompt. The other major family is the graphical debugger built into an IDE, where the breakpoints, the call stack, the source and the variable inspector are all on the screen a

footnote: [Cursor](https://cursor.com/download) is a fork of VSCode, so everything that applies to VSCode in this section applies to Cursor alike.

case study: In 2020 I ported my first model architecture to HF Transformers and I was using 2 PyCharm instances to step through the original model code in fairseq and the new model code in Transformers - I was matching the tensor values on 

[Credential-related source omitted]

cProfile is a built-in profiler that is quite easy to use. The full documentation is [here](https://docs.python.org/3/library/profile.html).

The only argument to `pr.print_stats` is how you want the output to be sorted, the example above uses `SortKey.CUMULATIVE` - cumulative time, `SortKey` is an `enum` variable, with other commonly useful values are `SortKey.TIME` (total time)

The argument to `stats.sort_stats` could be a string name or the `SortKey` enum like in the previous example. The complete table of options is [here](https://docs.python.org/3/library/profile.html#pstats.Stats.sort_stats).

Finally, `py-spy record -f speedscope` writes the profile for [speedscope](https://www.speedscope.app/) instead of an SVG, which adds a time-ordered view of the samples that a flame graph deliberately discards. Here is what it looks like (w

Executable file

Executable file

It was written against https://github.com/python/cpython/blob/3.8/Lib/trace.py. If you're

[Credential-related source omitted]

# string with commands to run - passed to exec()

Moreover, large models often require massive resources, which are typically expensive and can also can make a debugging process super complicated. For example any debugger can handle a single process, but if your model doesn't fit and requi

Important: given their popularity and the well designed simple API I will be discussing HF [`transformers`](https://github.com/huggingface/transformers/) models. But the same principle can be applied to any other model.

Now let's go through the actual code and convert ["google/mt5-small"](https://huggingface.co/google/mt5-small/tree/main) into its tiny random counterpart.

Also please note that you don't need any GPUs to do that and you could do this even on a huge 176B parameter model like [BLOOM-176B](https://huggingface.co/bigscience/bloom). Since you never load the actual original model, except its config

The original ["google/mt5-small"](https://huggingface.co/google/mt5-small/tree/main) model file was 1.2GB. With the above changes (and vocab shrinking as explained in the following sections) we got it down to 126MB.

If you're dealing with a multi-level nested config, you will have to update each sub-level's config object separately. For example in [IDEFICS](https://huggingface.co/HuggingFaceM4/idefics-9b/blob/main/config.json) we have 1 main and 2 nest

A related trap: architectures that interleave block types spell the pattern out in a `layer_types` list with one entry per layer. [Qwen/Qwen-AgentWorld-35B-A3B](https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B/blob/main/config.json#L19) 

[Anthony Moi](https://github.com/n1t0)'s tokenizer shrinker:

[Lysandre Debut](https://github.com/LysandreJik)' shrinker using `train_new_from_iterator`:

Here is the full version of [make_tiny_model.py](https://huggingface.co/stas/tiny-random-llama-2/blob/main/make_tiny_model.py) which includes both the model and the tokenizer shrinking.

git clone https://github.com/google/sentencepiece

[Credential-related source omitted]

tokenizer_orig.save_pretrained(tmp_dir) with open(vocab_orig_path, 'rb') as f: data = f.read() # adapted from https://blog.ceshine.net/post/trim-down-sentencepiece-vocabulary/

with open(vocab_orig_path, 'rb') as f: data = f.read() # adapted from https://blog.ceshine.net/post/trim-down-sentencepiece-vocabulary/ m = model.ModelProto()

# adapted from https://blog.ceshine.net/post/trim-down-sentencepiece-vocabulary/

If we continue the MT5 code from earlier in this chapter and add the tokenizer shrinking code from the previous section, we end up with this script [mt5-make-tiny-model.py](https://huggingface.co/stas/mt5-tiny-random/blob/main/mt5-make-tiny

Here is another example  [fsmt-make-super-tiny-model.py](https://huggingface.co/stas/tiny-wmt19-en-ru/blob/main/fsmt-make-super-tiny-model.py) - here you can see I'm creating a totally new tiny vocab from scratch.

Also be aware that since HF `transformers` needs tiny models for their testing, you are very likely to already find one for each architecture available mostly from https://huggingface.co/hf-internal-testing (except they didn't include the c

footnote: as of [`datasets` 4.0](https://github.com/huggingface/datasets/releases/tag/4.0.0), Hub-hosted Python loading scripts are no longer executed by `load_dataset`. Keep any builder/unpacker `.py` in the repo as documentation of how th

- [stas/oscar-en-10k](https://huggingface.co/datasets/stas/oscar-en-10k/blob/main/oscar-en-10k.py)

- [stas/c4-en-10k](https://huggingface.co/datasets/stas/c4-en-10k/blob/main/c4-en-10k.py)

- [stas/openwebtext-10k](https://huggingface.co/datasets/stas/openwebtext-10k/blob/main/openwebtext-10k.py)

And here are some examples of synthetic datasets, where instead of just shrinking the original tarball, I untar'ed it, manually chose the representative examples and then wrote a script to build any size of desired dataset based on those fe

- [stas/general-pmd-synthetic-testing](https://huggingface.co/datasets/stas/general-pmd-synthetic-testing/blob/main/general-pmd-synthetic-testing.py) and the [unpacker](https://huggingface.co/datasets/stas/general-pmd-synthetic-testing/blob

- [stas/cm4-synthetic-testing](https://huggingface.co/datasets/stas/cm4-synthetic-testing/blob/main/cm4-synthetic-testing.py) - and the [unpacker](https://huggingface.co/datasets/stas/cm4-synthetic-testing/blob/main/m4-ds-unpack.py)

And here is for example the instructions of deploying these scripts for [stas/general-pmd-synthetic-testing](https://huggingface.co/datasets/stas/general-pmd-synthetic-testing/):

# prep dataset repo https://huggingface.co/new-dataset => stas/general-pmd-synthetic-testing

https://huggingface.co/new-dataset => stas/general-pmd-synthetic-testing

git clone https://huggingface.co/datasets/stas/general-pmd-synthetic-testing

rm -rf data

rm image.jpg text.txt

Similar to tiny models, you will find many tiny datasets under https://huggingface.co/hf-internal-testing.

perl -lne 'while (/(https.*?.py)\)/g) { $x=$1; $x=~s/blob/raw/; print qq[wget $x] }' make-tiny-models.md

Therefore in this section we will discuss how to reduce the model's number of hidden layers from many to just 1-2. If the layers aren't identical (e.g. some MoE models alternate between 2 different block configurations) then ensure you incl

The config entry that we want to change is [`num_hidden_layers`](https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507/blob/e67ac5d/config.json#L24). Some architectures pair it with a companion entry that has to be trimmed in lockstep - a

git clone https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507

After finding the desired model on https://huggingface.co/, clone its git repo to the local disk, modify the `num_hidden_layers` entry in `config.json`, and then load the model from the local clone (same as we have just shown when measuring

git clone https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507

git clone https://github.com/huggingface/transformers/tree/main/src/transformers

Continuing the example of working with `Qwen/Qwen3-30B-A3B-Instruct-2507` model, we find the place where its architecture modeling code lives in the HF Transformers code base. For example, we can look at the `architectures` field in the [mo

When I was debugging 15M sequence length training using [ALST](https://arxiv.org/abs/2506.13996) - I would only run self-attention in the last layer and then skip it in the previous layers - this reduced my testing time from hours to minute

Luckily, PyTorch has a special [autograd mechanism that is used to detect anomalies](https://docs.pytorch.org/docs/stable/autograd.html#debugging-and-anomaly-detection). When enabled it'll augment the `backward` traceback with the `forward`

When developing [Arctic Long Sequence Training](https://arxiv.org/abs/2506.13996) a lot of the work involved hard (`contiguous`) reshaping of tensors for Sequence Parallelism, which lead to a very poor HBM utilization, because the reshapes 

Indeed, H200 is about 140GiB, so this checks out. You might wonder what's the point of this. This discovery code is useful when you have to deal with multiple systems accessing the same GPU - e.g. VLLM and VeRL sharing the same GPUs and PyT

The solution: to prevent CPU from stealing GPU memory switch to the CDMM mode (from the default NUMA mode), as explained in a paper linked from [this post](https://developer.nvidia.com/blog/understanding-memory-management-on-hardware-cohere

$ echo options nvidia NVreg_CoherentGPUMemoryMode=driver | sudo tee /etc/modprobe.d/nvidia-openrm.conf

At this stage you'd typically go to https://docs.pytorch.org/memory_viz and drop the memory profile pickle file you generated in Stage 1 into the browser at that URL. The most useful feature there is the 'Active memory profile` drop-down me

To get a feeling for what it looks like, here is an example of a memory profile rendering for a memory leak I discovered while I was working on a tricky implementation of a [`TiledMLP` `torch.autograd.Function`](https://github.com/deepspeed

As long as you're in the `forward` function it's relatively easy to find where in the code leaks comes from. But if it's a `backward` it becomes much more complicated unless you're debugging a custom autograd function. Still it should give 

Besides profiling memory leaks, this functionality is also useful for showing how different implementations of the same algorithm use a different amount of gpu memory. For example, the following visualization I prepared for the [Arctic Long

Additional resources: - [Understanding GPU Memory 1: Visualizing All Allocations over Time](https://pytorch.org/blog/understanding-gpu-memory-1/)

- [Understanding GPU Memory 1: Visualizing All Allocations over Time](https://pytorch.org/blog/understanding-gpu-memory-1/)

- HF folks made an [improved rendering version](https://huggingface.co/spaces/Leiyre/memory-viz).

While external memory profilers can be very useful, often having control over when you take a sample of GPU and CPU memory usage is needed. `see-mem-usage` debug util has been developed by the [DeepSpeed project](https://github.com/deepspee

For example, here is how I found a memory leak in `all_gather_object`, which you can see from this [Issue](https://github.com/pytorch/pytorch/issues/150798). And there were several other similar leaks in PyTorch I discovered using this tool

which is very difficult to notice. This is typically performed by an `oom-kill` via [cgroups](https://docs.kernel.org/admin-guide/cgroup-v2.html). The `SIGKILL` is not trappable, so the program itself can't report anything - but the kernel 

footnote: Moreover in some situations, as in recent kubernetes implementations, the user gets kicked out from the job allocation, which makes it even more difficult to debug. [Kubernetes Silent Pod Killer](https://itnext.io/kubernetes-silen

[`cgmemtime`](https://github.com/gsauthof/cgmemtime) is a little gem of a C program that uses cgroups v2 to measure the peak CPU memory usage of a process and all of its descendants no matter how many generations follow it.

git clone https://github.com/gsauthof/cgmemtime

Until recently Python couldn't tell us the name of the variable that was passed to a function, hence we need to pass the variable name as a string. Since Python 3.14 there is a way to overcome this, using [template strings (t-strings)](http

There is also the [lovely-tensors](https://github.com/xl0/lovely-tensors/) library that is designed to automate the process of making the tensor dump more useful and provide shapes, statistics, nan/inf counts, tensor memory usage, etc.

See also [Numerical instabilities](https://github.com/stas00/ml-engineering/blob/master/training/instabilities/README.md#numerical-instabilities) in the training chapter, which covers training-level causes and remedies for `inf`/`nan` value

Let's use fp16 floating point representation for demonstrating how we end up with Infinity numbers. 65504 is the largest normal floating point number that can be represented in the fp16 precision. This is slightly below `2**16` due to how t

The first tensor is fine, but the last one overflows when I added `50` to it and we get `inf`. If you remember back in the day, models were trained in fp16 mixed precision regime and this `inf` happened a lot, thus a special scaler was used

As of `transformers==4.57.1` the above will give you `loss=tensor(nan)`. The issue here is that the special `-100` label masks tokens to be excluded from the loss calculation and in the above example, we have 0 tokens that aren't masked, si

You can see it in context [here](https://github.com/deepspeedai/DeepSpeed/blob/df59f203f40c8a292dd019ae68c9e6c88f107026/deepspeed/runtime/sequence_parallel/ulysses_sp.py#L1184-L1186). Though the code has evolved since then, and you can find

Let's match the report to the code from [`models/t5/modeling_t5.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/modeling_t5.py):

See also [Reproducibility](https://github.com/stas00/ml-engineering/blob/master/training/reproducibility/README.md#achieve-determinism-in-randomness-based-software) for achieving determinism across different software and hardware setups.

For example, see this [discussion](https://github.com/deepspeedai/DeepSpeed/issues/4932) - the users reported that when doing Llama-2-7b inference they were getting quite different logits depending on how the model was initialized. To clari

This snapshot and the commentary come from this [PyTorch Issue thread](https://github.com/pytorch/pytorch/issues/84936#issuecomment-1246084645).

If you're curious where I pulled this code from - this is a simplified reduction of this original code in [modeling_llama.py](https://github.com/huggingface/transformers/blob/3f69f415adcbdaedec154ba8eac220ef3276975d/src/transformers/models/

If you suspect a crash is caused by faulty hardware rather than your code, see also [Troubleshooting NVIDIA GPUs](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/nvidia/debug.md#errors-and-diagnostics).

In theory enabling this variable should make everything run really slow, but in reality it really depends on your software. We did the whole of BLOOM-176B training using `CUDA_LAUNCH_BLOCKING=1` with [`Megatron-DeepSpeed`](https://github.co

Note: [NCCL==2.14.3 coming with `pytorch==1.13` hangs](https://github.com/NVIDIA/nccl/issues/750) when `CUDA_LAUNCH_BLOCKING=1` is used. So don't use it with that version of pytorch. The issue has been fixed in `nccl>=2.17` which should be 

Primary reference: [NVIDIA Compute Sanitizer](https://docs.nvidia.com/compute-sanitizer/).

Reference: [NVIDIA cuda-gdb](https://docs.nvidia.com/cuda/cuda-gdb/).

- `PID` is the process id of the hanging python process. - `-n` is useful if you want to see stack traces from python extensions written in C, C++, etc., as the program may hang in one of the extensions - you may need to add `sudo` before t

- you may need to add `sudo` before the command - for more details see [this note](https://github.com/benfred/py-spy/blob/master/README.md#when-do-you-need-to-run-as-sudo).

sudo echo 0 > /proc/sys/kernel/yama/ptrace_scope

which will allow you running `py-spy` (and `strace`) without needing `sudo`. Beware of the possible [security implications](https://wiki.ubuntu.com/SecurityTeam/Roadmap/KernelHardening#ptrace_Protection) - but typically if your compute node

sudo sysctl -w kernel.yama.ptrace_scope=0

sudo bash -c "echo 0 > /proc/sys/kernel/yama/ptrace_scope"

PDSH_RCMD_TYPE=ssh pdsh -w nodename-[5,8] 'sudo bash -c "source ~/.pdshrc; pgrep -P $(pgrep -o python) | xargs -I {} py-spy dump --pid {}"'

[Credential-related source omitted]

[Credential-related source omitted]

ds_ssh -f hostfile "source ~/.pdshrc; ps aux | grep python | grep -v grep | grep `whoami` | awk '{print \$2}' | xargs -I {} sudo py-spy dump --pid {} "

Primary reference: [ProcessGroupNCCL environment variables](https://docs.pytorch.org/docs/stable/torch_nccl_environment_variables.html).

[Credential-related source omitted]

sudo sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t

See [Troubleshooting AMD GPUs](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/amd/debug.md#hangs-or-slow-multi-gpu-runs-and-iommu) in ML Engineering.

For diagnosing NCCL connectivity problems between GPUs and nodes (the layer below PyTorch), see also [How to diagnose NCCL multi-gpu and multi-node connectivity issues](https://github.com/stas00/ml-engineering/blob/master/network/debug/READ

Once you need to use more than one node to scale your training, e.g., if you want to use DDP to train faster, you have to get the nodes to talk to each other, so that communication collectives could send data to each other. This is typicall

telnet localhost:6000 nmap -p 6000 localhost nc -zv localhost 6000

nmap -p 6000 localhost nc -zv localhost 6000 curl -v telnet://localhost:6000

nc -zv localhost 6000

curl -v telnet://localhost:6000

$ sudo ifconfig

Then there is the inter-node interface which can be InfiniBand, EFA, OPA, HPE Slingshot, etc. ([more information](https://github.com/stas00/ml-engineering/blob/master/network/README.md#inter-node-networking)). There could be one or dozens o

$ sudo ifconfig

$ sudo ifconfig

Here `ib` typically tells us it's an InfiniBand card, but really it can be any other vendor. I have seen [OmniPath](https://github.com/stas00/ml-engineering/blob/master/network/README.md#omni-path) using `ib` for example. Again `inet` tells

Another popular way is to use [Message Passing Interface (MPI)](https://en.wikipedia.org/wiki/Message_Passing_Interface). There are a few open source implementations of it available.

There could be a variety of reason for this failing, but of the most likely one is when you're on the cloud and the 2 nodes weren't provisioned so that their IB is connected. So your Ethernet inter-node connectivity works, but it's too slow

"""     PDB Subclass for debugging multi-processed code     Suggested in: https://stackoverflow.com/questions/4716533/how-to-attach-debugger-to-a-python-subproccess

PDB Subclass for debugging multi-processed code     Suggested in: https://stackoverflow.com/questions/4716533/how-to-attach-debugger-to-a-python-subproccess     """

Suggested in: https://stackoverflow.com/questions/4716533/how-to-attach-debugger-to-a-python-subproccess

`dist.breakpoint()` and `ForkedPdb` above both require you to plant the breakpoint in the source before the run. Python 3.14 adds the missing case: attaching to a process that is *already running* and that you did not prepare in advance, gi

Primary reference: [Python 3.14 pdb](https://docs.python.org/3.14/library/pdb.html).

[hyperfine](https://github.com/sharkdp/hyperfine) runs a given command multiple times and reports how long it took to run: the mean, the standard deviation, and the fastest and slowest run. It can also compare two or more commands and repor

sudo apt-get install -y hyperfine

$ hyperfine -p 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' 'python -c "import torch"'

What an import actually reads, and how to build a file system benchmark out of timings like these, is covered in [Usability perception IO benchmarks](https://github.com/stas00/ml-engineering/blob/master/storage/README.md#usability-perceptio

- `ht.hpu.Event(enable_timing)` - Intel Gaudi - `torch.xpu.Event(enable_timing)` - Intel dGPUs - Apple MPS has a different API, but we can emulate the same API as can seen in [mamf-finder.py](https://github.com/stas00/ml-engineering/blob/00

- Apple MPS has a different API, but we can emulate the same API as can seen in [mamf-finder.py](https://github.com/stas00/ml-engineering/blob/0099885db36a8f06556efe1faf552518852cb1e0/compute/accelerator/benchmarks/mamf-finder.py#L159).

Here are good practical examples of measuring time durations with the help of device events: - [mamf-finder](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/benchmarks/mamf-finder.py)

- [mamf-finder](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/benchmarks/mamf-finder.py)

- [torch-dist-bench](https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/torch-dist-bench.py)

Here are some excellent articles going into deeper explanations and examples: - [How to Accurately Time CUDA Kernels in Pytorch](https://www.speechmatics.com/company/articles-and-news/timing-operations-in-pytorch)

- [How to Accurately Time CUDA Kernels in Pytorch](https://www.speechmatics.com/company/articles-and-news/timing-operations-in-pytorch)

For more options please refer to the [`torch.profiler` doc](https://docs.pytorch.org/docs/stable/profiler.html).

Additionally, here is [an excellent introduction to `torch.profiler` from the HuggingFace team](https://huggingface.co/blog/torch-profiler).

I assume you already have the correct version of torch installed, if not head [here](https://pytorch.org/get-started/locally/).

Now let's use it to profile 2 forward calls using [Liger Kernel](https://github.com/linkedin/Liger-Kernel):

[Credential-related source omitted]

[Credential-related source omitted]

footnote: since the culprit here was `torch.compile`, you could also have spotted the recompilation directly with `TORCH_LOGS=recompiles` (and `TORCH_LOGS=graph_breaks` for eager fallbacks); see [PyTorch compiler troubleshooting](https://do

#!/usr/bin/env python

#!/usr/bin/env python

#!/usr/bin/env python

#!/usr/bin/env python

#!/usr/bin/env python

#!/usr/bin/env python

Executable file

Executable file

Executable file

# You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0

# #     http://www.apache.org/licenses/LICENSE-2.0 #

#     http://www.apache.org/licenses/LICENSE-2.0

The full 1TB+ dataset is at https://huggingface.co/datasets/c4.

_URL = "https://cdn-datasets.huggingface.co/nlp/datasets/c4/c4-en-10k.tar.xz"

description=_DESCRIPTION,             features=datasets.Features({"text": datasets.Value("string")}),             homepage="https://huggingface.co/datasets/allenai/c4/",

features=datasets.Features({"text": datasets.Value("string")}),             homepage="https://huggingface.co/datasets/allenai/c4/",             citation=_CITATION,

homepage="https://huggingface.co/datasets/allenai/c4/",

Executable file

rm -rf data

# rm texts*

# prep dataset repo https://huggingface.co/new-dataset => HuggingFaceM4/cm4-synthetic-testing

https://huggingface.co/new-dataset => HuggingFaceM4/cm4-synthetic-testing

git clone https://huggingface.co/datasets/HuggingFaceM4/cm4-synthetic-testing

_DESCRIPTION = """This dataset is designed to be used in testing. It's derived from cm4-10k dataset""" _HOMEPAGE = "https://huggingface.co/datasets/HuggingFaceM4/cm4-synthetic-testing"

_HOMEPAGE = "https://huggingface.co/datasets/HuggingFaceM4/cm4-synthetic-testing"

_URL = "https://huggingface.co/datasets/HuggingFaceM4/cm4-synthetic-testing/resolve/main/data.tar.gz"

#!/usr/bin/env python

# You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0

# #     http://www.apache.org/licenses/LICENSE-2.0 #

#     http://www.apache.org/licenses/LICENSE-2.0

Executable file

list2range('1,2,5-7,10')     [1, 2, 5, 6, 7, 10]     # from https://stackoverflow.com/a/6405711/9201239

[1, 2, 5, 6, 7, 10]     # from https://stackoverflow.com/a/6405711/9201239     """

# from https://stackoverflow.com/a/6405711/9201239

Executable file

# prep dataset repo https://huggingface.co/new-dataset => HuggingFaceM4/general-pmd-synthetic-testing

https://huggingface.co/new-dataset => HuggingFaceM4/general-pmd-synthetic-testing

git clone https://huggingface.co/datasets/HuggingFaceM4/general-pmd-synthetic-testing

rm -rf data

rm image.jpg text.txt

rm -rf data

_DESCRIPTION = """This dataset is designed to be used in testing. It's derived from general-pmd-10k dataset""" _HOMEPAGE = "https://huggingface.co/datasets/HuggingFaceM4/general-pmd-synthetic-testing"

_HOMEPAGE = "https://huggingface.co/datasets/HuggingFaceM4/general-pmd-synthetic-testing"

_URL = "https://huggingface.co/datasets/HuggingFaceM4/general-pmd-synthetic-testing/resolve/main/data.tar.gz"

#!/usr/bin/env python

Executable file

list2range('1,2,5-7,10')     [1, 2, 5, 6, 7, 10]     # from https://stackoverflow.com/a/6405711/9201239

[1, 2, 5, 6, 7, 10]     # from https://stackoverflow.com/a/6405711/9201239     """

# from https://stackoverflow.com/a/6405711/9201239

#!/usr/bin/env python

# You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0

# #     http://www.apache.org/licenses/LICENSE-2.0 #

#     http://www.apache.org/licenses/LICENSE-2.0

# To build: # 1. clone sentencepiece into this dir # git clone https://github.com/google/sentencepiece

# 1. clone sentencepiece into this dir # git clone https://github.com/google/sentencepiece #

# git clone https://github.com/google/sentencepiece

# adapted from https://blog.ceshine.net/post/trim-down-sentencepiece-vocabulary/

Executable file

# You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0

# #     http://www.apache.org/licenses/LICENSE-2.0 #

#     http://www.apache.org/licenses/LICENSE-2.0

title={OpenWebText Corpus},   author={Aaron Gokaslan*, Vanya Cohen*, Ellie Pavlick, Stefanie Tellex},   howpublished{\\url{http://Skylion007.github.io/OpenWebTextCorpus}},

author={Aaron Gokaslan*, Vanya Cohen*, Ellie Pavlick, Stefanie Tellex},   howpublished{\\url{http://Skylion007.github.io/OpenWebTextCorpus}},   year={2019}

howpublished{\\url{http://Skylion007.github.io/OpenWebTextCorpus}},

The full 8M-record dataset is at https://huggingface.co/datasets/openwebtext

_URL = "https://cdn-datasets.huggingface.co/nlp/datasets/openwebtext/openwebtext-10k.tar.xz"

description=_DESCRIPTION,             features=datasets.Features({"text": datasets.Value("string")}),             homepage="https://skylion007.github.io/OpenWebTextCorpus/",

features=datasets.Features({"text": datasets.Value("string")}),             homepage="https://skylion007.github.io/OpenWebTextCorpus/",             citation=_CITATION,

homepage="https://skylion007.github.io/OpenWebTextCorpus/",

Executable file

# You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0

# #     http://www.apache.org/licenses/LICENSE-2.0 #

#     http://www.apache.org/licenses/LICENSE-2.0

address   = {Mannheim},   doi       = {10.14618/ids-pub-9021},   url       = {http://nbn-resolving.de/urn:nbn:de:bsz:mh39-90215},

doi       = {10.14618/ids-pub-9021},   url       = {http://nbn-resolving.de/urn:nbn:de:bsz:mh39-90215},   pages     = {9 -- 16},

url       = {http://nbn-resolving.de/urn:nbn:de:bsz:mh39-90215},

The full 1TB+ dataset is at https://huggingface.co/datasets/oscar.

_URL = "https://cdn-datasets.huggingface.co/nlp/datasets/oscar/oscar-en-10k.tar.xz"

description=_DESCRIPTION,             features=datasets.Features({"text": datasets.Value("string")}),             homepage="https://oscar-corpus.com/",

features=datasets.Features({"text": datasets.Value("string")}),             homepage="https://oscar-corpus.com/",             citation=_CITATION,

homepage="https://oscar-corpus.com/",

#!/usr/bin/env python

[Credential-related source omitted]

Executable file

# You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0

# #     http://www.apache.org/licenses/LICENSE-2.0 #

#     http://www.apache.org/licenses/LICENSE-2.0

If you use [`tmux`](https://github.com/tmux/tmux) `Ctrl-b->[` will enter scrollback mode, and vi/emacs (depending on your settings) keys can be used for search/scroll/etc.

Anything that has to do with the current status is very helpful. If you work with git a lot and have to switch back and forth between different branches and forks. A tool like [bash-git-prompt](https://github.com/magicmonty/bash-git-prompt)

cd ~ git clone https://github.com/magicmonty/bash-git-prompt ~/.bash-git-prompt --depth=1

git clone https://github.com/magicmonty/bash-git-prompt ~/.bash-git-prompt --depth=1 cd ~/.bash-git-prompt/themes wget https://raw.githubusercontent.com/stas00/the-art-of-debugging/master/unix/bash-git-prompt/Stas.bgptheme

wget https://raw.githubusercontent.com/stas00/the-art-of-debugging/master/unix/bash-git-prompt/Stas.bgptheme

The possible exit codes are listed [here](https://tldp.org/LDP/abs/html/exitcodes.html). and `127` corresponds to "command not found" error - which is indeed the case here. Though most of the time the exit code will be `1`, which is a catch

$ sudo dmesg -T | grep -i "killed process"

sudo dmesg -l err,crit   # only errors and worse

sudo dmesg -w            # follow new messages, like tail -f

`oom_group_kill` is a different axis: it counts the times the kernel took the whole cgroup down together instead of picking one victim, which it only does when `memory.oom.group` is set to 1. Kubernetes sets that by default, which is how on

$ sudo dmesg -T | grep -iE "oom-kill|out of memory" | tail -2

$ sudo mkdir /sys/fs/cgroup/oomtest

$ echo 50M | sudo tee /sys/fs/cgroup/oomtest/memory.max

$ echo 0 | sudo tee /sys/fs/cgroup/oomtest/memory.swap.max

$ sudo bash -c 'echo $$ > /sys/fs/cgroup/oomtest/cgroup.procs; exec python3 -c "bytearray(300*2**20)"'

$ sudo rmdir /sys/fs/cgroup/oomtest

If you're chasing a GPU fault, the NVIDIA driver logs `Xid` errors here too - see [Xid Errors](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/nvidia/debug.md#xid-errors) in the companion book for how to read them.

In order that we have something to work with, let's take the `Makefile` from [ipyexperiments](https://github.com/stas00/ipyexperiments):

git clone https://github.com/stas00/ipyexperiments

This is probably the least interesting section of this guide, but it is what it is. For more information please see [GNU make](https://www.gnu.org/software/make/manual/make.html).

# symbols: https://unicodelookup.com/#circle/1