stas00/ml-engineering

[Credential-related source omitted]

Executable file

find . -name "*html" -exec rm {} \;

This repo is an ongoing brain dump of my experiences training Large Language Models (LLM) (and VLMs); a lot of the know-how I acquired while training the open-source [BLOOM-176B](https://huggingface.co/bigscience/bloom) model in 2022 and [I

1. **[And more debugging](https://github.com/stas00/the-art-of-debugging)**

I announce any significant updates on my twitter channel [https://twitter.com/StasBekman](https://twitter.com/StasBekman).

You can download various ebook formats of this book: * [PDF](https://huggingface.co/stas/ml-engineering-book/resolve/main/Stas%20Bekman%20-%20Machine%20Learning%20Engineering.pdf?download=true)

* [PDF](https://huggingface.co/stas/ml-engineering-book/resolve/main/Stas%20Bekman%20-%20Machine%20Learning%20Engineering.pdf?download=true)

* [EPUB](https://huggingface.co/stas/ml-engineering-book/resolve/main/Stas%20Bekman%20-%20Machine%20Learning%20Engineering.epub?download=true)

Thanks to HuggingFace for giving me permission to host my book's ebook formats at the [HF hub](https://huggingface.co/stas/ml-engineering-book).

See also the companion skills: [The Art of Debugging](https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md) and [Stas' Python Cookbook](https://github.com/stas00/python-cookbook/blob/master/SKILL.md).

- [Building resilient ML Engineering skills](https://www.youtube.com/watch?v=IBJUt9JPKHk) given on 2026-01-10 for the [GPU Mode community](https://github.com/gpu-mode). Only had time to discuss performance reality of accelerators, network a

If you want to discuss something related to ML engineering this repo has the [community discussions](https://github.com/stas00/ml-engineering/discussions) available - so please don't hesitate to share your experience or start a new discussi

Special thanks go to [Thom Wolf](https://github.com/thomwolf) who proposed that I lead the BLOOM-176B training back when I didn't know anything about large scale training. This was the project that catapulted me into the intense learning pr

Recently, I continued expanding my knowledge and experience while training models and building scalable training/inference systems at [Contextual.AI](https://contextual.ai/) and I'm grateful for that opportunity to Aman and Douwe.

If you found a bug, typo or would like to propose an improvement please don't hesitate to open an [Issue](https://github.com/stas00/ml-engineering/issues) or contribute a PR.

- [The Art of Debugging Open Book](https://github.com/stas00/the-art-of-debugging) — methodologies and recipes for debugging Unix, Python and PyTorch programs.

- [Stas' Python Cookbook](https://github.com/stas00/python-cookbook) — everyday Python and standard-library recipes.

publisher = {Stasosphere Online Inc.},   journal = {GitHub repository},   url = {https://github.com/stas00/ml-engineering}

journal = {GitHub repository},   url = {https://github.com/stas00/ml-engineering} }

url = {https://github.com/stas00/ml-engineering}

✔ **Books:**  [Machine Learning Engineering](https://github.com/stas00/ml-engineering) |

[Machine Learning Engineering](https://github.com/stas00/ml-engineering) |

[The Art of Debugging](https://github.com/stas00/the-art-of-debugging) |

[Stas' Python Cookbook](https://github.com/stas00/python-cookbook)

✔ **Applications:**  [ipyexperiments](https://github.com/stas00/ipyexperiments)

[ipyexperiments](https://github.com/stas00/ipyexperiments)

✔ **Tools and Cheatsheets:**  [bash](https://github.com/stas00/bash-tools) |

[bash](https://github.com/stas00/bash-tools) |

[conda](https://github.com/stas00/conda-tools) |

[git](https://github.com/stas00/git-tools) |

[jupyter-notebook](https://github.com/stas00/jupyter-notebook-tools) |

[make](https://github.com/stas00/make-tools) |

[python](https://github.com/stas00/python-tools) |

[tensorboard](https://github.com/stas00/tensorboard-tools) |

[unix](https://github.com/stas00/unix-tools)

✔ **Other Machine Learning:**  [ML ways](https://github.com/stas00/ml-ways) |

[ML ways](https://github.com/stas00/ml-ways) |

[Porting](https://github.com/stas00/porting)

Canonical URL: https://github.com/stas00/ml-engineering/blob/master/skills/ml-engineering/SKILL.md

2. Download the free version of [Prince XML](https://www.princexml.com/download/). It's used to build the pdf version of this book.

9. A measurement's provenance belongs in the book; the run's setup does not. What tool version, what flags, what hardware, what date - that is provenance, and a reader needs it to judge the number. That the package was missing and had to be

wget --user-agent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:39.0) Gecko/20100101 Firefox/39.0" URL

10. A guessed URL that returns a page is not necessarily the page you wanted - check the title and body before quoting it. If a guess lands on a listing or tag page, follow the site's own links, or read its `/specifications`-style index, to

11. **Do not propose adding an unmaintained GitHub repository** (archived, README says "not actively maintained" / "research artifact only", last meaningful commit years stale with no successor named). Prefer the live successor when one exi

9. A `000` result is bot mitigation, not death - `amd.com`, `hpe.com`, `microsoft.com` and `nasa.gov` all reject a bare client. Confirm with the browser user-agent recipe in `Sources and citations` item 9 before calling anything dead. 10. W

10. Watch the URL extractor itself. Markdown wraps URLs in ways that corrupt a naive regex: a closing paren ends the match, so `Hopper_(microarchitecture)` truncates and then 404s, and a URL inside backticks picks the backtick up as its tai

11. Two moves are invisible to an HTTP check, and `build/check-redirects.py` now reports both. A **`<meta http-equiv="refresh">`** is markup rather than a status, so `curl -L` does not follow it and the stale URL answers `200` forever. Conc

#!/usr/bin/env python3

# network. But one defect is detectable locally and cheaply, so it is caught on every run: a URL # wrapped into another URL. An archive citation legitimately carries two schemes # (web.archive.org/web/<ts>/https://original), so three or mor

# wrapped into another URL. An archive citation legitimately carries two schemes # (web.archive.org/web/<ts>/https://original), so three or more means a replacement was applied to a # URL that was a substring of a longer one. Concrete near-

# (web.archive.org/web/<ts>/https://original), so three or more means a replacement was applied to a

#!/usr/bin/env bash

for url in "$@"; do     case "$url" in         http://*|https://*) ;;

case "$url" in         http://*|https://*) ;;         *)

http://*|https://*) ;;

if result=$(curl "${curl_args[@]}" "$url" 2>&1); then

printf 'LIVE %s -> %s\n' "$url" "$result"     elif result=$(curl --http1.1 "${curl_args[@]}" "$url" 2>&1); then         printf 'LIVE %s -> %s\n' "$url" "$result"

elif result=$(curl --http1.1 "${curl_args[@]}" "$url" 2>&1); then

#!/usr/bin/env bash

#!/usr/bin/env python3

Two mechanisms are checked. HTTP redirects, which curl follows for us, and <meta refresh>,

# A <meta http-equiv="refresh"> is a redirect that curl does not follow, because it is markup

u = re.sub(r'^(https://github\.com/[^/]+/[^/]+)\.git$', r'\1', u)

try:         r = subprocess.run(['curl', '-sL', '--max-time', '20', '-A', BROWSER_UA,

r = subprocess.run(['curl', '-sL', '--max-time', '20', '-A', BROWSER_UA,

"""curl never sends the #fragment, so `final` always lacks it. Carry it over, or a naive

#!/usr/bin/env python3

#!/usr/bin/env python3

("|[" + "n" * 20 + "](https://example.com/" + "n" * 80 + ")|\n|---|\n|x|", False),

#!/usr/bin/env python3

return (         '<?xml version="1.0" encoding="UTF-8" standalone="no"?>\n'         '<svg xmlns="http://www.w3.org/2000/svg" '

'<?xml version="1.0" encoding="UTF-8" standalone="no"?>\n'         '<svg xmlns="http://www.w3.org/2000/svg" '         'xmlns:xlink="http://www.w3.org/1999/xlink" '

'<svg xmlns="http://www.w3.org/2000/svg" '

'xmlns:xlink="http://www.w3.org/1999/xlink" '

'xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" '

#!/usr/bin/env python

# rtfm https://linkchecker.github.io/linkchecker/man/linkcheckerrc.html

Executable file

my_repo_url = "https://github.com/stas00/ml-engineering/blob/master"

with open(cover_md_file, "w") as f:         f.write(f"""         PDF Conversion was done with https://www.princexml.com

f.write(f"""         PDF Conversion was done with https://www.princexml.com <div class="page_break"></div>

PDF Conversion was done with https://www.princexml.com

This is an ebook version of [Machine Learning Engineering Open Book by Stas Bekman](https://github.com/stas00/ml-engineering/) generated on {date}.

As this book is constantly being updated, if you downloaded it as a pdf or an epub file and the date isn't recent, chances are that it's already outdated - make sure to check the latest version at [https://github.com/stas00/ml-engineering](

Executable file

#!/usr/bin/env python3

Executable file

Executable file

# - empty links (i.e. just local anchor to the same doc)     # - skip explicit .md links     # - external links like https://...

# - external links like https://...

# skip:     # - empty links (i.e. just local anchor to the same doc)     # - external links like https://...

# - external links like https://...

# # run to test some of these utils     # para = 'bb [Markdown text](foo.md#tar) aaa bb [Markdown text2](foo/#bar) aaa [Markdown text3](http://ex.com/foo/#bar)'

# para = 'bb [Markdown text](foo.md#tar) aaa bb [Markdown text2](foo/#bar) aaa [Markdown text3](http://ex.com/foo/#bar)'

/*   CSS style sheet for prince html2pdf system (http://www.princexml.com/)

CSS style sheet for prince html2pdf system (http://www.princexml.com/)

prince --no-author-style -s prince_style.css http://en.wikipedia.org/wiki/Winter_war -o foo.pdf

@import url(http://www.princexml.com/fonts/gentium/index.css);

There exist two main ML workloads - training and inference. There is also the finetuning workload which is usually the same as training, unless a much lighter [LORA-style](https://arxiv.org/abs/2106.09685) finetuning is performed. The latte

[Additional glossary @ Modal](https://modal.com/gpu-glossary)

[source](https://www.nvidia.com/en-us/data-center/a100/)

This is one of the key characteristics that the accelerators are judged by. The term TFLOPS defines how many trillions of FloatingPointOperations the chip can perform in a second. The more the better. There is a different definition for dif

[source](https://developer.nvidia.com/blog/cuda-11-features-revealed/)

First, let's extract the necessary accelerator specs from [wiki](https://en.wikipedia.org/wiki/Hopper_(microarchitecture)#H100_accelerator_and_DGX_H100).

The tricky part was to find the FMAs ops per Tensor Core per clock cycle for BF16 (half precision). I found them [here](https://forums.developer.nvidia.com/t/how-to-calculate-the-tensor-core-fp16-performance-of-h100/244727/2). Most are comi

1. [Google Cloud TPU v5e documentation](https://docs.cloud.google.com/tpu/docs/v5e)

2. [Google Cloud TPU v4 documentation](https://docs.cloud.google.com/tpu/docs/v4)

3. Oddly NVIDIA A100 PCIe and SXM revisions [spec](https://www.nvidia.com/en-us/data-center/a100/) are reported to have the same TFLOPS, which is odd considering the SXM version uses 30% more power and uses a 5% faster HBM.

4. [NVIDIA A100 specifications](https://www.nvidia.com/en-us/data-center/a100/) - covers both the SXM and PCIe rows

5. [AMD Instinct MI250 specifications](https://www.amd.com/en/products/accelerators/instinct/mi200/mi250.html)

6. [NVIDIA L40S specifications](https://www.nvidia.com/en-us/data-center/l40s/)

7. [AMD Instinct MI250X specifications](https://www.amd.com/en/products/accelerators/instinct/mi200/mi250x.html)

8. Intel Gaudi2 and 3 only have partial TFLOPS [specs](https://www.intel.com/content/www/us/en/content-details/817486/intel-gaudi-3-ai-accelerator-white-paper.html) published, but it does support FP32, TF32, BF16, FP16 & FP8, INT8 and INT16

9. [Intel Gaudi 2 AI accelerator white paper](https://www.intel.com/content/www/us/en/content-details/839363/intel-gaudi-2-ai-accelerators-white-paper.html)

10. [Google Cloud TPU v5p documentation](https://docs.cloud.google.com/tpu/docs/v5p)

11. Trainium2 also supports FP8/FP16/BF16/TF32 @ 2563TFLOPS w/ 4:1 sparsity. See [AWS Trainium2 architecture documentation](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trainium2.html).

12. [NVIDIA H100 specifications](https://www.nvidia.com/en-us/data-center/h100/) - covers both the SXM and PCIe rows

13. [Google Cloud TPU v6e documentation](https://docs.cloud.google.com/tpu/docs/v6e)

14. GH200 - same note as GB200 - this is 2 GPUs in one package, so the table includes specs per chip w/o sparsity. See [NVIDIA GH200 Grace Hopper Superchip specifications](https://www.nvidia.com/en-us/data-center/grace-hopper-superchip/).

15. I didn't include `NVIDIA H100 dual NVL` as it's, well, 2x GPUs - so it won't be fair - it's the same FLOPS as H100 but 2x everything, plus at has a bit more memory (94GiB per chip, as compared to 80GiB H100) and the memory is a bit fast

16. H200 is the same as H100 but has 141GiB vs 80GiB of HBM memory, and its memory is faster, HBMe@4.8TBps vs HBM@3.35TBps - so basically H200 solves the memory-bandwidth and memory-capacity bottlenecks of H100. See [NVIDIA H200 specificati

17. [AMD Instinct MI300X specifications](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html)

18. MI325X is the same compute as MI300X, but has more memory and more power (more efficient compute). See [AMD Instinct MI325X specifications](https://www.amd.com/en/products/accelerators/instinct/mi300/mi325x.html).

19. Gaudi3 as of 2026-08 is running at 1600MHz (MME) and not the planned 1750MHz, therefore its BF16 TFLOPS are 1677 and not 1835 as per whitepaper spec. Same goes for fp8 which runs at the same TFLOPS as BF16. 20. [NVIDIA DGX B200 datashee

20. [NVIDIA DGX B200 datasheet](https://resources.nvidia.com/en-us-dgx-systems/dgx-b200-datasheet) - `FP4 Tensor Core: 144|72` petaFLOPS sparse|dense for 8 GPUs, so 9000 dense per GPU; the [HGX page](https://www.nvidia.com/en-us/data-center

21. [NVIDIA DGX B300 datasheet](https://resources.nvidia.com/en-us-dgx-systems/dgx-b300-datasheet) - `FP4 Tensor Core 144|108` PFLOPS sparse|dense for 8 GPUs, so 13500 dense per GPU, the "1.5x dense FP4" over B200 the datasheet advertises; 

22. [AMD Instinct MI355X specifications](https://www.amd.com/en/products/accelerators/instinct/mi350/mi355x.html) - these are AMD's dense figures. AMD also publishes `with Structured Sparsity` variants at exactly 2x - 10.1PFLOPS for OCP-FP8

23. Since GB200 is 2x B200 chips the table includes TFLOPS per chip for a fair comparison - you'd 2x it for the real GB200 - it also seems to run the B200 chips a bit faster so higher specs than standalone B200. This also means that instead

24. GB200 NVL72 and GB300 NVL72 seem to be the same but faster fp4 and more memory for the latter. See [NVIDIA GB300 NVL72 specifications](https://www.nvidia.com/en-us/data-center/gb300-nvl72/).

25. [Google Cloud TPU v7x documentation](https://docs.cloud.google.com/tpu/docs/tpu7x) - Google calls it "the latest TPU available on Google Cloud" and documents using it through GKE or Compute Engine, so it is treated as available like eve

26. [NVIDIA Vera Rubin NVL72 specifications](https://www.nvidia.com/en-us/data-center/vera-rubin-nvl72/) - pre-release, so these are announced figures rather than a shipped part's spec sheet

27. AMD's MI455X figures come from its [product page](https://www.amd.com/en/products/accelerators/instinct/mi400/mi455x.html), read on 2026-08-06. `fp32` 315, `fp16` 5000, `bf16` 5000 and `int8` 5000 are AMD's own dense numbers. `fp8`, `fp

28. Ascend 950DT figures are from the Atlas 950 SuperPoD product-spec table on [Ascend Community](https://www.hiascend.com/en/hardware/cluster?tag=950) (read 2026-08-10): `114.1 / 58.8 / 31.1` PFLOPS @ `mxFP4` / `mxFP8/HiF8` / `FP16/BF16` f

29. H200 NVL is the PCIe card version of the H200: the same chip and the same 141GB of HBM3e at 4.8TBps as the H200 SXM, but with a 600W power limit instead of 700W, and NVIDIA rates its compute about 16% lower. The `tf32`, `fp16`, `bf16`, 

30. Intel GPU D270 (code-named Crescent Island) is the first product in Intel's Xe3P-based GPU D-series - an inference-only data-center accelerator (32 Xe3P cores, 256 XMX matrix engines, 32MB L2), announced at OCP 2026 (2026-10-12) with cu

6. NVIDIA B200 SXM, BF16: MAMF ~274–300 W @ 1965 MHz; MSMF 1439 @ 3072x18944x4608, ~978–987 W @ 1399–1443 MHz, 61–74°C 7. AMD MI355X, BF16: `PYTORCH_TUNABLEOP_ENABLED=0` 8. AMD MI350X, BF16: `PYTORCH_TUNABLEOP_ENABLED=1`, 1000 W; the `amdsm

7. AMD MI355X, BF16: `PYTORCH_TUNABLEOP_ENABLED=0` 8. AMD MI350X, BF16: `PYTORCH_TUNABLEOP_ENABLED=1`, 1000 W; the `amdsmi` sample of ~340 W @ 2181 MHz lags the MAMF burst, as on MI300X (note 10); MSMF 1195 @ 4096x4096x8192, 1000 W, 74–100°

8. AMD MI350X, BF16: `PYTORCH_TUNABLEOP_ENABLED=1`, 1000 W; the `amdsmi` sample of ~340 W @ 2181 MHz lags the MAMF burst, as on MI300X (note 10); MSMF 1195 @ 4096x4096x8192, 1000 W, 74–100°C. This node's cooling limited its MSMF: the 4 GPUs

9. AMD MI325X, BF16: `PYTORCH_TUNABLEOP_ENABLED=1`, `numa_balancing=0`, 1000 W. MSMF 787 @ 9216x9728x8192, 1000 W @ 1577 MHz, 73°C. 10. AMD MI300X, BF16: `PYTORCH_TUNABLEOP_ENABLED=1`, 750 W; MAMF is not a boost burst - the card reaches its

10. AMD MI300X, BF16: `PYTORCH_TUNABLEOP_ENABLED=1`, 750 W; MAMF is not a boost burst - the card reaches its 750 W cap within the first ~4 ms kernel, and the `amdsmi` sample of ~198 W @ 2088 MHz lags it and reads close to idle (~180 W @ 209

12. NVIDIA H200 SXM, FP8: MAMF ~154–163 W @ 1980 MHz; MSMF 1354 @ 1536x2816x10240, ~683–689 W @ 1767–1848 MHz, 63–72°C 13. NVIDIA B200 SXM, FP8: MAMF ~270–312 W @ 1965 MHz; MSMF 2788 @ 3072x18944x6144, ~975–982 W @ 1399–1447 MHz, 61–73°C 14

13. NVIDIA B200 SXM, FP8: MAMF ~270–312 W @ 1965 MHz; MSMF 2788 @ 3072x18944x6144, ~975–982 W @ 1399–1447 MHz, 61–73°C 14. AMD MI300X, FP8: `float8_e4m3fnuz`, `PYTORCH_TUNABLEOP_ENABLED=1`, 750 W; MAMF is not a boost burst, as in note 10; M

14. AMD MI300X, FP8: `float8_e4m3fnuz`, `PYTORCH_TUNABLEOP_ENABLED=1`, 750 W; MAMF is not a boost burst, as in note 10; MSMF 1182 @ 8192x4864x4096, 750 W @ 1322–1378 MHz, 73–86°C. Measured by [Amy Sartran](https://github.com/stas00/ml-engin

- Rows are not always exactly comparable with each other. Each is the best shape found for *that* accelerator, and often on a different `torch`/CUDA or ROCm version, so any gap between two rows mixes hardware, shape and software. But it's a

- Of the rows with an MSMF, H200, B200 and A100 PCIe were measured with benchmark version 4 of `mamf-finder.py` (the report header prints it), and B300 and MI300X with an earlier version that defined both headlines differently, so compare t

- MI325X was measured on 2026-10-05 with the unchanged version 4 finder at [`30ddd970`](https://github.com/stas00/ml-engineering/commit/30ddd97054eb01035c0a5079a58f011df603b525), with siblings idle. These are best-MAMF single-run pairs, not

update: this new metric is starting to catch on. AMD published this graph and [explanations of why the efficiency of accelerators is going down as they get faster](https://rocm.blogs.amd.com/software-tools-optimization/Understanding_Peak_an

[source](https://rocm.blogs.amd.com/software-tools-optimization/Understanding_Peak_and_Max-Achievable_FLOPS/README.html)

The accelerators use [High Bandwidth Memory](https://en.wikipedia.org/wiki/High_Bandwidth_Memory) (HBM) which is a 3D version of SDRAM memory. For example, the 80GiB A100-SXM comes with HBM2e at 2.0TBps, and the 80GiB H100-SXM comes with HB

- While I was researching this table I found a wide variation of the above numbers. I think it's because either there were different implementations or the specs changed several times and different publications caught different specs. The t

* The listed sizes preserve vendor-published `GB` labels. As of 2026-07-29, NVIDIA's [HGX AI Factory components](https://docs.nvidia.com/enterprise-reference-architectures/hgx-ai-factory/latest/components.html) page lists B200 at 180GB, and

* The `NVIDIA Rubin SXM` row is the standard Rubin GPU (the `288`GB HBM4 part, sometimes called `R200`). `19.20 TBps` is NVIDIA's current per-GPU figure on the [Vera Rubin NVL72](https://www.nvidia.com/en-us/data-center/vera-rubin-nvl72/) s

1. AMD MI350X / MI355X [cache specification](https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/optimization/workload-optimization.html) and [announcement](https://newsroom.amd.com/news/amd-unveils-vision-for-an-open-ai-ecosystem-det

2. AMD MI300X / MI325X [cache specification](https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/optimization/workload-optimization.html) and [announcement](https://www.amd.com/en/newsroom/press-releases/2024-10-10-amd-delivers-leader

3. [AMD Instinct MI455X specifications and announcement](https://www.amd.com/en/products/accelerators/instinct/mi400/mi455x.html)

4. NVIDIA Blackwell [cache specification](https://docs.nvidia.com/cuda/blackwell-tuning-guide/index.html) and [announcement](https://nvidianews.nvidia.com/news/nvidia-blackwell-platform-arrives-to-power-a-new-era-of-computing)

5. Intel Gaudi3 [cache specification](https://www.intel.com/content/www/us/en/content-details/817486/intel-gaudi-3-ai-accelerator-white-paper.html) and [announcement](https://newsroom.intel.com/artificial-intelligence/vision-2024-enterprise

6. NVIDIA Hopper [cache specification](https://docs.nvidia.com/cuda/hopper-tuning-guide/index.html) and [announcement](https://nvidianews.nvidia.com/news/nvidia-announces-dgx-h100-systems-worlds-most-advanced-enterprise-ai-infrastructure)

7. NVIDIA Ampere [cache specification](https://docs.nvidia.com/cuda/ampere-tuning-guide/index.html) and [announcement](https://nvidianews.nvidia.com/news/nvidias-new-ampere-data-center-gpu-in-full-production)

1. [AMD Instinct MI455X specifications and announcement](https://www.amd.com/en/products/accelerators/instinct/mi400/mi455x.html)

2. AMD MI350X / MI355X [cache specification](https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/optimization/workload-optimization.html) and [announcement](https://newsroom.amd.com/news/amd-unveils-vision-for-an-open-ai-ecosystem-det

3. AMD MI300X / MI325X [cache specification](https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/optimization/workload-optimization.html) and [announcement](https://www.amd.com/en/newsroom/press-releases/2024-10-10-amd-delivers-leader

4. Intel Gaudi3 [cache specification](https://www.intel.com/content/www/us/en/content-details/817486/intel-gaudi-3-ai-accelerator-white-paper.html) and [announcement](https://newsroom.intel.com/artificial-intelligence/vision-2024-enterprise

5. Intel Gaudi2 [architecture specification](https://docs.habana.ai/en/latest/Gaudi_Overview/Gaudi_Architecture.html) and [announcement](https://www.intel.com/content/www/us/en/developer/articles/technical/habana-gaudi2-processor-for-deep-l

6. NVIDIA Blackwell [cache specification](https://docs.nvidia.com/cuda/blackwell-tuning-guide/index.html) and [announcement](https://nvidianews.nvidia.com/news/nvidia-blackwell-platform-arrives-to-power-a-new-era-of-computing)

7. NVIDIA Hopper [cache specification](https://docs.nvidia.com/cuda/hopper-tuning-guide/index.html) and [announcement](https://nvidianews.nvidia.com/news/nvidia-announces-dgx-h100-systems-worlds-most-advanced-enterprise-ai-infrastructure)

8. NVIDIA Ampere [cache specification](https://docs.nvidia.com/cuda/ampere-tuning-guide/index.html) and [announcement](https://nvidianews.nvidia.com/news/nvidias-new-ampere-data-center-gpu-in-full-production)

1. [AMD Instinct MI300 Series / MI350 Series workload optimization](https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/optimization/workload-optimization.html)

2. [Intel Gaudi architecture](https://docs.habana.ai/en/latest/Gaudi_Overview/Gaudi_Architecture.html)

3. [Intel Gaudi3 white paper](https://www.intel.com/content/www/us/en/content-details/817486/intel-gaudi-3-ai-accelerator-white-paper.html)

4. [NVIDIA Ampere Tuning Guide](https://docs.nvidia.com/cuda/ampere-tuning-guide/index.html)

5. [NVIDIA Blackwell Tuning Guide](https://docs.nvidia.com/cuda/blackwell-tuning-guide/index.html)

6. [NVIDIA Hopper Tuning Guide](https://docs.nvidia.com/cuda/hopper-tuning-guide/index.html)

7. [NVIDIA Hopper architecture white paper](https://resources.nvidia.com/en-us-hopper-architecture/nvidia-h100-tensor-c)

8. [NVIDIA H200 specifications](https://www.nvidia.com/en-us/data-center/h200/)

Also known as [clock rate](https://en.wikipedia.org/wiki/Clock_rate) this spec tells us at which frequency the card runs. As hardware becomes faster newer generations will typically increase the clock speed.

1. [AMD Instinct MI355X specifications](https://www.amd.com/en/products/accelerators/instinct/mi350/mi355x.html)

2. [AMD Instinct MI455X specifications](https://www.amd.com/en/products/accelerators/instinct/mi400/mi455x.html)

3. [AMD Instinct MI300X specifications](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html)

4. [AMD Instinct MI325X specifications](https://www.amd.com/en/products/accelerators/instinct/mi300/mi325x.html)

5. [NVIDIA System Management Interface documentation](https://docs.nvidia.com/deploy/nvidia-smi/)

6. [Intel Gaudi `hl-smi` documentation](https://docs.habana.ai/en/latest/Management_and_Monitoring/Embedded_System_Tools_Guide/System_Management_Interface_Tool.html)

7. [NVIDIA A100 architecture white paper](https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf)

8. [NVIDIA GB300 NVL72 specifications](https://www.nvidia.com/en-us/data-center/gb300-nvl72/)

9. [NVIDIA Vera Rubin NVL72 specifications](https://www.nvidia.com/en-us/data-center/vera-rubin-nvl72/)

10. [Intel Gaudi architecture documentation](https://docs.habana.ai/en/latest/Gaudi_Overview/Gaudi_Architecture.html)

11. Intel Gaudi exposes separate clocks for its Matrix Multiplication Engine (MME) and Tensor Processing Core (TPC). 12. 1830MHz is NVIDIA's official compute clock for H200 and H100, superseding the 1980MHz that was widely published earlier

12. 1830MHz is NVIDIA's official compute clock for H200 and H100, superseding the 1980MHz that was widely published earlier. It is also self-consistent: it reproduces the 989TFLOPS bf16 spec exactly through the [TFLOPS calculation](#how-to-

13. [NVIDIA H200 specifications](https://www.nvidia.com/en-us/data-center/h200/) publish no clock for H200 NVL. Its 835TFLOPS bf16 spec works back to about 1545MHz through the [TFLOPS calculation](#how-to-calculate-theoretical-tflops), if i

14. [AMD Instinct MI350X specifications](https://www.amd.com/en/products/accelerators/instinct/mi350/mi350x.html)

- [NVIDIA defines TGP](https://docs.nvidia.com/mission-control/docs/systems-administration-guide/2.3.0/wpps/concepts.html#tgp-total-graphics-power) as the total electrical power for the GPU core, onboard memory, and supporting board circuit

- [AMD lists MI250X](https://www.amd.com/en/products/accelerators/instinct/mi200/mi250x.html) with 500W TDP and 560W peak TBP, so for this OAM module the published peak TBP is 60W above its TDP.

- NVIDIA's [GB300 rack-planning example](https://docs.nvidia.com/mission-control/docs/systems-administration-guide/2.3.0/prs/faq.html) adds four 1.4kW GPUs and a 1kW CPU into a 6.6kW managed node budget, adds 0.9kW of static node power, the

1. [NVIDIA Mission Control power-management FAQs](https://docs.nvidia.com/mission-control/docs/systems-administration-guide/2.3.0/prs/faq.html)

2. [AMD Instinct MI355X specifications](https://www.amd.com/en/products/accelerators/instinct/mi350/mi355x.html)

3. [AMD Instinct MI325X specifications](https://www.amd.com/en/products/accelerators/instinct/mi300/mi325x.html)

4. [AMD Instinct MI350X specifications](https://www.amd.com/en/products/accelerators/instinct/mi350/mi350x.html)

5. [NVIDIA DGX B200 power capping](https://docs.nvidia.com/dgx/dgxb200-user-guide/power-capping.html)

6. [NVIDIA DGX B300 power capping](https://docs.nvidia.com/dgx/dgxb300-user-guide/power-capping.html)

7. [Intel Gaudi 3 AI accelerator white paper](https://www.intel.com/content/www/us/en/content-details/817486/intel-gaudi-3-ai-accelerator-white-paper.html)

8. [AMD Instinct MI300X specifications](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html)

9. [NVIDIA H100 specifications](https://www.nvidia.com/en-us/data-center/h100/)

10. [NVIDIA H200 specifications](https://www.nvidia.com/en-us/data-center/h200/)

11. [AMD Instinct MI350P specifications](https://www.amd.com/en/products/accelerators/instinct/mi350/mi350p.html)

12. [Intel Gaudi 2 AI accelerator white paper](https://www.intel.com/content/www/us/en/content-details/839363/intel-gaudi-2-ai-accelerators-white-paper.html)

13. [NVIDIA RTX PRO 6000 Blackwell Server Edition specifications](https://www.nvidia.com/en-us/data-center/rtx-pro-6000-blackwell-server-edition/)

14. [AMD Instinct MI250X specifications](https://www.amd.com/en/products/accelerators/instinct/mi200/mi250x.html)

15. [NVIDIA A100 specifications](https://www.nvidia.com/en-us/data-center/a100/)

16. [NVIDIA L40S specifications](https://www.nvidia.com/en-us/data-center/l40s/)

17. [AMD Instinct MI455X specifications](https://www.amd.com/en/products/accelerators/instinct/mi400/mi455x.html)

18. [AWS Trainium2 architecture documentation](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trainium2.html)

19. [NVIDIA Vera Rubin NVL72 specifications](https://www.nvidia.com/en-us/data-center/vera-rubin-nvl72/)

20. Google doesn't publish power consumption specs for recent TPUs, the older ones can be found [here](https://en.wikipedia.org/wiki/Tensor_Processing_Unit#Products)

NVIDIA: - [Vera Rubin NVL72](https://www.nvidia.com/en-us/data-center/vera-rubin-nvl72/) -- the 72 GPUs supernode at NVLink speed with Grace Rubin - 36x blocks of 2x Rubin GPU + Vera CPU

- [Vera Rubin NVL72](https://www.nvidia.com/en-us/data-center/vera-rubin-nvl72/) -- the 72 GPUs supernode at NVLink speed with Grace Rubin - 36x blocks of 2x Rubin GPU + Vera CPU

- [GB300 NVL72](https://www.nvidia.com/en-us/data-center/gb300-nvl72/) - the 72 GPUs supernode at NVLink speed with B300s (Grace Blackwell - 36x blocks of 2x B300 + Grace CPU)

- [GB200 NVL72](https://www.nvidia.com/en-us/data-center/gb200-nvl72/) - the 72 GPUs supernode at NVLink speed with B200s. (Grace Blackwell - 36x blocks of 2x B200 + Grace CPU)

- [B300](https://www.nvidia.com/en-us/data-center/hgx/) - NVIDIA now publishes official HGX B300 platform specs and lists it as shipping, as of 2026-07-31. Note that those are 8-GPU board figures (144PFLOPS FP4 sparse, 2.1TB total memory, 1

- [B200](https://www.nvidia.com/en-us/data-center/hgx/) - same story: official HGX B200 platform specs published and shipping (144PFLOPS FP4 sparse, 1.4TB total memory, 14.4TBps total NVLink), with the [DGX B200 datasheet](https://resources

- [H200](https://www.nvidia.com/en-us/data-center/h200/) - mainly the same as H100, but with more and faster memory! Widely available as of 2026-07-31.

- [H100](https://www.nvidia.com/en-us/data-center/h100) - 2-3x faster than A100 (half precision), 6x faster for fp8, has been available on all Tier-1 compute clouds since Q4-2023.

- [GH200](https://www.nvidia.com/en-us/data-center/grace-hopper-superchip/) - 2 chips on one card - (1) H100 w/ 96GiB HBM3 or 144GiB HBM3e + (2) Grace CPU w/ 624GiB RAM - available as of 2026-07-31. Do not confuse with H200, which is a diff

- [L40S](https://www.nvidia.com/en-us/data-center/l40s/) - a powerful card that is supposed to be more than 2x cheaper than H100, and it's more powerful than A100.

- [A100](https://www.nvidia.com/en-us/data-center/a100/#specifications) - huge availability, but already getting outdated. But given the much lower cost than H100 this is still a great GPU.

AMD: - [MI455X](https://www.amd.com/en/products/accelerators/instinct/mi400/mi455x.html) ~= Rubin, a little above it on the published numbers - but those are AMD Performance Labs projections carrying "Results subject to change when products

- [MI455X](https://www.amd.com/en/products/accelerators/instinct/mi400/mi455x.html) ~= Rubin, a little above it on the published numbers - but those are AMD Performance Labs projections carrying "Results subject to change when products are 

- [MI355X](https://www.amd.com/en/products/accelerators/instinct/mi350/mi355x.html) ~= B200 - just starting to emerge, mainly on Tier-2 clouds

- [MI350X](https://www.amd.com/en/products/accelerators/instinct/mi350/mi350x.html) ~= B200 - the same silicon as MI355X, air-cooled at 1000W and 2200MHz; rentable on demand on Tier-2 clouds as of 2026-10-07

- [MI325X](https://www.amd.com/en/products/accelerators/instinct/mi300/mi325x.html) ~= H200 - available mainly on Tier-2 clouds

- [MI300X](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html) ~= H100 - available mainly on Tier-2 clouds (lots of new startups)

- [MI250](https://www.amd.com/en/products/accelerators/instinct/mi200/mi250.html) ~= A100 - very few clouds have them

Intel: - Falcon Shores - canceled - [Gaudi3](https://habana.ai/products/gaudi3/), somewhat below B200 theoretical TFLOPS-wise - already available on Intel cloud - [spec](https://www.intel.com/content/www/us/en/content-details/817486/intel-g

- Falcon Shores - canceled - [Gaudi3](https://habana.ai/products/gaudi3/), somewhat below B200 theoretical TFLOPS-wise - already available on Intel cloud - [spec](https://www.intel.com/content/www/us/en/content-details/817486/intel-gaudi-3-

- [Gaudi3](https://habana.ai/products/gaudi3/), somewhat below B200 theoretical TFLOPS-wise - already available on Intel cloud - [spec](https://www.intel.com/content/www/us/en/content-details/817486/intel-gaudi-3-ai-accelerator-white-paper.

- [Gaudi2](https://habana.ai/products/gaudi2/) somewhere between A100 and H100 theoretical TFLOPS-wise [spec](https://docs.habana.ai/en/latest/Gaudi_Overview/Gaudi_Architecture.html) - available on Intel cloud. AWS has the older Gaudi1 via 

Amazon: - [Trainium2](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trainium2.html) < H100 - available on AWS (works via PyTorch XLA)

- [Trainium2](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trainium2.html) < H100 - available on AWS (works via PyTorch XLA)

Google: - [TPU7x](https://docs.cloud.google.com/tpu/docs/tpu7x) (Ironwood) - 2307TFLOPS bf16 and 192GB HBM per chip, in 4-chip VMs up to a 9216-chip pod. Rent-only on GCP, like every TPU generation

- [TPU7x](https://docs.cloud.google.com/tpu/docs/tpu7x) (Ironwood) - 2307TFLOPS bf16 and 192GB HBM per chip, in 4-chip VMs up to a 9216-chip pod. Rent-only on GCP, like every TPU generation

- [TPU v6e](https://docs.cloud.google.com/tpu/docs/v6e) (Trillium) and [TPU v5p](https://docs.cloud.google.com/tpu/docs/v5p) - the previous two generations, both still offered

SambaNova: - [SN40L](https://sambanova.ai/products/rdu-ai-chips) - fourth-generation RDU, 16 per SambaRack; inference-oriented rather than a training part

- [SN40L](https://sambanova.ai/products/rdu-ai-chips) - fourth-generation RDU, 16 per SambaRack; inference-oriented rather than a training part

- [SN50](https://sambanova.ai/blog/introducing-the-sn50-rdu-purpose-built-for-agentic-inference) - fifth generation, announced Feb-2026, "will start shipping to customers in the second half of 2026"

[Cerebras](https://www.cerebras.ai/) cluster and systems based on WaferScale Engine (WSE).

Google [TPUs](https://cloud.google.com/tpu), [specs](https://docs.cloud.google.com/tpu/docs/system-architecture-tpu-vm) - lock-in, can't switch to another vendor like NVIDIA -> AMD

Recent architecture specs: - [v7x](https://docs.cloud.google.com/tpu/docs/tpu7x)

- [v7x](https://docs.cloud.google.com/tpu/docs/tpu7x)

- [v6e](https://docs.cloud.google.com/tpu/docs/v6e)

- [v5p](https://docs.cloud.google.com/tpu/docs/v5p)

Cerebras: - [Cloud](https://www.cerebras.ai/cloud)

- [Cloud](https://www.cerebras.ai/cloud)

- [TensTorrent](https://tenstorrent.com), [n150s/n300s specs](https://docs.tenstorrent.com/aibs/wormhole/index.html#specificationsrequirements)

- [d-Matrix](https://www.d-matrix.ai), [specs](https://www.d-matrix.ai/product/)

For the baseline prices it should be easy to find a few good sites that provide an up-to-date public price comparisons across clouds - just search for something like [cloud gpu pricing comparison](https://www.google.com/search?q=cloud+gpu+p

Also Lantern Works tracks [on-demand GPU cloud rental prices](https://thelanternworks.com/gpu/price-comparison/) across multiple providers.

[AMD's table comparing its high-end gpus](https://rocm.docs.amd.com/en/latest/reference/gpu-specs.html)

[Architecture](https://docs.habana.ai/en/latest/Gaudi_Overview/Gaudi_Architecture.html)

There are no official TFLOPS information published (and from talking to an Intel representative they have no intention to publish any.) They publish the [following benchmarks](https://www.intel.com/content/www/us/en/developer/platform/gaudi

The Xe3P XMX engines natively support the full precision ladder - FP64, FP32, TF32, FP16, BF16, dual-format FP8, FP4, the OCP MX formats (MXFP4/MXFP8) and int8/int4 - but as of launch the only compute figure Intel has published is **1.3 PFL

Specs: [Trainium2](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trainium2.html), [Trn2 instances](https://aws.amazon.com/ec2/instance-types/trn2/).

Specs: [TPU7x](https://docs.cloud.google.com/tpu/docs/tpu7x), [TPU system architecture](https://docs.cloud.google.com/tpu/docs/system-architecture-tpu-vm).

Two things to be careful about. The TFLOPS and memory comparison-table rows for Ascend 950DT use the **64-card** Atlas 950 SuperPoD product-spec table on the English Ascend Community page (`114.1 / 58.8 / 31.1` PFLOPS @ mxFP4 / mxFP8-HiF8 /

Specs: [Ascend Community](https://www.hiascend.com/en). Be aware that both Huawei sites are JavaScript applications that serve only a page shell to a plain downloader - `e.huawei.com` returns 255KB of HTML containing not one occurrence of `

Specs: [Cerebras WSE](https://www.cerebras.ai/chip).

Specs: [RDU AI chips](https://sambanova.ai/products/rdu-ai-chips), [SambaRack](https://sambanova.ai/products/sambarack).

NVIDIA GPUs run on [CUDA](https://developer.nvidia.com/cuda/toolkit)

AMD GPUs run on [ROCm](https://www.amd.com/en/products/software/rocm.html) - note that PyTorch intentionally reuses the same `torch.cuda` Python API on ROCm, so many programs that only use PyTorch APIs can run unchanged on supported AMD GPU

The API is via [Habana SynapseAI® SDK](https://habana.ai/intel-gaudi-software/) which supports PyTorch and TensorFlow.

Useful integrations: - [HF Optimum Habana](https://github.com/huggingface/optimum-habana) which also includes - [DeepSpeed](https://github.com/deepspeedai/DeepSpeed) integration.

- [HF Optimum Habana](https://github.com/huggingface/optimum-habana) which also includes - [DeepSpeed](https://github.com/deepspeedai/DeepSpeed) integration.

- [MLPerf via MLCommons](https://mlcommons.org/) publishes various hardware benchmarks that measure training, inference, storage and other tasks' performance. The round numbers advance a few times a year, so head to the suite pages rather t

On Linux, AMD’s isolation docs lean on `ROCR_VISIBLE_DEVICES` when you want stack-wide isolation; for ordinary single-framework PyTorch work, `HIP_VISIBLE_DEVICES` / `CUDA_VISIBLE_DEVICES` is enough. See [GPU isolation techniques](https://r

- **Modern Instinct (MI300X, MI325X, MI350X, MI355X)**: AMD's system-acceptance guides want the IOMMU **enabled** in BIOS and running in pass-through mode on the kernel command line - `iommu=pt` together with `amd_iommu=on` (or `intel_iommu

- **Legacy last resort**: on older systems where neither pass-through nor the BIOS default helped, fully disabling it via `amd_iommu=off` (or the softer `iommu=soft`) was the historical workaround - see AMD's [IOMMU advisory for multi-GPU e

- [AMD Instinct MI300X system optimization](https://instinct.docs.amd.com/projects/system-acceptance/en/latest/gpus/mi300x.html)

- [AMD Instinct MI300X workload optimization](https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/optimization/workload-optimization.html)

[omniperf](https://github.com/ROCm/rocprofiler-compute) - Advanced Profiling and Analytics for AMD Hardware - e.g. can plot a roofline performance of your AMD accelerator and many other things.

1. [How to Accurately Time CUDA Kernels in PyTorch](https://www.speechmatics.com/company/articles-and-news/timing-operations-in-pytorch)

2. [How to Benchmark Code on CUDA Devices?](https://salykova.github.io/sgemm-gpu#2-how-to-benchmark-code-on-cuda-devices) - this one is different from (1) in that it suggests to set both GPU and Memory clocks, whereas (1) only locks the GPU

Dynamic power is roughly proportional to clock frequency times the number of transistors that flip. High-entropy or "unpredictable" bit patterns flip more transistors per cycle than sparse, sorted, or all-zero patterns, so the same-shape ma

Maximum Achievable Matmul FLOPS (MAMF) Benchmark: [mamf-finder.py](./mamf-finder.py) was derived from research found in [The Case for Co-Designing Model Architectures with Hardware](https://arxiv.org/abs/2401.14489) paper.

sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'

This will make the first iteration very slow, while it's searching for the best GEMM algorithm in the BLAS libraries for each `matmul` shape it encounters, but subsequent operations are likely to be significantly faster than the baseline. S

**Intel dGPUs (A770, A750, B580, etc.)** - Follow Intel Extension for PyTorch [installation steps](https://pytorch-extension.intel.com/installation?platform=gpu)

- Follow Intel Extension for PyTorch [installation steps](https://pytorch-extension.intel.com/installation?platform=gpu)

Peak GEMM shapes are not mysterious; they follow the paper's recipe ([arXiv:2401.14489](https://arxiv.org/abs/2401.14489)):

If you get a better MAMF or MSMF than the [table](../README.md#maximum-achievable-and-sustainable-matmul-flops-comparison-table) shows, e.g. with a more recent `torch`, CUDA or ROCm, BLAS library or driver, or you measured an accelerator or

#!/usr/bin/env python

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

#!/usr/bin/env python

For the auto search, discussion, and important nuances see: https://github.com/stas00/ml-engineering/tree/master/compute/accelerator/benchmarks#maximum-achievable-and-sustainable-matmul-flops-finder

https://github.com/stas00/ml-engineering/tree/master/compute/accelerator/benchmarks#maximum-achievable-and-sustainable-matmul-flops-finder

Results table: https://github.com/stas00/ml-engineering/tree/master/compute/accelerator#maximum-achievable-and-sustainable-matmul-flops-comparison-table

https://github.com/stas00/ml-engineering/tree/master/compute/accelerator#maximum-achievable-and-sustainable-matmul-flops-comparison-table

Credits: - Parts of this benchmark have been derived from https://github.com/EleutherAI/cookbook/tree/main/benchmarks/sizing

- Parts of this benchmark have been derived from https://github.com/EleutherAI/cookbook/tree/main/benchmarks/sizing

(highly recommended!) - Imtiaz Sajwani: HPU porting - Xiaoyu Zhang https://github.com/BBuf - flexible dtype support

- Imtiaz Sajwani: HPU porting - Xiaoyu Zhang https://github.com/BBuf - flexible dtype support - Oren Leung https://github.com/OrenLeung - flagging the lack of cache/dest-matrix reset and suggesting a fix - also

- Xiaoyu Zhang https://github.com/BBuf - flexible dtype support

- Oren Leung https://github.com/OrenLeung - flagging the lack of cache/dest-matrix reset and suggesting a fix - also

proposing geomean - Ivan Fioravanti https://github.com/ivanfioravanti - MPS support """

- Ivan Fioravanti https://github.com/ivanfioravanti - MPS support

[Credential-related source omitted]

# # l2_cache: written between iterations to emulate cache reset. On AMD this is really L3/LLC — 256MiB is the highest # across recent accelerators so far (https://github.com/stas00/ml-engineering/tree/master/compute/accelerator#caches).

# l2_cache: written between iterations to emulate cache reset. On AMD this is really L3/LLC — 256MiB is the highest # across recent accelerators so far (https://github.com/stas00/ml-engineering/tree/master/compute/accelerator#caches). # C_r

# across recent accelerators so far (https://github.com/stas00/ml-engineering/tree/master/compute/accelerator#caches).

# Instead of brute-forcing a 3D grid of MxNxK shapes, `--search auto` constructs a small set of shapes the accelerator # should run at/near peak on, using the three rules from "The Case for Co-Designing Model Architectures with Hardware" # 

# should run at/near peak on, using the three rules from "The Case for Co-Designing Model Architectures with Hardware" # (https://arxiv.org/abs/2401.14489): #   1. Tensor-core alignment: M, N, K are multiples of `128 bytes / dtype_size` ele

# (https://arxiv.org/abs/2401.14489):

[Credential-related source omitted]

sudo grep Xid /var/log/syslog

sudo dmesg -T | grep Xid

The full list of Xid Errors and their interpretation can be found [here](https://docs.nvidia.com/deploy/xid-errors/latest/index.html).

Many Xid log lines carry far more fields than the one above, and they are cryptic. [Xid 31 MMU Faults: What Causes Them and How to Fix Production GPU Crashes](https://www.abhik.ai/articles/gpu-xid31-mmu-faults) is a good example of how to d

NVIDIA's [Xid Catalog](https://docs.nvidia.com/deploy/xid-errors/analyzing-xid-catalog.html) goes further for Ampere and newer GPUs: for each Xid it lists which GPUs can log it, what triggers it, and what to do about it - an immediate actio

Now, there are 2 types of errors - Correctable and Uncorrectable. The correctable one is a Single Bit ECC Error (SBE) where despite memory being faulty the driver can still recover the correct value. The uncorrectable one is where more than

To dive even deeper into the GPU debugging, please refer to [this document](https://docs.nvidia.com/deploy/gpu-debug-guidelines/index.html) - it includes a useful triage chart which helps to determine when to RMA GPUs. This document has add

When that happens it shows up in the same `Remapped Rows` block as above, with the failure flag set and a bank that has run out of spares. This is [Crusoe Cloud's published signature](https://docs.crusoecloud.com/resources/troubleshooting) 

NVIDIA® Data Center GPU Manager (DCGM) is documented [here](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/index.html) and can be downloaded from [here](https://github.com/NVIDIA/DCGM#quickstart).

$ sudo dmesg | grep -i 'limited by'

Row remapping is how Ampere and later GPUs deal with memory that has gone bad - a degrading bank row is replaced by one of the spares that every HBM bank reserves for the purpose. It is [NVIDIA's replacement for the page retirement scheme](

As explained [here](https://arthurchiao.art/blog/understanding-gpu-performance/) the `GPU-Util` column in the `nvidia-smi` output isn't really telling you the GPU Utilization. What it's telling you is the percentage of time during which one

What you want to measure instead is GPU's utilization of the available capacity, otherwise known as "saturation". Alas, this information isn't provided by `nvidia-smi`. In order to get this information you need to install [dcgm-exporter](ht

git clone https://github.com/NVIDIA/dcgm-exporter.git

$ sudo cmd/dcgm-exporter/dcgm-exporter -c 500 -f dcp-metrics-custom.csv

watch -n 0.5 "curl http://localhost:9400/metrics"

This is a quick way of doing that, but the intention is to use it with [Prometheus](https://prometheus.io/) which will give you nice charts. E.g. the article included an example where you can see the SM occupancy, Tensor core, FP16 and FP32

([source](https://arthurchiao.art/blog/understanding-gpu-performance/))

- Loading the model weights, unless they are loaded directly onto the GPUs - this is usually a transitory memory usage that goes back to zero once the model has been moved to GPUs. - Saving the model weights. In some situations each GPU wri

- Saving the model weights. In some situations each GPU writes its own checkpoint directly to the disk, in other cases the model is recomposed on the CPU before it's written to disk - this too is a transitory memory usage. - Possible parame

- Possible parameter and optimizer state offloading when using frameworks like [DeepSpeed](https://www.deepspeed.ai/tutorials/zero-offload/). In which case quite a lot of CPU memory might be needed.

- If the `DataLoader` uses HF `datasets` in `mmap` mode the Resident memory usage may appear to be using a huge amount of CPU memory as it'll try to map out the whole datasets to the memory. Except this is misleading, since if the memory is

To score what a model generated, RL tool-use workloads have to run it - a test suite, a compilation, a package install, a symbolic math check. That runs in a sandbox: an isolated environment, since generated code is untrusted by constructio

1. Isolation taxes system calls, not arithmetic. gVisor adds no cost to CPU instructions, but routes every system call through its own user-space kernel, so in some situations file- and process-heavy work could slow by multiples - and a tes

footnote: A [Nebius agentic-RL writeup](https://nebius.com/blog/posts/in-agentic-rl-faster-tokens-are-not-enough) (2026-09) measured this on a Terminal-Bench workload, GLM-5.2 on 32 B300s. With the number of in-flight trajectories held at 1

Some frameworks, like [DeepSpeed](https://www.deepspeed.ai/tutorials/zero-offload/) and [FSDP](https://docs.pytorch.org/docs/stable/fsdp.html) can offload some compute work to CPUs without creating a bottleneck. In which case you'd want add

[Hyper-Threads](https://en.wikipedia.org/wiki/Hyper-threading) double the CPU cores number, by virtualizing each physical core into 2 virtual ones, allowing 2 threads to use the same CPU core at the same time. Depending on the type of workl

1. Some of them did it via PRs, and are thus listed automatically [here](https://github.com/stas00/ml-engineering/graphs/contributors)

- [Adam Moody](https://github.com/adammoody)

- [Alex Rogozhnikov](https://github.com/arogozhnikov)

- [Bowei Liu](https://github.com/boweiliu)

- [Darrick Horton](https://www.linkedin.com/in/darrick-horton/)

- [Elio VP](https://www.linkedin.com/in/eliovp/)

- [Garrett Goon](https://github.com/garrett361)

- [Horace He](https://github.com/Chillee)

- [Ivan Yashchuk](https://github.com/IvanYashchuk)

- [Jack Dent](https://github.com/jackdent)

- [Jeff Rasley](https://github.com/jeffra/)

- [Jon Stevens](https://github.com/jon-hotaisle)

- [Jordan Nanos](https://github.com/JordanNanos)

- [Mark Saroufim](https://github.com/msaroufim)

- [Olatunji Ruwase](https://github.com/tjruwase)

- Oren Leung - [Quentin Anthony](https://github.com/Quentin-Anthony) - [Ross Wightman](https://github.com/rwightman)

- [Quentin Anthony](https://github.com/Quentin-Anthony)

- [Ross Wightman](https://github.com/rwightman)

- [Samyam Rajbhandari](https://github.com/samyam)

- [Satyaki Upadhyay](https://github.com/satyaki-up)

- [Shikib Mehri](https://github.com/Shikib)

- [Siddharth Singh](https://github.com/siddharth9820)

- [Stéphane Requena](https://twitter.com/s_requena)

- [Zhiqi Tao](https://www.linkedin.com/in/zhiqitao/)

- [Machine Learning Engineering (MLE)](../../README.md) - [The Art of Debugging (AoD)](https://github.com/stas00/the-art-of-debugging)

- [The Art of Debugging (AoD)](https://github.com/stas00/the-art-of-debugging)

From [The Art of Debugging](https://github.com/stas00/the-art-of-debugging):

📖 [The power of one-liner programs](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#the-power-of-one-liner-programs)

📖 [Finding a breaking commit by bisecting revisions](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#finding-a-breaking-commit-by-bisecting-revisions)

📖 [Get the backtrace from the core file](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#get-the-backtrace-from-the-core-file)

📖 [Get the backtrace from the still running process](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#get-the-backtrace-from-the-still-running-process) + [py-spy](https://github.com/stas00/the-art-of-de

📖 [Run the program under gdb](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#run-the-program-under-gdb)

📖 [nm](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#nm)

*All lessons are drawn from [Machine Learning Engineering](../../README.md) and [The Art of Debugging](https://github.com/stas00/the-art-of-debugging) by Stas Bekman, distributed under [CC BY-SA 4.0](../../LICENSE-CC-BY-SA).*

Executable file

It was written against https://github.com/python/cpython/blob/3.8/Lib/trace.py. If you're

[Credential-related source omitted]

# string with commands to run - passed to exec()

#!/usr/bin/env python

#!/usr/bin/env python

#!/usr/bin/env python

#!/usr/bin/env python

#!/usr/bin/env python

#!/usr/bin/env python

Executable file

Moreover, large models often require massive resources, which are typically expensive and can also can make a debugging process super complicated. For example any debugger can handle a single process, but if your model doesn't fit and requi

Important: given their popularity and the well designed simple API I will be discussing HF [`transformers`](https://github.com/huggingface/transformers/) models. But the same principle can be applied to any other model.

Now let's go through the actual code and convert ["google/mt5-small"](https://huggingface.co/google/mt5-small/tree/main) into its tiny random counterpart.

Also please note that you don't need any GPUs to do that and you could do this even on a huge 176B parameter model like [BLOOM-176B](https://huggingface.co/bigscience/bloom). Since you never load the actual original model, except its config

The original ["google/mt5-small"](https://huggingface.co/google/mt5-small/tree/main) model file was 1.2GB. With the above changes (and vocab shrinking as explained in the following sections) we got it down to 126MB.

If you're dealing with a multi-level nested config, you will have to update each sub-level's config object separately. For example in [IDEFICS](https://huggingface.co/HuggingFaceM4/idefics-9b/blob/main/config.json) we have 1 main and 2 nest

A related trap: architectures that interleave block types spell the pattern out in a `layer_types` list with one entry per layer. [Qwen/Qwen-AgentWorld-35B-A3B](https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B/blob/main/config.json#L19) 

[Anthony Moi](https://github.com/n1t0)'s tokenizer shrinker:

[Lysandre Debut](https://github.com/LysandreJik)' shrinker using `train_new_from_iterator`:

Here is the full version of [make_tiny_model.py](https://huggingface.co/stas/tiny-random-llama-2/blob/main/make_tiny_model.py) which includes both the model and the tokenizer shrinking.

git clone https://github.com/google/sentencepiece

[Credential-related source omitted]

tokenizer_orig.save_pretrained(tmp_dir) with open(vocab_orig_path, 'rb') as f: data = f.read() # adapted from https://blog.ceshine.net/post/trim-down-sentencepiece-vocabulary/

with open(vocab_orig_path, 'rb') as f: data = f.read() # adapted from https://blog.ceshine.net/post/trim-down-sentencepiece-vocabulary/ m = model.ModelProto()

# adapted from https://blog.ceshine.net/post/trim-down-sentencepiece-vocabulary/

If we continue the MT5 code from earlier in this chapter and add the tokenizer shrinking code from the previous section, we end up with this script [mt5-make-tiny-model.py](https://huggingface.co/stas/mt5-tiny-random/blob/main/mt5-make-tiny

Here is another example  [fsmt-make-super-tiny-model.py](https://huggingface.co/stas/tiny-wmt19-en-ru/blob/main/fsmt-make-super-tiny-model.py) - here you can see I'm creating a totally new tiny vocab from scratch.

Also be aware that since HF `transformers` needs tiny models for their testing, you are very likely to already find one for each architecture available mostly from https://huggingface.co/hf-internal-testing (except they didn't include the c

footnote: as of [`datasets` 4.0](https://github.com/huggingface/datasets/releases/tag/4.0.0), Hub-hosted Python loading scripts are no longer executed by `load_dataset`. Keep any builder/unpacker `.py` in the repo as documentation of how th

- [stas/oscar-en-10k](https://huggingface.co/datasets/stas/oscar-en-10k/blob/main/oscar-en-10k.py)

- [stas/c4-en-10k](https://huggingface.co/datasets/stas/c4-en-10k/blob/main/c4-en-10k.py)

- [stas/openwebtext-10k](https://huggingface.co/datasets/stas/openwebtext-10k/blob/main/openwebtext-10k.py)

And here are some examples of synthetic datasets, where instead of just shrinking the original tarball, I untar'ed it, manually chose the representative examples and then wrote a script to build any size of desired dataset based on those fe

- [stas/general-pmd-synthetic-testing](https://huggingface.co/datasets/stas/general-pmd-synthetic-testing/blob/main/general-pmd-synthetic-testing.py) and the [unpacker](https://huggingface.co/datasets/stas/general-pmd-synthetic-testing/blob

- [stas/cm4-synthetic-testing](https://huggingface.co/datasets/stas/cm4-synthetic-testing/blob/main/cm4-synthetic-testing.py) - and the [unpacker](https://huggingface.co/datasets/stas/cm4-synthetic-testing/blob/main/m4-ds-unpack.py)

And here is for example the instructions of deploying these scripts for [stas/general-pmd-synthetic-testing](https://huggingface.co/datasets/stas/general-pmd-synthetic-testing/):

# prep dataset repo https://huggingface.co/new-dataset => stas/general-pmd-synthetic-testing

https://huggingface.co/new-dataset => stas/general-pmd-synthetic-testing

git clone https://huggingface.co/datasets/stas/general-pmd-synthetic-testing

rm -rf data

rm image.jpg text.txt

Similar to tiny models, you will find many tiny datasets under https://huggingface.co/hf-internal-testing.

perl -lne 'while (/(https.*?.py)\)/g) { $x=$1; $x=~s/blob/raw/; print qq[wget $x] }' make-tiny-models.md

When debugging PyTorch workflows, as explained in [using small payload](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#2-small-payload) you'd normally try to use tiny random models (as covered [above](#fast

Therefore in this section we will discuss how to reduce the model's number of hidden layers from many to just 1-2. If the layers aren't identical (e.g. some MoE models alternate between 2 different block configurations) then ensure you incl

The config entry that we want to change is [`num_hidden_layers`](https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507/blob/e67ac5d/config.json#L24). Some architectures pair it with a companion entry that has to be trimmed in lockstep - a

git clone https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507

After finding the desired model on https://huggingface.co/, clone its git repo to the local disk, modify the `num_hidden_layers` entry in `config.json`, and then load the model from the local clone (same as we have just shown when measuring

git clone https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507

git clone https://github.com/huggingface/transformers/tree/main/src/transformers

Continuing the example of working with `Qwen/Qwen3-30B-A3B-Instruct-2507` model, we find the place where its architecture modeling code lives in the HF Transformers code base. For example, we can look at the `architectures` field in the [mo

When I was debugging 15M sequence length training using [ALST](https://arxiv.org/abs/2506.13996) - I would only run self-attention in the last layer and then skip it in the previous layers - this reduced my testing time from hours to minute

Luckily, PyTorch has a special [autograd mechanism that is used to detect anomalies](https://docs.pytorch.org/docs/stable/autograd.html#debugging-and-anomaly-detection). When enabled it'll augment the `backward` traceback with the `forward`

When developing [Arctic Long Sequence Training](https://arxiv.org/abs/2506.13996) a lot of the work involved hard (`contiguous`) reshaping of tensors for Sequence Parallelism, which lead to a very poor HBM utilization, because the reshapes 

Indeed, H200 is about 140GiB, so this checks out. You might wonder what's the point of this. This discovery code is useful when you have to deal with multiple systems accessing the same GPU - e.g. VLLM and VeRL sharing the same GPUs and PyT

The solution: to prevent CPU from stealing GPU memory switch to the CDMM mode (from the default NUMA mode), as explained in a paper linked from [this post](https://developer.nvidia.com/blog/understanding-memory-management-on-hardware-cohere

$ echo options nvidia NVreg_CoherentGPUMemoryMode=driver | sudo tee /etc/modprobe.d/nvidia-openrm.conf

At this stage you'd typically go to https://docs.pytorch.org/memory_viz and drop the memory profile pickle file you generated in Stage 1 into the browser at that URL. The most useful feature there is the 'Active memory profile` drop-down me

To get a feeling for what it looks like, here is an example of a memory profile rendering for a memory leak I discovered while I was working on a tricky implementation of a [`TiledMLP` `torch.autograd.Function`](https://github.com/deepspeed

As long as you're in the `forward` function it's relatively easy to find where in the code leaks comes from. But if it's a `backward` it becomes much more complicated unless you're debugging a custom autograd function. Still it should give 

Besides profiling memory leaks, this functionality is also useful for showing how different implementations of the same algorithm use a different amount of gpu memory. For example, the following visualization I prepared for the [Arctic Long

Additional resources: - [Understanding GPU Memory 1: Visualizing All Allocations over Time](https://pytorch.org/blog/understanding-gpu-memory-1/)

- [Understanding GPU Memory 1: Visualizing All Allocations over Time](https://pytorch.org/blog/understanding-gpu-memory-1/)

- HF folks made an [improved rendering version](https://huggingface.co/spaces/Leiyre/memory-viz).

While external memory profilers can be very useful, often having control over when you take a sample of GPU and CPU memory usage is needed. `see-mem-usage` debug util has been developed by the [DeepSpeed project](https://github.com/deepspee

For example, here is how I found a memory leak in `all_gather_object`, which you can see from this [Issue](https://github.com/pytorch/pytorch/issues/150798). And there were several other similar leaks in PyTorch I discovered using this tool

which is very difficult to notice. This is typically performed by an `oom-kill` via [cgroups](https://docs.kernel.org/admin-guide/cgroup-v2.html). The `SIGKILL` is not trappable, so the program itself can't report anything - but the kernel 

note: Moreover in some situations, as in recent kubernetes implementations, the user gets kicked out from the job allocation, which makes it even more difficult to debug. [Kubernetes Silent Pod Killer](https://itnext.io/kubernetes-silent-po

[`cgmemtime`](https://github.com/gsauthof/cgmemtime) is a little gem of a C program that uses cgroups v2 to measure the peak CPU memory usage of a process and all of its descendants no matter how many generations follow it.

git clone https://github.com/gsauthof/cgmemtime

Until recently Python couldn't tell us the name of the variable that was passed to a function, hence we need to pass the variable name as a string. Since Python 3.14 there is a way to overcome this, using [template strings (t-strings)](http

There is also the [lovely-tensors](https://github.com/xl0/lovely-tensors/) library that is designed to automate the process of making the tensor dump more useful and provide shapes, statistics, nan/inf counts, tensor memory usage, etc.

Let's use fp16 floating point representation for demonstrating how we end up with Infinity numbers. 65504 is the largest normal floating point number that can be represented in the fp16 precision. This is slightly below `2**16` due to how t

As of `transformers==4.57.1` the above will give you `loss=tensor(nan)`. The issue here is that the special `-100` label masks tokens to be excluded from the loss calculation and in the above example, we have 0 tokens that aren't masked, si

You can see it in context [here](https://github.com/deepspeedai/DeepSpeed/blob/df59f203f40c8a292dd019ae68c9e6c88f107026/deepspeed/runtime/sequence_parallel/ulysses_sp.py#L1184-L1186). Though the code has evolved since then, and you can find

Let's match the report to the code from [`models/t5/modeling_t5.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/modeling_t5.py):

For example, see this [discussion](https://github.com/deepspeedai/DeepSpeed/issues/4932) - the users reported that when doing Llama-2-7b inference they were getting quite different logits depending on how the model was initialized. To clari

This snapshot and the commentary come from this [PyTorch Issue thread](https://github.com/pytorch/pytorch/issues/84936#issuecomment-1246084645).

If you're curious where I pulled this code from - this is a simplified reduction of this original code in [modeling_llama.py](https://github.com/huggingface/transformers/blob/3f69f415adcbdaedec154ba8eac220ef3276975d/src/transformers/models/

In theory enabling this variable should make everything run really slow, but in reality it really depends on your software. We did the whole of BLOOM-176B training using `CUDA_LAUNCH_BLOCKING=1` with [`Megatron-DeepSpeed`](https://github.co

Note: [NCCL==2.14.3 coming with `pytorch==1.13` hangs](https://github.com/NVIDIA/nccl/issues/750) when `CUDA_LAUNCH_BLOCKING=1` is used. So don't use it with that version of pytorch. The issue has been fixed in `nccl>=2.17` which should be 

Primary reference: [NVIDIA Compute Sanitizer](https://docs.nvidia.com/compute-sanitizer/).

Reference: [NVIDIA cuda-gdb](https://docs.nvidia.com/cuda/cuda-gdb/).

The general mechanics - enabling core dumps (`ulimit -c unlimited` and `kernel.core_pattern`), loading a core file into `gdb`, and the `bt` / `bt full` / `thread apply all bt` commands - are covered in [Segmentation fault, core files and gd

If you want to reproduce a disk-full failure on demand rather than wait for one, see [emulating an almost full disk partition](https://github.com/stas00/the-art-of-debugging/tree/master/methodology#emulating-an-almost-full-disk-partition).

- `PID` is the process id of the hanging python process. - `-n` is useful if you want to see stack traces from python extensions written in C, C++, etc., as the program may hang in one of the extensions - you may need to add `sudo` before t

- you may need to add `sudo` before the command - for more details see [this note](https://github.com/benfred/py-spy/blob/master/README.md#when-do-you-need-to-run-as-sudo).

sudo echo 0 > /proc/sys/kernel/yama/ptrace_scope

which will allow you running `py-spy` (and `strace`) without needing `sudo`. Beware of the possible [security implications](https://wiki.ubuntu.com/SecurityTeam/Roadmap/KernelHardening#ptrace_Protection) - but typically if your compute node

sudo sysctl -w kernel.yama.ptrace_scope=0

sudo bash -c "echo 0 > /proc/sys/kernel/yama/ptrace_scope"

PDSH_RCMD_TYPE=ssh pdsh -w nodename-[5,8] 'sudo bash -c "source ~/.pdshrc; pgrep -P $(pgrep -o python) | xargs -I {} py-spy dump --pid {}"'

[Credential-related source omitted]

[Credential-related source omitted]

ds_ssh -f hostfile "source ~/.pdshrc; ps aux | grep python | grep -v grep | grep `whoami` | awk '{print \$2}' | xargs -I {} sudo py-spy dump --pid {} "

Primary reference: [ProcessGroupNCCL environment variables](https://docs.pytorch.org/docs/stable/torch_nccl_environment_variables.html).

[Credential-related source omitted]

sudo sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t

Once you need to use more than one node to scale your training, e.g., if you want to use DDP to train faster, you have to get the nodes to talk to each other, so that communication collectives could send data to each other. This is typicall

telnet localhost:6000 nmap -p 6000 localhost nc -zv localhost 6000

nmap -p 6000 localhost nc -zv localhost 6000 curl -v telnet://localhost:6000

nc -zv localhost 6000

curl -v telnet://localhost:6000

$ sudo ifconfig

$ sudo ifconfig

$ sudo ifconfig

Another popular way is to use [Message Passing Interface (MPI)](https://en.wikipedia.org/wiki/Message_Passing_Interface). There are a few open source implementations of it available.

There could be a variety of reason for this failing, but of the most likely one is when you're on the cloud and the 2 nodes weren't provisioned so that their IB is connected. So your Ethernet inter-node connectivity works, but it's too slow

"""     PDB Subclass for debugging multi-processed code     Suggested in: https://stackoverflow.com/questions/4716533/how-to-attach-debugger-to-a-python-subproccess

PDB Subclass for debugging multi-processed code     Suggested in: https://stackoverflow.com/questions/4716533/how-to-attach-debugger-to-a-python-subproccess     """

Suggested in: https://stackoverflow.com/questions/4716533/how-to-attach-debugger-to-a-python-subproccess

`dist.breakpoint()` and `ForkedPdb` above both require you to plant the breakpoint in the source before the run. Python 3.14 adds the missing case: attaching to a process that is *already running* and that you did not prepare in advance, gi

Primary reference: [Python 3.14 pdb](https://docs.python.org/3.14/library/pdb.html).

[hyperfine](https://github.com/sharkdp/hyperfine) runs a given command multiple times and reports how long it took to run: the mean, the standard deviation, and the fastest and slowest run. It can also compare two or more commands and repor

sudo apt-get install -y hyperfine

$ hyperfine -p 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' 'python -c "import torch"'

- `ht.hpu.Event(enable_timing)` - Intel Gaudi - `torch.xpu.Event(enable_timing)` - Intel dGPUs - Apple MPS has a different API, but we can emulate the same API as can seen in [mamf-finder.py](https://github.com/stas00/ml-engineering/blob/00

- Apple MPS has a different API, but we can emulate the same API as can seen in [mamf-finder.py](https://github.com/stas00/ml-engineering/blob/0099885db36a8f06556efe1faf552518852cb1e0/compute/accelerator/benchmarks/mamf-finder.py#L159).

Here are some excellent articles going into deeper explanations and examples: - [How to Accurately Time CUDA Kernels in Pytorch](https://www.speechmatics.com/company/articles-and-news/timing-operations-in-pytorch)

- [How to Accurately Time CUDA Kernels in Pytorch](https://www.speechmatics.com/company/articles-and-news/timing-operations-in-pytorch)

In [Python Profilers](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#profilers) we have seen how to use general Python profilers to detect where the program is slow. For PyTorch there are specially designed prof

For more options please refer to the [`torch.profiler` doc](https://docs.pytorch.org/docs/stable/profiler.html).

Additionally, here is [an excellent introduction to `torch.profiler` from the HuggingFace team](https://huggingface.co/blog/torch-profiler).

I assume you already have the correct version of torch installed, if not head [here](https://pytorch.org/get-started/locally/).

Now let's use it to profile 2 forward calls using [Liger Kernel](https://github.com/linkedin/Liger-Kernel):

[Credential-related source omitted]

[Credential-related source omitted]

footnote: since the culprit here was `torch.compile`, you could also have spotted the recompilation directly with `TORCH_LOGS=recompiles` (and `TORCH_LOGS=graph_breaks` for eager fallbacks); see [PyTorch compiler troubleshooting](https://do

Executable file

# You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0

# #     http://www.apache.org/licenses/LICENSE-2.0 #

#     http://www.apache.org/licenses/LICENSE-2.0

The full 1TB+ dataset is at https://huggingface.co/datasets/c4.

_URL = "https://cdn-datasets.huggingface.co/nlp/datasets/c4/c4-en-10k.tar.xz"

description=_DESCRIPTION,             features=datasets.Features({"text": datasets.Value("string")}),             homepage="https://huggingface.co/datasets/allenai/c4/",

features=datasets.Features({"text": datasets.Value("string")}),             homepage="https://huggingface.co/datasets/allenai/c4/",             citation=_CITATION,

homepage="https://huggingface.co/datasets/allenai/c4/",

Executable file

rm -rf data

# rm texts*

# prep dataset repo https://huggingface.co/new-dataset => HuggingFaceM4/cm4-synthetic-testing

https://huggingface.co/new-dataset => HuggingFaceM4/cm4-synthetic-testing

git clone https://huggingface.co/datasets/HuggingFaceM4/cm4-synthetic-testing

_DESCRIPTION = """This dataset is designed to be used in testing. It's derived from cm4-10k dataset""" _HOMEPAGE = "https://huggingface.co/datasets/HuggingFaceM4/cm4-synthetic-testing"

_HOMEPAGE = "https://huggingface.co/datasets/HuggingFaceM4/cm4-synthetic-testing"

_URL = "https://huggingface.co/datasets/HuggingFaceM4/cm4-synthetic-testing/resolve/main/data.tar.gz"

#!/usr/bin/env python

# You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0

# #     http://www.apache.org/licenses/LICENSE-2.0 #

#     http://www.apache.org/licenses/LICENSE-2.0

Executable file

list2range('1,2,5-7,10')     [1, 2, 5, 6, 7, 10]     # from https://stackoverflow.com/a/6405711/9201239

[1, 2, 5, 6, 7, 10]     # from https://stackoverflow.com/a/6405711/9201239     """

# from https://stackoverflow.com/a/6405711/9201239

Executable file

# prep dataset repo https://huggingface.co/new-dataset => HuggingFaceM4/general-pmd-synthetic-testing

https://huggingface.co/new-dataset => HuggingFaceM4/general-pmd-synthetic-testing

git clone https://huggingface.co/datasets/HuggingFaceM4/general-pmd-synthetic-testing

rm -rf data

rm image.jpg text.txt

rm -rf data

_DESCRIPTION = """This dataset is designed to be used in testing. It's derived from general-pmd-10k dataset""" _HOMEPAGE = "https://huggingface.co/datasets/HuggingFaceM4/general-pmd-synthetic-testing"

_HOMEPAGE = "https://huggingface.co/datasets/HuggingFaceM4/general-pmd-synthetic-testing"

_URL = "https://huggingface.co/datasets/HuggingFaceM4/general-pmd-synthetic-testing/resolve/main/data.tar.gz"

#!/usr/bin/env python

Executable file

list2range('1,2,5-7,10')     [1, 2, 5, 6, 7, 10]     # from https://stackoverflow.com/a/6405711/9201239

[1, 2, 5, 6, 7, 10]     # from https://stackoverflow.com/a/6405711/9201239     """

# from https://stackoverflow.com/a/6405711/9201239

#!/usr/bin/env python

# You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0

# #     http://www.apache.org/licenses/LICENSE-2.0 #

#     http://www.apache.org/licenses/LICENSE-2.0

# To build: # 1. clone sentencepiece into this dir # git clone https://github.com/google/sentencepiece

# 1. clone sentencepiece into this dir # git clone https://github.com/google/sentencepiece #

# git clone https://github.com/google/sentencepiece

# adapted from https://blog.ceshine.net/post/trim-down-sentencepiece-vocabulary/

Executable file

# You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0

# #     http://www.apache.org/licenses/LICENSE-2.0 #

#     http://www.apache.org/licenses/LICENSE-2.0

title={OpenWebText Corpus},   author={Aaron Gokaslan*, Vanya Cohen*, Ellie Pavlick, Stefanie Tellex},   howpublished{\\url{http://Skylion007.github.io/OpenWebTextCorpus}},

author={Aaron Gokaslan*, Vanya Cohen*, Ellie Pavlick, Stefanie Tellex},   howpublished{\\url{http://Skylion007.github.io/OpenWebTextCorpus}},   year={2019}

howpublished{\\url{http://Skylion007.github.io/OpenWebTextCorpus}},

The full 8M-record dataset is at https://huggingface.co/datasets/openwebtext

_URL = "https://cdn-datasets.huggingface.co/nlp/datasets/openwebtext/openwebtext-10k.tar.xz"

description=_DESCRIPTION,             features=datasets.Features({"text": datasets.Value("string")}),             homepage="https://skylion007.github.io/OpenWebTextCorpus/",

features=datasets.Features({"text": datasets.Value("string")}),             homepage="https://skylion007.github.io/OpenWebTextCorpus/",             citation=_CITATION,

homepage="https://skylion007.github.io/OpenWebTextCorpus/",

Executable file

# You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0

# #     http://www.apache.org/licenses/LICENSE-2.0 #

#     http://www.apache.org/licenses/LICENSE-2.0

address   = {Mannheim},   doi       = {10.14618/ids-pub-9021},   url       = {http://nbn-resolving.de/urn:nbn:de:bsz:mh39-90215},

doi       = {10.14618/ids-pub-9021},   url       = {http://nbn-resolving.de/urn:nbn:de:bsz:mh39-90215},   pages     = {9 -- 16},

url       = {http://nbn-resolving.de/urn:nbn:de:bsz:mh39-90215},

The full 1TB+ dataset is at https://huggingface.co/datasets/oscar.

_URL = "https://cdn-datasets.huggingface.co/nlp/datasets/oscar/oscar-en-10k.tar.xz"

description=_DESCRIPTION,             features=datasets.Features({"text": datasets.Value("string")}),             homepage="https://oscar-corpus.com/",

features=datasets.Features({"text": datasets.Value("string")}),             homepage="https://oscar-corpus.com/",             citation=_CITATION,

homepage="https://oscar-corpus.com/",

git clone https://github.com/huggingface/transformers

So for example if you know that `transformers==4.33.0` was good and `transformers@main` (`HEAD`) is bad, find which commit is corresponding to the tag `4.33.0` by visiting [the releases page](https://github.com/huggingface/transformers/rele

You can then go to `https://github.com/huggingface/transformers/commit/` and append the commit SHA to that url which will take you to the commit, (e.g. `https://github.com/huggingface/transformers/commit/57f44dc4288a3521bd700405ad41e90a4687

#!/usr/bin/env python

[Credential-related source omitted]

Executable file

# You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0

# #     http://www.apache.org/licenses/LICENSE-2.0 #

#     http://www.apache.org/licenses/LICENSE-2.0

<?xml version="1.0" encoding="UTF-8" standalone="no"?> <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" width="1275" height="1650" viewBox="0 0 1

<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" width="1275" height="1650" viewBox="0 0 1275 1650">

See [NVIDIA's inference optimization overview](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/) and [LLM benchmarking metric definitions](https://developer.nvidia.com/blog/llm-benchmarking-fundamental-conc

The [OCP microscaling (MX) formats](https://github.com/openxla/xla/discussions/18085) use one 8-bit E8M0 scale per block of 32 values. Including the shared scale, their packed payload sizes are:

([source](https://developer.nvidia.com/blog/accelerated-inference-for-large-transformer-models-using-nvidia-fastertransformer-and-nvidia-triton-inference-server/))

You can get these dimensions from `config.json` inside the model's folder or from an equivalent file if it's different. e.g. [meta-llama/Meta-Llama-3.1-8B](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B/blob/main/config.json).

[DeepSeek v3](https://arxiv.org/abs/2412.19437) introduced Multi-Latent Attention (MLA) which compresses the Key and Value into a latent vector, which further reduces the KV-cache size.  See section 2.1.1 of the paper for the specific detai

[source](https://arxiv.org/abs/2405.04434)

[SwiftKV](https://arxiv.org/abs/2410.03960) was invented to deal with the common situation of 10:1 ratio of prefill vs decode use-cases, reducing inference computation during prompt processing rather than just compressing memory. By combini

KV cache while saving recomputation has a big negative impact on inference's performance. Here is a quote from [Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference](https://arxiv.org/abs/2403.09636):

([source](https://www.hopsworks.ai/dictionary/llm-temperature))

For any positive temperature, scaling doesn't change the logit rank order, so a pure greedy argmax chooses the same token. It still changes sampling probabilities. With Top-K, the same K candidates remain eligible but their relative samplin

For more on decoding methods, see this [HuggingFace blog](https://huggingface.co/blog/how-to-generate).

There are multiple implementations of this technique. As of 2026-08 vLLM supports four of them as [structured output backends](https://docs.vllm.ai/en/latest/features/structured_outputs/), and picks one automatically unless you override it:

- https://github.com/mlc-ai/xgrammar

- https://github.com/guidance-ai/llguidance

- https://github.com/dottxt-ai/outlines

- https://github.com/noamgat/lm-format-enforcer

Here is a good indepth dive into this subject: [Assisted Generation: a new direction toward low-latency text generation](https://huggingface.co/blog/assisted-generation).

One other much simpler solution for [input-grounded tasks](#input-grounded-tasks), is to use [ngram prompt lookup decoding](https://github.com/apoorvumang/prompt-lookup-decoding). In this approach there is no need for a draft model, instead

One of the solutions is called Fully [Homomorphic Encryption](https://en.wikipedia.org/wiki/Homomorphic_encryption) (FHE).

Have a look at one such implementation, [concrete-ml](https://github.com/zama-ai/concrete-ml) that rewrites the model to be able to have the client run part of the model themselves, then the intermediary encrypted activations are sent to th

There are various other approaches, e.g. this paper: [LLMs Can Understand Encrypted Prompt: Towards Privacy-Computing Friendly Transformers](https://arxiv.org/abs/2305.18396v3) goes into a custom solution based on Secure Multi-Party Computa

According to wiki there are [3 types of reading](https://en.wikipedia.org/wiki/Speed_reading#Types_of_reading) and the reading speed is measured in words per minute (WPM).

If you read benchmarks and run into p50, p75, p90, p95, and p99, these percentiles are values in an ordered distribution. A p95 value is the threshold at or below which 95% of observations fall; the remaining 5% are above it. The desirable 

For example, let's look at partial output of a system loading report generated by [k6](https://github.com/grafana/k6) on an inference server:

Please refer to [Percentile](https://en.wikipedia.org/wiki/Percentile) for a much more indepth explanation.

- vLLM supports the `--load-format` flag, where one could choose options like `npcache` (numpy format caching) or `tensorizer` using  CoreWeave’s [Tensorizer](https://github.com/coreweave/tensorizer).  ([recipe](https://docs.vllm.ai/en/stab

At the moment I mainly use [TTFT](#time-to-first-token) and [decode throughput](#decode-throughput) benchmarks. TTFT measures the client-observed time from sending the request until receiving the first non-empty generated chunk. The inter-t

TTFT is not pure prefill time. It includes request transport, queueing, scheduling, prompt processing, first-token generation, and response transport. Dividing the prompt-token count by TTFT therefore does not produce prefill throughput; th

note: I've discovered that when I use the openAI client it doesn't scale well and with many concurrent requests the openAI client creates a bottleneck and doesn't measure the real server performance - I am yet to figure out if it's an issue

- https://github.com/vllm-project/vllm/blob/main/benchmarks/benchmark_throughput.py - my favorite tool so far

- https://github.com/grafana/k6 - useful for load testing to simulate multiple concurrent clients - uses JavaScript clients.

[vLLM](https://github.com/vllm-project/vllm)

[DeepSpeed-FastGen](https://github.com/deepspeedai/DeepSpeed/tree/master/blogs/deepspeed-fastgen) from [the DeepSpeed team](https://github.com/deepspeedai/DeepSpeed).

[TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) (also integrated what used to be `FasterTransformer`)

[SGLang](https://github.com/sgl-project/sglang)

[llama.cpp](https://github.com/ggml-org/llama.cpp) - local / single-box serving (CPU, CUDA including datacenter GPUs, Metal, …); not a multi-tenant production peer to vLLM/SGLang.

[LightLLM](https://github.com/ModelTC/lightllm)

[LMDeploy](https://github.com/InternLM/lmdeploy)

[MLC-LLM](https://github.com/mlc-ai/mlc-llm)

- **Intel Gaudi (HPU)**: the [vllm-gaudi](https://github.com/vllm-project/vllm-gaudi) plugin (the older [HabanaAI/vllm-fork](https://github.com/HabanaAI/vllm-fork) is being retired), plus [Optimum for Intel Gaudi](https://github.com/hugging

- **AWS Trainium/Inferentia**: the [AWS Neuron SDK](https://github.com/aws-neuron/aws-neuron-sdk), which serves through standard vLLM APIs and [Optimum Neuron](https://github.com/huggingface/optimum-neuron).

- **Google TPU**: [tpu-inference](https://github.com/vllm-project/tpu-inference) - the vLLM TPU plugin unifying JAX and PyTorch, and the successor to the now-archived [JetStream](https://github.com/AI-Hypercomputer/JetStream); [MaxText](htt

- **Modular MAX**: [MAX](https://github.com/modular/modular) - a portable serving stack that also runs on CPUs and NVIDIA/AMD GPUs.

- **Apple Silicon**: [MLX](https://github.com/ml-explore/mlx) / [mlx-lm](https://github.com/ml-explore/mlx-lm), plus llama.cpp's Metal backend.

- **Tenstorrent**: [tt-metal](https://github.com/tenstorrent/tt-metal) and its vLLM integration.

Finally, some vendors expose inference only as a hosted service on their own hardware, rather than as an installable framework - e.g. [Cerebras Inference](https://www.cerebras.ai/inference), Groq and [SambaNova](https://sambanova.ai/).

For example, here is a snapshot of [vLLM](https://github.com/vllm-project/vllm)'s stats as of 2024-08-24, which is one of the most popular inference frameworks as of 2026-08.

- [Groq](https://groq.com/)

- [A Survey on Efficient Inference for Large Language Models (2024)](https://arxiv.org/abs/2404.14294)

- Since ML does a lot of parallel processing ([SIMD](https://en.wikipedia.org/wiki/Single_instruction,_multiple_data)) GPUs were used at the beginning, but now you additionally have TPUs, FPGAs, HPUs, QPUs, RDUs, etc. Recent CPUs are becomi

- The [Dot-com bubble](https://en.wikipedia.org/wiki/Dot-com_bubble) occurred during 1995-2000. And a very similar situation is happening right now in the AI space.

Pros: - Managed for you - all you need is your software to do the training and a bit of [SLURM](../orchestration/slurm) know-how to launch jobs - Often sponsored by the local government/university - probably could get the job done for less 

- Often sponsored by the local government/university - probably could get the job done for less $$ or even free (e.g. we trained [BLOOM-176B](https://huggingface.co/bigscience/bloom) for free on [JeanZay HPC](http://www.idris.fr/en/)!)

Imagine a steam locomotive - the engine is great, but if the [fireman](https://en.wikipedia.org/wiki/Fireman_(steam_engine)) isn't fast enough to shovel the coal in, the train won't move fast.

[source](https://commons.wikimedia.org/wiki/File:Baureihe52Heizer.jpg)

- Accelerators get ~2x faster every 2 years ([Moore's law](https://en.wikipedia.org/wiki/Moore%27s_law))

- If a single iteration requires 624 Tera floating point operations and it took 4 secs to run then we know that we get: `624/4=156` actual TFLOPS - now BF16@A100 is [advertised as 312TFLOPS](https://www.nvidia.com/en-us/data-center/a100/) s

- now BF16@A100 is [advertised as 312TFLOPS](https://www.nvidia.com/en-us/data-center/a100/) so `156/312=0.5` gives us 50% MFU.

- There is not much that can be done about the accelerator memory since its bandwidth is what it is - one can only write more efficient software to make data move faster to/from the accelerator - hint: fused and custom written kernels (like

- NVIDIA GPUs: all based on [CUDA](https://developer.nvidia.com/cuda/toolkit), which most training frameworks support. You can easily moved between different NVIDIA GPUs and most things would work the same.

- AMD MI250/MI3xx: with PyTorch using [ROCm](https://pytorch.org/blog/pytorch-for-amd-rocm-platform-now-available-as-python-package/) you can run most CUDA-based software as is. This is really the only inter-operable accelerator with the NV

- Intel Gaudi2/Gaudi3: if you use HF Transformers/Diffusers you can use [optimum-habana](https://github.com/huggingface/optimum-habana). If you use HF Trainer with NVIDIA GPUs it should be relatively easy to switch to train/infer on Gaudi2.

- Google TPU and AWS Trainium: both via [XLA](https://openxla.org/xla) - PyTorch works, custom CUDA kernels don't

- Cerebras: is also working on PyTorch support via [Cerebras Software Platform (CSoft) via XLA](https://www.cerebras.ai/blog/supporting-pytorch-on-the-cerebras-wafer-scale-engine/).

Also in general most ML code could be compiled into cross-platform formats like [Open Neural Network Exchange (ONNX)](https://en.wikipedia.org/wiki/Open_Neural_Network_Exchange) which can be run on a variety of accelerators. This approach i

Learning how to [debug efficiently](https://github.com/stas00/the-art-of-debugging/tree/master/methodology) should also make this process much less painful.

As a continuation of a previous section, a [Service Level Agreement](https://en.wikipedia.org/wiki/Service-level_agreement) (SLA) is an agreement between a service providers and a customer that define various guarantees and expectations wit

The other term is Service Level Objective (SLO) where SLA is quantified. For example, an SLO may define a Monthly Uptime Percentage to 99.5%, if the uptime is less than 99.5% the provider credits the customer to a certain percentage of the 

I'm yet to find a golden reference point, but, for example,  [SemiAnalysis](https://newsletter.semianalysis.com/p/ai-neocloud-playbook-and-anatomy#%C2%A7cluster-deployment-and-acceptance-test) suggests that OEM provider performs a 3-4 weeks

But, of course, only you know what your workloads' specific requirements are. Also consider the relationship between local storage and remote (shared) storage, as some providers will reduce the size and performance of local drives to save m

In general, CSPs follow NVIDIA's [DGX SuperPOD Reference Architecture](https://docs.nvidia.com/dgx-superpod/reference-architecture-scalable-infrastructure-h100/latest/abstract.html) which provides a lot of detail on how to build a rail-opti

However, many of the largest GPU clusters in the world now run RoCEv2 instead of InfiniBand. Meta has [proven](https://engineering.fb.com/2024/08/05/data-center-engineering/roce-network-distributed-ai-training-at-scale/) that you can train 

- semianalysis.com created a ClusterMax CSP rating system and includes excellent explanations of the different criteria and plans to continue ranking many CSPs. [2025](https://newsletter.semianalysis.com/p/the-gpu-cloud-clustermax-rating-sy

#!/usr/bin/env python3

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

#!/usr/bin/env python3

#!/usr/bin/env python3

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

#!/usr/bin/env bash

DEEPSPEED=0.19.2 LIGER=0.8.0 CU130=https://download.pytorch.org/whl/cu130

CU130=https://download.pytorch.org/whl/cu130

sudo update-alternatives --set cuda /usr/local/cuda-13.0 2>/dev/null || true

echo "== installing torch==$TORCH (cu130) ==" uv pip install --upgrade "torch==$TORCH" --index-url "$CU130"

uv pip install --upgrade "torch==$TORCH" --index-url "$CU130"

rm -rf ~/.cache/torch_extensions/* 2>/dev/null || true

uv pip install --no-deps flash-attn-3 --index-url "$CU130"

#!/usr/bin/env python3

#!/usr/bin/env bash

- algbw: algorithm bandwidth - payload size divided by elapsed time - Bi-directional, Duplex: a transmission from one point to another in both directions A <-> B, typically 2x the one-direction (unidirectional) rate - Bisection Width: minim

- Bi-directional, Duplex: a transmission from one point to another in both directions A <-> B, typically 2x the one-direction (unidirectional) rate - Bisection Width: minimum number of links cut to divide the network into two parts (not nec

- Bisection Width: minimum number of links cut to divide the network into two parts (not necessarily equal). The bandwidth of those links is known as Bisection Bandwidth - which is often used as a metric for real network bandwidth). Sometim

Here is a [good overview article](https://www.fs.com/blog/a-quick-look-at-the-differences-rdma-vs-tcpip-2945.html).

2. AWS publishes `NeuronLink-v3 ... 1.28 TB/sec bandwidth per chip` for Trainium2 without declaring directionality, so it's halved here per the directionality note below. AWS doesn't publish a per-link rate or a link count, so Trainium2 can

3. PCIe 6 is listed under `Announced, availability not confirmed` because no shipping accelerator attaches at Gen6 as of 2026-07-31 - current parts are Gen5 x16 (NVIDIA H200 SXM lists `PCIe Gen5`, AMD MI350X/MI355X list `PCIe 5.0 x16`), so 

4. Google publishes a "Bidirectional inter-chip interconnect (ICI) bandwidth per chip (GBps)" of 1200 for TPU7x, halved here per the note above. Peer-to-peer is the per-axis figure, "bi-directional bandwidth of 200 GBps per axis" - and 6 ne

[PCIe](https://en.wikipedia.org/wiki/PCI_Express) is a high-speed serial computer expansion bus standard that can be found even on the cheapest computer desktop.

footnote: a released specification is not shipping silicon. As of 2026-07-30 [PCI-SIG](https://pcisig.com/specifications) lists `PCI Express Base Specification Revision 7.0` as released, while PCIe 8.0 exists only as a members-only draft, t

- [NVLink](https://en.wikipedia.org/wiki/NVLink) is a wire-based serial multi-lane near-range communications link developed by NVIDIA. Here is the [What Is NVLink](https://blogs.nvidia.com/blog/what-is-nvidia-nvlink/) blog post with more ba

There is a good overview of evolution of NVLink (1 to 4) [here](https://www.naddod.com/blog/unveiling-the-evolution-of-nvlink).

As of 2026-08 there is no public spec of the speed, but I found 450GBps unidirectional mentioned [here](https://newsletter.semianalysis.com/p/gb200-hardware-architecture-and-component#%C2%A7the-4-rack-scale-form-factors-of-blackwell) for GB

[NVSwitch](https://www.nvidia.com/en-us/data-center/nvlink/) can connect more than 8 GPUs at the speed of [NVLink](#nvlink). It's advertised to connect up to 256 GPUs in the future generations of the switch.

The [NVIDIA DGX H100](https://web.archive.org/web/20231208180425/https://developer.nvidia.com/blog/upgrading-multi-gpu-interconnectivity-with-the-third-generation-nvidia-nvswitch/) connects eight H100 GPUs through four third-generation NVSw

A [DGX H100 SuperPOD](https://docs.nvidia.com/dgx-superpod/reference-architecture-scalable-infrastructure-h100/latest/index.html) scalable unit contains 32 DGX H100 systems (256 GPUs), but each DGX remains its own in-node NVSwitch domain. T

Recently [GB200 NVL72](https://www.nvidia.com/en-us/data-center/gb200-nvl72/) has been introduced, which uses NVSwitch to put 72 Blackwell GPUs into a single node all inter-connected at NVLink 5 900GBps unidirectional speed. So instead of h

Platform specs: - [MI250X](https://www.amd.com/en/products/accelerators/instinct/mi200/mi250x.html)

- [MI250X](https://www.amd.com/en/products/accelerators/instinct/mi200/mi250x.html)

- [MI300x](https://www.amd.com/en/products/accelerators/instinct/mi300/platform.html)

- [MI325X](https://www.amd.com/en/products/accelerators/instinct/mi300/mi325x.html)

- [MI350X](https://www.amd.com/en/products/accelerators/instinct/mi350/mi350x.html)

- [MI355X](https://www.amd.com/en/products/accelerators/instinct/mi350/mi355x.html)

According to [Gaudi2 spec](https://habana.ai/wp-content/uploads/2023/10/HLS-Gaudi2_Datasheet_10_23.pdf), these nodes provide the same 100GbE RoCE v2 RDMA hardware for inter- and intra-node connectivity (24x 100Gbps per card).

According to [Gaudi3 spec](https://www.intel.com/content/www/us/en/content-details/817486/intel-gaudi-3-ai-accelerator-white-paper.html), these nodes provide the same setup as Gaudi2 except the cards are 2x faster using 200GbE RoCE v2 RDMA 

NeuronLink v3 ([spec](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trn2-arch.html)) is the intra-node equivalent of NVLink for AWS Trainium2, but it's a point-to-point architecture, like AMD MI* 

1. [NVIDIA DGX B300 specifications](https://www.nvidia.com/en-us/data-center/dgx-b300/)

2. [AWS EC2 accelerated-computing network specifications](https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html)

3. [Intel Gaudi3 white paper](https://www.intel.com/content/www/us/en/content-details/817486/intel-gaudi-3-ai-accelerator-white-paper.html)

4. [NVIDIA DGX H100/H200 user guide](https://docs.nvidia.com/dgx/dgxh100-user-guide/introduction-to-dgxh100.html)

5. [Intel Gaudi2 data sheet](https://habana.ai/wp-content/uploads/2023/10/HLS-Gaudi2_Datasheet_10_23.pdf)

6. [Google Cloud GPUDirect-TCPX/TCPXO](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/gpu-bandwidth-gpudirect-tcpx)

7. [HPE Slingshot 11 QuickSpecs](https://www.hpe.com/us/en/collaterals/collateral.a50002546enw.html)

8. [Cornelis CN-100HFA specifications](https://www.cornelis.com/product/cornelis-omni-path-accelerated-host-fabric-adapter-cn-100hfa)

9. [InfiniBand Trade Association XDR specification release](https://infinibandta.org/ibta-unveils-xdr-infiniband-specification-to-enable-the-next-generation-of-ai-and-scientific-computing/)

10. [NVIDIA NDR cabling guide](https://docs.nvidia.com/dgx-superpod/design-guide-cabling-data-centers/latest/ndr-overview.html)

11. 2 ports of 4 XDR lanes each (4x200Gbps = 800Gbps per port); a node with this layout is illustrative, not a documented product. 12. 1 port of 4 NDR lanes (4x100Gbps = 400Gbps). 13. Cornelis [CN6000 product page](https://www.cornelis.com/

12. 1 port of 4 NDR lanes (4x100Gbps = 400Gbps). 13. Cornelis [CN6000 product page](https://www.cornelis.com/products/cn6000?product_range=supernics) and [AMD MI400 reference architecture](https://www.cornelis.com/stories/cornelis-announces

13. Cornelis [CN6000 product page](https://www.cornelis.com/products/cn6000?product_range=supernics) and [AMD MI400 reference architecture](https://www.cornelis.com/stories/cornelis-announces-new-reference-architecture-for-ai-inference-trai

14. 800Gbps product sampling in 2026; GA target Q4-2026; eight-adapter node remains illustrative. 15. [InfiniBand Trade Association roadmap](https://infinibandta.org/infiniband-roadmap/) 16. GDR is on the InfiniBand Trade Association roadma

15. [InfiniBand Trade Association roadmap](https://infinibandta.org/infiniband-roadmap/)

16. GDR is on the InfiniBand Trade Association roadmap; no GDR product has shipped. The 2-port node is illustrative. 17. [Cornelis CN5000 launch](https://www.cornelis.com/stories/cornelis-launches-cn5000-industry-leading-ai-and-hpc-scale-ou

17. [Cornelis CN5000 launch](https://www.cornelis.com/stories/cornelis-launches-cn5000-industry-leading-ai-and-hpc-scale-out-network)

29. Google publishes a higher headline number for these machine types - 1800Gbps max network bandwidth for A3 Mega and 1000Gbps for A3 High. Those totals include the VPC/frontend NIC; the figures above count only the accelerator fabric, so 

31. The `v1`-`v4` EFA generation labels are AWS's own, taken from the [supported instance types](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html#efa-instance-types) tables, which group instances under `Nitro v3 (EFA v1)` throug

1. [NVIDIA InfiniBand adapters](https://www.nvidia.com/en-us/networking/infiniband-adapters/) - NVIDIA states "up to 1.6 terabits per second (Tb/s) throughput" for ConnectX-9, "up to 800 gigabits per second (Gb/s) of data throughput" for Co

2. [AMD Pensando networking](https://www.amd.com/en/products/network-interface-cards/pensando.html) - "up to 800 Gbps of Ethernet connectivity" for Vulcano and "up to 400 Gbps Ethernet speeds" for Pollara 400.

6. AMD advertises "up to 2.4 Tbps of scale-out bandwidth per GPU" for Vulcano. That's three 800Gbps NICs attached to one GPU - a platform configuration, not a single faster NIC. Same trap as the node-aggregate columns elsewhere in this chap

7. These two sit under `Announced, availability not confirmed` because the vendor page names the product but states no availability, not because the vendor says it is unreleased. NVIDIA lists ConnectX-9 in its adapter portfolio without a sh

8. [Cornelis CN6000 SuperNIC adapters](https://www.cornelis.com/product/cornelis-cn6000-omni-path-adapters) - Bandwidth `800G`; Performance `1.6Tbps bidirectional (800Gbps Tx and 800Gbps Rx)`; Dual QSFP-DD (`x4` @ 112 Gbps/lane); PCIe 6.0 x

[InfiniBand](https://en.wikipedia.org/wiki/InfiniBand) (IB) has been around for a few decades so there are many available configurations that can be found out there. So that if someone says they have InfiniBand that is insufficient informat

1. [InfiniBand Trade Association XDR specification release](https://infinibandta.org/ibta-unveils-xdr-infiniband-specification-to-enable-the-next-generation-of-ai-and-scientific-computing/)

2. [NVIDIA NDR cabling guide](https://docs.nvidia.com/dgx-superpod/design-guide-cabling-data-centers/latest/ndr-overview.html)

3. [InfiniBand Trade Association roadmap](https://infinibandta.org/infiniband-roadmap/)

1. [InfiniBand Trade Association roadmap](https://infinibandta.org/infiniband-roadmap/)

2. [InfiniBand Trade Association historical 8X and 12X roadmap](https://infinibandta.org/infiniband-trade-association-ibta-announces-updated-infiniband-roadmap-projecting-data-speeds-of-104gb-s-per-4x-port-in-2011/)

InfiniBand provides [RDMA](https://en.wikipedia.org/wiki/Remote_direct_memory_access).

- One configuration of NVIDIA DGX H100 comes with 8x NVIDIA ConnectX-7 (CX7) Ethernet/InfiniBand ports each of 400Gbps, for a total of 3.2Tbps (0.4TBps) unidirectional to connect with other DGX servers. - For DGX H100 SuperPOD the ConnectX-

- For DGX H100 SuperPOD the ConnectX-7s across all 32 DGX servers and associated InfiniBand switches provide 12.8TBps of unidirectional bandwidth (25.6TBps counting both directions) for use within the pod or for scaling out to multiple Supe

- NVIDIA DGX GB200 NVL72 compute trays provide four single-port ConnectX-7 interfaces at up to 400Gbps each, corresponding to one 400Gbps connection per GPU ([networking documentation](https://docs.nvidia.com/dgx/dgxgb200-user-guide/network

[InfiniBand](https://en.wikipedia.org/wiki/InfiniBand) used to have multiple manufacturers, but at the moment it's just NVIDIA (purchased Mellanox) - which makes it a single-vendor interconnect, a lock-in consideration when weighing it agai

Practical links: - [InfiniBand Utilities](https://networking-docs.nvidia.com/mlnxofedswum/24.10-5.1.6.1lts/infiniband-fabric-utilities) (the link could be outdated as it's versioned) - these are useful when debugging an IB setup.

- [InfiniBand Utilities](https://networking-docs.nvidia.com/mlnxofedswum/24.10-5.1.6.1lts/infiniband-fabric-utilities) (the link could be outdated as it's versioned) - these are useful when debugging an IB setup.

[NVIDIA Quantum-2 InfiniBand Platform](https://www.nvidia.com/en-us/networking/quantum2/) is the switch side of NDR, pairing with the ConnectX-7 adapters in [Network adapters](#network-adapters). It supports 400Gbps per port, provides RDMA,

[NVIDIA Quantum-X800 InfiniBand Platform](https://www.nvidia.com/en-us/networking/products/infiniband/quantum-x800/) is the switch side of XDR, pairing with ConnectX-8, and supports 800Gbps per port and includes in-network computing with [S

[Spectrum-X](https://www.nvidia.com/en-us/networking/spectrumx/) is NVIDIA's Ethernet answer to its own InfiniBand line, for shops that want RoCE rather than IB. NVIDIA describes it as "the tight coupling of the NVIDIA Spectrum-X Ethernet s

Source: [NVIDIA Ethernet switching](https://www.nvidia.com/en-us/networking/ethernet-switching/), as of 2026-07-30. NVIDIA does not state per-model availability there, so `GA` is `?` for the Spectrum-6 SN6000 systems - the newer co-packaged

Note that NVIDIA does not attach a Quantum product name to the InfiniBand version on its [InfiniBand page](https://www.nvidia.com/en-us/networking/products/infiniband/) - only the generic photonics framing - so there is no generation to put

[Elastic Fabric Adapter (EFA)](https://aws.amazon.com/hpc/efa/) is a recent inter-node networking technology created by AWS.

The generation labels above are AWS's own: its [supported instance types](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html#efa-instance-types) page groups every EFA-capable instance under a `Nitro v3 (EFA v1)` through `Nitro v6 

According to [Gaudi2 spec](https://habana.ai/wp-content/uploads/2023/10/HLS-Gaudi2_Datasheet_10_23.pdf), these nodes provide `3*8=24` NICs of 100GbE RoCE v2 RDMA for a total of 2.4Tbps of inter-node connectivity with other Gaudi2 nodes.

According to [Gaudi3 spec](https://www.intel.com/content/www/us/en/content-details/817486/intel-gaudi-3-ai-accelerator-white-paper.html), these nodes provide `3*8=24` NICs of 200GbE RoCE v2 RDMA for a total of 4.8Tbps of inter-node connecti

[HPE Slingshot interconnect](https://www.hpe.com/ca/en/compute/hpc/slingshot-interconnect.html) seems to be used by HPCs. As of 2026-08 it provides 200Gbps per link. Some HPCs use 4 of those links to build 800Gbps interconnects, and, of cou

GPUDirect-TCPX is an RDMA-over-TCP software solution developed by Google for A3 instances. GPUDirect-TCPXO is a hardware-accelerated extension of TCPX available only on A3 Mega. The docs are scarce, but here is some information on [TCPX](ht

[Omni-Path Architecture](https://en.wikipedia.org/wiki/Omni-Path) (OPA). Originally by Intel, the technology got sold to Cornelis Networks. It's also known as Omni-Path Express (OPX).

case study: I used this technology at JeanZay HPC in France in 2022. It was only 135Gbps and while the vendor tried to fix it a year later it was still the same speed. Hopefully the issue has been resolved and the speed is much faster nowad

[Cornelis Omni-Path Accelerated Host Fabric Adapter CN-100HFA](https://www.cornelis.com/product/cornelis-omni-path-accelerated-host-fabric-adapter-cn-100hfa) 100Gbps NICs have been around for many years now - and until 2025 this was the onl

[CN5000](https://www.cornelis.com/product/cornelis-cn5000-omni-path-adapters) 400Gbps NICs began shipping in June 2025 and have been broadly available since Q3-2025 - see note 18 under the [inter-node networking](#inter-node-networking) tab

footnote: Cornelis's marketing pages sometimes say "400Gbps bidirectional", but the [product-family docs](http://docs.cornelis.com/en/cn5000-product-family-descriptions/fabric-hardware-components/cn5000-supernic.html) state "400 Gbps (4 x 1

[CN6000](https://www.cornelis.com/product/cornelis-cn6000-omni-path-adapters) is the next generation - an 800Gbps SuperNIC on PCIe 6.0 that runs Omni-Path, RoCEv2, and Ultra Ethernet concurrently (each of its dual QSFP-DD ports independentl

Omni-Path provides [RDMA](https://en.wikipedia.org/wiki/Remote_direct_memory_access).

[The UALink initiative](https://ualinkconsortium.org/) is an attempt to create an open standard to compete with [NVLink](#nvlink). Supposedly it'll be based on AMD's [Infinity Fabric](#infinity-fabric--xgmi). The consortium published the `U

caveat on sourcing: use Huawei's Chinese pages, not the English ones. Both are JavaScript applications, but the Chinese product pages inline the full specification table while the English equivalents inline almost nothing - the English Atla

NVIDIA [Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)](https://networking-docs.nvidia.com/sharpum/300) - allows performing data reductions and aggregations on the network itself (in-network computing). This is very useful

To understand the importance of this technology - for `all-reduce` operations, instead of 2N sends, it will only need N+1 sends - so for a large N - it almost doubles the effective all-reduce throughput. (N is the number of communicating ra

case study: I discovered SHARP accidentally when an H100 intra-node NVLink 4.0 [torch-collective-bench](benchmarks/torch-dist-bench.py) benchmark reported 480GBps for a 4GiB payload when the theoretical spec was only 450GBps! We figured out

NVIDIA's own documentation draws a 4-GPU line, though it takes one step to see why: NVLink SHARP is implemented with multicast - the switch reduces and then fans the result back out to every participant - which is why NCCL's setup log for `

In the case of NVL36, NVL72 and others bigger than NVL8 the granularity is likely 4 GPUs rather than 8, since a compute tray holds 2 GB200 modules and each module is 1 Grace CPU + 2 Blackwell GPUs. The [NVIDIA GB200 NVL Partition User Guide

[source](https://www.nvidia.com/en-us/on-demand/session/gtc25-s72583/)

If the experimental model still contains 2B params like in the previous section and grads are in fp32 then the training program needs to send 8GB (`2B * 4B`) of data on every iteration. Moreover, since syncing the gradients requires an [`al

([source](https://docs.pytorch.org/tutorials/intermediate/dist_tuto.html#collective-communication))

oh and this whole synchronization protocol is called DDP ([DistributedDataParallel](https://docs.pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html)) in the PyTorch lingo.

For example, when you read, the [A100 spec](https://www.nvidia.com/en-us/data-center/a100/#specifications) the TFLOPS there means TeraFloatingPointOperations per second.

Also see the [wiki page](https://en.wikipedia.org/wiki/FLOPS) for more clarifications.

For GPT-family of decoder transformers models we can use the math described in this [BLOOM-176 docs](https://github.com/bigscience-workshop/bigscience/tree/master/math#calculate-tflops):

So if we do a mixed half-precision training and most of the operations are done in half-precision then we can roughly say that we do [312TFLOPS on A100](https://www.nvidia.com/en-us/data-center/a100/#specifications) and usually a well optim

footnote: It's a ~3x [989TFLOPS on H100](https://www.nvidia.com/en-us/data-center/h100) (scroll to the end) and also it shows a misleading 2x numbers for sparsity so you have to mentally divide it by 2.

This benchmark run an `all_reduce` collective for various payload sizes from 32KiB to 16GiB. The value that we care about is the `busbw` - this column tells us the real network throughput as explained [here](https://github.com/NVIDIA/nccl-t

note: [NVIDIA/nvbandwidth](https://github.com/NVIDIA/nvbandwidth) is supposed to be a more detailed and correct benchmark to replace `p2pBandwidthLatencyTest`, but I found the latter to provide very similar results.

![nccl all-reduce scan benchmark](images/nccl-all-reduce-scan.png) ([source](https://arxiv.org/abs/2411.13055))

([source](https://arxiv.org/abs/2411.13055))

![nccl all-reduce nvlstree scan benchmark](images/nccl-all-reduce-scan-nvlstree.png) [source](https://www.nvidia.com/en-us/on-demand/session/gtc24-s62129/)

[source](https://www.nvidia.com/en-us/on-demand/session/gtc24-s62129/)

![Low-level Uni-directional Bandwidth Measurements](images/ccgrid11-uni-direction-bandwidth.png) ([source](https://ieeexplore.ieee.org/document/5238655))

([source](https://ieeexplore.ieee.org/document/5238655))

[source](https://www.nvidia.com/en-us/on-demand/session/gtc25-s72583/)

Another tool for bandwidth measurements on NVIDIA GPUs is [NVIDIA/nvbandwidth](https://github.com/NVIDIA/nvbandwidth).

Normally NCCL passes each payload through its own internal staging buffers, since it knows nothing about where the peers keep their data. If instead every rank's buffer sits at the same offset of a registered NCCL symmetric memory window, e

A collective can't tell you this. An `all-reduce` across nodes will always lean on the intra-node fabric, which is the whole point of this section. To measure the wire itself, use a point-to-point RDMA benchmark. [`ib_write_bw`](https://man

Fabrics with their own userspace stack rather than verbs - [Slingshot](#hpe-slingshot-interconnect), [Omni-Path](#omni-path) via OPX, [GPUDirect-TCPX](#gpudirect-tcpx) - need their own tools. The nearest equivalent there is `fi_rma_bw` from

footnote: RDMA-write-over-[SRD](#glossary-and-concepts) support was contributed to `perftest` by AWS ([PR 206](https://github.com/linux-rdma/perftest/pull/206)). Confirmed on a live instance on 2026-08-07: on an 8x H200 `p5en.48xlarge` node

footnote: it is tempting to instead keep all the accelerators and take the intra-node fabric away with [`NCCL_P2P_DISABLE=1`](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html), but that only pushes intra-node traffic down 

![Low-level Latency Measurements](images/ccgrid11-low-level-latency.png) ([source](https://ieeexplore.ieee.org/document/5238655))

([source](https://ieeexplore.ieee.org/document/5238655))

- **It gets the only tree.** NCCL implements its tree algorithm for all-reduce only; the other collectives use rings. A NCCL maintainer models the tree's latency as `2*(G-1)*l + 2*log2(N)*L`, which grows with the log of the node count, and 

- **The other collectives lack an equivalent.** NCCL 2.23 added the PAT algorithm for all-gather and reduce-scatter, but only for one GPU per node, and a maintainer says it's "not yet as low-latency as the Tree allreduce" ([NCCL #1473](http

Proprietary network hardware vendors like AWS (EFA) don't disclose their secrets and therefore the public libraries like [nccl](https://github.com/NVIDIA/nccl) cannot support those out of the box. These vendors have to supply their own vers

Originally proprietary hardware vendors used the trick of telling the users to use `LD_LIBRARY_PATH` and/or `LD_PRELOAD` to dynamically overload `libnccl.so` to get their custom version loaded into PyTorch or another framework. But recently

On EFA that plugin is [aws-ofi-nccl](https://github.com/aws/aws-ofi-nccl). NCCL has no transport for that NIC, so it cannot run over EFA without the plugin.

For more information about dynamic library loading see [this section](https://github.com/stas00/the-art-of-debugging/tree/master/compiled-programs#shared-libraries-ldsoconf-nm-unresolved-symbols-ldd-ld_library_path-ld_preload).

[MSCCL++](https://github.com/microsoft/mscclpp) implements the NCCL collective API in `libmscclpp_nccl.so`, which takes the place of `libnccl.so`. The application keeps its NCCL calls, and the collective runs an MSCCL++ schedule on the GPU.

[UCCL-collective](https://github.com/uccl-project/uccl), also called UCCL-Tran, is a [NCCL Net Plugin](#proprietary-network-hardware-and-nccl). NCCL still chooses the algorithm. The plugin replaces the transport: it sprays packets over many

[DeepEP](https://github.com/deepseek-ai/DeepEP) is DeepSeek's library for [expert-parallel](../training/model-parallelism/README.md#expert-parallelism) dispatch and combine. A call moves the hidden states of the tokens a rank holds, and onl

- Azure: [availability set](https://learn.microsoft.com/en-us/azure/virtual-machines/availability-set-overview?source=recommendations)

- AWS: [cluster placement groups](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html)

- GCP: [compact placement policies](https://docs.cloud.google.com/compute/docs/instances/use-compact-placement-policies)

Read the normalized `busbw` column rather than `algbw` - [PERFORMANCE.md](https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md#bandwidth) explains why. Like `nccl-tests`, the benchmark measures a unidirectional bandwidth, so 

![all-reduce multi node bandwidth](images/all-reduce-multi-node-bandwidth.png) [source](https://www.nvidia.com/en-us/on-demand/session/gtc24-s62129/)

[source](https://www.nvidia.com/en-us/on-demand/session/gtc24-s62129/)

If you're benchmarking a different collective the expected bandwidth can be very different from the above all-reduce results. [This presentation](https://www.nvidia.com/en-us/on-demand/session/gtc24-s62129/) also gives point-to-point commun

- Each call's time is the time from the end of the previous queued call to its own end, as in `nccl-tests`' per-iteration timing (`-I 1`), which it reports in its `i_*` columns, and as the median in its `-J` JSON output. The first call of a

- A call isn't done until it's done on every rank, so each call's time is that of its slowest rank, as in `nccl-tests`' per-iteration median and p99. Its `time` column is instead each rank's time per call averaged over the ranks, which is c

- Small payloads get up to 1000 calls each, so that the p99 is the 10th slowest of them, and larger payloads fewer, once 1000 calls would move more than 40MiB - [Intel MPI Benchmarks](https://github.com/intel/mpi-benchmarks) counts its repe

The payload, `algbw` and `busbw` follow `nccl-tests`, whose [PERFORMANCE.md](https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md) explains them. The payload is the larger of a rank's input and output buffers, and `busbw` is 

- [nccl-tests](https://github.com/NVIDIA/nccl-tests) - NVIDIA's benchmark of every NCCL collective through the NCCL C API; `time` is the mean per call, `-I 1` adds per-call min, max and p99. One 32MiB payload by default, while its README sw

- [rccl-tests](https://github.com/ROCm/rccl-tests) - the same benchmark for AMD's RCCL, with the same flags and columns.

- [OSU Micro-Benchmarks](https://mvapich.cse.ohio-state.edu/benchmarks/) - the MPI benchmarks from the MVAPICH team; `osu_allreduce` and its siblings report the average, min and max latency per call, 1B to 1MiB by default.

- [Intel MPI Benchmarks](https://github.com/intel/mpi-benchmarks) - `IMB-MPI1 Allreduce` and the other collectives report `t_min`, `t_max` and `t_avg` per call, 0B to 4MiB by default.

- [PARAM](https://github.com/facebookresearch/param) - Meta's benchmarks, whose `train/comms/pt/comms.py` times collectives through `torch.distributed` like this benchmark does and reports p50, p75 and p95 latency, 8B to 64B by default.

- [UCC perftest](https://github.com/openucx/ucc) - `ucc_perftest` times collectives through UCC, which can run over NCCL, and reports the average, min and max latency; one 128-element payload by default.

- [MSCCL++](https://github.com/microsoft/mscclpp) - its `python/mscclpp_benchmark/allreduce_bench.py` times MSCCL++'s all-reduce kernels against NCCL's from 4KiB to 1GiB. See [Other collective libraries](../README.md#other-collective-librar

- [DeepEP](https://github.com/deepseek-ai/DeepEP) - DeepSeek's MoE expert-parallel all-to-all library, whose tests time its dispatch and combine kernels at realistic token counts rather than sweeping payload sizes. See [Other collective lib

[NVIDIA/nccl-tests](https://github.com/NVIDIA/nccl-tests) benchmarks collectives - `all-reduce`, `all-gather`, `reduce-scatter` and the rest - and reports the same `busbw`/`algbw` columns as [torch-dist-bench.py](torch-dist-bench.py).

git clone https://github.com/NVIDIA/nccl-tests

[NVIDIA/nvbandwidth](https://github.com/NVIDIA/nvbandwidth) measures point-to-point bandwidth between hosts and accelerators - the closest thing to a direct reading of a single link, as opposed to a collective's aggregate:

git clone https://github.com/NVIDIA/nvbandwidth

[p2pBandwidthLatencyTest](https://github.com/NVIDIA/cuda-samples/tree/master/cpp/5_Domain_Specific/p2pBandwidthLatencyTest) from CUDA samples is a low-level accelerator-to-accelerator benchmark:

git clone https://github.com/NVIDIA/cuda-samples/

For example, you need to find a way to prevent the network usage from fluctuations. When we were doing performance optimizations for [108B pre-BLOOM experiments](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr8-104B-w

Note: The [EAI cookbook](https://github.com/EleutherAI/cookbook) contains a set of [communication benchmarks](https://github.com/EleutherAI/cookbook/tree/main/benchmarks/communication) for each collective that you can use to quickly measure

On CSPs that have enabled [SLURM Pyxis Container Plugin](https://github.com/NVIDIA/pyxis), such as CoreWeave, Crusoe, AWS, Oracle, Azure, GCP, etc, `torch-dist-bench.py` can be easily ran & reproduced via the following command:

Frameworks that shard weights and optim stages like [DeepSpeed](https://github.com/deepspeedai/DeepSpeed) w/ ZeRO Stage-3 do a lot more traffic than frameworks like [Megatron-DeepSpeed](https://github.com/bigscience-workshop/Megatron-DeepSp

[full details](https://github.com/deepspeedai/DeepSpeed/issues/2928#issuecomment-1463041491)

As it's often not easy to benchmark hundreds of nodes, often we try to benchmark interconnect performance using, say, 4 nodes. I wasn't sure whether this would give the correct indication for when 40 or 400 nodes will be used so I asked abo

For the optimal GDS performance, disable ACS by following these instructions [here](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs). Here are some [additional notes](https://do

While NCCL is excellent at automatically figuring out the best performance for any given network, sometimes it needs some help, in which case the following NCCL env vars are used to tune up performance. Let's look at a few common ones you m

I was asking questions about how a user can do the optimization and was told at [this NCCL Issue](https://github.com/NVIDIA/nccl/issues/790) that basically the user shouldn't try to optimize anything as NCCL has a ton of smart algorithms in

There is also an algo named `NVLS`, which uses NVLink SHARP to do the reduction inside the switch and can therefore report more than the wire spec - with NVLink 4.0 (450GBps) an `all-reduce` benchmark clocks 480GBps. `NVLSTree` (NCCL 2.18+)

Cooperative Thread Array (CTA) implements CUDA thread blocks - You can read about it [here](https://docs.nvidia.com/cuda/parallel-thread-execution/#thread-hierarchy).

The CTA value is derived algorithmically by NCCL, but the default behavior can be overridden by setting the lower and upper limits via the env vars: [`NCCL_MIN_CTAS`](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html?highli

For nuances see this paper: [Adaptive Routing in InfiniBand Hardware](https://web-backend.simula.no/sites/default/files/publications/files/adaptive_routing_in_infiniband_hardware.pdf).

#!/usr/bin/env python

[Credential-related source omitted]

#!/usr/bin/env python

[Credential-related source omitted]

#!/usr/bin/env python

Its new location is: https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/torch-dist-bench.py

new_url = "https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/torch-dist-bench.py"

[Credential-related source omitted]

#!/usr/bin/env python

# the 2*(n-1)/n busbw correction factor specific to all-reduce is explained here:     # https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md#allreduce

# https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md#allreduce

[Credential-related source omitted]

#!/bin/bash

#!/usr/bin/env python

The latest version of this program can be found at https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/torch-dist-bench.py

https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/torch-dist-bench.py

This benchmark is very similar to https://github.com/NVIDIA/nccl-tests, but much easier to set up, since all

For detailed documentation please refer to: https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/README.md#torch-dist-bench

https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/README.md#torch-dist-bench

This script: - has been derived from @jeffra's gist: https://gist.github.com/jeffra/b5e80466b4c86be00ea3b6f130fb7a36

- has been derived from @jeffra's gist: https://gist.github.com/jeffra/b5e80466b4c86be00ea3b6f130fb7a36

- which in turn is derived from the logic in https://github.com/NVIDIA/nccl-tests

* Indu Thangakrishnan https://github.com/indhub to handle timing correctly using cuda events

# the busbw correction factors of nccl-tests, explained in # https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md#bus-bandwidth - busbw reflects how optimally

# https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md#bus-bandwidth - busbw reflects how optimally

# 1. device_id arg was added in torch==2.3     # 2. setting device_id leads to hanging in 2.6.0<torch<2.7.1 https://github.com/pytorch/pytorch/issues/153960

# 2. setting device_id leads to hanging in 2.6.0<torch<2.7.1 https://github.com/pytorch/pytorch/issues/153960

[Credential-related source omitted]

PyTorch has `send` and `recv` for blocking, `isend` and `irecv` for non-blocking p2p comms. [more](https://docs.pytorch.org/tutorials/intermediate/dist_tuto.html#id1).

Reference: [PyTorch 2.14 release blog](https://pytorch.org/blog/pytorch-2-14-release-blog/).

![broadcast](images/collective-broadcast-1.png) [source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

![broadcast](images/collective-broadcast-2.png) [source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

`dist.broadcast(tensor, src, group)`: Copies `tensor` from `src` to all other processes. [doc](https://docs.pytorch.org/docs/stable/distributed.html#torch.distributed.broadcast).

![gather](images/collective-gather-1.png) [source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

![gather](images/collective-gather-2.png) [source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

`dist.gather(tensor, gather_list, dst, group)`: Copies one `tensor` from every process in the group into `gather_list` on `dst`. [doc](https://docs.pytorch.org/docs/stable/distributed.html#torch.distributed.gather)

![all-gather](images/collective-all-gather-1.png) [source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

![all-gather](images/collective-all-gather-2.png) [source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

`dist.all_gather(tensor_list, tensor, group)`: Copies `tensor` from all processes to `tensor_list`, on all processes. [doc](https://docs.pytorch.org/docs/stable/distributed.html#torch.distributed.all_gather)

![reduce](images/collective-reduce-1.png) [source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

![reduce](images/collective-reduce-2.png) [source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

`dist.reduce(tensor, dst, op, group)`: Applies `op` to every `tensor` and stores the result in `dst`. [doc](https://docs.pytorch.org/docs/stable/distributed.html#torch.distributed.reduce)

PyTorch supports multiple reduction operations like: `avg`, `sum`, `product`, `min`, `max`, `band`, `bor`, `bxor`, and others - [full list](https://docs.pytorch.org/docs/stable/distributed.html#torch.distributed.ReduceOp).

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

For example, this collective is used in [DDP](https://docs.pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html) to reduce gradients between all participating ranks.

`dist.all_reduce(tensor, op, group)`: Same as reduce, but the result is stored in all processes. [doc](https://docs.pytorch.org/docs/stable/distributed.html#torch.distributed.all_reduce)

![scatter](images/collective-scatter-1.png) [source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

![scatter](images/collective-scatter-2.png) [source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

`dist.scatter(tensor, scatter_list, src, group)`: Copies the `i`-th tensor `scatter_list[i]` to the `i`-th process. [doc](https://docs.pytorch.org/docs/stable/distributed.html#torch.distributed.scatter)

![reduce-scatter](images/collective-reduce-scatter.png) [source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

`dist.reduce_scatter(output, input_list, op, group, async_op)`: Reduces, then scatters a list of tensors to all processes in a group. [doc](https://docs.pytorch.org/docs/stable/distributed.html#torch.distributed.reduce_scatter)

![all-to-all](images/collective-all-to-all-1.png) [source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

![all-to-all](images/collective-all-to-all.png) [source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

`dist.all_to_all(output_tensor_list, input_tensor_list, group)`: Scatters list of input tensors to all processes in a group and return gathered list of tensors in output list. [doc](https://docs.pytorch.org/docs/stable/distributed.html#torc

![ring-based broadcast](images/broadcast-ring.png) [source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

![ring-based all-reduce chunk 1](images/all-reduce-ring-chunk1.png) [source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

![ring-based all-reduce chunk 2](images/all-reduce-ring-chunk2.png) [source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

[source](https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf)

Sources: 1. [Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithms](https://arxiv.org/abs/2507.04786) (2025), Table I.

1. [Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithms](https://arxiv.org/abs/2507.04786) (2025), Table I.

- [UvA Deep Learning Tutorials](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/DL2/High-performant_DL/Multi_GPU/hpdlmultigpu.html#Communication-Primitives)

- [Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithms](https://arxiv.org/abs/2507.04786) (2025) - this paper presents a comprehensive analysis of NCCL, focusing on its communication protocol variants (Simp

This section is definitely non-exhaustive and is meant to cover some of the most common setup issues that I have often encountered. For more complex problems please research the [NCCL repo Issues](https://github.com/NVIDIA/nccl/issues) or f

tells us that [nccl-fastsocket](https://github.com/google/nccl-fastsocket) transport layer plugin is used and it discovered 8 `ibs*` network interfaces (NIC cards). If you're using Google Cloud this is correct, and your NCCL is likely setup

Here, you can see that IB is used with 8 `mlx5_*` interfaces for collective comms, and one OOB, which stands for Out-Of-Band, and is used for doing bootstrapping the connections and is usually using a slower Ethernet NIC (at times [several 

The full doc is [here](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-socket-ifname).

The full doc is [here](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-ib-hca).

* Give enough resources by adding to the docker `run` these additional args: `--shm-size=1g --ulimit memlock=-1` ([more details](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#sharing-data))

For consumer-level GPUs there could be a variety of reasons for your GPU not being supported, often it's the IOMMU and/or ACS features being enabled. At other times it's just the driver version. And if you spend some time searching you migh

To check if PCI Access Control Services (ACS) are enabled and to disable those follow [this guide](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs).

The following env vars are most useful during debugging NCCL-related issues such as hanging and crashing. The full list of those can be found [here](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html).

A worked example of using the exclusion form to route around a single problem adapter is in [How to diagnose NCCL multi-gpu and multi-node connectivity issues](#how-to-diagnose-nccl-multi-gpu-and-multi-node-connectivity-issues). The full do

- [Slinky](https://github.com/slinkyproject) - a framework from the creators of SLURM.

- [Soperator](https://github.com/nebius/soperator) - from Nebius

- CoreWeave never released their [SUNK](https://www.coreweave.com/products/sunk) open source, so it's only available to its customers.

- [Kubernetes](kubernetes/) - also known as K8s, is an open source system for automating deployment, scaling, and management of containerized applications. Here is a good [comparison between SLURM and K8s](https://web.archive.org/web/202503

- [dstack](https://github.com/dstackai/dstack) is a lightweight, open-source alternative to Kubernetes & SLURM, simplifying AI container orchestration with multi-cloud & on-prem support. It natively supports NVIDIA, AMD, & TPU.

- [SkyPilot](https://github.com/skypilot-org/skypilot) is a framework for running AI and batch workloads on any infra, offering unified execution, high cost savings, and high GPU availability.

- [OpenHPC](https://github.com/openhpc/ohpc) provides a variety of common, pre-built ingredients required to deploy and manage an HPC Linux cluster including provisioning tools, resource management, I/O clients, runtimes, development tools,

- [run.ai](https://www.nvidia.com/en-us/software/run-ai/) - got acquired by NVIDIA and is planned to be open sourced soon.

- [Docker Swarm](https://docs.docker.com/engine/swarm/) is a container orchestration tool.

- [IBM Platform Load Sharing Facility (LSF)](https://www.ibm.com/products/hpc-workload-management) Suites is a workload management platform and job scheduler for distributed high performance computing (HPC).

Here is how this is done. Kubernetes 1.32 introduced the kubelet flag [`singleProcessOOMKill`](https://github.com/kubernetes/kubernetes/pull/126096), which allows you to set `memory.oom.group = 0`.

[Credential-related source omitted]

[Credential-related source omitted]

PDSH_RCMD_TYPE=ssh pdsh -w node-[21,23-26]  "nvidia-smi --query-compute-apps=pid --format=csv,noheader | sort | uniq | xargs -n1 sudo kill -9"

sudo scontrol show config

sudo scontrol reconfigure

sudo cat /var/log/slurm/slurmctld.log

#!/bin/bash

#!/bin/bash

#!/bin/bash

- [torchrun](torchrun-launcher.slurm) - to be used with [PyTorch distributed](https://github.com/pytorch/pytorch).

- [accelerate](accelerate-launcher.slurm) - to be used with [HF Accelerate](https://github.com/huggingface/accelerate).

- [lightning](lightning-launcher.slurm) - to be used with [Lightning](https://lightning.ai/) (“PyTorch Lightning” and “Lightning Fabric”).

#!/bin/bash

#!/bin/bash

#!/bin/bash

#!/bin/bash

#!/bin/bash

PDSH_RCMD_TYPE=ssh pdsh -w $n "nvidia-smi --query-compute-apps=pid --format=csv,noheader | sort | uniq | xargs -n1 sudo kill -9"

sudo scontrol update NodeName=$n State=idle Reason="undrained by $USER"

By default, if the cpu has [Hyper-Threads](https://en.wikipedia.org/wiki/Hyper-threading) (HT) enabled, SLURM will use it. If you don't want to use HT you have to specify `--hint=nomultithread`.

More details https://slurm.schedmd.com/job_array.html

But if you want to use something where you can disconnect and reconnect and continue using the program normally you'd have to use a [terminal multiplexer](https://en.wikipedia.org/wiki/Terminal_multiplexer) program like [`tmux`](https://git

[Credential-related source omitted]

[Credential-related source omitted]

find $WORK/cron -name "*.out" -mtime +7 -exec rm -f {} +

I'm yet to find why this is so, but so far we have been using a kill switch workaround. You have to code it in your framework. For example, see how it was implemented in [Megatron-DeepSpeed](https://github.com/bigscience-workshop/Megatron-D

rm $WORK/tmp/training17-kill-switch

- Boris Dayma wrote [A Recipe for Training Large Models](https://wandb.ai/craiyon/report/reports/Recipe-Training-Large-Models--VmlldzozNjc4MzQz)

- The HuggingFace team published [The Ultra-Scale Playbook: Training LLMs on GPU Clusters](https://huggingface.co/spaces/nanotron/ultrascale-playbook).

- Jacob Austin maintains [How to Scale Your Model: A Systems View of LLMs on TPUs](https://jax-ml.github.io/scaling-book/)

- The Harvard Edge group publishes [Machine Learning Systems](https://github.com/harvard-edge/cs249r_book) (CS249r textbook; site: [mlsysbook.ai](http://mlsysbook.ai/)).

- [@StellaAthena](https://github.com/StellaAthena) created the [Common LLM Settings spreadsheet](https://docs.google.com/spreadsheets/d/14vbBbuRMEHoqeuMHkTfw3uiZVmyXNuoSp8s-aHvfvZk/edit#gid=0) which can be a super-useful resource when you'r

- A few years back I started compiling information on [which dtype the models were trained in](https://discuss.huggingface.co/t/model-pre-training-precision-database-fp16-fp32-bf16/5671) - it only contains a handful of models but if you're 

If you know of a public LLM/VLM training logbook, or a technical report with such a section, that is not on this list please kindly let me know via [an Issue](https://github.com/stas00/ml-engineering/issues) or add it via a PR. Thank you!

- BigScience pre-BLOOM 108B training experiments (2021): [chronicles](https://github.com/bigscience-workshop/bigscience/blob/master/train/tr8-104B-wide/chronicles.md) |

[chronicles](https://github.com/bigscience-workshop/bigscience/blob/master/train/tr8-104B-wide/chronicles.md) |

[the full spec and discussions](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr8-104B-wide)

(backup: [1](https://github.com/stas00/bigscience-backup/blob/master/train/tr8-104B-wide/chronicles.md) | [2](https://github.com/stas00/bigscience-backup/tree/master/train/tr8-104B-wide))

[1](https://github.com/stas00/bigscience-backup/blob/master/train/tr8-104B-wide/chronicles.md) |

[2](https://github.com/stas00/bigscience-backup/tree/master/train/tr8-104B-wide))

- BigScience BLOOM-176B (2022): [chronicles-prequel](https://github.com/bigscience-workshop/bigscience/blob/master/train/tr11-176B-ml/chronicles-prequel.md) |

[chronicles-prequel](https://github.com/bigscience-workshop/bigscience/blob/master/train/tr11-176B-ml/chronicles-prequel.md) |

[chronicles](https://github.com/bigscience-workshop/bigscience/blob/master/train/tr11-176B-ml/chronicles.md) |

[the full spec and discussions](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr11-176B-ml)

(backup: [1](https://github.com/stas00/bigscience-backup/blob/master/train/tr11-176B-ml/chronicles-prequel.md) | [2](https://github.com/stas00/bigscience-backup/blob/master/train/tr11-176B-ml/chronicles.md) |

[1](https://github.com/stas00/bigscience-backup/blob/master/train/tr11-176B-ml/chronicles-prequel.md) |

[2](https://github.com/stas00/bigscience-backup/blob/master/train/tr11-176B-ml/chronicles.md) |

[3](https://github.com/stas00/bigscience-backup/tree/master/train/tr11-176B-ml))

- Meta OPT-175B (2022):  [logbook](https://github.com/facebookresearch/metaseq/tree/main/projects/OPT/chronicles) | [Video](https://www.youtube.com/watch?v=p9IxoSkvZ-M) (backup: [1](https://github.com/stas00/metaseq-backup/tree/main/project

[logbook](https://github.com/facebookresearch/metaseq/tree/main/projects/OPT/chronicles) | [Video](https://www.youtube.com/watch?v=p9IxoSkvZ-M) (backup: [1](https://github.com/stas00/metaseq-backup/tree/main/projects/OPT/chronicles))

- THUDM GLM-130B (2022): [en logbook](https://github.com/zai-org/GLM-130B/blob/main/logs/main-log-en.md) | [Mandarin version](https://github.com/zai-org/GLM-130B/blob/main/logs/main-log.md) (backup:  [1](https://github.com/stas00/GLM-130B-b

- HuggingFace IDEFICS-80B multimodal (Flamingo repro) (2023): [Learning log](https://github.com/huggingface/m4-logs/blob/master/memos/README.md) | [Training Chronicles](https://github.com/huggingface/m4-logs/blob/master/tr-190-80b/chronicle

- BloombergGPT 50B LLM - section C in [BloombergGPT: A Large Language Model for Finance](https://arxiv.org/abs/2303.17564)

- [MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs](https://arxiv.org/abs/2402.15627) - the paper covers various training issues and their resolution - albeit on models that are proprietary yet just as instructiona

- Imbue's [From bare metal to a 70B model: infrastructure set-up and scripts](https://imbue.com/blog/70b-infrastructure) very detailed technical post covers many training-related issues that they had to overcome while training a proprietary

- Meta's [The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783) devotes [§3.3 Infrastructure, Scaling, and Efficiency](https://arxiv.org/pdf/2407.21783#page=12) to the training stack, including §3.3.4 Reliability and Operational Cha

- Project Apertus (ETH Zürich, EPFL, CSCS) - [Apertus v1 Technical Report: Democratizing Open and Compliant LLMs for Global Language Environments](https://arxiv.org/abs/2509.14233) - a fully open 8B/70B suite trained on 15T tokens across 18

- [Supercharging Olmo-core for Efficient and Scalable MoE Training](https://allenai.org/papers/olmocore3) - practical recommendations for training large scale MoE models in 2026!

- [Marin 535B-A23B MoE live](https://mtracker.oa.dev/hero-run-535b) - each short entry goes to a details GitHub Issue where you can read the details. Finished trainings retrospectives: [8B](https://marin.readthedocs.io/en/latest/reports/mar

- Project Apertus - [Apertus v1.5 Technical Report: From Open Text Models to Open Multimodal Reasoners](https://arxiv.org/abs/2610.15542) - continued multimodal pretraining of the Apertus 8B/70B models; like v1 it includes *Infrastructure, 

- Imbue published a detailed log of how they have set up a 512-node IB-fat-tree cluster and made it to work: [From bare metal to a 70B model: infrastructure set-up and scripts](https://imbue.com/blog/70b-infrastructure), they also open-sour

- SemiAnalysis published a great detailed writeup about [what it takes to set up a Neocloud cluster](https://newsletter.semianalysis.com/p/ai-neocloud-playbook-and-anatomy).

- [Machine Learning Engineering](ml-engineering/SKILL.md) — train and operate large-scale ML models (accelerators, storage, network, orchestration, throughput, instabilities, checkpointing, inference). Canonical: https://github.com/stas00/m

- [Evaluate a GPU cluster](evaluate-cluster/SKILL.md) — trial-node / acceptance loop: environment dump, sequential matmul FLOPS, intra- and inter-node all-reduce, `fio-scan` on local disk and shared FS, dated markdown report. Canonical: htt

Companion skills in other books: [The Art of Debugging](https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md), [Stas' Python Cookbook](https://github.com/stas00/python-cookbook/blob/master/SKILL.md).

Canonical copy: https://github.com/stas00/ml-engineering/blob/master/skills/evaluate-cluster/SKILL.md

- This skill: https://github.com/stas00/ml-engineering/blob/master/skills/evaluate-cluster/SKILL.md

- Raw: https://raw.githubusercontent.com/stas00/ml-engineering/master/skills/evaluate-cluster/SKILL.md

This file is the whole runbook. You do not need any other document from the author's books. Scripts and spec tables are on GitHub; fetch them (commands below). Optional background: [How to evaluate the cluster](https://github.com/stas00/ml-

| -------------------- | -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

| **MAMF**             | Maximum Achievable Matmul FLOPS: the highest any **matmul shape** (M×N×K) reaches on that GPU. The winning shapes are short bursts — the clock sits at boost and the board stays far below its power limit — so this is

| **MSMF**             | Maximum Sustainable Matmul FLOPS: what is left once the matmul shape is big enough to pin the board at its power limit and the clock settles below boost. This is what back-to-back dense matmuls actually get, provide

| **lemon GEMM**       | A one-off inline `torch.matmul` (not a published script) on every GPU with the same large shape, to catch a dead, slow, or throttled card.                                                                             

| **all-reduce**       | Collective: every rank ends up with the sum of all ranks' tensors. Used to stress NVLink and the network.                                                                                                              

| **busbw / algbw**    | Printed by [`torch-dist-bench.py`](https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/torch-dist-bench.py). **busbw** = unidirectional bus bandwidth (what the fabric moves). **algbw** = algorith

| **hostfile**         | Text file, one peer address per line (`10.0.0.1` or `10.0.0.1 slots=8`). DeepSpeed `-H` and `pdsh -w` read it.                                                                                                         

| **pdsh**             | Parallel SSH: run one command on many hosts.                                                                                                                                                                           

| **fio-scan**         | Wrapper around `fio` ([script](https://github.com/stas00/ml-engineering/blob/master/storage/fio-scan)): six timed runs (16 KiB / 1 MiB / 1 GiB × read/write). Needs [`fio-json-extract.py`](https://github.com/stas00/m

RAW=https://raw.githubusercontent.com/stas00/ml-engineering/master curl -fsSL "$RAW/compute/accelerator/benchmarks/mamf-finder.py" -o mamf-finder.py

curl -fsSL "$RAW/compute/accelerator/benchmarks/mamf-finder.py" -o mamf-finder.py

curl -fsSL "$RAW/compute/accelerator/benchmarks/mamf-finder-all-gpus.py" -o mamf-finder-all-gpus.py

# mamf-finder-all-gpus.py runs ./mamf-finder.py from its own directory — keep both in one directory. curl -fsSL "$RAW/network/benchmarks/torch-dist-bench.py" -o torch-dist-bench.py curl -fsSL "$RAW/debug/torch-distributed-gpu-test.py" -o to

curl -fsSL "$RAW/network/benchmarks/torch-dist-bench.py" -o torch-dist-bench.py

curl -fsSL "$RAW/debug/torch-distributed-gpu-test.py" -o torch-distributed-gpu-test.py

curl -fsSL "$RAW/storage/fio-scan" -o fio-scan

curl -fsSL "$RAW/storage/fio-json-extract.py" -o fio-json-extract.py

Official BF16 peaks: [TFLOPS comparison table](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#tflops-comparison-table). NVLink gens: [NVLink](https://github.com/stas00/ml-engineering/blob/master/network/R

**Links in the report** to those scripts and tables are absolute GitHub URLs (`https://github.com/stas00/ml-engineering/blob/master/…`). Never `../` relatives — the report may be copied elsewhere. Sibling `raw-<name>/` stays relative. Exter

| GPUs per node                              | default 8                      | `nvidia-smi -L` if unknown                                                                                                   | | Local disk path + shared-FS pat

| Advertised uni bandwidth / official TFLOPS | optional                       | else [accelerator tables](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md) + `nvidia-smi` |

# 3. Peer → peer TCP (rdzv will need this). From node 1, hit node 0's IP:22 or the container sshd port. MASTER_ADDR=$(grep -v '^#' "$HOSTFILE" | grep -v '^$' | head -1 | cut -d' ' -f1) timeout 5 bash -c "echo >/dev/tcp/$MASTER_ADDR/${SSH_PO

timeout 5 bash -c "echo >/dev/tcp/$MASTER_ADDR/${SSH_PORT:-22}" && echo OPEN

timeout 5 bash -c "echo >/dev/tcp/$MASTER_ADDR/${SSH_PORT:-22}" && echo OPEN

python3 -m venv "$EVAL_VENV"   # $HOME/venvs/cluster-eval, or shared FS so every node sees it "$EVAL_VENV/bin/pip" install -U pip "$EVAL_VENV/bin/pip" index versions torch --index-url https://download.pytorch.org/whl/cu130   # -> LATEST: x.

"$EVAL_VENV/bin/pip" install -U pip "$EVAL_VENV/bin/pip" index versions torch --index-url https://download.pytorch.org/whl/cu130   # -> LATEST: x.y.z "$EVAL_VENV/bin/pip" install torch numpy packaging nvidia-ml-py matplotlib \

"$EVAL_VENV/bin/pip" index versions torch --index-url https://download.pytorch.org/whl/cu130   # -> LATEST: x.y.z

"$EVAL_VENV/bin/pip" install torch numpy packaging nvidia-ml-py matplotlib    --index-url https://download.pytorch.org/whl/cu130

--index-url https://download.pytorch.org/whl/cu130

sudo apt-get update

sudo apt-get install -y infiniband-diags fio iproute2 rdma-core pdsh

sudo dnf install -y infiniband-diags fio iproute rdma-core pdsh

Open Compute with the **GPUs that are here** (name, count per node, SM count, HBM GiB, TDP, official BF16 TFLOPS + [table](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#tflops-comparison-table)). Do not 

Intra-node (header of [`torch-dist-bench.py`](https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/torch-dist-bench.py)):

Requires passwordless SSH from the launch node to every hostfile IP. Port is cluster-specific (a wrapper on PATH may already add `-p`). If `ssh` hits the **machine** sshd (not the container), start `sshd` in the container on an unused port,

1. **Nobody hosts the store.** With no `is_host`, each agent calls `_matches_machine_hostname(endpoint_host)` ([`elastic/rendezvous/utils.py`](https://github.com/pytorch/pytorch/blob/main/torch/distributed/elastic/rendezvous/utils.py)), whi

ss -ltnp | grep "$MASTER_PORT" # 3. can a worker reach it?                          -> expect OPEN timeout 5 bash -c "echo > /dev/tcp/$MASTER_ADDR/$MASTER_PORT" && echo OPEN

# 3. can a worker reach it?                          -> expect OPEN timeout 5 bash -c "echo > /dev/tcp/$MASTER_ADDR/$MASTER_PORT" && echo OPEN # 4. is the advertised master resolvable everywhere?

timeout 5 bash -c "echo > /dev/tcp/$MASTER_ADDR/$MASTER_PORT" && echo OPEN

timeout 5 bash -c "echo > /dev/tcp/$MASTER_ADDR/$MASTER_PORT" && echo OPEN

Do this **before** quoting any all-reduce number. `busbw` is already unidirectional ([directionality](https://github.com/stas00/ml-engineering/blob/master/network/README.md#unidirectional-vs-bidirectional-duplex)). Tables: [intra-node](http

Map NVLink gen from per-link payload rate (or GPU family): ~53 GBps/link → **NVLink 5** (900 GBps unidirectional, B200-class); ~26.5 → **NVLink 4** (450, H100/H200-class); slower ~12.5-class is older NVLink ([table](https://github.com/stas0