Hanyuyuan6/remote-gpu-trainer

contact_links:   - name: Contributing guide     url: https://github.com/Hanyuyuan6/remote-gpu-trainer#contributing

- name: Contributing guide     url: https://github.com/Hanyuyuan6/remote-gpu-trainer#contributing     about: The bar a new gotcha or profile has to clear (root-caused, reproduced, generalizable) before it lands.

url: https://github.com/Hanyuyuan6/remote-gpu-trainer#contributing

[Credential-related source omitted]

[Credential-related source omitted]

[![evals](https://github.com/Hanyuyuan6/remote-gpu-trainer/actions/workflows/evals.yml/badge.svg)](https://github.com/Hanyuyuan6/remote-gpu-trainer/actions/workflows/evals.yml)

[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)

remote-gpu-trainer is an [Agent Skill](https://agentskills.io) for Claude Code, Codex and other coding agents. It lets your agent launch, monitor, resume and stop training on AutoDL, RunPod, vast.ai, Lambda, Paperspace, four other Chinese G

- **Paying for idle machines.** A stopped RunPod Pod bills its volume at twice the running rate, a stopped vast.ai instance bills its disk indefinitely, and `sudo shutdown` on Lambda leaves the meter running. The skill knows which action ac

Install the remote-gpu-trainer agent skill. Read https://raw.githubusercontent.com/Hanyuyuan6/remote-gpu-trainer/main/README.md

https://raw.githubusercontent.com/Hanyuyuan6/remote-gpu-trainer/main/README.md

git clone https://github.com/Hanyuyuan6/remote-gpu-trainer.git \

git clone https://github.com/Hanyuyuan6/remote-gpu-trainer.git \

| Lambda | `terminate`, which wipes the local NVMe | No stop state; `sudo shutdown` keeps billing; NFS bills until deleted |

| Host (linked to its docs) | User skills directory | |---|---| | [Claude Code](https://code.claude.com/docs/en/skills) | `~/.claude/skills/` |

|---|---| | [Claude Code](https://code.claude.com/docs/en/skills) | `~/.claude/skills/` | | [OpenAI Codex](https://learn.chatgpt.com/docs/build-skills) | `~/.agents/skills/` |

| [Claude Code](https://code.claude.com/docs/en/skills) | `~/.claude/skills/` |

| [OpenAI Codex](https://learn.chatgpt.com/docs/build-skills) | `~/.agents/skills/` |

| [Cursor](https://cursor.com/docs/skills) | `~/.agents/skills/` |

| [Gemini CLI](https://github.com/google-gemini/gemini-cli/blob/main/docs/cli/skills.md) | `~/.agents/skills/` |

| [GitHub Copilot](https://docs.github.com/en/copilot/concepts/agents/about-agent-skills) | `~/.agents/skills/` |

git -C "$SKILLS_DIR/remote-gpu-trainer" pull --ff-only else   git clone https://github.com/Hanyuyuan6/remote-gpu-trainer.git \

else   git clone https://github.com/Hanyuyuan6/remote-gpu-trainer.git \     "$SKILLS_DIR/remote-gpu-trainer"

git clone https://github.com/Hanyuyuan6/remote-gpu-trainer.git \

- Keep the folder name exactly `remote-gpu-trainer`. The Agent Skills spec requires it to match the skill's `name`. - If a non-empty folder of that name exists but is not a git clone, `git clone` stops with an error. Ask the user before rep

- If a non-empty folder of that name exists but is not a git clone, `git clone` stops with an error. Ask the user before replacing that folder. - Without git, download `https://github.com/Hanyuyuan6/remote-gpu-trainer/archive/refs/heads/mai

- Without git, download `https://github.com/Hanyuyuan6/remote-gpu-trainer/archive/refs/heads/main.tar.gz`, extract it into the skills directory, and rename `remote-gpu-trainer-main` to `remote-gpu-trainer`.

The skill is self-contained. Where a separate skill deepens a step, it names one: [superpowers](https://github.com/obra/superpowers) for verification before completion and parallel dispatch, and [Hugging Face skills](https://github.com/hugg

Issues and pull requests are welcome, and outside contributions have been merged: [PR #6](https://github.com/Hanyuyuan6/remote-gpu-trainer/pull/6) by [@hector25](https://github.com/hector25) added teardown and billing gotchas for RunPod, va

Report vulnerabilities privately through **Security → Advisories → Report a vulnerability** on this repository, not in a public issue, and never include a live secret. Details are in [SECURITY.md](https://github.com/Hanyuyuan6/remote-gpu-tr

title  = {remote-gpu-trainer: an Agent Skill for the DL experiment lifecycle on owned or rented GPUs},   year   = {2026},   url    = {https://github.com/Hanyuyuan6/remote-gpu-trainer}

year   = {2026},   url    = {https://github.com/Hanyuyuan6/remote-gpu-trainer} }

url    = {https://github.com/Hanyuyuan6/remote-gpu-trainer}

{"id": "node-idles-on-permission-prompt", "prompt": "I launched remote training from my local agent session; the agent then ran some local cleanup that stopped on a permission prompt. Nobody answered overnight and the rented GPU billed for 

#!/usr/bin/env python3

#!/usr/bin/env python3

[Credential-related source omitted]

[Credential-related source omitted]

expandable — prefer expand over silently shrinking the experiment (principle #9); present exact `rm -rf`

`snapshot_download` hangs or `ConnectTimeout` on big `.safetensors` shards. *Root cause:* (a) HF's Xet CAS backend is not mirror-proxied; (b) `no_proxy` lists `modelscope.com` not `modelscope.cn` (domestic source forced through internationa

backend is not mirror-proxied; (b) `no_proxy` lists `modelscope.com` not `modelscope.cn` (domestic source forced through international proxy = slower); (c) a curl test run without turbo measures the wrong path. *Fix:* `export HF_HUB_DISABLE

forced through international proxy = slower); (c) a curl test run without turbo measures the wrong path.

*Fix:* `export HF_HUB_DISABLE_XET=1` (or `pip uninstall -y hf_xet`) with `HF_ENDPOINT=https://hf-mirror.com`,

**AD7 — platform TensorBoard pinned to `/root/tf-logs`; events elsewhere invisible.** *Symptom:* the events file is non-empty and `curl http://127.0.0.1:6007/` returns 200, but the AutoPanel TB tile shows

events file is non-empty and `curl http://127.0.0.1:6007/` returns 200, but the AutoPanel TB tile shows

/root/tf-logs` and the tile proxies that pid; `--logdir` is hard-pinned and not reconfigurable in-container. *Fix:* write `SummaryWriter(log_dir="/root/tf-logs/<run>")`, or `ln -sfn <your-tb> /root/tf-logs/<run>` (the pinned TB's `--reload=

*Fix:* write `SummaryWriter(log_dir="/root/tf-logs/<run>")`, or `ln -sfn <your-tb> /root/tf-logs/<run>` (the pinned TB's `--reload=5` picks it up in ~5 s); verify with `curl … /data/runs`, not `ss`. (Also: restart the TB server to evict STA

(the pinned TB's `--reload=5` picks it up in ~5 s); verify with `curl … /data/runs`, not `ss`. (Also:

SCRATCH='latest.pth'                  # mutable resume anchor; active only, never export it HF_HOME=$CACHE_ROOT/huggingface       # regenerable cache stays in the exact project layout (AD5) HF_ENDPOINT=https://hf-mirror.com     # + HF_HUB_D

HF_HOME=$CACHE_ROOT/huggingface       # regenerable cache stays in the exact project layout (AD5) HF_ENDPOINT=https://hf-mirror.com     # + HF_HUB_DISABLE_XET=1 (AD2) DETACH=tmux                           # nohup fallback when tmux is absen

HF_ENDPOINT=https://hf-mirror.com     # + HF_HUB_DISABLE_XET=1 (AD2)

[Credential-related source omitted]

**The entire mirror / proxy / resumable-download story is shared across all CN platforms and lives in `references/run-remote/china-network.md` — do NOT duplicate it here.** That reference owns the mirrors table (PyPI/conda/HF), `HF_ENDPOINT

`references/run-remote/china-network.md` — do NOT duplicate it here.** That reference owns the mirrors table (PyPI/conda/HF), `HF_ENDPOINT=https://hf-mirror.com`, the ModelScope fallback, the resumable-download retry ladder, the `hf_transfe

(PyPI/conda/HF), `HF_ENDPOINT=https://hf-mirror.com`, the ModelScope fallback, the resumable-download retry

- **Gpushare — has a real academic proxy** (the closest analog to AutoDL's `/etc/network_turbo`):   `export https_proxy=http://turbo.gpushare.com:<PORT> http_proxy=http://turbo.gpushare.com:<PORT>`

`export https_proxy=http://turbo.gpushare.com:<PORT> http_proxy=http://turbo.gpushare.com:<PORT>`

[Credential-related source omitted]

- **Detect a stuck / throttled download.** `du -sh <cache-dir>` twice ~30 s apart — flat size = stalled (often   the GFW or a left-on Gpushare turbo proxy restricting a non-whitelisted host, GS2). Cross-check with   `curl -sI -x "$https_pro

the GFW or a left-on Gpushare turbo proxy restricting a non-whitelisted host, GS2). Cross-check with   `curl -sI -x "$https_proxy" https://hf-mirror.com` / `env | grep -i proxy`; `unset http_proxy https_proxy`   and retry on the mirror.

`curl -sI -x "$https_proxy" https://hf-mirror.com` / `env | grep -i proxy`; `unset http_proxy https_proxy`

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

- **Egress/proxy:** host-dependent; there is no platform proxy hook. If the box sits behind the GFW,   set the mirror manually — `export HF_ENDPOINT=https://hf-mirror.com` (or `HF_HUB_ENABLE_HF_TRANSFER=1`

set the mirror manually — `export HF_ENDPOINT=https://hf-mirror.com` (or `HF_HUB_ENABLE_HF_TRANSFER=1`

DATA_DIR=$HOME/proj    (working dir / data disk on the box) DURABLE_DIR=$HOME/proj (durable mount; restore canonical bytes to an independent consumer before teardown) PROXY_HOOK=        (none by default; set HF_ENDPOINT=https://hf-mirror.co

DURABLE_DIR=$HOME/proj (durable mount; restore canonical bytes to an independent consumer before teardown) PROXY_HOOK=        (none by default; set HF_ENDPOINT=https://hf-mirror.com only if behind the GFW) CRED_FILE=~/.netrc on the box's lo

PROXY_HOOK=        (none by default; set HF_ENDPOINT=https://hf-mirror.com only if behind the GFW)

[Credential-related source omitted]

[Credential-related source omitted]

meter_stop_verb: terminate    # the ONLY action that stops billing; sudo shutdown does NOT

- **Web console** at `cloud.lambda.ai` → Instances → Launch (pick GPU type + region, attach a filesystem   here if one is needed — see §2; attach any per-instance firewall ruleset here too — see §3/LAM4). - **REST API** — `https://cloud.lam

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

`terminate`, or an **improper `sudo shutdown`** that pushes the box to **Alert** while still billing

> shutdown -h now` or `sudo systemctl poweroff` … These commands will not work as expected and will cause

AutoDL tmux pattern. Install if absent (`sudo apt install -y tmux`); fall back to

- **LAM3 — `sudo shutdown` / `poweroff` keeps the meter running (Alert state).**

`for so in /usr/lib/python3/dist-packages/tensorflow/libcudnn*; do sudo ln -s "$so" /usr/lib/x86_64-linux-gnu/; done`

`torch==2.2.0` fails to resolve. → Root cause: GH200 is aarch64; the default PyPI torch wheel for   aarch64 is CPU-only. → Fix: use Lambda Stack's pre-compiled ARM PyTorch (e.g. 2.4.1) as-is, or install   from the CUDA index `pip install to

aarch64 is CPU-only. → Fix: use Lambda Stack's pre-compiled ARM PyTorch (e.g. 2.4.1) as-is, or install   from the CUDA index `pip install torch --index-url https://download.pytorch.org/whl/cu128` (aarch64 GPU   wheels live there), or compil

from the CUDA index `pip install torch --index-url https://download.pytorch.org/whl/cu128` (aarch64 GPU

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

weights/datasets directly; there is no fatter "on-box pipe" to prefer because the box *is* local. - **China mirror:** only if **your** connection sits behind the GFW — then `export   HF_ENDPOINT=https://hf-mirror.com` (detail: `references/r

- **China mirror:** only if **your** connection sits behind the GFW — then `export   HF_ENDPOINT=https://hf-mirror.com` (detail: `references/run-remote/china-network.md`); otherwise irrelevant. - **Port exposure:** services are already on l

HF_ENDPOINT=https://hf-mirror.com` (detail: `references/run-remote/china-network.md`); otherwise irrelevant.

then open `http://localhost:6006`. No SSH tunnel, no console port-forward.

DATA_DIR=$HOME/proj         (working dir / data on your local disk) DURABLE_DIR=$HOME/proj      (durable = your own disk; it IS local — no "pull to local" step, but back up irreplaceables yourself) PROXY_HOOK=                 (none; set HF_

DURABLE_DIR=$HOME/proj      (durable = your own disk; it IS local — no "pull to local" step, but back up irreplaceables yourself) PROXY_HOOK=                 (none; set HF_ENDPOINT=https://hf-mirror.com only if YOUR connection is behind the

PROXY_HOOK=                 (none; set HF_ENDPOINT=https://hf-mirror.com only if YOUR connection is behind the GFW)

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

- **Spot / interruptible** — set `interruptible: true` (REST) or bid via legacy GraphQL. Roughly **~50% cheaper** than On-Demand (verified runpod.io/blog/spot-vs-on-demand-instances-runpod: e.g. A6000 spot $0.232 vs on-demand $0.491/gpu/hr 

- **On-Demand** — non-interruptible while running, but carries the sneakier **zero-GPU-on-restart** trap (RP1): a stopped Pod is pinned to its host, and if that GPU is rented away the Pod can only restart **with zero GPUs** ("there are no G

- **Terminate is the meter-stop verb AND it is destructive.** "Terminating permanently deletes all data not stored in a network volume. Export important data first." (verified docs.runpod.io/pods/manage-pods 2026-06). Move every needed arti

- **Low-balance auto-stop → silent deletion.** When the account balance can no longer cover remaining runtime, RunPod **auto-stops all Pods**; storage then keeps accruing on the stopped volume disk, and **a depleted balance can have Pods + 

- **Billing granularity:** compute + container/volume disk bill **per second**; Network Volumes bill **hourly** (verified https://docs.runpod.io/references/billing-information 2026-06).

- **RP1 — Zero-GPU-on-restart.** Symptom: a stopped Pod restarts with no GPU attached and refuses compute work ("Zero GPU Pods"). Root cause: a stopped Pod stays bound to its physical host; another user rented that GPU while it was stopped.

[Credential-related source omitted]

[Credential-related source omitted]

**Entry points** (all equivalent): web console (`cloud.vast.ai`), the `vastai` CLI / Python SDK, the REST API (`https://console.vast.ai/api/v1/...`, Bearer token), and SSH into the running container. The CLI is the

API (`https://console.vast.ai/api/v1/...`, Bearer token), and SSH into the running container. The CLI is the

[Credential-related source omitted]

**China relevance: none at the platform level.** No China datacenters, no `/etc/network_turbo` equivalent, no built-in HF mirror. The HF-unreachable problem still exists at the *workload* level from some hosts, but the fix is the job's **ow

built-in HF mirror. The HF-unreachable problem still exists at the *workload* level from some hosts, but the fix is the job's **own** `HF_ENDPOINT=https://hf-mirror.com` / `hf_transfer`, not a platform script — see `references/run-remote/go

fix is the job's **own** `HF_ENDPOINT=https://hf-mirror.com` / `hf_transfer`, not a platform script — see

[Credential-related source omitted]

DURABLE_DIR=off-box              # no destroy-surviving mount: vastai copy / vastai cloud copy before destroy (§5) # PROXY_HOOK — none at platform level (no /etc/network_turbo). HF mirror is the JOB's own env if needed: PROXY_HOOK=''       

# PROXY_HOOK — none at platform level (no /etc/network_turbo). HF mirror is the JOB's own env if needed: PROXY_HOOK=''                    # set HF_ENDPOINT=https://hf-mirror.com in the job env only if a host can't reach HF # CRED_FILE — emp

PROXY_HOOK=''                    # set HF_ENDPOINT=https://hf-mirror.com in the job env only if a host can't reach HF

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

| Channel | Set | Endpoint(s) | |---|---|---| | **PyPI** | `pip config set global.index-url <url>` or `pip install -i <url> pkg` | Tsinghua TUNA `https://pypi.tuna.tsinghua.edu.cn/simple` · Aliyun `https://mirrors.aliyun.com/pypi/simple` · 

|---|---|---| | **PyPI** | `pip config set global.index-url <url>` or `pip install -i <url> pkg` | Tsinghua TUNA `https://pypi.tuna.tsinghua.edu.cn/simple` · Aliyun `https://mirrors.aliyun.com/pypi/simple` · USTC `https://pypi.mirrors.ustc.

| **PyPI** | `pip config set global.index-url <url>` or `pip install -i <url> pkg` | Tsinghua TUNA `https://pypi.tuna.tsinghua.edu.cn/simple` · Aliyun `https://mirrors.aliyun.com/pypi/simple` · USTC `https://pypi.mirrors.ustc.edu.cn/simple`

| **conda** | channels in `~/.condarc` (TUNA Anaconda) | `https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main` + `.../free` + the `cloud/` channels (pytorch, conda-forge) | | **HuggingFace** | `export HF_ENDPOINT=https://hf-mirror.com` 

| **HuggingFace** | `export HF_ENDPOINT=https://hf-mirror.com` | drop-in reverse proxy — identical repo IDs, identical `hf download` / `from_pretrained` calls |

Source: HF-Mirror `https://hf-mirror.com/`; TUNA PyPI `https://mirrors.tuna.tsinghua.edu.cn/help/pypi/`;

TUNA Anaconda `https://mirrors.tuna.tsinghua.edu.cn/help/anaconda/`; ModelScope client

`https://github.com/modelscope/modelscope_hub`.

# --- mirror routing --- export HF_ENDPOINT=https://hf-mirror.com           # MUST precede any HF import (see trap below)

export HF_ENDPOINT=https://hf-mirror.com           # MUST precede any HF import (see trap below)

# Inline on the command — the env is set before the interpreter starts: HF_ENDPOINT=https://hf-mirror.com python train.py

HF_ENDPOINT=https://hf-mirror.com python train.py

export HF_ENDPOINT=https://hf-mirror.com   # then later: python -m src.train ...

Source: HF-Mirror `https://hf-mirror.com/`; ModelScope client

`https://github.com/modelscope/modelscope_hub`.

Source: hf CLI resume `https://github.com/huggingface/huggingface_hub/issues/3580`; hf_transfer hang `https://github.com/huggingface/hf_transfer/issues/30`; ModelScope download

`https://github.com/huggingface/hf_transfer/issues/30`; ModelScope download

`https://deepwiki.com/modelscope/modelscope/3.1-model-download-and-caching`.

Source: requests `no_proxy` `https://github.com/psf/requests/issues/4871`; no_proxy guide

`https://www.browserstack.com/guide/no_proxy-environment-variable`; Clash pip ProxyError

`https://github.com/clash-verge-rev/clash-verge-rev/issues/2607`.

1. **hf-mirror first** — `HF_ENDPOINT=https://hf-mirror.com`. Drop-in, same repo IDs, no proxy, no

Source: HF-Mirror `https://hf-mirror.com/`; ModelScope `https://github.com/modelscope/modelscope_hub`.

[Credential-related source omitted]

[Credential-related source omitted]

(`profiles/generic-ssh.md`, `profiles/lambda.md`). Before blaming the panel, verify ground truth: the event file is non-empty (`ls -la <logdir>; du -sh <logdir>`) and TB answers locally (`curl -s localhost:<port>/ | head`). For curves that 

panel, verify ground truth: the event file is non-empty (`ls -la <logdir>; du -sh <logdir>`) and TB answers locally (`curl -s localhost:<port>/ | head`). For curves that must **survive teardown**, don't depend on a box-local panel at all → 

answers locally (`curl -s localhost:<port>/ | head`). For curves that must **survive teardown**, don't

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

- [ ] **The node runs itself before anything local can wait on a human.** Start the on-node chain first — probe → gate → dead-man (`scripts/run_one.sh.template`) → run → power-off — and only then take local steps that can stop on a question

URL: https://support.crusoecloud.com/hc/en-us/articles/46061806112155-NCCL-Hangs-and-Multi-Node-Training-Stalls-Caused-by-Failed-nvidia-fabricmanager

URL: https://github.com/NVIDIA/nccl/issues/1580

URL: https://repost.aws/questions/QURXddiuikQLesRDGz39RhIw/nccl-socket-timeout-when-using-large-dataset-in-multi-node-pretraining

URL: https://github.com/moby/moby/issues/4378

URL: https://docs.pytorch.org/tutorials/beginner/ddp_series_fault_tolerance.html

URL: https://horovod.readthedocs.io/en/stable/elastic_include.html

| Platform | Detection signal | Grace window | Implication | |---|---|---|---| | **AWS EC2 Spot** | IMDS `http://169.254.169.254/latest/meta-data/spot/instance-action` (404 = none, 200 = pending); rebalance-recommendation fires ~10–20 min e

|---|---|---|---| | **AWS EC2 Spot** | IMDS `http://169.254.169.254/latest/meta-data/spot/instance-action` (404 = none, 200 = pending); rebalance-recommendation fires ~10–20 min earlier | **~120 s** | On-signal flush of a *small* checkpoint

| **AWS EC2 Spot** | IMDS `http://169.254.169.254/latest/meta-data/spot/instance-action` (404 = none, 200 = pending); rebalance-recommendation fires ~10–20 min earlier | **~120 s** | On-signal flush of a *small* checkpoint is viable; still 

URLs: AWS [spot-instance-termination-notices](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-instance-termination-notices.html),

[rebalance-recommendations](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/rebalance-recommendations.html);

GCP [preemptible](https://docs.cloud.google.com/compute/docs/instances/preemptible),

[spot](https://docs.cloud.google.com/compute/docs/instances/spot);

Azure [scheduled-events](https://learn.microsoft.com/en-us/azure/virtual-machines/windows/scheduled-events);

Slurm [sbatch `--signal`](https://slurm.schedmd.com/sbatch.html);

RunPod [spot-vs-on-demand](https://www.runpod.io/blog/spot-vs-on-demand-instances-runpod);

vast.ai [Rental-Types](https://vast.ai/article/Rental-Types).

URLs: Young/Daly [robustness paper, INRIA](https://people.bordeaux.inria.fr/gaupy/ressources/pub/confs/icpp20_robustness.pdf),

[Optimal Checkpointing Period, LAWN 281](https://www.netlib.org/lapack/lawnspdf/lawn281.pdf),

[Optimal Checkpointing for Iterative Applications, IEEE](https://ieeexplore.ieee.org/document/9495174/).

URLs: [Check-N-Run, arXiv](https://arxiv.org/pdf/2010.08679),

[SkyPilot training-guide](https://docs.skypilot.co/en/latest/reference/training-guide.html),

[SageMaker resume-from-checkpoint](https://docs.aws.amazon.com/sagemaker/latest/dg/model-checkpoints-resume.html).

region/cloud to chase capacity, then re-runs the task). Caveat: it auto-recovers **only**   preemption/hardware failures — a user-code non-zero exit is **not** auto-recovered.   [managed-jobs](https://docs.skypilot.co/en/latest/examples/man

preemption/hardware failures — a user-code non-zero exit is **not** auto-recovered.   [managed-jobs](https://docs.skypilot.co/en/latest/examples/managed-jobs.html). - **AWS SageMaker Managed Spot** — set `use_spot_instances=True` + `checkpo

[managed-jobs](https://docs.skypilot.co/en/latest/examples/managed-jobs.html).

**`max_wait` must be greater than `max_run`** — `max_wait` covers wait-for-capacity *plus* run time   *plus* interruption gaps; set it too tight and the job is killed mid-resume.   [managed-spot docs](https://docs.aws.amazon.com/sagemaker/l

[managed-spot docs](https://docs.aws.amazon.com/sagemaker/latest/dg/model-managed-spot-training.html).

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]

**Symptom**: a fine-tuned causal LM produces garbage, or DPO/batched-generation logprobs disagree with single-example decoding. **Root cause**: causal-LM **training** wants **right-padding** (pad lands after content, attention mask zeroes i

**Fix**: `tokenizer.padding_side="right"` for SFT collation; `"left"` for generation/eval/DPO — set it per phase, not globally. ([HF causal-LM](https://huggingface.co/docs/transformers/tasks/language_modeling), [TRL DPO](https://huggingface

**Symptom**: SFT "trains" but parrots the prompt / barely follows instructions; loss plausible but flat. **Root cause**: HF LM loss is `CrossEntropyLoss(ignore_index=-100)` — only `-100` positions are skipped. Leaving the prompt-prefix labe

**Fix**: set labels to `-100` at **both** the prompt prefix (train only on the completion) and all padding positions. TRL `SFTTrainer` `completion_only_loss` / `DataCollatorForCompletionOnlyLM` does the prefix masking — verify it fired (dec

**Symptom**: throughput jumps after enabling packing, but quality drops vs unpacked; the model "completes" one doc with content from a packed neighbor. **Root cause**: naive packing concatenates examples into one `max_len` sequence; a vanil

**Fix**: **document masking** — emit `position_ids` that reset per sub-sequence + an attention impl that honors boundaries. TRL/HF `DataCollatorWithFlattening` packs into one stream, returns `position_ids`, and sets each example's first lab

**Symptom**: a 4k-context model is incoherent past ~4k at inference even when trained on longer sequences; or long-context finetune won't converge. **Root cause**: RoPE frequencies are calibrated to the pretrain context; longer positions ex

**Fix**: set `rope_scaling={"type":"linear"|"dynamic"|"yarn","factor":<target/orig>}` in config and finetune **with scaling active** (`"yarn"` for big jumps, `"linear"` only ≤4×). Train-time vs inference-time `rope_scaling` mismatch is a si

**Symptom**: pretraining/long-SFT loss is stable then spikes; in bf16/fp16 it can NaN; logits grow unboundedly over training. **Root cause**: the softmax normalizer `log Z` drifts from 0 as logits grow → low-precision overflow + gradient in

**Fix**: add **z-loss** `1e-4 · log²(Z)` (the PaLM / ST-MoE coefficient) to pull `log Z` toward 0. The general divergence ladder (warmup, grad-clip, skip-the-batch, qk-norm, bf16-over-fp16) is **references/training/precision-stability.md** 

**Symptom**: reported PPL implausible, or differs from a published number on the same checkpoint+data. **Root cause**: PPL = `exp(mean NLL over scored tokens)`. Including pad/prompt tokens, or a sliding window that double-counts overlap con

**Fix**: score only non-`-100` positions; for long docs use the HF strided window where overlap tokens are `-100` (context, not scored). Whether the number is comparable across runs is metric validity, not mechanics. ([HF perplexity](https:

**Symptom**: TRL trainer runs but learns nothing, or errors on a missing column; preference data in an SFT trainer (or vice versa) silently mistrains. **Root cause**: **SFT** = prompt+completion (train on completion); **DPO/preference** = `

**Fix**: match trainer to schema; for conversational data confirm the chat template fired (decode one example — look for role tags). Prefer the **explicit-prompt** form `{prompt, chosen, rejected}` over implicit. Recommended order SFT → DPO

**Symptom**: `rewards/margins` ~0 or `rewards/accuracies` ~0.5; or `logps/chosen` and `logps/rejected` both plunge (suppresses everything). **Root cause**: `beta` controls deviation from the frozen reference — too **small** → policy drifts 

**Fix**: start `beta=0.1`, raise to 0.3–0.5 if text degrades, lower if the margin won't move. Use `learning_rate≈1e-6` (TRL DPO default; `≈1e-5` for LoRA) — too high is the classic collapse. Health signal: `rewards/margins` ↑, `rewards/accu

**Symptom**: `401`/`GatedRepoError` when the Trainer loads a Llama/Gemma/Mistral base despite granted access; or `push_to_hub` can't write. **Root cause**: the token must be visible to the process **before** the gated `from_pretrained` / Tr

[Credential-related source omitted]

[Credential-related source omitted]

**Symptom**: training loss falls but val/test is near-chance; or a fine-tuned backbone is far worse than its pretrained eval. **Root cause**: pretrain `(mean,std)` (ImageNet `mean=[.485,.456,.406] std=[.229,.224,.225]`) differs between trai

**Fix**: use the **exact** pretrain normalization, identically in train and eval; match channel order. Print one input tensor's per-channel mean/std — should be ~`N(0,1)`; a gap that survives this fix is a real effect, not the bug. ([why-no

**Symptom**: detection loss decreases normally but mAP is exactly 0 (or ~0) at every eval. **Root cause**: a format mismatch the loss tolerates but eval doesn't — (1) box format `cxcywh`/`xywh` vs evaluator's `xyxy`, or normalized `[0,1]` v

**Fix**: assert the eval pipeline's box format + class indexing, convert explicitly (`torchvision.ops.box_convert`), and visualize 2–3 predicted boxes before trusting the metric. mAP=0 with healthy loss is almost never the model — it's eval

**Symptom**: seg loss trains but mIoU is 0 (or a class NaN); or loss is NaN from step 1. **Root cause**: label/class-index inconsistency — a void value (commonly `255`) not excluded → treated as a class id ≥ `num_classes` (out-of-range / po

**Fix**: set the **same** `ignore_index` in **both** loss and metric — `CrossEntropyLoss(ignore_index=255)` and mIoU mask `(label != 255)`; confirm `label.max() < num_classes` after any shift; apply reduction identically. mIoU=0 with fallin

**Symptom**: the noise-prediction MSE converges nicely but samples are blurry, mode-collapsed, or wrong. **Root cause**: diffusion loss (predict noise at a random timestep) is **weakly correlated with sample quality** — good average noise-p

**Fix**: the textbook **is-the-number-real** fork → cross-link **references/verifying/methodology.md** (**REQUIRED**; it owns loss-low-output-bad). Mechanically walk DF2→DF6; the single most common miss is evaluating **raw** weights instead

**Symptom**: samples from the just-saved checkpoint look worse than expected; quality jumps with an EMA checkpoint. **Root cause**: diffusion quality depends heavily on EMA — a running average (`decay≈0.999`, ~1000-update window) that denoi

**Fix**: maintain EMA during training and **sample/evaluate from EMA weights** (`diffusers` `EMAModel`); verify the eval path actually loaded EMA. Checkpointing the EMA shadow weights (and what breaks when they are lost) → `references/train

**Symptom**: latent-diffusion (SD-style) training is unstable / produces noise or blank output; or a swapped-in custom VAE degrades everything. **Root cause**: the model assumes ~unit-variance latents; the VAE output is multiplied by a cali

**Fix**: scale by the VAE's `config.scaling_factor` on encode, divide on decode. For a **custom** VAE, **measure** empirical latent std on a sample and set factor `1/std` — don't inherit `0.18215`. Print latent mean/std before training (~0/

**Symptom**: changing `guidance_scale` at inference barely changes output, or the model ignores conditioning. **Root cause**: CFG needs a learned **unconditional** path, trained by randomly replacing the condition with a null embedding for 

**Fix**: during training drop the condition `p_drop≈0.1` (replace with null embedding). At inference use `guidance_scale` ~5–15 for T2I (higher = more prompt adherence, lower diversity). Model-ignores-input is DF1's fork; the training-side 

**Symptom**: samples are systematically blurry / low-detail despite long training. **Root cause**: uniform loss weight over-weights high-noise (easy) steps relative to low-noise steps that carry fine detail; the gradient is dominated by the

**Fix**: apply **Min-SNR-γ** weighting (γ≈5) so low-noise steps get their share; `diffusers` scripts expose `--snr_gamma`. Compounds with DF2/DF3 — fix these details together, not in isolation. ([Min-SNR](https://arxiv.org/abs/2303.09556))

**Symptom**: training or fine-tuning an SD3 / Flux-class **flow-matching** model; the loss is low and stable but samples are noisy/structureless — and noticeably worse away from the training resolution. **Root cause**: flow matching predict

**Fix**: keep training and sampling on the **identical** schedule — same flow-matching scheduler, same `shift`/`use_dynamic_shifting`, and set `mu` from the actual image sequence length at inference. A mismatched scheduler/`prediction_type`

**Symptom**: average reward suddenly drops, responses get short/repetitive or refuse, length collapses. **Root cause**: reward hacking / over-optimization — the policy exploits the reward model's blind spots and drifts far, often after the 

**Fix**: strengthen the KL penalty (raise the KL coefficient), lower LR (`≈1e-6` for LLM PPO), reduce PPO update epochs/batch, add a length penalty if length is gamed. Watch reward **and** KL together — a reward jump with a KL spike is hack

**Symptom**: KL grows without bound; generations go incoherent; "diverges" though loss isn't NaN. **Root cause**: the KL penalty is too weak (or adaptive-KL target too loose); aggressive updates push the policy far, and a huge KL term can d

**Fix**: use an **adaptive KL controller** with a target (e.g. 6), or a fixed coefficient large enough to hold KL bounded; clip the ratio (`cliprange≈0.2`); cap update epochs. Confirm the frozen reference isn't being updated. Same axis DPO'

**Symptom**: high-variance, brittle training; small reward-scale changes destabilize everything. **Root cause**: raw reward/advantage magnitudes vary wildly across batches; PPO gradients are scale-sensitive. **Fix**: **whiten** advantages p

**Fix**: **whiten** advantages per minibatch (subtract mean, divide by std) — the standard PPO trick, more stabilizing than plain reward normalization (double-normalizing reward **and** advantage is often redundant). For classic-control RL,

**Symptom**: identical hyperparameters + different seeds give non-overlapping curves; an ablation "win" disappears on re-run. **Root cause**: extreme seed variance from algorithm, policy sampling, and environment stochasticity (comparing 5-

**Fix**: report aggregate over **≥5 seeds** (more for noisy envs), use **IQM** (interquartile mean) over mean/median, show CIs. A single-seed delta is not a result — squarely **references/verifying/methodology.md** territory (bug-vs-real-ef

**Symptom**: a LLaVA-style VLM doesn't ground on images (ignores visual tokens), or text quality collapses after multimodal finetune. **Root cause**: VLM training is **staged**, each stage freezing different towers. Stage 1 (alignment): fre

**Fix**: set `requires_grad` per tower per stage per the recipe; print trainable-param counts at each stage start to confirm the freeze took. "Ignores its input image" is model-ignores-input → X5. ([LLaVA recipe](https://rohitbandaru.github

**Symptom**: stage-1 alignment loss is unstable or the projector output is garbage. **Root cause**: the projector (linear or 2-layer MLP, often concatenating groups of vision tokens into the LLM embedding space) is **randomly initialized**;

**Fix**: stage 1 trains the projector **alone** against a frozen LLM so it learns the LLM's embedding space first; only then (stage 2) unfreeze the LLM. Confirm projector output dim == LLM hidden size (else a silent shape-broadcast bug). ([

**Symptom**: a shared LR either over-updates the pretrained vision encoder (forgets visual features) or leaves projector/LLM undertrained. **Root cause**: towers are at different maturities — a pretrained vision encoder needs a tiny LR (or 

**Fix**: **parameter groups** with per-group LRs (`AdamW([{"params":proj.parameters(),"lr":1e-3},{"params":llm.parameters(),"lr":2e-5}])`), or freeze the vision encoder. Log each group's LR to confirm. ([LLaVA LRs](https://rohitbandaru.gith

**Symptom**: VLM training errors on image-token-count mismatch, or intermittently ignores images on long examples. **Root cause**: image placeholders expand into many tokens (hundreds per image); a `max_length` truncation cuts them, breakin

**Fix**: set `max_length=None` (or large enough never to truncate image tokens) for VLM trainers, or verify truncation never removes placeholders across the whole dataset. Count image tokens vs expected per example as a smoke check. ([TRL V

**Fix**: every checkpoint must carry the full state (PyTorch tutorial [saving multiple / general checkpoint](https://docs.pytorch.org/tutorials/recipes/recipes/saving_and_loading_a_general_checkpoint.html);

[saving multiple / general checkpoint](https://docs.pytorch.org/tutorials/recipes/recipes/saving_and_loading_a_general_checkpoint.html);

offloading each to CPU on rank 0, so peak GPU memory stays bounded and non-rank-0 workers skip the GPU→CPU copy entirely ([HF Accelerate FSDP guide](https://huggingface.co/docs/accelerate/en/usage_guides/fsdp),

copy entirely ([HF Accelerate FSDP guide](https://huggingface.co/docs/accelerate/en/usage_guides/fsdp), [Lightning issue #11207](https://github.com/Lightning-AI/pytorch-lightning/issues/11207)). The full dict

([HF Accelerate FSDP guide](https://huggingface.co/docs/accelerate/en/usage_guides/fsdp),

[Lightning issue #11207](https://github.com/Lightning-AI/pytorch-lightning/issues/11207)). The full dict

**Fix**: use `torch.distributed.checkpoint` (DCP), the current PyTorch-2.x sharded-checkpoint API ([DCP recipe](https://docs.pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html),

([DCP recipe](https://docs.pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html),

[2.12 reference](https://docs.pytorch.org/docs/2.12/distributed.checkpoint.html)). **Save**: get canonical

**Fix** ([DeepSpeed model-checkpointing](https://deepspeed.readthedocs.io/en/stable/model-checkpointing.html),

[ZeRO tutorial](https://www.deepspeed.ai/tutorials/zero/)):

**Fix** ([Trainer docs](https://huggingface.co/docs/transformers/main/en/main_classes/trainer)):

**Fix** ([Accelerate checkpoint guide](https://huggingface.co/docs/accelerate/en/usage_guides/checkpoint)):

`.ckpt` is a full dump — epoch, global step, LightningModule `state_dict`, **all** optimizer + LR-scheduler states, callback states, loop state, and the 16-bit scaling factor (AMP) ([Lightning checkpointing basics](https://lightning.ai/docs

([Lightning checkpointing basics](https://lightning.ai/docs/pytorch/stable/common/checkpointing_basic.html)).

**Root cause**: the `GradScaler` holds dynamic state — `scale`, `growth_factor`, `backoff_factor`, `growth_interval`, `_growth_tracker` — that evolves during training; dropping it resets the scaler ([PyTorch AMP recipe](https://docs.pytorch

`growth_interval`, `_growth_tracker` — that evolves during training; dropping it resets the scaler ([PyTorch AMP recipe](https://docs.pytorch.org/tutorials/recipes/recipes/amp_recipe.html), [forum: No inf checks were recorded](https://discu

([PyTorch AMP recipe](https://docs.pytorch.org/tutorials/recipes/recipes/amp_recipe.html),

[forum: No inf checks were recorded](https://discuss.pytorch.org/t/resume-training-with-mixed-precision-lead-to-no-inf-checks-were-recorded-for-this-optimizer/115828)).

**Symptom**: pre-resume eval (using EMA weights) is good; post-resume eval drops sharply, then recovers

- **It isn't learning at all (start here)** — O1 overfit-one-batch · O2 params-not-in-optimizer · O3 loss-detached-from-graph · O4 zero_grad/backward/step-order · O5 train()/eval()-mode

**Symptom**: training "runs" (no error, normal throughput) but loss plateaus near its init value or wanders without trending down, across LRs/optimizers/architectures. You're tuning hyperparameters blind because nothing proves the loop can 

**Fix**: take ONE fixed mini-batch (2 examples is enough) and loop forward/backward/step on **that same batch** for hundreds of iters — a correct loop drives train loss → ~0. Turn **off** augmentation, shuffling, dropout, and weight decay f

**Symptom**: overfit-one-batch fails; a snapshotted param is unchanged before/after `optimizer.step()`; grad-norm may even be nonzero. No error. **Root cause**: the optimizer updates a **different** set of tensors than the model forwards th

**Fix**: build the optimizer **after** `model.to(device)`. Assert it sees every trainable param: `opt_ids={id(p) for g in optimizer.param_groups for p in g['params']}; assert all(id(p) in opt_ids for p in model.parameters() if p.requires_gr

**Symptom**: overfit fails with every `p.grad is None`; or `loss.backward()` raises *"element 0 of tensors does not require grad and does not have a grad_fn"*. **Root cause**: the loss tensor was severed from autograd before `backward`. Com

**Fix**: before `backward`, `assert loss.requires_grad and loss.grad_fn is not None`. Keep the differentiable loss tensor distinct from logging scalars (log `loss.item()`, back-prop the raw tensor). Reserve `no_grad`/`inference_mode` for ev

**Symptom**: overfit fails; weights never move, or training is erratic despite nonzero grads. **Root cause**: PyTorch's contract is *"gradients by default add up; to prevent double-counting we explicitly zero them each iteration"*, `backwar

**Fix**: the canonical order, exactly — `optimizer.zero_grad(set_to_none=True)` → forward → `loss=loss_fn(out,y)` → `loss.backward()` → `optimizer.step()` (under AMP: `scaler.scale(loss).backward()` → `scaler.step(optimizer)` → `scaler.upda

**Symptom**: two faces — (1) trained under `eval()`: BN uses frozen running stats and never updates them, Dropout is off → underfits / loss barely moves; (2) evaluated under `train()`: BN uses noisy per-batch stats and Dropout fires → val l

**Symptom**: two faces — (1) trained under `eval()`: BN uses frozen running stats and never updates them, Dropout is off → underfits / loss barely moves; (2) evaluated under `train()`: BN uses noisy per-batch stats and Dropout fires → val l

**Root cause**: `train()`/`eval()` set a per-module flag that *"has an effect only on certain modules ... e.g. Dropout, BatchNorm"* (`eval()` == `train(False)`). In eval mode BN switches to stored `running_mean/var` and **stops** updating t

**Fix**: bracket phases explicitly — `model.train()` atop each train epoch; `model.eval()` + `with torch.no_grad():` for every val/test pass; `model.train()` again before resuming. After build/load, `assert model.training` before the train 

**Symptom**: `weight_decay` on `torch.optim.Adam` barely regularizes (or hurts) vs the literature's AdamW recipe; or a from-scratch transformer/CNN trains worse than a reference at the "same" wd; or small models destabilize when LayerNorm/B

**Fix**: use `torch.optim.AdamW`, not `Adam(weight_decay=...)`. Split into two param groups with `weight_decay=0.0` on the no-decay group — nanoGPT's rule: decay `p.dim()>=2` (matmul/embedding weights), no-decay `p.dim()<2` (all biases + al

**Symptom**: no divergence, no NaN, grads finite — loss just falls glacially or plateaus high; throughput is fine but the model "won't learn." Often after copying an LR from a different-batch/optimizer recipe or defaulting to a tiny "safe" 

**Fix**: run an **LR range test** (Smith) — from a tiny LR, multiply it geometrically each batch over ~100–1000 steps, plot loss vs LR, pick ~1 decade below where loss starts to diverge. Tools: `pytorch-lr-finder` `LRFinder(model,opt,crit).

**Symptom**: PyTorch warns *"Detected call of `lr_scheduler.step()` before `optimizer.step()`"* and the LR curve is off-by-one; OR a cosine/warmup schedule sized in optimizer steps barely moves (stepped per-epoch) or decays to ~0 in one epo

**Fix**: order it `optimizer.step(); scheduler.step()`. Step at the granularity its `total_steps` was computed in — per optimizer step for warmup/cosine/OneCycle (inside the `if (i+1)%accum==0` block, **not** every micro-batch), per epoch o

**Symptom**: switching from batch `B` to (micro-batch `B/N` × N accumulation) "at the same config" trains hotter/diverges — loss/grad magnitude ~N× too big, i.e. you silently get N× the LR. For token tasks the accumulated loss also differs 

**Fix**: divide before backward — `loss = loss_fn(out,y) / accum_steps; loss.backward()`, with `step()`/`zero_grad()` only on the boundary. For token-level losses, normalize by the **total** non-pad tokens across the accumulation window (ac

**Symptom**: a run stable in fp32 develops update spikes/NaNs once optimizer math is half precision; or AdamW behaves as if `eps=0` (huge updates where the second moment `v` is small). Most visible with fp16 optimizer states or foreach/8-bi

**Fix**: raise eps for half-precision optimizer math — `eps=1e-7` (proposed in pytorch#26218 for fp16) up to `1e-6` for bf16; or keep optimizer states / master weights in **fp32** (FSDP `MixedPrecision`, DeepSpeed bf16 keep an fp32 master) 

**Symptom**: `AdamW(fused=True)` raises (e.g. on `_foreach_sub_` of `device_found_inf`) or mis-steps under GradScaler / bf16-mixed / FSDP; **or** the default `foreach` path OOMs at the optimizer step on a model that fit during forward/backw

**Fix**: on a fused error/suspicious step under AMP/FSDP/bf16-mixed, fall back to `fused=False` (lets `foreach` default) or upgrade past the fixed issue — confirm a parity loss-curve before trusting fused for a real datapoint. If the **step

**Symptom**: a model with a softmax (or log_softmax) final layer trains far slower than expected, plateaus high, or barely learns; loss is sluggish but not NaN. Classic when porting a Keras/TF model (expects probabilities) to PyTorch, or af

**Fix**: pass **raw logits** of shape `(N,C)` — remove any `nn.Softmax`/`F.log_softmax`/`nn.LogSoftmax` from the head. Apply softmax only at inference (for probabilities) or argmax (for the class). If you genuinely need log-probs in-graph, 

**Symptom**: a binary / multi-label head shows NaN or inf loss (often once outputs saturate toward 0/1), or spiky loss; the model has an explicit `torch.sigmoid` before `nn.BCELoss`. Under imbalance it also collapses to always predicting th

**Fix**: feed **raw logits** to `nn.BCEWithLogitsLoss` — it fuses sigmoid+BCE with the log-sum-exp trick, avoiding `log(0)`. Remove the explicit sigmoid (apply only at inference). For imbalance pass `pos_weight = #neg/#pos` per class (`>1` 

**Symptom**: any of — `RuntimeError: 0D or 1D target tensor expected, multi-target not supported` (one-hot target); `expected scalar type Long but found Float`; `IndexError: Target N is out of bounds` / CUDA `device-side assert ... t >= 0 &

**Fix**: hard labels → `targets.long()` of shape `(N,)`; remap ids to contiguous `0..C-1` (`{orig:i for i,orig in enumerate(sorted(set(labels)))}`; subtract 1 if 1-indexed); `assert targets.min()>=0 and targets.max()<C`. Don't one-hot the s

**Symptom**: a seq/NLP model's loss looks suspiciously small from step 0 and scales with how much padding is in the batch (more pad → lower loss); the model under-learns real tokens; changing batch size or max-length changes the loss magnit

**Fix**: skip padding. Easiest: `nn.CrossEntropyLoss(ignore_index=PAD_ID)` — *"the loss is averaged over non-ignored targets"* (sums valid positions, divides by valid count). Otherwise compute `reduction='none'`, multiply by a 0/1 mask, and

**Symptom**: a model uses `nn.NLLLoss` but has no `LogSoftmax`/`F.log_softmax` before it (or a plain `Softmax`): training "runs" with no error but loss is nonsensical / won't converge, accuracy stuck near chance. **Root cause**: `nn.NLLLoss

**Fix**: put `F.log_softmax(logits, dim=1)` (or an `nn.LogSoftmax(dim=1)` final layer) immediately before `nn.NLLLoss` (class dim = 1 for `(N,C)`). Simpler and less error-prone: drop NLLLoss+LogSoftmax and use `nn.CrossEntropyLoss` on raw l

**Symptom**: you set `requires_grad=False` on the backbone (or set it **after** building the optimizer over `model.parameters()`), yet the frozen weights keep moving every step; pretrained features drift and degrade though no real gradient 

**Fix**: exclude frozen params from the optimizer at construction — `optim.SGD([p for p in model.parameters() if p.requires_grad], lr=...)` (or per-module param groups). If you froze after building the optimizer, rebuild it, or set `param.g

**Symptom**: the backbone is "frozen" (`requires_grad=False`) yet val accuracy is unstable / worse than train, or `eval()` vs `train()`-mode inference disagree; small fine-tuning batches make it worse. The frozen features keep shifting batc

**Symptom**: the backbone is "frozen" (`requires_grad=False`) yet val accuracy is unstable / worse than train, or `eval()` vs `train()`-mode inference disagree; small fine-tuning batches make it worse. The frozen features keep shifting batc

**Fix**: put the frozen Norm layers in eval mode after `model.train()`: `for m in backbone.modules():\n    if isinstance(m,(nn.BatchNorm1d,nn.BatchNorm2d,nn.BatchNorm3d)): m.eval()` — or build them `track_running_stats=False`. Re-apply ever

**Symptom**: fine-tuning with a single LR either (too high) destroys the pretrained representations on the first updates and accuracy collapses below a frozen-feature baseline, or (too low) the random new head can't move. Both are the same 

**Fix**: discriminative fine-tuning — per-layer param groups with LR decaying toward the input (head highest, stem lowest), e.g. `AdamW([{'params':head,'lr':1e-3},{'params':backbone,'lr':1e-5}])`. Combine with **gradual unfreezing** (train 

**Symptom**: you replaced the classifier for a new `num_classes` and pass `strict=False` expecting it to skip the head, but loading still crashes: `RuntimeError: ... size mismatch for fc.weight: copying a param with shape [1000,...] ..., th

**Fix**: drop the incompatible head entries before loading, then load non-strict — `sd={k:v for k,v in ckpt.items() if not k.startswith('fc.')}; missing,unexpected = model.load_state_dict(sd, strict=False)` — and inspect `missing/unexpected

**Symptom**: after `get_peft_model(...)`, `print_trainable_parameters()` shows ~0% (or far fewer than expected) and loss won't drop; or a PEFT classifier reloads with a **random** head / shifted metrics after `save_pretrained`. **Root cause

**Fix**: enumerate real names with `[n for n,_ in model.named_modules()]` and set `LoraConfig(target_modules=[...])` (or `'all-linear'`); confirm with `model.print_trainable_parameters()` and that you see `lora.Linear` layers. Add the new h

**Symptom**: loss barely moves (under-stepping) or is jittery (over-stepping), and the bare grad-norm can't tell you which — it isn't scale-relative to the weights. **Root cause**: what matters is the size of the **actual update** relative 

**Fix**: log it every K steps — snapshot `w0={n:p.detach().clone() for n,p in model.named_parameters()}` before `step()`, then `(p.detach()-w0[n]).norm()/(w0[n].norm()+1e-12)` per name after. Lever: `≪1e-3` → raise that group's LR; `≫1e-3` 

**Symptom**: you log `cfg.lr` (a constant) so the dashboard LR is flat — yet you're on warmup+cosine. You can't see warmup, decay, a restart, or a frozen scheduler; LR-related loss behavior (spike on ramp, stall at the floor) is invisible. 

**Fix**: log `scheduler.get_last_lr()` (a list — one per param group; log them all if you use differential LRs) or read `optimizer.param_groups[0]['lr']` directly, every step. Don't use `get_lr()` for logging. If the logged LR plateaus when

**Symptom**: capacity quietly vanishes — a layer's outputs are increasingly all-zero, loss plateaus above where it should, and adding width doesn't help. No crash; it just under-fits. Worst case the net degenerates toward a constant functio

**Fix**: instrument the zero/saturation fraction per activation with a forward hook — `(out==0).float().mean()` for ReLU (or `|out|>0.99` for tanh/sigmoid), logged every K steps per layer. Healthy: a stable modest dead fraction (ReLU is spa

**Symptom**: scalar dashboards (loss, one grad-norm) look fine yet the model under-performs or destabilizes — a mean/norm hides the shape: activations drifting to a saturated tail, weights collapsing to a spike at 0 (a layer dying, O25), or

**Fix**: periodically (every few hundred steps — histograms aren't free) log `SummaryWriter.add_histogram(tag, values, global_step)` for each layer's **weights**, its **gradients** (after `backward`, before `zero_grad`), and key **activatio

**Symptom**: with `num_workers>0`, different workers emit the **same** random augmentation parameters (same crop coords, flips, noise) within a batch, and the exact same random sequence repeats **every epoch**. Augmentation diversity collap

**Fix**: pass a `worker_init_fn` that reseeds numpy from torch's already-per-worker seed: `def wif(_): np.random.seed(torch.initial_seed() % 2**32)`. `torch.initial_seed()` = `base_seed+worker_id` and `base_seed` is redrawn each epoch, givi

**Symptom**: an `IterableDataset` with `num_workers=N` yields each sample **N times** (an "epoch" is N× too long, samples repeat within a batch); and under DDP every **rank** streams the **same** data, so `all_reduce` averages identical gra

**Fix**: shard by **both** dimensions inside `__iter__`. Workers: `wi=torch.utils.data.get_worker_info()`, then keep records where `idx % wi.num_workers == wi.id` (or contiguous ranges). Ranks: fold in `dist.get_rank()`/`get_world_size()` —

**Symptom**: after correctly sharding an `IterableDataset` by rank, training intermittently **hangs at the last batch** of an epoch (NCCL collective timeout), or some ranks run one extra step. **Root cause**: streaming shards rarely divide 

**Fix**: make every rank run the **same** number of steps — (a) compute a global min steps/epoch and stop all ranks there (drop the ragged tail), (b) pad short shards by cycling samples, or (c) wrap the loop in the DDP **Join** context mana

**Symptom**: iteration crashes at batch assembly — `RuntimeError: stack expects each tensor to be equal size, but got [..] at entry 0 and [..] at entry 1` — for variable-length sequences, variable bbox counts, or differently-sized images/ma

**Fix**: pass `DataLoader(..., collate_fn=my_collate)`. Sequences: `pad_sequence(seqs, batch_first=True, padding_value=pad_id)` + emit a length/attention mask (then mask the loss → O15, by-domain L2). Detection-style ragged targets: keep th

**Symptom**: after wrapping batches in a custom class (a `Batch` object, a graph batch, a dataclass) and setting `pin_memory=True`, the async H2D copy (`.to('cuda', non_blocking=True)`) no longer overlaps — throughput regresses to a blockin

**Fix**: implement `def pin_memory(self): self.x=self.x.pin_memory(); self.y=self.y.pin_memory(); return self` (return `self`) — the pin worker calls it per batch. Then keep `pin_memory=True` and transfer with `.to(device, non_blocking=True

**Symptom**: on Windows/macOS, `num_workers>0` raises `AttributeError: Can't pickle local object '<locals>.<lambda>'`; OR worse, it proceeds but transforms silently vanish (samples come back un-augmented). The identical code runs fine with 

**Fix**: make everything the worker reconstructs a top-level importable callable — replace `collate_fn=lambda b: ...` and lambda transforms with module-level `def`s; bind args with `functools.partial(top_level_fn, ...)` not a closure; for p

**Symptom**: either (a) `IndexError`/`KeyError` from `__getitem__` partway through an epoch, or (b) no error but training quietly sees duplicated/skipped samples — when `__getitem__` does `self.items[idx % len(...)]` or indexes a shorter li

**Fix**: compute `__len__` and `__getitem__` from the **same** collection (materialize the kept-index list in `__init__`, index through it). Remove any `idx % N`/clamping — let an out-of-range index raise. Sanity once: `assert len(ds)==<exp

**Symptom**: training runs most of an epoch then dies at the **last** batch with `ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, C, ...])`. Happens when `len(dataset) % batch_size == 1`. **Ro

**Fix**: `DataLoader(..., drop_last=True)` on the **train** loader discards the incomplete final batch (the standard fix). Alternatives if you can't drop data: swap BatchNorm → `nn.GroupNorm`/`nn.LayerNorm` (no batch-stat dependence), or fr

**Fix**: `DataLoader(..., drop_last=True)` on the **train** loader discards the incomplete final batch (the standard fix). Alternatives if you can't drop data: swap BatchNorm → `nn.GroupNorm`/`nn.LayerNorm` (no batch-stat dependence), or fr

**Symptom**: RAM climbs steadily across iters/epochs until a bare `Killed` (exit 137, no traceback) — typically from a Dataset that lazy-caches decoded samples (`if idx not in self.cache: self.cache[idx]=load(idx)`). With `num_workers>0` th

**Fix**: don't accumulate unbounded Python objects in `__getitem__`. Options: (a) precompute to a single `np.memmap` / Arrow / LMDB / `.npy` in `__init__` and read slices (the OS page cache **is** shared across forked workers); (b) bound th

**Symptom**: the model loads and runs without error, but a pretrained backbone scores far below its reported number, or your own val accuracy is a few points under train for no obvious reason; predictions are systematically biased (reds↔blu

**Root cause**: the per-channel mean/std are correct numbers applied in the wrong space or order. (1) Stats must be computed on the **train split only** and reused verbatim at eval (the sklearn contract: `fit_transform` on train, `transform

**Fix**: compute stats once on train and reuse the same constants/transform at eval. For a torchvision pretrained model don't hand-roll it — use `weights.transforms()` (e.g. `ResNet50_Weights.IMAGENET1K_V2.transforms()`), which bundles resi

**Symptom**: a pipeline mixing `cv2` for I/O and a PIL/torchvision-trained (RGB) model: no exception, but color-sensitive predictions degrade; visualizing the array shows reds appearing blue. Often surfaces only when you switch the loader (

**Fix**: convert immediately after a cv2 load — `img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)` (or `img = img[:, :, ::-1].copy()` — the `.copy()` matters, a negative-stride view breaks `torch.from_numpy`). Or switch I/O to `torchvision.io.rea

**Symptom**: loss is huge or NaN from step 0, or activations/gradients are enormous, when the input came from a float numpy array, a `.npy`, a 16-bit/HDR image (PIL mode `I`/`F`), or a tensor already in `[0,1]`. The same model works fine on

**Fix**: don't rely on `ToTensor` for non-uint8 scaling. For float inputs scale explicitly: `t = torch.from_numpy(arr).float() / 255.0` (or the correct max for 16-bit). In the v2 API prefer `transforms.v2.ToImage()` + `transforms.v2.ToDtype

**Symptom**: dataset construction or the first `__getitem__` raises `TypeError: tensor should be a torch tensor. Got <class 'PIL.Image.Image'>` (or `img should be Tensor`). **Root cause**: `transforms.Normalize` operates on a float tensor s

**Fix**: order the pipeline — PIL ops → `ToTensor()` → `Normalize(mean, std)`, e.g. `Compose([Resize(256), CenterCrop(224), ToTensor(), Normalize([0.485,0.456,0.406],[0.229,0.224,0.225])])`. `mean`/`std` lengths must equal the channel count

**Symptom**: two shuffle failures — (a) `ValueError: sampler option is mutually exclusive with shuffle` the moment you add any sampler; (b) no error, but in DDP every epoch iterates the data in the **identical** order, so the train-loss cur

**Fix**: (a) when you must use a sampler (DistributedSampler, WeightedRandomSampler), set `shuffle=False` and let the sampler own ordering. (b) call `train_sampler.set_epoch(epoch)` at the **start of each epoch** before iterating (Lightning

**Symptom**: `DataLoader worker (pid N) is killed by signal: Bus error`, or `RuntimeError: unable to write to file </torch_...>` / `received 0 items of ancdata` — on a **rented container** while the identical code runs fine on your workstat

**Fix**: enlarge it at launch — `docker run --shm-size=8g` (or `--ipc=host`); where you can't set that (a fixed rental), switch the IPC strategy `torch.multiprocessing.set_sharing_strategy("file_system")` (fd-passing, slower but uncapped) a

**Fix**: launch through `torchrun`, which sets the full contract per process ([torchrun docs](https://docs.pytorch.org/docs/2.12/elastic/run.html)):

([torchrun docs](https://docs.pytorch.org/docs/2.12/elastic/run.html)):

[Credential-related source omitted]

**Fix** ([torchrun docs](https://docs.pytorch.org/docs/2.12/elastic/run.html)):

[Credential-related source omitted]

**Root cause**: both jobs default to `MASTER_PORT=29500`; the c10d TCP store can't bind a port the first job holds ([pytorch#85604](https://github.com/pytorch/pytorch/issues/85604)).

first job holds ([pytorch#85604](https://github.com/pytorch/pytorch/issues/85604)).

**Root cause**: Accelerate wraps the same env contract (D1) but sources it from `~/.cache/huggingface/accelerate/default_config.yaml` (written by `accelerate config`) or CLI flags ([launch docs](https://huggingface.co/docs/accelerate/en/bas

([launch docs](https://huggingface.co/docs/accelerate/en/basic_tutorials/launch)).

(`worker-1 slots=8`), distinct from torchrun's rendezvous. Under HF it's usually cleaner to let `accelerate launch` (with a DeepSpeed plugin/config) drive it ([HF DeepSpeed](https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed))

([HF DeepSpeed](https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed)).

- **Model does NOT fit (params+optim+grads ≈ 18 B/param, see `references/training/oom-memory.md` M1)** → shard it: **FSDP** (PyTorch-native) or **DeepSpeed ZeRO** (richer offload). Sharding-to-fit ladder → `references/training/oom-memory.md

- **HF ecosystem / Trainer** → **Accelerate** as the launcher; flip a config field to choose DDP/FSDP/ZeRO. - **Need CPU/NVMe offload of params *and* optimizer separately, or ZeRO-Infinity** → **DeepSpeed** (FSDP**1** offload is all-or-noth

- **Need CPU/NVMe offload of params *and* optimizer separately, or ZeRO-Infinity** → **DeepSpeed** (FSDP**1** offload is all-or-nothing; FSDP2/`fully_shard` is finer-grained; [HF concept guide](https://github.com/huggingface/accelerate/blob

**Symptom**: `RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. ... parameters that were not used in producing loss` ([HF discuss](https://discuss.huggingface.co/t/runtimeerror-expected-to-h

([HF discuss](https://discuss.huggingface.co/t/runtimeerror-expected-to-have-finished-reduction-in-the-prior-iteration-before-starting-a-new-one-this-error-indicates-that-your-module-has-parameters-that-were-not-used-in-producing-loss/64760

**Fix — in priority order**: 1. **Best**: make every output participate in the loss (often the real bug is a dropped/detached head). 2. If a branch is *legitimately* unused some steps, `DDP(model, find_unused_parameters=True)` — but it adds

1. **Best**: make every output participate in the loss (often the real bug is a dropped/detached head). 2. If a branch is *legitimately* unused some steps, `DDP(model, find_unused_parameters=True)` — but it adds a full graph traversal each 

2. If a branch is *legitimately* unused some steps, `DDP(model, find_unused_parameters=True)` — but it adds a full graph traversal each step and **can be drastically slower** ([PyTorch forum](https://discuss.pytorch.org/t/process-got-stuck-

**Symptom**: training completes most of an epoch then **freezes on the final batch**; one rank had fewer samples and exited the loop while the others wait in allreduce forever ([PyTorch forum](https://discuss.pytorch.org/t/understanding-dis

([PyTorch forum](https://discuss.pytorch.org/t/understanding-distributedsampler-and-dataloader-drop-last/206271)).

`from torch.distributed.algorithms.join import Join; with Join([model]): for batch in loader: ...`   — which mirrors the missing ranks' collectives so finished ranks don't deadlock   ([Join tutorial](https://docs.pytorch.org/tutorials/advan

— which mirrors the missing ranks' collectives so finished ranks don't deadlock   ([Join tutorial](https://docs.pytorch.org/tutorials/advanced/generic_join.html)). `model.join()` is the   legacy spelling of the same context manager.

([Join tutorial](https://docs.pytorch.org/tutorials/advanced/generic_join.html)). `model.join()` is the

**Root cause**: with no `auto_wrap_policy`, FSDP makes the **entire model one FSDP unit** — it must all-gather all parameters at once, defeating sharding ([FSDP tutorial](https://docs.pytorch.org/tutorials/intermediate/FSDP_tutorial.html)).

([FSDP tutorial](https://docs.pytorch.org/tutorials/intermediate/FSDP_tutorial.html)).

Under Accelerate set `fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP` + `fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer` ([HF FSDP](https://huggingface.co/docs/accelerate/en/usage_guides/fsdp)). FSDP2 (`fully_shard`) is the

`fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer` ([HF FSDP](https://huggingface.co/docs/accelerate/en/usage_guides/fsdp)). FSDP2 (`fully_shard`) is the current API; the wrapping principle is identical.

([HF FSDP](https://huggingface.co/docs/accelerate/en/usage_guides/fsdp)). FSDP2 (`fully_shard`) is the

`reduce_dtype=fp32` (or vice versa) changes gradient-reduction precision; FSDP keeps fp32 master weights and casts to bf16 for forward ([pytorch#146114](https://github.com/pytorch/pytorch/issues/146114)).

([pytorch#146114](https://github.com/pytorch/pytorch/issues/146114)).

**Root cause**: DeepSpeed reads from the JSON, and several Accelerate/Trainer fields are **ignored** once a `deepspeed_config_file` is supplied ([HF Accelerate DeepSpeed](https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed)).

([HF Accelerate DeepSpeed](https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed)).

config *and* the JSON to non-`"auto"` values multiplies them. (b) With DeepSpeed's own AMP, gradient scaling lives inside the engine — calling bare `loss.backward()` instead of `model_engine.backward(loss)` skips scaling ([DeepSpeed engine]

skips scaling ([DeepSpeed engine](https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/runtime/engine.py)).

**Fix — set these and relaunch the hang**: - `export TORCH_DISTRIBUTED_DEBUG=DETAIL` + `export TORCH_CPP_LOG_LEVEL=INFO` → on mismatch PyTorch prints `Detected mismatch between collectives on ranks`, naming the op + sequence number per rank

- `export TORCH_DISTRIBUTED_DEBUG=DETAIL` + `export TORCH_CPP_LOG_LEVEL=INFO` → on mismatch PyTorch prints `Detected mismatch between collectives on ranks`, naming the op + sequence number per rank ([PyTorch forum](https://discuss.pytorch.o

SyncBN, or a metric `all_reduce`) placed inside a rank-conditional branch. Rank 0 calls it; others skip it; everyone deadlocks. The classic is "save/log on rank 0 only" where the save path triggers a collective ([Lightning#19604](https://gi

collective ([Lightning#19604](https://github.com/Lightning-AI/pytorch-lightning/issues/19604)).

> `parallelize_module(model, m["tp"], plan)` mapping submodules to `ColwiseParallel` / `RowwiseParallel` / > `SequenceParallel`; then shard the data dim with FSDP2 `fully_shard(model, mesh=m["dp"])`. > ([PyTorch TP docs](https://docs.pytorc

> ([PyTorch TP docs](https://docs.pytorch.org/docs/stable/distributed.tensor.parallel.html))

Source: HF model-memory-anatomy (https://huggingface.co/docs/transformers/en/model_memory_anatomy) gives

Sources: PyTorch forums thread on the trace fields (https://discuss.pytorch.org/t/torch-outofmemoryerror-cuda-out-of-memory/217669); the reserved-vs-allocated

(https://discuss.pytorch.org/t/torch-outofmemoryerror-cuda-out-of-memory/217669); the reserved-vs-allocated

sub-batches before one optimizer step — same math, lower peak activation memory. Keep micro-batch as large as fits (batch 4 × accum 16 beats batch 1 × accum 64 — better GPU utilization). Source: https://huggingface.co/docs/transformers/main

as fits (batch 4 × accum 16 beats batch 1 × accum 64 — better GPU utilization). Source: https://huggingface.co/docs/transformers/main/en/perf_train_gpu_one (gradient accumulation). Caveat: accumulation also multiplies the effective LR unles

Source: https://huggingface.co/docs/transformers/main/en/perf_train_gpu_one (gradient accumulation).

and fp32 weight copies); **prefer bf16 over fp16** where supported → `references/training/precision-stability.md` P1. Source: https://huggingface.co/docs/transformers/main/en/perf_train_gpu_one (mixed precision; bf16 needs

`references/training/precision-stability.md` P1. Source: https://huggingface.co/docs/transformers/main/en/perf_train_gpu_one (mixed precision; bf16 needs Ampere+).

Source: https://huggingface.co/docs/transformers/main/en/perf_train_gpu_one (mixed precision; bf16 needs

`torch.utils.checkpoint.checkpoint(...)` manually. Only a subset of activations is stored; the rest are **recomputed** during backward — cuts activation memory substantially at **~20–30% slower** training. Source: https://huggingface.co/doc

**recomputed** during backward — cuts activation memory substantially at **~20–30% slower** training. Source: https://huggingface.co/docs/transformers/main/en/perf_train_gpu_one ("~20% slower"). Gotcha: with HF generate/caching set `model.c

Source: https://huggingface.co/docs/transformers/main/en/perf_train_gpu_one ("~20% slower"). Gotcha: with

starts; `PYTORCH_CUDA_ALLOC_CONF` is the canonical name). It backs segments with CUDA VMM so they grow/shrink instead of each `cudaMalloc` being an unmergeable block — which is the root of fragmentation. Source: PyTorch CUDA notes (https://

instead of each `cudaMalloc` being an unmergeable block — which is the root of fragmentation. Source: PyTorch CUDA notes (https://docs.pytorch.org/docs/stable/notes/cuda.html) and the allocator devlog (https://docs.pytorch.org/devlogs/eager

Source: PyTorch CUDA notes (https://docs.pytorch.org/docs/stable/notes/cuda.html) and the allocator devlog

(https://docs.pytorch.org/devlogs/eager/2026-06-01-cuda-caching-allocator/). Alternative knob if fragmenting

- **ZeRO-3 / FSDP `FULL_SHARD`** — also partition **parameters** (the 6 B). Maximum savings; needs   high-bandwidth interconnect (NVLink/NVSwitch) because params are all-gathered per layer. Source: DeepSpeed ZeRO tutorial (https://www.deeps

high-bandwidth interconnect (NVLink/NVSwitch) because params are all-gathered per layer. Source: DeepSpeed ZeRO tutorial (https://www.deepspeed.ai/tutorials/zero/) and HF DeepSpeed integration (https://huggingface.co/docs/transformers/en/de

Source: DeepSpeed ZeRO tutorial (https://www.deepspeed.ai/tutorials/zero/) and HF DeepSpeed integration

(https://huggingface.co/docs/transformers/en/deepspeed); FSDP's `ShardingStrategy` maps 1:1 to these stages.

**ZeRO-Infinity** offloads to both for huge models. QLoRA's **paged optimizers** (M12) are a lighter form — optimizer state pages to CPU only on memory spikes. Source: DeepSpeed ZeRO docs (https://deepspeed.readthedocs.io/en/stable/zero3.ht

optimizer state pages to CPU only on memory spikes. Source: DeepSpeed ZeRO docs (https://deepspeed.readthedocs.io/en/stable/zero3.html); ZeRO-Infinity (https://www.deepspeed.ai/2021/03/07/zero3-offload.html). Cost: PCIe/NVMe bandwidth becom

(https://deepspeed.readthedocs.io/en/stable/zero3.html); ZeRO-Infinity

(https://www.deepspeed.ai/2021/03/07/zero3-offload.html). Cost: PCIe/NVMe bandwidth becomes the bottleneck —

`optim="adafactor"` (stores row/column moments instead of per-element → much less memory, **slower convergence**). **Paged** variants additionally page optimizer state to CPU on spikes to survive transient peaks. Source: https://huggingface

convergence**). **Paged** variants additionally page optimizer state to CPU on spikes to survive transient peaks. Source: https://huggingface.co/docs/transformers/main/en/perf_train_gpu_one (optimizers) and model-memory-anatomy ("quantized 

peaks. Source: https://huggingface.co/docs/transformers/main/en/perf_train_gpu_one (optimizers) and

optimizers), train fp16/bf16 adapters on top — reported to finetune a **65B model on a single 48 GB GPU** with no accuracy degradation vs 16-bit. Source: QLoRA paper (https://arxiv.org/abs/2305.14314) and repo (https://github.com/artidoro/q

with no accuracy degradation vs 16-bit. Source: QLoRA paper (https://arxiv.org/abs/2305.14314) and repo (https://github.com/artidoro/qlora). Note: LoRA *changes model capacity* — it is a different optimization target, not a free OOM trick, 

(https://arxiv.org/abs/2305.14314) and repo (https://github.com/artidoro/qlora). Note: LoRA *changes model

**Fix**: set `per_device_eval_batch_size` explicitly (don't inherit a too-large value); set **`eval_accumulation_steps=N`** so predictions move to CPU every N steps instead of piling on the GPU (https://huggingface.co/docs/transformers/main

**`eval_accumulation_steps=N`** so predictions move to CPU every N steps instead of piling on the GPU (https://huggingface.co/docs/transformers/main_classes/trainer). In a custom loop: wrap eval in `torch.no_grad()` / `torch.inference_mode(

(https://huggingface.co/docs/transformers/main_classes/trainer). In a custom loop: wrap eval in

not at construction. Peak therefore jumps after step 1. Reserved memory also climbs as the allocator caches backward buffers. Source: the memory-snapshot timeline shows optimizer state appearing after iter 1 (https://pytorch.org/blog/unders

(https://pytorch.org/blog/understanding-gpu-memory-1/).

`torch.cuda.max_memory_allocated()` for the true peak (reset with `reset_peak_memory_stats()` between phases to isolate forward vs backward vs optimizer). Source: https://docs.pytorch.org/docs/stable/notes/cuda.html (memory management funct

https://docs.pytorch.org/docs/stable/notes/cuda.html (memory management functions).

Drag `oom_snapshot.pickle` onto **https://pytorch.org/memory_viz** (snapshots are not uploaded server-side),

**parameters / gradients / optimizer state / activations / temporaries** separately, so the tallest band at the OOM moment names the bucket to attack (→ map back to M5–M13). Available PyTorch **2.1+**. Source: https://pytorch.org/blog/under

the OOM moment names the bucket to attack (→ map back to M5–M13). Available PyTorch **2.1+**. Source: https://pytorch.org/blog/understanding-gpu-memory-1/. On a remote box: dump the pickle, `scp` it down (references/run-remote/ssh_transport

https://pytorch.org/blog/understanding-gpu-memory-1/. On a remote box: dump the pickle, `scp` it down

anyway). It only helps a *second* process on the same GPU, or it reduces fragmentation at a ~10% speed   cost (HF `torch_empty_cache_steps=N` runs it every N steps —   https://huggingface.co/docs/transformers/main/en/perf_train_gpu_one). It

cost (HF `torch_empty_cache_steps=N` runs it every N steps —   https://huggingface.co/docs/transformers/main/en/perf_train_gpu_one). It is **not** a fix for a model that   genuinely doesn't fit.

https://huggingface.co/docs/transformers/main/en/perf_train_gpu_one). It is **not** a fix for a model that

bf16 handles large dot-products / attention logits better than fp16, which saturates and triggers scaler-step-skipping. URLs: https://docs.pytorch.org/docs/2.12/amp.html ·

scaler-step-skipping. URLs: https://docs.pytorch.org/docs/2.12/amp.html ·

https://www.runpod.io/articles/guides/fp16-bf16-fp8-mixed-precision-speed-up-my-model-training

`"high"` = TF32; `"highest"` = true fp32 (default); `"medium"` = even coarser. HF Trainer exposes `--tf32 1`. Most nets converge identically with TF32 as with fp32. URLs: https://github.com/pytorch/pytorch/pull/76509 ·

Most nets converge identically with TF32 as with fp32. URLs: https://github.com/pytorch/pytorch/pull/76509 · https://docs.pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html ·

https://github.com/pytorch/pytorch/pull/76509 ·

https://docs.pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html ·

https://docs.pytorch.org/docs/2.12/notes/numerical_accuracy.html

On V100/T4 use fp16+GradScaler (P5). FP8 (H100) is opt-in via Transformer Engine / `torchao`, not plain autocast (out of scope). Record the card next to `nvidia-smi` in Phase 0. URL: https://www.e2enetworks.com/blog/nvidia-a100-vs-h100-vs-h

URL: https://www.e2enetworks.com/blog/nvidia-a100-vs-h100-vs-h200-gpu-comparison

...)` / `torch.amp.GradScaler("cuda")` API — `torch.cuda.amp.*` is **deprecated** in PyTorch 2.x. autocast state is thread-local (re-enter it inside each DDP/DataParallel worker thread). URL: https://docs.pytorch.org/docs/2.12/amp.html

URL: https://docs.pytorch.org/docs/2.12/amp.html

gradients don't underflow — carrying a GradScaler into a bf16 run is dead weight (wasted overhead, not a crash), and bf16 runs plain `loss.backward(); optimizer.step()`. URLs: https://github.com/pytorch/pytorch/blob/main/docs/source/notes/a

crash), and bf16 runs plain `loss.backward(); optimizer.step()`. URLs: https://github.com/pytorch/pytorch/blob/main/docs/source/notes/amp_examples.rst · https://docs.pytorch.org/docs/2.12/amp.html

https://github.com/pytorch/pytorch/blob/main/docs/source/notes/amp_examples.rst ·

https://docs.pytorch.org/docs/2.12/amp.html

`unscale_` is idempotent-per-step (call it once). For bf16, just `clip_grad_norm_` directly — no unscale. URL: https://github.com/pytorch/pytorch/blob/main/docs/source/notes/amp_examples.rst

URL: https://github.com/pytorch/pytorch/blob/main/docs/source/notes/amp_examples.rst

URLs: https://docs.pytorch.org/docs/stable/generated/torch.log.html ·

https://medium.com/better-ml/loss-spikes-in-training-causes-detection-and-mitigations-ed66e591b1a1

The docs warn it "will slow down your program" (roughly an order of magnitude) — enable to *locate*, then   turn OFF for the real run, never ship it on. URL: https://docs.pytorch.org/docs/2.12/autograd.html

turn OFF for the real run, never ship it on. URL: https://docs.pytorch.org/docs/2.12/autograd.html

scores, the loss) in **fp32** via a nested `torch.autocast("cuda", enabled=False)` region, and apply z-loss (P15) / qk-norm (P16) to stop the logits growing. URL: https://medium.com/better-ml/loss-spikes-in-training-causes-detection-and-mit

URL: https://medium.com/better-ml/loss-spikes-in-training-causes-detection-and-mitigations-ed66e591b1a1

apart with the replay diagnostic (P14). Smoke the data first — smoke *content* is owned by **references/verifying/methodology.md** (cross-link **REQUIRED**). URL: https://arxiv.org/pdf/2311.03938

URL: https://arxiv.org/pdf/2311.03938

URLs: https://arxiv.org/pdf/2309.14322 ·

https://apxml.com/courses/how-to-build-a-large-language-model/chapter-24-identifying-mitigating-training-instabilities/stabilization-techniques-revisited

*all* grads down, so one embedding-heavy batch can throttle everything else that step — consider per-module clipping if embeddings dominate. URL: https://medium.com/better-ml/loss-spikes-in-training-causes-detection-and-mitigations-ed66e591

URL: https://medium.com/better-ml/loss-spikes-in-training-causes-detection-and-mitigations-ed66e591b1a1

**Tell a *data* spike from an *optimizer-state* one by replay:** re-run the same batches from an *earlier* checkpoint — if the spike does **not** reproduce, it is a state interaction, not dirty data, so roll back (~100 steps) and skip those

checkpoint — if the spike does **not** reproduce, it is a state interaction, not dirty data, so roll back (~100 steps) and skip those batches rather than blaming the data ([PaLM §5.1](https://arxiv.org/abs/2204.02311) saw ~20 such spikes, n

(~100 steps) and skip those batches rather than blaming the data ([PaLM §5.1](https://arxiv.org/abs/2204.02311)

saw ~20 such spikes, none reproducing on replay). If a spike resists all of the above, suspect fp16 *range*: BLOOM's 104B fp16 pretraining diverged irrecoverably where the 176B **bf16** run stayed stable — prefer bf16 ([BLOOM §3.4.3](https:

BLOOM's 104B fp16 pretraining diverged irrecoverably where the 176B **bf16** run stayed stable — prefer bf16 ([BLOOM §3.4.3](https://arxiv.org/abs/2211.05100)). Adaptive spike-clipping (ZClip) and momentum-reset on spike (SPAM) automate thi

([BLOOM §3.4.3](https://arxiv.org/abs/2211.05100)).

https://arxiv.org/pdf/2504.02507 · https://arxiv.org/pdf/2501.06842

Coefficient **1e-4** is the PaLM / ST-MoE value; too large lets z-loss dominate. Standard in LLM pretraining; also the recommended fix for MoE router instability. URLs: https://medium.com/dair-ai/papers-explained-50-palm-480e72fa3fd5 · http

also the recommended fix for MoE router instability. URLs: https://medium.com/dair-ai/papers-explained-50-palm-480e72fa3fd5 · https://arxiv.org/pdf/2202.08906 · https://arxiv.org/pdf/2309.14322

https://medium.com/dair-ai/papers-explained-50-palm-480e72fa3fd5 · https://arxiv.org/pdf/2202.08906 ·

https://arxiv.org/pdf/2309.14322

**Fix**: apply **QK-LayerNorm** — LayerNorm query and key per-head before the dot-product. Combined with z-loss (P15) + warmup (P12), it lets small models train to similar loss across *orders of magnitude* of LR, i.e. removes most LR-sensit

i.e. removes most LR-sensitivity. URL: https://arxiv.org/pdf/2309.14322

**Fix**: scale residual-branch init by `1/√(2·n_layers)` (GPT-2-style); prefer pre-LN over post-LN for deep transformers; init embeddings at small std (~0.02). When unsure, copy a *known-good* config's init+norm scheme rather than tuning bl

scheme rather than tuning blind. URL: https://arxiv.org/pdf/2309.14322

A grad-norm trace is the cheapest, highest-signal stability instrument — log it from step 1. URL: https://apxml.com/courses/how-to-build-a-large-language-model/chapter-24-identifying-mitigating-training-instabilities/stabilization-technique

URL: https://apxml.com/courses/how-to-build-a-large-language-model/chapter-24-identifying-mitigating-training-instabilities/stabilization-techniques-revisited

`references/run-remote/gotchas_universal.md`. This layer only ensures the knobs are *set and logged*. Determinism costs speed — enable for the datapoint that must be clean, not every throwaway run. URL: https://docs.pytorch.org/docs/stable/

URL: https://docs.pytorch.org/docs/stable/notes/randomness.html

**Fix — classify with one cheap reading each** (heuristic: util consistently >90% ⇒ GPU-bound; low/fluctuating ⇒ elsewhere; both CPU+GPU low ⇒ I/O — https://apxml.com/courses/planning-optimizing-ai-infrastructure/chapter-5-strategies-for-pe

low/fluctuating ⇒ elsewhere; both CPU+GPU low ⇒ I/O — https://apxml.com/courses/planning-optimizing-ai-infrastructure/chapter-5-strategies-for-performance-optimization/identifying-performance-bottlenecks): - **GPU-bound** (the good case): u

https://apxml.com/courses/planning-optimizing-ai-infrastructure/chapter-5-strategies-for-performance-optimization/identifying-performance-bottlenecks):

**Root cause**: with `num_workers=0` "the data will be loaded in the main process" — no overlap between data prep and compute (https://docs.pytorch.org/docs/2.12/data.html).

data prep and compute (https://docs.pytorch.org/docs/2.12/data.html).

**Root cause**: default `persistent_workers=False` shuts down all workers after the dataset is consumed once and **re-forks them next epoch** — re-importing, re-opening files, rebuilding the dataset object each time (https://docs.pytorch.or

time (https://docs.pytorch.org/docs/2.12/data.html).

**Root cause**: a pageable-memory tensor must be staged through a pinned buffer by the driver before DMA; a synchronous `.to(device)` blocks the step. "When using a GPU it's better to set `pin_memory=True`" (https://docs.pytorch.org/tutoria

(https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html).

**Symptom**: with workers on, the GPU still periodically stalls — every *Nth* step (N = `num_workers`) has a long idle gap because all workers were busy producing the next batch when the GPU asked (https://docs.pytorch.org/tutorials/interme

(https://docs.pytorch.org/tutorials/intermediate/tensorboard_profiler_tutorial.html).

**Root cause**: `prefetch_factor` defaults to **2** when `num_workers>0` (None when 0) — "2 means there will be a total of `2 * num_workers` batches prefetched across all workers" (https://docs.pytorch.org/docs/2.12/data.html). A shallow qu

will be a total of `2 * num_workers` batches prefetched across all workers" (https://docs.pytorch.org/docs/2.12/data.html). A shallow queue can't absorb a variance spike in per-sample fetch/decode time.

(https://docs.pytorch.org/docs/2.12/data.html). A shallow queue can't absorb a variance spike in

**Root cause**: by default cuDNN picks a generic conv algorithm; the autotuner benchmarks variants on the first batch of each new shape and caches the fastest (https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html).

(https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html).

`model = model.to(memory_format=torch.channels_last)` and `x = x.to(memory_format=torch.channels_last)`. Optimizes convolutional networks with Tensor Cores + AMP (https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html). Marked

Optimizes convolutional networks with Tensor Cores + AMP (https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html). Marked experimental and CNN-specific (no benefit for pure transformers). No numerics change — purely a layout s

(https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html). Marked experimental and CNN-specific

- **`optimizer.zero_grad(set_to_none=True)`** (the **default** since PyTorch 2.0) over zero-filling —   assigning `None` skips a memory-write kernel per param and lets the next backward write fresh   (https://docs.pytorch.org/tutorials/reci

assigning `None` skips a memory-write kernel per param and lets the next backward write fresh   (https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html). Edge case: code reading `.grad`   between steps must tolerate `None`.

(https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html). Edge case: code reading `.grad`

`torch.autograd.profiler.profile`, `gradcheck` add per-op bookkeeping (anomaly detection is ~10× slower,   precision-stability P9). Grep `detect_anomaly` / leftover `with profile(` wrappers before a long launch   (https://docs.pytorch.org/t

(https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html); easy to leave on after a NaN hunt.

**silently falls back to the slow `math` backend** when the fused kernel's input constraints aren't met (wrong dtype, head dim, mask shape) — "if a fused implementation is not available, a warning will be raised" (https://docs.pytorch.org/d

raised" (https://docs.pytorch.org/docs/2.12/generated/torch.nn.functional.scaled_dot_product_attention.html).

`<96 GB` RAM ninja over-parallelizes and OOMs the build — cap `MAX_JOBS=4 pip install flash-attn   --no-build-isolation`. Prefer a **prebuilt wheel** matching the `cuXX/torchYY/cpZZ` triple   (https://github.com/Dao-AILab/flash-attention/is

--no-build-isolation`. Prefer a **prebuilt wheel** matching the `cuXX/torchYY/cpZZ` triple   (https://github.com/Dao-AILab/flash-attention/issues/1038, https://pypi.org/project/flash-attn/). A   torch/CUDA mismatch is **`references/run-remo

(https://github.com/Dao-AILab/flash-attention/issues/1038, https://pypi.org/project/flash-attn/). A

**Fix**: wrap the model — `model = torch.compile(model)`. Modes (https://huggingface.co/docs/transformers/en/perf_torch_compile):

(https://huggingface.co/docs/transformers/en/perf_torch_compile):

Reported ~2.2× mean-inference speedups; training gains real but model-dependent. **First step(s) are slow** — compilation is lazy on first call (https://huggingface.co/docs/transformers/en/perf_torch_compile); exclude

— compilation is lazy on first call (https://huggingface.co/docs/transformers/en/perf_torch_compile); exclude

`cache_size_limit`) Dynamo **stops compiling that function and runs it eagerly** — paying all the compile cost and getting none of the benefit (https://docs.pytorch.org/docs/stable/compile/programming_model.recompilation.html,

cost and getting none of the benefit (https://docs.pytorch.org/docs/stable/compile/programming_model.recompilation.html, https://github.com/pytorch/pytorch/issues/93457).

(https://docs.pytorch.org/docs/stable/compile/programming_model.recompilation.html,

https://github.com/pytorch/pytorch/issues/93457).

- **See it**: `TORCH_LOGS=recompiles python train.py` logs which function recompiled and the failed guard;   `TORCH_LOGS=graph_breaks` and `torch._dynamo.explain(...)` locate graph breaks   (https://docs.pytorch.org/docs/stable/torch.compil

`TORCH_LOGS=graph_breaks` and `torch._dynamo.explain(...)` locate graph breaks   (https://docs.pytorch.org/docs/stable/torch.compiler_troubleshooting.html). - **Tame shapes**: pad/bucket to a few fixed shapes so guards stop firing; or mark 

(https://docs.pytorch.org/docs/stable/torch.compiler_troubleshooting.html).

one shape-generic graph instead of one per size. `dynamic=False` forces a fresh recompile per distinct   size (use only with truly few shapes)   (https://docs.pytorch.org/docs/stable/compile/programming_model.html).

size (use only with truly few shapes)   (https://docs.pytorch.org/docs/stable/compile/programming_model.html). - **Last resort**: raise `torch._dynamo.config.recompile_limit` only if a handful of *stable* extra shapes

(https://docs.pytorch.org/docs/stable/compile/programming_model.html).

**Fix — scheduled profile of a few steps** (https://docs.pytorch.org/tutorials/recipes/recipes/profiler_recipe.html):

(https://docs.pytorch.org/tutorials/recipes/recipes/profiler_recipe.html):

**Read it**: large **GPU-timeline gaps** with CPU busy in `DataLoader`/transforms during the gap ⇒ **data-bound** (T4–T8); the TensorBoard "Performance Recommendation" panel names the DataLoader directly (https://docs.pytorch.org/tutorials/

**data-bound** (T4–T8); the TensorBoard "Performance Recommendation" panel names the DataLoader directly (https://docs.pytorch.org/tutorials/intermediate/tensorboard_profiler_tutorial.html). Densely-packed GPU timeline ⇒ GPU-bound; sort by 

(https://docs.pytorch.org/tutorials/intermediate/tensorboard_profiler_tutorial.html). Densely-packed GPU

memcpy, NVTX ranges, OS-runtime — so it exposes launch-bound stalls and CPU↔GPU sync that PyTorch can't. "Periodic gaps in the CUDA HW row are moments when the GPU is idle — a red flag" (https://docs.lxp.lu/howto/pytorch-profiling-with-nsig

(https://docs.lxp.lu/howto/pytorch-profiling-with-nsight/).

**Fix — profile a bounded window on the box, view locally** (canonical PyTorch recipe, https://gist.github.com/mcarilli/376821aa1a7182dfcf59928a7cde3223):

https://gist.github.com/mcarilli/376821aa1a7182dfcf59928a7cde3223):

`scp` the `.nsys-rep` down, open in the Nsight Systems GUI. Nsight **Systems** finds *which* kernel is slow; Nsight **Compute** (`ncu`) finds *why* (occupancy, bandwidth, warp stalls) — but `ncu` is heavy, reserve it for one hot kernel (htt

for one hot kernel (https://www.spheron.network/blog/gpu-profiling-ai-workloads-nsight-compute-pytorch-profiler-guide/).

**Fix — attach by PID, zero instrumentation** (https://github.com/benfred/py-spy):

For *what allocated the memory* (not time), the `torch.cuda.memory._record_memory_history` snapshot + https://pytorch.org/memory_viz timeline is owned by **references/training/oom-memory.md M19/M18**. It is a

https://pytorch.org/memory_viz timeline is owned by **references/training/oom-memory.md M19/M18**. It is a

**Root cause**: DDP overlaps gradient all-reduce with backward by bucketing gradients and launching each bucket's reduce on a separate CUDA stream as soon as it's ready (https://github.com/pytorch/pytorch/issues/67570). Overlap *breaks* whe

bucket's reduce on a separate CUDA stream as soon as it's ready (https://github.com/pytorch/pytorch/issues/67570). Overlap *breaks* when something forces a sync: an unused-parameter recompute, an off-by-default `find_unused_parameters=True`

(https://github.com/pytorch/pytorch/issues/67570). Overlap *breaks* when something forces a sync: an

image as independent images. A paired analysis can detect an effect despite overlapping marginal   bars. A wide interval means unresolved precision, not proof of zero effect; tiny or degenerate   resamples do not create information. See [NI

bars. A wide interval means unresolved precision, not proof of zero effect; tiny or degenerate   resamples do not create information. See [NIST paired observations](https://www.itl.nist.gov/div898/handbook/prc/section3/prc311.htm)   and [tw

resamples do not create information. See [NIST paired observations](https://www.itl.nist.gov/div898/handbook/prc/section3/prc311.htm)

and [two-sample inference](https://www.itl.nist.gov/div898/handbook/eda/section3/eda353.htm).

threshold, not MDE at a specified power. If the design cannot resolve the actionable effect,   change the authorized design or report that limitation; an observed small/non-significant result   is not automatically a null finding. [NIST sam

change the authorized design or report that limitation; an observed small/non-significant result   is not automatically a null finding. [NIST sample-size and power derivation](https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm

is not automatically a null finding. [NIST sample-size and power derivation](https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm).

#!/usr/bin/env bash

#!/usr/bin/env python3

#!/usr/bin/env python3

#!/usr/bin/env python3

#!/usr/bin/env bash

rm -f "$incoming"

rm -f "$LOCAL_TARGET/$MANIFEST_NAME"

#!/usr/bin/env bash

#!/usr/bin/env bash

#!/usr/bin/env python3

#!/usr/bin/env bash

#!/usr/bin/env python3

#!/usr/bin/env bash

#!/usr/bin/env python3

#!/usr/bin/env bash

#!/usr/bin/env bash

[Credential-related source omitted]

if ! eval "${PROXY_HOOK}" 2>/dev/null; then

[Credential-related source omitted]

rm -f "$CKPT_DIR/latest.pth"

#!/usr/bin/env bash

eval "${PROXY_HOOK}" 2>/dev/null || true

#!/usr/bin/env bash

export HF_ENDPOINT="${HF_ENDPOINT:-https://hf-mirror.com}"

pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple 2>/dev/null \

|| export PIP_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple

echo "[setup-china-mirrors] HF_ENDPOINT=$HF_ENDPOINT  HF_HOME=$HF_HOME"

#!/usr/bin/env python3

#!/usr/bin/env python3

#!/usr/bin/env python3

#!/usr/bin/env python3

[Credential-related source omitted]

[Credential-related source omitted]

[Credential-related source omitted]