---

name: training description: "Route MMAudio DDP training, smoke runs, checkpoint resume, EMA synthesis, and training-side output inspection." disable-model-invocation: true metadata: disco-role: operating license: MIT

Training

Use this route when a user wants to launch, resume, or debug MMAudio training on CUDA, or needs a bounded smoke command for a quick environment check. This route assumes precomputed feature memmaps and external model assets already exist. It does not extract features or run arbitrary batch evaluation.

Read first

• references/training-workflow.md for the end-to-end lifecycle, command patterns, expected outputs, and resume precedence.
• references/configuration.md for Hydra keys, supported model variants, batch-size semantics, and path conventions.
• references/troubleshooting.md for NCCL, env-var, checkpoint, model-variant, and asset failures.
• scripts/build_train_command.py to render a smoke-safe or full torchrun command without launching training.

What this route owns

• DDP training launch on one node.
• example_train smoke runs and bounded one-iteration checks.
• Checkpoint resume, pretrained weights loading, and checkpoint precedence.
• EMA synthesis, TensorBoard logging, and the built-in post-training sample path.
• Interpreting artifacts under output/<exp_id>/.

What this route excludes

• Feature extraction and memmap creation.
• Batch evaluation on arbitrary datasets.
• Demo or Gradio inference for pretrained models.

Before you launch

• Use torchrun; train.py imports mmaudio.sample at module load, so plain python train.py --help can fail unless the distributed env vars are present.
• Confirm CUDA is available. This route has no CPU fallback.
• Install av_bench; the training loop and final sample use it when evaluation runs.
• Populate ext_weights/ with the required empty-string, VAE, Synchformer, and vocoder assets for the chosen model.
• Make sure the example memmap fixtures, training memmaps, and any required validation/test caches exist before relying on example_train or a real training run.

Typical path

1. Build a command with scripts/build_train_command.py.
2. Launch with OMP_NUM_THREADS=4 torchrun --standalone --nproc_per_node=<N> train.py ....
3. Watch output/<exp_id>/ for checkpoints, EMA files, logs, and sampled audio/video.
4. If you are resuming, decide whether you want an exact checkpoint or only model weights; do not pass both.
5. After the run, inspect the final EMA and the sample/eval outputs before deciding whether to continue training.

Common gotchas

• The runtime code divides batch_size by the number of GPUs, so the value you pass must be compatible with the world size.
• large_44k_v2 is not a training target in this code path.
• The built-in post-training sample is fixed to the extracted VGGSound test path defined in the config; for other evaluation datasets or metrics, use the evaluation route.
• The smoke command still reaches the built-in sample path at the end, so a "successful" smoke run needs the example fixtures plus the test cache to survive that final step.
• mini_train is not a reliable standalone smoke flag in the current loader logic; use example_train=True instead.