---

name: training description: "Configure, validate, launch, and resume Motus three-stage training on a single node or SLURM cluster without confusing fine-tuning, checkpoint resume, or backbone initialization." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Motus training

Use this skill when a user needs to prepare a Motus training configuration, check it safely, construct a torchrun/DeepSpeed command, or reason about resuming a run. It covers the three-stage intent, not dataset conversion or inference.

• Read configuration.md for the config contract, derived values, and checkpoint-mode rules.
• Read workflows.md for single-node, SLURM, and checkpoint export procedures. Launchers in the repository are reference-only: they are long-running and may submit external cluster jobs.
• Read troubleshooting.md when validation, imports, CUDA, flash-attn, NCCL, logging, data, or checkpoint loading fails.
• For dataset directory/layout preparation, use the sibling data-preparation skill. For inference, use model-inference.

Operating procedure

1. Identify intent before editing YAML. Select pretrain, finetune, or resume. Record whether this is Stage 1 VGM training, Stage 2 Motus pretraining with latent actions, or Stage 3 target-robot SFT with actions. A checkpoint resume is a fourth execution mode, not a new training stage.
2. Choose an example config. Use configs/latent_action.yaml for the explicit training_mode: pretrain latent-action pipeline; use the embodiment config matching the dataset (robotwin, ac_one, aloha_agilex_2, or lerobot) for fine-tuning. Replace all example paths with paths valid on the target machine.
3. Run safe checks before any launch. Parse YAML, check the required sections and enum-like fields, calculate action_chunk_size, verify local dataset/checkpoint/VAE/config paths where available, and check GPU count, CUDA, bfloat16, DeepSpeed, and flash-attn. A parser/import check is allowed on CPU; model construction, VAE access, and training require CUDA and the large WAN/VLM checkpoints.
4. Make checkpoint intent mutually clear. Never set a resume checkpoint and a fine-tune checkpoint accidentally. Resume loads the complete Accelerator state and continues its step number; fine-tune partially initializes a finetune model and skips action input/decoder weights; scratch loads WAN and VLM backbones from model.*.checkpoint_path.
5. Review the rendered command. Check node count, GPUs per node, master address/port, config path, DeepSpeed JSON, logging mode, output/checkpoint directory, and the absence of accidental shell comments after \\. Prefer a short, bounded smoke configuration before an expensive run.
6. Launch only with explicit approval. torchrun, sbatch, srun, WAN/VLM downloads, and full training are external, long-running, or expensive operations. This skill explains and constructs them but does not silently execute them. Use the repository launchers only as templates; do not copy their placeholder paths into a public skill or run them as validation.
7. Verify expected signals. Rank 0 should report the selected dataset and training mode, checkpoint directory, logging backends, model creation, dataloader creation, and step/loss lines. A saved checkpoint directory should contain Accelerator/DeepSpeed state and the filtered config.json. Use scripts/export_config_json.py to create that JSON deliberately for an existing checkpoint directory.

Hard boundary

This skill does not prescribe dataset layouts, conversion, or inference CLI flags. It does not promise CPU training: Motus uses CUDA autocast and a very large WAN/VLM stack, and the documented training requirement is more than 80 GB VRAM (typically eight 80-GB GPUs for the supplied examples). Keep missing model checkpoints, datasets, optional packages, cluster topology, and unverified hardware as explicit blockers rather than guessing.