---

name: distributed description: "Composer distributed launch, rank, device, sampler, auto microbatching, and parallelism workflows." metadata: disco-role: operating disable-model-invocation: true license: Apache 2.0

Composer Distributed

Use this sub-skill when a task needs Composer-specific distributed launch, rank or world-size reasoning, sampler wiring, device/backend selection, auto microbatching caveats, or FSDP/FSDP2/tensor-parallel configuration basics.

Route here

• Convert a single-process training script into a composer launcher command.
• Choose --nproc, --world_size, --node_rank, --base_rank, --master_addr, and --master_port for single-node or multi-node runs.
• Capture non-rank-zero stdout/stderr with filename placeholders such as {rank} or {local_rank}.
• Read rank values through composer.utils.dist instead of parsing environment variables by hand.
• Add dist.get_sampler(dataset, shuffle=True) for map-style datasets.
• Decide between device="cpu", device="gpu", MPS, TPU, HPU, and Neuron device helpers.
• Understand when device_train_microbatch_size="auto" can recover from CUDA OOMs.
• Prepare basic parallelism_config for DDP/FSDP/FSDP2/tensor-parallel work.

Reroute

• Core Trainer fit/eval loops, optimizer stepping, checkpoint scheduling, or non-distributed resume: use ../training/SKILL.md.
• Algorithm/method behavior and method-specific side effects: use ../methods/SKILL.md.
• Loggers, profiler setup, tracing, and performance-analysis tooling: use ../observability/SKILL.md.
• Export-only, inference, ONNX, TorchScript, or deployment flows: use ../inference-export/SKILL.md.

Read first

• Launcher and ranks: CLI launcher, rank environment variables, output capture, and sampler wiring.
• Parallelism and backends: device selection, FSDP/FSDP2/TP configuration, auto microbatching, and backend prerequisites.
• Troubleshooting: launch/backend/distributed failures and concrete recovery checks.
• Device probe: safe local Composer/Torch device surface check.
• Launcher help: safe launcher usage printer that does not start training.

Quick launch patterns

Single node with every visible GPU:

composer -n 8 train.py --config config.yaml

Single node with rank-specific child logs:

composer -n 8 --stdout stdout_{rank}.log --stderr stderr_{rank}.log train.py

Module mode, useful when the training entry point is importable:

composer -n 4 -m package.train --arg value

Multi-node requires explicit topology:

composer -n 8 --world_size 16 --node_rank 1 --master_addr host0 --master_port 29500 train.py

If world_size > nproc, provide enough information for Composer to infer the global ranks on this node.

In-script distributed pattern

from torch.utils.data import DataLoader
from composer.utils import dist

sampler = dist.get_sampler(dataset, shuffle=True)
loader = DataLoader(dataset, batch_size=32, sampler=sampler)

if dist.get_global_rank() == 0:
    print("rank zero logs once")

Use get_world_size(), get_local_rank(), get_global_rank(), get_node_rank(), and get_local_world_size() for diagnostics that also work in non-distributed runs.

Device and backend rules

1. Start CPU-only if the issue is model logic, dataloader schema, or loss/metric behavior.
2. Move to device="gpu" only after PyTorch reports CUDA availability and the requested precision is supported.
3. Treat MPS/TPU/HPU/Neuron as backend-specific paths; verify the vendor runtime before claiming support.
4. For FSDP/FSDP2/tensor parallelism, align the world size with the requested sharding/replication/tensor degrees.
5. CPU importability is not proof that GPU launch, FSDP, or auto-microbatching is healthy.

Auto microbatching decision guide

Set device_train_microbatch_size="auto" when a CUDA training run OOMs during forward/backward and the full batch should be split dynamically. It does not fix dataloader memory, CPU OOM, model construction OOM, callback OOM, or algorithm work that runs outside the Trainer forward/backward path. Be careful with BatchNorm when auto microbatching makes microbatches very small.

Safe workflow

1. Run python scripts/device_probe.py to inspect import/device availability.
2. Run python scripts/launcher_help.py before composing launch commands.
3. Verify the training script works as python train.py or python -m package.train before multi-rank launch.
4. Add dist.get_sampler for map-style datasets.
5. Add rank-specific file placeholders before increasing ranks.
6. Add FSDP/FSDP2/TP only after ordinary DDP launch works.
7. Keep a tiny synthetic job for backend smoke; do not debug distributed launch on a full production run first.

Ask or stop before proceeding

• The user asks for multi-node launch but cannot provide master address, rank topology, scheduler details, or network reachability.
• Required accelerator hardware or vendor runtime is unavailable.
• The task would launch a long training run, write to shared storage, or consume all GPUs without explicit approval.
• A sharded checkpoint must be loaded on a different world size and the checkpoint format/backends are unknown.
• FSDP/TP degrees conflict with visible world size or model architecture.