---

name: training-finetuning description: "Create, inspect, adapt, and troubleshoot LLM Foundry pretraining and fine-tuning Composer YAMLs." metadata: disco-role: operating disable-model-invocation: true license: Apache 2.0

LLM Foundry training-finetuning

Use this sub-skill when a task involves LLM Foundry training configuration, launch commands, YAML overrides, pretraining, supervised or instruction fine-tuning, domain adaptation, sequence-length adaptation, checkpointing, resumption, training callbacks/loggers/optimizers/schedulers, or bounded smoke runs.

Start here

1. Identify the requested workflow:
   • pretraining or continued pretraining/domain adaptation with a text MDS loader;
   • supervised/instruction fine-tuning with a finetuning loader;
   • a CPU smoke run that should be tiny and local;
   • a GPU, multi-GPU, or multi-node scaling adaptation;
   • checkpoint save/load/autoresume, callback/logger, optimizer/scheduler, or run-stability troubleshooting.
2. Read references/workflows.md for command patterns, override syntax, CPU/GPU scaling, MDS data paths, and MCLI/platform adaptation.
3. Read references/configuration-reference.md when creating or reviewing a training YAML. It maps required TrainConfig fields, common optional fields, dataloaders, model/tokenizer/optimizer/scheduler sections, evaluation hooks, FSDP/TP, precision, and batch-size fields.
4. Read references/checkpointing-and-callbacks.md for save/load/autoresume, checkpoint naming, HF checkpoint callbacks, run monitors, loggers, optimizers, and schedulers.
5. Read references/troubleshooting.md when launch, data, tokenizer/model, optional dependency, memory, FSDP/TP, or checkpoint upload errors appear.
6. Before launching a long run, run the safe bundled probe:

python scripts/llmfoundry_config_probe.py <config.yaml> \
  variables.data_local=<data-local> \
  train_loader.dataset.split=train_small \
  max_duration=2ba

The probe only parses the YAML and override strings. It does not train, download models or data, initialize distributed state, or write checkpoints. Diagnostic flags such as --json or --strict may appear before or after the YAML path; OmegaConf overrides remain ordinary key=value tokens.

Primary public CLI

Use the installed public CLI for package-level training:

llmfoundry train <config.yaml> [overrides...]

Examples:

llmfoundry train <pretrain.yaml> \
  variables.data_local=<data-local> \
  train_loader.dataset.split=train_small \
  eval_loader.dataset.split=val_small \
  max_duration=10ba \
  eval_interval=0 \
  save_folder=<save-folder>

llmfoundry train <sft.yaml> \
  train_loader.dataset.hf_kwargs.data_dir=<data-local> \
  train_loader.dataset.preprocessing_fn=<python.module>:<function> \
  max_duration=1ep

For multi-process training, the public command still represents the training entry point, but the cluster or launcher must start the required processes and provide distributed environment variables. Platform job YAMLs are reference-only adaptation material; do not copy account-, cluster-, or credential-specific values blindly.

Boundaries

This sub-skill owns:

• Composer YAML creation and adaptation for pretraining, SFT/instruction tuning, domain adaptation, sequence-length adaptation, and small smoke runs.
• TrainConfig required/optional fields that affect training, evaluation hooks inside training, callbacks, algorithms, loggers, optimizer/scheduler, save/load/autoresume, precision, FSDP, TP, and batch sizing.
• Training launch commands and CLI override strings.
• Training-specific troubleshooting.

Route elsewhere:

• Data conversion schemas, prompt/response conversion tools, and MDS writing details: data-preparation.
• Standalone eval task schema, ICL task authoring, and Eval Gauntlet details: evaluation.
• Checkpoint export to HF/ONNX/FasterTransformer and text generation: inference-conversion.
• Model registry internals, MPT configuration depth, custom package registrations, and API extension design: package-apis-configuration.

Safety rules

• Do not start an unbounded training run while inspecting a config. Probe first, then set explicit max_duration, eval_interval, save_interval, data split, and checkpoint destination.
• Do not use local source checkout paths in runtime instructions. Use placeholders such as <config.yaml>, <data-local>, <data-remote>, <save-folder>, <load-checkpoint>, <hf-model>, and <run-name>.
• Treat model and tokenizer downloads, gated Hugging Face models, object-store checkpoints, MCLI jobs, CUDA kernels, Flash Attention, TransformerEngine, and MegaBlocks as environment-dependent.
• Keep conversion and export out of this sub-skill even when a training checkpoint is the input or output of that later workflow.