---

name: observability description: "Configure Composer logging, callbacks, artifact uploading, profiling, environment reports, and monitoring for training runs." metadata: disco-role: operating disable-model-invocation: true license: Apache 2.0

Composer Observability

Use this sub-skill when a Composer task is about making run state visible: metrics/loggers, callback-based monitoring, file and artifact uploads, profiler traces, TensorBoard, environment reports, or diagnosing why observability outputs are missing.

Route here

• Add local metrics capture with InMemoryLogger, ConsoleLogger, ProgressBarLogger, or FileLogger.
• Configure external experiment trackers such as Weights & Biases, MLflow, Comet ML, Neptune, Slack, or TensorBoard.
• Decide which callback owns a monitoring concern: speed, memory, learning-rate, optimizer, NaN/OOM, early stopping, threshold stopping, runtime estimate, image visualization, or MLPerf reporting.
• Upload Composer-generated files such as checkpoints, logs, TensorBoard event files, profiler traces, or custom artifacts.
• Configure Profiler, JSONTraceHandler, cyclic_schedule, and short trace windows.
• Run or interpret composer_collect_env output when debugging hardware, Python, PyTorch, or CUDA availability.
• Diagnose missing log files, empty metric collections, absent uploaded artifacts, invalid remote paths, or profiler option conflicts.

Reroute

• Training loop construction, optimizer/dataloader setup, checkpoint save/load/autoresume semantics: use ../training/SKILL.md.
• Algorithm method behavior, method-specific events, or speedup recipe choices: use ../methods/SKILL.md.
• Distributed launcher, rank topology, backend initialization, FSDP/TP config, or automatic microbatching: use ../distributed/SKILL.md.
• Model export APIs and export callback ownership: use ../inference-export/SKILL.md.

Read first

• Loggers and callbacks: choose logger destinations, monitoring callbacks, run-name placeholders, and upload-capable destinations.
• Profiling and artifacts: configure Profiler, JSONTraceHandler, file upload routing, trace retention, and composer_collect_env.
• Troubleshooting: optional dependencies, credentials, upload gaps, URI parsing, placeholder mistakes, profiler conflicts, and remote validation.
• Logger smoke and profiler smoke: tiny CPU/no-network sanity checks future agents can copy or run in a prepared Composer environment.

Logger decision guide

1. Keep the default progress bar for quick interactive experiments.
2. Add log_to_console=True and console_log_interval="100ba" when stdout/stderr logs are easier to archive than progress bars.
3. Use InMemoryLogger() when a test, smoke script, or agent needs assertions over metrics and hyperparameters after the run.
4. Use FileLogger(filename="{run_name}/logs-rank{rank}.txt") for durable per-rank local text logs.
5. Use TensorboardLogger(log_dir=...) when users expect TensorBoard event files.
6. Use tracker loggers only after installing the matching optional extra and confirming credentials or offline/debug mode.
7. Use RemoteUploaderDownloader(bucket_uri=...) when the task is about object-store upload/download rather than experiment tracking.

Artifact upload checklist

• A class must generate a local file first, such as a checkpoint callback, file logger, TensorBoard logger, profiler trace handler, or custom trainer.logger.upload_file(...) call.
• At least one logger destination must implement upload; otherwise the file remains local.
• Keep rank and run-name placeholders in filenames so multi-rank outputs do not clobber each other.
• If autoresume=True, avoid unsupported concurrent upload settings and keep a stable run_name plus checkpoint latest filename.
• For cloud/object stores, validate credentials and URI scheme separately from Trainer construction.

Profiler decision guide

Use a short window before expanding trace collection:

from composer.profiler import Profiler, JSONTraceHandler, cyclic_schedule

profiler = Profiler(
    schedule=cyclic_schedule(skip_first=0, wait=0, warmup=1, active=4, repeat=1),
    trace_handlers=[JSONTraceHandler(folder="traces", overwrite=True)],
    torch_prof_memory_filename=None,
)

Attach it as Trainer(..., profiler=profiler). Prefer one or two batches first, then increase active or enable stack/memory options when the tiny trace is usable.

Safe workflow

1. Start with a tiny max_duration="1ba" run and InMemoryLogger().
2. Add one durable destination such as FileLogger or TensorBoard.
3. Add only one remote tracker or uploader at a time.
4. If files are generated but missing remotely, inspect upload-capable destinations before changing callbacks.
5. If profiler output is missing, check the schedule first; it may have stayed in SKIP or WARMUP.
6. Record composer_collect_env output when debugging PyTorch/CUDA/device issues, but do not put machine-specific paths into reusable project code.

Bundled smoke scripts

Run from this sub-skill directory:

python scripts/logger_smoke.py
python scripts/profiler_smoke.py

Both scripts use random CPU tensors and temporary directories; they do not contact external services.

Ask or stop before proceeding

• The user wants to log to a credentialed service and no token/offline mode is available.
• The task requires mutating external object storage or deleting existing remote artifacts.
• Profiling must run on a production-scale job rather than a tiny isolated reproduction.
• The issue depends on multi-rank launch behavior rather than logger/profiler configuration.
• Optional logger packages are absent and installation policy is unclear.