---

name: big-model-inference description: "Use Accelerate big-model inference utilities for meta initialization, device-map planning, checkpoint dispatch, CPU/disk offload, hooks, pipeline inference, and memory sizing without triggering downloads or heavyweight runs." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Big Model Inference

Use this sub-skill when an agent needs to load, inspect, partition, or troubleshoot models that may not fit in a single device's memory. It covers Accelerate's init_empty_weights, infer_auto_device_map, load_checkpoint_and_dispatch, dispatch_model, CPU/disk offload helpers, hook behavior, prepare_pippy, and model-size estimation concepts.

Route First

• For generic accelerate launch, accelerate config, or accelerate estimate-memory CLI syntax, route to ../configuration-and-cli/.
• For DeepSpeed ZeRO-3 training/inference tradeoffs, route to ../distributed-training-backends/.
• For normal Accelerator.prepare training loops, route to the training-oriented sub-skill instead of using big-model dispatch primitives.

Core References

• Read references/workflows.md for safe workflows: meta-device construction, device-map planning, checkpoint dispatch, CPU/disk offload, distributed inference, and pipeline parallelism.
• Read references/api-reference.md for public APIs, key parameters, and import locations.
• Read references/troubleshooting.md when device maps, offload folders, checkpoint indexes, meta tensors, split points, or optional model libraries fail.
• Use scripts/big_model_api_smoke.py --help or run the script directly to verify tiny CPU/meta-device API availability without downloads.

Safety Defaults

• Prefer tiny synthetic torch.nn.Module models for inspection and examples.
• Do not call from_pretrained, snapshot_download, or heavyweight model examples unless the user explicitly wants network/model downloads.
• Treat disk offload as requiring a real writable offload folder and clean it up after experiments.
• Never call .to("cpu"), .cuda(), or a forward pass on a model still initialized on meta; load or set real tensors first.