---

name: petals description: "Use Petals for distributed large-language-model inference, prompt tuning, server swarms, block-level internals, benchmarks, and package troubleshooting." disable-model-invocation: true metadata: disco-role: operating license: MIT

Petals Repo Skill

Use this skill when a task involves Petals, the Python package for collaborative distributed inference and prompt tuning of large language models whose transformer blocks are hosted by remote peers.

This skill is self-contained. Do not rely on a repository checkout being present unless the user explicitly says they are maintaining one; use the bundled references and scripts below for ordinary package workflows.

Start here

1. For install, import, dependency, cache, model-access, and backend readiness, read references/installation-and-environment.md.

2. Run the safe root checker when you need a no-network local sanity check:

   python scripts/check_petals_environment.py --pretty
   

   A passing checker means local imports, API discovery, and CLI parser checks worked. It does not prove public swarm health, model access, GPU serving performance, or adapter execution.

3. Route the user request to the nearest sub-skill below.

4. For cross-cutting failures, read references/troubleshooting.md.

5. Before refreshing this skill against a newer Petals release, read references/repo-provenance.md.

Sub-skill routing

User intent	Read this
Write client code for distributed generation, sequence classification, public/private swarm clients, AutoDistributedModel*, .generate(), inference_session, routing or retry options	sub-skills/client-inference/SKILL.md
Host model blocks, start a server, bootstrap a private DHT, construct python -m petals.cli.run_server / run_dht commands, reason about ports, identities, cache, device, dtype, quantization, adapters, or reachability	sub-skills/server-swarms/SKILL.md
Adapt a distributed model with tuning_mode="ptune" or "deep_ptune", train prompt embeddings/classifier heads, plan causal-LM or classification prompt tuning, or troubleshoot PEFT adapter safety	sub-skills/prompt-tuning/SKILL.md
Work with RemoteSequential, block slices, hidden-state tensors, load_pretrained_block, QuantType, tensor parallel conversion, speculative Llama internals, dtype resolution, or block-level errors	sub-skills/distributed-blocks/SKILL.md
Build benchmark command templates, plan tiny smoke checks, select focused native maintainer checks, or interpret benchmark/CI health signals	sub-skills/benchmarks-maintenance/SKILL.md

Core operating model

• Petals model classes are Transformers-compatible wrappers. Embeddings and heads live on the client; transformer blocks are discovered through a Hivemind DHT and executed by remote Petals servers.
• The default public swarm is external: availability, hosted block coverage, peer latency, and gated model access can change independently of local package health.
• Private swarms require consistent initial_peers, dht_prefix, model identifier, and block ranges across clients and servers.
• Generation needs a remote attention-cache budget. Without an active session, pass exactly one of max_new_tokens or max_length; for interactive continuation, explicitly create and reuse inference_session(max_length=...).
• Prompt tuning trains local prompt embeddings and, for classification models, local heads. The client does not update remote transformer blocks hosted by servers.
• Server-side quantization and adapters are optional backend features. Verify the actual torch/CUDA/ROCm/MPS/bitsandbytes stack before claiming they work.

Install and minimal import

python -m pip install petals
python - <<'PY'
import petals
from petals import AutoDistributedModelForCausalLM
print(petals.__version__)
PY

For a checkout maintained by the user, use that checkout's package metadata to choose the Python version and dependencies, then run the same import check from outside the checkout directory. Do not treat import success as proof that a public swarm, gated model, GPU serving, or adapter loading is available.