---

name: quantization-integrations description: "Choose and validate Transformers quantization methods, backend integrations, device placement, PEFT/Accelerate/DeepSpeed/FSDP/kernel interactions, and no-download config smoke checks." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Transformers Quantization Integrations

Use this sub-skill when the task is to reduce model memory, load or create quantized checkpoints, choose a quantization backend, combine quantization with PEFT/training/serving, or diagnose backend-specific optional dependency and device placement failures.

Transformers inspected version: 5.13.0.dev0. Import package: transformers. A minimal environment can import base Transformers, but most quantization workflows require torch plus backend packages such as accelerate, bitsandbytes, gptqmodel, autoawq, torchao, peft, deepspeed, or hardware-specific kernels.

Route Here

• Choose between BitsAndBytesConfig, GPTQConfig, AwqConfig, TorchAoConfig, GGUF/GGML loading, compressed-tensors, Quanto, HQQ, AQLM, EETQ, FP8, Quark, VPTQ, SpQR, SINQ, Metal, or other quantization_config methods.
• Decide whether a user needs on-the-fly quantization, a pre-quantized checkpoint, or a calibration/quantization job.
• Configure AutoModelForCausalLM.from_pretrained(..., quantization_config=..., device_map=..., dtype="auto") without downloading models during planning.
• Combine quantized models with PEFT/QLoRA, LoRA adapters, Accelerate device_map, CPU/disk offload, DeepSpeed, FSDP, tensor parallelism, or custom kernels.
• Check install and hardware constraints for CPU, CUDA, ROCm, MPS/Metal, Intel XPU, Gaudi/HPU, and torch.compile.
• Diagnose quantized checkpoint mismatch, missing optional packages, unsupported kernels, serialization limits, offload conflicts, or continuous batching incompatibilities.

Route Elsewhere

• Inference task selection, pipeline(...) kwargs, and pipeline output behavior: inference-pipelines.
• Decoding controls, GenerationConfig, streamers, and generation-only continuous batching concepts: generation.
• Trainer, TrainingArguments, datasets, collators, and ordinary fine-tuning loops: training.
• Tokenizer, processor, chat template, image/audio/video preprocessing, and special-token choices: tokenizers-processors.
• transformers CLI commands, HTTP serving, and endpoint routing: serving-cli.
• Adding new architectures, configs, model files, or processors: model-extension.

Fast Decision Checklist

1. Identify the goal: memory-only loading, faster inference, CPU-only inference, QLoRA/PEFT fine-tuning, pre-quantized checkpoint loading, or creating a new quantized checkpoint.
2. Identify hardware: CPU, CUDA GPU generation, ROCm, Apple Silicon/Metal/MPS, Intel XPU, Gaudi/HPU, multi-GPU, or no backend available.
3. Prefer easy on-the-fly paths first: bitsandbytes for common 4/8-bit GPU workflows, torchao for PyTorch/compile-oriented paths, HQQ/SINQ/Quanto when their backend matches the host.
4. Prefer pre-quantized paths when calibration is expensive: AWQ and GPTQ are often loaded as existing 4-bit checkpoints; creating them requires calibration data and time.
5. Keep dtype="auto" and device_map="auto" deliberate: they are useful for large models but can conflict with tensor parallelism, offload, or unsupported backends.
6. Validate config construction locally before downloads with scripts/quantization_config_smoke.py.
7. Separate config validation from model execution: constructing a config does not prove kernels, GPU drivers, checkpoint metadata, or memory capacity.
8. For quantized training, update only extra parameters unless the backend explicitly supports broader training; QLoRA usually means frozen 4-bit base weights plus PEFT adapters.

Common Starting Points

• Choose a method and backend matrix: quantization methods.
• Combine with Accelerate, PEFT, DeepSpeed, FSDP, tensor parallelism, kernels, or serving: integration workflows.
• Check optional package, hardware, compile, serialization, and device placement constraints: compatibility.
• Diagnose failures and expected signals: troubleshooting.

Canonical Loading Pattern

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="bfloat16",
)
model = AutoModelForCausalLM.from_pretrained(
    "org/model",
    quantization_config=quantization_config,
    device_map="auto",
    dtype="auto",
)

Use this as a shape, not a universal default. Replace BitsAndBytesConfig when the checkpoint already declares another method, when the host is CPU-only and GGUF is the intended format, or when the backend requires calibration, specialized kernels, or a different package.

No-download Config Smoke Check

Run the bundled helper from this sub-skill directory or pass its path explicitly:

python scripts/quantization_config_smoke.py --method bitsandbytes-4bit --print-json
python scripts/quantization_config_smoke.py --method gptq --bits 4 --dataset c4
python scripts/quantization_config_smoke.py --method awq --bits 4 --fuse-max-seq-len 512

Expected success signal: OK quantization config validated plus a summary of the config class and options. Expected optional-dependency signal: MISSING optional dependency with install guidance. Expected validation failure: nonzero exit with ERROR and the invalid option.

Output Expectations For Agent Answers

For quantization tasks, return the selected method, required packages, required hardware/backend, exact quantization_config or loading skeleton, device_map/offload decisions, PEFT/training/serving interaction notes, smoke-check command, expected success signals, and remaining risks such as calibration time, unsupported kernels, serialization limits, or checkpoint metadata mismatch.

Optional Dependency Boundary

A base transformers import is not enough for real quantized model execution. Many config classes can be inspected or constructed without downloading a model, but model loading typically requires torch, accelerate, and the backend package. Missing dependencies commonly raise optional dependency ImportErrors at class import, config validation, or model load time.