---

name: tokenizers-processors description: "Use Transformers tokenizers, processors, image/audio/video processors, chat templates, special tokens, BatchEncoding alignment, and preprocessing save/load validation." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Tokenizers And Processors

Use this sub-skill when an agent needs to prepare model inputs or inspect preprocessing artifacts with Transformers 5.13.0.dev0. Prefer AutoTokenizer, AutoProcessor, AutoImageProcessor, and model-specific processor classes over hand-built preprocessing unless the model docs or local files prove a custom class is required.

Route Here

Route tokenizer and processor work here when the task asks to:

• Load or validate a tokenizer with AutoTokenizer.from_pretrained(...), including local directories and Hub ids.
• Choose fast vs slow tokenizer behavior, alignment APIs, or optional tokenization backends.
• Configure padding, truncation, special tokens, chat templates, or decode/postprocess behavior.
• Use AutoProcessor, ProcessorMixin, image processors, feature extractors, video processors, or multimodal processors.
• Validate that save_pretrained(...) and from_pretrained(...) round-trip preprocessing files.
• Preflight tokenizer/processor inputs before training, generation, pipelines, or serving.

Route Elsewhere

• Inference pipeline assembly, task selection, device, dtype, or device_map: use ../inference-pipelines/SKILL.md.
• Decoding strategy, GenerationConfig, streamers, or model.generate(...): use ../generation/SKILL.md.
• Trainer, TrainingArguments, datasets, collators, or distributed training: use ../training/SKILL.md.
• Server and CLI behavior, including the transformers console script: use ../serving-cli/SKILL.md.
• Quantization packages, adapters, accelerate, or backend integrations: use ../quantization-integrations/SKILL.md.
• New model classes, configs, generated modular files, or embedding resize implementation: use ../model-extension/SKILL.md.

Canonical APIs

• AutoTokenizer.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs) selects the tokenizer class from local or remote metadata.
• tokenizer(...) returns a BatchEncoding with model inputs such as input_ids, attention_mask, and optional masks or offsets.
• tokenizer.decode(...) and tokenizer.batch_decode(...) convert ids back to text; use skip_special_tokens=True for user-visible text.
• tokenizer.add_special_tokens(...) and extra_special_tokens=... register structural or multimodal tokens.
• tokenizer.apply_chat_template(...) formats role/content messages for chat models.
• AutoProcessor.from_pretrained(...) loads multimodal processors that wrap tokenizers plus image, audio, or video preprocessing.
• AutoImageProcessor.from_pretrained(...) loads image preprocessing configs, optionally with backend="torchvision" or backend="pil" where supported.
• processor.apply_chat_template(...) can format multimodal chat and return text-only or model-ready dictionaries depending on options.
• save_pretrained(...) and from_pretrained(...) validate portable tokenizer/processor artifacts.

See references/api-reference.md for argument choices and object behavior.

Safe Loading Pattern

Prefer local-first loading during validation so a task does not unexpectedly require network access:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    model_or_path,
    local_files_only=True,
    use_fast=True,
)

If local files are unavailable and network access is explicitly allowed, retry with local_files_only=False. Use trust_remote_code=True only when the user accepts custom repository code execution and the source is trusted.

Tokenization Decisions

• Set padding=True for dynamic batch padding or padding="max_length" with max_length=... for fixed-size export/evaluation flows.
• Set truncation=True and an explicit max_length when overflow must be controlled; inspect special_tokens_mask or sequence length when debugging.
• Use return_tensors="pt", "np", or another supported framework only when that dependency is installed.
• Use return_offsets_mapping=True and alignment helpers only with fast tokenizers; slow/Python tokenizers may not expose identical mappings.
• Avoid calling the tokenizer again with add_special_tokens=True after apply_chat_template(tokenize=False) unless you have verified the template lacks BOS/EOS tokens.

Chat Template Pattern

Chat inputs are a list of dictionaries, usually with role and content keys:

messages = [
    {"role": "system", "content": "You are concise."},
    {"role": "user", "content": "Summarize tokenization."},
]
encoded = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
)

Use continue_final_message=True for prefill/continuation workflows instead of add_generation_prompt=True; do not set both. For multimodal chat, prefer processor.apply_chat_template(...) and pass modality payloads in the schema expected by that processor.

Special Tokens And Resizing

Adding new tokens changes tokenizer vocabulary. If a model will consume the new token ids, the model embedding matrix must also be resized. This sub-skill helps detect and explain the tokenizer side; route model mutation to ../model-extension/SKILL.md or the owning generation/training workflow.

Validation signals:

• added = tokenizer.add_special_tokens({...}) returns the number of newly added tokens.
• len(tokenizer) increases when new tokens are added.
• Named multimodal tokens from extra_special_tokens={"image_token": "<image>"} become accessible where the tokenizer exposes model-specific token attributes.
• Saved tokenizer directories include tokenizer config, special token metadata, and backend vocabulary files required by that tokenizer class.

Processor Decisions

Use processors when the model expects non-text inputs or a coupled tokenizer/image/audio/video contract. Common patterns:

• Text plus images: AutoProcessor.from_pretrained(...), then processor(text=..., images=..., return_tensors=...).
• Vision-only preprocessing: AutoImageProcessor.from_pretrained(...), then image_processor(images=..., return_tensors=...).
• Audio/video processors: load with AutoProcessor or the model-specific processor and check optional dependencies before tensor conversion.
• Save/load checks: call processor.save_pretrained(tmp_dir) and reload with AutoProcessor.from_pretrained(tmp_dir, local_files_only=True) when the processor class supports auto loading.

Processor details and modality formats are in references/data-formats.md and references/workflows.md.

Bundled Smoke Check

Use the bundled script to inspect tokenizer loading and basic behavior without requiring network by default:

python scripts/tokenizer_smoke.py --model-or-path ./tokenizer_dir --text "Hello world" --local-files-only

Allow Hub resolution only when the user permits network access:

python scripts/tokenizer_smoke.py --model-or-path bert-base-uncased --text "Hello" --allow-remote

The script prints load metadata, special tokens, tokenized keys/lengths, decoded text, and optional chat-template output.

Expected Checks

Before handing tokenizer/processor work to a sibling workflow, verify:

• from_pretrained(...) loads with intended local_files_only, use_fast, and trust settings.
• Encoded outputs contain the keys the downstream model or pipeline expects.
• Padding/truncation produce rectangular batches when tensors are requested.
• Added special tokens are saved and reloadable.
• Chat templates format the intended role order and do not duplicate BOS/EOS tokens.
• Optional modality dependencies are installed or gracefully skipped.

Troubleshooting Index

• Backend package errors (tokenizers, sentencepiece, tiktoken, mistral-common): references/troubleshooting.md.
• Alignment or offset surprises: references/api-reference.md and references/troubleshooting.md.
• Multimodal processor schemas and image/audio/video inputs: references/data-formats.md.
• Save/load and preflight workflows: references/workflows.md.