---

name: local-inference-decoding description: "Use OptiLLM local HuggingFace and MLX inference, LoRA adapters, backend checks, reasoning tokens, logprobs, and advanced decoding methods safely." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Local Inference and Decoding

Use this sub-skill when OptiLLM itself loads and serves local models, or when a task involves local decoding methods and backend troubleshooting.

Read first for these tasks

• Use OPTILLM_API_KEY=optillm to enable the built-in local inference client.
• Serve HuggingFace models, private models, or model strings with LoRA adapters separated by +.
• Configure active_adapter, logprobs, token limits, or reasoning-token accounting.
• Use local decoding modes: cot_decoding, entropy_decoding, thinkdeeper, deepconf, or autothink.
• Diagnose CUDA, MPS, MLX, bitsandbytes, PEFT, transformers, HuggingFace cache, or model download issues.

Route external provider/server setup to ../proxy-server/SKILL.md. Route core approach selection to ../optimization-approaches/SKILL.md. Route JSON/router/deepthink plugins to ../plugins-and-tools/SKILL.md when the plugin is the user-facing surface.

Core workflow

1. Confirm local inference is intended. OPTILLM_API_KEY activates the local engine; unset it for external provider proxying.
2. Probe backend before loading models. Run python scripts/check_local_backend.py --json to inspect torch/CUDA/MPS/transformers/MLX imports without downloads.
3. Choose model string. Use a HuggingFace model id, optionally with LoRA adapters appended by +.
4. Bound generation. Set request max_tokens or OPTILLM_MAX_TOKENS for small models that may not emit EOS.
5. Add decoding only when supported. cot_decoding, entropy_decoding, thinkdeeper, deepconf, and autothink require local model internals/logits, not just an external provider.
6. Validate response fields. Check usage.completion_tokens_details.reasoning_tokens when using <think> tags or reasoning methods.

Minimal local server example

export OPTILLM_API_KEY=optillm
optillm --model meta-llama/Llama-3.2-1B-Instruct

Client:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="optillm")
response = client.chat.completions.create(
    model="meta-llama/Llama-3.2-1B-Instruct",
    messages=[{"role": "user", "content": "Say hello."}],
    max_tokens=64,
)

LoRA model string:

base-model+adapter-one+adapter-two

Select adapter through request body:

extra_body={"active_adapter": "adapter-one"}

References and helper

• references/local-inference.md covers local engine APIs, model/LoRA strings, logprobs, response classes, and request knobs.
• references/decoding-methods.md covers cot_decoding, entropy_decoding, thinkdeeper, deepconf, and autothink.
• references/hardware-and-backends.md covers CUDA/MPS/MLX checks and backend claims.
• references/troubleshooting.md maps model/cache/backend errors to recovery steps.
• Run python scripts/check_local_backend.py --help before loading a real model.

Validation checklist

• Local inference was requested explicitly; external provider env vars are not being shadowed by OPTILLM_API_KEY.
• torch imports and reports the expected backend.
• Private HuggingFace models have a non-empty token set.
• Model and adapter identifiers are accessible or cached.
• Token limits are bounded for smoke tests.
• MLX-specific paths are used only on macOS Apple Silicon.