---

name: chat-and-demos description: "Guides ChatGLM2-6B local generation, streaming chat, CLI and Gradio/Streamlit demos, quantization, CPU/MPS fallback, and multi-GPU placement." disable-model-invocation: true metadata: disco-role: operating license: NOASSERTION

ChatGLM2-6B Chat and Demos

Use this route when the task asks to load ChatGLM2-6B locally, generate a response, preserve conversation history, run the terminal/Gradio/Streamlit examples, use quantization, or split a model across GPUs. Use api-serving for HTTP endpoints, ptuning for tuned checkpoints, and evaluation for C-Eval.

First decisions

1. Identify the model source: a Hub id such as THUDM/chatglm2-6b, a pinned Hub revision such as v1.0, or a complete local model directory. Model weights are not bundled with this skill.
2. Choose the backend before writing a command. The repository's official demos call .cuda(). CPU inference is slower and memory-heavy; Mac MPS requires a local model path and a compatible PyTorch build.
3. Check memory, model variant, and context limits. FP16/BF16 is documented as roughly 13 GB; INT4 reduces the reported minimum to about 5–6 GB but uses quantization kernels and can change quality.
4. Run the bundled environment check before loading weights: python sub-skills/chat-and-demos/scripts/check_chatglm2_environment.py --backend auto.
5. For a multi-GPU plan, inspect the deterministic placement first: python sub-skills/chat-and-demos/scripts/inspect_device_map.py --num-gpus 2.

Common loading shape

Use the Transformers remote-code contract and keep the model in evaluation mode:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_OR_PATH, trust_remote_code=True, revision=REVISION  # optional
)
model = AutoModel.from_pretrained(
    MODEL_OR_PATH, trust_remote_code=True, revision=REVISION  # optional
).eval()

For a normal CUDA model, move the model to CUDA after loading. For CPU use .float() and expect much slower generation. For Apple MPS, load from a local path and use .to("mps"); the repository documents this as a Mac-specific path, not a Linux fallback.

The usual single-turn call is model.chat(tokenizer, query, history=[]) and returns (response, history). For incremental output use model.stream_chat(...); pass the returned history into the next turn. The CLI and web demos also retain past_key_values when the model implementation supports it. Do not mix a prefix-tuned checkpoint with the base-model route; use ptuning for that distinction.

Demo routing

• CLI: use the interactive loop only after model loading succeeds. Enter a prompt, clear to reset history, and stop to exit. This is a blocking interactive process, not a smoke test.
• Gradio: use the legacy-compatible dependency range documented in demo-workflows.md. The source calls the deprecated .style() method; recent Gradio 6 releases fail before launch.
• Streamlit: launch with streamlit run and keep model loading in the cached resource path. The sidebar controls max_length, top_p, and temperature; the session stores history and past key values.
• Quantization: use a model/checkpoint that supports the requested 4- or 8-bit route. Do not claim that a CPU or MPS run supports CUDA-only INT4 kernels; read the compatibility notes before choosing a device.
• Multi-GPU: call the bundled map inspector first. The repository's helper keeps embeddings, final layer norm, output layer, rotary positions, and lm_head on GPU 0, then distributes encoder layers. Use accelerate for dispatch and verify the actual map on the target model.

Read model-loading.md for loading and device choices, demo-workflows.md for command recipes, and troubleshooting.md when a model, backend, UI, or memory error appears. The bundled helpers are safe preflight tools; they never download weights or start a server.