---

name: api-serving description: "Guides ChatGLM2-6B FastAPI and OpenAI-compatible serving, request validation, conversation-history mapping, streaming SSE, and safe local deployment." disable-model-invocation: true metadata: disco-role: operating license: NOASSERTION

ChatGLM2-6B API Serving

Use this route for the repo's HTTP services, curl/OpenAI client integration, SSE streaming, endpoint contracts, or service troubleshooting. Use chat-and-demos for local UI/CLI generation and ptuning when the model is a prefix/full fine-tuned checkpoint.

Choose the endpoint

• Simple API: a FastAPI POST / accepting prompt, optional history, max_length, top_p, and temperature. It returns response, updated history, status, and a timestamp.
• OpenAI-compatible API: GET /v1/models plus POST /v1/chat/completions. Use this for clients that already speak the Chat Completions protocol. The repository supports normal JSON responses and Server-Sent Event streaming.

Read api-contracts.md before writing a client. Read deployment.md before starting Uvicorn; the service loads model weights once, keeps global model/tokenizer state, and binds a listener. Run the bundled validators first:

python sub-skills/api-serving/scripts/validate_api_payload.py --json '{"prompt":"你好","history":[]}'
python sub-skills/api-serving/scripts/validate_openai_messages.py --json '{"model":"chatglm2-6b","messages":[{"role":"user","content":"你好"}]}'

Serve safely

1. Select a complete model directory or Hub id and a backend; the repo's launch path calls .cuda() and assumes model weights fit on one or more GPUs.
2. Install fastapi, uvicorn, and sse-starlette for the OpenAI streaming route in addition to the model runtime.
3. Bind to localhost while testing. Use one worker unless you deliberately duplicate model memory and have an external process policy.
4. Validate request roles/history before sending traffic. For streaming, parse JSON chunks until the final finish_reason chunk and [DONE].
5. Treat CORS and allow_origins=["*"] as development defaults, not a production security policy. Add authentication, origin restrictions, rate limits, and timeouts outside this legacy sample.

The source route introspection is safe and confirms /, /v1/models, and /v1/chat/completions; it does not prove that a model can load. Use troubleshooting.md for model, port, role, SSE, and memory failures.