---

name: models-and-serving description: "Configure DB-GPT 0.8.1 model providers and local backends, validate model TOML, operate model controller/worker/API-server roles, use model-management CLI commands, and diagnose serving failures without mistaking an import or registry record for a live deployment." metadata: disco-role: operating disable-model-invocation: true license: MIT

Models and serving

Use this route when a DB-GPT task mentions model/provider TOML, models.llms, embeddings or rerankers, OpenAI-compatible endpoints, Ollama, Hugging Face, vLLM, llama.cpp, MLX, model workers, controller, model API server, model CLI, model registration, health/heartbeat, GPU memory, quantization, or serving logs.

This route owns model selection and the model-service topology. Route general installation, profile creation, workspace paths, and generic start webserver setup to setup-and-cli; route agents/AWEL to agents-and-awel; route public client CRUD and sandbox calls to apis-client-and-sandbox; route document, embedding-pipeline, vector-store, and knowledge workflows to data-and-rag.

Operating contract

• Work against the public DB-GPT 0.8.1 package interface, not a source checkout.
• Treat a TOML parse, Python import, controller registry row, or healthy value observed before a real request as configuration evidence only. Deployment is successful only after the intended service is reachable and a suitable model operation succeeds.
• Keep provider credentials in environment variables. Never print resolved secrets, copy them into generated files, or put them in shell history.
• Choose one model role explicitly: llm, text2vec (embedding), or reranker. A chat model cannot silently substitute for an embedding model.
• CPU is the verified baseline for this route. CUDA/local-model, vLLM, bitsandbytes, llama.cpp acceleration, and MLX are optional backend paths; do not claim they work from a CPU import or from host GPU visibility alone.

Read the focused contracts before making changes:

• model-api-reference.md — configuration dataclasses, model/provider fields, worker types, and OpenAI-compatible API surface.
• provider-and-backend-matrix.md — provider credentials, endpoint conventions, embedding pairing, optional dependencies, and backend boundaries.
• serving-workflows.md — proxy/local TOML, unified webserver, split controller/worker/API-server deployment, and CLI operations.
• backends.md — local-model, quantization, VRAM, and hardware-specific decision rules.
• troubleshooting.md — symptom-to-check diagnosis and recovery boundaries.

Standard workflow

1. Identify the topology and model roles. Decide whether the user needs a unified webserver process, a split cluster, a standalone model API server, or only a provider configuration. Record the LLM, embedding, and optional reranker names separately.

2. Select the provider/backend. Prefer a remote proxy for a CPU-first smoke test. For a local model, require a real model path or an explicitly allowed model download and select hf, vllm, llama.cpp, llama.cpp.server, or mlx deliberately. Check the matrix for the provider-specific key and endpoint.

3. Write a redacted TOML. Put model entries under [models] using repeated [[models.llms]], [[models.embeddings]], and [[models.rerankers]] tables. Use ${env:NAME} or ${env:NAME:-default} placeholders. Set default_llm, default_embedding, or default_reranker only when multiple entries make selection ambiguous. Keep web/database configuration owned by the setup route unless the serving topology requires the related model service section.

4. Validate without network or model loading. Run the bundled checker:

   python scripts/model_config_check.py path/to/config.toml
   

   It parses TOML, checks model-table shape, provider/role requirements, duplicate names, placeholder syntax, endpoint URLs, and obvious service-port collisions. It never resolves secrets, calls a provider, contacts a controller, downloads a model, or starts a process. Use --json for a machine-readable report and --allow-missing-embeddings only when the configuration is intentionally chat-only; RAG/knowledge workflows must not use that exception.

5. Run a safe package-level check. Use dbgpt --version, the relevant static --help, and import/config-parser checks. Do not use dbgpt model start --help or another dynamic model command as an offline gate: model command discovery can query a controller and return a timeout/502 when no controller is reachable.

6. Start only the chosen topology. Use the exact commands in serving-workflows.md. Do not start both a unified model manager and an independent worker on the same ports. For split mode, start and verify the controller before workers, then verify workers before the API/web layer.

7. Verify reachability and model capability. Check the controller health and registry, then issue a minimal request through the intended API or provider. dbgpt model list is a registry/heartbeat view, not proof that generation or embedding works. An unreachable controller, failed heartbeat, missing model, authentication failure, or failed embedding request is a deployment failure; report it as such with the exact next check.

8. Diagnose from the right boundary. Inspect the command's resolved config path, service logs, bind address, port, controller address, provider base URL, model name, and optional-backend imports in that order. Redact API keys and authorization headers in reports.

CLI routing facts

The DB-GPT CLI registers model commands under dbgpt model and service commands under dbgpt start / dbgpt stop:

dbgpt model [--address CONTROLLER_URL] list|start|stop|restart|chat
dbgpt start controller --config FILE [--daemon]
dbgpt start worker --config FILE [--daemon]
dbgpt start apiserver --config FILE [--daemon]
dbgpt start webserver [web options]
dbgpt stop controller|worker|apiserver [--port PORT]

CONTROLLER_ADDRESS can supply the controller address used by model-management commands when --address is omitted. The dynamic model start command obtains supported-model metadata from the controller before constructing some options; therefore a failed remote request during help or invocation is not a successful start. For model-service launch commands, the configuration file is the source of worker/model parameters; do not rely on stale examples that show model flags which are not present in the installed command tree.

Validation and boundaries

A valid result for this route includes:

• provider/model/role entries parse with no secret resolution or network call;
• an embedding entry exists and is compatible with the intended RAG path;
• each selected local backend's dependency, path, device, and VRAM assumptions are explicit;
• service roles have distinct, reachable addresses and non-conflicting ports;
• controller registry and heartbeat checks are separated from generation, completion, embedding, or reranking checks;
• optional CUDA/GPU/provider/service dependencies are labeled unverified when they were not executed.

Do not claim that dbgpt model list proves deployment, that --daemon proves a healthy child process, that a provider key exists merely because a placeholder parsed, or that an A100-visible host proves CUDA/torch/vLLM/bitsandbytes support. Do not add API CRUD, agent orchestration, or RAG chunking instructions here.