---

name: lightllm description: "Route LightLLM serving, deployment, and validation workflows into bundled references and sub-skills." metadata: disco-role: operating disable-model-invocation: true license: Apache 2.0

LightLLM

LightLLM is a GPU-accelerated LLM serving framework with OpenAI-compatible, Anthropic-compatible, TGI-style, and LightLLM-native request paths. Use this skill when a user wants to start the server, understand its HTTP surface, choose a supported model/backend combination, or validate the runtime with repo-native benchmark and smoke flows.

Quick verification

If the package is not already installed, install the public lightllm distribution into a CUDA-capable Python environment first, for example with python -m pip install lightllm, then confirm that the installation is usable.

A healthy inspection environment should be able to do all of the following:

1. Import the installed lightllm distribution.
2. Pass python -m pip check.
3. Report CUDA readiness from the target environment.
4. Print python -m lightllm.server.api_server --help.

Use the bundled helper scripts when you need a fast check:

• scripts/inspect_cuda.py for a tiny torch/CUDA smoke.
• scripts/inspect_start_args.py for the live StartArgs surface.
• scripts/inspect_api_surface.py for route and request-schema inspection.
• scripts/request_smoke.py for a local /generate or /v1/* request against a running server.

Route map

serving-api

Use this sub-skill for HTTP serving, endpoint payloads, streaming, function calling, reasoning parsers, multimodal requests, health/readiness, metrics, and profiler control.

Read:

• sub-skills/serving-api/SKILL.md
• references/api-reference.md
• references/cli-reference.md

model-runtime

Use this sub-skill for supported model families, registry behavior, quantization/backend selection, multimodal model toggles, reward/RL model support, and adding a new model support path.

Read:

• sub-skills/model-runtime/SKILL.md
• sub-skills/model-runtime/references/model-support.md

deployment-topologies

Use this sub-skill for single-node and multi-node launch topologies, pd_master / prefill / decode layouts, config server flows, multimodal worker placement, cache modes, RDMA/NIXL/UCX caveats, and startup sequencing.

Read:

• sub-skills/deployment-topologies/SKILL.md
• sub-skills/deployment-topologies/references/deployment-matrix.md

benchmark-validation

Use this sub-skill for repository benchmarks, service throughput/QPS checks, static inference tests, prompt-cache timing, scenario regressions, and selected accuracy/validation scripts.

Read:

• sub-skills/benchmark-validation/SKILL.md
• sub-skills/benchmark-validation/references/benchmark-catalog.md

Cross-cutting references

• references/repo-provenance.md records the source commit, dirty state, and evidence set.
• references/repo-routing-metadata.json captures router placement and selection guidance.
• references/troubleshooting.md summarizes install, import, CUDA, optional dependency, and startup failures that cut across sub-skills.

Runtime notes

• Treat the bundled references and scripts as the public operating context. Do not depend on the original checkout remaining available.
• Prefer the smallest workflow that answers the user’s question: serving requests go to serving-api, model support questions go to model-runtime, topology and startup sequencing go to deployment-topologies, and benchmark or regression questions go to benchmark-validation.
• For questions that mix multiple areas, start with the sub-skill that owns the user-facing action, then read the cross-cutting references above.
• If a workflow needs a running server, use deployment-topologies first and then return here for the API or benchmark route.
• If a question is about profiler control only, treat it as part of serving-api and benchmark-validation, not as a separate route.

Staleness check

If the repository version, package version, or supported backend story changes, refresh the provenance and routing metadata before reusing this skill.