---

name: benchmark-validation description: "Validate LightLLM with repo-native benchmarks, regressions, static inference, and smoke flows." metadata: disco-role: operating disable-model-invocation: true license: Apache 2.0

benchmark-validation

Use this sub-skill when the user wants to benchmark LightLLM, compare throughput or latency, replay a regression scenario, or interpret one of the repo-native accuracy and performance scripts.

Covers

• Service benchmarks for throughput, QPS, and multi-turn latency.
• Static inference benchmarks for text and vision models.
• Prompt-cache and PD-related benchmark flows.
• Accuracy / regression / scenario scripts under test/acc.
• Log and result-file conventions for repeatable benchmark runs.

Does not cover

• How to start the server topology itself.
• How to choose a model family or backend.
• Endpoint payload details beyond the smoke request used to warm up a run.

Read first

• references/benchmark-catalog.md
• references/troubleshooting.md
• ../../references/troubleshooting.md
• ../../references/api-reference.md

Use this route when the user says

• “benchmark LightLLM”
• “measure QPS or latency”
• “run the prompt-cache test”
• “validate the model with the repo benchmark scripts”
• “compare a performance run or regression result”

Minimal working sequence

1. Confirm the service or deployment route is already up.
2. Choose the smallest benchmark that matches the user’s question.
3. Run a local smoke request before a larger benchmark or scenario script.
4. Record inputs, model path, endpoint, and log directory before collecting results.
5. Decide whether the benchmark is throughput-focused, multi-turn, or accuracy/regression-focused.

Decision points

• Use service QPS and latency scripts for serving performance.
• Use static inference scripts for model-side inference timing.
• Use prompt-cache scripts when the user cares about cache reuse.
• Use test/acc scripts when the task is regression or scenario validation.
• Keep profiler control in the serving route unless the benchmark itself needs profiler coordination.

Related helpers

• ../../scripts/request_smoke.py is the small local request used before a larger benchmark.
• ../../scripts/inspect_api_surface.py helps map the benchmark target to the right endpoint family.

Troubleshooting highlights

• The benchmark can start before the service is actually ready.
• lm_eval and similar tools may slow down if caches are missing or online downloads are allowed unexpectedly.
• Model/tokenizer mismatches often show up as benchmark setup failures rather than bad performance numbers.
• A benchmark against the wrong endpoint family gives misleading latency or throughput numbers.
• Proxy leakage can make localhost benchmark traffic fail in confusing ways.

Review standard

This sub-skill is complete when a future agent can:

• pick the correct benchmark family,
• understand the expected logs and summary files,
• run a local smoke before the larger benchmark,
• and explain a suspicious result without reopening the source repository.