---

name: vlm-eval-kit description: "Use VLMEvalKit (vlmeval) for vision-language model evaluation, model/API configuration, benchmark authoring, result inspection, and troubleshooting." metadata: disco-role: operating disable-model-invocation: true license: Apache 2.0

VLMEvalKit (vlmeval)

Use this repo skill when a task involves VLMEvalKit, the vlmeval Python package, OpenCompass multimodal/VLM evaluation workflows, supported VLM/API model names, benchmark datasets, run.py, vlmutil, LMUData, prediction/result files, or VLMEvalKit-specific errors.

VLMEvalKit is an evaluation toolkit for large vision-language models. It provides one-command benchmark inference/evaluation, model/API registries, dataset loaders, answer-extraction/judging utilities, video benchmark presets, and helper scripts for data/result workflows.

Start here

1. Install or inspect the package. Use installation and environment for install modes, cache roots, optional dependencies, credentials, and format-related environment variables. Run scripts/check_vlmeval_install.py as a safe local diagnostic when a Python environment is available.
2. Choose the workflow route. Use the sub-skill map below instead of reading all references.
3. Keep verification boundaries explicit. Live API calls need credentials, local VLM runs need model weights/hardware, and dataset builds may download large files. Do not claim those succeeded unless the current task actually runs them.
4. Troubleshoot cross-cutting failures first. For install/import, missing .env, dependency, cache, output-format, or backend failures, read troubleshooting before narrowing to a sub-skill.

Sub-skill routes

• evaluation: run or resume evaluations, build run.py/torchrun/vlmutil commands, use async API mode, configure judges, scan failed API outputs, summarize status.json, and interpret prediction/evaluation artifacts.
• model-development: configure existing VLM/API names, use JSON configs, use OpenAI-compatible/LMDeploy/LiteLLM providers, implement model/API wrappers, register supported_VLM entries, and add prompt adapters.
• benchmark-authoring: create or adapt benchmark TSV/video data, implement dataset classes, define build_prompt and evaluate, register image/text/video datasets, add video presets, and write converters.

Common entry points

Task signal	Best route	First check
"Run MMBench/MME/Video-MME on a model"	evaluation	Decide python run.py vs torchrun; pick --data, --model, --work-dir, --mode, --reuse
"Evaluate an OpenAI-compatible endpoint"	evaluation then model-development if adapter/config fails	--base-url, --key, --custom-prompt, --api-mode, provider media support
"Add a new VLM or API provider"	model-development	BaseModel/BaseAPI contract and supported_VLM registration
"Add a benchmark or custom TSV"	benchmark-authoring	Data columns, build_prompt, evaluate, build_dataset, --data-config
"Why are results missing or failed?"	evaluation	status.json, prediction file, checkpoints, failure markers, reuse settings
"Package won't import or dependencies conflict"	troubleshooting	Python version, broad requirements, optional backend deps, .env behavior

Minimal public import checks

Use these only as diagnostics, not as proof that GPU/API/data workflows are ready:

python - <<'PY'
import vlmeval
print("vlmeval version:", getattr(vlmeval, "__version__", "unknown"))
from vlmeval.api.litellm_api import LiteLLMAPI
from vlmeval.inference_api import APIEvalPipeline, DatasetConfig
print(LiteLLMAPI.__name__, APIEvalPipeline.__name__, DatasetConfig.__name__)
PY

vlmutil dlist l1
vlmutil mlist all | head

A missing .env log during import is expected when API keys are not stored in a repository-local .env; it is not by itself an import failure.

Output and verification boundary

This generated skill is self-contained operating guidance. It distills source evidence from VLMEvalKit docs, package code, tests, and safe helper scripts. It includes bundled scripts for install checks, safer torchrun command review, failure scanning, run-summary extraction, and a LongDocURL-style TSV converter. It does not bundle the full evaluation runner, model weights, datasets, API credentials, or service deployments.

Read repo provenance when checking whether this skill is stale relative to a newer VLMEvalKit checkout. Router metadata for managed import lives in repo routing metadata.