---

name: model-development description: "Configure and extend VLMEvalKit model wrappers, API providers, registries, and prompt adapters." metadata: disco-role: operating disable-model-invocation: true license: Apache 2.0

VLMEvalKit model development

Use this sub-skill when the task is to configure an existing VLMEvalKit model name, add or adapt a local VLM wrapper, use LiteLLM/LMDeploy/OpenAI-compatible APIs, register model aliases, or implement dataset-aware prompt adapters. VLMEvalKit is the distribution/import package vlmeval; its main evaluation CLI is run.py, and the console helper is vlmutil.

Route here when

• A user needs to choose between an existing supported_VLM name, JSON model config, --base-url, or a new wrapper/adapter.
• A wrapper must satisfy BaseModel.generate() / generate_inner() / optional chat_inner() contracts.
• An API provider must satisfy BaseAPI.generate() / generate_inner() ret-code contracts.
• A LiteLLM, LMDeploy, OpenAI-compatible, or custom prompt-adapter setup fails before or during model invocation.
• A model-specific prompt, image/video conversion, device split, dependency version, or thinking-output issue must be diagnosed.

Route elsewhere

• Running evaluation jobs, reuse/resume behavior, output files, status.json, scans, and summaries: evaluation.
• Dataset/benchmark authoring, TSV/video formats, prompt construction on the dataset side, metrics, judges, and converters: benchmark-authoring.

First decision: configure or implement?

1. Existing registry name: discover names with vlmutil mlist all or narrower categories, then use the exact key from vlmeval/config.py.
2. JSON config: use run.py --config when the user needs local aliases or per-model kwargs without editing package files.
3. OpenAI-compatible endpoint: use run.py --base-url when a local/remote service already exposes a chat-completions-compatible endpoint; this route constructs LMDeployAPI without editing config.py.
4. New wrapper or adapter: implement the smallest contract in vlmeval/vlm/, vlmeval/api/, or vlmeval/api/adapters/, then register it in vlmeval/config.py or a JSON config.

Reference map

• API reference: BaseModel, BaseAPI, LiteLLM, LMDeploy/OpenAI-compatible invocation contracts.
• Model registry: supported_VLM, model_groups, vlmutil mlist, JSON config, and --base-url selection rules.
• Prompt adapters: adapter registry, use_custom_prompt, process_inputs, process_payload, postprocess, and thinking split patterns.
• Troubleshooting: missing deps/keys, bad base URLs, media formats, custom prompt mismatches, device/WORLD_SIZE issues, compatibility notes, and thinking-output failures.

Validation checklist

Before routing a model-development change back to evaluation:

• Confirm the chosen model name appears in supported_VLM or is defined in the JSON model section, unless --base-url is intentionally used.
• Confirm wrappers accept VLMEvalKit message lists with allowed types text, image, and when supported video.
• Confirm local VLM wrappers implement generate_inner(message, dataset=None); only add chat_inner when multi-turn chat is supported.
• Confirm API wrappers return (ret_code, answer, log) from generate_inner and let BaseAPI.generate() handle retry/failure behavior.
• Confirm prompt adapters are registered and selected by name when using --custom-prompt or custom_prompt kwargs.
• Treat CUDA, live provider calls, dataset downloads, Gradio services, and large model evaluations as unverified unless the current task explicitly verifies them.

Source evidence distilled

Primary evidence: docs/en/Development.md, docs/en/Quickstart.md, docs/en/EvalByLMDeploy.md, docs/en/ConfigSystem.md, README.md, run.py, vlmeval/vlm/base.py, vlmeval/vlm/__init__.py, vlmeval/api/base.py, vlmeval/api/litellm_api.py, vlmeval/api/lmdeploy.py, vlmeval/api/openai_sdk.py, vlmeval/api/adapters/base.py, vlmeval/api/adapters/internvl2.py, vlmeval/api/adapters/internvl3.py, vlmeval/api/adapters/interns1_1.py, vlmeval/api/__init__.py, vlmeval/config.py, and tests/test_litellm_api.py.