---

name: benchmark-authoring description: "Author and adapt VLMEvalKit benchmark datasets, TSV/video schemas, converters, dataset registration, prompts, and evaluators." metadata: disco-role: operating disable-model-invocation: true license: Apache 2.0

VLMEvalKit Benchmark Authoring

Use this sub-skill when a task is about adding or adapting a VLMEvalKit benchmark, building a TSV or video metadata artifact, registering a dataset or video preset, implementing build_prompt / evaluate, or diagnosing dataset-construction failures.

Route first

• Need to run a completed benchmark evaluation, resume predictions, scan API failures, or summarize outputs? Use evaluation.
• Need to add model wrappers, API wrappers, custom model-side prompts, or LiteLLM/OpenAI-compatible model configuration? Use model-development.
• Need to create benchmark data, a dataset class, a video preset, an evaluator, or a converter? Stay here.

Operating workflow

1. Choose the authoring path.
   • One-off local TSV: put <DatasetName>.tsv under LMUData or use a --data-config JSON data entry.
   • Reusable in-package benchmark: implement or reuse an ImageBaseDataset, TextBaseDataset, VideoBaseDataset, or task-specific subclass such as ImageMCQDataset.
   • Video benchmark: add a preset in supported_video_datasets with exactly one of nframe or fps.
   • Conversion task: build a TSV/image tree first; do not start full model inference from this sub-skill.
2. Author and validate the data artifact. Use data formats for TSV columns, multimodal fields, cache roots, and local validation probes.
3. Implement the dataset contract. Use dataset API reference for load_data, prepare_tsv, dump_image, build_prompt, evaluate, registries, custom fallback behavior, and MCQ evaluation helpers.
4. Write or adapt converters safely. Use converter patterns and the bundled LongDocURL TSV helper as the small, runnable pattern. Treat large downloads, archives, and Gradio browsers as optional/reference-only workflows.
5. Dry-check before handoff. Prefer import/build/schema checks (build_dataset, column probes, tiny converter fixtures). Hand off any full evaluation command to evaluation.
6. Troubleshoot from symptoms. Use troubleshooting for missing columns, broken image paths, unsupported-dataset fallback surprises, video frame-configuration errors, cache/download issues, and evaluator output problems.

Validation boundary

The production evidence for this skill covered source inspection and lightweight runtime checks around package imports, build_dataset, dataset registries, CLI help/listing, native LiteLLM/API pipeline tests, and a CUDA smoke probe. It did not verify live API calls, dataset downloads, Gradio services, or large model evaluations. Keep those as user-authorized optional steps.