---

name: evaluation description: "Operate Qwen-VL official benchmark evaluation, scoring, and data utilities." metadata: disco-role: operating disable-model-invocation: true license: NOASSERTION

Qwen-VL Evaluation Sub-skill

Use this sub-skill when the task is to prepare or run the official Qwen-VL benchmark evaluations, convert benchmark data, score generated answers, or format benchmark submissions.

Route here for

• Captioning benchmarks: Flickr30K and Nocaps.
• VQA and document/chart/text QA: VQAv2, OKVQA, TextVQA, VizWiz, DocVQA, ChartQA, GQA, OCRVQA, and AI2D.
• Referring-expression grounding: RefCOCO, RefCOCO+, and RefCOCOg.
• Multiple-choice benchmarks: ScienceQA, MMBench, and SEED-Bench.
• Reference-only benchmark workflows: MME and TouchStone.
• CPU-only data conversion or scoring utilities bundled under scripts/.

Route elsewhere

• For direct ad-hoc multimodal chat, image grounding prompts, model loading, checkpoint choice, or quantization behavior, use ../inference/SKILL.md.
• For evaluating a newly trained adapter after a finetuning run, use this sub-skill for benchmark execution but use ../finetuning/SKILL.md for training and adapter preparation.
• For web demos or OpenAI-compatible service deployment, use ../serving/SKILL.md.

First actions

1. Identify the benchmark name and split, then open references/benchmark-reference.md for the owner script, dataset key, metric, output file, and submission route.
2. Validate the local data/ layout against references/data-layouts.md. Do not assume external datasets, annotations, or model checkpoints are already present.
3. If the workflow runs model inference, confirm CUDA/NCCL availability and launch with the distributed arguments in references/benchmark-reference.md. These official inference scripts use device_map='cuda', .cuda(), torch.distributed, and NCCL.
4. If the task is only conversion or scoring, prefer the CPU-capable bundled utilities in scripts/mmbench/ or scripts/infographicsvqa_eval.py when their input files are already available.
5. When model downloads, benchmark downloads, API-judged scoring, or full multi-GPU runs are not approved, report the exact skipped prerequisite and prepare commands/files without executing the expensive step.

Bundled entry points

• Core Qwen-VL evaluation scripts: scripts/evaluate_caption.py, scripts/evaluate_vqa.py, scripts/evaluate_grounding.py, and scripts/evaluate_multiple_choice.py.
• VQA and ANLS scoring helpers: scripts/vqa.py, scripts/vqa_eval.py, and scripts/infographicsvqa_eval.py.
• MMBench conversion, scoring, and submission helpers: scripts/mmbench/.
• SEED-Bench prompt conversion and multiple-choice inference: scripts/seed_bench/trans.py and scripts/seed_bench/eval.py.

Safety and limitations

• Do not run a full benchmark unless the user has provided a checkpoint, datasets, model-download approval or cached weights, enough GPU memory, and output destination expectations.
• Do not use MME get_images.py from the source repository as-is; this sub-skill documents MME as reference-only because the original helper mutates directories and assumes external hard-coded dataset paths.
• SEED-Bench image conversion is CPU-friendly; video conversion additionally needs video decoding packages and local video assets. See references/troubleshooting.md.
• TouchStone requires an external judge workflow and is documented as methodology/reference-only here.