---

name: benchmark-evaluation description: "Run a bounded ChestAgentBench or Eurorad multimodal evaluation through an OpenAI-compatible chat client, with local or URL-backed images, manifest checks, JSONL logging, interruption handling, and cautious result interpretation." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Benchmark evaluation

Use this skill to evaluate a vision-capable chat model on an already prepared, local ChestAgentBench-style manifest or Eurorad-derived case set. This is an evaluation runner, not a dataset builder and not a clinical decision tool. Route dataset scraping, question generation, and exploratory experiments to their sibling skills. Route MedRAX agent/tool initialization to the agent/tool runtime skill.

Safety and scope gate

• Start with a bounded plan: select the manifest, image mode, model, output prefix, and a small --max-cases value. Treat one case as the smoke run and expand only after its log and image resolution are sound.
• Network access, credentials, API spend, provider-side inference, and URL image downloads are deferred until the user explicitly authorizes the run. A manifest check never needs an API key or network access.
• ChestAgentBench/Eurorad accuracy is a benchmark observation, not clinical validation, patient-specific advice, or evidence of deployment safety.
• Do not place API keys, bearer headers, inline base64 image payloads, or unredacted prompts in a report or shared log. Preserve only the redacted JSONL contract described in data-formats.md.

Quickstart route

1. Confirm that the supplied benchmark runner exposes the flags documented in cli-reference.md. Do not silently substitute a dataset-generation command.
2. Validate a local JSONL manifest and image root before any client creation:

   python scripts/validate_case_manifest.py cases/metadata.jsonl --root cases
   

   Nested image arrays are supported. Missing local images are errors; a case with no image references is reported as a skip candidate. Use --help to inspect all bounded-check options.
3. For a local-image smoke run, after authorization and local data readiness:

   python quickstart.py --model "$MODEL" --temperature 0.2 \
     --log-prefix smoke-local --max-cases 1
   

   The runner loads a JSONL benchmark split, creates an OpenAI-compatible client, embeds local images as data URLs, and logs one JSON object per case.
4. For URL-backed images, plan exactly one case first:

   python quickstart.py --model "$MODEL" --temperature 0.2 \
     --log-prefix smoke-url --max-cases 1 --use-urls
   

   URL mode still fetches and base64-embeds the images before the API call; it is therefore both network work and inference work. Stop if the URL fetch is not explicitly authorized or cannot be bounded.
5. On success, inspect the redacted log for question_id, status, answer, duration, usage, and image-source counts before increasing the case limit. Never infer benchmark quality from a single case.

Prompt and answer contract

The multimodal request contains a medical-case preamble, the case's multiple choice question, and an instruction to answer only from the supplied images and case information. The system role asks for only one choice letter among A/B/C/D/E/F. The expected answer is the manifest's answer field. Preserve the model's raw answer for audit, but normalize only a separate analysis field; do not rewrite the source answer or silently score prose as a letter.

A valid case should have a stable question_id, non-empty question, an expected answer, and images for local mode or image_source_urls for URL mode. Image fields may be a string, a list, or nested lists. Flatten them in order and preserve that order in the log. Empty or missing images do not call the model: record status: "skipped" and reason: "no_images".

Configuration and bounded execution

• --model is the effective request model and takes precedence over the runner's default. OPENAI_MODEL is a useful provider/configuration convention, but do not assume the runner reads it automatically; pass its value explicitly when needed.
• OPENAI_API_KEY is required by the reference quickstart. OPENAI_BASE_URL, when set, is passed to the OpenAI-compatible client. Use a provider's exact compatible endpoint and model name; a custom base URL does not make an incompatible model multimodal.
• Keep temperature low and record it. --max-cases limits the selected prefix of the dataset; it is not a random sample and is not a confidence interval.
• Use a writable, non-sensitive log prefix. The timestamped output filename is derived from the prefix and current time; check that it exists and is non-empty after the run.
• The reference request retries transient-looking failures up to three times with exponential waits (approximately 4–10 seconds between attempts). A final exception is logged and raised, so persistent provider errors can stop the loop rather than being silently counted as wrong answers.
• SIGINT/SIGTERM sets a shutdown event checked between cases. Expect the current request to finish or fail; interruption is not a promise to cancel a provider request. Keep the log as a partial run and record processed/skipped counts.

Interpretation and handoff

Separate these quantities: attempted cases, successful model responses, no-image skips, request errors, valid-choice answers, and exact answer matches. Report the manifest version/source description, image mode, model/base-url class (not secrets), temperature, case limit, interruption state, and log completeness. Do not call skipped cases incorrect unless the evaluation plan explicitly defines that policy. A benchmark score is sensitive to image availability, prompt wording, model/version, provider behavior, and case ordering; compare runs only when those controls are held constant.

For detailed schemas, workflows, flags, and predictable failures, read:

• workflows.md
• data-formats.md
• cli-reference.md
• troubleshooting.md