---

name: benchmarks-and-evaluation description: "Prepare and troubleshoot MobileAgent GUI benchmark and evaluation workflows: AndroidWorld, OSWorld, WebArena/WebVoyager/VisualWebArena, grounding, GUI knowledge, and GUI-Critic-R1." metadata: disco-role: operating disable-model-invocation: true license: MIT

Benchmarks and Evaluation

Use this sub-skill when a user asks about evaluation, benchmark task suites, trajectories, score/judge models, GUI-Critic-R1 datasets, AndroidWorld, OSWorld, WebArena, WebVoyager, VisualWebArena, grounding benchmarks, or GUI knowledge benchmarks.

This route builds and validates evaluation commands/data safely. Live benchmark execution still requires emulators, VMs, browsers, datasets, checkpoints, APIs, and explicit user authorization.

Route map

Prompt signal	Workflow	Read / run
AndroidWorld, MiniWoB, emulator ports, suite_family, run_ma35.py, run_ma3.py	AndroidWorld/MiniWoB	references/androidworld-osworld.md, scripts/build_androidworld_command.py
OSWorld, VM path, domains, parallel environments, run_multienv_*	OSWorld	references/androidworld-osworld.md, scripts/build_osworld_command.py
WebArena, WebVoyager, VisualWebArena, web benchmark task id, judge, main_for_eval.py	Web benchmarks	references/web-benchmarks.md, scripts/build_web_benchmark_command.py
GUI-Critic-R1 JSONL, score tags, critic labels, hard-coded key risk	GUI-Critic-R1	references/grounding-and-gui-critic.md, scripts/validate_gui_critic_dataset.py
Grounding benchmark, GUI knowledge benchmark, model_path, ds_path, eval_benchmark_type	Local VLM checkpoint evaluation	references/grounding-and-gui-critic.md, scripts/build_grounding_eval_command.py
Emulators, browsers, checkpoints, APIs, score interpretation, skip policy	Troubleshooting	references/troubleshooting.md

Safe usage pattern

1. Identify the benchmark family and required backend/service.
2. Build a command with the relevant script. Example AndroidWorld:

python sub-skills/benchmarks-and-evaluation/scripts/build_androidworld_command.py \
  --version v35 \
  --model-env GUI_OWL_MODEL \
  --api-key-env GUI_OWL_API_KEY \
  --base-url-env GUI_OWL_BASE_URL \
  --grpc-port 8554 \
  --console-port 5554 \
  --tasks ContactsAddContact,ClockStopWatchRunning \
  --n-task-combinations 2 \
  --fixed-task-seed \
  --traj-output-path runs/androidworld-traj

3. For data/schema tasks, validate locally before any inference. Example GUI-Critic:

python sub-skills/benchmarks-and-evaluation/scripts/validate_gui_critic_dataset.py --jsonl sample.jsonl

4. Report live prerequisites separately from safe checks. Do not claim scores from generated templates.

Boundaries

• Ordinary current v3.5 mobile/desktop/browser task execution belongs to ../current-gui-owl/SKILL.md.
• Mobile-Agent-E task evolution belongs to ../mobile-agent-e/SKILL.md.
• UI-S1 training/eval JSONL and checkpoint merge belongs to ../ui-s1-training/SKILL.md.
• Generic LLM benchmark harnesses unrelated to GUI agents are outside this skill.

Verification stance

Command builders and validators are safe CPU checks. Live benchmark results require the real backend and should be recorded as live evidence only when actually run.