---

name: run-evaluate description: "Operate Harbor jobs and trials: resolve tasks, select agents/models/environments, configure execution, run locally or on hosted infrastructure, preserve artifacts and trajectories, and resume or hand off sessions." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Run and evaluate Harbor workloads

Use this skill when the user wants to execute an existing Harbor task or dataset, evaluate one or more agents/models, run a single trial, or recover an interrupted execution. The task definition, dataset manifest, adapter, and verifier must already exist or be resolvable. This skill operates the evaluation; it does not author benchmark inputs and it does not perform result analysis or framework extension work.

Routing gate

Choose the smallest workflow before touching a command:

• Create or repair a task, dataset, adapter, verifier, RewardKit check, or registry package: route to author-benchmarks.
• Compile arbitrary files into tasks or run map/reduce: route to exec-map-reduce, not ordinary run flags.
• Inspect, compare, view, export, upload, publish, or explain completed results/trajectories: route to analyze-publish.
• Implement or register a custom agent/environment/plugin/bridge/MCP integration: route to integrations. Passing an already-implemented custom import path to a run is still covered here.

When a request mixes these surfaces, finish the smallest safe prerequisite first and hand off the resulting path or config; do not duplicate the sibling workflow. A task-side MCP server or skill directory belongs to the benchmark configuration, while run-time --mcp-config and --skill injection is execution configuration.

First checks and safety gates

1. Confirm the installed command surface without assuming that the inspected source snapshot and installed distribution are identical:

   harbor --version
   harbor run --help
   harbor trial start --help
   

   If harbor is not on PATH, ask how it was installed rather than inventing an invocation. Older installs may omit newer commands or expose older trajectory semantics; prefer the live help and config-model validation.

2. Identify exactly one input mode for a job: --path for a local task or dataset, --task for one registry/package task, or --dataset for a dataset. Do not combine task and dataset flags. For a trial, use --path or a complete --config.

3. Before a costly or credentialed run, show the resolved agent, model, environment, task count, attempts, concurrency, timeout policy, output directory, network policy, and artifact plan. Use --print-config for a job config; it validates and prints without starting a job. A --launch dry run validates a hosted launch without queuing it.

4. Treat model keys, provider credentials, registry access, Docker/cloud daemons, network access, private images, and hosted quota as explicit gates. Never claim that a provider or model was executed merely because it appears in a factory enum or help panel. Do not expose secret values in YAML, logs, or a handoff.

5. Prefer a tiny local/CPU or install-only preflight when the user wants a compatibility check. --install-only performs agent setup/install, skips the agent run and verification, and is not a score-producing evaluation.

Choose the execution shape

One job: run / job start

Use a job for a task or dataset × agent/model × attempt matrix. The canonical aliases are equivalent:

harbor run -p ./tasks/hello -a claude-code -m <model>
harbor run -d <org/dataset>@<ref> -a <agent> -m <model> -n 8
harbor job start --config ./job.yaml --print-config

harbor run is the public shortcut for harbor job start; a config file must conform to JobConfig for a local run and to the installed hosted config model when --launch is used. CLI flags are applied after a loaded config, so a flagged value overrides the corresponding config value. -m/--model is repeatable for jobs; each configured model becomes an agent configuration.

A job resolves datasets and task packages before creating trials. In the ordinary (non-regrade) case it creates one TrialConfig for every task, agent configuration, and attempt (n_attempts), then submits the remaining trials through a bounded queue. Job results are written under jobs_dir/job_name and trial directories are children of that job directory. The job and each trial persist config.json, a resolved lock.json, logs, results, agent/verifier outputs, and artifacts as applicable.

Useful job flags are grouped in the references:

• matrix and output: --job-name, --jobs-dir, -k/--n-attempts, -n/--n-concurrent, --n-concurrent-agents, --quiet, --debug;
• input: -p/--path, -t/--task, -d/--dataset, --repo, registry options, include/exclude task names, and --n-tasks;
• execution: -a/--agent, repeatable -m/--model, -e/--env, config files, timeout multipliers, resource overrides, --artifact, extra instructions, and verification controls;
• hosted/share gates: --launch, --dry-run, --credential-mode, secret grants, --upload, visibility, organization/user sharing.

Do not use --launch and --upload together. --launch moves execution to Harbor-managed infrastructure; --upload runs locally and uploads the result afterward. --n-concurrent-agents is a local-only per-agent cap and is rejected for a hosted launch.

One trial: trial start

Use a trial when isolating one task/agent/model configuration or debugging a single rollout:

harbor trial start -p ./tasks/hello -a <agent> -m <model> \
  --trials-dir ./trials --trial-name smoke
harbor trial start --config ./trial.yaml

A trial requires --path or a YAML/JSON config. Its TrialConfig.task is a TaskConfig identifying a local path, Git task (path plus git_url and optional commit), or package task (name: org/name plus optional ref). A trial's generated name is task-based plus a short unique suffix when no name is supplied. Trial CLI flags modify the loaded model, including explicit seconds for agent/verifier/setup timeouts and run-time trajectory loading.

Trial.create() resolves the task, selects MultiStepTrial when the task has [[steps]], otherwise SingleStepTrial, initializes the environment and agent, then run() performs setup, agent execution, artifact recovery, verification, final persistence, and teardown. A failed trial normally records exception_info and returns a result; a cancellation is recorded and then re-raised so the job can preserve cancellation state.

Job/trial management commands

• harbor job resume -p <job-dir> continues a persisted local job; it does not restart completed trials.
• harbor job download <job-uuid> and harbor trial download <trial-uuid> materialize hosted archives locally and require platform authentication. --overwrite is an explicit replacement of an existing target.
• harbor job share <job-uuid> --org <name> or --user <name> mutates sharing on an already-uploaded job; require confirmation and Hub auth. General publishing, comparisons, and result browsing route to analyze-publish.
• job summarize and trial summarize are removed command shims in the current source surface; use analyze-publish for summaries rather than teaching a future agent a command that exits with an error.
• trial handoff, job regrade, and trial regrade are covered below and in trajectory-lifecycle.md.

Job and trial lifecycle commands

Use the singular command groups exposed by the installed CLI:

• harbor job start is the config-oriented form of harbor run; use --print-config to validate without execution.
• harbor job resume --job-path JOB_DIR continues an interrupted job from its persisted config/lock. It is not an agent-session resume; see the trajectory reference.
• harbor job download JOB_UUID --output-dir DIR and harbor trial download TRIAL_UUID --output-dir DIR materialize credentialed Hub records locally. They do not execute an agent. Treat the downloaded directory as read-only until a deliberate derived operation is chosen.
• harbor job share JOB_UUID --org ORG or --user USER changes sharing on an already-uploaded job. It requires authentication and explicit sharing confirmation; never run it as part of a local evaluation.
• harbor job regrade SOURCE ... and harbor trial regrade SOURCE ... create new verifier-only derived outputs. They never rerun the agent and require the single-step/separate-verifier/artifact-manifest conditions described in trajectory-lifecycle.md.
• harbor job summarize and harbor trial summarize are removed compatibility shims in current Harbor. Use harbor analyze for completed-result inspection instead of relying on an old summarize command.
• harbor trial start runs one trial; there is no generic trial resume. Use --load-trajectory for a new seeded trial or trial handoff for a supported local CLI continuation.

Credentialed download/share operations and result inspection are normally handed to analyze-publish after the execution choice is made. This skill still records their execution gates so a run plan does not accidentally relaunch or mutate a completed result.

Configure, run, and hand off

Read the focused references progressively:

1. job-and-trial-config.md for schemas, config layering, task expansion, retries, and resume.
2. agents-and-environments.md for factory selection, providers, agent options, and preflight.
3. resources-network-artifacts.md for resource policies, timeouts, network phases, artifacts, and logs.
4. trajectory-lifecycle.md for multi-step sessions, loading, handoff, downloads, and regrade boundaries.
5. troubleshooting.md for symptom-driven recovery before relaunching anything.

For multi-step execution, make the session policy explicit. By default each step starts a fresh conversation. resume_trajectory: true changes the sequence to (fresh, resume, resume, ...), but only for agents that advertise native resume support; a failure is intended to happen before environment spend. load_trajectory seeds the first step, and can be combined with resume. A native session is lossless but agent-specific; an ATIF JSON trajectory is portable but converted and therefore not lossless. Only claim handoff when the installed agent supports it and the trial has one session.

Use harbor job resume --job-path <job-dir> for an interrupted job. It reads that job's config.json, reconciles completed trial configs, and continues the remaining work. It rejects a changed config or mismatched lock and, by default, removes trials whose recorded error type is CancelledError; pass repeated --filter-error-type options deliberately if another cleanup policy is wanted. A completed job is not restarted by accident. There is no generic trial resume command: use trajectory loading, trial handoff, or a new trial config.

Use regrade only when the intent is to recompute verification without rerunning the agent. harbor trial regrade and harbor job regrade never modify the source and require a completed single-step source with readable artifacts and an artifact manifest; the replacement verifier must run in separate mode. Multi-step regrade is unsupported. Route score inspection, comparisons, and read-only result browsing to analyze-publish after the execution decision.

Stop conditions

Stop before creating a job when the task/dataset cannot be resolved, the resolved config is invalid, a required custom import cannot load, the selected provider's preflight fails, a required model credential is absent, a required backend is unsupported, or a network/resource policy cannot be enforced. Do not paper over these with --disable-verification, a larger timeout, or a provider fallback unless the user explicitly approves the changed experiment.

--disable-verification is an intentional no-score mode, not a fix for a bad verifier. If the task itself, its tests, its registry metadata, or an adapter must change, hand off to author-benchmarks.