---

name: evaluation description: "Validate completed Visual Pushing and Grasping sessions and produce source-compatible metrics or headless training curves from transition logs." disable-model-invocation: true metadata: disco-role: operating license: BSD 2-Clause

Evaluation

Use this route after a VPG test or training session has produced a session folder. It owns offline session-log validation, the completion/grasp/action- efficiency measurements, and the training-curve plot. It does not train a model, start V-REP/CoppeliaSim, move a UR5, or capture camera data.

Choose the operation

Let <skill-root> mean the directory containing the root SKILL.md.

• Report a completed test session: run the bundled helper <skill-root>/sub-skills/evaluation/scripts/evaluate_session.py. It accepts the source-compatible flags --session_directory, --method, and --num_obj_complete; add --format json for machine-readable output or --output PATH to save the text/JSON report.
• Plot one or more training sessions: run the bundled helper <skill-root>/sub-skills/evaluation/scripts/plot_session.py with one or more positional session directories and --output PATH. It uses Matplotlib's Agg backend and never requires a display. By default it uses the source interval of 200 steps and x-axis maximum of 2,500 steps; override them with --interval-size and --max-plot-iteration.

Read references/metrics-and-logs.md before interpreting a result, and use references/troubleshooting.md when a log layout, method, or plotting error is reported. The two helpers are bundled standalone artifacts adapted from the historical evaluation and plotting programs; they are not a claim that the original full training loop is modern-runtime compatible.

Required session contract

SESSION is the root directory created for one logging run. It must contain:

SESSION/
  transitions/
    executed-action.log.txt
    reward-value.log.txt
    clearance.log.txt       # required by evaluate_session.py
  models/                   # needed for plot auto method detection

logger.py writes the transition files as whitespace-separated text. Each executed-action row has at least four values; column 0 is the primitive ID: 0 for push and 1 for grasp. Reward rows are scalar values aligned to the same action history. A clearance value is an integer action/iteration index at which the workspace is reset for the next trial. In the documented test interpretation, completion is expected before more than 10 consecutive failed/no-change attempts trigger that reset. The evaluator prepends a zero boundary, so trial i is the half-open range [clearance[i-1], clearance[i]).

The helper validates finite values, aligned action/reward lengths, legal primitive IDs, monotonic in-range clearance boundaries, and nonempty trials. Failures name the offending file or boundary and explain the expected layout. Paths are resolved from the caller's current directory, so both helpers run from an arbitrary working directory and do not import repository modules.

Method and threshold

Pass --method reactive for the supervised/classification policy or --method reinforcement for the Q-learning policy. The same grasp action has different success evidence:

• reactive: a grasp is successful when its reward/class value equals 0;
• reinforcement: a grasp is successful when its reward value is at least 0.5 (the source evaluator's threshold).

Set --num_obj_complete N to the positive number of successful grasps needed to count a trial as completed. Do not infer this number from the number of objects in a preset. A trial is valid for the three aggregate metrics only when its counted grasp successes are at least N; pushing contributes actions but never success count.

Report interpretation

The evaluator preserves the source formulas and prints the three README metrics (all percentages, higher is better):

1. average completion/clearance rate over all trials;
2. average grasp success rate, restricted to completed trials;
3. average action efficiency, 100 * mean(N / actions_in_completed_trial).

For source compatibility it also reports the diagnostic grasp-to-push ratio, 100 * mean(grasp_attempts / all_actions) over completed trials. If no trial completes, the restricted metrics are reported as undefined/null with a warning rather than emitting an unhelpful divide-by-zero or NaN. This is a valid negative result, not evidence that the logs are malformed.

Plot interpretation

plot_session.py SESSION [SESSION ...] --output PATH draws a solid grasp success curve and a dashed push-then-grasp success curve for each session. The curves use the preceding interval_size action attempts and preserve the source's warm-up scaling for steps before 200. The plot uses at most min(number_of_action_rows - 2, max_plot_iteration) rows, matching the source program's exclusion of two trailing action rows whose rewards may not yet be available. A model filename containing reactive or reinforcement selects the method; pass --method to override detection when no model snapshot is available. The output is a PNG by default (or the format implied by the output suffix) and is written without plt.show().

Boundaries and provenance

This operating route was distilled from source commit 580e2334beec0d83b49e6ca89d7542b79d1d4350, especially the historical evaluate.py, plot.py, logger.py, README evaluation instructions, and main.py clearance logging. Bounded checks with a current Python numerical stack can validate the bundled offline helpers, but do not establish a modern full-loop training, simulator, or robot run. See references/metrics-and-logs.md for formulas and references/troubleshooting.md for recovery steps.