---

name: visualization-benchmarking description: "This skill guides an agent through MapTR prediction visualization, qualitative video assembly, inference timing, and log interpretation without overstating unavailable evidence." disable-model-invocation: true metadata: disco-role: operating license: MIT

Visualization and benchmarking

Use this skill when a MapTR config and checkpoint should produce inspectable prediction artifacts, when existing artifacts should be assembled into a video, or when timing/log outputs need a cautious interpretation. Read the linked references before running a command:

• visualization procedure and artifact contract
• benchmark and log-analysis contract
• symptoms, recovery, and stop conditions

This skill routes data conversion, dataset layout, and annotation generation to data-preparation. It routes model/config changes to model-configuration and training or distributed evaluation launches to training-evaluation. It does not train a model, download data or checkpoints, build the legacy CUDA extension, or claim native execution when those prerequisites are absent.

Fast routing

1. Need rendered predictions? First verify a compatible config, checkpoint, prepared test annotations, six camera image paths per sample, and the documented legacy runtime. Then use tools/maptr/vis_pred.py.
2. Need a video from rendered samples? Inspect the sample directories and required filenames. Prefer the safe local helper linked below; it sorts sample directories, validates images, rejects unsafe output names, and does not access the network.
3. Need FPS? Record GPU, batch size, number of camera views, precision, config, checkpoint, sample count, warm-up policy, software/extension state, and whether data loading is included. Compare only matched conditions.
4. Need curves or iteration timing? Use analyze_logs.py only with line-delimited JSON logs and state whether the first iteration of each epoch was excluded.

Prediction visualization

Run commands from the project root and use an explicit threshold when a result must be reproducible. The documented example is:

PYTHONPATH=. python tools/maptr/vis_pred.py \
  projects/configs/maptr/maptr_tiny_r50_24e.py path/to/checkpoint.pth \
  --score-thresh 0.3 --show-cam \
  --show-dir work_dirs/maptr_tiny/vis_pred \
  --gt-format fixed_num_pts

The positional arguments are config and checkpoint. Useful options are --score-thresh FLOAT, --show-cam, --show-dir DIR, and one or more values for --gt-format. In the checked source, the documentation says the default threshold is 0.3 while the parser currently defaults to 0.4; pass the value explicitly. The --show-cam flag is accepted, but the current implementation copies camera images and builds surroud_view.jpg regardless of that flag.

With no --show-dir, the native default is work_dirs/<config-stem>/vis_pred. Each sample is a child directory named from the lidar filename. Typical outputs include CAM_*.jpg, the misspelled surroud_view.jpg, PRED_MAP_plot.png, and a GT map image. The native script also dumps a copy of the config into the visualization directory.

Choose GT representation deliberately:

• fixed_num_pts: resampled points used by the standard saved artifact GT_fixednum_pts_MAP.png; this is the safest choice for the native video layout.
• polyline_pts: original Shapely coordinates, saved as GT_polyline_pts_MAP.png; useful for inspecting annotation detail.
• se_pts: start/end point arrows. The parser accepts it, but this source revision does not save a separate se_pts image.
• bbox: an enclosing rectangle. The parser accepts it, but this source revision does not save a separate bbox image.

A visualization run is not evidence of model quality by itself. Preserve the config, checkpoint identity, threshold, GT format, dataset split, and skipped sample count beside any review. If the dataset, checkpoint, or required extension is unavailable, stop after prerequisite reporting rather than inventing images or metrics.

Safe video assembly

The native command is:

python tools/maptr/generate_video.py path/to/vis_pred \
  --fps 10 --video-name demo --sample-name SAMPLE_VIS.jpg

It writes <parent-of-vis_pred>/demo.mp4, assumes six camera files plus PRED_MAP_plot.png and GT_fixednum_pts_MAP.png in every sample directory, and uses the mp4v codec at 1680x450. Its directory iteration is not sorted and it can fail obscurely on missing or undecodable images. Use the bundled, source-adapted helper for a safe fixture or an existing artifact set:

python <skill-root>/sub-skills/visualization-benchmarking/scripts/make_video.py \
  path/to/vis_pred --fps 10 --video-name demo --sample-name SAMPLE_VIS.jpg
python <skill-root>/sub-skills/visualization-benchmarking/scripts/make_video.py \
  --self-check

The helper ignores non-directory entries, sorts frame directory names, reports and skips incomplete samples, generates the requested sample image (so a pre-existing SAMPLE_VIS.jpg is not required), writes a fixed-size MP4, and fails if no complete sample can be decoded. It refuses path traversal in --video-name or --sample-name; add --overwrite to replace an existing video. A successful fixture check must report two ordered frames and a video path. Codec availability is still platform-dependent; a successful write does not prove that every media player can decode the file.

Timing and logs

The native benchmark contract and interpretation rules are in benchmarking.md. In short, tools/maptr/benchmark.py requires a config, optionally loads a checkpoint, uses one GPU and batch size one, synchronizes CUDA around model inference, warms up five iterations, and prints periodic and overall img / s. It is not a video frame rate and does not establish end-to-end latency. The sibling analysis benchmark has the same core timing assumptions but no plugin import step.

Do not compare an FPS value across different GPU models, batch sizes, view counts, precision modes, configs, checkpoints, warm-up/sample policies, or extension builds. The README's published reference is specifically batch one with six views on an RTX3090; it is context, not a result for another setup. When logs or hardware metadata are missing, label the comparison blocked.

Difficult synthetic cases

• Mixed visualization names: create frame-b/ and frame-a/ with the required images, add a non-directory and an incomplete child, and omit SAMPLE_VIS.jpg. Assert that the helper reports the incomplete child, writes the requested sample image, and reports ORDER frame-a,frame-b. Recovery is to repair or intentionally exclude the skipped child; never infer order from native os.listdir output.
• Incompatible benchmark request: if two proposed FPS values have different GPU or batch size and the logs/manifests are missing, assert status: blocked and produce no numeric comparison. Recovery is to provide matched manifests/logs or rerun both cases under one fixed protocol.

For JSON training logs:

python tools/analysis_tools/analyze_logs.py plot_curve run_a.json run_b.json \
  --keys mAP_0.25 --legend run-a run-b --out curves.png
python tools/analysis_tools/analyze_logs.py cal_train_time run.json

plot_curve supports --mode train|eval, --interval, --backend, --style, and --out. cal_train_time excludes each epoch's first timing value unless --include-outliers is supplied. Check that every log is line-delimited JSON, ends in .json, contains epoch fields, and has the requested metric before interpreting a plot. Missing logs, incompatible metrics, or an unavailable display backend are stop conditions, not reasons to fill in a fabricated curve.

Completion gate

A safe handoff states the exact commands, artifact directory, threshold and GT format, frame count/order, video codec/size/fps, benchmark conditions, log files/metrics, and every skipped or unverified item. It must explicitly say when model visualization or benchmark execution was deferred because data, checkpoint, CUDA compatibility, or logs were not available.