---

name: evaluation-and-mapping description: "Run and interpret Gaussian-SLAM trajectory, rendering, Replica mesh, global-map, and ScanNet++ novel-view evaluation from a completed checkpoint; verify artifact contracts, preserve partial-stage evidence, and recover common CUDA/Open3D/data failures." disable-model-invocation: true metadata: disco-role: operating license: MIT

Evaluation and mapping

Use this sub-skill when a Gaussian-SLAM run already has a checkpoint and the Researcher needs metrics, a Replica reconstruction, or a refined global map. It covers run_evaluation.py and the Evaluator lifecycle. It does not train or resume SLAM, download datasets, repair checkpoints, or claim that a partial metric set is a complete evaluation.

Read the focused reference before acting:

• api-reference.md: public entry points, checkpoint schema, stage behavior, and implementation defaults.
• workflows.md: preflight, command, stage, and recovery procedures.
• outputs-and-metrics.md: artifact and metric interpretation, including dataset gates and units.
• troubleshooting.md: failure diagnosis and safe recovery.

Operating contract

Inputs

Require all of the following before an expensive run:

1. A completed or intentionally partial checkpoint directory.
2. config.yaml in that directory, or an explicit compatible config path.
3. estimated_c2w.ckpt and a submaps/ directory containing numbered submap checkpoints. Each submap should contain gaussian_params and submap_keyframes as produced by Gaussian-SLAM.
4. The dataset named by the config and its ground-truth assets at the paths resolved by that config.
5. A CUDA-capable runtime for the Evaluator: rendering, Replica mesh reconstruction, and global-map merging/refinement all use CUDA-bound model and rasterizer/FAISS code. A CPU-only trajectory helper check is useful support validation, but is not a substitute for a full evaluation.

Treat checkpoint files as untrusted inputs. Do not overwrite, prune, rewrite, move, or repair them unless the user explicitly requests a separate operation. The repository evaluator writes results inside the checkpoint directory; make a copy or select a disposable output tree when preserving the source run matters.

Safe preflight

Run the bundled scripts before importing the repository evaluator:

python skills/disco/gaussian-slam/sub-skills/evaluation-and-mapping/scripts/check_cli.py \
  --checkpoint /path/to/checkpoint
python skills/disco/gaussian-slam/sub-skills/evaluation-and-mapping/scripts/validate_checkpoint.py \
  --checkpoint /path/to/checkpoint

These scripts are read-only, do not import PyTorch/Open3D, do not access the network, do not open GUI windows, and do not write a report. They check paths, configuration text, and the file-level schema only; they cannot prove that a serialized tensor can be loaded on CUDA. Use --json for machine-readable output. Fix missing required artifacts before starting run_evaluation.py.

Standard invocation

From the repository root, with the project environment active:

python run_evaluation.py \
  --checkpoint_path /path/to/checkpoint \
  --config_path /path/to/checkpoint/config.yaml

If --config_path is omitted, the wrapper uses <checkpoint_path>/config.yaml. The evaluator has no dry-run flag and no separate output directory option. save_render is false in this wrapper, but JSON, plots, meshes, and NVS images are still written under the checkpoint. Do not run the command merely to test --help or schema validity.

Lifecycle and stage interpretation

Evaluator.__init__ loads the merged config, seeds the run, constructs the configured dataset, fixes device="cuda", reads estimated_c2w.ckpt, and discovers submaps. It then runs these independent stages:

1. Trajectory — compares estimated and ground-truth camera translations, writes raw and Horn-aligned ATE, and plots trajectories.
2. Rendering — loads every *.ckpt submap and renders its keyframes for PSNR, LPIPS, SSIM, and train-view depth L1.
3. Reconstruction — only for dataset_name: replica; integrates rendered RGB-D frames into a mesh and evaluates it against Replica assets.
4. Global map — merges and deduplicates submap points, refines a Gaussian map for 10,000 iterations, saves a PLY, and, only for dataset_name: scannetpp, renders test-split NVS images and reports NVS PSNR to stdout.

Each call in Evaluator.run() is wrapped in its own broad exception handler. A later stage can therefore succeed after an earlier stage fails. Always inspect the output files and logs stage by stage; a process that reaches the end is not proof that all four stages completed. Conversely, constructor failures happen before those per-stage catches and prevent the run entirely.

Required distinctions

• Reconstruction is explicitly skipped for every non-Replica dataset.
• Global-map creation is attempted for all datasets, but NVS evaluation is explicitly supported only for ScanNet++ (dataset_name: scannetpp).
• depth_L1 in the reconstruction result is not the same as rendering depth_l1_train_view. The former needs Replica ground-truth mesh and unseen points plus headless Open3D; the latter is a rendered training-view average.
• Headless Open3D and the evaluate_3d_reconstruction package are required for the full Replica reconstruction path. A CPU trajectory result alone cannot validate those dependencies.
• CUDA is required for rendering/global-map paths. Do not silently downgrade them to CPU and report success.

Procedure

1. Identify the dataset, scene, checkpoint provenance, and whether preserving the source checkpoint is required.
2. Run both read-only preflight scripts and record warnings.
3. Check that the configured input dataset and, for Replica reconstruction, data/Replica-SLAM/cull_replica/<scene>.ply plus <scene>_pc_unseen.npy are present. Do not download missing assets in this sub-skill.
4. Confirm CUDA, the differential Gaussian rasterizer, FAISS GPU, Open3D, and evaluate_3d_reconstruction in the intended environment. Use the environment contract in workflows.md.
5. Run the standard command once. Do not concurrently evaluate the same checkpoint because stages write shared files.
6. Record which stage files exist, which stages printed exceptions, and which metrics are absent or null. Preserve traceback text for recovery.
7. Interpret values using outputs-and-metrics.md, retaining dataset gates and implementation caveats.
8. If a stage fails, apply the smallest non-mutating recovery in troubleshooting.md. Re-run against a copy if the original output must remain reproducible.
9. Report complete, partial, skipped, and unverified stages separately.

Safe scripting rules

Bundled scripts must remain read-only by default. They must not run the expensive evaluator, import heavy CUDA/Open3D modules, open Open3D windows, download weights or datasets, or mutate user checkpoints. Keep review logs and verification cases outside this runtime skill directory.