---

name: data-preparation description: "Preprocess nuPlan scenarios into Diffusion Planner .npz records and a JSON filename manifest, and diagnose path, schema, feature-cap, and normalization failures. Use for nuPlan preprocessing, Diffusion Planner training data, scenario extraction, lane/route/agent feature preparation, or data schema debugging." disable-model-invocation: true metadata: disco-role: operating license: NO_LICENSE

Diffusion Planner data preparation

Use this sub-skill when the Researcher needs to turn nuPlan scenarios into the fixed-size records consumed by Diffusion Planner training. It covers raw nuPlan data and map prerequisites, scenario filtering, ego/agent/static/map feature extraction, .npz output, the JSON filename manifest, and normalization compatibility. It does not cover distributed optimization, checkpoint loading, closed-loop simulation, or guidance authoring.

Fast route

1. Confirm that the active Python environment can import both the project package and the nuPlan devkit. A missing devkit is a hard stop; do not try to fabricate scenarios or replace the builder with a different data format.
2. Confirm separate readable directories for the nuPlan DB root and map root, and a writable processed-data directory. Use the bundled adapter's explicit --log-names and --manifest-output paths so the manifest is not tied to a working-directory convention.
3. Choose caps before processing. The training-compatible defaults are 32 agents, 5 static objects, 70 lanes × 20 points, and 25 route lanes × 20 points. The 2-second history and 8-second future are sampled at 10 Hz.
4. Run the safe validator in scripts/validate_preprocessed_data.py against a manifest, processed directory, and normalization file before launching a large extraction. It can create and validate a tiny synthetic fixture; it never imports nuPlan or touches raw data.
5. For real extraction, use the bundled path-explicit adapter scripts/run_preprocessing.py with --data-path, --map-path, --save-path, --log-names, --manifest-output, scenario-limit, shuffle, and feature-cap arguments. Start with a very small total limit, inspect one record and its manifest, then scale up. The source shell workflow is evidence only: it contains unsafe placeholders and an expensive million- scenario default.

See workflow and CLI semantics, data formats and normalization, and troubleshooting. Once a valid dataset exists, hand it to model-training; for downstream runtime use, see closed-loop-planning. The graph entry point is diffusion-planner.

Operating contract

Inputs

• A nuPlan DB/data root and the matching map root for the same map version.
• The project preprocessing environment: Python 3.9-compatible dependencies, diffusion_planner 1.0.0, nuPlan-devkit 1.2.2, and compatible NumPy, Shapely, GeoPandas, Rasterio, and PyTorch installations.
• A JSON array of training log names. The standard project manifest contains 13,180 log names; it is an input selector, not a list of processed .npz files.
• A writable output directory and a copy or path to the matching normalization.json.

Outputs

Each successful scenario produces one .npz named from its map name and scenario token. The generated JSON manifest is an array of those .npz filenames. A usable output must have no missing manifest entries, no path traversal or absolute names in the manifest, finite floating-point values, and consistent fixed shapes. Treat an empty output directory or an empty manifest as a failed preprocessing run, even if the process exited successfully.

Safe boundaries

• Never start a full run until --help, path checks, a tiny scenario limit, and one-record schema validation have passed.
• Do not infer that a missing .npz means a valid zero-scenario result; inspect builder/filter logs and the manifest location.
• Do not change feature caps after generating data without regenerating or revalidating every record and the model/training configuration.
• Do not normalize raw global coordinates. The processor stores ego-centric coordinates; the training path converts future headings to cosine/sine and applies the normalizers at training time.
• Full preprocessing is I/O- and map-query-intensive and requires real nuPlan DBs/maps. A synthetic fixture proves parser and schema behavior only; it is not evidence of nuPlan extraction correctness.