---

name: data-preparation description: "This skill prepares and preflights MapTR nuScenes and Argoverse2 data layouts, annotation contracts, and vector-map schemas before model execution." disable-model-invocation: true metadata: disco-role: operating license: MIT

MapTR data preparation

Use this route before training, evaluation, or visualization. It turns a raw nuScenes or Argoverse2 download into a checked layout plan and tells the next agent which generated files and vector schema MapTR expects. It does not download data, run a full converter, install packages, or modify a user data root.

Fast path

1. Choose the dataset and make an explicit absolute data root.

2. Run the bundled CPU-only layout checker. It checks directories and file names; it does not import MapTR, mmcv, nuscenes, av2, or torch.

   python <skill-root>/scripts/check_dataset_layout.py \
     --dataset nuscenes --root /data/maptr/nuscenes --canbus-root /data/maptr \
     --check-annotations
   python <skill-root>/scripts/check_dataset_layout.py \
     --dataset av2 --root /data/maptr/argoverse2/sensor \
     --check-annotations
   

3. Resolve every ERROR before starting a converter. Use --json when a machine-readable report is needed. --self-test exercises valid and deliberately broken tiny fixtures without accessing external data.

4. Select a converter only after the raw layout passes. Record the exact command, version, output root, and worker count; preserve converter logs.

5. Re-run with --check-annotations after conversion. A passing layout check is not proof that images, calibration, transforms, or map geometry are numerically correct.

The checker returns exit code 0 only when all requested checks pass. It returns 1 for a missing required path, wrong placement, malformed map archive, or missing requested annotation file. It never creates a directory in the data root.

Installation and safety gates

The documented dependency target is Python 3.8, PyTorch 1.9.1+cu111, mmcv-full==1.4.0, mmdet==2.14.0, mmsegmentation==0.14.1, timm, mmdetection3d==0.17.2, shapely==1.8.5.post1, and the av2 API for AV2. The project also documents installing its mmdetection3d component and the Geometric Kernel Attention extension. Treat these versions as a compatibility plan, not as a verified installation on the current machine. A legacy CUDA compiler, matching mmcv-full, and a successful custom extension build are not assumed here.

Install prerequisites in an isolated environment and prove imports before conversion. Do not start with a large converter just to discover an import or ABI mismatch. See the workflow reference for the ordered probes and documented commands. Route model and extension choices to model-configuration, and route execution to training-evaluation.

Dataset choices and converter contracts

nuScenes

The documented raw root is data/nuscenes/, with maps/, samples/, sweeps/, and version metadata such as v1.0-trainval/ and v1.0-test/. The CAN bus expansion belongs beside the dataset root: with --root-path /data/maptr/nuscenes --canbus /data/maptr, the converter looks for /data/maptr/can_bus/, not /data/maptr/nuscenes/can_bus/.

Reference command for the full release:

python tools/create_data.py nuscenes \
  --root-path /data/maptr/nuscenes --out-dir /data/maptr/nuscenes \
  --extra-tag nuscenes --version v1.0 --canbus /data/maptr \
  --max-sweeps 10

tools/create_data.py dispatches nuscenes twice for a non-mini version: it passes v1.0-trainval, then v1.0-test, to the nuScenes converter. The trainval call writes nuscenes_infos_temporal_train.pkl and nuscenes_infos_temporal_val.pkl; the test call writes nuscenes_infos_temporal_test.pkl. A v1.0-mini request is handled once and uses the mini split. The converter also performs 2D annotation export. MapTR's map dataset consumes the temporal infos records and derives local vectors from the map expansion at runtime.

Argoverse2 Sensor

The documented root is data/argoverse2/sensor/ with exactly the split level train/, val/, and test/. Each log directory needs the sensor data used by the loader and one map archive matching map/log_map_archive_*.json. The converter is:

python tools/data_converter/av2_converter.py \
  --data-root /data/maptr/argoverse2/sensor --nproc 8

The CLI defaults to 64 workers; choose a bounded value for a first run. Its __main__ path processes train, val, and test, removes a fixed list of known failing logs, queries lidar timestamps, chooses the closest image for seven ring cameras, and discards samples with unsynchronized lidar/camera paths. It loads one JSON map archive per log and writes, at the sensor root, av2_map_infos_train.pkl, av2_map_infos_val.pkl, and av2_map_infos_test.pkl. The config normally consumes train and val files and uses load_interval=4 for validation.

Full converters are reference-only for this route: they require downloaded private data, third-party APIs, map parsing, multiprocessing, and writes to large output files. The checker is the safe preflight; do not replace it with a full conversion run on synthetic folders.

MapTR data contract

For nuScenes configs, set data_root to the nuScenes directory, ann_file to the matching temporal train/val pkl, and leave map_ann_file for the map evaluation-format JSON (for example, nuscenes_map_anns_val.json). map_ann_file is not a substitute for the raw map expansion or the temporal annotation pkl. For AV2, set data_root to the sensor directory and use av2_map_infos_{train,val}.pkl; the dataset loader expects each pkl to contain samples and id2map, sorted by sample timestamp. map_ann_file similarly names the map-format output used by validation/test formatting.

Both custom datasets expose three map classes in the standard MapTR configs: divider (label 0), ped_crossing (label 1), and boundary (label 2). VectorizedLocalMap obtains nuScenes divider lines, pedestrian-crossing polygons, and road/lane contour boundaries from the four supported map locations. VectorizedAV2LocalMap obtains divider polylines, pedestrian polygons, and drivable-area boundary polygons from the converter's id2map. Unexpected vector class names are errors; do not silently remap them.

The configured fixed-point representation is 20 points per ground-truth and predicted line. A nonempty LiDARInstanceLines.fixed_num_sampled_points has shape [N, 20, 2], in local x/y coordinates, clamped to the patch. Shifted representations may have shape [N, shifts, 20, 2]; closed polygons receive cyclic shifts, open lines receive forward/reverse variants, and padding uses -10000. Empty vector sets must remain representable; never fabricate a placeholder line. The patch is derived from pc_range: height is y_max-y_min, width is x_max-x_min. For the common nuScenes range [-15,-30,-2,15,30,2], the local patch is (60,30) and x/y limits are approximately [-15,15]/[-30,30]. AV2's common range reverses those planar extents.

Handoff and routing

Report: dataset and release, absolute roots, raw checks, converter command planned or run, generated pkl names, map-archive count, map classes, fixed point count, and any unresolved dependency or data-quality issue. Do not claim native conversion or model execution from a filesystem-only pass.

For configuration values and pc_range consistency use model-configuration. For train/test and runtime commands use training-evaluation. For map annotation rendering and benchmark plots use visualization-benchmarking. Use workflows, data formats, and troubleshooting for detailed recovery.