---

name: "trajectories-and-datasets" description: "Record ManiSkill episodes, replay and convert HDF5 trajectories, manage demos/assets, and plan motion-planning or teleoperation data workflows." metadata: disco-role: operating disable-model-invocation: true license: NOASSERTION

Trajectories and Datasets

Use this sub-skill from the mani-skill root when the task is about recorded ManiSkill data: collecting episodes, downloading demonstration or asset bundles, replaying HDF5 trajectories, converting trajectories, loading trajectory data in Python, running motion-planning data generation examples, or collecting teleoperation demonstrations.

Route away from this sub-skill when the request is mainly about:

• General environment creation, observation modes, vectorization, rendering, or controller use: load ../environment-usage/.
• Writing or changing task definitions, assets, rewards, registration, or custom scenes: load ../custom-environments/.
• RL/IL baseline training loops, evaluation sweeps, or benchmark hyperparameters: use the learning/baseline sub-skill instead.

Fast routes

User need	Use
Record episodes from an already-created environment	RecordEpisode; see CLI/API notes.
Understand .h5/.json trajectory bundles	data layout.
Validate a trajectory bundle before doing anything expensive	python scripts/inspect_trajectory_bundle.py PATH/trajectory.h5.
Plan a replay command without running replay	python scripts/plan_replay_command.py PATH/trajectory.h5 ....
Download official demos or required assets	download planning and python scripts/preview_download_options.py.
Replay demonstrations to add observations/rewards/videos or convert control modes	replay workflows.
Convert ManiSkill trajectories to LeRobot format	LeRobot conversion and python scripts/plan_lerobot_conversion.py.
Load data with PyTorch	ManiSkillTrajectoryDataset; see CLI/API notes.
Generate data from built-in motion-planning examples	motion-planning workflow.
Collect click+drag teleoperation data	teleoperation workflow.
Diagnose failed replay, conversion, recording, downloads, or display issues	troubleshooting.

Core operating rules

1. Treat a ManiSkill trajectory as a paired bundle: a .h5 file plus a sibling .json file with the same stem. Replay, dataset loading, and LeRobot conversion all rely on the JSON metadata.
2. Prefer source metadata over filename guesses. File names commonly encode <obs_mode>.<control_mode>.<sim_backend>, but the JSON env_info and each episode's control_mode, reset_kwargs, and info are authoritative.
3. Never launch networked downloads, long replays, full conversions, or GUI teleoperation without explicit user intent. Use the bundled planning scripts first when the next action is unclear.
4. Control-mode conversion is limited and robot/controller dependent. Panda trajectories are the best-supported path; GPU-parallel replay does not support converting to a different control mode.
5. Use source_type and source_desc when recording new demos so future agents can distinguish motion-planning, teleoperation, RL-policy, or other sources.
6. Keep training-policy details out of this sub-skill. This sub-skill prepares data; it does not teach imitation or reinforcement learning baselines.

Minimal recipes

Record a trajectory bundle

from mani_skill.utils.wrappers.record import RecordEpisode

env = RecordEpisode(
    env,
    output_dir="demos/PickCube-v1/manual-check",
    trajectory_name="trajectory",
    save_trajectory=True,
    save_video=False,
    source_type="manual",
    source_desc="short human-readable collection note",
)

Close the wrapper to flush metadata, then validate the bundle:

python scripts/inspect_trajectory_bundle.py demos/PickCube-v1/manual-check/trajectory.h5

Replay or reprocess a bundle

python -m mani_skill.trajectory.replay_trajectory \
  --traj-path demos/PickCube-v1/teleop/trajectory.h5 \
  --save-traj -o state -c pd_joint_delta_pos

For exact visual/state reproduction, prefer --use-env-states; for CPU-to-GPU or GPU-to-CPU replay attempts, consider --use-first-env-state and keep the original backend available when precision matters.

Convert to LeRobot format

python scripts/plan_lerobot_conversion.py \
  demos/PickCube-v1/teleop/trajectory.h5 \
  converted/pickcube_lerobot

The planner only prints the command and dependency status. Run the printed python -m mani_skill.trajectory.convert_to_lerobot ... command only after the user confirms the conversion and output directory.

Bundled references and scripts

• workflows: end-to-end recording, download, replay, conversion, motion-planning, and teleoperation flows.
• CLI/API notes: signatures, options, and module entry points for RecordEpisode, replay, converter, downloader, dataset, and generation examples.
• data layout: HDF5/JSON schema, replay naming, LeRobot output shape, dataset-loader behavior, and trajectory utilities.
• troubleshooting: concrete fixes for missing pairs, conversion limits, optional dependencies, download prompts, video recording, and display issues.
• scripts/inspect_trajectory_bundle.py: safe HDF5/JSON bundle validator.
• scripts/plan_replay_command.py: safe replay-command planner.
• scripts/plan_lerobot_conversion.py: safe LeRobot conversion-command planner and dependency checker.
• scripts/preview_download_options.py: safe demo/asset UID and command planner; it does not download anything.