---

name: datapipes description: "Build and debug PhysicsNeMo reader-to-Dataset-to-DataLoader data pipelines for TensorDict, mesh, streaming, and Hydra-configured workflows." metadata: disco-role: operating disable-model-invocation: true license: Apache 2.0

PhysicsNeMo Datapipes Operating Skill

Use this sub-skill when the user needs to load scientific ML data into PhysicsNeMo, build reader → dataset → transform → DataLoader workflows, validate TensorDict payloads, choose file formats, configure Hydra datapipes, or debug data-format/collation/device/streaming failures.

Trigger phrases

Use this sub-skill for requests that mention any of:

• PhysicsNeMo datapipes, readers, Dataset, DataLoader, TensorDict, MultiDataset, MeshDataset, or IterableDatasetBase.
• Loading NPZ/NumPy, HDF5, Zarr/TensorStore, VTK/STL, .pmsh, or .pdmsh data into a model.
• Normalize, SubsamplePoints, Compose, KNearestNeighbors, MeshToTensorDict, custom readers/transforms/collators, Hydra ${dp:...} configs, or data-pipeline reproducibility.
• Errors involving missing fields, wrong file patterns, mismatched TensorDict keys/shapes, optional dependencies, CPU↔CUDA transfer, stream prefetching, or online/iterable generators.

Fast route

1. Classify the data source. Use the reader selection notes below, then open references/data-formats.md for exact layout expectations.
2. Build the smallest pipeline. Start with Reader, inspect len(reader), reader.field_names, and reader[0]; then wrap with Dataset, then DataLoader. Recipes are in references/datapipe-workflows.md.
3. Validate the payload. Confirm keys, shapes, dtypes, metadata, device, batch dimension, and collation strategy before connecting to a model.
4. Add transforms last. Apply Normalize, SubsamplePoints, spatial/mesh transforms, or custom transforms only after raw reader output is correct. Use references/api-reference.md for exact constructor names.
5. Debug synchronously first. For hard failures, set prefetch_factor=0 or call loader.disable_prefetch() before investigating CUDA streams or threaded I/O. See references/troubleshooting.md.
6. Run a tiny smoke when in doubt. The bundled helper scripts/create_tiny_npz_datapipe.py creates a temporary NPZ fixture and exercises NumpyReader → Dataset → DataLoader on CPU or CUDA.

Reader selection

• Single .npz with sample axis in the first dimension: use NumpyReader(path, fields=..., index_key=...).
• Directory of .npz sample files: use NumpyReader(directory, file_pattern="sample_*.npz", fields=...); this mode supports reader-side coordinated subsampling.
• Single or directory HDF5: use HDF5Reader; it reads root-level datasets by key and indexes the first dimension in single-file mode.
• Zarr groups: use ZarrReader for local Zarr groups, attributes-as-tensors, store caching, and single-group or directory mode.
• Large/local Zarr with async reads: use TensorStoreZarrReader when tensorstore is installed and you want explicit cache/concurrency controls.
• STL-style VTK surface samples: use VTKReader only when the optional VTK/PyVista stack is installed and the sample directories contain supported mesh files.
• Native PhysicsNeMo mesh archives: use MeshReader for .pmsh single meshes and DomainMeshReader for .pdmsh domain meshes; pair with MeshDataset and mesh transforms.
• No stable length or index: implement IterableDatasetBase and let DataLoader use the main-thread generator path.

Validation checklist

Before handing a datapipe to a training or inference loop, verify:

• len(reader) is correct for map-style data, or len(loader) intentionally raises for iterable data.
• reader.field_names or the first sample contains the expected keys.
• reader[0] returns (TensorDict, metadata) for tensor readers, or (Mesh/DomainMesh, metadata) for mesh readers.
• Dataset(reader, device=...) moves tensors to the intended device before transforms.
• Transform input keys exist and have compatible first dimensions for coordinated subsampling.
• The selected collator matches sample shape: default stacking for fixed shapes; concat/custom collation for variable-length point clouds or graph objects.
• DataLoader(..., collate_metadata=True) returns metadata only if the training loop needs source indices, filenames, or dataset_index.
• seed= and loader.set_epoch(epoch) are used when stochastic readers/transforms must be reproducible.

Route to sibling sub-skills

• For which model family or example to pair with a datapipe, route to model-selection.
• For mesh creation, mesh validation, repair, geometry calculations, or mesh serialization beyond loading, route to mesh-and-geometry.
• For distributed samplers, DDP/FSDP2, ShardTensor, domain parallel input scatter, or multi-GPU launch issues, route to distributed-and-domain-parallel.
• For diffusion/generative datasets tied to sampler or preconditioner workflows, route to diffusion-and-generative after the data path is validated.
• For active-learning orchestration, logging/checkpoints, or ONNX export after data loading, route to active-learning-and-deployment.

Operating cautions

• Treat readers as CPU I/O adapters. Do not put GPU kernels, model calls, downloads, credential prompts, or long preprocessing inside a Reader._load_sample implementation.
• Transforms mutate/update TensorDict-style payloads; order matters.
• PhysicsNeMo's DataLoader is thread/stream oriented rather than PyTorch multiprocessing oriented. Pinned reader output plus device="cuda" is what enables asynchronous host-to-device transfer.
• Never assume large domain examples are smoke tests. Use tiny generated fixtures for validation, then document external-data and optional-dependency requirements separately.