---

name: "read-datasets" description: "Routes Petastorm dataset-reading workflows, including plain Python iteration, TensorFlow and PyTorch adapters, Spark RDD reads, row-group selection, NGrams, and throughput benchmarking." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Read Datasets

Use this route when the user wants to open an existing Petastorm dataset or plain Parquet store and read samples back in Python, TensorFlow, PyTorch, or Spark. This route also owns predicates, sharding, row-group selectors, NGrams, shuffling, caching, and throughput measurement.

Typical triggers

• "How do I read this Petastorm dataset?"
• "How do I use make_reader or make_batch_reader?"
• "How do I select row groups or filter rows?"
• "How do I get TensorFlow tensors or a PyTorch DataLoader?"
• "How do I benchmark reader throughput?"

What belongs here

• make_reader
• make_batch_reader
• Reader
• row predicates and selectors
• sharding and epoch control
• NGram
• WeightedSamplingReader
• tf_tensors
• make_petastorm_dataset
• DataLoader, BatchedDataLoader, and InMemBatchedDataLoader
• dataset_as_rdd
• petastorm-throughput.py
• cache and cleanup behavior for reads

What does not belong here

• dataset writing and schema construction
• metadata repair
• copy and filter workflows
• Spark DataFrame conversion into cached datasets

Use sub-skills/create-datasets/ for those tasks. Use sub-skills/spark-converter/ when the starting point is a Spark DataFrame cache.

Read this first

• references/workflows.md for end-to-end read recipes
• references/api-reference.md for verified signatures and semantics
• references/cli-reference.md for the throughput CLI
• references/troubleshooting.md for import-order, metadata, and adapter failures
• scripts/smoke_read_minimal_dataset.py for a tiny read-back smoke test
• scripts/smoke_read_plain_parquet.py for the plain Parquet batch-reader path
• scripts/smoke_read_tensorflow.py when TensorFlow support needs a quick check
• scripts/smoke_read_torch.py when PyTorch support needs a quick check

How to choose a reading path

1. Petastorm dataset or plain Parquet?
   • Use make_reader for Petastorm datasets.
   • Use make_batch_reader for plain Parquet stores or when batch output is acceptable.
2. Need row-level logic?
   • Use predicate, rowgroup_selector, shuffle_rows, shuffle_row_groups, or transform_spec.
3. Need framework integration?
   • Use tf_tensors or make_petastorm_dataset for TensorFlow.
   • Use DataLoader, BatchedDataLoader, or InMemBatchedDataLoader for PyTorch.
4. Need aggregation or mixed sampling?
   • Use NGram or WeightedSamplingReader.
5. Need Spark RDD access or timing?
   • Use dataset_as_rdd or the throughput CLI.

Boundary reminders

• make_reader expects a Petastorm dataset and will warn if the store lacks Petastorm metadata.
• make_batch_reader is the route for non-Petastorm Parquet data.
• DataLoader and BatchedDataLoader are not interchangeable: the batched variant is optimized for tensorable fields and the in-memory variant is single-use.
• PyTorch users should import pyarrow before importing torch.
• TensorFlow dataset repetition should usually be controlled through the reader's num_epochs, not by repeating the dataset blindly.

Common tasks

Read a Petastorm dataset in plain Python

1. Call make_reader(dataset_url, ...).
2. Use reader_pool_type='dummy' for the smallest deterministic smoke checks.
3. Iterate rows and close the reader with a context manager.

Read a non-Petastorm Parquet store

1. Call make_batch_reader(dataset_url_or_urls, ...).
2. Limit schema_fields to the columns you actually need.
3. Treat returned objects as batches rather than single rows.

Connect to TensorFlow

1. Build a reader.
2. Call tf_tensors(reader) or make_petastorm_dataset(reader).
3. Consume the tensors inside a tf.Session() style loop.

Connect to PyTorch

1. Import pyarrow before torch.
2. Wrap the reader in DataLoader, BatchedDataLoader, or InMemBatchedDataLoader.
3. Avoid nullable or string-heavy fields when you want tensor-only batches.

Measure throughput

1. Use petastorm-throughput.py with a real dataset URL.
2. Start with the dummy or thread pool on a tiny dataset.
3. Increase cycles only after the basic path is working.

When to stop and switch routes

• If the problem is schema creation, field encoding, or dataset layout, switch to create-datasets.
• If the problem is converting a Spark DataFrame cache into a loader, switch to spark-converter.
• If the problem is a URL or filesystem resolution issue, read the root filesystem reference first.