---

name: processing-streaming description: "Transform map-style and iterable datasets with map/filter/batching, formatting, caching, combining, and streaming-aware workflows." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Processing and Streaming

Use this sub-skill when a task is about preparing an already-loaded Dataset, IterableDataset, DatasetDict, or IterableDatasetDict for analysis, training, export, or further processing.

For initial loading from local files or the Hub, route to ../loading-local-hub/SKILL.md. For schema declarations, feature casting, and modality feature types, route to ../features-formats/SKILL.md. For persistent cache locations, offline settings, and CLI environment concerns, route to ../sharing-cli-cache/SKILL.md.

What This Covers

• Map-style transforms: map, filter, select, shuffle, sort, shard, train_test_split, flatten_indices, and to_iterable_dataset.
• Iterable/streaming transforms: lazy map, filter, shuffle, skip, take, shard, reshard, set_epoch, and framework iteration.
• Batching and multiprocessing: batched row-count changes, remove_columns, features, num_proc, with_indices, and with_rank.
• Formatting and conversions: set_format, with_format, framework formatters, iter, to_pandas, to_polars, and TensorFlow/PyTorch handoff patterns.
• Dataset composition: concatenate_datasets, interleave_datasets, map-style versus iterable shard behavior.
• Cache/fingerprint behavior for Arrow-backed processing and why iterable processing does not reuse map-style cache files.

Reference Routing

• Read references/processing-workflows.md for practical decision trees and end-to-end patterns for map-style, streaming, batching, framework handoff, and combining datasets.
• Read references/api-reference.md for concrete parameter guidance, return semantics, and method differences across Dataset, IterableDataset, and dictionary containers.
• Read references/troubleshooting.md when processing fails with Arrow length/schema errors, multiprocessing pickling issues, cache reuse surprises, shuffle limitations, formatter dependency errors, or mixed type failures.
• Run scripts/processing_smoke.py --help to see a self-contained smoke check. Run it without arguments to verify local datasets processing behavior without downloading data.

Quick Guidance

Prefer map-style Dataset when you need random access, exact length, deterministic splits, cached Arrow transforms, or repeated inspection. Prefer IterableDataset when data is too large to materialize, when streaming from files or the Hub, or when fast approximate shuffling is sufficient for training.

For batched map, every returned column must have the same length within a batch. If the function changes row counts, overwrite existing columns or pass remove_columns for columns whose old length no longer matches. If the output type should be stable, pass explicit features; for mixed Python types, review on_mixed_types in references/api-reference.md.

Use with_format(...) for non-mutating formatting and set_format(...) for in-place formatting. Framework formatters may require optional packages such as PyTorch, TensorFlow, JAX, pandas, polars, image, or audio extras; do not assume those are installed.