---

name: data-pipelines description: "Build and troubleshoot Ray Data pipelines for loading, transforming, writing, and tuning datasets." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Ray Data Pipelines

Use this sub-skill when the task involves ray.data, Dataset, dataset readers such as read_csv, read_parquet, read_images, read_text, in-memory creation with from_items, row or batch transforms, write_parquet, schema or batch-format issues, optional Data dependencies, block sizing, concurrency, or Ray Data memory/performance behavior.

Fast Routing

• For Ray Core task/actor internals, object refs, placement groups, or cluster resources outside Data APIs, route to ../core-runtime/SKILL.md.
• For feeding datasets into Ray Train or Tune workflows, use this sub-skill for dataset construction and then route to ../train-tune/SKILL.md for trainers, tuners, scaling configs, and checkpoints.
• For Serve request batching or online inference deployment, route to ../serve-deployments/SKILL.md after any generic Ray Data preprocessing.
• LLM-specific data ingestion is out of default scope unless the solution only uses generic Ray Data APIs described here.

Working Pattern

1. Confirm the environment has the narrow extra needed for Data work, usually ray[data]; avoid recommending ray[all] by default.
2. Pick a source pattern: local/cloud file reader, from_items, from_pandas, from_numpy, or from_arrow.
3. Choose the transform level: map for one row to one row, flat_map for one row to many rows, and map_batches for vectorized pandas, NumPy, or PyArrow logic.
4. Keep output and validation explicit: call take, schema, materialize, stats, or a writer such as write_parquet to trigger lazy execution.
5. Tune only after the basic pipeline works: inspect blocks with materialize().stats(), then adjust override_num_blocks, concurrency, ray_remote_args, batch_size, and DataContext options.

Bundled References

• references/api-reference.md lists the core Ray Data APIs, signatures, parameters, and validation checks.
• references/workflows.md gives self-contained recipes for local/cloud/in-memory reads, batch transforms, writes, and Core/Train/Tune handoff.
• references/performance-and-troubleshooting.md covers block sizing, fusion/materialization, optional dependencies, schema errors, OOMs, and path/cloud issues.
• scripts/data_pipeline_smoke.py is a safe helper adapted from Ray Data key-concept examples; it defaults to --help/argument parsing and only runs a tiny local in-memory pipeline when --run is passed.

Minimal Sanity Check

Run the bundled smoke helper before debugging larger pipelines:

python scripts/data_pipeline_smoke.py --help
python scripts/data_pipeline_smoke.py --run --rows 4 --num-blocks 2

Add --write-parquet only when you want the smoke helper to create a temporary Parquet output as part of the check.