---

name: "data-preparation" description: "Routes PFLlib dataset generation, client-split construction, and dataset-layout validation for label-skew, feature-shift, and real-world scenarios." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Data Preparation

Use this route when you need to create or validate PFLlib client-split datasets. It covers the built-in scenario generators for label-skew, feature-shift, and real-world data layouts.

Use this when

• You need MNIST, CIFAR, Fashion-MNIST, AG News, Sogou News, Amazon Review, HAR, or another built-in PFLlib split.
• You need to regenerate a dataset after changing train_ratio, alpha, or a dataset-specific preprocessing step.
• You need to confirm that a dataset/<name>/ tree has the expected config.json, train/, and test/ layout before running experiments.
• You want to understand how PFLlib stores per-client .npz files and how read_client_data() turns them back into tensors.

Read these references

• references/data-formats.md for the generated dataset tree and file schema.
• references/workflows.md for the common generation and validation flows.
• references/troubleshooting.md for download, split, and layout failures.

Run these helpers

• scripts/run_dataset_generator.py to launch a repo dataset generator from a checkout without hard-coding source paths.
• scripts/validate_dataset_layout.py to check an existing split tree for missing files, stale metadata, or mismatched client counts.

What belongs here

Include dataset creation, split validation, dataset-root inspection, and scenario selection for:

• label skew: MNIST, EMNIST, FEMNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, AG News, Sogou News, Tiny-ImageNet, Country211, Flowers102, GTSRB, Shakespeare, Stanford Cars, COVIDx, and kvasir
• feature shift: Amazon Review, Digit5, and DomainNet
• real-world: Camelyon17, iWildCam, Omniglot, HAR, and PAMAP2

What does not belong here

• Launching federated learning experiments or summarizing training results; route that to experiments.
• Adding new algorithms, models, or optimizers; route that to extension.
• CUDA or package installation issues that happen before dataset generation; use the root troubleshooting and install checker.

Common workflow

1. Confirm the target dataset family and split style.
2. Run the bundled generator launcher against a checkout.
3. Validate the output tree with the bundled layout checker.
4. Only then hand the dataset root to the experiment runner.

If the generator reports that the dataset already exists, use the validator to confirm the tree instead of forcing a rewrite.