---

name: data-preparation description: "Prepare and route BigGAN-PyTorch ImageFolder, CIFAR10/CIFAR100, ImageNet HDF5, and Inception-moment data with the repository's exact mappings, transforms, and resource safeguards." disable-model-invocation: true metadata: disco-role: operating license: MIT

BigGAN-PyTorch data preparation

Use this sub-skill when preparing a dataset for train.py, building an ImageNet HDF5 cache, checking a cache, or generating the reference Inception moments used by FID. This is runtime guidance for the checked-out BigGAN-PyTorch repository. Run repository commands from the repository root; invoke bundled helpers through the root skill directory, and keep large data and metric outputs in an explicit data/output location.

Set SKILL_ROOT to the directory containing the root SKILL.md and REPO_ROOT to the BigGAN-PyTorch checkout before using the commands below.

Safety and route selection

• ImageNet downloads and dataset acquisition are network-dependent and expensive. Do not download ImageNet or CIFAR implicitly as a smoke test. CIFAR's torchvision download is also network-dependent; use download=False only after the archive has been installed and verified.
• HDF5 conversion is expensive. It reads every source image, writes a potentially very large file, and consumes substantial time and disk. The repository's scripts/utils/prepare_data.sh is a recipe, not a safe smoke test. Inspect with --help or use the bundled metadata-only validator before approving a conversion.
• Inception moments require CUDA in the source implementation. calculate_inception_moments.py hard-codes device = 'cuda', walks the full dataset, and accumulates activations in host RAM. It is a full metric job, not a quick validation.
• Choose one representation and make its data root explicit:
  • I32, I64, I128, or I256: ImageFolder/ImageNet directories.
  • I32_hdf5, I64_hdf5, I128_hdf5, or I256_hdf5: preprocessed ILSVRC_HDF5 files.
  • C10 or C100: the repository's CIFAR loaders at native 32x32.

Read references/data-formats.md for schemas and root mappings, references/workflows.md for command sequences and resource choices, and references/troubleshooting.md for failure diagnosis. The references are relative to this data-preparation sub-skill directory. Use the validator via $SKILL_ROOT/sub-skills/data-preparation/scripts/validate_hdf5.py so the command remains resolvable when copied from the skill root.

Canonical command sequence

From the checkout, after an ImageNet tree has been deliberately provisioned under data/ImageNet, the README's default 128px recipe is:

python make_hdf5.py --dataset I128 --batch_size 256 --data_root data
python calculate_inception_moments.py --dataset I128_hdf5 --data_root data

The first command creates data/ILSVRC128.hdf5; the second reads that file and writes I128_inception_moments.npz in the current working directory. The recipe does not download ImageNet. Confirm free disk, RAM, CUDA, and whether the output moment file should instead be moved to the configured metrics root before starting either expensive command.

For a small, already-installed dataset, first inspect a loader or run a one-batch diagnostic with num_workers=0, then scale batch_size and workers. Do not claim a full data-preparation verification from --help alone.

Loader behavior that must be preserved

utils.get_data_loaders() appends root_dict[dataset] to data_root, builds one training DataLoader, and returns it in a one-element list. The dataset key controls resolution, class count, root name, and model metadata. Use --shuffle for training unless deliberately reproducing ordered traversal; drop_last=True is the default in normal loader use. pin_memory is enabled by default and can be disabled with --no_pin_memory.

• ImageFolder data is decoded as RGB with PIL (or accimage when selected by torchvision), then transformed. Supported suffixes are .jpg, .jpeg, .png, .ppm, .bmp, and .pgm. Class directories and nested image files are traversed in sorted order, so classes receive stable lexicographic indices. An index listing is cached as <dataset>_imgs.npz in the process's current working directory; remove the cache after changing classes or files.
• CIFAR10 and CIFAR100 are read from torchvision-style pickle archives. Each sample becomes a PIL image before the configured transform. Both are native 32x32; C10 has 10 classes and C100 has 100. The custom CIFAR loader reads the archive arrays into memory regardless of its load_in_mem argument. val_split and train='validate' are supported by the class but are not exposed by the normal get_data_loaders convenience call.
• HDF5 samples are stored as NCHW uint8 and are converted by ILSVRC_HDF5 to float tensors with ((pixel / 255) - 0.5) * 2, i.e. approximately [-1, 1]. The HDF5 loader intentionally ignores the torchvision transform argument; --augment therefore does not add random crops or flips to an _hdf5 dataset.

Transform and resource decisions

For non-HDF5 ImageNet, no augmentation means center-crop the shorter square from the image's long-edge geometry, resize to the selected resolution, then ToTensor() and normalize each channel with mean/std [0.5, 0.5, 0.5]. With --augment, use a random long-edge square crop, resize to the selected resolution, and random horizontal flip before the same tensor conversion and normalization. CIFAR without augmentation applies only tensor conversion and that normalization; CIFAR augmentation is RandomCrop(32, padding=4) plus RandomHorizontalFlip().

Run the bundled checker from the repository root as python "$SKILL_ROOT/sub-skills/data-preparation/scripts/validate_hdf5.py" "$REPO_ROOT/data/ILSVRC128.hdf5" --resolution 128 --check-label-range; it is metadata-only unless label range checking is requested.

--load_in_mem has different practical consequences:

• ImageFolder decodes and transforms every image during dataset construction and retains the transformed samples and labels. If the transform is random, its random crop/flip is sampled once at construction rather than anew each epoch. This is fast but can consume far more RAM than the compressed source.
• HDF5 loads both imgs and labels arrays into RAM. Omitting it causes the loader to open the HDF5 file and read the requested sample for each __getitem__; this saves RAM but can make I/O and multi-worker behavior slower. The README reports roughly a 64GB I128 file and recommends about 96GB+ RAM for the training recipe that loads it.
• More num_workers can hide image decode/I/O latency, but each worker adds process overhead and HDF5 access pressure. Start HDF5 with 0 or a small worker count, then increase only after measuring. pin_memory improves host-to-CUDA transfer at the cost of pinned host memory.

For a custom dataset, add a loader in datasets.py and keep dset_dict, imsize_dict, root_dict, nclass_dict, and classes_per_sheet_dict in utils.py synchronized. Verify root, image size, channels, class ids, and normalization before training or moments generation.