---

name: data-pipelines description: "Build timm data configs, datasets, transforms, loaders, ImageNet metadata, mixup/random erasing, and NaFlex data pipelines for train, eval, and inference." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

timm Data Pipelines

Use this sub-skill when a task needs timm data input handling rather than model discovery or optimizer/training-loop design. It covers create_dataset, ImageDataset, IterableImageDataset, resolve_data_config, create_transform, create_loader, ImageNet metadata helpers, mixup/random erasing, prefetcher/device/dtype choices, and NaFlex loader settings.

Route by Task

• Model-aligned inference/eval transforms: start with resolve_data_config(model=model) or resolve_data_config(pretrained_cfg=...), then pass the result into create_transform. See references/data-api.md and references/transform-loader-recipes.md.
• Folder, tar, TorchVision, HF, TFDS, or WebDataset inputs: use create_dataset with a reader-style name, root, split, search_split, and optional class_map. See references/data-api.md.
• Training and validation loaders: use create_loader to attach transforms to a dataset and create a PyTorch DataLoader or PrefetchLoader. Set device=torch.device('cpu') or use_prefetcher=False on CPU-only systems. See references/transform-loader-recipes.md.
• Augmentation policy choices: configure train-time create_transform/create_loader args such as scale, ratio, hflip, color_jitter, auto_augment, re_prob, re_mode, and re_count. Use Mixup or FastCollateMixup when labels need soft targets.
• NaFlex variable-resolution pipelines: use create_transform(..., naflex=True, patch_size=..., max_seq_len=..., patchify=...) for transform-only work or create_naflex_loader for loader-managed patchified batches. See references/transform-loader-recipes.md.
• Class names and ImageNet subsets: use ImageNetInfo and infer_imagenet_subset when mapping ImageNet indices, synsets, or model class counts to labels. See references/data-api.md.

Common Workflow

1. Create or inspect the model in the model-library sub-skill when model selection is in scope.
2. Resolve data settings with resolve_data_config(args=overrides, model=model).
3. Build an eval/inference transform with create_transform(**data_config) or a training transform with create_transform(is_training=True, **data_config, ...).
4. Create the dataset with create_dataset(...) only when files or dataset services are needed.
5. Create the loader with create_loader(dataset, **data_config, batch_size=..., is_training=...), explicitly deciding device, img_dtype, and use_prefetcher.

Bundled Probe

Run scripts/data_config_probe.py to inspect a model's resolved data config and summarize eval/train transform objects without requiring a dataset. The script prints JSON-safe output and is useful before wiring loaders.

Cross-Links

• Use the model-library sub-skill for selecting model names, pretrained configs, and NaFlex-compatible model creation.
• Use the training-workflows sub-skill for optimizer, loss, AMP, distributed training, and how mixup labels affect losses.
• Use the cli-workflows sub-skill for full train.py, validate.py, or inference.py command construction.

Troubleshooting First Stops

• CPU-only loader fails: set device=torch.device('cpu') when use_prefetcher=True, or disable the prefetcher.
• Accuracy is unexpectedly poor: re-check input_size, interpolation, mean, std, crop_pct, and crop_mode from the model's pretrained config.
• Dataset cannot find classes or split: check root, split, search_split, folder names, and class_map file format.
• NaFlex fails shape assertions: confirm both model and loader/transform are NaFlex-compatible and agree on patch_size, max_seq_len, and patchify_channels_last.

See references/troubleshooting.md for detailed failure modes.