---

name: training-and-datasets description: "Prepare, validate, adapt, and troubleshoot ControlNet training datasets and tutorial training workflows for SD1.5 and SD2.1." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

ControlNet Training And Datasets

Use this sub-skill when the task is about preparing a ControlNet training dataset, validating Fill50K-style rows, writing a custom PyTorch Dataset, adapting the tutorial training pattern for SD1.5 or SD2.1, tuning training knobs, or debugging dataset/training startup failures.

Do not use this sub-skill for image-generation UI workflows, checkpoint-conversion internals, or detector preprocessing. Route those requests to:

• Initial ControlNet checkpoint creation and model/config architecture: model-and-weight-utilities.
• Detector-generated control maps and preprocessing pipelines: annotators-and-preprocessing.
• Gradio inference apps and qualitative generation tests: gradio-inference-apps.

Fast Routing

• Validate Fill50K or a Fill50K-like dataset before training with the bundled validator; it checks prompt.json, row keys, image paths, image readability, shape consistency, and tutorial normalization ranges without importing ControlNet or starting training.
• Author custom datasets by matching the ControlNet data contract: jpg is the RGB target image normalized to [-1, 1], hint is the RGB control/source image normalized to [0, 1], and txt is the prompt string.
• Adapt SD1.5 training with models/cldm_v15.yaml and an initialized control_sd15_ini.ckpt; adapt SD2.1 training with models/cldm_v21.yaml and an initialized control_sd21_ini.ckpt or equivalent.
• Start with conservative training knobs (batch_size=1 to 4, learning_rate=1e-5, sd_locked=True, only_mid_control=False) and add gradient accumulation or low-VRAM mode only after the dataset and checkpoint load cleanly.
• Treat full tutorial imports/runs as unsafe preflight checks: they require data, checkpoints, GPU memory, and may trigger model/tokenizer downloads.

Bundled References And Tools

• Read dataset-format.md when validating Fill50K layout, writing a custom Dataset, checking prompt.json schema, or confirming jpg/txt/hint ranges.
• Read training-workflows.md when adapting the SD1.5 or SD2.1 tutorial training pattern, choosing configs/checkpoints, setting knobs, or planning low-memory runs.
• Read troubleshooting.md when a dataset, checkpoint, CUDA, Lightning, CLIP, or low-VRAM issue blocks training startup.
• Run validate_fill50k_dataset.py for safe dataset preflight checks or to create a tiny local fixture that demonstrates the expected Fill50K structure.

Safe Validation Command

From any working directory, point the validator at the dataset root that contains prompt.json, source/, and target/:

python path/to/training-and-datasets/scripts/validate_fill50k_dataset.py --dataset-root path/to/training/fill50k --max-items 100

Use --max-items 0 for a full scan before a long training job. Use --write-example-fixture path/to/tmp-fill50k --validate-written-fixture to create and validate a tiny self-contained fixture without network access, checkpoint loading, image generation, or training.