---

name: model-training description: "Configure, preflight, run, resume, and troubleshoot DeepDanbooru 1.0.0 model training with a CPU-first TensorFlow workflow." disable-model-invocation: true metadata: disco-role: operating license: MIT

Model training

Use this sub-skill after a DeepDanbooru project and Danbooru-style dataset exist. It covers configuration, a read-only preflight, training dispatch, checkpoint resume, and saved-model artifacts for DeepDanbooru 1.0.0.

CPU TensorFlow is the required verified backend. GPU execution is optional and unverified; never infer GPU readiness from device discovery or a CPU success. Full training is intentionally skip-expensive during skill verification: the ResNet variants, image pipeline, multiple epochs, checkpoint I/O, and model exports can consume substantial compute, memory, and storage. Verify structure and a bounded data sample first, then launch training only with an explicit runtime and resource decision.

Route the request

Need	Read or run
Create project.json, tags.txt, or the SQLite/image layout	project-data-setup
Validate configuration values without training	validate_project_config.py
Validate config, tags, SQLite rows, image paths, checkpoints, and packages	training_preflight.py
Start, resume, or initialize from a saved model	Training workflows
Understand row selection, image paths, labels, and augmentation	Training data pipeline
Choose a model, optimizer, loss, schedule, or precision policy	Model and loss reference
Diagnose empty runs, missing images, unsupported values, or resume surprises	Troubleshooting
Evaluate the resulting Keras model	inference-evaluation
Convert to TFLite or create Grad-CAM output	post-training-tools

Supported dispatch values

Use these exact lowercase strings in project.json:

• Models: resnet_152, resnet_custom_v1, resnet_custom_v2, resnet_custom_v3, resnet_custom_v4.
• Optimizers: adam, sgd, rmsprop.
• Losses: binary_crossentropy, focal_loss.

Any other value fails before training. The default project uses resnet_custom_v2, adam, binary_crossentropy, and an image shape of 299 by 299 with three channels. Shape order in the model is (image_height, image_width, 3) even though the JSON fields list width and height separately.

Safe preflight

From this sub-skill directory, run:

python scripts/validate_project_config.py PROJECT --check-paths
python scripts/training_preflight.py PROJECT --check-packages

Both helpers are read-only, deterministic, offline, and use the standard library for configuration, filesystem, tag, and SQLite checks. They do not launch training. Package and TensorFlow import probes are opt-in so basic validation still works when optional runtime packages are absent.

The preflight checks a bounded, ID-ordered image sample by default. Use --max-image-checks 0 only when a complete filesystem scan is intended and its I/O cost is acceptable. Use --probe-imports --probe-tensorflow to test the installed modules and confirm that TensorFlow exposes a CPU; this still does not train or verify a GPU.

Do not launch training unless all of these hold:

1. project.json parses and all dispatch values and numeric ranges are valid.
2. tags.txt is non-empty and is the intended fixed output vocabulary.
3. The SQLite posts table has the required columns and at least one eligible row at minimum_tag_count.
4. Derived image paths exist for the checked rows, and the image format is actually decodable by the training environment.
5. Checkpoint state is understood: an existing latest checkpoint causes an automatic resume.
6. The CPU TensorFlow environment can import DeepDanbooru, TensorFlow, and TensorFlow I/O.

Launch boundary

The side-effecting command is:

deepdanbooru train-project PROJECT_PATH

To load initial weights/model structure from an existing saved model:

deepdanbooru train-project PROJECT_PATH --source-model MODEL.keras

--source-model is not a no-resume flag. After the model is loaded, any latest checkpoint under PROJECT_PATH/checkpoints/ is restored and can supersede its state. Preserve or deliberately relocate checkpoints before changing training lineage. Also confirm source-model input/output compatibility with the image shape and number of tags; the training function does not perform a friendly compatibility check.

Runtime behavior to preserve

Training compiles with precision and recall metrics, shuffles records once per epoch from a checkpointed seed, applies an optional epoch learning-rate schedule, trains in checkpoint-sized slices, and saves after each slice. Checkpoint state includes optimizer/model variables plus used_epoch, used_minibatch, used_sample, offset, and random_seed; only the latest three checkpoints are retained.

Periodic exports are named:

model-MODEL.eEPOCH.keras

The final export is:

model-MODEL.keras

Exports omit optimizer state. Resume therefore depends on checkpoints/, not on a .keras export. export_model_per_epoch: 0 means export every epoch; otherwise an epoch export occurs when the completed epoch is divisible by that positive interval.

With mixed_precision: true, training uses a mixed_float16 model and a loss-scaled optimizer, then attempts separate float32 exports ending in .float32.keras. The 1.0.0 implementation reconstructs a float32 model and restores checkpointed model variables into it. Treat those exports as needing explicit load/inference validation. In particular, combining --source-model with mixed precision leaves the float32 export model unconstructed and can fail during export; the bundled preflight rejects that combination.

Failure semantics

A missing image is not reliably fatal at the dataset boundary because Dataset.ignore_errors() can drop load/decode failures. A run with many bad paths can therefore appear to progress while using fewer samples; a run with no usable records can produce checkpoints or model files without meaningful updates. Treat missing files, empty eligible selections, empty tag records, and zero usable batches as failures, not warnings to ignore.

Unsupported model, optimizer, or loss strings fail during dispatch. A checkpoint_frequency_mb of zero can prevent offset progress; a console_logging_frequency_mb of zero causes modulo-by-zero; non-positive batch/image/epoch values are invalid for an intended training run. Use the validators rather than discovering these conditions after allocating a long job.

Handoff

After training, retain project.json, the exact ordered tags.txt, the final .keras model, and any checkpoints needed for further resume. Validate the saved model through inference-evaluation before conversion or Grad-CAM work in post-training-tools. If project/data checks fail, return to project-data-setup instead of changing model hyperparameters to mask a data problem.