---

name: training description: "Author DeepMedic 0.8.4 training configurations, launch CPU/GPU training, resume or fine-tune checkpoints, and inspect metrics and outputs safely." disable-model-invocation: true metadata: disco-role: operating license: NOASSERTION

DeepMedic training

Use this skill when the task is to configure or operate a DeepMedic 0.8.4 training session. The canonical training entry point is deepMedicRun; this skill covers the training session, not model architecture authoring, raw NIFTI preparation, or test-only inference.

Operating boundary

• Training is expensive. A full 3-D DeepMedic run can take hours or days and consumes substantial CPU/GPU memory. Start with a tiny configuration, one or a few cases, one or two epochs, few subepochs, and a small segment count as a smoke test. Only then scale to production values.
• The verified runtime is Python 3.8.20, TensorFlow 2.6.2, NumPy 1.19.5, SciPy 1.7.3, pandas 1.2.5, NiBabel 3.2.2, matplotlib 3.5.3, and protobuf 3.20.3. The package imports successfully and both deepMedicRun -h and the progress plotter help path exit successfully. TensorFlow is not declared by setup.py, so install and verify it separately in the runtime environment.
• The tested TensorFlow CUDA build is 11.2 and sees eight A100 GPUs (compute capability 8.0). This is evidence about the verified environment, not a requirement that every deployment have these GPUs. A GPU deployment must still use a TensorFlow/CUDA/cuDNN combination compatible with its host.
• Input images, labels, and optional masks must already be valid, co-registered NIFTI data. Follow data preparation. For downstream segmentation or test-only output semantics, use inference.

Fast routing checklist

1. Confirm the model configuration and its number of output classes. Labels include background class 0 and use contiguous class ids.
2. Create a Python-syntax train config. At minimum, provide folderForOutput, channelsTraining, gtLabelsTraining, and batchsize_train; sessionName is strongly recommended. Paths in a train config are interpreted relative to the config file, not the caller's current directory.
3. Decide CPU versus GPU explicitly. Use -dev cpu for a small smoke test; use -dev cuda for TensorFlow's visible GPUs or -dev cudaN to select one numbered host GPU. See workflows.
4. Run a smoke test before enabling validation, affine augmentation, multiprocessing, full-volume validation, or a large segment budget.
5. Inspect the read-only output report from scripts/check_training_outputs.py, then inspect the text log. Plot with scripts/plot_training_progress.py or enable TensorBoard.
6. For continuation, load a complete checkpoint prefix. For fine-tuning a pretrained network with a fresh schedule, add -resetopt; this preserves network weights but resets the trainer and optimizer state.

CLI contract

The training invocation has this shape (the executable must be available in PATH):

deepMedicRun -model MODEL_CFG -train TRAIN_CFG [-load CHECKPOINT_PREFIX] [-resetopt] [-dev cpu|cuda|cudaN]

• -model is required by the front end and points to the architecture config.
• -train selects training and must follow -model conceptually; it points to the Python-syntax train config.
• -load is optional and overrides cnnModelFilePath in the train config. It accepts a TensorFlow checkpoint prefix, not an individual shard.
• -resetopt is only valid with -train and -load/a configured checkpoint. It reinitializes trainer variables while retaining trainable net/* variables.
• -dev cpu sets CUDA_VISIBLE_DEVICES empty and binds the graph to /CPU:0. -dev cuda leaves CUDA devices visible and lets TensorFlow choose the available GPU. -dev cudaN masks to host GPU N and binds the process' visible GPU 0. An invalid device string is rejected.

Use deepMedicRun -h before a run to verify the installed entry point. GPU selection is not proof of GPU execution: check the session log's device list and the host's accelerator monitor, and treat CUDA/cuDNN initialization errors as a backend failure rather than silently trusting the requested flag.

Data contract at the training boundary

The direct file-list form is:

• channelsTraining: a Python list with one file-list path per input modality. Every list file has one NIFTI path per case; all modality lists must have the same number and order of cases.
• gtLabelsTraining: one file-list path containing one label NIFTI per case. It is required for supervised training. Labels are integer-like and must be in [0, numberOfOutputClasses - 1].
• roiMasksTraining: optional file-list path, one mask per case. With a mask, sampling is restricted to positive-mask voxels; without one, the full volume is eligible according to the sampling type.

The current parser also supports dataframe_train: a CSV with alphabetically sorted channel_* columns and a required ground_truth column (optional roi_mask). If it is supplied, it is used instead of the three direct training lists. Keep this alternative explicit; do not mix it accidentally with stale direct-list settings.

Validation is disabled by default. If either performValidationOnSamplesThroughoutTraining or performFullInferenceOnValidationImagesEveryFewEpochs is true, provide channelsValidation; provide gtLabelsValidation for sample validation and for validation metrics; provide roiMasksValidation only when available. Full validation output needs namesForPredictionsPerCaseVal when segmentation, probability maps, or feature maps are saved. Names are case names, not paths.

What happens in one run

The session creates the output tree, builds the TensorFlow graph, initializes or restores variables, then repeats epochs and subepochs. Each subepoch samples cases and patches, validates first when requested, trains in complete batches, and reports metrics. At epoch end it updates the schedule, increments the saved epoch counter, saves a checkpoint, and optionally performs full-volume validation. At normal completion it writes a final checkpoint. Sampling can be sequential (num_processes_sampling = -1 or default thread behavior 0) or parallel (>0 child processes); begin with -1/0 when diagnosing failures.

numberTrainingSegmentsLoadedOnGpuPerSubep is a sample budget, not a batch count. The routine computes optimization batches as the number of extracted segments integer-divided by batchsize_train; leftover segments are ignored. Keep the segment budget divisible by the batch size to avoid an unexpectedly short subepoch. numberOfSubepochs controls metric reports and fresh sampling, while numberOfEpochs is the training-state stopping target.

Checkpoint, resume, and fine-tune rules

TensorFlow writes a checkpoint set such as:

<model>.<session>.<timestamp>.model.ckpt.index
<model>.<session>.<timestamp>.model.ckpt.data-00000-of-00001

The value passed to -load or cnnModelFilePath must be the shared prefix ending exactly in .model.ckpt, not the .index file and not the .data-... shard. A complete checkpoint set is required. Use the same model architecture config when resuming; variable-name or shape mismatches indicate a different architecture or incompatible checkpoint.

• Resume interrupted training: load an epoch checkpoint without -resetopt. The network, optimizer accumulators, current schedule state, and saved num_epochs_trained are restored. Training continues until the configured total epoch count is reached; if the checkpoint already reached that count, no training epoch runs.
• Fine-tune: use the matching architecture and -load a pretrained prefix, usually with -resetopt. This restores only network weights and initializes optimizer variables, epoch counter, and schedule state from the new config. It does not reinitialize trainable network weights.
• Freeze selected layers: layersToFreezeNormal, layersToFreezeSubsampled, and layersToFreezeFC are one-based config layer numbers. The implementation converts them to zero-based internal indices; an omitted subsampled list mirrors the normal list.

Do not delete or overwrite a checkpoint while a run is active. The supplied output checker is read-only and can identify usable prefixes.

Metrics and progress

Each session log is <folderForOutput>/logs/<sessionName>.txt. Training and sample-validation metrics are reported per subepoch and epoch. Class 0 is reported as merged foreground in the accuracy monitor (not as ordinary background); class-specific reports for other classes are one-vs-all. Full validation Dice is computed from whole-volume inference and is distinct from sample Dice.

With tensorboard_log = True, TensorBoard event files are placed below <folderForOutput>/tensorboard/<sessionName>/train and /val. The logger records sample accuracy/cost and per-class accuracy, sensitivity, precision, specificity, and Dice; full-volume validation records Dice1, Dice2, and Dice3. TensorBoard files are binary and are not a substitute for the text log.

Use the bundled plotter with explicit log paths, for example:

python sub-skills/training/scripts/plot_training_progress.py LOG.txt --detailed --moving-average 20 --save-figure --output-dir PLOTS

The plotter parses only known DeepMedic log sentences, carries forward N/A metric values as the original utility does, smooths sample metrics but not full-volume Dice, and never evaluates log content as Python. --help documents all flags. Use --no-show in headless jobs. Basic mode plots overall accuracy; detailed mode supports --classes and plots validation/training accuracy, sensitivity, precision, sample Dice, and validation full-volume Dice.

Handoff

For the long config matrix, schedule behavior, augmentation dictionaries, output layout, and failure diagnosis, read the linked files:

• train-config.md
• workflows.md
• troubleshooting.md

Run the two bundled scripts with --help; they are safe wrappers and do not modify training outputs. A successful parser/help check is not a successful training run: verify the log reaches session completion and that checkpoint prefixes and expected prediction/event folders exist.