---

name: evaluation description: "Evaluate Medical SAM Adapter checkpoints through in-training validation or the independent val.py path, interpret threshold-averaged IoU/Dice, handle REFUGE and multi-class outputs, and diagnose 3D chunking, visualization, CUDA, and checkpoint compatibility." metadata: disco-role: operating disable-model-invocation: true license: GPL 3.0

Evaluation

Use this route to score a Medical SAM Adapter model, inspect a checkpoint before loading it, interpret IoU/Dice output, or reduce validation memory. The canonical graph entry is the Medical SAM Adapter root. This route covers both the periodic validation_sam call made by train.py and the independent val.py command. It does not own optimizer/adaptation choices, raw dataset layout, or standalone object-aware detector inference.

Safe route selection

1. Confirm a CUDA-enabled PyTorch runtime and a selected GPU. CUDA is required for actual validation: the evaluation code creates cuda:<gpu_device> tensors and moves the network and batches there. CPU parser/import checks are diagnostics only, not a supported evaluation substitute.

2. Route missing files, NIfTI/NRRD/3D shape or dataset-layout questions to data preparation. Route model, encoder, and adaptation decisions to training. Route the separately documented detector/box path to MobileSAMv2 inference.

3. Inspect a supplied checkpoint first:

   python scripts/inspect_checkpoint.py --checkpoint /path/to/file.pth
   

   The helper is read-only, maps to CPU, prefers weights_only=True, does not import repository code, and never constructs a model. Read checkpoint schema for the result and the distributed-prefix decision.

4. Keep the base-model file (-sam_ckpt) separate from the trained wrapper file (-weights). Select matching -net, -encoder, -multimask_output, -image_size, and adaptation structure before launching val.py.

5. Supply a positive integer to -vis. The parser default is None, but validation_sam evaluates ind % args.vis unconditionally; -vis 0 also fails. Preserve the exact checkpoint, dataset split, device, and output policy when comparing runs.

Choose the workflow

• Training validation: train.py calls function.validation_sam on the in-memory network during the first five epochs, at positive -val_freq intervals, and on the final epoch. It uses the current validation loader and writes metrics and optional samples in that run's log/sample directories.
• Independent evaluation: val.py builds a fresh network, loads the wrapper from -weights, creates a new timestamped logs/<exp_name>_* directory, and evaluates the test loader once. It still needs the matching -sam_ckpt, local data, and CUDA. It is not a source-independent metric implementation; it shares the repository loader, preprocessing, decoder, and validation_sam implementation.
• Metric interpretation: read metrics first. Predictions and targets are thresholded at five fixed thresholds and the per-threshold IoU/Dice values are averaged; a printed value is not simply a 0.5-threshold score.
• 3D: use -thd True for slice-wise evaluation of [B,C,H,W,D] data and -evl_chunk to split the depth dimension. -chunk controls the MONAI training crop, not independent validation. See workflows for remainder and aggregation behavior.

Exact command shape

val.py parses the shared CLI with the exact -flag spellings and defaults listed in workflows. A normal 2D invocation is:

python val.py -net sam -encoder vit_b -mod sam_adpt \
  -weights /path/to/adapter-wrapper.pth \
  -sam_ckpt /path/to/sam-base.pth \
  -dataset isic -data_path /path/to/data \
  -image_size 256 -out_size 256 -vis 50 -gpu_device 0

For REFUGE, use the implemented case-sensitive dataset name and two outputs:

python val.py -net sam -encoder vit_b -mod sam_adpt \
  -weights /path/to/refuge-wrapper.pth \
  -sam_ckpt /path/to/sam-base.pth \
  -dataset REFUGE -multimask_output 2 -vis 50

These commands are templates only: paths, weights, data, and compatibility must be checked first. Full evaluation is not a preflight and is not started without the user's real artifacts.

Critical source behaviors

• Several booleans use argparse type=bool; a non-empty string such as False is truthy. Do not assume -thd False or -gpu False disables a feature; omit the option or verify the parsed configuration.
• With original SAM, multimask_output > 1 causes the validation decoder call to request multiple masks. EfficientSAM and MobileSAM explicitly request multimask_output=False in validation_sam, regardless of the shared flag.
• The REFUGE loader returns channel 0 cup and channel 1 disc. -multimask_output 2 and two-channel metrics are therefore reported separately; do not collapse them into one score. See the channel-order caveat in metrics.
• In a non-none -distributed mode, get_network wraps the model in DataParallel; val.py adds one module. prefix to every stored key. The saver writes net.module.state_dict(), so the expected source checkpoint is unprefixed. Do not feed an already-prefixed mapping into this path.

For exact defaults, output paths, metric formulas, checkpoint keys, and recovery steps, use the four bundled references. Escalate shared environment issues through the root troubleshooting route when that root reference is available.