---

name: inference-and-evaluation description: "Use the MedicalDetectionToolkit Predictor, post-processing, Evaluator, and plotting contracts to run bounded inference, consolidate detections, score ROI/patient results, and inspect saved outputs." metadata: disco-role: operating disable-model-invocation: true license: Apache 2.0

Inference and evaluation

Use this node after a valid experiment configuration, data-loader batch, model output, or saved prediction pickle exists. It explains the source-backed prediction/evaluation pipeline; it does not provide a checkpoint, dataset, trained-model performance claim, or a replacement for the legacy CUDA path.

Evidence and compatibility boundary

The operating facts below were distilled from the repository's predictor, evaluator, plotting, execution, defaults, and README evidence. Public signatures were inspected with a modern isolated stack; that inspection did not prove the historical custom CUDA operators or model inference.

• Source-backed: function signatures, mode branches, array/key conventions, file names, thresholds, matching rules, metric formulas, and output-writing behavior described in the linked references.
• Verified portable boundary: pure-Python/NumPy/Pandas/scikit-learn helper behavior is suitable for small CPU probes, subject to the legacy API caveats in troubleshooting.
• Unverified: accuracy, runtime, memory use, checkpoint quality, dataset-specific thresholds, and exact detector execution. The source requirements pin Python-3.6-era packages including torch 0.4.1; the repository is unmaintained and README points to nnDetection, whose APIs must not be assumed compatible.

Route and workflow

1. Start with configuration and data contracts. Confirm cf.dim, cf.class_dict, cf.patch_size, cf.batch_size, cf.fold_dir, cf.test_n_epochs, cf.test_aug, cf.wcs_iou, cf.merge_2D_to_3D_preds, cf.merge_3D_iou, cf.ap_match_ious, cf.report_score_level, cf.min_det_thresh, and output paths. Route configuration questions to configuration-and-experiments and batch/patch shape questions to data-and-preprocessing.
2. Choose the Predictor mode. Use Predictor(cf, net, logger, mode='val') for patient validation/monitoring, mode='test' for checkpoint inference, and mode='analysis' only for load_saved_predictions in exec.py's analysis branch. See prediction workflows.
3. Preserve raw predictions before post-processing. predict_test_set collects epochs and XY mirrors, writes raw_pred_boxes_list.pickle or raw_pred_boxes_hold_out_list.pickle, and optionally applies WBC and 2D-to-3D merging. Use load_saved_predictions(apply_wbc=True) for repeatable analysis without a network forward. See consolidation.
4. Evaluate only schema-valid results. Evaluator.evaluate_predictions matches det boxes to gt boxes class-by-class at each configured IoU, creates the internal dataframe, and then return_metrics computes ROI AP and patient AP/AUC. Validate records with scripts/validate_evaluation_records.py before expensive analysis; see evaluation.
5. Write or inspect artifacts. score_test_df writes fold dataframes/results text and aggregates all fold pickles when available. Hold-out analysis calls utils.exp_utils.create_csv_output; monitoring/curves use monitoring and outputs. Do not interpret a plot or score as a performance guarantee without dataset and checkpoint provenance.

Result contract at a glance

A patient result is [boxes_by_batch_instance, pid]. boxes_by_batch_instance is a list of lists; 3D uses one dummy batch instance, while unmerged 2D uses one list per slice/batch element. Each detection box needs box_type: 'det', box_coords ([y1, x1, y2, x2] or [y1, x1, y2, x2, z1, z2]), box_score in the model's confidence scale (normally [0, 1]), and integer box_pred_class_id. Ground truth entries use box_type: 'gt', box_coords, and box_label. Raw patch detections additionally carry patch_id, box_patch_center_factor, and box_n_overlaps; WBC consumes these and emits consolidated detections.

A batch passed to predict_patient must include model input data, pid, original_img_shape, and patient targets/labels when validation needs ground truth; patched batches additionally include patch_crop_coords. Prediction outputs contain boxes, seg_preds, and (in validation) monitor_values. Segmentation averaging is implemented in tiling, but instance/semantic segmentation evaluation is explicitly not implemented by Evaluator.

Cross-links and progressive disclosure

• Read prediction workflows for mode prerequisites, patching, mirroring, checkpoint/epoch ensembling, and saved-prediction lifecycle.
• Read consolidation for WBC math, wcs_iou, n_ens, patch-center/overlap weights, and 2D-to-3D connected-slice merging.
• Read evaluation for dataframe records, ROI/patient matching, AP/AUC/ROC/PRC semantics, and fold aggregation.
• Read monitoring and outputs for CSV, pickle, text, histogram, stat-curve, and sampled prediction outputs.
• Read troubleshooting before changing thresholds or attempting old CUDA-backed detectors; route model/custom-op failures to models-and-architectures and cuda-extensions.
• Use scripts/validate_wbc_inputs.py for small, dependency-free structural checks on WBC inputs and scripts/validate_evaluation_records.py for dependency-free record checks. They never download models, import the checkout, or run inference.

Safe-stop rules

Stop and fix the input rather than silently continuing when patient IDs, batch-instance nesting, class IDs, coordinate dimensionality, box_type, or required patch metadata disagree. Stop before claiming a result when epoch ranking/checkpoints are missing, fold counts do not match, custom CUDA imports fail, or metric inputs contain only one patient class for AUC. Keep raw pickle outputs immutable while experimenting with WBC/thresholds so analysis remains reproducible.