---

name: data-and-preprocessing description: "Prepare, inspect, and safely adapt MedicalDetectionToolkit array datasets, labels, preprocessing, patching, and batchgenerators pipelines for toy, LIDC, and PET-CT-style experiments." metadata: disco-role: operating disable-model-invocation: true license: Apache 2.0

Data and preprocessing

Use this sub-skill when a Researcher must turn a preprocessed medical-image case into the dictionaries and arrays expected by MedicalDetectionToolkit (MDT), or when adapting one of the example loaders without silently changing axes, label semantics, or patch coordinates. This is an operating reference, not a data acquisition recipe. The repository is legacy and explicitly no longer maintained; reproduce the contracts below and verify the installed environment before changing them.

Operating contract

Given an experiment config cf, a logger, a manifest/data directory, and preprocessed image/segmentation arrays, produce one of these observable contracts:

• load_dataset returns an OrderedDict keyed by pid. Each value contains paths under data and seg, the pid, and a class_target list. LIDC and PET-CT additionally carry fg_slices; toy uses one array path for a packed two-plane [image, segmentation] file and does not carry foreground-slice metadata.
• A training batch contains data, seg, pid, and class_target. data is batched channels-first and seg has a singleton segmentation-channel axis. The spatial axes are (x, y) for 2D or (x, y, z) for 3D at the point the network sees them; the exact loader transposes are documented in data-formats.
• After ConvertSegToBoundingBoxCoordinates, a batch also contains bb_target and roi_labels. Patient iterators preserve patient-level targets as patient_bb_target and patient_roi_labels, plus original_img_shape; tiled batches additionally carry patch_crop_coords.

Before loading a real case, validate a portable image/segmentation pair with validate_preprocessed_case.py. The validator accepts only .npy/.npz arrays and optional JSON metadata, has hard size/voxel bounds, performs no repository imports, and never downloads or modifies data. Its JSON metadata format is described in troubleshooting.

Required workflow

1. Select the evidence-backed family. Use the toy loader only for small 2D onboarding data; use LIDC for per-patient CT volumes with instance-valued ROIs and optional 2D/3D operation; use PET-CT for the two-channel CT/PET classification-style example with a binary foreground segmentation. Read preprocessing-workflows before changing normalization, resampling, or label creation.
2. Freeze the manifest contract. The configured input_df_name is normally info_df.pickle. Its rows must agree with the filenames derived from pid. Do not infer a different axis order from a filename. Check class_target, fg_slices (where present), and spacing/shape metadata before training.
3. Validate arrays and metadata. Use the bundled validator on a copied or synthetic case. Never pass arbitrary pickle files to a new utility. The source loaders read pandas pickle manifests, so treat them as trusted local inputs only and keep their paths outside portable skills.
4. Construct the generator. Instantiate the appropriate BatchGenerator, then use create_data_gen_pipeline. Training composes optional mirror and SpatialTransform; validation/testing uses CenterCropTransform; both then call ConvertSegToBoundingBoxCoordinates. Exact flags and version-sensitive signatures are in data-loader-contracts.
5. Check labels after every geometric transform. Image and segmentation must be transformed together. Use interpolation order 0 / background 0 for segmentation. Confirm the resulting bb_target coordinates fit the post-transform patch and that class labels still correspond to ROI ids.
6. Record reversible customization. Keep a small manifest schema note and a before/after shape example for every custom loader. If a change affects axes, label mode, patch size, or channel count, update config and tests together rather than patching a downstream model.

Label and geometry rules

• Background is label 0; positive integer ROI ids are foreground. In LIDC, each nonzero ROI id is an individual lesion and class_target is aligned to those ids after the preprocessing script's rix assignment. Do not relabel ROIs without updating the aligned class vector.
• A pixelwise instance map has increasing ROI ids and needs get_rois_from_seg_flag=False. A binary map has foreground 1 and does not identify individual lesions; use get_rois_from_seg_flag=True so the batchgenerators converter performs connected-component labeling. The PET-CT pipeline uses this binary mode. A box-only source must first be represented as a pseudo-mask if it is to go through the same spatial augmentation path.
• class_specific_seg_flag controls whether converted segmentation targets preserve class-specific channels/labels. It is not a substitute for making a binary mask into instance ids. Check cf.head_classes and cf.num_seg_classes when changing it.
• The loader may return no foreground in a sampled patch even when the patient has lesions. Do not treat an empty patch as a corrupt case; distinguish it from an invalid all-negative patient or an out-of-range fg_slices entry.
• Coordinates are local to the current patch. Patient-level targets must be retained separately when tiling so predictions can be mapped/consolidated.

Dimensions, channels, and patching

• Toy is 2D only: its packed case stores image at index 0 and segmentation at index 1; BatchGenerator adds a batch axis and a singleton data/seg channel axis. It uses 320 x 320 pre-crop and patch settings in the supplied config.
• LIDC preprocessing writes (z, y, x) image and ROI arrays. The loader transposes them to (y, x, z) before sampling; 2D mode selects a z slice, while 3D mode retains the volume. The supplied config uses 300 x 300 pre-crops / 288 x 288 2D patches, or 156 x 156 x 96 pre-crops / 128 x 128 x 64 3D patches, and enables 2D-to-3D prediction merging.
• PET-CT preprocessing writes a two-channel image (c, z, y, x) and a binary (z, y, x) mask. The loader transposes the image to (c, y, x, z) and samples 2D slices or 3D volumes. The supplied config is 3D with 280 x 280 x 48 pre-crops and 192 x 192 x 32 patches, two channels, and 2D-to-3D merging enabled if 2D is selected.
• If an input is smaller than the configured pre-crop or patient patch, the loaders call pad_nd_image; padding is centered and defaults to constant zero in the shown paths. If larger, training samples a pre-crop (with foreground-biased sampling) and patient inference creates a coordinate grid using get_patch_crop_coords. A patch size of 1 in z means one patch per slice, not a three-dimensional slab.
• n_3D_context adds neighboring slices as channels in 2D mode. It changes cf.n_channels and requires matching model input configuration; padding is applied at the volume boundary. Do not enable it without checking channel count and the context coordinate adjustment.

Safety boundaries

• The example raw data roots, mounted paths, rsync copies, and cloud/server branches are environment-specific. LIDC-IDRI, PET/CT/TNM, and the toy data directories are external datasets; the PET/CT paths and clinical labels may be private. Do not claim access, redistribute them, or run acquisition from this skill. The supplied examples are evidence of schema and behavior only.
• experiments/lidc_exp/pack_dataset.py and utils/dataloader_utils.py::unpack_dataset/delete_npy are storage helpers, not safe validation steps. Packing/unpacking can create many files and delete_npy is destructive. Treat pack, delete, rsync, multiprocessing preprocessing, and raw-data conversion as non-runnable unless a user explicitly reviews paths, backups, and a bounded dry run outside this skill.
• Do not run native final cases, download data, or import this operating graph during construction. Use signatures/import checks and bounded synthetic validator fixtures only.

References and bundled script

• Data formats and array axes
• Data-loader and batchgenerators contracts
• Preprocessing workflows
• Data utilities: padding, tiling, balancing, packing boundaries
• Troubleshooting and safe customization
• Standalone preprocessed-case validator

All references link back here and to the other references so the graph remains usable when a Researcher opens a leaf directly. Source evidence is the checked- out README.md, the three experiment configs.py/data_loader.py pairs, LIDC preprocessing.py/pack_dataset.py, PET-CT preprocessing.py, and utils/dataloader_utils.py.

Verification gate

Accept a customized data path only when: (a) the three example loaders import with the documented loader dependency/API variant; (b) the validator passes a bounded synthetic valid case and rejects mismatched shapes, non-integral labels, or oversized input; (c) the manifest-to-file naming and class_target/ROI semantics are recorded; and (d) no native dataset case or destructive helper was run. Unresolved backend, private-data, axis, or label ambiguities stay explicit in the handoff.