---

name: data-preparation description: "Guides SimpleDet annotation conversion and roidb validation for COCO-like, VOC, CrowdHuman, and custom JSON detection or instance-segmentation data." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Data preparation

Use this route for annotation conversion, data/cache/*.roidb, image layouts, class IDs, polygon masks, split names, or validation. Read data-formats.md before conversion and troubleshooting.md after a failure.

Workflow

1. Choose COCO-like JSON, VOC XML, CrowdHuman ODGT, or custom JSON.
2. Confirm the image/annotation layout and exact split names.
3. Use the bundled convert_roidb.py with an explicit --output; it never downloads data.
4. Run validate_roidb.py on a tiny result before using the cache for a model workflow.
5. Match the output basename to DatasetParam.image_set.
6. For masks, validate gt_poly, polygon lengths, and static config limits.

Data contract

• gt_class uses positive foreground IDs; 0 is background and CrowdHuman ignore regions use -2.
• gt_bbox is (N,4) xyxy, not COCO xywh.
• flipped should be false in cached records; loader-time augmentation flips.
• image_url, positive h/w, and float32-safe im_id are required.
• Mask records need one raw polygon-list entry per instance.

Bundled helpers

python <skill-root>/sub-skills/data-preparation/scripts/convert_roidb.py --help
python <skill-root>/sub-skills/data-preparation/scripts/validate_roidb.py --help
python <skill-root>/sub-skills/data-preparation/scripts/validate_roidb.py --input data/cache/custom.roidb --check-images --max-records 3

The converter supports --format json|voc|coco|crowdhuman; it requires only its format-specific public dependencies and writes only the explicit output. The validator is read-only and can check JSON/JSONL without NumPy; pickle roidbs require NumPy.

Route onward

• Runtime/compiler/backend issues: setup-and-operations.
• Detector/config/checkpoint execution: detection-workflows.
• Architecture or tensor-contract changes: model-customization.

Read workflows.md for format-specific commands and cache handoff details.