---

name: dataset-curation description: "Guide fastdup workflows for local image cleanup, duplicate removal, gallery generation, and binary feature round-trips." metadata: disco-role: operating disable-model-invocation: true license: NOASSERTION

dataset-curation

Use this sub-skill for fastdup's core local-corpus workflows: duplicates, outliers, broken images, components, stats, and disk cleanup.

Use when

The request mentions any of the following:

• cleaning a folder of images
• finding duplicates or near-duplicates
• spotting broken images or bad files
• generating duplicate, outlier, component, similarity, or stats galleries
• removing duplicates from disk
• working with features.dat, features.dat.csv, or binary feature round-trips
• using fd.run(...) or fastdup.run(...) on a plain image folder

Typical workflow

1. Confirm the input is a local folder, file list, or a synthetic fixture.
2. Choose the run path:
   • fastdup.create(...); fd.run(...) for object-oriented use
   • fastdup.run(...) for one-shot use
   • fastdup.remove_duplicates(...) when the goal is cleanup
3. Set the analysis knobs that matter most:
   • num_images for smoke runs
   • threshold for similarity sensitivity
   • lower_threshold for outlier selection
   • lazy_load for large galleries
   • distance / ccthreshold when duplicate cleanup is the real goal
4. Inspect the galleries and CSV outputs.
5. If the workflow needs a tiny deterministic fixture, run the bundled synthetic-image script first.

What to read

• ../../references/api-reference.md for the main API and gallery methods
• ../../references/data-formats.md for file names, output names, and feature-vector layout
• ../../references/workflows.md for the core dataset-curation sequence
• ../../references/troubleshooting.md for package-wide install/import and image-decoding issues
• references/troubleshooting.md in this sub-skill for local cleanup pitfalls

Bundled scripts

Run these when you need a fast smoke check or a reproducible fixture:

• ../../scripts/make_synthetic_image_data.py — create a small image corpus with valid, corrupted, duplicated, and missing rows
• ../../scripts/run_core_analysis_smoke.py — run a full tiny analysis and generate galleries
• ../../scripts/run_feature_vector_smoke.py — verify binary feature save/load round-trips

Common decisions

• Use remove_duplicates only when deleting files from disk is the actual goal.
• Use fastdup.run(..., num_images=...) or fd.run(..., num_images=...) for smoke tests.
• Use show=False in gallery helpers when you only need file generation.
• Keep filenames aligned with the input folder layout; path mismatches are the most common failure mode.
• If a gallery returns no interesting pairs, lower the threshold or use a richer fixture.

Outputs to expect

• similarity.csv
• outliers.csv
• stats.csv
• component_info.csv
• connected_components.csv
• HTML galleries under a galleries/ directory

Known limitation

The workflow assumes the package's core CPU path. GPU hardware is not required for the selected fastdup dataset-curation scope.