name: shard-writing
description: "Route WebDataset tar-shard creation, rollover, sample encoding,
and local read-back validation."
metadata:
disco-role: operating
disable-model-invocation: true
license: BSD 3-Clause
shard-writing
Use this sub-skill when an agent needs to create WebDataset tar shards, choose shard names or rollover limits, encode common sample values, or verify what was written by reading it back.
Route here for
TarWriter or ShardWriter usage.- Sample dict conventions with
__key__ and extension-suffixed fields.
- Encoder support for
txt, json, pyd, pth, npy, npz, tenbin/tb, images, and *.gz suffixes.
- Shard rollover,
post hooks, and deterministic tiny dataset generation.
- Read-after-write checks using the reader sub-skill or the bundled smoke script.
Route elsewhere for
- Loading, decoding, batching, mixing, or consuming shards -> reading-pipelines.
pipe: URLs, stream/open behavior, cache rules, or security posture -> io-caching-security.- Full Transformers, Ray, OCR, or cloud upload execution -> reference-only; keep only the local write loop here.
Working rule
- Decide whether the task needs a single tar stream (
TarWriter) or a rollover writer (ShardWriter).
- Keep sample keys stable and extension-driven; use
encoder=True unless the payloads are already bytes or a custom encoder is required.
- Prefer fixed metadata such as
mtime=0 when reproducibility matters.
- If the source recipe came from a notebook, keep only the stable write or transform loop and move it into the bundled reference or helper script.
- Validate by reading the written shard(s) back locally before handing off.
- If the task depends on pipe URLs, remote streams, or gopen behavior, stop and hand off to the security sub-skill instead of expanding that behavior here.
Bundled assets