---

name: data-preparation description: "Prepare and validate WeNet data manifests, dictionaries, tokenizers, CMVN inputs, and raw-versus-shard dataset layouts before training or decoding." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

WeNet Data Preparation

Use this sub-skill when the task is to convert speech data into WeNet training or evaluation inputs, validate wav.scp/text, create data.list, build a character dictionary, choose raw versus shard data, or debug tokenizer/CMVN setup.

Start here

1. Read references/data-formats.md to confirm the expected wav.scp, text, data.list, dictionary, and tokenizer shapes.

2. Use the bundled raw-manifest helper for small or medium custom datasets:

   python sub-skills/data-preparation/scripts/make_wenet_raw_manifest.py \
     --wav-scp wav.scp --text text --output data.list
   

3. Use the bundled dictionary helper for character-token recipes:

   python sub-skills/data-preparation/scripts/make_wenet_dict.py \
     --text text --output units.txt
   

4. Read references/workflows.md before planning CMVN, shard data, tokenizer selection, or recipe-stage conversion.

5. Read references/troubleshooting.md when dataset loading filters everything, keys mismatch, audio cannot be decoded, dictionaries have wrong reserved IDs, or tokenizer files cannot be found.

Route by task

• Single-audio package transcription belongs in ../package-transcription/SKILL.md.
• Training, checkpoint averaging, recognition, and WER/CER scoring belong in ../training-and-decoding/SKILL.md.
• Exporting a trained checkpoint belongs in ../model-export/SKILL.md.

Key decisions

• Choose raw data for ordinary small/medium datasets and custom debugging. Choose shard only when the dataset is large enough that tar shards improve throughput and the user can tolerate extra packaging steps.
• Use absolute or stable audio paths in wav.scp and data.list. Relative paths often fail when training is launched from a different directory.
• For Mandarin character recipes, dictionary files reserve <blank> 0, <unk> 1, and <sos/eos> as the final or recipe-defined special token. Keep the active training config and tokenizer initialization consistent.
• Treat CMVN computation and feature extraction as data-prep stages, but do not run large audio scans unless the user has approved the data path, runtime, and storage cost.