---

name: data-preparation description: "Routes OpenNMT-py corpus YAML, vocabulary building, transforms, tokenizers, source features, and data preparation failure recovery." disable-model-invocation: true metadata: disco-role: operating license: MIT

Data preparation

Use this sub-skill when an OpenNMT-py task is about shaping corpora before training or inference: corpus YAML, onmt_build_vocab, transform lists, tokenizer models, source features, alignment-aware data, and data/config validation failures.

Use this route when

• The user asks to create, validate, or repair a YAML config with data, path_src, path_tgt, path_txt, path_align, src_vocab, tgt_vocab, save_data, n_sample, or share_vocab.
• The user needs to build vocabularies with onmt_build_vocab or understand the plain-text vocab files that training later consumes.
• The request mentions transforms such as filtertoolong, prefix, suffix, clean, normalize, inferfeats, onmt_tokenize, sentencepiece, bpe, bart, switchout, tokendrop, tokenmask, or insert_mask_before_placeholder.
• The user is debugging source features, feature defaults, subword regularization, tokenizer model paths, missing target files, corpus weights, or alignment file compatibility.

Do not use this route when

• The primary task is model architecture, optimizer schedules, checkpoint resume, LoRA, embeddings, or GPU training orchestration; route that to ../training/.
• The primary task is translation output, scoring, REST serving, dynamic inference, or CTranslate2 runtime selection; route that to ../inference/.
• The primary task is checkpoint conversion, checkpoint release, model averaging, or external model import; route that to ../conversion/.

Start here

1. Read references/data-formats.md for the self-contained corpus YAML schema, vocabulary fields, transform ownership, tokenizer options, source-feature rules, and examples.

2. Run the bundled validator before launching an expensive build or training job:

   python scripts/validate_data_config.py --config CONFIG.yaml --root DATA_ROOT --mode build-vocab
   

   Use --mode train when checking that already-built vocab files exist for onmt_train; use --mode generic for a partial data-only YAML.

3. Build vocabularies only after the config is clean enough for the intended mode:

   onmt_build_vocab -config CONFIG.yaml -n_sample 5000
   

4. If n_src_feats is greater than zero, confirm every corpus has inferfeats in transforms and that src_feats_defaults has exactly one ￨-separated value per source feature.

5. If alignment learning is involved, validate every path_align and avoid on-the-fly tokenization or token-deleting transforms in the aligned training config.

What this sub-skill owns

• Corpus YAML structure: top-level data mapping, train corpus ids, valid, file path fields, weights, source-only language-model layouts, and blockwise path_txt data.
• Vocabulary preparation: src_vocab, tgt_vocab, share_vocab, save_data, overwrite, n_sample, dump_samples, learn_subwords, and feature vocab side outputs.
• Transform and tokenizer preparation: per-corpus transform lists, global default transforms, subword model paths, pyonmttok kwargs, regularization knobs, transform order, and transforms that require existing vocabularies.
• Source feature preparation: feature delimiter, n_src_feats, src_feats_defaults, inferfeats, and common mixed-feature-line failures.
• Data-preparation troubleshooting: YAML shape errors, relative path confusion, missing optional tokenizer dependencies, incompatible alignments, invalid vocab modes, and config-vs-command option placement.

Bundled helper

scripts/validate_data_config.py is a self-contained validator. It parses YAML with PyYAML, checks OpenNMT-py data mappings and paths, validates vocab fields for build or train modes, checks feature defaults and sampled feature annotations, warns about transform/tokenizer pitfalls, and never imports the source checkout.

Useful invocations:

python scripts/validate_data_config.py --config CONFIG.yaml --root . --mode generic
python scripts/validate_data_config.py --config CONFIG.yaml --root . --mode build-vocab --strict
python scripts/validate_data_config.py --config CONFIG.yaml --root . --mode train --sample-lines 50

Troubleshooting entry point

If onmt_build_vocab or onmt_train fails before model computation starts, read references/troubleshooting.md first. Most failures are config shape, relative paths, missing vocab fields, source-feature count mismatches, tokenizer model paths, or transforms that are incompatible with alignments or build-vocab mode.