---

name: data-preparation description: "Validate and prepare XTuner SFT, MLLM, pretraining, and RL JSONL data, tokenizer/chat-template configuration, caching, packing, and local conversion workflows." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

XTuner data-preparation operating skill

Use this sub-skill when the task is about XTuner V1 dataset records, JSONL validation, tokenizer/chat-template selection, dataset caching, packing, or safe local data conversion before training.

Route here

• Validate OpenAI-style SFT JSONL, including bare message-list lines and object lines with messages or dialogs.
• Validate multimodal SFT/pretraining JSONL with image/video references, media_root, and class_name="VLMJsonlDataset".
• Validate RL/GSM8K-style reward JSONL with prompt, data_source, and reward_model.ground_truth.
• Choose or document DatasetConfig, DataloaderConfig, OpenaiTokenizeFunctionConfig, and FTDPTokenizeFnConfig fields.
• Explain dataset cache behavior (cache_dir, cache_tag) and packing choices (pack_level, preset pack/sampler files).
• Convert local GSM8K question/answer files into XTuner RL JSONL without network access.

Route elsewhere

• Training launch commands, torchrun resources, FSDP/checkpoint/resume, and training logs belong to the training sub-skill.
• RL trainer, rollout, Ray, judger service, advantage, replay buffer, and inference-backend configuration belong to the reinforcement-learning sub-skill.
• Model family, tokenizer/model path validity, dense/MoE/VLM backend, attention, FP8, or optional accelerator choices belong to the model-backends sub-skill.
• Legacy preprocess CLI usage belongs to the cli-and-tools sub-skill unless only schema context is needed here.

Start with the safe local helpers

python ./scripts/validate_xtuner_jsonl.py ./data/openai_sft.jsonl --mode sft
python ./scripts/validate_xtuner_jsonl.py ./data/mllm.jsonl --mode mllm --media-root ./media
python ./scripts/convert_gsm8k_jsonl.py --input-dir ./gsm8k_raw --out-dir ./gsm8k_xtuner
python ./scripts/validate_xtuner_jsonl.py ./gsm8k_xtuner/train.jsonl --mode rl

The bundled scripts use only the Python standard library. They do not import XTuner, a tokenizer, PyTorch, image libraries, or any original repository checkout.

References to load

• references/data-formats.md: SFT, MLLM, pretraining, RL/GSM8K JSONL schemas and config snippets.
• references/tokenization-and-packing.md: chat templates, tokenization configs, cache invalidation, and packing/sampler options.
• references/troubleshooting.md: failure signatures and fixes for JSONL, media, truncation, cache, preset packing, and RL reward fields.

Operating rules

1. Validate JSONL before proposing a training launch.
2. For multimodal data, always resolve local image/video references with an explicit media_root and fail fast on missing files.
3. Treat the validator's --max-length output as an approximation only; real truncation depends on the selected tokenizer and chat template.
4. When pack_level="preset", ensure both pack_config_path and sampler_config_path are set before handing off to training.
5. For RL data, reward_model.ground_truth is required for GSM8K-style rule reward; do not hand off RL data with missing or empty ground truth.