---

name: data-preparation description: "Guides pix2tex formula extraction, tokenizer and dataset pickle creation, LaTeX rendering, scraping boundaries, and data-format troubleshooting." disable-model-invocation: true metadata: disco-role: operating license: MIT

Data Preparation

Use this sub-skill when the user needs to prepare image/LaTeX pairs for LaTeX-OCR, extract formulas from TeX-like documents, create a tokenizer, build Im2LatexDataset pickle files, render formulas to PNGs, or plan safe data acquisition. Use this before training and evaluation.

Quick Route

1. Read references/data-formats.md for equation text files, image naming, tokenizer JSON, and dataset pickle expectations.
2. Read references/rendering-and-preprocessing.md before rendering LaTeX to PNG or normalizing formulas.
3. Read references/data-acquisition.md before web, Wikipedia, StackExchange, arXiv, or Google Drive acquisition.
4. Read references/troubleshooting.md for missing system tools, bad formulas, naming mismatch, and import quirks.
5. Use scripts/extract_math_snippets.py for safe local formula extraction, and scripts/check_dataset_inputs.py before running the package dataset builder.

Typical Dataset Build Flow

# Validate inputs first; this does not create a pickle.
python scripts/check_dataset_inputs.py --equations math.txt --images images/

# Then build the package dataset pickle when the validation looks right.
python -m pix2tex.dataset.dataset \
  --equations math.txt \
  --images images/ \
  --tokenizer tokenizer.json \
  --out dataset.pkl

To train a new tokenizer from formulas only:

python -m pix2tex.dataset.dataset --equations math.txt --vocab-size 8000 --out tokenizer.json

Boundaries

• Training/evaluation commands belong in ../training-and-evaluation/SKILL.md.
• OCR inference and image capture belong in ../ocr-inference/SKILL.md.
• Network scraping, large downloads, TeX rendering, and dataset pickle creation can be long or system-dependent; confirm the user's budget and tools before running them.