---

name: data-preparation description: "Prepare, validate, and visualize HRM ARC, Sudoku, and Maze puzzle datasets in the repository's shared NumPy layout." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

HRM Data Preparation

Use this sub-skill when the task is to create, inspect, validate, or visualize Hierarchical Reasoning Model (HRM) datasets for ARC, Sudoku, or Maze workflows. The original repo builders produce a shared directory layout consumed by PuzzleDataset and by pretrain.py/evaluate.py.

When to use

• The user asks to build ARC-AGI, ARC-AGI-2, ConceptARC, Sudoku Extreme, or Maze datasets for HRM.
• The task names dataset/build_arc_dataset.py, build_sudoku_dataset.py, build_maze_dataset.py, PuzzleDataset, PuzzleDatasetMetadata, identifiers.json, or all__inputs.npy style files.
• A training or evaluation run fails before model execution because dataset paths, metadata, .npy arrays, split/subset names, token ranges, or puzzle index arrays are wrong.
• The user wants a browser visualizer for converted puzzle folders.

Route map

1. Read references/data-formats.md for the exact shared dataset directory structure, metadata fields, array meanings, and validation invariants.
2. Read references/workflows.md for ARC, Sudoku, Maze, and visualization command recipes, including network/submodule prerequisites and safe bounded variants.
3. Use references/troubleshooting.md when a builder cannot find raw data, Hugging Face downloads fail, token ranges look invalid, or PuzzleDataset yields no batches.
4. Run scripts/validate_dataset_layout.py against a converted dataset before long training or evaluation.
5. Use scripts/run_dataset_builder.py when a future task needs a wrapper that forwards arguments to an HRM checkout from any current working directory.
6. Open scripts/puzzle_visualizer.html in a browser for local inspection of generated puzzle folders. The required npyjs asset is bundled under scripts/assets/.

Core workflow

# ARC-1 from initialized ARC-AGI + ConceptARC submodules
python dataset/build_arc_dataset.py

# ARC-2 from ARC-AGI-2 raw data
python dataset/build_arc_dataset.py \
  --dataset-dirs dataset/raw-data/ARC-AGI-2/data \
  --output-dir data/arc-2-aug-1000

# Sudoku 1K augmented demo
python dataset/build_sudoku_dataset.py \
  --output-dir data/sudoku-extreme-1k-aug-1000 \
  --subsample-size 1000 \
  --num-aug 1000

# Maze 30x30 hard 1K
python dataset/build_maze_dataset.py --output-dir data/maze-30x30-hard-1k

After building, validate the shared layout before training:

python <skill>/sub-skills/data-preparation/scripts/validate_dataset_layout.py \
  data/sudoku-extreme-1k-aug-1000 --splits train test

Replace <skill> with the root of this generated skill directory. The helper is self-contained; it does not need the original HRM checkout unless you are running the original builder scripts.

Boundaries

• This sub-skill owns dataset builders, converted layout, schema validation, and local visualization.
• Use model-architecture for HRM ACT v1 config fields, losses, model class routing, FlashAttention layers, and sparse puzzle embeddings.
• Use training-evaluation for pretrain.py, evaluate.py, checkpoints, W&B, CUDA training, and ARC prediction post-processing.
• Do not run full dataset downloads or raw-data submodule cloning unless the user has accepted network and storage costs. Use tiny fixtures and the bundled validator for bounded checks.

Validation expectations

• Builder --help commands are safe and should list the documented options.
• Full Sudoku and Maze conversion require network access to Hugging Face datasets.
• Full ARC conversion requires the ARC/ConceptARC raw-data submodules or an equivalent local directory.
• Converted train/test split directories must contain dataset.json plus <subset>__inputs.npy, <subset>__labels.npy, <subset>__puzzle_identifiers.npy, <subset>__puzzle_indices.npy, and <subset>__group_indices.npy for each subset in metadata sets.