---

name: data-preparation description: "Prepare and validate MatterGen crystal CSV data, released MP-20 or Alex-MP-20 archives, cache splits, and custom property labels before training, fine-tuning, or evaluation." disable-model-invocation: true metadata: disco-role: operating license: MIT

MatterGen data preparation

Use this sub-skill when a MatterGen workflow needs a released MP-20 or Alex-MP-20 dataset, CSV-to-cache preprocessing, a custom property column, or a preflight before training/fine-tuning/evaluation. The parent MatterGen skill should route here before invoking a trainer or property adapter.

This is a preparation and validation route, not a downloader, data scientist, or model-quality evaluator. It does not silently fetch LFS objects, unzip archives, convert data, repair malformed CIFs, invent columns, or import unverified caches.

Fast route

1. Identify the dataset source, version, split names, property labels, storage location, and license/provenance record. Keep the CSV folder limited to the splits intended for this run.
2. Check available storage before downloading or unzipping a large release. Alex-MP-20 preprocessing is documented as taking about an hour and may need archive, extracted CSV, and cache space at the same time.
3. Run the safe, read-only CSV preflight script over the complete CSV folder. Add one --property NAME for every selected label. Use --limit-rows only for a preliminary probe, never as the final gate.
4. Resolve every ERROR, review WARN output for sparse labels, and rerun the full preflight. A pass means the package converter can be attempted; it does not prove scientific correctness or label comparability.
5. Run the actual csv-to-dataset package CLI with all three required options. It processes every .csv in the folder and writes one cache directory per filename stem. It is not the same operation as the bundled validator.
6. Inspect the generated core arrays and property JSON files using the cache check in data formats. Use a fresh cache root after interruption or schema changes.
7. Point the selected data-module config at the verified dataset root, then hand off to training/fine-tuning/evaluation with the provenance and unresolved limits recorded.

Exact input and output contract

The converter consumes CSV rows with exact lowercase cif and material_id columns. cif is parsed into a primitive pymatgen structure; material_id becomes the structure identifier. The converter copies only exact property IDs registered by the installed MatterGen package. Similar names such as band_gap, e_above_hull, or spacegroup.number are not silently mapped.

For a folder containing ordinary train.csv, val.csv, and test.csv, the logical output is:

<CACHE_FOLDER>/<DATASET_NAME>/train/
<CACHE_FOLDER>/<DATASET_NAME>/val/
<CACHE_FOLDER>/<DATASET_NAME>/test/

The required core cache files are pos.npy, cell.npy, atomic_numbers.npy, num_atoms.npy, and structure_id.npy. Registered properties are written as <property>.json. See data formats for array semantics, split expectations, recognized property IDs, and a read-only cache validation command.

The package data configs use these split contracts:

• mp_20: train, val, and test; default properties are empty and common supported labels are documented in its data-module config.
• alex_mp_20: train and val; default properties are empty and the config also documents ml_bulk_modulus, hhi_score, space_group, and energy_above_hull.

Both configs apply lattice symmetrization and set a chemical-system string as a sample transform. Both apply filter_sparse_properties as a dataset transform; when properties are selected, rows missing any selected value are filtered.

Released data route

MP-20 and Alex-MP-20 are separate training releases. Download and unzip only with explicit user approval. The documented commands and storage-aware route are in released-data workflows. Use a user-supplied release archive or hydrated download; the skill does not assume a particular checkout layout. In summary, the normal sequence is:

# Explicitly acquire and verify <MP20_ARCHIVE>.zip first.
unzip <MP20_ARCHIVE>.zip -d <DATA_ROOT>
python <mattergen-skill-root>/sub-skills/data-preparation/scripts/validate_dataset_csv.py \
  --csv-folder <DATA_ROOT>/mp_20
csv-to-dataset --csv-folder <DATA_ROOT>/mp_20 \
  --dataset-name mp_20 --cache-folder <CACHE_ROOT>

For Alex-MP-20, substitute the user-supplied archive and extracted folder:

unzip <ALEX_MP20_ARCHIVE>.zip -d <DATA_ROOT>
python <mattergen-skill-root>/sub-skills/data-preparation/scripts/validate_dataset_csv.py \
  --csv-folder <DATA_ROOT>/alex_mp_20
csv-to-dataset --csv-folder <DATA_ROOT>/alex_mp_20 \
  --dataset-name alex_mp_20 --cache-folder <CACHE_ROOT>

If the archive is stored in Git LFS, hydrate only the explicitly requested release object using the user's checkout-aware command, then verify that the result is a real ZIP rather than an LFS pointer before extraction.

Run the bundled validator between extraction and conversion. An LFS pointer is not a ZIP. Do not download the Alex-MP reference energy archives, presentation CIF archive, or measurement files for this training-data route.

Custom properties and split discipline

For an existing property, pass its exact registry name to the validator and place the same column in every relevant CSV split. Sparse labels may be intentional, but an entirely blank selected column or a missing column in one split is a blocking preparation problem.

For a new property, register its exact name in MatterGen's property-source registry before conversion, add the column, rebuild the cache, add a compatible property-embedding configuration, and select it in the data module. A column that merely passes CSV parsing will otherwise be ignored by the converter. Read custom properties for the supported value/provenance rules and fine-tuning handoff.

Storage, config, and handoff

Before Alex-MP-20, use df -h and du -sh on the archive, extracted CSV, and intended cache locations. If disk is constrained, place --cache-folder on a larger filesystem and override data_module.root_dir to the resulting <CACHE_FOLDER>/<DATASET_NAME> before training. Do not launch a full conversion merely to discover that the cache destination is too small.

The package CLI is CPU-oriented for parsing, although its import stack includes PyTorch and PyTorch Geometric. GPU/backend compatibility matters later for training and diffusion data utilities, not for the safe preflight itself. Use csv-to-dataset --help and the troubleshooting guide for import, optional-backend, LFS, CLI, config, cache, and workflow failures.

Record the dataset release/source version, citations and CC BY 4.0 obligations, repository preprocessing modifications, property units and calculation method, split/ID policy, command parameters, cache location, and validation result outside this runtime tree. The code is MIT-licensed, but that does not replace the separate MP-20/Alexandria dataset terms.

Verification gates

A data-preparation handoff is ready only when:

• csv-to-dataset --help succeeds in the intended environment;
• the full validator exits zero, or all warnings are explicitly accepted;
• every intended split has the five core arrays and matching structure/atom counts;
• every selected property JSON exists and has one value per structure;
• the configured data-module split paths resolve to those caches; and
• provenance, licensing, storage decisions, and unresolved scientific limits are recorded.

If conversion is interrupted, a required dependency is broken, a CIF cannot be parsed, a selected property schema differs across splits, or a cache check fails, stop and route through troubleshooting. Do not claim that training or fine-tuning is ready.