---

name: datasets-configs description: "Understand SaProt LMDB datasets, JSONL-to-LMDB conversion, task YAML paths, and safe config validation." disable-model-invocation: true metadata: disco-role: operating license: MIT

SaProt Datasets and Configs

Use this sub-skill when the task is about SaProt LMDB layout, JSON row fields, dataset initialization paths, tokenizer/model asset locations, or editing a task YAML before a training or evaluation run.

Quick Routing

• Convert JSONL rows into a SaProt-style LMDB with scripts/jsonl_to_lmdb.py; read references/data-formats.md first for the row schema.
• Validate a YAML task config without importing PyTorch, Transformers, or Lightning with scripts/validate_config.py; read references/configuration.md for path rules.
• Route full training, evaluation, Lightning launch behavior, WandB logging, checkpoint selection, and expensive benchmark execution to training-evaluation.
• Route model loading, tokenization behavior, embeddings, mutation scoring, and forward-pass details to model-inference.
• Route PDB/mmCIF-to-SaProt sequence conversion, Foldseek setup, chain selection, and pLDDT masking from structures to structure-sequences.

Core Facts

• LMDBDataset(train_lmdb=None, valid_lmdb=None, test_lmdb=None, dataloader_kwargs=None) opens one LMDB directory per stage and expects an integer-like length key plus numeric row keys from 0 to length - 1.
• DataInterface.init_dataset(dataset_py_path, **kwargs) dynamically imports a dataset module path such as saprot/saprot_regression_dataset; do not include the leading dataset/ directory or the .py suffix in YAML.
• SaProt config paths are conventionally resolved from the run base: LMDB data under LMDB/, Hugging Face tokenizer/model directories under weights/PLMs/, model modules under model/, and dataset modules under dataset/.
• LMDB row values are UTF-8 JSON object strings; each dataset class expects different fields, so validate JSONL with the matching --dataset-type before writing a database.
• Keep data/config checks separate from execution: a config can be syntactically valid while still requiring model weights, LMDB assets, compatible GPUs, and the training stack before a real run.

Safe Workflows

JSONL to LMDB

1. Choose the target dataset schema from references/data-formats.md.
2. Validate and write rows with the bundled converter: python scripts/jsonl_to_lmdb.py --jsonl rows.jsonl --lmdb-dir LMDB/MyTask/foldseek/train --dataset-type regression --map-size-gb 10240
3. Confirm the converter reports info, length, and numeric keys.
4. Repeat for train, valid, and test directories if the YAML will use all three stages.

YAML validation

1. Edit paths relative to the intended SaProt run base, not relative to the config file location unless those are the same.
2. Check module paths, asset paths, LMDB keys, and Trainer backend choices: python scripts/validate_config.py --config task.yaml --base-dir .
3. Add --require-assets when the machine is expected to already contain LMDB and model/tokenizer directories.
4. Do not launch training until validation errors are resolved and remaining warnings are intentionally accepted.

Reference Map

• references/data-formats.md: LMDB key layout, dataset constructor behavior, JSON row fields, and converter usage.
• references/configuration.md: YAML section meanings, module path resolution, asset placement, CPU smoke-test adaptation, and backend validation.
• references/troubleshooting.md: common LMDB, JSON field, tokenizer path, pLDDT, map-size, relative path, and GPU-device failures.