---

name: data-pipeline description: "Prepare, validate, preprocess, filter, standardize, and deduplicate SMILES datasets with REINVENT4 reinvent_datapre." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Data Pipeline

Use this sub-skill when a task mentions data_pipeline.toml, reinvent_datapre, SMILES preprocessing, CSV-to-SMI conversion, token filtering, allowed elements, RDKit standardization, transform_file, deduplication, or cleaning a dataset before sampling, scoring, transfer learning, staged learning, or enumeration.

Read First

• references/data-formats.md for accepted input files, required config keys, CLI invocation, output files, and helper usage.
• references/preprocessing-workflow.md for a safe build-validate-run-scale workflow and what each regex/RDKit filter does.
• references/troubleshooting.md for invalid SMILES, unexpected elements, bad delimiters/headers, transform-file mistakes, output writes, multiprocessing, and dependency triage.
• scripts/validate_data_pipeline_config.py to parse a TOML config, verify input/output fields and filter structure, and sample-check the selected SMILES column without running reinvent_datapre.
• scripts/make_tiny_smiles_fixture.py to generate a small CSV/SMI fixture plus matching data_pipeline.toml for smoke checks.

Fast Path

1. Create a TOML config with input_csv_file, smiles_column, separator, output_smiles_file, num_procs, chunk_size, and a [filter] table.
2. Use a headered CSV/TSV for tabular data, or .smi/.smi.gz for headerless SMILES input. For .smi, smiles_column names the synthetic column used internally.
3. Validate before preprocessing:

   python sub-skills/data-pipeline/scripts/validate_data_pipeline_config.py data_pipeline.toml --sample-rows 20
   

4. Run a small preprocessing smoke check:

   reinvent_datapre data_pipeline.toml --log-filename data_pipeline.log
   

5. Inspect output_smiles_file, the log, and the side-effect regex.smi in the working directory before scaling to a large dataset.

Scope Boundaries

• This sub-skill owns reinvent_datapre, data_pipeline.toml, CSV/SMI inputs, smiles_column, separator, regex token filtering, allowed elements, RDKit cleanup, standardization transforms, deduplication, output .smi, and tiny-fixture validation.
• For molecule generation after preprocessing, use the sampling sub-skill.
• For scoring preprocessed molecules, use the scoring sub-skill.
• For transfer learning or staged reinforcement learning on cleaned data, use the learning sub-skill.
• For peptide or molecule enumeration, use the enumeration sub-skill.