---

name: data-storage description: "Use ClearML Dataset, StorageManager, clearml-data CLI, storage backends, cache behavior, and HyperDataset/DataView routing for dataset versioning and retrieval." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

ClearML Data and Storage

Use this sub-skill when the user needs to create, version, upload, finalize, retrieve, verify, or troubleshoot ClearML datasets, build clearml-data commands, manage remote storage URIs, or reason about StorageManager caching and HyperDataset/DataView routing.

Route First

• Use this sub-skill for Dataset, StorageManager, clearml-data, dataset parents/versions, local copies, external links, storage credentials/extras, cache limits, and HyperDataset/DataView selection.
• Route experiment task setup, scalar/image reporting, and non-dataset Task.upload_artifact() usage to ../experiment-tracking/SKILL.md.
• Route queued task execution, agents, clearml-task, Task.execute_remotely(), or remote job launch to ../remote-execution-cli/SKILL.md.
• Route pipelines that consume datasets as pipeline inputs or steps to ../automation-pipelines/SKILL.md.

Core Dataset Pattern

Use a two-stage mental model: a dataset version is mutable until finalized, and after finalization it should be retrieved or used as a parent for a new version.

from clearml import Dataset

base = Dataset.create(
    dataset_project="datasets/examples",
    dataset_name="images",
    dataset_version="1.0.0",
    output_uri="s3://bucket/clearml/images",
)
base.add_files(path="data/train", dataset_path="train")
base.upload(output_url="s3://bucket/clearml/images", verbose=True)
base.finalize()

child = Dataset.create(
    dataset_project="datasets/examples",
    dataset_name="images",
    dataset_version="1.1.0",
    parent_datasets=[base.id],
)
child.sync_folder(local_path="data/train", dataset_path="train", verbose=True)
child.upload(verbose=True)
child.finalize()

Prefer explicit IDs after creation. Dataset.get() can search by project/name/version, but dataset_id avoids ambiguity when multiple semantic versions or recent updates match.

CLI Planning

Use clearml-data when the user wants shell commands or quick dataset lifecycle operations. The CLI stores the most recent dataset ID in ~/.clearml_data.json; commands without --id may silently target the last accessed dataset, so production instructions should include --id after create unless the user intentionally wants one-shell stateful use.

clearml-data create --project datasets/examples --name images --version 1.0.0 --storage s3://bucket/clearml/images
clearml-data add --id DATASET_ID --dataset-folder train --files data/train
clearml-data upload --id DATASET_ID --storage s3://bucket/clearml/images --verbose
clearml-data close --id DATASET_ID --verbose

Use scripts/dataset_cli_plan.py to generate safe command sequences without running ClearML calls. Use scripts/validate_dataset_plan.py to check JSON plan invariants before translating a plan to Python or CLI.

Storage Guidance

• Dataset.create(..., output_uri=...), Dataset.upload(output_url=...), and CLI --storage accept s3://..., gs://..., azure://..., a shared filesystem path, or the default ClearML files server.
• Cloud storage support requires the matching package extras: install ClearML with s3, gs, or azure extras when those schemes are used.
• Credentials are read from ClearML configuration and provider defaults; do not put secrets in generated commands, scripts, dataset names, or skill content.
• Use StorageManager.get_local_copy() for remote file caching/downloading and StorageManager.upload_file() or upload_folder() for direct storage operations that are not dataset-versioning workflows.

References

• references/api-reference.md documents Dataset, StorageManager, and HyperDataset/DataView signatures and behavior notes.
• references/cli-reference.md documents clearml-data subcommands, required flags, state behavior, and safe command construction.
• references/workflows.md gives Python and CLI workflows for create/add/sync/upload/finalize/get/parents/external links.
• references/troubleshooting.md covers credentials, storage extras, stale CLI state, missing IDs, pending uploads, parent mistakes, and cache/local-copy pitfalls.
• scripts/dataset_cli_plan.py builds offline clearml-data command plans.
• scripts/validate_dataset_plan.py validates JSON dataset plan invariants with the Python standard library only.