---

name: io-conversion description: "Open, convert, export, and validate local and optional remote tabular data with Vaex." disable-model-invocation: true metadata: disco-role: operating license: MIT

IO Conversion

Use this sub-skill when the task is about getting tabular data into or out of Vaex: opening local files, converting CSV/Arrow/Parquet/Feather/HDF5, exporting DataFrames, checking roundtrips, or diagnosing format-specific failures.

Route first

• For generic DataFrame construction, lazy DataFrame semantics, virtual columns, selections, and basic inspection, route to ../dataframe-core/SKILL.md.
• For overall vaex command discovery, settings, aliases, environment variables, and non-conversion CLI commands, route to ../cli-settings/SKILL.md.
• For expression syntax, filters used in conversion, groupby/binby/aggregation, joins, and analytic validation, route to ../expressions-analytics/SKILL.md.
• Stay here for vaex.open, open_many, from_csv, from_csv_arrow, from_pandas, from_arrow_table, df.export*, df.to_arrow_table, local astro file openers, cloud path caveats, and vaex convert data semantics.

Read order

1. references/io-workflows.md for actionable open/convert/export recipes, API signatures, vaex convert options, and roundtrip checks.
2. references/data-formats.md for format selection, memory-mapping behavior, schema caveats, and Pandas/Arrow/HDF5 handoff details.
3. references/astro-and-cloud-io.md for optional local astro formats and remote/cloud/TAP skip conditions. Do not run network or credentialed checks unless the user explicitly authorizes them.
4. references/troubleshooting.md when vaex.open fails, a plugin is missing, a CSV conversion is memory-heavy, conversion cleanup is surprising, or a file opens in Pandas but not in Vaex.

Safe bundled helpers

• scripts/io_roundtrip_smoke.py creates tiny local data, exercises Vaex export and open roundtrips, and validates Pandas/Arrow table handoffs. It never uses network paths.
• scripts/convert_csv_hdf5.py converts a local CSV to Vaex HDF5 with optional column selection, filtering, sorting, fractioning, chunk size, listing, dry run, and safe partial-output handling. It rejects remote paths.

Run both helpers with --help first. They are safe from arbitrary working directories and use installed public Vaex APIs rather than source-checkout files.

Operating rules

• Prefer memory-mappable Vaex HDF5 or Arrow-family files for repeated large-data work; treat CSV as an ingestion/interchange format and convert when the same data will be queried repeatedly.
• Use vaex.open(path) for lazy file-backed access when the format is supported; use vaex.open(path, convert="data.hdf5") or the bundled conversion helper for large CSVs that need a faster local binary representation.
• Validate every conversion with at least: row count, column names, a small sample or checksum-like aggregate, and the exact group or file pattern used.
• Treat cloud storage, TAP, and credentialed URLs as optional trust-boundary work. Document the plan and skip by default in automated scripts.
• Never assume a Pandas-created HDF5 file is Vaex-readable. Convert through vaex.from_pandas(...).export_hdf5(...) when the source HDF5 layout is row oriented or otherwise Pandas-specific.