---

name: expressions-analytics description: "Use Vaex expressions, virtual columns, selections, statistics, groupby/binby grids, joins, sorting, accessors, and analytic validation." disable-model-invocation: true metadata: disco-role: operating license: MIT

expressions-analytics

Use this sub-skill when the task is to transform, aggregate, validate, or join data that is already represented as a Vaex DataFrame. Keep the DataFrame lazy where possible: express calculations as Vaex expressions or virtual columns, then ask Vaex for statistics, grouped results, grids, or a bounded evaluated slice.

Route first

• DataFrame construction, basic lazy semantics, and minimal inspection: ../dataframe-core/SKILL.md.
• File import/export/conversion or materializing a persisted dataset: ../io-conversion/SKILL.md.
• Plotting histograms, heatmaps, scatter plots, or Jupyter widgets from analytic grids: ../visualization-jupyter/SKILL.md.
• ML feature pipelines, encoders, scalers, predictors, and pipeline persistence: ../ml-pipelines/SKILL.md.

Operating rules

1. Prefer Vaex expressions over eager arrays: df.x + df.y, df['column with spaces'], and NumPy ufuncs on expressions stay lazy.
2. Add reusable derived data as virtual columns with df['new'] = expr or df.add_virtual_column('new', expr); do not call .values, .to_numpy(), np.array(df), or to_pandas_df() unless the requested result is known to fit in memory.
3. Validate expressions on a slice before full execution: df.evaluate(expr, i1=0, i2=min(5, len(df))) or expr.evaluate(i1=0, i2=5).
4. Use selections for repeated subset statistics and filtered DataFrames for Pandas-like narrowing. Remember that joins ignore filters unless a filtered DataFrame is first extracted.
5. For aggregations, use Vaex statistics (mean, sum, std, count, minmax, correlation, percentile_approx, mutual_information) and df.groupby(..., agg=...)/vaex.agg rather than materializing intermediate arrays.
6. For high-cardinality groupby or join keys, check cardinality (unique, value_counts, row_limit) and missing-value semantics before running a large job.
7. Joins default to left joins and build lookup/index structures rather than copying the whole right table. Diagnose duplicate right keys before using allow_duplication=True.

Reference map

• references/expressions-and-selections.md: expression objects, virtual columns, selections/filters, bounded evaluation, string/datetime/struct/geo accessors, and derived-column recipes.
• references/analytics-reference.md: statistics, groupby/binby grids, joins, sorting, API signatures, and analytic validation patterns.
• references/troubleshooting.md: expression parse/name errors, non-identifier columns, missing/NaN/category groups, materialization surprises, join cardinality, grid shape/limit issues, and accessor failures.

Bundled check

Run a tiny installed-package smoke check after editing this sub-skill or when diagnosing an environment:

python scripts/analytics_smoke.py --help
python scripts/analytics_smoke.py

The script creates in-memory DataFrames only, uses public Vaex APIs, and asserts expression, virtual-column, selection, statistic, groupby, binby, join, sorting, and accessor behavior.