---

name: core-dataframes description: "Create and manipulate Koalas DataFrame, Series, and Index objects with pandas-like APIs and conversion guidance." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Koalas core DataFrames router

Use this sub-skill when the task is about creating, converting, indexing, reshaping, cleaning, typing, or summarizing Koalas DataFrame, Series, and Index objects with pandas-like APIs.

Handle here

• Constructors and conversion: databricks.koalas as ks, ks.DataFrame, ks.Series, ks.Index, ks.from_pandas, ks.range, to_pandas, and to_spark(index_col=...) when the focus is preserving or avoiding a default index.
• Top-level dataframe functions: ks.concat, ks.merge, ks.melt, ks.get_dummies, ks.to_datetime, ks.date_range, ks.isna / isnull / notna / notnull, and ks.to_numeric.
• Selection and indexing: df[...], attribute column access when safe, loc, iloc, at, iat, head, tail, where, mask, query, set_index, and reset_index.
• Core object APIs: dtype inspection and astype, missing-data operations, string accessor Series.str, datetime accessor Series.dt, categorical accessor Series.cat, index and MultiIndex basics, and descriptive statistics.
• Pandas migration decisions: when a pandas snippet can be translated directly, when duplicate/case-sensitive/reserved columns must be fixed, and when to_pandas() is acceptable only for bounded data.

Route elsewhere

• Spark sessions, storage formats, Spark SQL, readers/writers, .spark accessor depth, JDBC, and production Spark IO: spark-io-sql.
• apply, transform, map_in_pandas, groupby, rolling, expanding, window, and return type-hint depth: apply-groupby-window.
• Koalas options, plotting, custom accessors/extensions, optional dependencies, and global performance switches: configuration-extensions.

Operating references

1. Start with API reference for supported constructors, methods, signatures, and known API gaps.
2. Use workflows for concrete migration, creation, conversion, indexing, reshape, dtype, missing, string, datetime, categorical, and statistics recipes.
3. Use troubleshooting for unsupported pandas APIs, duplicate/case-sensitive columns, reserved __column__-style names, pandas collection risks, non-iterability, default-index cost, and dtype gotchas.
4. For a safe runtime probe, run scripts/koalas_dataframe_quickstart.py with --check import, --check dataframe, or --check all.

Default decision rules

• Keep data in Koalas/Spark for distributed work; use to_pandas() only after a bounded head, sample, filter, or other proof that the result fits driver memory.
• Prefer ks.from_pandas(pdf) for migration tests and tiny fixtures. For Spark interop, preserve an index with to_spark(index_col="...") and sdf.to_koalas(index_col="...") when an existing column can serve as the index.
• Reject or rename duplicate columns, case-only duplicate columns such as a/A, and names wrapped in double underscores before relying on selection or Spark conversion.
• For pandas code that iterates over a Series or expects local arrays, rewrite to Koalas vectorized APIs or explicitly collect with to_numpy() / to_pandas() after bounding data size.