界面已本地化;技能介绍与说明文档按所选语言由 AI 自动翻译,代码、命令、路径和专业术语保留原文,以原作者文档为准。
探索技能
数据与研究 · SKILL.md

kedro

kedro

【数据与研究】Use Kedro to create, configure, run, extend, inspect, and troubleshoot production-ready data and analytics pipeline projects.

独立 GitHub 快照 · 自动分类 · 以原作者文档为准

技能说明

GitHub

name: kedro description: "Use Kedro to create, configure, run, extend, inspect, and troubleshoot production-ready data and analytics pipeline projects." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Kedro

Use this repo skill when the task involves Kedro data or analytics pipelines: node(), Pipeline, DataCatalog, OmegaConfigLoader, kedro run, project scaffolding, KedroSession, hooks/plugins, runners, inspection snapshots, or the optional server API.

Kedro is a Python framework for production-ready data engineering and data science pipelines. The source evidence for this skill targets Kedro 1.4.0, whose distribution and import name are both kedro and whose console script is kedro.

Quick Checks

  • Install/import check: python -c "import kedro; print(kedro.__version__)".
  • CLI check without telemetry: KEDRO_DISABLE_TELEMETRY=1 kedro --version and KEDRO_DISABLE_TELEMETRY=1 kedro --help.
  • Environment diagnostic: run scripts/check_kedro_environment.py --help, then scripts/check_kedro_environment.py --check-cli.
  • Core package only includes Kedro framework primitives; concrete datasets such as pandas.CSVDataset usually require kedro-datasets and relevant extras.
  • Optional server workflows need the server extra or equivalent fastapi/uvicorn dependencies.

Route by Task

  • Use sub-skills/pipelines-and-nodes/SKILL.md for graph authoring with node(), Node, Pipeline, pipeline(), namespaces, tags, modular pipelines, slicing semantics, preview payloads, and graph validation.
  • Use sub-skills/data-catalog-and-config/SKILL.md for DataCatalog, dataset YAML, credentials, versioning, catalog factories, OmegaConfigLoader, parameters, globals, and config merge behavior.
  • Use sub-skills/project-cli-and-sessions/SKILL.md for kedro new, starters, project layout, global/project CLI command availability, project metadata, settings, bootstrap_project(), configure_project(), KedroSession, packaging, IPython, and Jupyter.
  • Use sub-skills/runners-and-execution/SKILL.md for kedro run, KedroSession.run(), SequentialRunner, ThreadRunner, ParallelRunner, async I/O, run slicing, load versions, runtime params, and missing-output resume.
  • Use sub-skills/hooks-and-extensions/SKILL.md for project hooks, plugin entry points, custom CLI commands, custom datasets, custom runners, notebook extension points, serving patterns, and deployment integrations.
  • Use sub-skills/inspection-and-server/SKILL.md for read-only project snapshots, kedro.inspection.get_project_snapshot, snapshot model interpretation, optional HTTP server endpoints, and server dependency/startup troubleshooting.

Current API Facts

  • Import pipeline APIs from kedro.pipeline: node, Node, Pipeline, pipeline, GroupedNodes, and optional preview/LLM helpers.
  • Use kedro.pipeline.pipeline(...) for reusable namespaced pipelines; do not use stale kedro.pipeline.modular_pipeline imports.
  • Construct catalogs with DataCatalog(...) or DataCatalog.from_config(catalog, credentials=None, load_versions=None, save_version=None).
  • Load project config with OmegaConfigLoader(conf_source, env=None, runtime_params=None, config_patterns=None, base_env=None, default_run_env=None, custom_resolvers=None, merge_strategy=None, ignore_hidden=True).
  • Create sessions with KedroSession.create(project_path=None, save_on_close=True, env=None, runtime_params=None, conf_source=None).
  • Built-in runners are SequentialRunner(is_async=False), ThreadRunner(max_workers=None, is_async=False), and ParallelRunner(max_workers=None, is_async=False).

Common Workflows

  1. For a new project, use project-cli-and-sessions to choose kedro new flags, validate naming, opt out of telemetry if required, and explain generated layout.
  2. For pipeline code, use pipelines-and-nodes to design nodes and reusable pipeline factories before choosing catalog entries or runners.
  3. For data/config files, use data-catalog-and-config to validate catalog structure, credentials indirection, runtime params, config environments, and optional dataset dependencies.
  4. For execution, use runners-and-execution to build the kedro run command or programmatic KedroSession.run() call and select the runner.
  5. For customization, use hooks-and-extensions to add hooks/plugins/custom commands or datasets, then return to sibling skills for config and run validation.
  6. For diagnostics, use inspection-and-server for read-only snapshots and optional server APIs; use execution routes only when the task explicitly runs nodes.

References

  • Read references/package-overview.md for package facts, command families, optional extras, public API map, and cross-skill workflow examples.
  • Read references/troubleshooting.md for cross-cutting install/import, CLI, telemetry, optional dependency, project detection, stale API, and secret-handling issues.
  • Read references/repo-provenance.md to see the source commit, dirty-state note, package version, and evidence paths used to create this skill.
  • references/repo-routing-metadata.json is structured metadata for managed repo-skills-router import.

Safety and Boundaries

  • Do not rely on the original repository checkout, docs, tests, templates, or scripts at runtime; this skill bundles distilled references and safe helper scripts.
  • Avoid printing credentials from credentials.yml, environment variables, or cloud auth helpers; validate the presence and shape of secrets without echoing values.
  • Set KEDRO_DISABLE_TELEMETRY=1 or DO_NOT_TRACK=1 for automated kedro CLI probes when telemetry/network side effects are undesirable.
  • Warn before kedro new with remote starters, --example=yes, deployment commands, server startup, or any command that writes projects, builds artifacts, opens browsers, or uses network access.
我的收藏