界面已本地化;技能介绍与说明文档按所选语言由 AI 自动翻译,代码、命令、路径和专业术语保留原文,以原作者文档为准。
探索技能
设计与创意 · SKILL.md

data-warehouse

data-warehouse

【设计与创意】Use when designing, building, or optimizing enterprise data warehouses — Kimball vs Inmon methodology, dimensional and normalized modeling, slowly changing dimensions, ETL/ELT pipeline design, partitioning and distribution strategies, materialized views, query optimization, data vault 2.0, warehouse platforms (PostgreSQL/Snowflake/Redshift/BigQuery/S…

独立 GitHub 快照 · 自动分类 · 以原作者文档为准

技能说明

GitHub

name: data-warehouse description: Use when designing, building, or optimizing enterprise data warehouses — Kimball vs Inmon methodology, dimensional and normalized modeling, slowly changing dimensions, ETL/ELT pipeline design, partitioning and distribution strategies, materialized views, query optimization, data vault 2.0, warehouse platforms (PostgreSQL/Snowflake/Redshift/BigQuery/Synapse/DB2 Warehouse), and operational patterns (incremental loads, CDC, reconciliation). Part of the data-* skill family. disambiguation: The ENTERPRISE layer — methodology choice, integration, SCD strategy, warehouse-wide modelling. One departmental subject-area mart is data-mart.

Enterprise Data Warehouse — Design & Implementation

Companion skills: python-data-engineer (SQLAlchemy, Alembic, query optimization), python-enterprise-connectors (DB2, Oracle, SQL Server, mainframe integration), rhel-databases / ubuntu-databases (PostgreSQL/MySQL administration).


Reference Files

Detailed code examples, patterns, and configuration are in the reference files below. Read the relevant file when working on that area.

FileCovers
etl-scd-optimization.mdData Vault 2.0, ETL/ELT design, slowly changing dimensions, partitioning/distribution strategies, and query optimization
methodology-modeling.mdKimball vs Inmon methodology, architecture patterns, dimensional modeling (star/snowflake schemas, fact/dimension tables)
operations-testing-platforms.mdplatform-specific guidance (Snowflake, BigQuery, Redshift, Databricks), operational patterns (refresh strategies, monitoring, disaster recovery), and testing/quality


Anti-Patterns

Anti-PatternWhy It FailsCorrect Approach
Using 3NF (normalized) design for user-facing query layersJoins across 10+ normalized tables destroy query performance; business users cannot write their own queriesUse dimensional modeling (star schema) for consumption layer; normalize only in staging/integration layers
No slowly changing dimension strategyOverwriting dimension attributes loses history; reports retroactively change when a customer moves or a product renamesChoose SCD type per attribute: Type 1 for corrections, Type 2 for historically significant changes, Type 3 for limited history
Loading the warehouse during business hoursETL competes with queries for resources; users experience slow dashboards; lock contention causes failuresSchedule loads during off-hours batch windows; use change data capture (CDC) for near-real-time if business requires it
No materialized views or pre-aggregated tables for common queriesDashboard queries recompute the same aggregations on millions of rows every time; slow and wastefulCreate materialized views for high-frequency query patterns; refresh on schedule; index aggregation columns
Skipping data quality checks between ETL stagesBad data from source systems propagates into production reports; executives lose trust in the numbersImplement data quality gates at each ETL stage: null checks, range validation, referential integrity, row count reconciliation

Related Skills

DomainSkill
SQLAlchemy, Alembic, query optimizationpython-data-engineer
DB2, Oracle, SQL Server, mainframe connectorspython-enterprise-connectors
PostgreSQL/MySQL/Redis on RHEL 9rhel-databases
PostgreSQL/MySQL/Redis on Ubuntu 24.04ubuntu-databases
Docker containers for dev/test warehousesdocker-compose-patterns
ETL/pipeline parallelism in Pythonpython-parallelism
Data pipeline profiling and load testingperformance
我的收藏