data-data-catalogUse this skill when asked about data catalog, DataHub, Amundsen, Apache Atlas, OpenMetadata, metadata management, data discovery, business glossary, data lineage, data ownership, or column-level lineage. This skill enforces: metadata ingestion pipelines, column-level lineage tracking, business glossary management, data discovery with search, catalog API usa…
Independent GitHub snapshot · Automated classification · Original documentation governs
Design and deploy a data catalog platform with metadata ingestion, column-level lineage, business glossary, data discovery, ownership model, and usage analytics.
Exact user phrases: "data catalog", "DataHub", "Amundsen", "Apache Atlas", "OpenMetadata", "metadata management", "data discovery", "business glossary", "data lineage", "data ownership", "column-level lineage", "metadata ingestion".
Data catalog architecture with platform selection, metadata ingestion pipeline design, governance model with glossary and ownership, and usage analytics framework.
# Catalog platform selection matrix
# Metadata ingestion pipeline
# Business glossary structure
# Ownership & stewardship model
# Usage analytics schema
No preamble. No postamble. No explanations. No filler/hedging/transitions. Compress output — why use many token when few do trick.
350 lines of configuration.
| Platform | Strengths | Weaknesses | Best For |
|---|---|---|---|
| DataHub | Column-level lineage, real-time push, GraphQL API | Complex deployment | ML-first orgs, real-time metadata |
| Amundsen | Simple search, good Tableau integration | Limited lineage, Python-only | Search-first use cases |
| OpenMetadata | Ingestion framework, role-based access, UI | Newer ecosystem, smaller community | Organizations wanting all-in-one |
| Apache Atlas | Hadoop-native, tag-based policies | Outdated UX, heavy | Existing Hadoop ecosystem |
Default: DataHub for lineage-heavy, OpenMetadata for governance-heavy. Amundsen only if search is primary need and lineage secondary.
Decision tree:
| Source | Method | Frequency | Tool |
|---|---|---|---|
| Snowflake/BigQuery/Redshift | JDBC connector | Daily | DataHub ingestion CLI |
| dbt | dbt artifacts (manifest.json) | On dbt run | DataHub dbt ingestion |
| Spark | OpenLineage + SparkListener | Real-time | OpenLineage Spark integration |
| Airflow | OpenLineage + Airflow plugin | Per DAG run | OpenLineage Airflow integration |
| Kafka | Schema Registry poll | Real-time | DataHub Kafka connector |
| Tableau/PowerBI | Metadata API | 6-hourly | DataHub BI ingestion |
source:
type: snowflake
config:
account_id: xy12345.us-east-1
warehouse: COMPUTE_WH
role: DATAHUB_ROLE
include_views: true
include_tables: true
profiling:
enabled: true
profile_table_level_only: false
capture_column_lineage:
enabled: true
sink:
type: datahub-rest
config:
server: "http://datahub-gms:8080"
source:
type: snowflake
serviceName: snowflake_prod
serviceConnectionType: Snowflake
sourceConfig:
config:
type: DatabaseMetadata
markDeletedTables: true
includeTables: true
includeViews: true
sink:
type: metadata-rest
config:
apiEndpoint: http://openmetadata-server:8585/api
Configure via OpenLineage for Spark/Airflow. Use dbt catalog.json + manifest.json for dbt models. For SQL-based ETL, use DataHub SQL parser to extract column-level lineage from query logs.
# DataHub lineage extraction from SQL
source:
type: glue
config:
aws_region: us-east-1
extract_transforms: true
sql_parser:
enabled: true
lineage_type: COLUMN
Column-level lineage enables impact analysis: "which dashboards break if I change this column?" and root cause analysis: "which source column caused this data quality issue?"
| Domain | Term | Definition | Steward |
|---|---|---|---|
| Commerce | Customer | Person or entity that places an order | data.owner@org.com |
| Finance | Revenue | Total income from sales before deductions | finance.owner@org.com |
| Marketing | Attribution | Assignment of credit to marketing touchpoints | marketing.owner@org.com |
Glossary terms linked to tables and columns via schemaField tags. Terms inherit from parent to child datasets.
| Role | Responsibility | Coverage |
|---|---|---|
| Data Owner | Quality, access, lifecycle of datasets | Every dataset must have owner |
| Data Steward | Glossary, documentation, metadata quality | Per domain |
| Data Custodian | Technical implementation, pipeline health | Per pipeline |
Ownership stored as metadata on each dataset entity. Stewards review metadata quarterly.
| Facet | Example Values |
|---|---|
| Platform | Snowflake, BigQuery, S3, Kafka |
| Domain | Commerce, Finance, Marketing |
| Owner | team-analytics, team-marketing |
| Tier | Critical, Important, Operational |
| Tag | PII, GDPR, Sensitive, Certified |
Search rankings: popularity > freshness > completeness > documentation score.
Track: dataset_queries, column_access_frequency, top_users, search_queries. Publish dashboard: most queried datasets, orphan datasets (no reads in 90 days), top searches without results.
from datahub.ingestion.graph.client import DataHubGraph
graph = DataHubGraph(
server="http://datahub-gms:8080",
token="<personal-access-token>",
)
# Search datasets
results = graph.search_entities(
type="dataset",
query="orders",
facets=["platform", "domain", "owner"],
start=0, count=20,
)
for dataset in results.entities:
print(dataset.urn, dataset.properties.name)
# Get lineage
lineage = graph.get_lineage(
urn="urn:li:dataset:(urn:li:dataPlatform:snowflake,PROD_DB.COMMERCE.FCT_ORDERS,O)",
depth=2, direction="UPSTREAM",
)
Implement data access request workflows through the catalog. When a consumer finds a dataset, they click "Request Access" which creates a ticket for the data owner. The owner approves or denies. On approval, permissions are provisioned automatically. Catalog tracks access grants and flags stale permissions for quarterly review.
Score each dataset on metadata completeness: description (exists, > 50 chars), owner (assigned), glossary terms (linked), lineage (column-level), documentation score (fields documented). Publish scores on catalog dashboard. Domain stewards responsible for improving low-scoring datasets.
Need data catalog?
├── Real-time column lineage required? → DataHub
├── Built-in governance + glossary workflows? → OpenMetadata
├── Existing Hadoop/Hive infrastructure? → Apache Atlas
├── Simple search, limited resources? → Amundsen
└── Need managed service? → DataHub Cloud / Atlan / Alation
Which sources first?
├── Tier 1: Critical BI datasets (Snowflake, BigQuery, Redshift)
├── Tier 2: Transformation layer (dbt, Spark, Airflow)
├── Tier 3: Streaming (Kafka, Schema Registry)
└── Tier 4: BI tools (Tableau, PowerBI, Looker)
| Feature | DataHub | OpenMetadata | Amundsen | Apache Atlas |
|---|---|---|---|---|
| Column lineage | Yes (real-time) | Yes | No | Limited |
| Business glossary | Yes | Yes (native) | Yes | Yes |
| Search | Full-text + faceted | Full-text + faceted | Simple full-text | Basic |
| Real-time ingestion | Push API + pull | Pull only | Pull only | Pull only |
| dbt integration | Native | Native | Custom | Custom |
| BI tool lineage | Tableau, PowerBI | Tableau, PowerBI | Tableau | None |
| Access control | RBAC | RBAC + SSO | Basic | Tag-based |
| API | GraphQL + REST | REST | REST | REST |
Data catalog vs schema registry: catalog manages business metadata, ownership, and discovery. Schema registry manages technical schema, compatibility, and serialization. They integrate: schema registry feeds schemas to the catalog as technical metadata.
Data catalog vs data quality tools: catalog is about discovery and governance. Quality tools (Great Expectations, Soda) validate data against expectations. They integrate: quality results feed into catalog as dataset health scores.
Scalability: DataHub scales horizontally by adding GMS instances. Search scales with Elasticsearch cluster. For 100K+ datasets, partition by domain and use tiered storage for older metadata.
| Tool | Purpose |
|---|---|
| DataHub | Metadata platform, lineage, search |
| OpenMetadata | All-in-one catalog with governance |
| Amundsen | Search-focused catalog |
| Apache Atlas | Hadoop-native metadata management |
| OpenLineage | Open standard for lineage collection |
| dbt | Transformation metadata source |
| Marquez | OpenLineage implementation |
| Atlan / Alation | Commercial catalog platforms |
| SQLGlot | SQL parser for lineage extraction |
| Great Expectations | Data quality for catalog integration |
# DataHub ingestion recipe: snowflake + dbt + tableau
datahub_ingestion:
source:
type: snowflake
config:
account_id: "xyz12345"
warehouse: "transforming"
include_views: true
include_tables: true
profiling:
enabled: true
profile_table_level_only: false
profile_pattern:
allow: ["analytics.*"]
source:
type: dbt
config:
manifest_path: "./target/manifest.json"
catalog_path: "./target/catalog.json"
sources_path: "./target/sources.json"
enable_meta_mapping: true
node_type_pattern:
allow: ["model", "source", "test"]
source:
type: tableau
config:
connect_uri: "https://10ay.online.tableau.com"
site: "my-site"
projects: ["default", "executive"]
extract_lineage: true
ingest_dashboards: true
sink:
type: datahub-rest
config:
server: "http://datahub-gms:8080"
metadata_model:
dataset:
required_fields:
- id: "fully qualified name"
- name: "human-readable name"
- schema: { columns: [{ name, type, description, nullable }] }
- owner: "team or individual"
- tier: [1, 2, 3]
- domain: "business domain tag"
optional_fields:
- glossary_terms: ["term1", "term2"]
- tags: ["PII", "FINANCIAL", "EXPERIMENTAL"]
- lineage: { upstream: [], downstream: [] }
- quality: { score: 0.95, last_checked: "2026-05-01" }
- usage: { weekly_queries: 500, top_users: [] }
- profile: { row_count: 5000000, size_bytes: 1073741824 }
glossary_term:
required_fields:
- id: "term_id"
- name: "term_name"
- description: "term definition"
- domain: "responsible domain"
optional_fields:
- synonyms: ["also_known_as"]
- related_terms: ["parent", "child"]
- stewardship: { owner: "data_governance@org.com" }
tag:
required_fields:
- name: "tag_name"
- description: "tag purpose"
optional_fields:
- color: "#hex"
- inherit_from_dataset: true
# DataHub GraphQL queries for catalog integration
import requests
def search_datasets(keyword, domain=None, tier=None):
query = """
query searchDatasets($query: String!, $filters: [FacetFilterInput!]) {
search(input: { type: DATASET, query: $query, filters: $filters }) {
total results { entity { urn type ...on Dataset { name platform { name } } } }
}
}
"""
variables = {"query": keyword, "filters": []}
if domain: variables["filters"].append({"field": "domains", "value": domain})
if tier: variables["filters"].append({"field": "tier", "value": str(tier)})
response = requests.post(
"http://datahub-gms:8080/api/graphql",
json={"query": query, "variables": variables},
headers={"Authorization": "Bearer <token>"}
)
return response.json()
def update_metadata(dataset_urn, metadata):
"""Update dataset description and tags via DataHub API"""
mutation = """
mutation updateDataset($urn: String!, $input: DatasetUpdateInput!) {
updateDataset(urn: $urn, input: $input) { urn }
}
"""
variables = {
"urn": dataset_urn,
"input": {
"editableProperties": {
"description": metadata.get("description", ""),
"tags": metadata.get("tags", [])
}
}
}
return requests.post("http://datahub-gms:8080/api/graphql",
json={"query": mutation, "variables": variables})
Team size and metadata requirements?
├── Small team (< 10), need quick data discovery
│ └── Amundsen (simple, search-focused)
├── Mid-size team, need lineage + governance
│ ├── Python/Spark/Airflow stack → DataHub (best lineage support)
│ └── dbt-native stack → OpenMetadata (dbt integration)
├── Large enterprise, compliance-driven
│ └── DataHub or Atlan (strong governance, RBAC, audit)
└── Hadoop/Hive-centric ecosystem
└── Apache Atlas (native Hive/HBase/Spark integration)
Catalog Platform Selection
├── Cloud-native SaaS desired?
│ ├── Yes → Atlan / Datahub Cloud / Alation
│ └── No → Open-source self-hosted (Datahub, Amundsen)
├── dbt integration required?
│ ├── Yes → Datahub (dbt ingestion native)
│ └── No → Evaluate feature completeness
├── Real-time metadata sync?
│ ├── Yes → Datahub with Kafka-based ingestion
│ └── No → Batch ingestion (daily cron)
└── Schema drift detection needed?
├── Yes → Datahub with column-level lineage
└── No → Basic table/column catalog (Amundsen)
Decision criteria: Evaluate metadata freshness, integration depth, operating cost, and team size for administration.
# data_catalog/ingestion.py
from dataclasses import dataclass
from datetime import datetime
@dataclass
class TableMetadata:
database: str
schema: str
table: str
columns: list[dict]
row_count: int
last_updated: datetime
description: str
class MetadataIngestor:
def __init__(self, catalog_api: str, token: str):
self.api = catalog_api
self.headers = {"Authorization": f"Bearer {token}"}
async def ingest_table(self, metadata: TableMetadata):
payload = {
"urn": f"urn:li:dataset:{metadata.database}.{metadata.schema}.{metadata.table}",
"schema": metadata.columns,
"last_updated": metadata.last_updated.isoformat(),
"description": metadata.description,
}
async with httpx.AsyncClient() as client:
resp = await client.post(f"{self.api}/entities", json=payload, headers=self.headers)
resp.raise_for_status()
# data_catalog/search_config.yml
index:
name: data_catalog
fields:
- name: table_name
type: text
boost: 3.0
- name: description
type: text
boost: 1.5
- name: tags
type: keyword
boost: 2.0
synonyms:
- ["user", "customer", "account"]
- ["revenue", "sales", "income"]
| Anti-Pattern | Consequence | Solution |
|---|---|---|
| Manual metadata entry | Stale catalog, low adoption | Automate all ingestion from sources |
| No column-level lineage | Trust issues with data | Enable field-level lineage tracking |
| Over-permissive discovery | Sensitive data exposed | Implement role-based access on catalog |
| No freshness indicators | Users distrust stale tables | Add last-updated badges and alert on staleness |
| Ignoring dbt model metadata | Catalog misses transformation logic | Ingest dbt manifest and docs |
data-data-platform for platform infrastructure. data-data-quality for linking quality metadata to catalog. data-data-observability for freshness monitoring. data-data-contracts for contract metadata in catalog.