name: batch-integrations
description: "Mage batch data integration pipelines, io_config.yaml, and
connector configuration."
metadata:
disco-role: operating
disable-model-invocation: true
license: Apache 2.0
Batch Integrations
Use this route for non-streaming data integration pipelines and connector configuration.
Owns
io_config.yaml layout and profile resolution- Batch data integration pipelines that move data between one source and one destination
- Source/destination configuration, schema settings, bookmarks, and stream selection
- Variable interpolation in integration configs
- Connector families exposed by the repo and the sibling
mage_integrations package
- Safe validation of connector configuration without running the connector against a live service
Excludes
- Batch block code itself, which belongs in
pipeline-authoring
- Real-time message streams, which belong in
streaming
- dbt orchestration, which belongs in
dbt-workflows
- CLI bootstrap, auth, logging, and deployment, which belong in
platform-ops
- AI generation, which belongs in
ai-workflows
Use this route when
- The user asks how to configure a source or destination profile
- The user wants to understand
io_config.yaml
- The user needs help with data integration pipeline schema, bookmark, or prefix behavior
- The user wants to extend or reason about connector configuration
Bundled reference
Read references/connectors-and-config.md for the config shape, interpolation rules, connector families, and source/destination workflow.
Bundled troubleshooting
Read references/troubleshooting.md for Docker, profile, credential, local-host, and S3 credential failures.
Bundled script
scripts/inspect_io_config.py — safely inspect profiles and key presence without printing secrets
Core reminders
- The docs and source treat data integrations as Docker-oriented because of connector dependencies.
io_config.yaml belongs in the project root.- Use this route for the configuration layer; use
pipeline-authoring for the block logic that consumes the integration output.