---

name: streaming description: "Mage streaming pipelines, real-time sources, sinks, and connector configuration." metadata: disco-role: operating disable-model-invocation: true license: Apache 2.0

Streaming

Use this route for real-time Mage pipelines and message-oriented connectors.

Owns

• Streaming pipeline structure: source, transformer, sink
• SourceFactory and SinkFactory connector selection
• Kafka, CDC, and other live streaming source/sink configs
• Streaming executor knobs such as executor_count and executor_type
• Message metadata, batch size, checkpointing, and live connector troubleshooting

Excludes

• Batch data integrations and io_config.yaml, which belong in batch-integrations
• Batch pipeline block code, which belongs in pipeline-authoring
• dbt orchestration, which belongs in dbt-workflows
• CLI/bootstrap/auth/logging, which belong in platform-ops
• AI generation, which belongs in ai-workflows

Use this route when

• The user asks about real-time pipelines
• The user mentions Kafka, CDC, Pub/Sub, Kinesis, RabbitMQ, or similar live connectors
• The user needs sink selection or source-selection guidance for streaming
• The user is debugging a streaming connector failure or runtime mode issue

Bundled reference

Read references/streaming-connectors.md for the source/sink matrix, connector patterns, executor settings, and message-handling notes.

Bundled troubleshooting

Read references/troubleshooting.md for connection, auth, live-broker, and test-environment failures.

Core reminders

• Streaming pipelines are different from batch integrations; do not route them through batch-integrations.
• The repo's streaming source class tests can try live connection checks unless the environment is treated as test.
• Many connectors require extra packages and external services that are not part of a plain CPU-only install.