Pipes and Filters
Compose sequential processing via independent filters connected by pipes; foundation for ETL, streaming, and reactive pipelines.
What it is
An integration style where data flows through a series of filters (transformations), connected by pipes (channels), enabling composability and reuse.
Benefits
- Loose coupling and composability
- Parallelization and scalability
- Testable, reusable processing stages
Trade-offs
- End-to-end latency across many stages
- Error handling and backpressure management
- Observability across distributed pipelines
When to use
- ETL/ELT and data transformation
- Streaming and event processing
- Workflow orchestration with reusable steps