Download
Broadway is a data processing library for Elixir designed to handle high-throughput, concurrent workloads with ease. It provides an abstraction for defining pipelines that consume data from sources like RabbitMQ, Kafka, Amazon SQS, or custom producers. Each pipeline is fault-tolerant and backpressure-aware, ensuring stable throughput even under load. The library integrates seamlessly with GenStage and OTP supervision trees, making it highly resilient in production. Developers can enrich pipelines with batching, concurrency control, and metrics reporting, simplifying the management of complex data ingestion and processing systems. Broadway is often used for event processing, stream handling, and background jobs, offering both performance and clarity in Elixir’s functional style.
Features
- Simplifies construction of concurrent data pipelines
- Supports multiple producers (e.g., Amazon SQS, Kafka, RabbitMQ, Google PubSub)
- Automatically manages GenStage topology under the hood
- Batching support for efficient processing
- Built‑in supervision and graceful shutdown via Terminator process
- Declarative configuration API minimizing boilerplate
Project Samples
