Broadway
Concurrent and multi-stage data ingestion and data processing
...It provides an abstraction for defining pipelines that consume data from sources like RabbitMQ, Kafka, Amazon SQS, or custom producers. Each pipeline is fault-tolerant and backpressure-aware, ensuring stable throughput even under load. The library integrates seamlessly with GenStage and OTP supervision trees, making it highly resilient in production. Developers can enrich pipelines with batching, concurrency control, and metrics reporting, simplifying the management of complex data ingestion and processing systems. Broadway is often used for event processing, stream handling, and background jobs, offering both performance and clarity in Elixir’s functional style.