Search Results for "streaming"

...With Spark Streaming (microbatches) and Structured Streaming, it delivers low-latency event processing suitable for real-time analytics. The built-in MLlib library provides scalable machine learning algorithms, while GraphX enables graph computations integrated with data pipelines. Spark supports multiple languages—Scala, Java, Python, R—and connects with many storage systems like HDFS, S3, Cassandra, and streaming platforms like Kafka, making it a versatile choice for big data workloads in analytics, ETL, and data science.

FS2 (“Functional Streams for Scala”) is a purely functional, effectful abstraction for stream processing on the JVM. Built on Cats Effect, it enables compositional resource-safe streaming workflows with robust error handling, back-pressure, pull/push semantics, and support for concurrent and interruptible pipelines.

Build powerful reactive, concurrent, and distributed applications more easily. Akka is a toolkit for building highly concurrent, distributed, and resilient message-driven applications for Java and Scala. Actors and Streams let you build systems that scale up, using the resources of a server more efficiently, and out, using multiple servers. Building on the principles of The Reactive Manifesto Akka allows you to write systems that self-heal and stay responsive in the face of failures. Up to...

The Alpakka project is an open source initiative to implement stream-aware and reactive integration pipelines for Java and Scala. It is built on top of Akka Streams and has been designed from the ground up to understand streaming natively and provide a DSL for reactive and stream-oriented programming, with built-in support for backpressure. Akka Streams is a Reactive Stream and JDK 9+ java.util.concurrent.Flow-compliant implementation and therefore fully interoperable with other implementations. As Kafka’s client protocol negotiates the version to use with the Kafka broker, you may use a Kafka client version that is different than the Kafka broker’s version.

Algebird is Twitter’s Apache‑licensed Scala library providing abstract algebra data structures and algorithms, especially for online/streaming aggregation. It includes Monoid, Approximate, HyperLogLog, CMS, BloomFilter, Min/Max, Averaged Value types, supporting efficient distributed aggregation and approximate analytics.

The opinionated microservices framework for moving away from the monolith. Lagom helps you decompose your legacy monolith and build, test, and deploy entire systems of Reactive microservices. Lagom is an open source framework for building systems of Reactive microservices in Java or Scala. Lagom builds on Akka and Play, proven technologies that are in production in some of the most demanding applications today. Lagom's integrated development environment allows you to focus on solving...

...Rather than requiring every application to embed Spark or manage Spark contexts manually, this server abstracts a long-lived service where clients can upload JARs, start and stop contexts, submit jobs synchronously or asynchronously, and manage named objects (RDDs / DataFrames) across job executions. It supports multiple modes (transient jobs, persistent contexts for reuse, streaming, SQL/Hive, etc.), and can be integrated with authentication/authorization systems (e.g. via Apache Shiro). The architecture isolates Spark contexts (optionally in separate JVMs), isolates job dependencies, and persists job / jar metadata via pluggable DAOs. It supports deployment across cluster managers (YARN, Mesos, etc.) and aims to simplify Spark-as-a-service scenarios.

Summingbird is a streaming + batch hybrid computation framework developed by Twitter. Its aim is to let developers express data aggregation pipelines in a unified way, where the same logic can run either in real time (stream) or in batch mode, and the results can be merged or reconciled. In effect, Summingbird abstracts over multiple execution engines (such as Storm, Scalding, etc.) to provide one high-level program that composes transformations and aggregations, and then executes them in different runtime contexts. ...

Spray is a suite of Scala libraries built on top of Akka that offers a modular, asynchronous, and non-blocking toolkit for building and consuming RESTful and HTTP services. The core philosophy behind Spray is that it should act as a “library” for integration and HTTP layers, not as a full application framework—it gives you the building blocks to handle HTTP and REST but doesn’t impose heavy structure. It includes modules for low-level HTTP I/O, routing, client and server APIs, HTTP model...