DownloadOryx 2 is a realization of the lambda architecture built on Apache Spark and Apache Kafka, but with specialization for real-time large-scale machine learning. It is a framework for building applications but also includes packaged, end-to-end applications for collaborative filtering, classification, regression and clustering. The application is written in Java, using Apache Spark, Hadoop, Tomcat, Kafka, Zookeeper and more. Configuration uses a single Typesafe Config config file, wherein applications configure an entire deployment of the system. This includes implementations of key interface classes which implement the batch, speed, and serving logic. Applications package and deploy their implementations with each instance of the layer binaries. Each of these is a runnable Java .jar which starts all necessary services.
Features
- Generic lambda architecture tier, providing batch/speed/serving layers, which is not specific to machine learning
- Specialization on top providing ML abstractions for hyperparameter selection, etc.
- End-to-end implementation of the same standard ML algorithms as an application (ALS, random decision forests, k-means) on top
- The data transport mechanism is an Apache Kafka topic
- The speed layer is implemented as a Spark Streaming process
- The batch layer is implemented as a Spark Streaming process on a Hadoop cluster
Project Samples
