Audience
Companies looking for a distributed Workflow Manager solution that helps solve the problem of Hadoop job dependencies
About Azkaban
Azkaban is a distributed Workflow Manager, implemented at LinkedIn to solve the problem of Hadoop job dependencies. We had jobs that needed to run in order, from ETL jobs to data analytics products. After version 3.0, we provide two modes: the stand alone “solo-server” mode and distributed multiple-executor mode. The following describes the differences between the two modes. In solo server mode, the DB is embedded H2 and both web server and executor server run in the same process. This should be useful if one just wants to try things out. It can also be used on small scale use cases. The multiple executor mode is for most serious production environment. Its DB should be backed by MySQL instances with master-slave set up. The web server and executor servers should ideally run in different hosts so that upgrading and maintenance shouldn’t affect users. This multiple host setup brings in robust and scalable aspect to Azkaban.
Integrations
Company Information
Azkaban
azkaban.github.io
Videos and Screen Captures
Data Virtuality
Connect and centralize data. Transform your existing data landscape into a flexible data powerhouse. Data Virtuality is a data integration platform for instant data access, easy data centralization and data governance. Our Logical Data Warehouse solution combines data virtualization and materialization for the highest possible performance. Build your single source of data truth with a virtual layer on top of your existing data environment for high data quality, data governance, and fast time-to-market. Hosted in the cloud or on-premises. Data Virtuality has 3 modules: Pipes, Pipes Professional, and Logical Data Warehouse. Cut down your development time by up to 80%. Access any data in minutes and automate data workflows using SQL. Use Rapid BI Prototyping for significantly faster time-to-market. Ensure data quality for accurate, complete, and consistent data. Use metadata repositories to improve master data management.
Learn more
Yandex Data Proc
You select the size of the cluster, node capacity, and a set of services, and Yandex Data Proc automatically creates and configures Spark and Hadoop clusters and other components. Collaborate by using Zeppelin notebooks and other web apps via a UI proxy. You get full control of your cluster with root permissions for each VM. Install your own applications and libraries on running clusters without having to restart them. Yandex Data Proc uses instance groups to automatically increase or decrease computing resources of compute subclusters based on CPU usage indicators. Data Proc allows you to create managed Hive clusters, which can reduce the probability of failures and losses caused by metadata unavailability. Save time on building ETL pipelines and pipelines for training and developing models, as well as describing other iterative tasks. The Data Proc operator is already built into Apache Airflow.
Learn more
StreamScape
Make use of Reactive Programming on the back-end without the need for specialized languages or cumbersome frameworks. Triggers, Actors and Event Collections make it easy to build data pipelines and work with data streams using simple SQL-like syntax, shielding users from the complexities of distributed system development. Extensible Data Modeling is a key feature that supports rich semantics and schema definition for representing real-world things. On-the-fly validation and data shaping rules support a variey of formats like XML and JSON, allowing you to easily describe and evolve your schema, keeping pace with changing business requirements. If you can describe it, we can query it. Know SQL and Javascript? Then you already know how to use the data engine. Whatever the format, a powerful query language lets you instantly test logic expressions and functions, speeding up development and simplifying deployment for unmatched data agility.
Learn more
Gravity Data
Gravity's mission is to make streaming data easy from over 100 sources while only paying for what you use. Gravity removes the reliance on engineering teams to deliver streaming pipelines with a simple interface to get streaming up and running in minutes from databases, event data and APIs. Everyone in the data team can now build with simple point and click so that you can focus on building apps, services and customer experiences. Full Execution trace and detailed error messaging for quick diagnosis and resolution. We have implemented new, feature-rich ways for you to quickly get started. From bulk set-up, default schemas and data selection to different job modes and statuses. Spend less time wrangling with infrastructure and more time analysing data while allowing our intelligent engine to keep your pipelines running. Gravity integrates with your systems for notifications and orchestration.
Learn more
You Might Also Like
With the world of work changed forever, it’s essential to manage your workplace and assets together to effectively create a high-performing environment. The Eptura experience combines the power of workplace management software with asset management, enabling you to effectively operate your building and facilitate hybrid work.
Product Details
Platforms Supported
SaaS
Training
Documentation
Support
Online