Search Results for "storage linux"

Apache Spark is a unified engine for large-scale data processing, offering APIs for batch jobs, streaming, machine learning, and graph computation. It builds on resilient distributed datasets (RDDs) and the newer DataFrame/Dataset abstractions to provide fault-tolerant, in-memory computation across clusters. Spark’s execution engine handles scheduling, shuffles, caching, and data locality so users can focus on transformations rather than infrastructure plumbing. With Spark Streaming...

Deequ is a library built atop Apache Spark that enables defining “unit tests for data” — that is, formal constraints or checks on datasets to ensure data quality along dimensions such as completeness, uniqueness, value ranges, correlations, etc. It can scale to large datasets (billions of rows) by translating those data checks into Spark jobs. Deequ supports advanced features like a metrics repository for storing computed statistics over time, anomaly detection of data quality metrics, and...

A container orchestration platform for Mesos and DC/OS. Deploy and manage containers (including Docker) on top of Apache Mesos at scale. Marathon is a production-grade container orchestration platform for Mesosphere’s Datacenter Operating System (DC/OS) and Apache Mesos. Marathon has first-class support for both Mesos containers (using cgroups) and Docker. Marathon runs as an active/passive cluster with leader election for 100% uptime. Marathon can bind persistent storage volumes to your...

Graphcool was an open-source framework for developing and deploying GraphQL-based backends. It acts as a “backend-as-a-service / framework” that lets you define your data model via GraphQL SDL (Schema Definition Language), and in turn generates a GraphQL CRUD API, supports nested mutations, filtering, pagination, and real-time subscriptions. Graphcool separates the business logic from stateful storage components, allowing the stateful parts (database, subscription engine) to scale...

Gizzard is a Scala framework originally developed by Twitter for building scalable, fault-tolerant, distributed key-value stores that can be sharded and replicated. It provides infrastructure for routing requests through shard trees, splitting or rebalancing shards dynamically, failover, and migrations. In Gizzard, data is stored in underlying storage shards (which could be databases or other stores) and Gizzard handles the process of routing requests correctly as the cluster topology...