Search Results for "library python"

Apache Spark is a high-performance, unified analytics engine for large-scale data processing. It supports distributed data workflows across clusters, enabling batch processing, stream processing, machine learning, graph computations, and SQL queries. Spark offers high-level APIs in multiple languages (Scala, Java, Python, etc.), integrates with many storage systems, and is oriented toward in-memory computation to drastically speed up large-scale data tasks.

SynapseML (previously MMLSpark) is an open source library to simplify the creation of scalable machine learning pipelines. SynapseML builds on Apache Spark and SparkML to enable new kinds of machine learning, analytics, and model deployment workflows. SynapseML adds many deep learning and data science tools to the Spark ecosystem, including seamless integration of Spark Machine Learning pipelines with the Open Neural Network Exchange (ONNX), LightGBM, The Cognitive Services, Vowpal Wabbit...

Deequ is a library built by Amazon (AWS Labs) on top of Apache Spark to enable automated data quality testing, constraint verification, and anomaly detection at scale. It lets users define assertions or constraints about data (e.g. completeness, uniqueness, min / max, correlations etc.), run metrics, verify that data meets expectations, suggest constraints, detect drift/anomaly, and integrate into data pipelines so that bad data is caught early before feeding downstream systems or ML.

Scala and polyglot Spark library (Scala, PySpark, SparkSQL, ... ) focused on reading OpenStreetMap Pbf files.

Assorted projects. General-purpose libraries for Python, C++, Scala, bash, and others. Meta-programming tools. System utilities. UI components. Web APIs. Configuration files. Benchmarks. Programming competition entries. And much more.

Java-based scientific graphics with support for Java, Groovy, MATLAB, Python, the R statistical environment, Scala and SciLab.