Deequ PySpark

Related Products

  • Parasoft
    4.6 out of 5 stars 152 Ratings
    Visit Website for Parasoft
  • Google Cloud Platform
    4.5 out of 5 stars 61,093 Ratings
    Visit Website for Google Cloud Platform
  • TIMi
    4.9 out of 5 stars 68 Ratings
    Visit Website for TIMi
  • Gearset
    4.7 out of 5 stars 305 Ratings
    Visit Website for Gearset
  • DropTrack
    4.5 out of 5 stars 191 Ratings
    Visit Website for DropTrack
  • pCloud Business
    4.3 out of 5 stars 189 Ratings
    Visit Website for pCloud Business
  • Virtuoso QA
    4.5 out of 5 stars 131 Ratings
    Visit Website for Virtuoso QA
  • RunMyJobs by Redwood
    4.8 out of 5 stars 429 Ratings
    Visit Website for RunMyJobs by Redwood
  • Flagsmith
    4.8 out of 5 stars 43 Ratings
    Visit Website for Flagsmith
  • Dragonfly
    4.9 out of 5 stars 16 Ratings
    Visit Website for Dragonfly
Deequ:

About

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets. We are happy to receive feedback and contributions. Deequ depends on Java 8. Deequ version 2.x only runs with Spark 3.1, and vice versa. If you rely on a previous Spark version, please use a Deequ 1.x version (legacy version is maintained in legacy-spark-3.0 branch). We provide legacy releases compatible with Apache Spark versions 2.2.x to 3.0.x. The Spark 2.2.x and 2.3.x releases depend on Scala 2.11 and the Spark 2.4.x, 3.0.x, and 3.1.x releases depend on Scala 2.12. Deequ's purpose is to "unit-test" data to find errors early, before the data gets fed to consuming systems or machine learning algorithms. In the following, we will walk you through a toy example to showcase the most basic usage of our library.
PySpark:

About

PySpark is an interface for Apache Spark in Python. It not only allows you to write Spark applications using Python APIs, but also provides the PySpark shell for interactively analyzing your data in a distributed environment. PySpark supports most of Spark’s features such as Spark SQL, DataFrame, Streaming, MLlib (Machine Learning) and Spark Core. Spark SQL is a Spark module for structured data processing. It provides a programming abstraction called DataFrame and can also act as distributed SQL query engine. Running on top of Spark, the streaming feature in Apache Spark enables powerful interactive and analytical applications across both streaming and historical data, while inheriting Spark’s ease of use and fault tolerance characteristics.
Deequ:

Platforms Supported

Windows Not Supported
Mac Not Supported
Linux Not Supported
Cloud Supported
On-Premises Not Supported
iPhone Not Supported
iPad Not Supported
Android Not Supported
Chromebook Not Supported
PySpark:

Platforms Supported

Windows Not Supported
Mac Not Supported
Linux Not Supported
Cloud Supported
On-Premises Not Supported
iPhone Not Supported
iPad Not Supported
Android Not Supported
Chromebook Not Supported
Deequ:

Audience

Anyone looking for an Unit Testing solution that measures data quality in large datasets
PySpark:

Audience

Application development solution for DevOps teams
Deequ:

Support

Phone Support Not Supported
24/7 Live Support Not Supported
Online Supported
PySpark:

Support

Phone Support Not Supported
24/7 Live Support Not Supported
Online Supported
Deequ:

API

Offers API Not Supported
PySpark:

API

Offers API Supported
Deequ:

Screenshots and Videos

PySpark:

Screenshots and Videos

Deequ:

Pricing

No information available.
Free Version Not Supported
Free Trial Not Supported
PySpark:

Pricing

No information available.
Free Version Not Supported
Free Trial Not Supported
Deequ:

Reviews/Ratings

Overall 0.0 / 5
ease 0.0 / 5
features 0.0 / 5
design 0.0 / 5
support 0.0 / 5

This software hasn't been reviewed yet. Be the first to provide a review:

Review this Software
PySpark:

Reviews/Ratings

Overall 0.0 / 5
ease 0.0 / 5
features 0.0 / 5
design 0.0 / 5
support 0.0 / 5

This software hasn't been reviewed yet. Be the first to provide a review:

Review this Software
Deequ:

Training

Documentation Supported
Webinars Not Supported
Live Online Not Supported
In Person Not Supported
PySpark:

Training

Documentation Supported
Webinars Not Supported
Live Online Not Supported
In Person Not Supported
Deequ:

Company Information

Deequ
github.com/awslabs/deequ
PySpark:

Company Information

PySpark
spark.apache.org/docs/latest/api/python/
Deequ:

Alternatives

Spark Streaming

Spark Streaming

Apache Software Foundation
PySpark:

Alternatives

Deequ:
PySpark:
Deequ:
Apache Spark

Apache Spark

Apache Software Foundation
PySpark:
Deequ:
MLlib

MLlib

Apache Software Foundation
PySpark:
Apache Spark

Apache Spark

Apache Software Foundation
Deequ:
Apache Mahout

Apache Mahout

Apache Software Foundation
PySpark:
Spark Streaming

Spark Streaming

Apache Software Foundation
Deequ:

Categories

Unit Testing Supported
PySpark:

Categories

Deequ:

Integrations

Apache Spark Supported
Amazon SageMaker Data Wrangler Not Supported
Comet LLM Not Supported
Feast Not Supported
Fosfor Decision Cloud Not Supported
Tecton Not Supported
Union Pandera Not Supported
PySpark:

Integrations

Apache Spark Supported
Amazon SageMaker Data Wrangler Supported
Comet LLM Supported
Feast Supported
Fosfor Decision Cloud Supported
Tecton Supported
Union Pandera Supported
Deequ:
Claim Deequ and update features and information
Claim Deequ and update features and information
PySpark:
Claim PySpark and update features and information
Claim PySpark and update features and information