Open Source Data Quality Tools - Page 2
Sort By:
Data Quality Tools
View 152 business solutions-
Train ML Models With SQL You Already KnowBuild and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
-
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
1
EZStacking
EZStacking is Jupyter notebook generator for machine learning
EZStacking is Jupyter notebook generator for supervised learning problems using Scikit-Learn pipelines and stacked generalization. EZStacking handles classification and regression problems for structured data. It can also be viewed as a development tool, because a notebook generated with EZStacking contains: -an exploratory data analysis (EDA) used to assess data quality - a modelling producing a reduced-size stacked estimator - a server returning a prediction, a measure of the quality of input data and the execution time. -
2
Encord Active
The toolkit to test, validate, and evaluate your models and surface
Encord Active is an open-source toolkit to test, validate, and evaluate your models and surface, curate, and prioritize the most valuable data for labeling to supercharge model performance. Encord Active has been designed as a all-in-one open source toolkit for improving your data quality and model performance. Use the intuitive UI to explore your data or access all the functionalities programmatically. Discover errors, outliers, and edge-cases within your data - all in one open source toolkit. Get a high level overview of your data distribution, explore it by customizable quality metrics, and discover any anomalies. Use powerful similarity search to find more examples of edge-cases or outliers. -
3
The Functional Genomics Data Society develops standards for biological research data quality, annotation and exchange. We define minimum information specs and create software that builds on these, helping scientists annotate and share data easily.
-
4
Feathr
A scalable, unified data and AI engineering platform for enterprise
Feathr is a data and AI engineering platform that is widely used in production at LinkedIn for many years and was open sourced in 2022. It is currently a project under LF AI & Data Foundation. Define data and feature transformations based on raw data sources (batch and streaming) using Pythonic APIs. Register transformations by names and get transformed data(features) for various use cases including AI modeling, compliance, go-to-market and more. Share transformations and data(features) across team and company. Feathr is particularly useful in AI modeling where it automatically computes your feature transformations and joins them to your training data, using point-in-time-correct semantics to avoid data leakage, and supports materializing and deploying your features for use online in production. -
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
5
FiftyOne
The open-source tool for building high-quality datasets
The open-source tool for building high-quality datasets and computer vision models. Nothing hinders the success of machine learning systems more than poor-quality data. And without the right tools, improving a model can be time-consuming and inefficient. FiftyOne supercharges your machine learning workflows by enabling you to visualize datasets and interpret models faster and more effectively. Improving data quality and understanding your model’s failure modes are the most impactful ways to boost the performance of your model. FiftyOne provides the building blocks for optimizing your dataset analysis pipeline. Use it to get hands-on with your data, including visualizing complex labels, evaluating your models, exploring scenarios of interest, identifying failure modes, finding annotation mistakes, and much more! Surveys show that machine learning engineers spend over half of their time wrangling data, but it doesn't have to be that way. -
6
Gretel Synthetics
Synthetic data generators for structured and unstructured text
Unlock unlimited possibilities with synthetic data. Share, create, and augment data with cutting-edge generative AI. Generate unlimited data in minutes with synthetic data delivered as-a-service. Synthesize data that are as good or better than your original dataset, and maintain relationships and statistical insights. Customize privacy settings so that data is always safe while remaining useful for downstream workflows. Ensure data accuracy and privacy confidently with expert-grade reports. Need to synthesize one or multiple data types? We have you covered. Even take advantage or multimodal data generation. Synthesize and transform multiple tables or entire relational databases. Mitigate GDPR and CCPA risks, and promote safe data access. Accelerate CI/CD workflows, performance testing, and staging. Augment AI training data, including minority classes and unique edge cases. Amaze prospects with personalized product experiences. -
7
A simple little engine to do fuzzy name & address searching. Helps improve data quality and avoids duplicate data entry.
-
8
-
9
MOIRAI
Simple Scientific Workflow System for CAGE Analysis
Cap analysis of gene expression (CAGE) is a sequencing based technology to capture the 5’ ends of RNAs in a biological sample. After mapping, a CAGE peak on the genome indicates the position of an active transcriptional start site (TSS) and the number of reads correspond to its expression level. CAGE is prominently used in both the FANTOM and ENCODE project. MOIRAI is a compact yet flexible workflow system designed to carry out the main steps in data processing and analysis of CAGE data. MOIRAI has a graphical interface allowing wet-lab researchers to create, modify and run analysis workflows. Embedded within the workflows are graphical quality control indicators allowing users assess data quality and to quickly spot potential problems. MOIRAI package comes with three main workflows allowing users to map, annotate and perform an expression analysis over multiple samples. -
$300 in Free Credit Towards Top Cloud ServicesStart your project in minutes. After credits run out, 20+ products include free monthly usage. Only pay when you're ready to scale.
-
10
Medrechaincode
Lifetime medical records in decentralized chain code-network
Medrechaincode uses a blockchain technology to store patient life time medical records in decentralized node with single true version of patient medical records” Medrechaincode provides a consensus based access of patient medical record to different healthcare professionals like R&D labs, doctors, hospital & pharmacist. This platform will also help the healthcare professional to use integrated data quality tool to remove duplicity of patient records , profiling of medical records, metadata discovery , data cleansing , classification of medical records, bucketization, anomaly discovery to find out the anomalous trend of medical records. m This platform will help patient to reduced their consultation time, monetize their medical records, for labs to get the historical records & come up with new medicine and make a clinical trial. -
11
NBi
NBi is a testing framework (add-on to NUnit)
NBi is a testing framework (add-on to NUnit) for Business Intelligence. It supports most of the relational databases (SQL server, MySQL, postgreSQL ...) and OLAP platforms (Analysis Services, Mondrian ...) but also ETL and reporting components (Microsoft technologies). The main goal of this framework is to let users create tests with a declarative approach based on an Xml syntax. By the means of NBi, you don't need to develop C# code to specify your tests! Either, you don't need Visual Studio to compile your test suite. Just create an Xml file and let the framework interpret it and play your tests. The framework is designed as an add-on of NUnit but with the possibility to port it easily to other testing frameworks. -
12
NGS data quality evaluation
Python tool to evaluate the quality of high-throughput sequencing data
ngsdataqeval is a Python tool to evaluate the quality of high-throughput sequencing data, used by Next Generation Sequencing. Unlike other tools that analyze raw data, this is designed to evaluate the quality of the processed reads after mapping to a reference genome. The evaluation is performed in a genomic region defined by the user, and it provides some statistics computed from the reads that map to that region (ie. a single gene). The program provides a graphical output embedded in an html file. The analysis contains the sequencing quality along the reads, the mapping quality distribution, the coverage of the defined region, the overall quality at each nucleotide position, and the distribution of the coverage as a function of the GC content in the reference genome. The results provided by this program can help to distinguish between sequence variation and sequencing errors. -
13
ODD Platform
First open-source data discovery and observability platform
Unlock the power of big data with OpenDataDiscovery Platform. Experience seamless end-to-end insights, powered by unprecedented observability and trust - from ingestion to production - while building your ideal tech stack! Democratize data and accelerate insights. Find data that fits your use case and discover hints left by your peers to leverage existing knowledge. Explore tags, ownership details, links to other sources and other information to shorten and simplify data discovery phase. Forget unnerved stakeholders and wasting too much time on digging the root cause of data issues when it fails. With ODD’s automatic company-wide ingestion-to-product lineage you’ll have answers in just seconds and stakeholders won’t need to wait. Sleep well, knowing all your data is in check. Forget manual testing, days of debugging, and weeks of worrying. Know the impact of each code change with automatic testing. Enjoy lineage and alerts powered with data quality information. -
14
Open Data Profiler is a an open source, extensible data profiler, which enables users to analyze and gather automatically data quality facts on data sources in various formats (XML, JDBC or CSV).
-
15
PrepMS is a simple-to-use graphical application for MS data preprocessing, peak detection, and visual data quality assessment. PrepMS is a compiled stand-alone application. Project homepage: http://stat.tamu.edu/~yuliya/prepMS.html
-
16
Qualitis
Qualitis is a one-stop data quality management platform
Qualitis is a data quality management platform that supports quality verification, notification, and management for various datasource. It is used to solve various data quality problems caused by data processing. Based on Spring Boot, Qualitis submits quality model task to Linkis platform. It provides functions such as data quality model construction, data quality model execution, data quality verification, reports of data quality generation and so on. At the same time, Qualitis provides enterprise-level features of financial-level resource isolation, management and access control. It is also guaranteed working well under high-concurrency, high-performance and high-availability scenarios. -
17
Restful APIs for Data Cleansing
This is sister project for osDQ which provide Restful APIs
(Beta Version) This is sister project for https://sourceforge.net/projects/dataquality/ . It provides Restful APIs for features for data quality and data preparation features. This project will help projects which want embed data quality and data preparation features in their project or UI using restful calls. Data Cleansing APIs Dockerfile: # Pull base image FROM frnde/jetty-9.4.2-jre8-alpine-cet ADD osdq-v0.0.1.war /var/lib/jetty/webapps/osdq.war EXPOSE 8080 Docker Image https://hub.docker.com/r/vreddym/osdq-web/tags -
18
SQLBucket
Lightweight library to write, orchestrate and test your SQL ETL
SQLBucket is a lightweight framework to help write, orchestrate and validate SQL data pipelines. It gives the possibility to set variables and introduces some control flow using the fantastic Jinja2 library. It also implements a very simplistic unit and integration test framework where you can validate the results of your ETL in the form of SQL checks. With SQLBucket, you can apply TDD principles when writing data pipelines. To start working, you need to instantiate your SQLBucket core object with the project_folder parameter. That folder will contain all your SQL ETL. The python file where you create your SQLBucket object is also a good place to instantiate your command line interface. -
19
An integrated pipeline for forensic analysis from SNP panel data. 1. SNP caller takes a FASTQ file and reference SNP panel as input and generates SNP calls. 2. Kinship analysis 3. Ancestry prediction 4. Data quality check 5. Replicate analysis 6. Mixture analysis module available by request
-
20
Like every other software, also for Mashup applications is important to ensure the Data quality in order to have more chance our software works in the desired way. Final goal: work out a software for Mashup data quality check.
-
21
Toolsverse ETL Framework
Open source Extract Transform Load engine written in Java
ETL Framework is a standalone Extract Transform Load engine written in Java. It includes executables for all major platforms and can be easily integrated into other applications. Key Features: * embeddable, open source and free * fast and scalable * uses target database features to do transformations and loads * manual and automatic data mapping * data streaming * bulk data loads * data quality features using SQL, JavaScript? and regex * data transformations Requirements * Java 1.6 and up * At least 4 MB of RAM New in 3.2 (01/18/2013) * Improved auto-update functionality * Bug fixes -
22
To create a framework to extract Web data and store in local RDBMS, to generate assessment reports on quality of the data being extract, and to publish the quality reports on the Web.
-
23
WhyLogs Java Library
Profile and monitor your ML data pipeline end-to-end
This is a Java implementation of WhyLogs, with support for Apache Spark integration for large scale datasets. Understanding the properties of data as it moves through applications is essential to keeping your ML/AI pipeline stable and improving your user experience, whether your pipeline is built for production or experimentation. WhyLogs is an open source statistical logging library that allows data science and ML teams to effortlessly profile ML/AI pipelines and applications, producing log files that can be used for monitoring, alerts, analytics, and error analysis. WhyLogs calculates approximate statistics for datasets of any size up to TB-scale, making it easy for users to identify changes in the statistical properties of a model's inputs or outputs. Using approximate statistics allows the package to run on minimal infrastructure and monitor an entire dataset, rather than miss outliers and other anomalies by only using a sample of the data to calculate statistics. -
24
apache spark data pipeline osDQ
osDQ dedicated to create apache spark based data pipeline using JSON
This is an offshoot project of open source data quality (osDQ) project https://sourceforge.net/projects/dataquality/ This sub project will create apache spark based data pipeline where JSON based metadata (file) will be used to run data processing , data pipeline , data quality and data preparation and data modeling features for big data. This uses java API of apache spark. It can run in local mode also. Get json example at https://github.com/arrahtech/osdq-spark How to run Unzip the zip file Windows : java -cp .\lib\*;osdq-spark-0.0.1.jar org.arrah.framework.spark.run.TransformRunner -c .\example\samplerun.json Mac UNIX java -cp ./lib/*:./osdq-spark-0.0.1.jar org.arrah.framework.spark.run.TransformRunner -c ./example/samplerun.json For those on windows, you need to have hadoop distribtion unzipped on local drive and HADOOP_HOME set. Also copy winutils.exe from here into HADOOP_HOME\bin -
25
