Search Results for "data cleaning"
Sort By:
Showing 27 open source projects for "data cleaning"
View related business solutions-
Save Up to 91% on Cloud Compute With Spot VMsRun batch jobs at 60-91% off with Spot VMs. Long-running workloads get automatic discounts with sustained use.
-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
1
AI Data Science Team
An AI-powered data science team of agents
AI Data Science Team is a Python library and agent ecosystem designed to accelerate and automate common data science workflows by modeling them as specialized AI “agents” that can be orchestrated to perform tasks like data cleaning, transformation, analysis, visualization, and machine learning. It provides a modular agent framework where each agent focuses on a step in the typical data science pipeline — for example, loading data from CSV/Excel files, cleaning and wrangling messy datasets, engineering predictive features, building models with AutoML, connecting to SQL databases, and producing visual outputs — all driven by natural language or programmatic instructions. ... -
2
The Data Engineering Handbook
Links to everything you'd ever want to learn about data engineering
...It includes beginner and intermediate boot camps, interview guides, data cleaning and transformation resources, and curated lists of newsletters and industry communities, making it useful both for self-study and technical interview preparation. The repository is actively maintained and widely starred, reflecting its role as a go-to reference for newcomers and experienced practitioners alike. -
3
Python for Data Analysis
Materials and IPython notebooks for "Python for Data Analysis"
Python for Data Analysis is the official companion repository for Python for Data Analysis, 3rd Edition by Wes McKinney. It contains the datasets, examples, and IPython notebooks used throughout the book. The repository helps readers practice Python data analysis concepts directly in Jupyter Notebook. Its chapters cover Python basics, NumPy, pandas, data loading, cleaning, wrangling, visualization, time series, modeling libraries, and full analysis examples. ... -
4
Agentic Data Scientist
An end-to-end Data Scientist
Agentic Data Scientist is an experimental AI-driven research framework that orchestrates data science workflows through autonomous agents that can reason, plan, and execute complex analytics tasks. Unlike traditional scripted pipelines, this project lets AI agents break down high-level research goals into sub-tasks such as data acquisition, cleaning, modeling, evaluation, and reporting, with minimal human direction. -
Cut Data Warehouse Costs by 54%BigQuery delivers 54% lower TCO with exabyte scale and flexible pricing. Free migration tools handle the SQL translation automatically.
-
5
ExtractThinker
ExtractThinker is a Document Intelligence library for LLMs
ExtractThinker is a tool designed to facilitate the extraction and analysis of information from various data sources, aiding in data processing and knowledge discovery. -
6
nb-clean
Clean Jupyter notebooks of outputs, metadata, and empty cells
nb-clean cleans Jupyter notebooks of cell execution counts, metadata, outputs, and (optionally) empty cells, preparing them for committing to version control. It provides both a Git filter and pre-commit hook to automatically clean notebooks before they're staged, and can also be used with other version control systems, as a command line tool, and as a Python library. It can determine if a notebook is clean or not, which can be used as a check in your continuous integration pipelines.... -
7
DOLMA
Data and tools for generating and inspecting OLMo pre-training data
DOLMA (Data Optimization and Learning for Model Alignment) is a framework designed to manage large-scale datasets for training and fine-tuning language models efficiently. -
8
PandasAI
PandasAI is a Python library that integrates generative AI
PandasAI is a Python library that adds Generative AI capabilities to pandas, the popular data analysis and manipulation tool. It is designed to be used in conjunction with pandas, and is not a replacement for it. PandasAI makes pandas (and all the most used data analyst libraries) conversational, allowing you to ask questions to your data in natural language. For example, you can ask PandasAI to find all the rows in a DataFrame where the value of a column is greater than 5, and it will... -
9
All-in-RAG
Big Model Application Development Practice 1
All-in-RAG is an open-source educational project designed to teach developers how to build applications using retrieval-augmented generation techniques. The repository provides a structured learning path that covers both theoretical foundations and practical implementation steps for RAG systems. It explains the full development pipeline required to create knowledge-aware AI assistants, including data preparation, document indexing, vector embedding generation, and retrieval strategies. The... -
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
10
NeMo Curator
Scalable data pre processing and curation toolkit for LLMs
NeMo Curator is a Python library specifically designed for fast and scalable dataset preparation and curation for large language model (LLM) use-cases such as foundation model pretraining, domain-adaptive pretraining (DAPT), supervised fine-tuning (SFT) and paramter-efficient fine-tuning (PEFT). It greatly accelerates data curation by leveraging GPUs with Dask and RAPIDS, resulting in significant time savings. The library provides a customizable and modular interface, simplifying pipeline... -
11
Open Interpreter
A natural language interface for computers
Open Interpreter is an open-source tool that provides a natural-language interface for interacting with your computer. It lets large language models (LLMs) run code locally (Python, JavaScript, shell, etc.), enabling you to ask your computer to do tasks like data analysis, file manipulation, browsing, etc. in human terms (“chat with your computer”), with safeguards. Runs locally or via configured remote LLM servers/inference backends, giving flexibility to use models you trust or have... -
12
Data Preprocessing Automate
Data Preprocessing Automation: A GUI for easy data cleaning & visualiz
Data Preprocessing Automation is a Python-based GUI application designed to simplify and automate data preprocessing tasks. It allows users to upload Excel files, automatically handle missing values, remove duplicates, and detect and remove outliers using statistical methods. The application provides data visualization tools, including box plots for distribution analysis and scatter plots for exploring relationships between variables. Users can download the processed data for further... -
13
Temp_Cleaner GUI
A free and open-source program to free up disk space
While most of us tend to ignore them, the truth is that browser history, cookies and cache take quite a lot of space on the disk. Deleting them does not only help you gain storage space, but it can also speed up the PC. Temp_Cleaner GUI Project is a simple and straightforward utility that enables you to clean your Windows-based computer of junk and obsolete files. The app comes with a single-window interface packed with a huge list of options. As you probably hinted, all you need to do is... -
14
MB Nexus Cockpit
Hardware Monitor & System Cleaner
⚡ [EN] MB Nexus Cockpit – Lightweight Windows 11 hardware monitor & system cleaner. Zero background services, 0% tracking. ℹ️ UI is currently German only, but fully visual & intuitive. Full English translation planned! Hobby dev here — feedback welcome, test it now! 📊 Hardware Monitor: Live CPU/GPU temps, RAM & voltages with color alerts. 🎮 Game Logger: Records background hardware stats while gaming (Min/Max/Avg). 🧹 Cleaner & SSD: Safe Windows 11 cleanup & NVMe S.M.A.R.T. health... -
15
Streamline Analyst
AI agent that streamlines the entire process of data analysis
Streamline Analyst is a cutting-edge, open-source application powered by Large Language Models (LLMs) designed to revolutionize data analysis. This Data Analysis Agent effortlessly automates all the tasks such as data cleaning, preprocessing, and even complex operations like identifying target objects, partitioning test sets, and selecting the best-fit models based on your data. With Streamline Analyst, results visualization and evaluation become seamless. -
16
sketch
AI code-writing assistant that understands data content
...The tool integrates directly into pandas dataframes through an extension, making it easy to use within existing Python workflows without requiring additional IDE plugins. Sketch supports a variety of tasks including data cleaning, feature engineering, visualization, and exploratory analysis, all driven by simple natural language prompts. It also includes advanced capabilities for generating structured outputs and applying transformations directly to datasets, reducing the need for manual coding. -
17
text-dedup
All-in-one text de-duplication
text-dedup is a Python library that enables efficient deduplication of large text corpora by using MinHash and other probabilistic techniques to detect near-duplicate content. This is especially useful for NLP tasks where duplicated training data can skew model performance. text-dedup scales to billions of documents and offers tools for chunking, hashing, and comparing text efficiently with low memory usage. It supports Jaccard similarity thresholding, parallel execution, and flexible deduplication strategies, making it ideal for cleaning web-scraped data, language model training datasets, or document archives. -
18
paramspider
Mine parameterized URLs from web archives for security testing
...These endpoints are commonly used during reconnaissance because parameters often expose inputs that may be vulnerable to issues like cross-site scripting, SQL injection, or server-side request forgery. ParamSpider automates the process of retrieving archived URLs, cleaning them, and preparing them for fuzzing or further probing. It can process a single domain or multiple domains from a list, making it useful for both targeted testing and large-scale reconnaissance. -
19
funNLP
Resources, corpora, and tools for Chinese natural language processing
...The repository is organized into categories such as sentiment analysis, text classification, named entity recognition, knowledge graphs, and various lexicons (e.g. sensitive words, emotion dictionaries, stopwords). It also includes links to academic papers, open-source model implementations, and practical utilities like word segmentation or text cleaning scripts. The project is highly community-oriented, frequently updated with contributions and new resources, and it’s widely used in both academic and applied NLP research. Its value lies in providing not just tools but also curated, domain-specific data, which can be hard to find elsewhere. -
20
SageMaker Experiments Python SDK
Experiment tracking and metric logging for Amazon SageMaker notebooks
...Each step in the workflow is described by a Trial Component. There is no relationship between Trial Components such as ordering. Trial Component: A description of a single step in a machine learning workflow. For example data cleaning, feature extraction, model training, model evaluation, etc. -
21
Tashkeela processed
Tashkeela dataset cleaned and normalized.
A version of the Tashkeela Arabic diacritized text dataset cleaned from the non-Arabic content and the undiacritized text, then divided into training, development, and testing sets. The cleaning process includes removing the XML tags and strange symbols, as well as fixing diacritics errors. After that, the tokenization is performed while focusing on the extraction of the Arabic words. The result is a space-separated tokens file, where the words and the numbers are separated, but not the... -
22
cocoNLP
A Chinese information extraction tool
cocoNLP is a lightweight natural-language processing toolkit geared toward practical information extraction from raw text, especially for Chinese and mixed Chinese–English content. Instead of requiring a heavy pipeline, it focuses on quick wins such as extracting names, places, organizations, emails, phone numbers, and dates directly from unstructured sentences. The project blends pattern-based methods with NLP heuristics, giving developers dependable results for real-world texts like chats,... -
23
MLBox
MLBox is a powerful Automated Machine Learning python library
MLBox is a powerful Automated Machine Learning python library. Fast reading and distributed data preprocessing/cleaning/formatting. Highly robust feature selection and leak detection. Accurate hyper-parameter optimization in high-dimensional space. State-of-the-art predictive models for classification and regression (Deep Learning, Stacking, LightGBM,...) Prediction with model interpretation. MLBox has been developed and used by many active community members. -
24
fooltrader
Quant framework for stock
Build a standard data schema, and then implement various connectors to import systems you are familiar with for analysis. fooltrader is a quantitative analysis trading system designed using big data technology, including data capture, cleaning, structuring, calculation, display, backtesting and trading. Its goal is to provide a unified framework for the whole market (stock, futures, bonds, foreign exchange, digital currency, macroeconomics, etc.) for research, backtesting, forecasting, and trading. ... -
25
Dual-Boot Filename Filter
Sustain data readibility in a dual-boot Windows-Linux environment
The Dual-Boot Filename Filter makes it easy to sustain data readability in a dual-boot Windows-Linux environment, or even when you want to write your data to a disc. Windows have much stricter rules for file naming than other operating systems, so it is not always easy to create data on Linux, which will be read in Windows. Even burning discs can be tiresome, if they have to be readable in both environments. Common disc burning programs are able to rename your files if their name do...
