Search Results for "data cleaning"
Sort By:
Showing 61 open source projects for "data cleaning"
View related business solutions-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
Demo Series - Small Business Backup By VeeamWatch this on-demand demo series and learn how to protect your Microsoft 365 data with clear, simple, actionable steps that are easy to implement for businesses of all sizes.
-
1
NYC Taxi Data
Import public NYC taxi and for-hire vehicle (Uber, Lyft)
The nyc-taxi-data repository is a rich dataset and exploratory project around New York City taxi trip records. It collects and preprocesses large-scale trip datasets (fares, pickup/dropoff, timestamps, locations, passenger counts) to enable data analysis, modeling, and visualization efforts. The project includes scripts and notebooks for cleaning and filtering the raw data, memory-efficient processing for large CSV/Parquet files, and aggregation workflows (e.g. trips per hour, heatmaps of pickups/dropoffs). ... -
2
AI Data Science Team
An AI-powered data science team of agents
AI Data Science Team is a Python library and agent ecosystem designed to accelerate and automate common data science workflows by modeling them as specialized AI “agents” that can be orchestrated to perform tasks like data cleaning, transformation, analysis, visualization, and machine learning. It provides a modular agent framework where each agent focuses on a step in the typical data science pipeline — for example, loading data from CSV/Excel files, cleaning and wrangling messy datasets, engineering predictive features, building models with AutoML, connecting to SQL databases, and producing visual outputs — all driven by natural language or programmatic instructions. ... -
3
The Data Engineering Handbook
Links to everything you'd ever want to learn about data engineering
...It includes beginner and intermediate boot camps, interview guides, data cleaning and transformation resources, and curated lists of newsletters and industry communities, making it useful both for self-study and technical interview preparation. The repository is actively maintained and widely starred, reflecting its role as a go-to reference for newcomers and experienced practitioners alike. -
4
Python for Data Analysis
Materials and IPython notebooks for "Python for Data Analysis"
Python for Data Analysis is the official companion repository for Python for Data Analysis, 3rd Edition by Wes McKinney. It contains the datasets, examples, and IPython notebooks used throughout the book. The repository helps readers practice Python data analysis concepts directly in Jupyter Notebook. Its chapters cover Python basics, NumPy, pandas, data loading, cleaning, wrangling, visualization, time series, modeling libraries, and full analysis examples. ... -
Go from Code to Production URL in SecondsSkip the Kubernetes configs. Cloud Run handles HTTPS, scaling, and infrastructure automatically. Two million requests free per month.
-
5
Agentic Data Scientist
An end-to-end Data Scientist
Agentic Data Scientist is an experimental AI-driven research framework that orchestrates data science workflows through autonomous agents that can reason, plan, and execute complex analytics tasks. Unlike traditional scripted pipelines, this project lets AI agents break down high-level research goals into sub-tasks such as data acquisition, cleaning, modeling, evaluation, and reporting, with minimal human direction. -
6
janitor
Simple tools for data cleaning in R
janitor provides simple, convenient tools for data cleaning, formatting, and exploration in R. It is especially useful for cleaning messy data frames, removing duplicates, formatting column names, and producing frequency tables in a tidy workflow. -
7
Perfect Roadmap To Learn Data Science
Basic To Intermediate Python data science guide
...What makes it particularly valuable is its holistic nature: rather than focusing only on modeling or theory, it also addresses the broader lifecycle of data-science work, data ingestion, cleaning, EDA, feature engineering, model building, validation, deployment, etc. -
8
litlyx
Analytics for developers, setup Analytics in 30 seconds
The easiest, developer-centric analytics tool. Litlyxis an open-source, self-hostable analytics solution for the modern framework. Litlyx offers a unique eyewear cleaning system that includes a special cleaning solution and reusable microfiber swabs. This system is designed to provide a more thorough and eco-friendly way to clean glasses, lenses, and screens. The brand emphasizes sustainability by reducing single-use plastics and promoting long-term use of their products. Their cleaning kit... -
9
Kudu
Free Windows, Mac and Linux cleaner, scanner, and more
...Real-time monitoring displays processor, memory, storage, network, and drive health information. Kudu also supports scheduled scans, restore points, cleaning history, one-click maintenance, command-line automation, and an interface translated into 30 languages. -
99.99% Uptime for MySQL and PostgreSQL DatabasesCloud SQL Enterprise Plus delivers near-zero downtime with 35 days of point-in-time recovery. Supports MySQL, PostgreSQL, and SQL Server.
-
10
ExtractThinker
ExtractThinker is a Document Intelligence library for LLMs
ExtractThinker is a tool designed to facilitate the extraction and analysis of information from various data sources, aiding in data processing and knowledge discovery. -
11
MeshLab
The open source mesh processing system
...The open source system for processing and editing 3D triangular meshes. It provides a set of tools for editing, cleaning, healing, inspecting, rendering, texturing and converting meshes. It offers features for processing raw data produced by 3D digitization tools/devices and for preparing models for 3D printing. -
12
CSV Lint
CSV Lint plug-in for Notepad++ for syntax highlighting
CSV Lint plug-in for Notepad++ for syntax highlighting, csv validation, automatic column and datatype detecting fixed width datasets, change datetime format, decimal separator, sort data, count unique values, convert to xml, json, sql etc. A plugin for data cleaning and working with messy data files. Use CSV Lint for metadata discovery, technical data validation, and reformatting on tabular data files. It is not meant to be a replacement for spreadsheet programs like Excel or SPSS, but rather it's a quality control tool to examine, verify or polish up a dataset before further processing. -
13
nb-clean
Clean Jupyter notebooks of outputs, metadata, and empty cells
nb-clean cleans Jupyter notebooks of cell execution counts, metadata, outputs, and (optionally) empty cells, preparing them for committing to version control. It provides both a Git filter and pre-commit hook to automatically clean notebooks before they're staged, and can also be used with other version control systems, as a command line tool, and as a Python library. It can determine if a notebook is clean or not, which can be used as a check in your continuous integration pipelines.... -
14
FDUPES
FDUPES is a program for identifying or deleting duplicate files
...Because it operates directly on file content rather than just filenames, fdupes can accurately detect true copies and guide cleaning operations in data cleanup or migration tasks. It’s a simple, efficient, and widely used utility on Unix-like systems, appreciated by administrators, developers, and power users. -
15
DOLMA
Data and tools for generating and inspecting OLMo pre-training data
DOLMA (Data Optimization and Learning for Model Alignment) is a framework designed to manage large-scale datasets for training and fine-tuning language models efficiently. -
16
Crowbook LaTeX
Converts books written in Markdown to HTML, LaTeX/PDF and EPUB
Crowbook's aim is to allow you to write a book in Markdown without worrying about formatting or typography and let the program generate HTML, PDF and EPUB output for you. Its focus is novels and fiction, and the default settings should (hopefully) generate readable books with correct typography without requiring you to worry about it. -
17
Java Tablesaw
Java dataframe and visualization library
Tablesaw is a dataframe and visualization library that supports loading, cleaning, transforming, filtering, and summarizing data. If you work with data in Java, it may save you time and effort. Tablesaw also supports descriptive statistics and can be used to prepare data for working with machine learning libraries like Smile, Tribuo, H20.ai, DL4J. Import data from RDBMS, Excel, CSV, TSV, JSON, HTML, or Fixed Width text files, whether they are local or remote (http, S3, etc.) ... -
18
PandasAI
PandasAI is a Python library that integrates generative AI
PandasAI is a Python library that adds Generative AI capabilities to pandas, the popular data analysis and manipulation tool. It is designed to be used in conjunction with pandas, and is not a replacement for it. PandasAI makes pandas (and all the most used data analyst libraries) conversational, allowing you to ask questions to your data in natural language. For example, you can ask PandasAI to find all the rows in a DataFrame where the value of a column is greater than 5, and it will... -
19
All-in-RAG
Big Model Application Development Practice 1
All-in-RAG is an open-source educational project designed to teach developers how to build applications using retrieval-augmented generation techniques. The repository provides a structured learning path that covers both theoretical foundations and practical implementation steps for RAG systems. It explains the full development pipeline required to create knowledge-aware AI assistants, including data preparation, document indexing, vector embedding generation, and retrieval strategies. The... -
20
NeMo Curator
Scalable data pre processing and curation toolkit for LLMs
NeMo Curator is a Python library specifically designed for fast and scalable dataset preparation and curation for large language model (LLM) use-cases such as foundation model pretraining, domain-adaptive pretraining (DAPT), supervised fine-tuning (SFT) and paramter-efficient fine-tuning (PEFT). It greatly accelerates data curation by leveraging GPUs with Dask and RAPIDS, resulting in significant time savings. The library provides a customizable and modular interface, simplifying pipeline... -
21
Miller
Miller is like awk, sed, cut, join, and sort for name-indexed data
Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed. With Miller, you get to use named fields without needing to count positional indices, using familiar formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed. Then, on the fly, you can add new fields which are functions of existing fields, drop fields, sort, aggregate statistically, pretty-print, and more. Miller operates on key-value-pair data while the... -
22
Open Interpreter
A natural language interface for computers
Open Interpreter is an open-source tool that provides a natural-language interface for interacting with your computer. It lets large language models (LLMs) run code locally (Python, JavaScript, shell, etc.), enabling you to ask your computer to do tasks like data analysis, file manipulation, browsing, etc. in human terms (“chat with your computer”), with safeguards. Runs locally or via configured remote LLM servers/inference backends, giving flexibility to use models you trust or have... -
23
stylish-haskell
Haskell code prettifier
A simple Haskell code prettifier. The goal is not to format all of the code in a file since I find that kind of tools often "get in the way". However, manually cleaning up import statements, etc. gets tedious very quickly. This tool tries to help where necessary without getting in the way. Aligns and sorts import statements. Groups and wraps {-# LANGUAGE #-} pragmas, can remove (some) redundant pragmas. Removes trailing whitespace. Aligns branches in case and fields in records. Converts line endings (customizable) Replaces tabs by four spaces (turned off by default) Replaces some ASCII sequences by their Unicode equivalents (turned off by default) Format data constructors and fields in records. -
24
LLM Datasets
Curated list of datasets and tools for post-training
...Licensing and provenance are surfaced to encourage compliant usage and to guide dataset selection in commercial settings. For practitioners, the repo is a practical “starting pantry” that accelerates experimentation and helps keep data wrangling from dominating the project timeline. -
25
Automated Tool for Optimized Modelling
Automated Tool for Optimized Modelling
During the exploration phase of a machine learning project, a data scientist tries to find the optimal pipeline for his specific use case. This usually involves applying standard data cleaning steps, creating or selecting useful features, trying out different models, etc. Testing multiple pipelines requires many lines of code, and writing it all in the same notebook often makes it long and cluttered.
