Search Results for "dataset" - Page 2
Sort By:
Showing 341 open source projects for "dataset"
View related business solutions-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
Train ML Models With SQL You Already KnowBuild and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
-
1
FlagEmbedding
Retrieval and Retrieval-augmented LLMs
...FlagEmbedding includes a family of models known as BGE (BAAI General Embedding), which are designed to achieve strong performance across multilingual and cross-lingual retrieval benchmarks. The toolkit provides infrastructure for inference, fine-tuning, evaluation, and dataset preparation, enabling developers to train custom embedding models for specific domains or applications. It also includes reranker models that refine search results by re-evaluating candidate documents using cross-encoder architectures, improving retrieval accuracy in complex queries. -
2
Step1X-3D
High-Fidelity and Controllable Generation of Textured 3D Assets
...The result is fully 3D assets — meshes + textures — which can be rendered from any viewpoint, textured consistently, and used in 3D applications. To achieve this, the project includes a massive curated dataset: among more than 5 million candidate 3D assets, it filters and standardizes to produce a high-quality 2 million–asset subset suitable for training. -
3
PyTorch Forecasting
Time series forecasting with PyTorch
PyTorch Forecasting aims to ease state-of-the-art time series forecasting with neural networks for both real-world cases and research alike. The goal is to provide a high-level API with maximum flexibility for professionals and reasonable defaults for beginners. A time series dataset class that abstracts handling variable transformations, missing values, randomized subsampling, multiple history lengths, etc. A base model class that provides basic training of time series models along with logging in tensorboard and generic visualizations such actual vs predictions and dependency plots. Multiple neural network architectures for timeseries forecasting that have been enhanced for real-world deployment and come with in-built interpretation capabilities. ... -
4
Weights and Biases
Tool for visualizing and tracking your machine learning experiments
...Use W&B to visualize results in real time, all in a central dashboard. Focus on the interesting ML. Spend less time manually tracking results in spreadsheets and text files. Capture dataset versions with W&B Artifacts to identify how changing data affects your resulting models. Reproduce any model, with saved code, hyperparameters, launch commands, input data, and resulting model weights. Set wandb.config once at the beginning of your script to save your hyperparameters, input settings (like dataset name or model type), and any other independent variables for your experiments. ... -
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
5
Ultravox
Fast multimodal LLM for real-time voice interaction and AI apps
...It supports use cases such as conversational AI agents, speech-to-speech translation, and analysis of spoken audio content. Ultravox also includes tooling and configuration systems for training, evaluation, and dataset integration. -
6
Materials Discovery: GNoME
AI discovers 520000 stable inorganic crystal structures for research
...The project centers on Graph Networks for Materials Exploration (GNoME), a message-passing neural network architecture trained on density functional theory (DFT) data to predict material stability and energy formation. Using GNoME, DeepMind identified 381,000 new stable materials, later expanding the dataset to include over 520,000 materials within 1 meV/atom of the convex hull as of August 2024. The repository provides datasets, model definitions, and interactive Colabs for exploring these materials, computing decomposition energies, and visualizing chemical families. Additionally, it includes JAX-based implementations of GNoME and Nequip—the latter being used to train interatomic potentials for dynamic simulations. -
7
Agent Apprenticeship
The living ecosystem where AI agents complete tasks
...Apprentice agents can be paired with mentor agents, human reviewers, or domain experts depending on the selected mode. The project supports Codex, Cursor, Claude Code, OpenClaw, OpenCode, Hermes Agent, and custom agent commands. Its seed dataset includes curated tasks, reusable lessons, execution traces, work episodes, and structured experience records. It is designed for long-horizon agent work, workflow evaluation, runtime training, and shared improvement across an agent ecosystem. -
8
Hugging Face Skills
Definitions for AI/ML tasks like dataset creation
Hugging Face Skills is a repository of standardized task definitions that package instructions, scripts, and resources so coding agents can reliably perform AI and machine learning workflows. Each skill is a self-contained folder with structured metadata and guidance that tells an agent how to execute tasks such as dataset creation, model training, evaluation, or Hub operations. The project is designed to be interoperable across major agent ecosystems, including Claude Code, OpenAI Codex, Gemini CLI, and Cursor, making it a cross-platform building block for agent automation. By formalizing best practices and workflows, Skills helps transform general-purpose coding agents into domain-aware assistants that can execute complex ML pipelines with less manual prompting. ... -
9
HyperTools
A Python toolbox for gaining geometric insights
...HyperTools is designed to facilitate dimensionality reduction-based visual explorations of high-dimensional data. The basic pipeline is to feed in a high-dimensional dataset (or a series of high-dimensional datasets) and, in a single function call, reduce the dimensionality of the dataset(s) and create a plot. -
Go from Code to Production URL in SecondsSkip the Kubernetes configs. Cloud Run handles HTTPS, scaling, and infrastructure automatically. Two million requests free per month.
-
10
AWS Data Wrangler
Pandas on AWS, easy integration with Athena, Glue, Redshift, etc.
An AWS Professional Service open-source python initiative that extends the power of Pandas library to AWS connecting DataFrames and AWS data-related services. Easy integration with Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON, and EXCEL). Built on top of other open-source projects like Pandas, Apache Arrow and Boto3, it offers abstracted functions to execute... -
11
Letterboxd Recommendations
Scraping publicly-accessible Letterboxd data for movie recommendations
...Their ratings are then combined with a sample of ratings from the top 4000 most active users on the site to create a collaborative filtering recommender model using singular value decomposition (SVD). All movies in the full dataset that the user has not rated are run through the model for predicted scores and the items with the top predicted scores are returned. Due to constraints in time and computing power, the maximum sample size that a user is allowed to select is 500,000 samples, though there are over five million ratings in the full dataset from the top 4000 Letterboxd users alone. -
12
FramePack
Lets make video diffusion practical
FramePack explores compact representations for sequences of image frames, targeting tasks where many near-duplicate frames carry redundant information. The idea is to “pack” frames by detecting shared structure and storing differences efficiently, which can accelerate training or inference on video-like data. By reducing I/O and memory bandwidth, datasets become lighter to load while models still see the essential temporal variation. The repository demonstrates both packing and unpacking... -
13
LongWriter
Unleashing 10,000+ Word Generation from Long Context LLMs
...Traditional large language models can process large inputs but often struggle to generate long outputs due to limitations in training data and alignment strategies. LongWriter addresses this challenge by introducing a specialized dataset and training approach that encourages models to produce longer responses. The system uses an agent-based pipeline called AgentWrite that decomposes large writing tasks into smaller subtasks, allowing the model to produce long documents section by section. Researchers also created the LongWriter-6k dataset containing thousands of examples with outputs ranging from a few thousand to tens of thousands of words. -
14
Magicoder
Empowering Code Generation with OSS-Instruct
Magicoder is an open-source family of large language models designed specifically for code generation and software development tasks. The project focuses on improving the quality and diversity of code generation by training models with a novel dataset construction approach known as OSS-Instruct. This technique uses open-source code repositories as a foundation for generating more realistic and diverse instruction datasets for training language models. By grounding training data in real open-source examples, Magicoder aims to reduce bias and improve the reliability of code generation results compared to models trained solely on synthetic instructions. ... -
15
MaxKB
Open-source platform for building enterprise-grade agents
...It focuses on practical deployments such as customer support, internal knowledge bases, research assistants, and education, bundling tools for data ingestion, chunking, embedding, retrieval, and answer synthesis. The system exposes flexible tool-use (including MCP), supports multi-model backends, and provides dashboards for dataset management and evaluation. It’s backed by an active org that also builds adjacent ops tooling, and there’s a dedicated documentation repo for configuration and contribution. Community posts describe “self-host your ChatGPT-style assistant” positioning, with integrations and workflows to move from demo to production. Security advisories are tracked publicly, with upgrade guidance when issues arise. -
16
Feast
Feature Store for Machine Learning
...Make features consistently available for training and serving by managing an offline store (to process historical data for scale-out batch scoring or model training), a low-latency online store (to power real-time prediction), and a battle-tested feature server (to serve pre-computed features online). Avoid data leakage by generating point-in-time correct feature sets so data scientists can focus on feature engineering rather than debugging error-prone dataset joining logic. This ensure that future feature values do not leak to models during training. Decouple ML from data infrastructure by providing a single data access layer that abstracts feature storage from feature retrieval, ensuring models remain portable as you move from training models to serving models, from batch model -
17
WhatsApp Beacon
OSINT tool for tracking WhatsApp online status via Web automation
...It uses Selenium automation to interact with the web interface and detect when a target account goes online or offline. By continuously monitoring these changes, WhatsApp Beacon records connectivity patterns and builds a historical dataset of activity sessions. The collected information is stored in logs and a local database, allowing users to review behavioral patterns over time. In addition, the project supports exporting collected data to spreadsheet formats for further analysis or reporting. WhatsApp Beacon is designed to run across multiple operating systems and can operate in the background using headless browser automation. ... -
18
doccano
Open source annotation tool for machine learning practitioners
...So, you can create labeled data for sentiment analysis, named entity recognition, text summarization and so on. Just create a project, upload data and start annotating. You can build a dataset in hours. -
19
dxy-covid-19-crawler
Realtime crawler for COVID-19 outbreak statistics from DXY data
DXY-COVID-19-Crawler is a Python-based project designed to collect real-time COVID-19 infection data from the public dataset provided by Ding Xiang Yuan (DXY). The crawler periodically retrieves pandemic statistics and stores them in a database so that historical changes in the outbreak can be preserved and analyzed later. It was created to make up-to-date infection data more accessible for developers, researchers, and analysts who wanted to build visualizations or conduct data analysis during the early stages of the pandemic. ... -
20
LLaMA-Mesh
Unifying 3D Mesh Generation with Language Models
...By serializing 3D geometry into text tokens, the approach allows existing transformer architectures to generate and interpret 3D models without requiring specialized visual tokenizers. The project includes a supervised fine-tuning dataset composed of interleaved text and mesh data, allowing the model to learn relationships between textual descriptions and 3D structures. As a result, the model can generate mesh models directly from text prompts, explain mesh structures in natural language, or output mixed text-and-mesh sequences. This unified representation enables a single model to operate across both textual and spatial domains. -
21
second-brain-ai-assistant-course
Learn to build your Second Brain AI assistant with LLMs
...Through a series of modules, the project explains how to design data pipelines, build retrieval-augmented generation systems, and implement agent-based reasoning workflows. The course also introduces practical techniques such as dataset generation, model fine-tuning, and deployment strategies for AI applications. Learners build a full system capable of retrieving information from stored resources and generating responses based on that data. -
22
TRELLIS.2
Native and Compact Structured Latents for 3D Generation
TRELLIS.2 is a cutting-edge open-source model and codebase for high-fidelity 3D asset generation from 2D images, developed to push forward the state of the art in image-to-3D generation. At its core is a novel sparse voxel structure called O-Voxel that jointly encodes both geometry and surface appearance, enabling reconstruction and generation of complex 3D shapes with arbitrary topology, open surfaces, and physically based rendering (PBR) textures. The system leverages a large... -
23
UCO3D
Uncommon Objects in 3D dataset
uCO3D is a large-scale 3D vision dataset and toolkit centered on turn-table videos of everyday objects drawn from the LVIS taxonomy. It provides about 170,000 full videos per object instance rather than still frames, along with per-video annotations including object masks, calibrated camera poses, and multiple flavors of point clouds. Each sequence also ships with a precomputed 3D Gaussian Splat reconstruction, enabling fast, differentiable rendering workflows and modern implicit/point-based modeling experiments. ... -
24
Deep Lake
Data Lake for Deep Learning. Build, manage, and query datasets
Deep Lake (or Deeplake, formerly known as Activeloop Hub) is a data lake for deep learning applications. Our open-source dataset format is optimized for rapid streaming and querying of data while training models at scale, and it includes a simple API for creating, storing, and collaborating on AI datasets of any size. It can be deployed locally or in the cloud, and it enables you to store all of your data in one place, ranging from simple annotations to large videos. -
25
Petastorm
Petastorm library enables single machine or distributed training
...This library enables single machine or distributed training and evaluation of deep learning models directly from datasets in Apache Parquet format. Petastorm supports popular Python-based machine learning (ML) frameworks such as Tensorflow, PyTorch, and PySpark. It can also be used from pure Python code. A dataset created using Petastorm is stored in Apache Parquet format. On top of a Parquet schema, petastorm also stores higher-level schema information that makes multidimensional arrays into a native part of a petastorm dataset. Petastorm supports extensible data codecs. These enable a user to use one of the standard data compressions (jpeg, png) or implement her own.