Search Results for "etl."
Sort By:
Showing 39 open source projects for "etl."
View related business solutions-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
Stop Cyber Threats with VM-Series Next-Gen Firewall on AzureGain integrated visibility across all traffic in a single pass. Deploy Palo Alto Networks VM-Series to determine application identity and content while automating security policy updates via rich APIs.
-
1
Ethereum ETL
Python scripts for ETL (extract, transform and load) jobs for Ethereum
Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery. Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases. -
2
Unstract
No-code LLM Platform to launch APIs and ETL Pipelines
...It integrates a visual Prompt Studio environment where users can iteratively design extraction schemas, compare outputs from different models, and monitor costs and accuracy side by side, making it easier to refine prompts and extraction logic before deploying at scale. Unstract supports deploying structured extraction as REST API endpoints or embedding it into data engineering ETL pipelines, which allows it to plug directly into data warehouses, cloud storage, or downstream analytics systems. Its platform works with a broad variety of file types — from PDFs and spreadsheets to images — and includes integrations with databases, cloud storage providers, and vector databases. -
3
AWS Data Wrangler
Pandas on AWS, easy integration with Athena, Glue, Redshift, etc.
...Easy integration with Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON, and EXCEL). Built on top of other open-source projects like Pandas, Apache Arrow and Boto3, it offers abstracted functions to execute usual ETL tasks like load/unload data from Data Lakes, Data Warehouses, and Databases. Convert the column name to be compatible with Amazon Athena and the AWS Glue Catalog. Run a query against AWS CloudWatchLogs Insights and convert the results to Pandas DataFrame. Get QuickSight dashboard ID given a name and fails if there is more than 1 ID associated with this name. ... -
4
NVIDIA Merlin
Library providing end-to-end GPU-accelerated recommender systems
...Each stage of the Merlin pipeline is optimized to support hundreds of terabytes of data, which is all accessible through easy-to-use APIs. For more information, see NVIDIA Merlin on the NVIDIA developer website. Transform data (ETL) for preprocessing and engineering features. Accelerate your existing training pipelines in TensorFlow, PyTorch, or FastAI by leveraging optimized, custom-built data loaders. Scale large deep learning recommender models by distributing large embedding tables that exceed available GPU and CPU memory. Deploy data transformations and trained models to production with only a few lines of code. -
$300 Free Credits to Build on Google CloudStart your next project with $300 in free Google Cloud credit. Spin up VMs, run containers, query petabytes in BigQuery, or build agents with Gemini Enterprise Agent Platform. Once your credits are used, keep building with 20+ always-free tier products including Compute Engine, Cloud Storage, GKE, and Cloud Run functions. No commitment required—just sign up and start building.
-
5
Prefect
Prefect is a workflow orchestration framework
Prefect is an open-source modern workflow orchestration tool for scheduling, monitoring, and managing data workflows and tasks. It enables Python-native pipeline definitions with robust retries, caching, observability, and a powerful UI—ideal for data engineering and ETL processes. -
6
Pathway
Python ETL framework for stream processing, real-time analytics, LLM
Pathway is an open-source framework designed for building real-time data applications using reactive and declarative paradigms. It enables seamless integration of live data streams and structured data into analytical pipelines with minimal latency. Pathway is especially well-suited for scenarios like financial analytics, IoT, fraud detection, and logistics, where high-velocity and continuously changing data is the norm. Unlike traditional batch processing frameworks, Pathway continuously... -
7
Hamilton DAGWorks
Helps scientists define testable, modular, self-documenting dataflow
...As shown below, it results in readable code that can always be visualized. Hamilton loads that definition and automatically builds the DAG for you. Hamilton brings modularity and structure to any Python application moving data: ETL pipelines, ML workflows, LLM applications, RAG systems, BI dashboards, and the Hamilton UI allows you to automatically visualize, catalog, and monitor execution. -
8
Pyper
Concurrent Python made simple
Pyper is a Python-native orchestration and scheduling framework designed for modern data workflows, machine learning pipelines, and any task that benefits from a lightweight DAG-based execution engine. Unlike heavier platforms like Airflow, Pyper aims to remain lean, modular, and developer-friendly, embracing Pythonic conventions and minimizing boilerplate. It focuses on local development ergonomics and seamless transition to production environments, making it ideal for small teams and... -
9
Superduper
Superduper: Integrate AI models and machine learning workflows
...Developers may leverage Superduper by building compositional and declarative objects that out-source the details of deployment, orchestration versioning, and more to the Superduper engine. This allows developers to completely avoid implementing MLOps, ETL pipelines, model deployment, data migration, and synchronization. Using Superduper is simply "CAPE": Connect to your data, apply arbitrary AI to that data, package and reuse the application on arbitrary data, and execute AI-database queries and predictions on the resulting AI outputs and data. -
Our Free Plans just got better! | Auth0You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
-
10
Trellis AI
All-in-one AI framework & toolkit for Claude Code & Cursor
...Trellis also includes tooling for monitoring, scheduling, and tracing the execution of complex multi-step jobs, helping teams maintain visibility into how work progresses and where bottlenecks emerge. The platform can integrate with external services, databases, and model endpoints, making it suitable for automation, ETL pipelines, AI-driven processes, and business logic orchestration. -
11
Instill Core
Instill Core is a full-stack AI infrastructure tool for data
...It provides an end-to-end solution that enables developers to build, deploy, and manage AI-powered applications without needing to manually stitch together multiple tools across the data and model lifecycle. The platform focuses heavily on handling unstructured data such as documents, images, audio, and video, transforming them into AI-ready formats through integrated ETL pipelines and processing workflows. Instill Core includes modular components such as pipelines, artifacts, and model services, which work together to enable flexible and scalable AI system design. It also supports retrieval-augmented generation workflows and model deployment without requiring complex GPU infrastructure management. -
12
DocETL
A system for agentic LLM-powered data processing and ETL
DocETL is an open-source system designed to build and execute data processing pipelines powered by large language models, particularly for analyzing complex collections of documents and unstructured datasets. The platform allows developers and researchers to construct structured workflows that extract, transform, and organize information from sources such as reports, transcripts, legal documents, and other text-heavy data. Instead of relying on single prompts or ad-hoc scripts, DocETL... -
13
AWS SDK for pandas
Easy integration with Athena, Glue, Redshift, Timestream, Neptune
...The library abstracts efficient patterns like partitioning, compression, and vectorized I/O so you get performant data lake operations without hand-rolling boilerplate. It also supports Redshift, OpenSearch, and other services, enabling ETL tasks that blend SQL engines and Python transformations. Operational helpers handle IAM, sessions, and concurrency while exposing knobs for encryption, versioning, and catalog consistency. The result is a productive workflow that keeps your analytics in Python while leveraging AWS-native storage and query engines at scale. -
14
Datapipe
Real-time, incremental ETL library for ML with record-level depend
Datapipe is a real-time, incremental ETL library for Python with record-level dependency tracking. Datapipe is designed to streamline the creation of data processing pipelines. It excels in scenarios where data is continuously changing, requiring pipelines to adapt and process only the modified data efficiently. This library tracks dependencies for each record in the pipeline, ensuring minimal and efficient data processing. -
15
# CSV Splitter Uma ferramenta para dividir arquivos CSV em múltiplos arquivos com base na quantidade de registros especificada, mantendo a integridade dos dados e permitindo configurações de charset, separador e formatação. Ideal para lidar com grandes arquivos CSV que precisam ser fragmentados para melhor manuseio e processamento. ## Funcionalidades - **Divisão de CSV**: Divide o arquivo original em múltiplos arquivos CSV, com o número de registros por arquivo definido pelo...
-
16
Mara Pipelines
A lightweight opinionated ETL framework, halfway between plain scripts
This package contains a lightweight data transformation framework with a focus on transparency and complexity reduction. Data integration pipelines as code: pipelines, tasks and commands are created using declarative Python code. PostgreSQL as a data processing engine. Extensive web ui. The web browser as the main tool for inspecting, running and debugging pipelines. GNU make semantics. Nodes depend on the completion of upstream nodes. No data dependencies or data flows. No in-app data... -
17
Originally a reimplementation of OpenGroupware's ZideStore. While compatible with legacy ZideStore Coils provides a sophisticated workflow system with ETL and integration capabilities and superior WebDAV/CalDAV features and compatibility. The workflow engine suppports processes described in BPML and provides integration with a variety of services include SSH, LPD, LDAP, and relation databases.
-
18
Tributary
Streaming reactive and dataflow graphs in Python
Tributary is a library for constructing dataflow graphs in Python. Unlike many other DAG libraries in Python (airflow, luigi, prefect, dagster, dask, kedro, etc), tributary is not designed with data/etl pipelines or scheduling in mind. Instead, tributary is more similar to libraries like mdf, loman, pyungo, streamz, or pyfunctional, in that it is designed to be used as the implementation for a data model. One such example is the greeks library, which leverages tributary to build data models for options pricing. -
19
SQLBucket
Lightweight library to write, orchestrate and test your SQL ETL
...It gives the possibility to set variables and introduces some control flow using the fantastic Jinja2 library. It also implements a very simplistic unit and integration test framework where you can validate the results of your ETL in the form of SQL checks. With SQLBucket, you can apply TDD principles when writing data pipelines. To start working, you need to instantiate your SQLBucket core object with the project_folder parameter. That folder will contain all your SQL ETL. The python file where you create your SQLBucket object is also a good place to instantiate your command line interface. -
20
cocoNLP
A Chinese information extraction tool
...The project blends pattern-based methods with NLP heuristics, giving developers dependable results for real-world texts like chats, comments, and user-generated content. Its API is intentionally simple, so you can drop it into scripts, ETL jobs, or dashboards without deep ML expertise. Because it aims at utility over complexity, it’s useful for prototyping data products or building lightweight text analytics where large models would be overkill. The repository also includes examples and test snippets to help you understand expected inputs and typical outputs, which shortens the learning curve for newcomers. -
21
CSV*Extractor for RDBMS (command line)
Extract table data is CSV format from 14 databases.
Spools data for a given query or table from 14 databases. Windows command-line application. -
22
Data Migrator for Oracle
Migrate/Copy your data between Oracle database and 13 major DBs.
Command line data Copy/Migration tool for Oracle. Supports Oracle 7.3, Oracle 8i, Oracle 9i, Oracle 10G, Oracle 11G and 13 major databases. 1. Exadata 2. Sybase ASE 3. Informix Innovator C 4. Sybase SQL Anywhere 5. DB2 UDB 6. CSV 7. SQLServer 8. MariaDB 9. Sybase IQ 10. PostgreSQL 11. MySQL 12. Informix IDS 13. TimesTen -
23
CSV*Loader PRO (Windows command line)
Loads CSV file to14 databases
Windows command line tool for CSV data load to 14 relational stores. -
24
CSV*Extractor Pro (Windows command line)
Spool you scalar data in CSV format from 14 major Databases.
Command line tool for data export from major relational data stores (RDBMS). DB2 Advanced Enterprise Server DB2 Advanced Workgroup Server DB2 Developer Edition DB2 Enterprise Server DB2 Express DB2 Express C DB2 Workgroup Server Exadata Infobright Informix IDS Informix Innovator C MariaDB MySQL Oracle Oracle XE PostgreSQL SAP Sybase ASE SQL Lite SQL Server Enterprise SQL Server Express Sybase IQ Sybase SQL Anywhere TimesTen -
25
Data Migration Tools for RDBMS
DataMigrator for 14 major databases
Touch and go Windows command line data migration tool for 14 databases: 1. Sybase ASE 2. Informix Innovator C 3. Sybase SQL Anywhere 4. DB2 UDB 5. SQLServer 6. MariaDB 7. Sybase IQ 8. PostgreSQL 9. MySQL 10. Informix IDS 11. TimesTen 12. Oracle 13. SQL Lite 14. Exadata