Search Results for "data integration"
Sort By:
Showing 72 open source projects for "data integration"
View related business solutions-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
Paessler: Easy to Use With Enterprise Power. Free TrialYou shouldn't have to choose between a monitoring tool that's easy to use and one that's powerful enough for a complex environment. PRTG's low-code interface lets any admin build dashboards, set alerts and monitor devices without scripting, while custom sensors and full API access are there when your team needs deeper control. One platform, no compromise. Download a free 30-day trial now.
-
1
AWS Data Wrangler
Pandas on AWS, easy integration with Athena, Glue, Redshift, etc.
An AWS Professional Service open-source python initiative that extends the power of Pandas library to AWS connecting DataFrames and AWS data-related services. Easy integration with Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON, and EXCEL). Built on top of other open-source projects like Pandas, Apache Arrow and Boto3, it offers abstracted functions to execute usual ETL tasks like load/unload data from Data Lakes, Data Warehouses, and Databases. ... -
2
AI Data Science Team
An AI-powered data science team of agents
AI Data Science Team is a Python library and agent ecosystem designed to accelerate and automate common data science workflows by modeling them as specialized AI “agents” that can be orchestrated to perform tasks like data cleaning, transformation, analysis, visualization, and machine learning. It provides a modular agent framework where each agent focuses on a step in the typical data science pipeline — for example, loading data from CSV/Excel files, cleaning and wrangling messy datasets,... -
3
Airbyte
Data integration platform for ELT pipelines from APIs, databases
...Our teams process more than 300 billion rows each month for ambitious businesses of all sizes. Enable your data engineering teams to focus on projects that are more valuable to your business. Building and maintaining custom connectors have become 5x easier with Airbyte. With an average response rate of 10 minutes or less and a Customer Satisfaction score of 96/100, our team is ready to support your data integration journey all over the world. -
4
Positron
Positron, a next-generation data science IDE
...The IDE supports notebook and script workflows, integration of data-app frameworks (such as Shiny, Streamlit, Dash), database and cloud connections, and built-in AI-assisted capabilities to help write code, explore data, and build models. -
Save Up to 91% on Cloud Compute With Spot VMsRun batch jobs at 60-91% off with Spot VMs. Long-running workloads get automatic discounts with sustained use.
-
5
CellTypist
A tool for semi-automatic cell type classification, harmonization
CellTypist is an automated tool for cell type classification, harmonization, and integration. Classification, transfer cell type labels from the reference to query dataset. Harmonization, match and harmonize cell types defined by independent datasets. integration, integrate cell and cell types with supervision from harmonization. CellTypist recapitulates cell type structure and biology of independent datasets. Regularised linear models with Stochastic Gradient Descent provide a fast and... -
6
Recap
Recap tracks and transform schemas across your whole application
Recap is a schema language and multi-language toolkit to track and transform schemas across your whole application. Your data passes through web services, databases, message brokers, and object stores. Recap describes these schemas in a single language, regardless of which system your data passes through. Recap schemas can be defined in YAML, TOML, JSON, XML, or any other compatible language. -
7
Pathway
Python ETL framework for stream processing, real-time analytics, LLM
Pathway is an open-source framework designed for building real-time data applications using reactive and declarative paradigms. It enables seamless integration of live data streams and structured data into analytical pipelines with minimal latency. Pathway is especially well-suited for scenarios like financial analytics, IoT, fraud detection, and logistics, where high-velocity and continuously changing data is the norm. -
8
Dagster
An orchestration platform for the development, production
Dagster is an orchestration platform for the development, production, and observation of data assets. Dagster as a productivity platform: With Dagster, you can focus on running tasks, or you can identify the key assets you need to create using a declarative approach. Embrace CI/CD best practices from the get-go: build reusable components, spot data quality issues, and flag bugs early. Dagster as a robust orchestration engine: Put your pipelines into production with a robust... -
9
Neuroglancer
WebGL-based viewer for volumetric data
...The viewer is built with a multi-threaded architecture, separating rendering and data processing to ensure smooth performance even with massive datasets. Extensively used in neuroscience research, Neuroglancer supports integration with tools. -
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
10
Lithops
A multi-cloud framework for big data analytics
...It abstracts cloud providers like IBM Cloud, AWS, Azure, and Google Cloud into a unified interface and turns your Python functions into scalable, event-driven workloads. Lithops is ideal for data processing, ML inference, and embarrassingly parallel workloads, giving you the power of FaaS (Function-as-a-Service) without vendor lock-in. It also supports hybrid cloud setups, object storage access, and simple integration with Jupyter notebooks. -
11
harmonypy
Integrate multiple high-dimensional datasets with fuzzy k-means
Harmony is an algorithm for integrating multiple high-dimensional datasets. harmonypy is a port of the harmony R package by Ilya Korsunsky. Harmony is a general-purpose R package with an efficient algorithm for integrating multiple data sets. It is especially useful for large single-cell datasets such as single-cell RNA-seq. -
12
AWS SDK for pandas
Easy integration with Athena, Glue, Redshift, Timestream, Neptune
aws-sdk-pandas (formerly AWS Data Wrangler) bridges pandas with the AWS analytics stack so DataFrames flow seamlessly to and from cloud services. With a few lines of code, you can read from and write to Amazon S3 in Parquet/CSV/JSON/ORC, register tables in the AWS Glue Data Catalog, and query with Amazon Athena directly into pandas. The library abstracts efficient patterns like partitioning, compression, and vectorized I/O so you get performant data lake operations without hand-rolling... -
13
JS Analyzer
Burp Suite extension for JavaScript static analysis
JS Analyzer is a powerful static analysis tool implemented as a Burp Suite extension that helps security researchers and web developers automatically uncover important artifacts in JavaScript files during web application testing. It parses JavaScript responses intercepted by Burp Suite and intelligently extracts API endpoints, full URLs (including cloud storage links), secrets like API keys or tokens, and email addresses while filtering out noise from irrelevant code patterns. The extension... -
14
AutoHedge
Build your autonomous hedge fund in minutes
...It is built around the concept of algorithmic hedging, where strategies are executed programmatically rather than manually, enabling faster and more consistent responses to market changes. The system supports integration with financial data sources, allowing it to process real-time or historical data for analysis and strategy execution. It also emphasizes modularity, enabling developers to customize strategies, risk parameters, and decision logic. AutoHedge is particularly useful for experimentation and research in algorithmic trading and financial automation. ... -
15
VisPy
Main repository for Vispy
Vispy is an open-source, high-performance interactive visualization library in Python, designed for creating scientific visualizations and interactive plots. It leverages the power of modern Graphics Processing Units (GPUs) through OpenGL to render large datasets efficiently. Vispy supports a wide range of visualization types, including 2D plots, 3D visualizations, volume rendering, and more, making it suitable for scientific research, data analysis, and educational purposes. -
16
Apache Airflow Provider
Great Expectations Airflow operator
Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+. If your Airflow version is 2.1.0, and you want to install this provider version, first upgrade Airflow to at least version 2.1.0. Otherwise, your Airflow package version will be upgraded automatically, and you will have to manually run airflow upgrade db to complete the migration. This operator currently works with the Great Expectations V3 Batch Request API only. If you would like to use the... -
17
PyVista
3D plotting and mesh analysis through a streamlined interface
3D plotting and mesh analysis through a streamlined interface for the Visualization Toolkit (VTK). PyVista is a helper module for the Visualization Toolkit (VTK) that takes a different approach on interfacing with VTK through NumPy and direct array access. This package provides a Pythonic, well-documented interface exposing VTK’s powerful visualization backend to facilitate rapid prototyping, analysis, and visual integration of spatially referenced datasets. This module can be used for... -
18
Cal.diy
Scheduling infrastructure for absolutely everyone
...Built with modern web technologies such as Next.js, React, and PostgreSQL, it offers a robust foundation for building custom scheduling workflows. The system is designed for developers and advanced users who prefer to host and secure their own data rather than depend on third-party providers. It also supports integration with external tools through APIs and authentication systems, allowing flexible customization. -
19
Open Wearables
Self-hosted platform to unify wearable health data
Open Wearables is an open-source initiative that aims to provide a community-driven ecosystem for wearable device software and interoperability by connecting sensor data, activity tracking, and health insights across multiple platforms and devices. Instead of relying on closed vendor ecosystems, the project provides standardized data models and APIs that let developers and hobbyists collect, sync, and analyze biometric and environmental data from wearables, DIY sensors, and open hardware... -
20
Luigi
Python module that helps you build complex pipelines of batch jobs
Luigi is a Python (3.6, 3.7, 3.8, 3.9 tested) package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more. The purpose of Luigi is to address all the plumbing typically associated with long-running batch processes. You want to chain many tasks, automate them, and failures will happen. These tasks can be anything, but are typically long running things like Hadoop jobs, dumping data to/from databases, running machine learning algorithms, or anything else. ... -
21
LinkdAPI Python
Powerful, lightweight B2B Data API for enriching professional profiles
LinkdAPI Python is a software development kit designed to simplify integration with the LinkdAPI platform, enabling developers to connect applications with external services through a unified API layer. It provides structured tools, libraries, and documentation that streamline authentication, request handling, and data exchange between systems. The SDK is built to reduce the complexity of working with multiple APIs by offering standardized methods for interacting with different endpoints. ... -
22
OpenBB Terminal
Investment research for everyone, anywhere
Fully written in python which is one of the most used programming languages due to its simplified syntax and shallow learning curve. It is the first time in history that users, regardless of their background, can so easily add features to an investment research platform. The MIT Open Source license allows any user to fork the project to either add features to the broader community or create their own customized terminal version. The terminal allows for users to import their own proprietary... -
23
SleepFM-Clinical
Improve human sleep through scientifically
SleepFM-Clinical is a specialized version of SleepFM designed for clinical and research environments, offering an adaptive audio modulation system aimed at improving human sleep through scientifically guided soundscapes. Rather than simply playing static white noise or ambient tracks, it uses a closed-loop, frequency-modulated framework that responds to user-specific sleep patterns and physiological signals to tailor sound in ways that can enhance sleep onset and depth. The clinical release... -
24
CUDOS Framework
Command Line Interface tool for Cloud Intelligence Dashboards
The AWS Cloud Intelligence Dashboards Framework is a set of open-source tools and templates designed to help organizations deploy and manage advanced data visualization dashboards that offer insights into cost, usage, governance, and operational health across AWS environments. It is part of the AWS Solutions Library and includes CloudFormation templates, CLI commands, and pre-built dashboards that collect, process, and visualize data from AWS billing, cost management, budgets, and usage... -
25
Blueprint MCP
Diagram generation for understanding codebases and system architecture
Blueprint MCP is a modular control plane designed for managing and orchestrating multiple game-server clusters in real time, giving operators fine-grained control over scaling, configuration, and deployment workflows across distributed infrastructure. It provides a central management REST API and dashboard where teams can view cluster health, adjust instance fleets, set auto-scaling policies, and monitor usage metrics in a unified interface. Blueprint-MCP also supports templated server...
