Search Results for "data processing"
Sort By:
251 projects for "data processing" with 2 filters applied:
-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
1
Algorithms & data structures project
A collection of algorithms and data structures
Algorithms is an educational collection of data structures and algorithms implemented primarily in Java. Its goal is to demonstrate common techniques through clear and elegant reference implementations. The repository covers trees, heaps, hash tables, graphs, dynamic programming, geometry, searching, sorting, strings, and many other topics. Complexity information is documented for many algorithms to help users understand performance characteristics. Implementations can be compiled with a... -
2
Kapacitor
Open source framework for processing, monitoring, and alerting
Open source framework for processing, monitoring, and alerting on time series data. Kapacitor is a real-time data processing engine for monitoring and alerting, specifically designed to work with time-series data from InfluxDB. -
3
NYC Taxi Data
Import public NYC taxi and for-hire vehicle (Uber, Lyft)
The nyc-taxi-data repository is a rich dataset and exploratory project around New York City taxi trip records. It collects and preprocesses large-scale trip datasets (fares, pickup/dropoff, timestamps, locations, passenger counts) to enable data analysis, modeling, and visualization efforts. The project includes scripts and notebooks for cleaning and filtering the raw data, memory-efficient processing for large CSV/Parquet files, and aggregation workflows (e.g. trips per hour, heatmaps of pickups/dropoffs). ... -
4
Siddhi Core Libraries
Stream Processing and Complex Event Processing Engine
Fully open source, cloud-native, scalable, micro streaming, and complex event processing system capable of building event-driven applications for use cases such as real-time analytics, data integration, notification management, and adaptive decision-making. Event processing logic can be written using Streaming SQL queries via graphical and source editors, to capture events from diverse data sources, process and analyze them, integrate with multiple services and data stores, and publish output to various endpoints in real time. ... -
Demo Series - Small Business Backup By VeeamWatch this on-demand demo series and learn how to protect your Microsoft 365 data with clear, simple, actionable steps that are easy to implement for businesses of all sizes.
-
5
Apache Spark
A unified analytics engine for large-scale data processing
Apache Spark is a unified engine for large-scale data processing, offering APIs for batch jobs, streaming, machine learning, and graph computation. It builds on resilient distributed datasets (RDDs) and the newer DataFrame/Dataset abstractions to provide fault-tolerant, in-memory computation across clusters. Spark’s execution engine handles scheduling, shuffles, caching, and data locality so users can focus on transformations rather than infrastructure plumbing. ... -
6
lxml
The lxml XML toolkit for Python
A Python library for efficient XML and HTML processing, known for speed and compatibility. The lxml XML toolkit is a Pythonic binding for the C libraries libxml2 and libxslt. It is unique in that it combines the speed and XML feature completeness of these libraries with the simplicity of a native Python API, mostly compatible but superior to the well-known ElementTree API. The latest release works with all CPython versions from 3.6 to 3.12. See the introduction for more information about the... -
7
Broadway
Concurrent and multi-stage data ingestion and data processing
Broadway is a data processing library for Elixir designed to handle high-throughput, concurrent workloads with ease. It provides an abstraction for defining pipelines that consume data from sources like RabbitMQ, Kafka, Amazon SQS, or custom producers. Each pipeline is fault-tolerant and backpressure-aware, ensuring stable throughput even under load. -
8
Hacks
A collection of hacks and one-off scripts
Hacks is a collection of experimental scripts, utilities, and one-off tools created to solve specific problems in security research, data processing, and automation. Rather than being a single cohesive application, it serves as a repository of practical command-line tools that can be used independently or combined into workflows. The scripts cover a wide range of tasks, including URL manipulation, parameter replacement, data extraction, and reconnaissance automation. ... -
9
Easy3D
Efficient library for processing 3D data
Easy3D is a lightweight, easy-to-use, and efficient library for processing and rendering 3D data, implemented in C++ with Python bindings. It is designed for tasks such as 3D modeling, geometry processing, and rendering, emphasizing simplicity and efficiency. Easy3D serves as a valuable tool for research, education, and the development of sophisticated 3D applications, providing a solid foundation for handling 3D data. -
Go from Code to Production URL in SecondsSkip the Kubernetes configs. Cloud Run handles HTTPS, scaling, and infrastructure automatically. Two million requests free per month.
-
10
OmniTools
Self-hosted collection of powerful web-based tools for everyday tasks
...It’s designed to replace the random assortment of “free online tools” people use for quick tasks, while avoiding ads, tracking, and the need to upload sensitive files to unknown servers. A key design choice is that file processing happens entirely on the client side, meaning your data stays in your browser instead of being sent to the backend. The tool catalog spans both technical and non-technical needs, including image, video, audio, PDF, text, date/time, math, and data format utilities like JSON/CSV/XML helpers. It’s also packaged for straightforward self-hosting, with a lightweight Docker image and simple run commands, so it can be deployed quickly on a homelab or internal network. -
11
Elixir
Dynamic, functional language designed for building scalable apps
...Elixir leverages the Erlang VM, known for running low-latency, distributed, and fault-tolerant systems. Elixir is successfully used in web development, embedded software, data ingestion, and multimedia processing, across a wide range of industries. All Elixir code runs inside lightweight threads of execution (called processes) that are isolated and exchange information via messages. Due to their lightweight nature, it is not uncommon to have hundreds of thousands of processes running concurrently in the same machine. ... -
12
Apache Sedona
Cluster computing framework for processing large-scale geospatial data
Apache Sedona™ is a cluster computing system for processing large-scale spatial data. Sedona extends existing cluster computing systems, such as Apache Spark and Apache Flink, with a set of out-of-the-box distributed Spatial Datasets and Spatial SQL that efficiently load, process, and analyze large-scale spatial data across machines. According to our benchmark and third-party research papers, Sedona runs 2X - 10X faster than other Spark-based geospatial data systems on computation-intensive query workloads. ... -
13
h265web.js
A HEVC/H.265 Web Player
...Its architecture separates parsing, decoding, and rendering, giving developers fine-grained control over how video data is handled and displayed. The system is designed to work with streaming data, allowing incremental feeding of video chunks and real-time decoding, which is useful for surveillance, streaming platforms, and custom media pipelines. -
14
Apache Flink
Stream processing framework with powerful stream
Apache Flink is a distributed engine for stateful computations over data streams and batches, designed for low-latency processing at scale. Its core runtime executes dataflow graphs with fine-grained backpressure and checkpointing, allowing applications to recover consistently from failures. Flink’s event-time model and watermarks enable accurate out-of-order processing, windowing, and complex time semantics that typical real-time systems struggle with. -
15
glom
Python's nested data operator
...Its declarative specification style lets users describe the output shape they want instead of manually building transformation logic. glom is especially useful for data processing, API response cleanup, configuration handling, and scripts that need reliable nested data manipulation. -
16
WebP Codec
Library to encode and decode images in WebP format
libwebp is the reference codec library for Google’s WebP image format, providing both encoding and decoding along with command-line tools. It supplies cwebp to compress images into WebP and dwebp to decompress them back, making it easy to test quality/size trade-offs across presets and tuning parameters. The GitHub repository is a mirror; the canonical source of truth lives on Chromium’s git, and developer docs are hosted on WebP’s portal. The project underpins WebP support across browsers,... -
17
Kotlin Dataframe
Structured data processing in Kotlin
Data frame is an abstraction for working with structured data. Essentially it’s a 2-dimensional table with labeled columns of potentially different types. You can think of it like a spreadsheet or SQL table, or a dictionary of series objects. The handiness of this abstraction is not in the table itself but in a set of operations defined on it. The Kotlin Dataframe library is an idiomatic Kotlin DSL defining such operations. The process of working with data frame is often called data... -
18
Lexbor
Lexbor is development of an open source HTML Renderer library
Lexbor is the development of a web browser engine available as a software library; it ships with a free license and has no extra dependencies. For us, speed is an absolute must-have. In our development process, we focus on fastest parsing techniques for HTML, CSS, and fonts, fastest data processing methods, and fastest ways to serve content to end users. Whether you are building a backend that handles millions of HTML documents or a UI-heavy user app, your software’s response rate always matters to users and developers alike. Lexbor’s code is optimized for ease of access in end-user applications and across programming languages. ... -
19
Zipline Reloaded
Zipline, a Pythonic Algorithmic Trading Library
Zipline Reloaded is a maintained Python library for event-driven backtesting of algorithmic trading strategies. It continues the original Zipline project after Quantopian ended operations. Developers write trading algorithms while the engine simulates orders, market events, portfolio changes, and strategy performance over historical data. Common statistics such as moving averages and linear regression are available within algorithm workflows. Pandas-based input and output integrate naturally... -
20
Bacalhau
Community-driven, simple, yet powerful framework
Bacalhau is a decentralized compute platform for running jobs on data stored across distributed networks, like IPFS or Filecoin, without moving the data to centralized cloud environments. It allows developers to run containerized workloads close to where the data lives, reducing latency, cost, and privacy risks. Bacalhau supports various runtime environments and is designed to make decentralized data processing as accessible as traditional cloud computing. ... -
21
Spring Batch
Spring Batch is a framework for writing batch applications using Java
A lightweight, comprehensive batch framework designed to enable the development of robust batch applications vital for the daily operations of enterprise systems. Spring Batch provides reusable functions that are essential in processing large volumes of records, including logging/tracing, transaction management, job processing statistics, job restart, skip, and resource management. It also provides more advanced technical services and features that will enable extremely high-volume and high... -
22
hash-wasm
Lightning fast hash functions using hand-tuned WebAssembly binaries
...The library supports a wide range of algorithms, including MD5, SHA variants, BLAKE, Argon2, bcrypt, and xxHash, making it suitable for applications ranging from security to data processing. By compiling optimized C implementations into WebAssembly, hash-wasm achieves significantly better performance compared to pure JavaScript alternatives while maintaining portability across platforms. It supports both simple one-shot hashing and advanced streaming modes, allowing developers to process large datasets incrementally. ... -
23
Acl
A powerful server and network library, including coroutine
The Acl (Advanced C/C++ Library) project a is powerful multi-platform network communication library and service framework, supporting LINUX, WIN32, Solaris, FreeBSD, MacOS, AndroidOS, iOS. Many applications written by Acl run on these devices with Linux, Windows, iPhone and Android and serve billions of users. There are some important modules in Acl project, including network communcation, server framework, application protocols, multiple coders, etc. The common protocols such as... -
24
MathPHP
Powerful modern math library for PHP
Math PHP is a library that brings advanced mathematical functions and data analysis capabilities to PHP applications. It covers a wide range of topics, including linear algebra, calculus, statistics, probability, and numerical analysis. Math PHP is designed for developers and data scientists who require precise and efficient mathematical computations in PHP, making it suitable for scientific computing and data processing. -
25
Oban
Robust job processing in Elixir, backed by modern PostgreSQL
...It provides a simple and consistent API for scheduling and performing jobs, and it is built to be fault-tolerant and easy to monitor. Oban is fundamentally different from other background job processing tools because it retains job data for historic metrics and inspection. You can leave your application running indefinitely without worrying about jobs being lost or orphaned due to crashes.
