Search Results for "data processing"
Sort By:
832 projects for "data processing" with 1 filter applied:
-
Demo Series - Small Business Backup By VeeamWatch this on-demand demo series and learn how to protect your Microsoft 365 data with clear, simple, actionable steps that are easy to implement for businesses of all sizes.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
Synthetic Data Generator
SDG is a specialized framework
...It also includes a data processing module capable of handling different data types, preprocessing columns, managing missing values, and converting formats automatically before model training. -
2
XY
Ultra-fast and customizable Python charts
XY is an interactive Python visualization library designed to handle both everyday charts and extremely large datasets. Charts can be created through a declarative API or familiar Matplotlib-style conventions. A Rust processing core dynamically determines how much data the current screen resolution actually needs to display. Large datasets can be rendered as bounded density or decimated views, while zooming retrieves finer details and original rows. Interactive capabilities include panning, zooming, hovering, selections, crosshairs, callbacks, and linked charts. ... -
3
Agentic Data Scientist
An end-to-end Data Scientist
...Each agent is designed to independently call functions, interact with data sources, and adapt to uncertainties during processing, enabling iterative refinement of models without manual coordination. The framework supports interoperability with existing data tools and libraries, letting the agents leverage libraries like pandas, scikit-learn, and visualization frameworks to perform real computations rather than mock demonstrations. -
4
Algorithms & data structures project
A collection of algorithms and data structures
Algorithms is an educational collection of data structures and algorithms implemented primarily in Java. Its goal is to demonstrate common techniques through clear and elegant reference implementations. The repository covers trees, heaps, hash tables, graphs, dynamic programming, geometry, searching, sorting, strings, and many other topics. Complexity information is documented for many algorithms to help users understand performance characteristics. Implementations can be compiled with a... -
Build Securely on AWS with Proven FrameworksMoving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.
-
5
Kapacitor
Open source framework for processing, monitoring, and alerting
Open source framework for processing, monitoring, and alerting on time series data. Kapacitor is a real-time data processing engine for monitoring and alerting, specifically designed to work with time-series data from InfluxDB. -
6
NYC Taxi Data
Import public NYC taxi and for-hire vehicle (Uber, Lyft)
The nyc-taxi-data repository is a rich dataset and exploratory project around New York City taxi trip records. It collects and preprocesses large-scale trip datasets (fares, pickup/dropoff, timestamps, locations, passenger counts) to enable data analysis, modeling, and visualization efforts. The project includes scripts and notebooks for cleaning and filtering the raw data, memory-efficient processing for large CSV/Parquet files, and aggregation workflows (e.g. trips per hour, heatmaps of pickups/dropoffs). ... -
7
Siddhi Core Libraries
Stream Processing and Complex Event Processing Engine
Fully open source, cloud-native, scalable, micro streaming, and complex event processing system capable of building event-driven applications for use cases such as real-time analytics, data integration, notification management, and adaptive decision-making. Event processing logic can be written using Streaming SQL queries via graphical and source editors, to capture events from diverse data sources, process and analyze them, integrate with multiple services and data stores, and publish output to various endpoints in real time. ... -
8
LOTUS
AI-Powered Data Processing: Use LOTUS to process all of your datasets
LOTUS is an open-source framework and query engine designed to enable efficient processing of structured and unstructured datasets using large language models. The system provides a declarative programming model that allows developers to express complex AI data operations using high-level commands rather than manually orchestrating model calls. It offers a Python interface with a Pandas-like API, making it familiar for data scientists and engineers already working with data analysis libraries. ... -
9
Apache Spark
A unified analytics engine for large-scale data processing
Apache Spark is a unified engine for large-scale data processing, offering APIs for batch jobs, streaming, machine learning, and graph computation. It builds on resilient distributed datasets (RDDs) and the newer DataFrame/Dataset abstractions to provide fault-tolerant, in-memory computation across clusters. Spark’s execution engine handles scheduling, shuffles, caching, and data locality so users can focus on transformations rather than infrastructure plumbing. ... -
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
10
jq
Lightweight and flexible command-line JSON processor
jq is like sed for JSON data - you can use it to slice, filter, map and transform structured data with the same ease that sed, awk, grep and friends let you play with text. jq is written in portable C, and it has zero runtime dependencies. You can download a single binary, scp it to a far away machine of the same type, and expect it to work. jq can mangle the data format that you have into the one that you want with very little effort, and the program to do so is often shorter and simpler... -
11
cobalt
Video and media downloader: Best way to save what you love
Cobalt is an open-source media downloader and tool designed to provide a high-performance and privacy-focused alternative for interacting with online media content, particularly focused on downloading and processing media from various platforms. It emphasizes speed, reliability, and a clean user experience, allowing users to retrieve media without unnecessary tracking, ads, or intrusive elements commonly found in web-based tools. The project is built with performance in mind, leveraging efficient backend processing to handle requests quickly and consistently. ... -
12
lxml
The lxml XML toolkit for Python
A Python library for efficient XML and HTML processing, known for speed and compatibility. The lxml XML toolkit is a Pythonic binding for the C libraries libxml2 and libxslt. It is unique in that it combines the speed and XML feature completeness of these libraries with the simplicity of a native Python API, mostly compatible but superior to the well-known ElementTree API. The latest release works with all CPython versions from 3.6 to 3.12. See the introduction for more information about the... -
13
PyAV
Pythonic bindings for FFmpeg's libraries
...PyAV integrates well with other Python libraries such as NumPy and Pillow, allowing seamless data transformation and analysis. It supports decoding, encoding, filtering, and resampling operations, covering a wide range of multimedia tasks. While powerful, it requires a solid understanding of FFmpeg concepts, as it prioritizes flexibility and control over abstraction. Overall, PyAV is a robust tool for developers building advanced video and audio processing systems in Python. -
14
Broadway
Concurrent and multi-stage data ingestion and data processing
Broadway is a data processing library for Elixir designed to handle high-throughput, concurrent workloads with ease. It provides an abstraction for defining pipelines that consume data from sources like RabbitMQ, Kafka, Amazon SQS, or custom producers. Each pipeline is fault-tolerant and backpressure-aware, ensuring stable throughput even under load. -
15
TeXworks
A simple interface for working with TeX documents
TeXworks is a free and simple working environment for authoring TeX (LaTeX, ConTeXt and XeTeX) documents. Inspired by Dick Koch's award-winning TeXShop program for Mac OS X, it makes entry into the TeX world easier for those using desktop operating systems other than OS X. It provides an integrated, easy-to-use environment for users on other platforms particularly GNU/Linux and Windows and features a clean, simple interface accessible to casual and non-technical users. -
16
PDFCraft
PDFCraft is a free, privacy-focused PDF toolkit
PDFCraft is an extensible toolkit for creating, editing, and transforming PDF documents with both a graphical interface and a scripting API, making it useful for users ranging from casual editors to automated document processors. At its core, the project provides a clean, modern UI where you can rearrange pages, annotate text, insert images, fill forms, and export to multiple formats, all without needing a heavyweight commercial PDF suite. But beyond manual editing, it also offers a... -
17
Hacks
A collection of hacks and one-off scripts
Hacks is a collection of experimental scripts, utilities, and one-off tools created to solve specific problems in security research, data processing, and automation. Rather than being a single cohesive application, it serves as a repository of practical command-line tools that can be used independently or combined into workflows. The scripts cover a wide range of tasks, including URL manipulation, parameter replacement, data extraction, and reconnaissance automation. ... -
18
Easy3D
Efficient library for processing 3D data
Easy3D is a lightweight, easy-to-use, and efficient library for processing and rendering 3D data, implemented in C++ with Python bindings. It is designed for tasks such as 3D modeling, geometry processing, and rendering, emphasizing simplicity and efficiency. Easy3D serves as a valuable tool for research, education, and the development of sophisticated 3D applications, providing a solid foundation for handling 3D data. -
19
DocETL
A system for agentic LLM-powered data processing and ETL
DocETL is an open-source system designed to build and execute data processing pipelines powered by large language models, particularly for analyzing complex collections of documents and unstructured datasets. The platform allows developers and researchers to construct structured workflows that extract, transform, and organize information from sources such as reports, transcripts, legal documents, and other text-heavy data. -
20
fluentbit
Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX
Fluent Bit is a super-fast, lightweight, and highly scalable logging and metrics processor and forwarder. It is the preferred choice for cloud and containerized environments. A robust, lightweight, and portable architecture for high throughput with low CPU and memory usage from any data source to any destination. Proven across distributed cloud and container environments. Highly available with I/O handlers to store data for disaster recovery. Granular management of data parsing and routing.... -
21
OmniTools
Self-hosted collection of powerful web-based tools for everyday tasks
...It’s designed to replace the random assortment of “free online tools” people use for quick tasks, while avoiding ads, tracking, and the need to upload sensitive files to unknown servers. A key design choice is that file processing happens entirely on the client side, meaning your data stays in your browser instead of being sent to the backend. The tool catalog spans both technical and non-technical needs, including image, video, audio, PDF, text, date/time, math, and data format utilities like JSON/CSV/XML helpers. It’s also packaged for straightforward self-hosting, with a lightweight Docker image and simple run commands, so it can be deployed quickly on a homelab or internal network. -
22
Elixir
Dynamic, functional language designed for building scalable apps
...Elixir leverages the Erlang VM, known for running low-latency, distributed, and fault-tolerant systems. Elixir is successfully used in web development, embedded software, data ingestion, and multimedia processing, across a wide range of industries. All Elixir code runs inside lightweight threads of execution (called processes) that are isolated and exchange information via messages. Due to their lightweight nature, it is not uncommon to have hundreds of thousands of processes running concurrently in the same machine. ... -
23
Scanopy
Clean network diagrams, One-time setup, zero upkeep
Scanopy is a powerful multi-modal data capture and analysis toolkit that enables users to collect, process, and visualize structured and unstructured information from a variety of sources in a flexible pipeline. It is built to handle complex scanning tasks — such as OCR, document analysis, audio transcription, network data capture, and image extraction — while providing unified APIs and workflows that make managing heterogeneous data sources seamless. Developers can compose custom pipelines... -
24
Sail
A drop-in Apache Spark replacement written in Rust
Sail is an open-source distributed computation framework designed to unify batch processing, stream processing, and AI workloads into a single, high-performance engine. It is built entirely in Rust, eliminating JVM overhead and enabling predictable performance, fast startup times, and improved memory safety compared to traditional big data frameworks. Sail is compatible with the Spark Connect protocol, which means existing Spark SQL and DataFrame workloads can run without code changes, making adoption seamless for teams already using Spark-based pipelines. ... -
25
Apache Sedona
Cluster computing framework for processing large-scale geospatial data
Apache Sedona™ is a cluster computing system for processing large-scale spatial data. Sedona extends existing cluster computing systems, such as Apache Spark and Apache Flink, with a set of out-of-the-box distributed Spatial Datasets and Spatial SQL that efficiently load, process, and analyze large-scale spatial data across machines. According to our benchmark and third-party research papers, Sedona runs 2X - 10X faster than other Spark-based geospatial data systems on computation-intensive query workloads. ...
