Search Results for "parallel compression"
Sort By:
Showing 33 open source projects for "parallel compression"
View related business solutions-
Our Free Plans just got better! | Auth0You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
-
Error to trace to log to deploy. One click. No SSH.AppSignal links every error to the trace, the trace to the log, the log to the deploy that shipped it.
-
1
compress
Optimized Go Compression Packages
This package provides various compression algorithms. zstandard compression and decompression in pure Go. S2 is a high-performance replacement for Snappy. Optimized deflate packages which can be used as a dropin replacement for gzip, zip and zlib. snappy is a drop-in replacement for github.com/golang/snappy offering better compression and concurrent streams. huff0 and FSE implementations for raw entropy encoding. gzhttp Provides client and server wrappers for handling gzipped requests efficiently. pgzip is a separate package that provides a very fast parallel gzip implementation. ... -
2
Vortex
The LLVM of columnar file formats
Vortex is a high-performance toolkit designed for working with compressed Apache Arrow arrays, providing functionality for in-memory, on-disk, and over-the-wire data handling. It aims to be an advanced successor to Apache Parquet, offering dramatically faster random access reads and scans, while maintaining similar compression ratios. Vortex's modular design allows for extensibility, enabling developers to implement custom encodings for efficient data management, particularly for large-scale... -
3
HDF5
Official HDF5® Library Repository
HDF5 (Hierarchical Data Format v5) is a widely-used data management library and file format for storing large and complex scientific data sets efficiently. -
4
Reliable PostgreSQL Backup & Restore
Reliable PostgreSQL backup and restore
pgBackRest aims to be a reliable, easy-to-use backup and restore solution that can seamlessly scale up to the largest databases and workloads by utilizing algorithms that are optimized for database-specific requirements. Compression is usually the bottleneck during backup operations but, even with now-ubiquitous multi-core servers, most database backup solutions are still single-process. pgBackRest solves the compression bottleneck with parallel processing. Utilizing multiple cores for compression makes it possible to achieve 1TB/hr raw throughput even on a 1Gb/s link. ... -
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
5
Koog
Koog is the official Kotlin framework for building AI agents
...It features pure Kotlin implementation, seamless Model Control Protocol (MCP) integration for enhanced model management, vector embeddings for semantic search, and a flexible system for creating and extending tools that access external systems and APIs. Ready‑to‑use components address common AI engineering challenges, while intelligent history compression optimizes token usage and preserves context. A powerful streaming API enables real‑time response processing and parallel tool calls. Persistent memory allows agents to retain knowledge across sessions and between agents, and comprehensive tracing facilities provide detailed debugging and monitoring. -
6
Av1an
Cross-platform command-line AV1 / VP9 / HEVC / H264 encoding framework
Av1an is a command-line encoding framework designed to efficiently encode video into the AV1 format using distributed and parallel processing techniques. It acts as an orchestrator that splits video files into chunks, processes them in parallel using encoders like SVT-AV1, AOM, or rav1e, and then merges the results into a final output. The tool is optimized for high-quality encoding while reducing total processing time through multi-core and multi-machine support. -
7
Google Highway
Performance-portable, length-agnostic SIMD with runtime dispatch
Google Highway is a high-performance C++ library designed to provide portable SIMD (Single Instruction, Multiple Data) vectorization across multiple CPU architectures while maintaining predictable and efficient behavior. It abstracts low-level vector intrinsics into a consistent API that maps closely to hardware instructions, allowing developers to write high-performance code without relying heavily on compiler auto-vectorization. Highway enables the same source code to run across different... -
8
FreeRDP
FreeRDP is a free remote desktop protocol library and clients
...Enjoy the freedom of using your software wherever you want, the way you want it, in a world where interoperability can finally liberate your computing experience. The nightly builds are installed into /opt/freerdp-nightly and can be installed in parallel with the distributions regular freerdp package. Network tracing of RDP can be quite a challenge due to a number of factors such as encryption, compression, and the fact that these protocol features cannot always be disabled or worked around. One of the easiest way of decrypting RDP traffic is to configure the server with a self-signed certificate for which the private key is known, and then use this certificate with a network tracing tool to automatically decrypt the packets. -
9
Barman
Backup and Recovery Manager for PostgreSQL
Barman (Backup and Recovery Manager) is an enterprise-grade tool for managing backups and disaster recovery of PostgreSQL databases. It supports both full and incremental backups, Point-In-Time Recovery (PITR), and remote backup via SSH. Barman is widely used by DBAs to ensure secure, reliable, and consistent backups in production environments. -
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
10
ClickHouse
A fast open-source OLAP database management system
ClickHouse® is a fast, open-source column-oriented database management system that can generate analytical data reports through SQL queries in real time. According to several independent benchmarks, it far exceeds other comparable column-oriented database management systems, working even up to 1000 times faster. It is able to process hundreds of millions to more than a billion rows and tens of gigabytes of data per single server per second. Apart from its blazing speed, ClickHouse is highly... -
11
LinDB
LinDB is a scalable, high performance, high availability database
LinDB is a scalable, high-performance, high-availability distributed time series database. A single server could easily support more than one million write TPS; With fundamental techniques like efficient compression storage and parallel computing, LinDB delivers highly optimized query performance. The multi-channel replication protocol supports any amount of nodes, and ensures the system's availability. Schema-free multi-dimensional data model with Metric, Tags, and Fields; The LinQL is flexible yet handy for real-time data analytics. Horizontal scalable is made simple by adding more new broker and storage nodes without too much thinking and manual operations. ... -
12
ERNIE
The official repository for ERNIE 4.5 and ERNIEKit
...The project also emphasizes optimization techniques for large-scale training, including mixed-precision and hybrid-parallel strategies that are commonly needed for multi-node GPU clusters. In addition to training, it includes guidance and example materials intended to help developers adopt ERNIE models for real product scenarios rather than only research demonstrations. -
13
MediaCoder
Universal media transcoding software
MediaCoder is a universal media transcoding software actively developed and maintained since 2005. It puts together most cutting-edge audio/video technologies into an out-of-box transcoding solution with a rich set of adjustable parameters which let you take full control of your transcoding. New features and latest codecs are added or updated constantly. MediaCoder might not be the easiest tool out there, but what matters here is quality and performance. It will be your swiss army knife for... -
14
OpenAS2
AS2 1.1 server implementation in Java
OpenAS2 is a java-based implementation of the EDIINT AS2 standard. It is intended to be used as a server. It is extremely configurable and supports a wide variety of signing and encryption algorithms. Supports very high traffic volume allowing parallel processing of files per partner. SUPPORT: Please use the Github issues and Discussions channels here: https://github.com/OpenAS2/OpenAs2App/ Requirements: - Java 11 or newer (tested with the LTS versions of Java up to 21) ... -
15
LocustDB
Massively parallel, high performance analytics database
An experimental analytics database aiming to set a new standard for query performance and storage efficiency on commodity hardware. See How to Analyze Billions of Records per Second on a Single Desktop PC and How to Read 100s of Millions of Records per Second from a Single Disk for an overview of current capabilities. Download the latest binary release, which can be run from the command line on most x64 Linux systems, including Windows Subsystem for Linux. When loading .csv or .csv.gz files... -
16
ADAMS
ADAMS is a workflow engine for building complex knowledge workflows.
ADAMS is a flexible workflow engine aimed at quickly building and maintaining data-driven, reactive workflows, easily integrated into business processes. Instead of placing operators on a canvas and manually connecting them, a tree structure and flow control operators determine how data is processed (sequentially/parallel). This allows rapid development and easy maintenance of large workflows, with hundreds or thousands of operators. Operators include machine learning (WEKA, MOA, MEKA)... -
17
Sqlite Index Blaster
Create huge Sqlite indexes at breakneck speeds
SQLite Blaster is an advanced SQLite extension that enhances database performance by enabling multi-threading, data compression, and memory optimizations. It is designed for applications that require fast local storage with improved query efficiency. -
18
Crunch
Insane(ly slow but wicked good) PNG image optimization
Crunch is an image compression tool for lossy PNG image file optimization. Using a combination of selective bit depth, color palette reduction and color type, as well as zopfli DEFLATE compression algorithm encoding that employs the pngquant and zopflipng PNG optimization tools, Crunch is effectively able to optimize and compress images with minimal decrease in image quality. While it may produce file size gains larger than those produced by lossless approaches, the impact on image quality... -
19
XZ
Multi-core, cross-platform XZ compression tool written in Go
Parallel implementation of XZ for modern multi-core machines written in Go. This project is licensed under the ISC License. Copyright © 2020-2021 ALBANESE Research Lab Source code: https://github.com/pedroalbanese/xz-2 Visit: http://albanese.atwebpages.com -
20
lzhw
LZHW Windows command line lossless compression tool for tabular files
LZHW Command Line Lossless Compression Tool is a Windows command line tool used to compress and decompress files from and to any form, csv, excel etc without any dependencies or installations. Using an optimized algorithm (LZHW) developed from Lempel-Ziv, Huffman and LZ-Welch algorithms. The tool can work in parallel and most of its code is written in Cython, so it is pretty fast. -
21
ZIPcrypt
Simple Encryption / Decryption Utility
ZIPcrypt is a simple encryption and compression software that uses patented XOTIC 512-bit+, or AES 256-bit encryption. Delivering the best encryption technology on the market today, ZIPcrypt was also designed to be fast, easy to use, and to work in parallel with the SCIFCOM Encryption as a Service website. -
22
Parallel MILC
Parallel decompression for the MILC algorithm
The MILC compression has been developed specifically for medical images and proven to be effective. This algorithm is a parallel version for the decompression phase, meant to exploit the parallel computing potential of the modern hardware. Developed with OpenCL framework (version 1.2), runs on CPU and GPU OpenCL devices. -
23
Icarus Scene Engine
3D .NET engine for Windows, Linux, Mac
Icarus Scene Engine is a cross-platform, CLS-compliant 3D framework, integrating open source APIs into a cohesive all-open-source, all .NET solution. For Windows, MacOSX & Linux. Uses OpenTK, OpenGL, OpenAL APIs for graphics and audio. Runs on Mono/.NET. Links to FFMpeg for streaming video, and FreeType as a temporary conversion mod to bring fonts into a native XML format. Binary builds of Icarus will be infrequent, roughly every 18 months or so, the GIT code updates regularly on a... -
24
COMRAD-MPI
Compression of Large Genomic Datasets using Parallel Computing
COMRAD-MPI is a parallel computing algorithm for reducing the computational time for compressing the large genomic data sets based on COMRAD algorithm. It captures the long range repeat redundancies in large genomes there by providing a way to compress the large DNA data set. Three stages- Substitution, Clean up and Huffman encoding have been parallelized using message passing library. -
25
Parallel DPCM
C# Parallel implementation of DPCM alogrithm
An implementation of the DPCM algorithm for compressing and decompressing an avi video file. It shows the compression size, as it is saved in a txt file near the avi original file. The decompression results in another avi file, saved near the original file. If the width and height of the frames aren't a multiple of 16, they are completed with black pixels. The pixel macroblocks of 16x16 are generated on separate threads for each frame and they are replaced on frames at decompression...