Search Results for "parallel compression"
Sort By:
Showing 26 open source projects for "parallel compression"
View related business solutions-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
Host LLMs in Production With On-Demand GPUsDeploy your model, get an endpoint, pay only for compute time. No GPU provisioning or infrastructure management required.
-
1
compress
Optimized Go Compression Packages
This package provides various compression algorithms. zstandard compression and decompression in pure Go. S2 is a high-performance replacement for Snappy. Optimized deflate packages which can be used as a dropin replacement for gzip, zip and zlib. snappy is a drop-in replacement for github.com/golang/snappy offering better compression and concurrent streams. huff0 and FSE implementations for raw entropy encoding. gzhttp Provides client and server wrappers for handling gzipped requests efficiently. pgzip is a separate package that provides a very fast parallel gzip implementation. ... -
2
HDF5
Official HDF5® Library Repository
HDF5 (Hierarchical Data Format v5) is a widely-used data management library and file format for storing large and complex scientific data sets efficiently. -
3
Vortex
The LLVM of columnar file formats
Vortex is a high-performance toolkit designed for working with compressed Apache Arrow arrays, providing functionality for in-memory, on-disk, and over-the-wire data handling. It aims to be an advanced successor to Apache Parquet, offering dramatically faster random access reads and scans, while maintaining similar compression ratios. Vortex's modular design allows for extensibility, enabling developers to implement custom encodings for efficient data management, particularly for large-scale... -
4
Av1an
Cross-platform command-line AV1 / VP9 / HEVC / H264 encoding framework
Av1an is a command-line encoding framework designed to efficiently encode video into the AV1 format using distributed and parallel processing techniques. It acts as an orchestrator that splits video files into chunks, processes them in parallel using encoders like SVT-AV1, AOM, or rav1e, and then merges the results into a final output. The tool is optimized for high-quality encoding while reducing total processing time through multi-core and multi-machine support. -
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
5
Koog
Koog is the official Kotlin framework for building AI agents
...It features pure Kotlin implementation, seamless Model Control Protocol (MCP) integration for enhanced model management, vector embeddings for semantic search, and a flexible system for creating and extending tools that access external systems and APIs. Ready‑to‑use components address common AI engineering challenges, while intelligent history compression optimizes token usage and preserves context. A powerful streaming API enables real‑time response processing and parallel tool calls. Persistent memory allows agents to retain knowledge across sessions and between agents, and comprehensive tracing facilities provide detailed debugging and monitoring. -
6
Barman
Backup and Recovery Manager for PostgreSQL
Barman (Backup and Recovery Manager) is an enterprise-grade tool for managing backups and disaster recovery of PostgreSQL databases. It supports both full and incremental backups, Point-In-Time Recovery (PITR), and remote backup via SSH. Barman is widely used by DBAs to ensure secure, reliable, and consistent backups in production environments. -
7
Google Highway
Performance-portable, length-agnostic SIMD with runtime dispatch
Google Highway is a high-performance C++ library designed to provide portable SIMD (Single Instruction, Multiple Data) vectorization across multiple CPU architectures while maintaining predictable and efficient behavior. It abstracts low-level vector intrinsics into a consistent API that maps closely to hardware instructions, allowing developers to write high-performance code without relying heavily on compiler auto-vectorization. Highway enables the same source code to run across different... -
8
ERNIE
The official repository for ERNIE 4.5 and ERNIEKit
...The project also emphasizes optimization techniques for large-scale training, including mixed-precision and hybrid-parallel strategies that are commonly needed for multi-node GPU clusters. In addition to training, it includes guidance and example materials intended to help developers adopt ERNIE models for real product scenarios rather than only research demonstrations. -
9
LinDB
LinDB is a scalable, high performance, high availability database
LinDB is a scalable, high-performance, high-availability distributed time series database. A single server could easily support more than one million write TPS; With fundamental techniques like efficient compression storage and parallel computing, LinDB delivers highly optimized query performance. The multi-channel replication protocol supports any amount of nodes, and ensures the system's availability. Schema-free multi-dimensional data model with Metric, Tags, and Fields; The LinQL is flexible yet handy for real-time data analytics. Horizontal scalable is made simple by adding more new broker and storage nodes without too much thinking and manual operations. ... -
Fully Managed MySQL, PostgreSQL, and SQL ServerCloud SQL handles your database ops end to end, so you can focus on your app.
-
10
MediaCoder
Universal media transcoding software
MediaCoder is a universal media transcoding software actively developed and maintained since 2005. It puts together most cutting-edge audio/video technologies into an out-of-box transcoding solution with a rich set of adjustable parameters which let you take full control of your transcoding. New features and latest codecs are added or updated constantly. MediaCoder might not be the easiest tool out there, but what matters here is quality and performance. It will be your swiss army knife for... -
11
OpenAS2
AS2 1.1 server implementation in Java
OpenAS2 is a java-based implementation of the EDIINT AS2 standard. It is intended to be used as a server. It is extremely configurable and supports a wide variety of signing and encryption algorithms. Supports very high traffic volume allowing parallel processing of files per partner. SUPPORT: Please use the Github issues and Discussions channels here: https://github.com/OpenAS2/OpenAs2App/ Requirements: - Java 17 or newer (tested with the LTS versions of Java up to 25) ... -
12
LocustDB
Massively parallel, high performance analytics database
An experimental analytics database aiming to set a new standard for query performance and storage efficiency on commodity hardware. See How to Analyze Billions of Records per Second on a Single Desktop PC and How to Read 100s of Millions of Records per Second from a Single Disk for an overview of current capabilities. Download the latest binary release, which can be run from the command line on most x64 Linux systems, including Windows Subsystem for Linux. When loading .csv or .csv.gz files... -
13
ADAMS
ADAMS is a workflow engine for building complex knowledge workflows.
ADAMS is a flexible workflow engine aimed at quickly building and maintaining data-driven, reactive workflows, easily integrated into business processes. Instead of placing operators on a canvas and manually connecting them, a tree structure and flow control operators determine how data is processed (sequentially/parallel). This allows rapid development and easy maintenance of large workflows, with hundreds or thousands of operators. Operators include machine learning (WEKA, MOA, MEKA)... -
14
Sqlite Index Blaster
Create huge Sqlite indexes at breakneck speeds
SQLite Blaster is an advanced SQLite extension that enhances database performance by enabling multi-threading, data compression, and memory optimizations. It is designed for applications that require fast local storage with improved query efficiency. -
15
Crunch
Insane(ly slow but wicked good) PNG image optimization
Crunch is an image compression tool for lossy PNG image file optimization. Using a combination of selective bit depth, color palette reduction and color type, as well as zopfli DEFLATE compression algorithm encoding that employs the pngquant and zopflipng PNG optimization tools, Crunch is effectively able to optimize and compress images with minimal decrease in image quality. While it may produce file size gains larger than those produced by lossless approaches, the impact on image quality... -
16
XZ
Multi-core, cross-platform XZ compression tool written in Go
Parallel implementation of XZ for modern multi-core machines written in Go. This project is licensed under the ISC License. Copyright © 2020-2021 ALBANESE Research Lab Source code: https://github.com/pedroalbanese/xz-2 Visit: http://albanese.atwebpages.com -
17
lzhw
LZHW Windows command line lossless compression tool for tabular files
LZHW Command Line Lossless Compression Tool is a Windows command line tool used to compress and decompress files from and to any form, csv, excel etc without any dependencies or installations. Using an optimized algorithm (LZHW) developed from Lempel-Ziv, Huffman and LZ-Welch algorithms. The tool can work in parallel and most of its code is written in Cython, so it is pretty fast. -
18
ZIPcrypt
Simple Encryption / Decryption Utility
ZIPcrypt is a simple encryption and compression software that uses patented XOTIC 512-bit+, or AES 256-bit encryption. Delivering the best encryption technology on the market today, ZIPcrypt was also designed to be fast, easy to use, and to work in parallel with the SCIFCOM Encryption as a Service website. -
19
Icarus Scene Engine
3D .NET engine for Windows, Linux, Mac
Icarus Scene Engine is a cross-platform, CLS-compliant 3D framework, integrating open source APIs into a cohesive all-open-source, all .NET solution. For Windows, MacOSX & Linux. Uses OpenTK, OpenGL, OpenAL APIs for graphics and audio. Runs on Mono/.NET. Links to FFMpeg for streaming video, and FreeType as a temporary conversion mod to bring fonts into a native XML format. Binary builds of Icarus will be infrequent, roughly every 18 months or so, the GIT code updates regularly on a... -
20
COMRAD-MPI
Compression of Large Genomic Datasets using Parallel Computing
COMRAD-MPI is a parallel computing algorithm for reducing the computational time for compressing the large genomic data sets based on COMRAD algorithm. It captures the long range repeat redundancies in large genomes there by providing a way to compress the large DNA data set. Three stages- Substitution, Clean up and Huffman encoding have been parallelized using message passing library. -
21
Parallel DPCM
C# Parallel implementation of DPCM alogrithm
An implementation of the DPCM algorithm for compressing and decompressing an avi video file. It shows the compression size, as it is saved in a txt file near the avi original file. The decompression results in another avi file, saved near the original file. If the width and height of the frames aren't a multiple of 16, they are completed with black pixels. The pixel macroblocks of 16x16 are generated on separate threads for each frame and they are replaced on frames at decompression... -
22
Multi Threaded PDF Merger
Have to merge huge set's of PDF's to a single one, well use this.
...Im Programmer and this wasn't acceptable for me.... This Libary (or Tutorial ?) is the result, its using IText for C# and it's made to use your physical CPU Cores and do Parallel Execution. I will make it short, 1000 PDF needs 15 seconds(On a 6 Core Machine with 50% Usage Peeks), also the Filesize is 50% smaller than Adobe and there is no visual difference (if there change compression...)! This save me a lot of time and since this is for my work also money. Adobe may be great for other tasks but not for simple merging. ... -
23
pigz for windows
pigz (parallel gzip) for OS Windows
Based on source from http://zlib.net/pigz/ -
24
pixz
Parallel compression utility
New releases of pixz live at the GitHub page: https://github.com/vasi/pixz/releases -
25
dspTransfer
complete file transfer command line client
dspFileTransfer can upload and download files using ftp, sftp (ftp over ssh), http and https. The command line utility offers many advanced features such as upload or download a file only if the file is newer, parallel downloads etc. In addition the program can perform compression, decompression, encryption and decryption of processed files. The program is a part of batch file processing suite called Data Stream Processor. For full download and support go to www.datastreamprocessor.com. To use the transfer service on its you need to specify '-nodbuse' option to the program so that it doesn't try use DSP db.
