Search Results for "chunk"

Version 1.0.9 contains a fix to "integer overflow" problem. This happens when "one-shot" decoding API is used (or input chunk for streaming API is not limited), input size (chunk size) is larger than 2GiB, and input contains uncompressed blocks. After the overflow happens, memcpy is invoked with a gigantic num value, that will likely cause the crash. Brotli is a generic-purpose lossless compression algorithm that compresses data using a combination of a modern variant of the LZ77 algorithm, Huffman coding and 2nd order context modeling, with a compression ratio comparable to the best currently available general-purpose compression methods. ...

gzip3: Compressor gzip multithread com chunk de 128 KB, work stealing entre 64 compartimentos NUMA, escritor centralizado sequencial. Escala até 512 threads. Ideal para dados "quentes" onde velocidade de compressão importa. zstd3: Compressor Zstandard multithread com chunk de 2 MB, fatias contíguas por thread, cada uma escreve em temp file próprio, main concatena no final.

...Unlike fixed-size blocks, content-defined chunks are cut at data-dependent boundaries, which means shifts and insertions don’t invalidate the rest of the file’s delta map. The tool maintains chunk fingerprints and a manifest so it can de-duplicate across versions and even across different files that share regions of content. This approach makes it well suited for large logs, VM images, and binary assets where small edits would otherwise force full re-uploads. The implementation focuses on being robust over unreliable networks with resumable transfers and integrity checks that verify every chunk. ...

Backs up Linux and Windows computers (via NFS, CIFS, SSHFS and local access) onto a central server. Provides filtration to avoid backing up unnecessary files. Database is chunk-based (compressed where worth it) and deduplicated (at chunk level). Most setting can be configured and have sensible defaults.

fft is a distributed file transfer tool designed to accelerate large file movement by coordinating multiple relay nodes in parallel rather than depending on a single server’s bandwidth ceiling. Its architecture splits a transfer into concurrent “workers” that fetch or push chunks across multiple paths, improving throughput on high-latency or bandwidth-constrained links. The project is implemented in Go and exposes a straightforward command-line interface so operators can stand up senders,...

mdadm is great for assembling RAID devices, but there's on edrawback: It can't work with chunk sizes below 4K. While this a rare case, it nevertheless exists. This where xraid may help you. It allows to assemble any number of RAID-0 member and lets you access the resulting RAID device via FUSE. Fast and without RAM nor storage space usage.

There are many online storage services, for example Dropbox. While these services claim to be secure, they should not be relied upon for security or reliability. SDFS aims to address this issue by providing a filesystem-like interface while encrypting and storing data redundantly in many online storage accounts. In the first alpha early release(0.0.0) we are planning to support dropbox only but if you want support for other services, or if you are a developer who wants to develop support for...