Search Results for "hash"

Advanced C/C++ library(ACL) for UNIX-like OS and WIN32 OS, including sync/async/ssl iostream for net/file, thread pool, process pool, db pool, server framework, event, memory, string, array/hash/ring/list, xml and json parser, http/smtp/icmp protocol, SSL/TLS, C unit test, etc

...Let's take a large linear data stream, split it into variable-sized chunks (the size of each being a function of the chunk's contents), and store these chunks in individual, compressed files in some directory, each file named after a strong hash value of its contents, so that the hash value may be used to as key for retrieving the full chunk data. Let's call this directory a "chunk store". At the same time, generate a "chunk index" file that lists these chunk hash values plus their respective chunk sizes in a simple linear array. The chunking algorithm is supposed to create variable, but similarly sized chunks from the data stream.