Re: [Cppqed-support] Compression of state files

[Andras V. <and...@ui...>]

Dear Raimar,
I think enabling compression as an option is a very good idea, especially
if you say that in cases it can lead to significant gains.
Perhaps a small rationale why you opted for bzip2 would be in order.
I think that compressing the individual archives is perhaps a better option
because if the zip-stream is split at continuation instead, then the
structure of the state file will depend on the sequence of continuations,
which may cause other problems in the future.
Best regards,
András


On Wed, Oct 22, 2014 at 4:25 PM, Raimar Sandner <rai...@ui...>
wrote:

> Dear András,
>
> how is everything going in Budapest? I hope you are well.
>
> Maybe you remember the time when we discussed the binary boost
> serialization,
> back then I was under the impression that compressing the state files did
> not
> gain much disk space.
>
> Now with my real simulations however I found that it makes a huge
> difference
> in disk space, and therefore I need to compress the state files with
> bzip2. I
> thought it would be nice if, in Python, I could read in the compressed
> state
> files directly. Therefore I added transparent bzip2 support for _reading_
> of
> state files in the branch called "compression". This is done by changing
> the
> interface to std::istream instead of std::ifstream and working with boost's
> filtering_istream, to which one can add a decompressor if necessary. This
> is
> actually quite nice. On the downside we need yet another binary boost
> library,
> boost_iostreams. This library is detected by cmake, and compression
> support is
> enabled or disabled accordingly.
>
> In principle the _writing_ of compressed state files does also work.
> However
> there is a problem with continuation: we end up with several full
> bzip2-files
> concatenated. This is actually legal, it is called a multi-stream bzip2
> file
> and I can decompress these files with bunzip2 without problems. However,
> unfortunately the boost decompressor has a bug and cannot handle these
> files.
> As a result, I have disabled the _writing_ of compressed state files at the
> moment. Instead, I just use the external bzip2 after the trajectory is
> done.
> If I want to continue a trajectory, I have to decompress first of course.
>
> One could solve this problem easily by compressing the individual archives
> inside the state file, not the file as a whole. But then one cannot use the
> external bzip2/bunzip2 utilities on the state files anymore.
>
> What do you think of compression as such, and of the implementation?
>
> Best regards
> Raimar
>
>
>
>
>
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Cppqed-support mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppqed-support
>