Re: [Gtkwave-users] vcd2lxt2 performance writing to NFS disk (fseek & zlib)

[<by...@nc...>]

I don't know if all filesystems are created equal. At work, I've never
seen performance problems and the filesystems are definitely not
local.

For #1 that is an old trick. I was discussing that on the Amiga FB
group yesterday regarding stereo audio files (where the left and right
channels are each dumped in their entirety) because the FFS filesystem
is terrible for seeks. You keep multiple file descriptors open in
order not to have to seek--or minimize it as much as possible.
For #2, FST actually has an option for compress on close.

As far as LXT2 goes, there is always something to update. I'd like to
retire it at some point as I consider it made obsolete by FST.

Using mmap() variants and treating the data as memory might actually
have better performance, mainly because there might be some form of
more persistent locking in the buffer cache. I'm not quite sure how it
works in the kernel.

-Tony

	-----------------------------------------From: "R B" 

To: gtk...@li...
Cc: 
Sent: Tuesday January 22 2019 6:17:04PM
Subject: Re: [Gtkwave-users] vcd2lxt2 performance writing to NFS disk
(fseek & zlib)

Thanks. I'd say seeks are a bottleneck for most or all NFS.More naive
speculations (are block attributes updated multiple times?):
!. If code kept two independent open files for same output file, with
unique offsets:first's offset is always at EOF and always used for
appends (no seeks);second's offset remains at current-block attributes
via pwrite(), and only seeks to move to next block.
2. Or can an LXT2 file be literally uncompressed to start (different I
think from --depth=0, so no per-block attributes to update), and
compressed only by gzip after the fact? Then can such be generated to
stdout and piped thru gzip (so large uncompressed intermediate LXT2
never written to disk, and net effect is no seeks, no block rewrites)?
On Mon, Jan 21, 2019 at 11:23 PM  wrote:
This is for a 400MB vcd file:

620:/tmp/q> strace vcd2lxt2 z.vcd z.lxt >& zzz_lxt_seek
621:/tmp/q> strace vcd2fst z.vcd z.fst > & zzz_fst_seek
622:/tmp/q> strace vfast z.vcd > & zzz_fsdb_seek
623:/tmp/q> cat zzz_lxt_seek | grep seek | wc -l
141
624:/tmp/q> cat zzz_fst_seek | grep seek | wc -l
67
625:/tmp/q> cat zzz_fsdb_seek | grep seek | wc -l
19003

...FST should perform better for you if seeks are a bottleneck on your
NFS. I was quite amazed to see how high the number of seeks was for
FSDB.

-Tony

	-----------------------------------------From: by...@nc... [2]
To: "gtk...@li... [3]"
Cc: "gtk...@li... [4]"
Sent: Monday January 21 2019 6:55:40PM
Subject: Re: [Gtkwave-users] vcd2lxt2 performance writing to NFS disk
(fseek & zlib)

I forgot to answer why all the seeks...mainly we're dealing with a
block-based format and certain structures/fields need to update
dynamically such as blocks finalizing. In order to finalize a block,
we have to rewind back to where the various status/attributes are
stored and update them, then zoom back to the end of file so we can
start writing the next block. All the seeking should be within the
same/current block. In a different implementation, the block would be
completely memory resident and emitted in one huge chunk.

The usage of zlib is that some structures are zlib compressed in order
to cut down on space. I learned my lesson with FST as far as zlib
speed goes and moved to LZ4. Mainly, if you have a large amount of
smallish streams to compress, LZ4 performs far better, even if you
have to run it twice--which you can do because LZ4 is not a bit-based
compression scheme. There is a non-trivial amount of compressor state
that zlib needs to initialize that LZ4 doesn't, so zlib is not so good
performance-wise for numerous small chunks of data to be compressed.

-Tony
 _______________________________________________
 Gtkwave-users mailing list
 Gtk...@li... [5]
  [6]https://lists.sourceforge.net/lists/listinfo/gtkwave-users [7]
  

Links:
------
[1] mailto:by...@nc...
[2] mailto:by...@nc...
[3] mailto:gtk...@li...
[4] mailto:gtk...@li...
[5] mailto:Gtk...@li...
[6] https://lists.sourceforge.net/lists/listinfo/gtkwave-users
[7] https://lists.sourceforge.net/lists/listinfo/gtkwave-users