Re: [Nfdump-discuss] Compression possibilities of nfdump (and one weird issue)

[Peter H. <ph...@us...>]

bzip2 is now part of 1.6.14. However, it's not recommended for realtime collecting flows, as it slows down too much in
high volume traffic environments.

use -z for lzo - still prefered, while collecting.
use -j for bzip2 for archiving data or lightly loaded servers

Change compression of file: -J <nun> 0: uncompressed, 1: LZO-1, 2: bzip2

	- Peter

On 06.09.15 13:35, Peter Haag wrote:
> Hi,
> the compress algorithm was used as best compromise between speed, result and CPU usage.
> I also did test bzip2 and - yes - you get more out of disk space, but it's more CPU intensive.
> 
> Anyway, it would be possible to add another compression along the existing one and let the user choose.
> So if this is a demand from other people too, we can happily have a lookk together.
> 
> Cheers
> 
> 	- Peter
> 
> 
> On 05/09/15 16:01, Miroslav Kratochvil wrote:
>> Hello list,
>>
>> I was wondering whether it is possible to use any better compression for nfdump files than the supplied LZO1X-1. By simple measurements at my site, the files compressed by the nfdump are shrank to around 38% of the original size, while using bzip2
>> (speed of which is still bearable for many environments) gets to around 20% of the original.
>>
>> For people who need to archive the netflow data, almost two times smaller archives could be a great space/time saver. CPU usage of the compression is IMHO not a real issue on current machines.
>>
>> Is there any possibility to replace the LZO compression by something else? I'd happily code a patch for bzip2 support -- is that a viable option, or is there some reason not to use this specific compression? (I've seen github migration is underway,
>> I'd just send a merge request if it'd been finished :] )
>>
>> Thanks a lot in advance,
>> -mk
>>
>>
>> PS.:
>> Also, on a related note, when trying to code the compression around, I've seen this weird error unexplainable by traditional means:
>>
>> $ cat < somefile |nfdump -r -
>> [... data are OK]
>>
>> $ bzip2 somefile
>> $ bzip2 -c -d < somefile.bz2 |nfdump -r -
>> Date flow start Duration Pr[....]
>> Read error in file '(null)': Success
>> No matched flows
>>
>> Any explanation?
>>
>>

-- 
Be nice to your netflow data. Use NfSen and nfdump :)