Re: [Nfdump-discuss] Multiple Input data files to nfdump

[Glenn F. F. L. <gl...@co...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The -R option is flexible enough, I think, but comes with certain 
constraints (:

   - the directories you want all need to be at the same hierarchical
      level ( as they are in your example below);

   - the directories you want need to be sequential (as I suspect they
      *aren't* in your example below - I presume each of BNE-BRD-1 and
      SYD-BRD-1 have an structure of YYYY/MM/DD below them, yes?);

   - the files you want need to go from the lexigraphically last in the
      first directory to the lexigraphically first in the last directory,
      i.e. you could go from BNE-BRD-1/.../nfcapd.200910151420 through
      all the remaining files BNE-BRD-1 that were later in the day than
      1420,  then all the files in BNE-SYD-1/ that were earlier in the day
      than 1420, and finally BNE-SYD-1/...nfcapd.200910151420 itself. I'm
      quite sure your datafiles and what you're trying to do don't line
      up this way.

If you don't mind a heavy dose of cruft, you could wrapper nfdump in your 
scripting language of choice, and write a script that would:

   - take a work direcory and a list of files as arguments;
   - create soft links in the work directory to each of the list of files;
   - use "nfdump -R" over the work directory to perform the processing;
   - clean the soft links out of the work directory and exit.

 	-g
- --
Glenn Forbes Fleming Larratt
Cornell University IT Security Office

On Wed, 14 Oct 2009, Jason Luxton wrote:

> Hi All,
>
> This seem like a simple request and I'm sure the answer is staring me in the face.
>
> How do I supply a list of data files collected by nfcapd  to processed by nfdump?
>
> I have tried to cat all the neccessary files together and pipe them into nfdump as follows but also get a 'corrupt data file' message.  The individual files are fine.
>
> <snip>
> jasonl@syd-netflow-01$ cat BNE-BRD-1/2009/10/15/nfcapd.200910151420 SYD-BRD-1/2009/10/15/nfcapd.200910151420 | nfdump -s dstip:p
> Can't process block type 20. Skip block.
> Skip corrupt data file '': 'Corrupt data file: Requested buffer size 759452226 exceeds max. buffer size.
> '
> Top 10 Dst IP Addr ordered by flows:
> 2009-10-15 14:13:43.910   667.061 UDP      xxx.xxx.xxx.xxx    24957( 4.6)    32953( 0.3)    4.1 M( 0.1)       49    49273   124
> 2009-10-15 14:12:45.521   720.938 TCP      xxx.xxx.xxx.xxx     8571( 1.6)   153038( 1.6)  145.2 M( 2.3)      212    1.6 M   948
> 2009-10-15 14:18:50.765   339.602 UDP      xxx.xxx.xxx.xxx     6666( 1.2)     6978( 0.1)   782377( 0.0)       20    18430   112
> ...
> </snip>
>
> I am using a snapshot of nfdump as below but have found the same problem on version 1.5.7.
>
> <snip>
> nfdump: Version: snapshot-1.6b-20090930 $LastChangedDate: 2009-09-30 10:04:28 +0200 (Wed, 30 Sep 2009) $
> $Id: nfdump.c 31 2009-09-30 08:04:28Z haag $
> </snip>
>
> I can't use multiple '-r' options and -R requires the files to be in the same directory.  Using the -M option to specify multiple directories doesn't help me either.  Maybe because the files have the same name but in different directories?
>
> I'm sure this is a user error but yet to find out how.
>
> Cheers
> Jason
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iEYEARECAAYFAkrW/hUACgkQLyw7nZwiKgQK6gCglX5SHgklXqxGDmrlSmCEXLYC
3gsAoKErleycV9OUIwsh0pWF+YCz/k9/
=3FK4
-----END PGP SIGNATURE-----