Re: [Nfdump-discuss] Multiple Input data files to nfdump
netflow collecting and processing tools
Brought to you by:
phaag
|
From: Glenn F. F. L. <gl...@co...> - 2009-10-15 10:49:23
|
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
The -R option is flexible enough, I think, but comes with certain
constraints (:
- the directories you want all need to be at the same hierarchical
level ( as they are in your example below);
- the directories you want need to be sequential (as I suspect they
*aren't* in your example below - I presume each of BNE-BRD-1 and
SYD-BRD-1 have an structure of YYYY/MM/DD below them, yes?);
- the files you want need to go from the lexigraphically last in the
first directory to the lexigraphically first in the last directory,
i.e. you could go from BNE-BRD-1/.../nfcapd.200910151420 through
all the remaining files BNE-BRD-1 that were later in the day than
1420, then all the files in BNE-SYD-1/ that were earlier in the day
than 1420, and finally BNE-SYD-1/...nfcapd.200910151420 itself. I'm
quite sure your datafiles and what you're trying to do don't line
up this way.
If you don't mind a heavy dose of cruft, you could wrapper nfdump in your
scripting language of choice, and write a script that would:
- take a work direcory and a list of files as arguments;
- create soft links in the work directory to each of the list of files;
- use "nfdump -R" over the work directory to perform the processing;
- clean the soft links out of the work directory and exit.
-g
- --
Glenn Forbes Fleming Larratt
Cornell University IT Security Office
On Wed, 14 Oct 2009, Jason Luxton wrote:
> Hi All,
>
> This seem like a simple request and I'm sure the answer is staring me in the face.
>
> How do I supply a list of data files collected by nfcapd to processed by nfdump?
>
> I have tried to cat all the neccessary files together and pipe them into nfdump as follows but also get a 'corrupt data file' message. The individual files are fine.
>
> <snip>
> jasonl@syd-netflow-01$ cat BNE-BRD-1/2009/10/15/nfcapd.200910151420 SYD-BRD-1/2009/10/15/nfcapd.200910151420 | nfdump -s dstip:p
> Can't process block type 20. Skip block.
> Skip corrupt data file '': 'Corrupt data file: Requested buffer size 759452226 exceeds max. buffer size.
> '
> Top 10 Dst IP Addr ordered by flows:
> 2009-10-15 14:13:43.910 667.061 UDP xxx.xxx.xxx.xxx 24957( 4.6) 32953( 0.3) 4.1 M( 0.1) 49 49273 124
> 2009-10-15 14:12:45.521 720.938 TCP xxx.xxx.xxx.xxx 8571( 1.6) 153038( 1.6) 145.2 M( 2.3) 212 1.6 M 948
> 2009-10-15 14:18:50.765 339.602 UDP xxx.xxx.xxx.xxx 6666( 1.2) 6978( 0.1) 782377( 0.0) 20 18430 112
> ...
> </snip>
>
> I am using a snapshot of nfdump as below but have found the same problem on version 1.5.7.
>
> <snip>
> nfdump: Version: snapshot-1.6b-20090930 $LastChangedDate: 2009-09-30 10:04:28 +0200 (Wed, 30 Sep 2009) $
> $Id: nfdump.c 31 2009-09-30 08:04:28Z haag $
> </snip>
>
> I can't use multiple '-r' options and -R requires the files to be in the same directory. Using the -M option to specify multiple directories doesn't help me either. Maybe because the files have the same name but in different directories?
>
> I'm sure this is a user error but yet to find out how.
>
> Cheers
> Jason
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
iEYEARECAAYFAkrW/hUACgkQLyw7nZwiKgQK6gCglX5SHgklXqxGDmrlSmCEXLYC
3gsAoKErleycV9OUIwsh0pWF+YCz/k9/
=3FK4
-----END PGP SIGNATURE-----
|