Re: [Nfdump-discuss] How to use the nfdump -t <timewin> option for RRDTool integration?

[Peter V. M. <pe...@mo...>]

I never got any reply to this thread, so I've created a patch based on the
patch in https://sourceforge.net/p/nfdump/bugs/22/. With this new patch, -t
timewin behaves as it always has, unless -S is given also. Then -t timewin
will only consider the start of a flow to determine whether it matches or
not.

Apparently we are several that would like -t to behave differently, and
this version of the patch does not introduce any backwards-compatibility
issues. Win-win for all, as I see it.

It is a git patch (that also applies cleanly with GNU patch) against
1.6.11. Attached and available here:
https://github.com/pmorch/nfdump/commit/d9ae3d94036639e9b08caa753bd1c867d39d27c8.

I hope it can be included. Or at least commented on.

If I don't hear anything by next week, I'll add it to sourceforge as a
patch and comment to bugs 25 and 22 where I found the original, older patch.

Sincerely,

Peter Mørch


On Fri, Nov 29, 2013 at 1:39 PM, Peter Valdemar Mørch <pe...@mo...>wrote:

> It looks like I have been bitten by the same behavior as described in
> http://sourceforge.net/p/nfdump/bugs/25/ and
> https://sourceforge.net/p/nfdump/bugs/22/. So I guess Peter understands
> the problem but has decided not to do anything.
>
> Ok.
>
> In the mean time, the patch in https://sourceforge.net/p/nfdump/bugs/22/can be used to achieve what I'm trying to achieve. (It fails because there
> are whitespaces differences since bugs/22/ until 1.6.11, but the logic is
> still the same.)
>
> Peter, if someone were to introduce two new options, e.g. -ts and -te that
> behave like -t but operate only on start and end, would you be open to
> accepting such a patch?
>
> Sincerely,
>
> Another :-) Peter
>
>
> On Fri, Nov 29, 2013 at 12:12 AM, Peter Valdemar Mørch <pe...@mo...>wrote:
>
>> Hi,
>>
>> I want to create statistics for every 5-minute interval based on a filter
>> for use with something similar to RRDTool and I'd like to use nfdump to get
>> it.
>>
>> The simple solution is to just use the appropriately named 5-minute file
>> nfcapd.201311281820, and run a filter on that. But I'm curious as to
>> whether I'm missing a way to do this with the -t option or some other
>> option.
>>
>> Naively, I tried -t 2013/11/28.18:10-2013/11/28.18:15 to get statistics
>> about all the data in the 18:10 - 18:15 interval. But that returned zero
>> flows. It looks like nfdump only includes flows where the entire period is
>> within the to-from times in -t. See "details" below. (Would be nice if man
>> nfdump went into a little more detail about -t)
>>
>> But, if some flows are short lived (e.g. webserver) and some are
>> long-lived (e.g. ssh-connection) I don't see how I can use the -t option to
>> get an idea of how much traffic occurred between 18:10 and 18:15.
>>
>> I guess I was hoping for a -t option that looked exclusively at e.g. flow
>> end, or some way to use flow-end in a filter. Then I could  get an idea
>> about traffic in a 5minute period. Where long-running flows would be
>> calculated as-if they occurred entirely at the flow-end time, but this is
>> exactly what you get by looking at one nfcapd.* file at a time, isn't it?
>>
>> The Rolls Royce would be that if a flow ran from 18:09-18:13, 75% of the
>> traffic from that flow would be added to the 18:10-18:15, because 75% of
>> the time is within that period, but hey, I can see that's a little wild.
>>
>> I also tried to experiment with +/-10, but could not get this to work at
>> all.
>>
>> What we *can* do is use only a single time for -t as in "-t
>> 2013/11/28.18:05" and then look at every flow and discard any that end
>> outside the 18:10-18:15 interval. But that is very time consuming
>> especially because one needs to comparisons on time strings such as e.g.
>> "2013-11-28 18:27:46.362". (It would be nice if there was something similar
>> to -N that printed times as unixtime, so the heavy conversion (also in
>> nfdump?) isn't necessary.)
>>
>> Can -t be used to get data from 18:10-18:15? Am I missing something
>> (else) obvious? ( Appart from "just use nfsen" - I have other reasons not
>> to, and I'm trying to understand -t in nfdump )
>>
>> Sincerely,
>>
>> Peter
>>
>> ==========
>>    Details
>> ==========
>>
>> I have a test nfcapd file[1], with NetFlow records from nothing but a
>> single long-running ssh connection.
>>
>> Looking at all the data in the file, I get this:
>>
>> > nfdump -r singleSSH.nfcapd -o 'fmt: %td %ts %te %sa:%sp %da:%dp'
>>   Duration Date flow start         Date flow end                Src IP
>> Addr Src Pt      Dst IP Addr Dst Pt
>>    <snip>
>>    300.716 2013-11-28 17:56:31.752 2013-11-28 18:01:32.468    1.2.3.4:
>> 2222   172.22.216.119: 43654
>>    300.710 2013-11-28 18:01:42.485 2013-11-28 18:06:43.195    1.2.3.4:
>> 2222   172.22.216.119: 43654
>>    300.796 2013-11-28 18:06:53.207 2013-11-28 18:11:54.003    1.2.3.4:
>> 2222   172.22.216.119: 43654
>>    310.813 2013-11-28 18:12:04.019 2013-11-28 18:17:14.832    1.2.3.4:
>> 2222   172.22.216.119: 43654
>>    <snip>
>> Summary: total flows: 21, total bytes: 12.1 M, total packets: 33103, avg
>> bps: 14730, avg pps: 5, avg bpp: 364
>> Time window: 2013-11-28 17:20:09 - 2013-11-28 19:09:22
>> Total flows processed: 21, Blocks skipped: 0, Bytes read: 1168
>> Sys: 0.008s flows/second: 2625.0     Wall: 0.000s flows/second: 30882.4
>>
>> And experimenting with the -t option, I see that
>> > nfdump -r singleSSH.nfcapd -o 'fmt: %td %ts %te %sa:%sp %da:%dp' -t
>> 2013/11/28.18:12:04-2013/11/28.18:17:14
>> is the tightest I can go with -t and still get any data around that time
>> period.
>>
>> Assuming I know that the NetFlow collector transmits every 5 minutes, I
>> guess I could do -t 18:10-18:20 and then know that I'll likely only get one
>> flow record from the long-running ssh connection, and then in the next
>> period use 18:15-18:25. That would work most of the time (tm) for the ssh
>> connection. But any short lived flows e.g. entirely inside 18:18 will be
>> counted in both intervals. :-(
>>
>> About +/-10: How am I supposed to use this?
>>
>> > nfdump -r singleSSH.nfcapd -o 'fmt: %td %ts %te %sa:%sp %da:%dp' -t +10
>> Time Window error: No time slot information available
>> > nfdump -r singleSSH.nfcapd -o 'fmt: %td %ts %te %sa:%sp %da:%dp' -t -10
>> Time Window error: No time slot information available
>> > nfdump -r singleSSH.nfcapd -o 'fmt: %td %ts %te %sa:%sp %da:%dp' -t
>> 2013/11/28.18:12:04+10
>> Time format error at '04+10': unexpected character: '+'.
>> > nfdump -r singleSSH.nfcapd -o 'fmt: %td %ts %te %sa:%sp %da:%dp' -t
>> 2013/11/28.18:12:04-10
>> Time format error: '10' unexpected.
>>
>> 1: attached and at http://ge.tt/5Rthyl41/v/0?c
>>
>> --
>> Peter Valdemar Mørch
>> http://www.morch.com
>>
>
>
>
> --
> Peter Valdemar Mørch
> http://www.morch.com
>



-- 
Peter Valdemar Mørch
http://www.morch.com