#75 Flow sampling

[Michael Allen]

Would like the ability to specify flow sampling for x out of y flows. This will help scale high volume environments.
Perhaps also send a sampling option template to help define the sample rate for the collector side.

[ABC]

Are you sure this will help scale high volume traffic? (I don't.) Can you elaborate on that?

GIT version of module is already very high performance (probably would outperform conntrack). Tested to work on 10Gbit linux router.

[ABC]

Also I note you say 'flow sampling' not 'packet sampling'. So flows still need to be distinguished. Well, with appropriate algo that could reduce load. But which sampling algorithm do you want?

[Michael Allen]

As far as the load goes, I'm not necessarily questioning the capabilities of ipt_netflow. I'm thinking along the lines of datacenter switches running Cumulus for example. Collection can be a challenge in environments like this(100k/flows per second), so flow sampling might be a handy option here. Sampling is a necessity in some larger environments.
Having both deterministic and random sampling are options with Cisco sampling and can each have benefits so maybe give both equal weight.
I don't want to digress into another feature request, but spreading export across multiple collectors would also help in ultra high flow environments.

[ABC]

So you basically want to reduce load on collectors. I'm reading rfc7014 on flow sampling.

[ABC]

I'm principally not against of this feature(s). But

1. It seems to require a lot of changes (for example option templates support), so I want make next release before adding bulk of new code. [done]
2. Nobody else requested flow sampling before. So it seems like you are single person who needs this. I don't want to complicate code in vain. What is chance that you will actually using it?

[ABC]

Btw, do you have other devices which export sampled flows? I would like to see example captured packets for reference. If you have, send to my email from README or attach files here.

[Michael Allen]

I have sent you an email containing a couple packet captures. Please let me know if you didn't get them.

[ABC]

I got them. Thanks!

[ABC]

I added Options Template support, [7d10d1485040007cae40a6581c8b7aae2b4f6b14]. You may test if you wish. Next will be flow sampling.

[ABC]

I implemented Flow Sampling (random and deterministic) in the latest git commit [ab8c0eeef0d0e969962f1c25a983040c0262c791]. Please test.

[ABC]

I was concentrating on v9/ipfix extensively and forgot about doing sampling properly for v5. Thus, in netflow v5 mode random/deterministic sampling does not work. Sorry. I will fix it later. I will also add exporting of interface names/descriptions in options templates. (Found exmaple in your packet captures, thax again!)

[ABC]

Just commited correction to flow sampling on v5. [1d728487866212694b07c77a07e2d836b416c84f]

I will also add exporting of interface names/descriptions in options templates. (Found exmaple in your packet captures, thax again!)

Done that too.