Would like the ability to specify flow sampling for x out of y flows. This will help scale high volume environments.
Perhaps also send a sampling option template to help define the sample rate for the collector side.
Also I note you say 'flow sampling' not 'packet sampling'. So flows still need to be distinguished. Well, with appropriate algo that could reduce load. But which sampling algorithm do you want?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
As far as the load goes, I'm not necessarily questioning the capabilities of ipt_netflow. I'm thinking along the lines of datacenter switches running Cumulus for example. Collection can be a challenge in environments like this(100k/flows per second), so flow sampling might be a handy option here. Sampling is a necessity in some larger environments.
Having both deterministic and random sampling are options with Cisco sampling and can each have benefits so maybe give both equal weight.
I don't want to digress into another feature request, but spreading export across multiple collectors would also help in ultra high flow environments.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm principally not against of this feature(s). But
It seems to require a lot of changes (for example option templates support), so I want make next release before adding bulk of new code. [done]
Nobody else requested flow sampling before. So it seems like you are single person who needs this. I don't want to complicate code in vain. What is chance that you will actually using it?
Last edit: ABC 2014-08-07
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Btw, do you have other devices which export sampled flows? I would like to see example captured packets for reference. If you have, send to my email from README or attach files here.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I was concentrating on v9/ipfix extensively and forgot about doing sampling properly for v5. Thus, in netflow v5 mode random/deterministic sampling does not work. Sorry. I will fix it later. I will also add exporting of interface names/descriptions in options templates. (Found exmaple in your packet captures, thax again!)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Are you sure this will help scale high volume traffic? (I don't.) Can you elaborate on that?
GIT version of module is already very high performance (probably would outperform conntrack). Tested to work on 10Gbit linux router.
Also I note you say 'flow sampling' not 'packet sampling'. So flows still need to be distinguished. Well, with appropriate algo that could reduce load. But which sampling algorithm do you want?
As far as the load goes, I'm not necessarily questioning the capabilities of ipt_netflow. I'm thinking along the lines of datacenter switches running Cumulus for example. Collection can be a challenge in environments like this(100k/flows per second), so flow sampling might be a handy option here. Sampling is a necessity in some larger environments.
Having both deterministic and random sampling are options with Cisco sampling and can each have benefits so maybe give both equal weight.
I don't want to digress into another feature request, but spreading export across multiple collectors would also help in ultra high flow environments.
So you basically want to reduce load on collectors. I'm reading rfc7014 on flow sampling.
Last edit: ABC 2014-08-05
I'm principally not against of this feature(s). But
Last edit: ABC 2014-08-07
Btw, do you have other devices which export sampled flows? I would like to see example captured packets for reference. If you have, send to my email from README or attach files here.
I have sent you an email containing a couple packet captures. Please let me know if you didn't get them.
I got them. Thanks!
I added Options Template support, [7d10d1485040007cae40a6581c8b7aae2b4f6b14]. You may test if you wish. Next will be flow sampling.
Related
Commit: [7d10d1]
I implemented Flow Sampling (random and deterministic) in the latest git commit [ab8c0eeef0d0e969962f1c25a983040c0262c791]. Please test.
Related
Commit: [ab8c0e]
I was concentrating on v9/ipfix extensively and forgot about doing sampling properly for v5. Thus, in netflow v5 mode random/deterministic sampling does not work. Sorry. I will fix it later. I will also add exporting of interface names/descriptions in options templates. (Found exmaple in your packet captures, thax again!)
Just commited correction to flow sampling on v5. [1d728487866212694b07c77a07e2d836b416c84f]
Done that too.
Related
Commit: [1d7284]