#75 Flow sampling

git version
open
nobody
None
5
2014-08-29
2014-08-04
No

Would like the ability to specify flow sampling for x out of y flows. This will help scale high volume environments.
Perhaps also send a sampling option template to help define the sample rate for the collector side.

Discussion

  • ABC

    ABC - 2014-08-05

    Are you sure this will help scale high volume traffic? (I don't.) Can you elaborate on that?

    GIT version of module is already very high performance (probably would outperform conntrack). Tested to work on 10Gbit linux router.

     
  • ABC

    ABC - 2014-08-05

    Also I note you say 'flow sampling' not 'packet sampling'. So flows still need to be distinguished. Well, with appropriate algo that could reduce load. But which sampling algorithm do you want?

     
  • Michael Allen

    Michael Allen - 2014-08-05

    As far as the load goes, I'm not necessarily questioning the capabilities of ipt_netflow. I'm thinking along the lines of datacenter switches running Cumulus for example. Collection can be a challenge in environments like this(100k/flows per second), so flow sampling might be a handy option here. Sampling is a necessity in some larger environments.
    Having both deterministic and random sampling are options with Cisco sampling and can each have benefits so maybe give both equal weight.
    I don't want to digress into another feature request, but spreading export across multiple collectors would also help in ultra high flow environments.

     
  • ABC

    ABC - 2014-08-05

    So you basically want to reduce load on collectors. I'm reading rfc7014 on flow sampling.

     
    Last edit: ABC 2014-08-05
  • ABC

    ABC - 2014-08-07

    I'm principally not against of this feature(s). But

    1. It seems to require a lot of changes (for example option templates support), so I want make next release before adding bulk of new code. [done]
    2. Nobody else requested flow sampling before. So it seems like you are single person who needs this. I don't want to complicate code in vain. What is chance that you will actually using it?
     
    Last edit: ABC 2014-08-07
  • ABC

    ABC - 2014-08-08

    Btw, do you have other devices which export sampled flows? I would like to see example captured packets for reference. If you have, send to my email from README or attach files here.

     
    • Michael Allen

      Michael Allen - 2014-08-08

      I have sent you an email containing a couple packet captures. Please let me know if you didn't get them.

       
      • ABC

        ABC - 2014-08-08

        I got them. Thanks!

         
  • ABC

    ABC - 2014-08-27

    I was concentrating on v9/ipfix extensively and forgot about doing sampling properly for v5. Thus, in netflow v5 mode random/deterministic sampling does not work. Sorry. I will fix it later. I will also add exporting of interface names/descriptions in options templates. (Found exmaple in your packet captures, thax again!)

     
  • ABC

    ABC - 2014-08-29

    Just commited correction to flow sampling on v5. [1d728487866212694b07c77a07e2d836b416c84f]

    I will also add exporting of interface names/descriptions in options templates. (Found exmaple in your packet captures, thax again!)

    Done that too.

     

    Related

    Commit: [1d7284]


Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks