Silk data and FlowMonitor data is diffrent

[Domagoj]

Hy Joe,

Can you please explain how FlowMonitor caclulates "flows active/second" and "flows initiated/seconds" for FlowMonitor types?
When I check data over silk command I get diffrent values.
For example monitornig flows aktive per second:

root@l01netflow1:/var/pipeline/log# rwfilter --type=in,inweb --start-date=2017/01/17:14:10 --dcidr=iprange/19 --pass=stdout | rwcount --bin-size=1 --load-scheme=end

           Date|        Records|    Bytes|          Packets|

2017/01/17T14:00:00| 66.00| 38444.00| 68.00|
2017/01/17T14:00:01| 56.00| 32879.00| 61.00|
2017/01/17T14:00:02| 63.00| 46436.00| 73.00|
2017/01/17T14:00:03| 61.00| 32178.00| 64.00|
2017/01/17T14:00:04| 64.00| 19454.00| 64.00|
2017/01/17T14:00:05| 62.00| 36001.00| 65.00|
2017/01/17T14:00:06| 65.00| 42697.00| 66.00|
2017/01/17T14:00:07| 68.00| 35647.00| 71.00|

In FlowMonitor I get 5 time bigger values(I didn't used multiplayes) for "flows active/second".
Also, I did used multipalyer 1024 for "bits/second" monitor, is that OK?

[Joe Loiacono]

Hi Domagoj

Sorry for the delay. It is a good question you have. In general, FlowMonitor_Collector will gather about 30 minutes worth of netflow data and focus on a 5-minute period. For 'Flows-Initiated' it will accumulate a flow into the 5 minute period (i.e., increase the observed value for the period by 1) only if it's start-time is in that period. For 'Flows-Active', it will accumulate the flow into the period (i.e., increase the observed value for the period by 1) if it is active during any time of the period. Byte counts are ignored altogether since we're looking at flows only.

In the example you show above, with loadcheme=end, you are only counting flows whose end-time is in the period (in the case of FlowMonitor this would be a 5-minute bucket.) Under the Flows-Active scheme FlowMonitor will count as well any flow which is 'active' at all during the 5-minute window. That would count more flows including any that started in the period (and went past the 5-minute period end) or started before the period and ended after the 5-minute period end.

Can you reconcile the counts that way?

Also: yes - any multiplier is fine.

Joe

[Domagoj]

Hy Joe,

When I calculated average value for 5 minute period for 'Flows-Initiated per second' my SiLK data and FlowMonitor data are identical. For SUM value i should use multiplier 300 (for 5 minutes) - this is consistent.

For 'Flows-Active per second' I can not get identical values I have tried with every silk load-scheme there is.

Basicly, my data 'Flows-Active per second' are always greater then Flows-Initiated per second', this corresponds to your explanation.

Thank you for help Joe.

Regards

[Domagoj]

P.S
Im using --load-scheme=5 in my SiLK query.
In my FlowViewer_Configuration.pm conf loadscheme are:
$silk_init_loadscheme = 1; # For Flows Initiated/Second - see SiLK rwcount documentation
$silk_active_loadscheme = 5; # For Flows Active/Second - see SiLK rwcount documentation

Im getting slightly bigger values in SiLK query (10 - 20 flows bigger) for 'flows-Active per second'

[Joe Loiacono]

Hi Domagoj,

I think the difference is due to the way FlowViewer looks at the notion of "flows active". Let's look at a 35 second flow and 'bucket size' equal to 5 seconds. For that single flow, Flowviewer will increase the count of active flows in seven buckets (7 * 5 = 35). SiLK still counts that as only one flow. The load-scheme asks 'where should I place the bytes associated with this flow. FlowViewer ignores bytes when creating graphs on Flows-initiated or Flows-active.

The Flows-Active option shows how many flows were ongoing at any time. Flows-Initiated shows only how many flows were initiated (start-time) during a time-period of length bucket-size. So in the above case, the 35 second flow would increase the count of only one bucket - namely the one that contains the flow start-time within the bucket period

So, Flows-Active shouldn't be used to measure total flows and will (almost) always be much bigger than Flow-Initiated. The exception would be if you only encountered shorter flows all of whose start and end times happened to fall completely within a bucket.