FlowMonitor Graphs not working redux
FlowViewer is a web-based netflow data analysis tool.
Brought to you by:
jloiacon
Menu
▾
▴
So the thread here seems to have died...:
http://sourceforge.net/p/flowviewer/discussion/general/thread/ef715db0/
So trying again.
Same problem as described there, FlowMonitor renders graphs but there's no data in them, all values are 0 and nothing is being graphed. FV & FG work fine and show the flows.
I've pasted what I think is all the relevant info below. Let me know if additional outputs or data is needed. Thanks!
Tim
DEBUG_MONITOR_C output:
FlowMonitor_Collector_info output:
DEBUG_MONITOR_G output:
Tim,
First my apologies ... new (very busy) work assignment and (required!) vacation.
This line of DEBUG is key:
TME Lab Gateway 1438634100:0
It shows that the update value, which will be put into RRDtool, unfortunately is '0'.
Next time you get a chance, look into DEBUG_MONITOR_C and execute the main command it is issuing for your FlowMonitor, from a command line and see what is happening. i.e.,:
rwfilter_command: /usr/local/bin/rwfilter --data-rootdir=/data --type=all --sensors=tme-lab-gwy --start-date=2015/08/03:11 --end-date=2015/08/03:13 --active=2015/08/03:13:30:00-2015/08/03:13:35:00 --pass=/var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Prefiltered_Site_alltme-lab-gwy
silk_command: /usr/local/bin/rwfilter --pass=stdout /var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Prefiltered_Site_alltme-lab-gwy | /usr/local/bin/rwcount --bin-size=300 --start-time=2015/08/03:13:30:00 --end-time=2015/08/03:13:35:00 --epoch-slots --no-titles > /var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Collector_output
You may have to adjust intermediate file names, but this can offer some insight.
Joe
Hi Joe - thank you for the reply!
So I ran the two commands in the DEBUG_C file, the the first rwfilter command works fine, but the second returns the error below:
tstevens@tstevens-silk:~/FV/Flow_Working$ /usr/local/bin/rwfilter --data-rootdir=/data --type=all --sensors=tme-lab-gwy --start-date=2015/08/14:14 --end-date=2015/08/14:17 --active=2015/08/14:16:50:00-2015/08/14:16:55:00 --pass=/var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Prefiltered_Site_alltme-lab-gwy
tstevens@tstevens-silk:~/FV/Flow_Working$ /usr/local/bin/rwfilter --pass=stdout /var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Prefiltered_Site_alltme-lab-gwy | /usr/local/bin/rwcount --bin-size=300 --start-time=2015/08/14:16:50:00 --end-time=2015/08/14:16:55:00 --epoch-slots --no-titles > /var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Collector_output
rwfilter: Must specify partitioning rules when using --pass-destination
Use 'rwfilter --help' for usage
rwcount: Error processing headers on file '-': Unexpected end of file while reading header
tstevens@tstevens-silk:~/FV/Flow_Working$
Thanks,
Tim
Hi Tim,
I'm wondering if FlowMonitor_Collector can get to one of the subroutines it needs: 'create_ipfix_filter'. This could be happening if FlowMonitor_Collector is not started out of the cgi-bin directory that FlowViewer_Utilities.pm is in. Can you make sure (e.g., ensuring flowmonitor_startup points to your cgi-bin directory: FlowViewer=/var/www/cgi-bin/FlowViewer_4.6) you start up FlowMonitor_Collector in the FlowViewer cgi-bin directory.
Thanks,
Joe
Hi Joe,
So from what I can see, everything's where it should be. I don't see the exact variables you're referring to, but what I have is the following line in FlowViewer_Configuration.pm:
$cgi_bin_directory = "/var/www/cgi-bin/FlowViewer";
Both FlowMonitor_Collector & FlowViewer_Utilities.pm are in that directory.
I found that if I do actually manually insert some partitioning switch into that second command, it is creating the file, like so (adding the --any-address=10.x.x.x switch in the 2nd command):
tstevens@tstevens-silk:/var/www/cgi-bin/FlowViewer/Flow_Working$ /usr/local/bin/rwfilter --data-rootdir=/data --type=all --sensors=tme-lab-gwy --start-date=2015/08/14:14 --end-date=2015/08/14:17 --active=2015/08/14:16:50:00-2015/08/14:16:55:00 --pass=/var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Prefiltered_Site_alltme-lab-gwy
tstevens@tstevens-silk:/var/www/cgi-bin/FlowViewer/Flow_Working$ /usr/local/bin/rwfilter --any-address=10.x.x.x --pass=stdout /var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Prefiltered_Site_alltme-lab-gwy | /usr/local/bin/rwcount --bin-size=300 --start-time=2015/08/14:16:50:00 --end-time=2015/08/14:16:55:00 --epoch-slots --no-titles > /var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Collector_output
tstevens@tstevens-silk:/var/www/cgi-bin/FlowViewer/Flow_Working$ cat /var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Collector_output
1439596200| 160.77| 545626.00| 5159.02|
1439596500| 5.36| 237015.70| 4553.49|
tstevens@tstevens-silk:/var/www/cgi-bin/FlowViewer/Flow_Working$
But the next time the periodic Grapher/Collector processes run, that file can't be found any more (presumably due to the "rwfilter: Must specify partitioning rules when using --pass-destination" error).
Any other suggestions appreciated, thanks for your time!
Tim
Tim,
Please replace your current FlowMonitor_Collector with the attached. Rename the existing one (e.g., FlowMonitor_Collector_orig) and then rename the new, attached, to FlowMonitor_Collector and run it.
Please then share the debug with me.
Hi Joe,
Thought I posted a reply here but looks like it didn't stick. So with this version of FM_C I do see the flow monitor graphs being populated now (note that the small 'dashboard' graphs are still not being populated with data). Below is the output of DEBUG_MONITOR_C.
Thanks,
Tim
tstevens@tstevens-silk:~/FV/Flow_Working$ cat DEBUG_MONITOR_C
from: end this_filter to: start next_filter elapsed seconds: 300.006994 running: 255002.609302
/var/www/cgi-bin/FlowMonitor_Files/FlowMonitor_Filters/f2e_fp_testbed.fil
from: start next_filter to: start SiLK_processing elapsed seconds: 0.003149 running: 255002.612451
partitioning_switches: --duration=0.0-
rwfilter_command: /usr/local/bin/rwfilter --data-rootdir=/data --type=all --sensors=tstevens-fp1,tstevens-fp2,tstevens-fp5,tstevens-fp6 --start-date=2015/09/17:05 --end-date=2015/09/17:07 --active=2015/09/17:07:25:00-2015/09/17:07:30:00 --pass=/var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Prefiltered_Site_alltstevens-fp1tstevens-fp2tstevens-fp5tstevens-fp6
silk_command: /usr/local/bin/rwfilter --duration=0.0- --pass=stdout /var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Prefiltered_Site_alltstevens-fp1tstevens-fp2tstevens-fp5tstevens-fp6 | /usr/local/bin/rwcount --bin-size=300 --start-time=2015/09/17:07:25:00 --end-time=2015/09/17:07:30:00 --epoch-slots --no-titles > /var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Collector_output
from: start SiLK_processing to: end SiLK_processing elapsed seconds: 0.204984 running: 255002.817435
from: end SiLK_processing to: start SiLK_BINS elapsed seconds: 0.000109 running: 255002.817544
from: start SiLK_BINS to: end SiLK_BINS elapsed seconds: 0.000144 running: 255002.817688
from: end SiLK_BINS to: start RRDtool_update elapsed seconds: 0.000978 running: 255002.818666
F2E FP Testbed 1442500200:2490145527
from: start RRDtool_update to: end RRDtool_update elapsed seconds: 0.002850 running: 255002.821516
from: end RRDtool_update to: end this_filter elapsed seconds: 0.000040 running: 255002.821556
from: end this_filter to: start next_filter elapsed seconds: 0.000019 running: 255002.821575
/var/www/cgi-bin/FlowMonitor_Files/FlowMonitor_Filters/f3_fp_testbed.fil
from: start next_filter to: start SiLK_processing elapsed seconds: 0.001309 running: 255002.822884
partitioning_switches: --duration=0.0-
rwfilter_command: /usr/local/bin/rwfilter --data-rootdir=/data --type=all --sensors=tstevens-7710,tstevens-7710a,tstevens-7706-1,tstevens-7706-1a --start-date=2015/09/17:05 --end-date=2015/09/17:07 --active=2015/09/17:07:25:00-2015/09/17:07:30:00 --pass=/var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Prefiltered_Site_alltstevens-7710tstevens-7710atstevens-7706-1tstevens-7706-1a
silk_command: /usr/local/bin/rwfilter --duration=0.0- --pass=stdout /var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Prefiltered_Site_alltstevens-7710tstevens-7710atstevens-7706-1tstevens-7706-1a | /usr/local/bin/rwcount --bin-size=300 --start-time=2015/09/17:07:25:00 --end-time=2015/09/17:07:30:00 --epoch-slots --no-titles > /var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Collector_output
from: start SiLK_processing to: end SiLK_processing elapsed seconds: 0.006106 running: 255002.828990
from: end SiLK_processing to: start SiLK_BINS elapsed seconds: 0.000033 running: 255002.829023
from: start SiLK_BINS to: end SiLK_BINS elapsed seconds: 0.000054 running: 255002.829077
from: end SiLK_BINS to: start RRDtool_update elapsed seconds: 0.001067 running: 255002.830144
F3 FP Testbed 1442500200:0
from: start RRDtool_update to: end RRDtool_update elapsed seconds: 0.002462 running: 255002.832606
from: end RRDtool_update to: end this_filter elapsed seconds: 0.000036 running: 255002.832642
from: end this_filter to: start next_filter elapsed seconds: 0.000016 running: 255002.832658
/var/www/cgi-bin/FlowMonitor_Files/FlowMonitor_Filters/tme_lab_gateway.fil
from: start next_filter to: start SiLK_processing elapsed seconds: 0.001301 running: 255002.833959
partitioning_switches: --duration=0.0-
rwfilter_command: /usr/local/bin/rwfilter --data-rootdir=/data --type=all --sensors=tme-lab-gwy --start-date=2015/09/17:05 --end-date=2015/09/17:07 --active=2015/09/17:07:25:00-2015/09/17:07:30:00 --pass=/var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Prefiltered_Site_alltme-lab-gwy
silk_command: /usr/local/bin/rwfilter --duration=0.0- --pass=stdout /var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Prefiltered_Site_alltme-lab-gwy | /usr/local/bin/rwcount --bin-size=300 --start-time=2015/09/17:07:25:00 --end-time=2015/09/17:07:30:00 --epoch-slots --no-titles > /var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Collector_output
from: start SiLK_processing to: end SiLK_processing elapsed seconds: 0.010462 running: 255002.844421
from: end SiLK_processing to: start SiLK_BINS elapsed seconds: 0.000037 running: 255002.844458
from: start SiLK_BINS to: end SiLK_BINS elapsed seconds: 0.000068 running: 255002.844526
from: end SiLK_BINS to: start RRDtool_update elapsed seconds: 0.001066 running: 255002.845592
TME Lab Gateway 1442500200:64857
from: start RRDtool_update to: end RRDtool_update elapsed seconds: 0.002150 running: 255002.847742
from: end RRDtool_update to: end this_filter elapsed seconds: 0.000032 running: 255002.847774
from: end this_filter to: start next_filter elapsed seconds: 0.000012 running: 255002.847786
/var/www/cgi-bin/FlowMonitor_Files/FlowMonitor_Filters/tstevens_lab_gateway.fil
from: start next_filter to: start SiLK_processing elapsed seconds: 0.001137 running: 255002.848923
partitioning_switches: --duration=0.0-
rwfilter_command: /usr/local/bin/rwfilter --data-rootdir=/data --type=all --sensors=tstevens-lab-gwy --start-date=2015/09/17:05 --end-date=2015/09/17:07 --active=2015/09/17:07:25:00-2015/09/17:07:30:00 --pass=/var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Prefiltered_Site_alltstevens-lab-gwy
silk_command: /usr/local/bin/rwfilter --duration=0.0- --pass=stdout /var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Prefiltered_Site_alltstevens-lab-gwy | /usr/local/bin/rwcount --bin-size=300 --start-time=2015/09/17:07:25:00 --end-time=2015/09/17:07:30:00 --epoch-slots --no-titles > /var/www/cgi-bin/FlowViewer/Flow_Working/FlowMonitor_Collector_output
from: start SiLK_processing to: end SiLK_processing elapsed seconds: 0.117235 running: 255002.966158
from: end SiLK_processing to: start SiLK_BINS elapsed seconds: 0.000122 running: 255002.966280
from: start SiLK_BINS to: end SiLK_BINS elapsed seconds: 0.000232 running: 255002.966512
from: end SiLK_BINS to: start RRDtool_update elapsed seconds: 0.001692 running: 255002.968204
tstevens Lab Gateway 1442500200:5040244
from: start RRDtool_update to: end RRDtool_update elapsed seconds: 0.005865 running: 255002.974069
from: end RRDtool_update to: end this_filter elapsed seconds: 0.000092 running: 255002.974161
tstevens@tstevens-silk:~/FV/Flow_Working$
Thanks for posting the debug Tim.
It appears that you have created FlowMonitors for which no filtering criteria have been set. For example, no Source IP addresses, no Dest IP addresses, no Source Interfaces, etc. have been specified. What this produces is a sum of all trafic going through all interfaces that are exporting netflow. In the case of a router with all interfaces exporting, it would be the sum of all traffic going in every direction during that 5 minute span. This may not be what you want.
As an example, filtering on interfaces is a good way to begin to understand the traffic through your devices.
Without any filtering, you have wound up with:
F2E FP Testbed 1442500200:2490145527
F3 FP Testbed 1442500200:0
TME Lab Gateway 1442500200:64857
tstevens Lab Gateway 1442500200:5040244
(Note: the 1442500200 is the epoch time at the beginning of teh 5-minute period; the number to the right is the value for that period)
The problem was SiLK requires some partitioning switches (i.e., filter fields) and will die without some. The change to the code was to add "--duration=0.0-" if the filter is empty. This should still permit everything through.
I really appreciate your help. I will add this change to the next version.
Joe
I take it back, the small dashboard graphs are indeed updating as well. Thanks!
Tim
BTW, thanks a lot for looking into it and adding that change!
I have a similar problem with FlowViewer 4.6 (and Silk 3.10). FlowViewer and FlowGrapher work properly, I can see flow information and graphs of the data coming from a node of mine running Yaf.
When I create a Monitor the following files and folders get created on the filesystem as expected:
I have checked, the rrd file is no more updated after creation and no image gets created under /var/www/html/FlowMonitor/test.
My configuration is:
$reports_directory = "/var/www/html/FlowViewer"; $reports_short = "/FlowViewer"; $graphs_directory = "/var/www/html/FlowGrapher"; $graphs_short = "/html/FlowGrapher"; $monitor_directory = "/var/www/html/FlowMonitor"; $monitor_short = "/html/FlowMonitor"; $cgi_bin_directory = "/usr/lib/cgi-bin/FlowViewer_4.6"; $cgi_bin_short = "/cgi-bin/FlowViewer_4.6"; $work_directory = "/usr/lib/cgi-bin/FlowViewer_4.6/Flow_Working"; $save_directory = "/var/www/html/FlowViewer_Saves"; $save_short = "/FlowViewer_Saves"; $names_directory = "/usr/lib/cgi-bin/FlowViewer_4.6"; $ipset_directory = "/usr/lib/cgi-bin/FlowViewer_4.6"; # Where FlowViewer can find IPset files $filter_directory = "/usr/lib/cgi-bin/FlowMonitor_Files/FlowMonitor_Filters"; $rrdtool_directory = "/usr/lib/cgi-bin/FlowMonitor_Files/FlowMonitor_RRDtool"; $dashboard_directory = "/var/www/html/FlowViewer_Dashboard"; $dashboard_short = "/html/FlowViewer_Dashboard"; @other_dashboards = (); # Set to empty if you have just the one nominal Dashboard #@other_dashboards = ("/var/www/html/SOC","/var/www/html/NetOps"); @dashboard_titles = (); # Set to empty if you have just the one nominal Dashboard #@dashboard_titles = ("Performance","SOC","NetOps"); # titles must be in the same order as the directories $flow_data_directory = "/data/flows/"; $exporter_directory = "/data/all_routers"; $flow_bin_directory = "/usr/local/flow-tools/bin"; $rrdtool_bin_directory = "/usr/bin"; # SiLK parameters $silk_data_directory = "/data/flows/"; $silk_bin_directory = "/usr/local/bin"; $site_config_file = "/data/silk.conf"; # If left blank, will look for silk.conf in specified Data Rootdir (see User's Guide) $sensor_config_file = "/data/sensors.conf"; $silk_compiled_localtime = ""; # Set to "Y" if you compiled SiLK with --enable-localtime switch $silk_capture_buffer_pre = (125 * 60); # Start of SiLK file concatenation $silk_capture_buffer_post= (5 * 60); # End of SiLK file concatenation $silk_init_loadscheme = 1; # For Flows Initiated/Second - see SiLK rwcount documentation $silk_active_loadscheme = 5; # For Flows Active/Second - see SiLK rwcount documentation $silk_class_default = ""; # General SiLK file structure info. silk.conf, sensor.conf $silk_flowtype_default = ""; # General SiLK file structure info. silk.conf, sensor.conf $silk_type_default = "all"; # General SiLK file structure info. silk.conf, sensor.conf $silk_sensors_default = ""; # General SiLK file structure info. silk.conf, sensor.conf $silk_switches_default = ""; # General SiLK file structure info. silk.conf, sensor.confAny idea of what's wrong with my installation?
Thanks in advance
PL
Last edit: Paolo Larcheri 2015-10-16
This may be caused by a filter that has no selections made at all. It is a bug.
Please try attached FlowMonitor_Collector script (rename the old one, and then rename the script to FlowMonitor_Collector and restart.
If it still fails please send me a copy of $Flow_Working/DEBUG_MONITOR_C
Thanks.
Joe
P.S. - double-check that you indeed want a filter that has nothing specified. Such a filter will account for all traffic going through the device in all directions at once.
Last edit: Joe Loiacono 2015-10-16
Hi Joe, this happens for every monitor, also those applying valid filters. Looks like RRD files are never updated, neither images created.
Hi have no DEBUG_MONITOR_C file:
I had already tried the modified version of FlowCollector you posted for Tim in a previous post, but it did not solve the problem.
Just let me know the info you need and I will provide you right away.
Thanks for the support.
PL
Last edit: Paolo Larcheri 2015-10-19
Sorry Joe, I read the doc again. I figured out I had totally missed the part concerning the 2 utilities FlowMonitor_Grapher and FlowMonitor_Collector to be launched by hand.
It works like a charm.
Thanks for the time
PL