I have a situation where Silk and FlowViewer seem to be installed correctly, silk is logging data from an ASA for testing and I have the flowFiewer dashboard up, however no data is showing up in reports and the drop-down menus such as "interfaces" are not populating with interface information.
My configuration is Centod 6.5 with Silk 3.8.1. Silk has all it's config files in /etc and data is being logged to /data.
The ASA is outputting v9 Netflow and the directories below /data are
/data/in
/data/inweb
/data/out
/data/outweb
The section of FlowViewer_Configuration.pm I have changed is
The SiLK collection directory structure needs to have devices immediately after the $silk_data_directory. So in your case that would be:
/data/hf01.syd01
You'll need to create that directory and give it enough permissions so that silk processes can write into it. Put a copy of silk.conf in there as well. rwflowpack will create the 'in', 'out', etc. subdirectories.
Kyle Kroskey has offered up diffs for a cleaner way to do this and we'll put that in the next release.
Regarding the Interface dropdowns. You have to first come up with a 'SNMP index to Interface name' mapping, and then put that in the NamedInterfaces_Devices file provided in the distro.
Joe
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Looks like the only difference from the default path-format is the %C, which is the "class name". Did you do this in place of manually creating a 'device' subdirectory under $silk_data_directory?
Can you show the resulting directory structure and your sensor.conf and silk.config files (anonimized) and the location of these?
Thanks,
Joe
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Any update how did you manage it?
In my setup, we have only silk with 3 sensors defined having Netflowv9 only and no IPFIX flows.
After, I changed path-format by adding %N at the start of default string, flowpack process of silk started creating files inside as required.
After that, I was having error "site configuration file not found" in httpd error log. The resolution was to copy silk.conf to all 3 subdirectories and the main silk data directory; but guess what, now when I run same command extracted from flowViewer DeBug log starting with rwfilter with a pipe to rwstats, I don't get any output on screen.
Note:- per flowviewer format, rwfilter doesn't pass sensor name but override data directory of each subdirectory. If I change it with senspr name, it works.
Please update how should i resolve this?
Rgds,
Shazad
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm running into a similar situation. I've followed the above directions and now have two v9 exporters and what I think is a standard tree:
/data/F5520N1/(silkdirs)
/data/R2811N1/(silkdirs)
they're in the FlowViewer_Configuration.pm file:
@ipfix_devices = ("F5520N1","R2811N1"); # for SiLK, if none: @ipfix_devices = ();
@ipfix_storage = ("F5520N1:15G","R2811N1:500M");
Now, the data-rootdir is /data (as referenced in the config files). If I take the output from the debug file and change "--data-rootdir" to "--sensor=R2811N1" I get output, so I know the data is there. I'm not sure what I've got messed up in my config that I can't get Flowviewer to manipulate it....
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Shazhad,Iamactivelyworkingthisaroundmydayjob:-).Hopetogetitsoon,andwillputoutanewversiontogetherwithabetterexplanationinthedocumentation.KyleKroskeyhasdoneanicejobcleaningthisup...I've included hischanges below. Note: he made the change for FlowTracker_Collector; I'vemodifiedittomatchFlowGrapher_Main.cgiwhichyoucanusetotestmorequickly.Ifyoudon't have graphics set up yet, make the change toFlowViewer_Main.cgi.Note also that he has named his probe, his sensor, and his device all thesame thing: S0. Thank you Kyle.Kyle'schanges:Bothsensor.confandsilk.confarein/netflow.The--data_rootdirwastherefromthegetgo.WhatIaddedwasthe--sensortotherwfiltercommand.Justbreaksitapartfromthe$data_root_dirvariableaboveit.root@maverick:/usr/lib/cgi-bin/FlowViewer#diff/usr/src/FlowViewer_4.3/FlowGrapher_Main.cgi/usr/lib/cgi-bin/FlowViewer/FlowGrapher_Main.cgi548c548<$data_root_dir=$silk_data_directory."/".$device_name;--->$data_root_dir=$silk_data_directory;580c580<$rwfilter_command="$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end $window_type=$time_window --pass=$prefiltered_file";--->$rwfilter_command="$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --sensors=$device_name --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end $window_type=$time_window --pass=$prefiltered_file";----root@maverick:/netflow#catsensor.confprobeS0ipfixlisten-on-port18001protocoltcpendprobesensorS0ipfix-probesS0internal-ipblocks192.168.1.0/2410.0.0.0/8224.0.0.0/8external-ipblocksremainderendsensor----root@maverick:/netflow#catsilk.conf# silk.conf for the "twoway" site# RCSIDENT("$SiLK: silk.conf 52d8f4f62ffd 2012-05-25 21:16:30Z mthomas $")# For a description of the syntax of this file, see silk.conf(5).# The syntactic format of this file# version 2 supports sensor descriptions, but otherwise identical to 1version2# NOTE: Once data has been collected for a sensor or a flowtype, the# sensor or flowtype should never be removed or renumbered. SiLK Flow# files store the sensor ID and flowtype ID as integers; removing or# renumbering a sensor or flowtype breaks this mapping.sensor0S0"BRAKJ_v9 Sensor"classallsensorsS0endclass# Editing above this line is sufficient for sensor definition.# Be sure you understand the workings of the packing system before# editing the class and type definitions below. In particular, if you# change or add-to the following, the C code in packlogic-twoway.c# will need to change as well.classalltype0inintype1outouttype2inwebiwtype3outwebowtype4innullinnulltype5outnulloutnulltype6int2intint2inttype7ext2extext2exttype8inicmpinicmptype9outicmpouticmptype10otherotherdefault-typesininwebinicmpendclassdefault-classall# The layout of the tree below SILK_DATA_ROOTDIR.# Use the default, which assumes a single class.path-format"%N/%T/%Y/%m/%d/%x"# The plug-in to load to get the packing logic to use in rwflowpack.# The --packing-logic switch to rwflowpack will override this value.# If SiLK was configured with hard-coded packing logic, this value is# ignored. packing-logic"/usr/local/lib/silk/packlogic-twoway.so"____root@maverick:/netflow#ls-ltotal20-rwxrwxrwx1rootroot6976Mar1619:23rwflowpack.confdrwxrwxrwx8rootroot4096Mar2019:27S0-rwxrwxrwx1rootroot196Mar2220:14sensor.conf-rwxrwxrwx1rootroot1703Mar1920:05silk.conf
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
With respect to User Authentication, FlowViewer itself does not provide it. However, you can put basic userid/password by using the normal 'htpasswd' capability.
Also - Peter had some questions about @exporters ... that should be left blank: @exporters = ();
Regards,
Joe
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for your help. This is what I was suspecting that we have to append --sensor in the code itself.
I changed both lines in FlowGrapher_Main.cgi, and it started showing data correctly.
When I checked FlowViewer_Main.cgi, there $rwfilter_command is in lot of places into the file.
Thinking your referring to FlowTracker_Collector file, I changed data_root_dir and rwfilter_command accordingly in it but still having problem.
The problem is that FlowGrapher is populating data from 3 different sensors (networks) properly but FlowViewer is showing data based on data_root_dir, meaning from all sensors at the same time with no filtering to specific sensor based on my appending "--sensor=$device_name".
Please update in which places should I append "--sensor=device_name" to fix the issue? OR do I append it everywhere in all 3 files i.e. FlowViewer_Main.cgi, FlowGrapher_Main.cgi and FlowTracker_Main.cgi?
Regards,
Shahzad
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I changed your suggested configurations in FlowViwer_main.cgi and FlowGrapher_main.cgi i.e. Excluding $device_name from $data_root_dir and adding --sensor=$device_name into $rwfilter_command.
Both are working fine as required, but FlowTracker_main.cgi doesn't have variables with such names in it to be changed ; and it is main part in flowviewer dashboard :)
Isn't there any single file where from we can control these all three things?
Rgds,
shahzad
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've applied these diffs and can now get data from the device. However, for some strange reason the grapher is inconsistent. Sometimes it draws graphs, sometimes it doesn't. Either way I get the raw source data listed in a table blow the (sometimes blank) graph so I know it's picking data to graph. I don't see anything useful in DEBUG_GRAPHER (it just says "done_FLOWS to create_graph to done" in about 3 seconds).
This only applies to my silk source, I have a V5 exporter that graphs fine down to the sub-minute level. The V9 exporter will graph if I select, say, a whole day but if I pick a one-hour slice I get the table data but no graph. I can't seem to pin down what does and doesn't make a graph come out. Any way I can help narrow it down?
Thanks,
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
So, my latest attempt after adding your debug for dates 6/3 14:00 - 15:00, generated 719 buckets, all valued at 0. The text representation of the data did populate under the blank graph. I also tried 01:00-05:00 and got a graph, 707 of 3587 buckets were populated.
Thanks,
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It looks like the firewall is logging in GMT even though the clock is set to EDT. The stamps on the data in the file is +4 from the file date/time (GMT vs EDT). Setting the FlowViewer_Configuration.pm timezone to GMT seems to have fixed it for the ASA, but breaks things for the V5 collector.......
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I am sure we are aware that SiLK uses default TZ UTC for both storing files and taking input/ giving output to users. We can control the second part i.e. input/output by some compilation parameters but still it will store the files per UTC timestamp.
Dear Joe,
Thanks a ton for your helping out in our case.
I am in GST TZ and I haven't changed TZ in FlowViewer_Configuration.pm and for me FlowGrahper is working fine and FlowViewer shows some inconsistency in "Detect Scanning" Report.
Using FLowViewer, when we extract "Detect Scanning" Report for last one hour i.e. 1300-1400 ZBL with no filters and include if: Entirely in Specified Time Span,
the start and end time show are in UTC i.e. 0900-1000. In SiLK, it is know that it keeps time in UTC but during compilation, it allows to change TimeZone to your region
only for Report inputs from users and Report outputs to the users, though it stores the files under same UTC format.
So, as per my configuration, when I run the rwfilter commands in my setup from CLI, it gives me RIGHT time but not on FlowViwer GUI.
If I check DEBUG_VIEWER, command is as follows:
rwfilter_command: /usr/local/bin/rwfilter --data-rootdir=/silk_data_repository --sensors=Zabeel-Secure-Colo --type=all --start-date=2014/06/04:05 --end-date=2014/06/04:07 --stime=2014/06/04:6:00:00-2014/06/04:7:00:00 --etime=2014/06/04:6:00:00-2014/06/04:7:00:00 --pass=/var/www/cgi-bin/FlowViewer_4.3/Flow_Working/FlowViewer_scanner_081416
Running same command as:
rwfilter_command: /usr/local/bin/rwfilter --data-rootdir=/silk_data_repository --sensors=Zabeel-Secure-Colo --type=all --start-date=2014/06/04:05 --end-date=2014/06/04:07 --stime=2014/06/04:6:00:00-2014/06/04:7:00:00 --etime=2014/06/04:6:00:00-2014/06/04:7:00:00 --pass=stdout | rwcut
shows me RIGHT timings
Secondly Joe, I changed your recommended configs into FlowTracker_Collector file at 2 places but still I can't see any tracker graphs on Flow Trackers.
here is the Debug (with no commands in it)
tail: DEBUG_TRACKER: file truncated
In FlowTracker_Display.cgi
FlowTracker_Display: filter_hash: TR_c_s filter_filename: c_s.fil
tail: DEBUG_TRACKER: file truncated
In FlowTracker_Display.cgi
FlowTracker_Display: filter_hash: TR_c_s filter_filename: c_s.fil
tail: DEBUG_TRACKER: file truncated
In FlowTracker_Display.cgi
FlowTracker_Display: filter_hash: TR_z_s_c filter_filename: z_s_c.fil
In FlowTracker_main.cgi action: Revise Tracking
FORM{start_date}: 01/01/2000 start_date: 01/01/2000 FORM{end_date}: 01/01/2000 end_date: 01/01/2000
This device is exporting IPFIX
Please help in both cases.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Comments in-line:
> From: "Shahzad" <shahzadenwar@users.sf.net>
> To: "[flowviewer:discussion] " <general@discussion.flowviewer.p.re.sf.net>
> Date: 06/04/2014 01:14 AM
> Subject: [flowviewer:discussion] FlowViewer with Silk
>
> Dear Peter / Joe,
> I am sure we are aware that SiLK uses default TZ UTC for both
> storing files and taking input/ giving output to users. We can
> control the second part i.e. input/output by some compilation
> parameters but still it will store the files per UTC timestamp.
Good to know.
> I am in GST TZ and I haven't changed TZ in
> FlowViewer_Configuration.pm and for me FlowGrahper is working fine
So the flow start and end times are correct? They are listed in GST time?
> and FlowViewer shows some inconsistency in "Detect Scanning" Report.
> Using FLowViewer, when we extract "Detect Scanning" Report for last
> one hour i.e. 1300-1400 ZBL with no filters and include if: Entirely
> in Specified Time Span,
> the start and end time show are in UTC i.e. 0900-1000. In SiLK, it
> is know that it keeps time in UTC but during compilation, it allows
> to change TimeZone to your region
> only for Report inputs from users and Report outputs to the users,
> though it stores the files under same UTC format.
> So, as per my configuration, when I run the rwfilter commands in my
> setup from CLI, it gives me RIGHT time but not on FlowViwer GUI.
> If I check DEBUG_VIEWER, command is as follows:
> rwfilter_command: /usr/local/bin/rwfilter --data-rootdir=/
> silk_data_repository --sensors=Zabeel-Secure-Colo --type=all --
> start-date=2014/06/04:05 --end-date=2014/06/04:07 --stime=2014/06/
> 04:6:00:00-2014/06/04:7:00:00 --etime=2014/06/04:6:00:00-2014/06/04:
> 7:00:00 --pass=/var/www/cgi-bin/FlowViewer_4.3/Flow_Working/
> FlowViewer_scanner_081416
> Running same command as:
> rwfilter_command: /usr/local/bin/rwfilter --data-rootdir=/
> silk_data_repository --sensors=Zabeel-Secure-Colo --type=all --
> start-date=2014/06/04:05 --end-date=2014/06/04:07 --stime=2014/06/
> 04:6:00:00-2014/06/04:7:00:00 --etime=2014/06/04:6:00:00-2014/06/04:
> 7:00:00 --pass=stdout | rwcut
> shows me RIGHT timings
OK, so that says the rwfilter_command is OK with respect to time zone.
You can confirm this by looking at the intermediate file:
host> rwcut < /var/www/cgi-bin/FlowViewer_4.3/Flow_Working/FlowViewer_scanner_143833
After this intermediate step, the next steps involve rwsort, and
then rwscan. You can see these in the DEBUG_VIEWER file.
Can you run each of those and see if and where the times are
getting changed?
Also - try the Printed Reports: 132 Columns
Also - can we see what happens to FV Scanner, and FG, when you set
the timezone variable in the config file?
> Secondly Joe, I changed your recommended configs into
> FlowTracker_Collector file at 2 places but still I can't see any
> tracker graphs on Flow Trackers.
See comment on running flowtracker_restart, plus for debug look into
DEBUG_TRACKER_C for FlowTracker_Collector, and DEBUG_TRACKER_G for FlowTracker_Grapher.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Additionally, I am not sure how can we restart the FlowTracker.
[root@euc010007013102 ~]# /var/www/cgi-bin/FlowViewer_4.3/tools/flowtracker_restart
Usage: /var/www/cgi-bin/FlowViewer_4.3/tools/flowtracker_restart {start|stop|restart} [root@euc010007013102 ~]# [root@euc010007013102 ~]# /var/www/cgi-bin/FlowViewer_4.3/tools/flowtracker_restart restart
Stopping FlowTracker processes: FlowTracker_Collector: no process killed
Problem stopping FlowTracker_Collector
FlowTracker_Grapher: no process killed
Problem stopping FlowTracker_Grapher
Done.
Starting FlowTracker processes: Starting FlowTracker_Collector:
/bin/su: user flowviewer does not exist
Problem starting FlowTracker_Collector
Starting FlowTracker_Grapher:
/bin/su: user flowviewer does not exist
Problem starting FlowTracker_Grapher
Done. [root@euc010007013102 ~]#
Rgds,
Shahzad
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Line 15 in the start-up script sets the userid that you want to use to run FlowTracker_Collector and FlowTracker_Grapher. You can change it however you like.
Then start the script with 'sudo' which will accomplish the 'su' to that user. Note: root privileges are not required. You can simply start it from a command line:
I have a situation where Silk and FlowViewer seem to be installed correctly, silk is logging data from an ASA for testing and I have the flowFiewer dashboard up, however no data is showing up in reports and the drop-down menus such as "interfaces" are not populating with interface information.
My configuration is Centod 6.5 with Silk 3.8.1. Silk has all it's config files in /etc and data is being logged to /data.
The ASA is outputting v9 Netflow and the directories below /data are
/data/in
/data/inweb
/data/out
/data/outweb
The section of FlowViewer_Configuration.pm I have changed is
$flow_data_directory = "/data/flows";
$exporter_directory = "/data/flows/all_routers";
$flow_bin_directory = "/usr/local/flow-tools/bin";
$rrdtool_bin_directory = "/usr/bin";
$silk_data_directory = "/data";
$silk_bin_directory = "/usr/local/bin";
$sensor_config_directory = "/etc";
General parameters
$version = "4.3";
$no_devices_or_exporters = "N";
@devices = (); # for flow-tools
@ipfix_devices = ("hf01.syd01"); # for SiLK, if none: @ipfix_devices = ();
@ipfix_storage = ("hf01.syd01:15G");
@exporters = ();
$flow_capture_interval = (35 * 60);
$flow_file_length = (15 * 60);
$start_offset = (90 * 60); # e.g., 90 minutes ago
$end_offset = (30 * 60); # e.g., 30 minutes ago
$use_even_hours = "Y";
$N = 3;
$use_NDBM = "N";
$pie_chart_default = 1; # 0 = None; 1 = With Others; 2 = Without Others
$number_slices = 7;
$maximum_days = "91";
$remove_workfiles_time = 286400;
$remove_graphfiles_time = 786400;
$remove_reportfiles_time = 7*86400;
$time_zone = ""; # If left empty, will use system time zone
$time_zone_dst_offset = (60 * 60); # Number of seconds of the Daylight Savings adjustment in your timezone
$date_format = "YMD"; # MDY=MM/DD/YYYY DMY=DD/MM/YYYY DMY2=DD.MM.YYYY YMD=YYYY-MM-DD
$labels_in_titles = "1"; # Set to "1" for labels in Tracker graph titles; "0" off
$sip_prefix_length = "16";
$dip_prefix_length = "16";
$silk_all_only = "N";
$silk_flows_loadscheme = 5;
Any help would be greatly appreciated.
Last edit: David Taylor 2014-04-16
Hi David,
The SiLK collection directory structure needs to have devices immediately after the $silk_data_directory. So in your case that would be:
/data/hf01.syd01
You'll need to create that directory and give it enough permissions so that silk processes can write into it. Put a copy of silk.conf in there as well. rwflowpack will create the 'in', 'out', etc. subdirectories.
Kyle Kroskey has offered up diffs for a cleaner way to do this and we'll put that in the next release.
Regarding the Interface dropdowns. You have to first come up with a 'SNMP index to Interface name' mapping, and then put that in the NamedInterfaces_Devices file provided in the distro.
Joe
Hi Joe,
Thanks again for you fast reply.
In terms of the silk path structure, in the silk.conf file I needed to update the "path-format" option as follows
path-format "%N/%T/%Y/%m/%d/%x"
I'm still working on the interface mapping.
D
Last edit: David Taylor 2014-04-17
Looks like the only difference from the default path-format is the %C, which is the "class name". Did you do this in place of manually creating a 'device' subdirectory under $silk_data_directory?
Can you show the resulting directory structure and your sensor.conf and silk.config files (anonimized) and the location of these?
Thanks,
Joe
Hi David,
Any update how did you manage it?
In my setup, we have only silk with 3 sensors defined having Netflowv9 only and no IPFIX flows.
After, I changed path-format by adding %N at the start of default string, flowpack process of silk started creating files inside as required.
After that, I was having error "site configuration file not found" in httpd error log. The resolution was to copy silk.conf to all 3 subdirectories and the main silk data directory; but guess what, now when I run same command extracted from flowViewer DeBug log starting with rwfilter with a pipe to rwstats, I don't get any output on screen.
Note:- per flowviewer format, rwfilter doesn't pass sensor name but override data directory of each subdirectory. If I change it with senspr name, it works.
Please update how should i resolve this?
Rgds,
Shazad
I'm running into a similar situation. I've followed the above directions and now have two v9 exporters and what I think is a standard tree:
/data/F5520N1/(silkdirs)
/data/R2811N1/(silkdirs)
they're in the FlowViewer_Configuration.pm file:
@ipfix_devices = ("F5520N1","R2811N1"); # for SiLK, if none: @ipfix_devices = ();
@ipfix_storage = ("F5520N1:15G","R2811N1:500M");
@devices is null
they're also in exporters:
@exporters = ("192.168.168.1:Border Firewall","10.20.30.1:IPV6 Tunnel Router");
but I don't think that should/needs to be there.
I can get data from the silk tools by hand.
However, I think I have something misconfigured because the debug files aren't referencing the right directories:
silk_command: /usr/local/bin/rwfilter --data-rootdir=/data/R2811N1 --type=all (etc)
Now, the data-rootdir is /data (as referenced in the config files). If I take the output from the debug file and change "--data-rootdir" to "--sensor=R2811N1" I get output, so I know the data is there. I'm not sure what I've got messed up in my config that I can't get Flowviewer to manipulate it....
Shazad and Peter,
I'm anxious to get this cleaned up. Kyle Kroskey has a better way to do this and I'd like to make his changes. I'll work on this now.
Cool, I'll wait until we hear back.
Thanks!
Hi Peter,
I have same problem as you do have, in command line rwfilter "-sensor" works but with "data-rootdir" it doesnt work.
Hello Joe,
Thank you for the update. How much time will it take?
Secondly, how can we restrict flowviewer to user based authentication?
Rgds,
Shahzad
Shahzad,
With respect to User Authentication, FlowViewer itself does not provide it. However, you can put basic userid/password by using the normal 'htpasswd' capability.
Also - Peter had some questions about @exporters ... that should be left blank: @exporters = ();
Regards,
Joe
Dear Joe,
Thanks for your help. This is what I was suspecting that we have to append --sensor in the code itself.
I changed both lines in FlowGrapher_Main.cgi, and it started showing data correctly.
When I checked FlowViewer_Main.cgi, there $rwfilter_command is in lot of places into the file.
Thinking your referring to FlowTracker_Collector file, I changed data_root_dir and rwfilter_command accordingly in it but still having problem.
The problem is that FlowGrapher is populating data from 3 different sensors (networks) properly but FlowViewer is showing data based on data_root_dir, meaning from all sensors at the same time with no filtering to specific sensor based on my appending "--sensor=$device_name".
Please update in which places should I append "--sensor=device_name" to fix the issue? OR do I append it everywhere in all 3 files i.e. FlowViewer_Main.cgi, FlowGrapher_Main.cgi and FlowTracker_Main.cgi?
Regards,
Shahzad
Dear Joe,
In FlowViewer_Main.cgi, I appended --sensor in 4 lines:
# Prepare rwfilter command including flow window parameters
Joe,
I changed your suggested configurations in FlowViwer_main.cgi and FlowGrapher_main.cgi i.e. Excluding $device_name from $data_root_dir and adding --sensor=$device_name into $rwfilter_command.
Both are working fine as required, but FlowTracker_main.cgi doesn't have variables with such names in it to be changed ; and it is main part in flowviewer dashboard :)
Isn't there any single file where from we can control these all three things?
Rgds,
shahzad
Hi,
I've applied these diffs and can now get data from the device. However, for some strange reason the grapher is inconsistent. Sometimes it draws graphs, sometimes it doesn't. Either way I get the raw source data listed in a table blow the (sometimes blank) graph so I know it's picking data to graph. I don't see anything useful in DEBUG_GRAPHER (it just says "done_FLOWS to create_graph to done" in about 3 seconds).
This only applies to my silk source, I have a V5 exporter that graphs fine down to the sub-minute level. The V9 exporter will graph if I select, say, a whole day but if I pick a one-hour slice I get the table data but no graph. I can't seem to pin down what does and doesn't make a graph come out. Any way I can help narrow it down?
Thanks,
First let's see if you're getting data to plot when it is not plotting.
Can you add the following debug line to FlowGrapher_Main.cgi (looks like it would be after line 1042):
1042 $buckets[$i] = $buckets[$i] / $bucket_size;
1043 print DEBUG "buckets[$i]: $buckets[$i]\n";
We'll see if it is a data problem, or a graphing problem.
Hi,
So, my latest attempt after adding your debug for dates 6/3 14:00 - 15:00, generated 719 buckets, all valued at 0. The text representation of the data did populate under the blank graph. I also tried 01:00-05:00 and got a graph, 707 of 3587 buckets were populated.
Thanks,
Hi Peter,
Could you check your router time? Sometimes if it is off, the files (based on host system time) don't sync up with the netflow that is exported.
You can look at the timestamps of the flows by (e.g.,):
rwcut out-S1_20140603.20 > ~/temp.out
They should be close to the file time.
It looks like the firewall is logging in GMT even though the clock is set to EDT. The stamps on the data in the file is +4 from the file date/time (GMT vs EDT). Setting the FlowViewer_Configuration.pm timezone to GMT seems to have fixed it for the ASA, but breaks things for the V5 collector.......
Might have to do the same time check with the device you are collecting v5 from.
I agree with your comment below. In the end, it has turned out for me to be easier overall just to use GMT everywhere.
Dear Peter / Joe,
I am sure we are aware that SiLK uses default TZ UTC for both storing files and taking input/ giving output to users. We can control the second part i.e. input/output by some compilation parameters but still it will store the files per UTC timestamp.
Dear Joe,
Thanks a ton for your helping out in our case.
I am in GST TZ and I haven't changed TZ in FlowViewer_Configuration.pm and for me FlowGrahper is working fine and FlowViewer shows some inconsistency in "Detect Scanning" Report.
Using FLowViewer, when we extract "Detect Scanning" Report for last one hour i.e. 1300-1400 ZBL with no filters and include if: Entirely in Specified Time Span,
the start and end time show are in UTC i.e. 0900-1000. In SiLK, it is know that it keeps time in UTC but during compilation, it allows to change TimeZone to your region
only for Report inputs from users and Report outputs to the users, though it stores the files under same UTC format.
So, as per my configuration, when I run the rwfilter commands in my setup from CLI, it gives me RIGHT time but not on FlowViwer GUI.
If I check DEBUG_VIEWER, command is as follows:
rwfilter_command: /usr/local/bin/rwfilter --data-rootdir=/silk_data_repository --sensors=Zabeel-Secure-Colo --type=all --start-date=2014/06/04:05 --end-date=2014/06/04:07 --stime=2014/06/04:6:00:00-2014/06/04:7:00:00 --etime=2014/06/04:6:00:00-2014/06/04:7:00:00 --pass=/var/www/cgi-bin/FlowViewer_4.3/Flow_Working/FlowViewer_scanner_081416
Running same command as:
rwfilter_command: /usr/local/bin/rwfilter --data-rootdir=/silk_data_repository --sensors=Zabeel-Secure-Colo --type=all --start-date=2014/06/04:05 --end-date=2014/06/04:07 --stime=2014/06/04:6:00:00-2014/06/04:7:00:00 --etime=2014/06/04:6:00:00-2014/06/04:7:00:00 --pass=stdout | rwcut
shows me RIGHT timings
Secondly Joe, I changed your recommended configs into FlowTracker_Collector file at 2 places but still I can't see any tracker graphs on Flow Trackers.
here is the Debug (with no commands in it)
tail: DEBUG_TRACKER: file truncated
In FlowTracker_Display.cgi
FlowTracker_Display: filter_hash: TR_c_s filter_filename: c_s.fil
tail: DEBUG_TRACKER: file truncated
In FlowTracker_Display.cgi
FlowTracker_Display: filter_hash: TR_c_s filter_filename: c_s.fil
tail: DEBUG_TRACKER: file truncated
In FlowTracker_Display.cgi
FlowTracker_Display: filter_hash: TR_z_s_c filter_filename: z_s_c.fil
In FlowTracker_main.cgi action: Revise Tracking
FORM{start_date}: 01/01/2000 start_date: 01/01/2000 FORM{end_date}: 01/01/2000 end_date: 01/01/2000
This device is exporting IPFIX
Please help in both cases.
Dear Joe,
Additionally, I am not sure how can we restart the FlowTracker.
[root@euc010007013102 ~]# /var/www/cgi-bin/FlowViewer_4.3/tools/flowtracker_restart
Usage: /var/www/cgi-bin/FlowViewer_4.3/tools/flowtracker_restart {start|stop|restart}
[root@euc010007013102 ~]#
[root@euc010007013102 ~]# /var/www/cgi-bin/FlowViewer_4.3/tools/flowtracker_restart restart
Stopping FlowTracker processes: FlowTracker_Collector: no process killed
Problem stopping FlowTracker_Collector
FlowTracker_Grapher: no process killed
Problem stopping FlowTracker_Grapher
Done.
Starting FlowTracker processes: Starting FlowTracker_Collector:
/bin/su: user flowviewer does not exist
Problem starting FlowTracker_Collector
Starting FlowTracker_Grapher:
/bin/su: user flowviewer does not exist
Problem starting FlowTracker_Grapher
Done.
[root@euc010007013102 ~]#
Rgds,
Shahzad
Shahzad,
Line 15 in the start-up script sets the userid that you want to use to run FlowTracker_Collector and FlowTracker_Grapher. You can change it however you like.
Then start the script with 'sudo' which will accomplish the 'su' to that user. Note: root privileges are not required. You can simply start it from a command line:
host>FlowTracker_Collector &
host>FlowTracker_Grapher &