Issues with multiple sources for netflow information

[Anders]

Hello,

I am trying to set up FlowViewer with SiLK and two Cisco ASA.
I have my /data/sensor.conf set up with two probes and two sensors for the different firewalls:

---

probe CiscoASA netflow-v9
listen-on-port 9901
protocol udp
accept-from-host 1.2.3.4
quirks zero-packets
end probe
probe CiscoASA2 netflow-v9
listen-on-port 9901
protocol udp
accept-from-host 2.3.4.5
quirks zero-packets
end probe
sensor CiscoASA
netflow-v9-probes CiscoASA
internal-ipblock 128.2.0.0/16
external-ipblock remainder
end sensor
sensor CiscoASA2
netflow-v9-probes CiscoASA2
internal-ipblock 128.2.0.0/16
external-ipblock remainder
end sensor

---

(External IPs of the firewalls changed here of course)

In silk.conf I have set:
sensor 0 CiscoASA
sensor 1 CiscoASA2

and:

sensors CiscoASA CiscoASA2

Otherwise no changes.

I have created the directories /data/CiscoASA and /data/CiscoASA2
In these directories I have created the sub-directories:

ext2ext/
in/
in2int/
innull/
out/
outweb/

I also have a copy of silk.conf in each of the two directories.

---

This is how FlowViewer_Configuration.pmis configured:

$flow_data_directory = "/data";
$exporter_directory = "/data/flows/all_routers";
$silk_data_directory = "/data";
$site_config_file = "/data/silk.conf";
$sensor_config_file = "/data/sensor.conf";

Now, in the FlowViewer GUI if I go to FlowViewer or FlowGrapher and select CiscoASA or CiscoASA2
from the Select Device drop-down, and type in /data as the Data Rootdir, I get the same output. It
does not seem to understand or see the difference between CiscoASA and CiscoASA2. If I type /data/CiscoASA
or /data/CiscoASA2 as the Data Rootdir I get nothing. There is however data present in the sub-directories
of both /data/CiscoASA and /data/CiscoASA2.

Can anyone see where the problem exists?

[Joe Loiacono]

Hmm, Looks OK.

Try a FlowViewer run and then look into $work_directory/DEBUG_VIEWER and examine the SiLK commands that are created. Then copy and paste one onto the CLI and see what responses you get.

Joe

[Anders]

I have now managed to get FlowViewer and FlowGrapher to display separate outputs when I choose /data/CiscoASA and /data/CiscoASA2, although I am not entirely sure how I managed it.

The problem I have now is that when I create FlowMonitors for the dashboard all I get are empty graphs no matter what rootdir I choose. Is there some separate configuration that needs to be done in order to get these graphs to work, since FlowGrapher works now just as it should?

[Joe Loiacono]

Well, I'm glad FV and FG are working!

The FlowMonitor issue has come up a couple of times recently and I getting concerned about it. One thing I have suggested to others is to make sure that the script FlowMonitor_Collector is tsarted out of the FlowViewer cgi-bin directory, since it needs access to FlowViewer_Utilities.pm (see flowmonitor_restart in /tools, or cd to the cgi-bin directory, and start the script there (i.e., ./FlowMonitor_Collector)

However, if you are starting it in that directory and still have the problem, please look into DEBUG_MONITOR_C and check the SiLK commands, I suspect you are missing the 'partitioning switches' and we'll need to find out why.

I will try to work closely with you to figure this out.

Thanks,

Joe

[Anders]

I checked out DEBUG_MONITOR_C and the two commands it runs are:

/usr/local/bin/rwfilter --site-config-file=/data/silk.conf --data-rootdir=/data/CiscoASA --type=all --start-date=2015/08/21:05 --end-wviewer/working/FlowMonitor_Prefiltered_CiscoASA_all

and

/usr/local/bin/rwfilter --site-config-file=/data/silk.conf --pass=stdout /usr/local/www/flowviewer/working/FlowMonitor_Prefiltered_Ct-time=2015/08/21:07:30:00 --end-time=2015/08/21:07:35:00 --epoch-slots --no-titles > /usr/local/www/flowviewer/working/FlowMonitor_Collector_output

I tried running them in the shell and the first one gives no output in the shell but the second command gives:

rwfilter: Must specify partitioning rules when using --pass-destination
Use 'rwfilter --help' for usage
rwcount: Error processing headers on file '-': Unexpected end of file while reading header

Also the file: /usr/local/www/flowviewer/working/FlowMonitor_Collector_output is empty, so it never outputs any data to it.
I am not sure which file the program means when it says Error processing headers on file.

Furthermore, and this is I guess a more general bug report, but I am running all this on a FreeBSD machine, and the flowmonitor_restart script and other scripts in that directory are non-functional on FreeBSD due to things like running /bin/su --shell=/bin/sh.
This fails because su is located in /usr/bin/su on FreeBSD and the --shell switch does not exist.
This is more of a FreeBSD port maintainer issue.

EDIT: the graphs on the flow monitors have started showing 0.00 bps instead of nan bps, for what it's worth.

[Joe Loiacono]

Anders,

The best thing would be to edit the flowmonitor_restart script for your environment. The script makes sure FlowMonitor_Collector starts in the proper directory so it can access subroutines in other FlowViewer packages (e.g., FlowViewer_Utilities.pm).

Joe