Hello,
I’ve set up FlowViewer version 4.6 with SiLK version 3.10.1 and have it working very well with a single Cisco router as a sensor. I’ve now added an ASA firewall as a second sensor and SiLK is collecting flow data but I can’t get FlowViewer to report on it. Both devices appear as Netflow sources in FlowViewer but I get the same report regardless of which on I select. I’m sure I missed a configuration step but it seems like I’m reporting on the entire “Site” when I only want to report on a single sensor. Can you provide any advice as to what I can look at?
Thank you
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Can you show what your SiLK directory structure looks like? Can you show your sensor.conf and silk.conf files as well (sanitized if you like, of course). Also, what does your @ipfix_devices array look like in FlowViewer_Configuration.pm?
Thanks,
Joe
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Joe,
Thanks for getting back to me. The root of my data directory is at /apps1/data/flows and the layout uses a path-format of path-format "%N/%T/%Y/%m/%d/%x". There I've created a directory per sensor and the subdirectories were created automatically following the date format when flow records arrived.
silk.conf and sensor.conf are both in the data directory and I also have a copy of silk.conf in each individual sensor directory. These filese are in the attachements
Both sensors, RCO3366-F5550-PAR-1A and RCO3366-R1004-WAN-1A are available in the drop down menus in flow viewer but I get the same data regardless of which one I choose. I'm at a loss here as to what I've missed.
1) You have both sensors receiving on the same port, 9905. I'm not sure how your rwflowpack's pull the data apart, but that could be related to it. Could you post those commands. I believe SiLK can do it - it's just that I have never done it on a single port and can't comment very well at this point.
2) Make a run with the different devices, and each time look into DEBUG_VIEWER (or _GRAPHER, etc.) and see what SiLK rwfilter command is getting created.
Thanks,
Joe
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
1) Yes I'm using the same port as ultimately I want to have a single configuration to apply to all devices which would include the standard port number. I read that this can work but can change if it's an issue
2) The rwfilter command is the same for both devices as it refers to the same data directory with the only difference being the FlowViewer_ouput file name. You can see these below
Both of the runs above gave the same output which seems to be the correct data for the router (RCO366-R1004-WAN-1A) but I never see data from the firewall.
I tried updating the Data Rootdir when running the queries to point directly at the individual sensor directories but this was unsuccessful as the report comes back blank. I'll change the port the firewall is sending to and will see if that helps.
Thanks
Doug
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I updated the port on the firewall, changed sensor.conf to reflect this, and restarted rwflowpack but I'm still having the same issue. I'll keep looking through it to see what else I can find.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
(Hmmm. Looks like I can't post to SourceForge from one of my VPNs)
Doug - can you replace you current FlowViewer_UI.pm script with the attached (renaming the attached to FlowViewer_UI.pm)? Then, make another run and post the DEBUG?
I've isolated the issue area. But before I put a mod in, can you try two tests where you set of a FlowViewer run, and then put each device name into the 'Sensor' field at the bottom for each run?
I want to see if that works in situations like yours, because I'll need to be careful for other users that may be handling the situation that way.
Thanks,
Joe
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Looks like that's the answer Joe. Adding the sensor name at the bottom gave me exactly what I'm looking for. I'm able to run reports per device. Do you want to see the debugs from them?
Thanks again
Doug
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Putting the sensors in the path-format as you have looks like maybe the most proper way to do it, actually.
I'd like to make it easier for you to do these reports without having to type that information in each time, although it will be preserved across multiple reports and between tool applications.
So, I may add a flag to control it.
I'll send over an update FlowViewer_UI which should allow it to work based on the Device pulldown, instead of copy/pasting into the Sensor field.
Joe
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
That's great! Thanks for putting all the effort into it. This is a great tool so far and will greatly help our team. If you're anywhere around the SF Bay Area I'd love to buy you a dinner sometime....
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello,
I’ve set up FlowViewer version 4.6 with SiLK version 3.10.1 and have it working very well with a single Cisco router as a sensor. I’ve now added an ASA firewall as a second sensor and SiLK is collecting flow data but I can’t get FlowViewer to report on it. Both devices appear as Netflow sources in FlowViewer but I get the same report regardless of which on I select. I’m sure I missed a configuration step but it seems like I’m reporting on the entire “Site” when I only want to report on a single sensor. Can you provide any advice as to what I can look at?
Thank you
Can you show what your SiLK directory structure looks like? Can you show your sensor.conf and silk.conf files as well (sanitized if you like, of course). Also, what does your @ipfix_devices array look like in FlowViewer_Configuration.pm?
Thanks,
Joe
Hi Joe,
Thanks for getting back to me. The root of my data directory is at /apps1/data/flows and the layout uses a path-format of path-format "%N/%T/%Y/%m/%d/%x". There I've created a directory per sensor and the subdirectories were created automatically following the date format when flow records arrived.
[root@RCOVLNX3026 flows]# ls -l /apps1/data/flows/
total 24
drwxrwx---. 2 nettools apache 4096 Apr 9 16:48 archive
drwxrwx---. 2 nettools apache 4096 Apr 22 00:00 log
drwxr-xr-x. 8 nettools apache 4096 Apr 21 09:13 RCO3366-F5550-PAR-1A
drwxr-xr-x. 8 nettools apache 4096 Apr 20 13:10 RCO3366-R1004-WAN-1A
-rwxr-xr-x. 1 nettools apache 612 Apr 20 16:17 sensor.conf
-rwxr-xr-x. 1 nettools apache 1806 Apr 10 08:47 silk.conf
[root@RCOVLNX3026 flows]# ls -l /apps1/data/flows/RCO3366-F5550-PAR-1A/
total 28
drwxr-xr-x. 3 nettools apache 4096 Apr 20 11:00 ext2ext
drwxr-xr-x. 3 nettools apache 4096 Apr 20 11:00 in
drwxr-xr-x. 3 nettools apache 4096 Apr 20 11:01 int2int
drwxr-xr-x. 3 nettools apache 4096 Apr 20 11:00 inweb
drwxr-xr-x. 3 nettools apache 4096 Apr 20 11:00 out
drwxr-xr-x. 3 nettools apache 4096 Apr 20 11:00 outweb
-rwx------. 1 nettools apache 1806 Apr 20 13:10 silk.conf
silk.conf and sensor.conf are both in the data directory and I also have a copy of silk.conf in each individual sensor directory. These filese are in the attachements
FlowViewer_Configuration.pm has the entry below:
@ipfix_devices = ("RCO3366-F5550-PAR-1A","RCO3366-R1004-WAN-1A","Site");
Both sensors, RCO3366-F5550-PAR-1A and RCO3366-R1004-WAN-1A are available in the drop down menus in flow viewer but I get the same data regardless of which one I choose. I'm at a loss here as to what I've missed.
Thanks for any help you can provide here
Doug
Hi Doug,
Two quick things:
1) You have both sensors receiving on the same port, 9905. I'm not sure how your rwflowpack's pull the data apart, but that could be related to it. Could you post those commands. I believe SiLK can do it - it's just that I have never done it on a single port and can't comment very well at this point.
2) Make a run with the different devices, and each time look into DEBUG_VIEWER (or _GRAPHER, etc.) and see what SiLK rwfilter command is getting created.
Thanks,
Joe
1) Yes I'm using the same port as ultimately I want to have a single configuration to apply to all devices which would include the standard port number. I read that this can work but can change if it's an issue
2) The rwfilter command is the same for both devices as it refers to the same data directory with the only difference being the FlowViewer_ouput file name. You can see these below
silk_command: /usr/local/bin/rwfilter --site-config-file=/apps1/data/flows/silk.conf --data-rootdir=/apps1/data/flows --type=all --start-date=2015/04/22:13 --end-date=2015/04/22:17 --active=2015/04/22:16:00:00-2015/04/22:17:00:00 --pass=stdout | /usr/local/bin/rwstats --site-config-file=/apps1/data/flows/silk.conf --fields=1,2 --values=Bytes,Packets,Records --no-titles --no-percents --delimited=" " --count=100 > /apps1/data/Flow_Working/FlowViewer_output_110045
silk_command: /usr/local/bin/rwfilter --site-config-file=/apps1/data/flows/silk.conf --data-rootdir=/apps1/data/flows --type=all --start-date=2015/04/22:13 --end-date=2015/04/22:17 --active=2015/04/22:16:00:00-2015/04/22:17:00:00 --pass=stdout | /usr/local/bin/rwstats --site-config-file=/apps1/data/flows/silk.conf --fields=1,2 --values=Bytes,Packets,Records --no-titles --no-percents --delimited=" " --count=100 > /apps1/data/Flow_Working/FlowViewer_output_111137
Both of the runs above gave the same output which seems to be the correct data for the router (RCO366-R1004-WAN-1A) but I never see data from the firewall.
I tried updating the Data Rootdir when running the queries to point directly at the individual sensor directories but this was unsuccessful as the report comes back blank. I'll change the port the firewall is sending to and will see if that helps.
Thanks
Doug
I updated the port on the firewall, changed sensor.conf to reflect this, and restarted rwflowpack but I'm still having the same issue. I'll keep looking through it to see what else I can find.
Sorry Doug, got caught up in something. Can you post your FlowViewer_Configuration file (sanitized as you like)?
oops .. and a full DEBUG_VIEWER file?
No need to apologize Joe. I appreciate you taking the time to help. The files are attached
Looks like I missed the debug
(Hmmm. Looks like I can't post to SourceForge from one of my VPNs)
Doug - can you replace you current FlowViewer_UI.pm script with the attached (renaming the attached to FlowViewer_UI.pm)? Then, make another run and post the DEBUG?
Thanks,
Joe
Here's the output. Thanks again for working on this
Hi Doug,
Can you try this file? Thanks.
Also can you post the DEBUG file, and the FlowViewer_Save_xxxxxx file?
Thanks,
Joe
Sorry for the delay Joe, I just got back into work for the day
Thanks
Doug
No problem Doug, actually I appreciate your quick responses for anything "non-overnight".
Can you please use this FlowViewer_Main? It was over-writing earlier debug.
Thanks,
Joe
Hopefully this one gives you more info..
Doug,
Can you:
1) Click on FlowViewer button
2) Click on "Reset Form Values"
3) Select one of your devices
And then let me know hat is in the field: Data Rootdir:
Thanks,
Joe
Both devices populate the Data Rootdir with /apps1/data/flows
Doug,
Can you try using this new FlowViewer_UI, then:
1) Click on FlowViewer button
2) Click on "Reset Form Values"
3) Select one of your devices
The post the DEBUG_VIEWER?
Thanks,
Joe
Ooops forgot to post the attachment ....
Here's the latest Debug Joe. Thanks again for all your effort on this. I really appreciate the help
Doug
Hi Doug,
I've isolated the issue area. But before I put a mod in, can you try two tests where you set of a FlowViewer run, and then put each device name into the 'Sensor' field at the bottom for each run?
I want to see if that works in situations like yours, because I'll need to be careful for other users that may be handling the situation that way.
Thanks,
Joe
Looks like that's the answer Joe. Adding the sensor name at the bottom gave me exactly what I'm looking for. I'm able to run reports per device. Do you want to see the debugs from them?
Thanks again
Doug
Thanks, Doug.
Putting the sensors in the path-format as you have looks like maybe the most proper way to do it, actually.
I'd like to make it easier for you to do these reports without having to type that information in each time, although it will be preserved across multiple reports and between tool applications.
So, I may add a flag to control it.
I'll send over an update FlowViewer_UI which should allow it to work based on the Device pulldown, instead of copy/pasting into the Sensor field.
Joe
That's great! Thanks for putting all the effort into it. This is a great tool so far and will greatly help our team. If you're anywhere around the SF Bay Area I'd love to buy you a dinner sometime....