FlowViewer and FlowGrapher do not show all flows
FlowViewer is a web-based netflow data analysis tool.
Brought to you by:
jloiacon
Menu
▾
▴
I Installed flow-tools and FlowViewer. All installations were without errors. I installed it on Ubuntu-22.04. I don't see any errors in the logs. If I run FlowViewer like this,
I get this.
If I run FlowGrapher with the same settings, they get something like this.
Only outputs streams with IP address 191.168.7.218
If in /etc/perl/FlowViewer_Utilities.pm here
print FILTER "filter-primitive start_flows\n";
print FILTER "type time-date\n";
print FILTER "permit ge $flows_start\n";
print FILTER "default deny\n";
print FILTER "filter-primitive end_flows\n";
print FILTER "type time-date\n";
print FILTER "permit lt $flows_end\n";
print FILTER "default deny\n";
print FILTER "\n";
print FILTER "filter-definition Flow_Filter\n";
I change default deny to default permit, then FlowViewer with the same launch parameters produces this,
So there are many flows on device at this time. Why doesn't FlowViewer show the other threads? Why it shows only flow with ip 191.168.7.218
Help.
Last edit: kuzma 2024-01-22
Hi Kuzma,
I don't really know why that is happening. Can you restore the FlowViewer_Utilities.pm file and we can start looking at the debug (DEBUG_VIEWER and DEBUG_GRAPHER)? I assume FlowViewer_Configuration.pm looks something like:
$debug_viewer = "Y";
$debug_grapher = "Y";
$debug_monitor = "Y";
$debug_group = "Y";
$debug_files = "N";
From documentation:
This parameter, if set to “Y”, will turn on debugging for FlowViewer. The debug output can be found in $flow_working/DEBUG_VIEWER.
Thanks for quick answer.
DEBUG_VIEWER and DEBUG_GRAPHER in attachment.
Regards, Kuzma
Can you share these files:
/var/www/cgi-bin/FlowViewer_4.6/Flow_Working/FlowViewer_filter_173805
/var/www/cgi-bin/FlowViewer_4.6/Flow_Working/FlowGrapher_filter_173835EW
If they're not there, you may have to set to "Y":
$debug_files = "Y";
... and run again.
Done!
Let's see what the underlying flow-tools command is producing. From the debug files (e.g., DEBUG_VIEWER) we can get the flow-tools command:
/usr/local/flow-tools/bin/flow-cat -a -t "01/22/2024 07:58:59" -T "01/22/2024 09:35:00" /var/flows/C6509/2024/2024-01/2024-01-22 | /usr/local/flow-tools/bin/flow-nfilter -f /var/www/cgi-bin/FlowViewer_4.6/Flow_Working/FlowGrapher_filter_173835EW -FFlow_Filter > /var/www/cgi-bin/FlowViewer_4.6/Flow_Working/FG_buckets_cat_173835EW
Try this command from the command line. If we look at the parts of the flow-tools command we see the 'cat' with the correct time-frame; then it is passed to flow-nfilter, etc. resulting in FG_buckets_cat_173835EW.
See if you get any hints, or where it is messing up. For example, if FG_buckets_cat_173835EW has everything, then we'll have to look into FlowViewer. If not, we'll have to look closer at flow-tools.
Hello, Joe.
Yesterday's net-flow files were overwritten. I am sending new ones, plus the result of the command.
In attachment my FlowViewer instalation steps. Maybe it will help.
Regards.
2 commands. First with nfilter, second without nfilter
First
/usr/local/flow-tools/bin/flow-cat -a -t "01/23/2024 14:58:59" -T "01/23/2024 15:05:01" /var/flows/C6509/2024/2024-01/2024-01-23 | /usr/local/flow-tools/bin/flow-nfilter -f /var/www/cgi-bin/FlowViewer_4.6/Flow_Working/FlowViewer_filter_172953 -FFlow_Filter | /usr/local/flow-tools/bin/flow-stat -f10 -S2 >/var/www/cgi-bin/FlowViewer_4.6/Flow_Working/FlowViewer_output_172953 2>>/var/www/cgi-bin/FlowViewer_4.6/Flow_Working/FlowViewer_output_172953
Second
/usr/local/flow-tools/bin/flow-cat -a -t "01/23/2024 14:58:59" -T "01/23/2024 15:05:01" /var/flows/C6509/2024/2024-01/2024-01-23 | /usr/local/flow-tools/bin/flow-stat -f10 -S2 >/var/www/cgi-bin/FlowViewer_4.6/Flow_Working/FlowViewer_output_172953 2>>/var/www/cgi-bin/FlowViewer_4.6/Flow_Working/FlowViewer_output_172953
Results in attachment. Such drama!
Regards
Yes, a little puzzling!
I'm starting to focus on time handling. See attached pic of the process. If I have the correct files, the times in the Filter are not matching the flow-cat times**. See if you can:
Verify that flow-tools is OK by working only from the command line with flow-tools only and modifying your Filter file by hand
A couple more FlowViewer runs, saving off the filter file, so that we can see if the Filter times are indeed being mis-calculated for some reason.
Just for reference, it would be nice to verify the captured flow start/end times. Use the "132 Columns" FlowViewer report. Pull the flow-tools command out of debug and run with no filter if FlowViewer still does it's weird filtering.
** The flow-cat times are supposed to start a little before your time of interest and end about 30 minutes after your end time to make sure all late-reporting flows that are in your period of interest are included)
Last edit: Joe Loiacono 2024-01-23
Hello Joe.
I think that I solved the problem. It was the wrong configured device. There was custom net flow record there. So net flow files format was wrong. Flow tools couldn't understand some fields. So result was unpredictable. When I changed record to default - all became look nice.
If sombody wants, I can send instruction, how install FlowViewer on Ubuntu.
Greate thanks for help.
Regards.
Nick (aka kuzma)
Great, Nick! By the way, nice install work. I know it gets harder with time as some of the dependencies get a little old. I'll add your install process to the SourceForge web site.
If so, you may add this file to my instruction. I hope, I collected all the packages that I downloaded.
Will do. The instructions are great.
Have you considered using SiLK as your flow collector? It is great code, very powerful, robust, still supported (Carnegie Mellon University) and free! It permits the collection of IPFIX data (where flow-tools is limited to netflow version 5.) Also, it will permit the viewing and analysis of IPv6 (As I recall, flow-tools doesn't do this.)
Great promotion :).
Yes, I am planing to try SiLK. if I can do it.
Hello, Joe.
Installed SiLK. Everything seems to be working fine. However, here's the question: When I used flow-tools, I could select the device from a dropdown menu (Netflow Source). When I use SiLK, I must enter sensor name manually and "Site" in the Netflow Source menu.
There are many devices. It is not comfortable. Is there any other way, using SiLK, to see the flows from the specific device?
Hi Nick,
Yes, you can treat SiLK devices just like flow-tools ones from the input screen perspective. Just make sure your SiLK directory structure looks like that in the attached image. Then make sure your FlowViewer_Configuration.pm file has entries like these below, they will put the SiLK devices in the Netflow Source pulldown. There's more explanation in the User Guide starting at page 15. I also wanted to mention that SiLK can handle 'sflow' data in case you come across it (some devices only have 'sflow' and not 'netflow')
@devices = ("ft_rtr1","ft_rtr2"); # or @devices=(); if not using flowtools
@ipfix_devices = ("silk_rtr1","silk_rtr2a","silk_rtr3","silk_rtr4","Site");
The configuration line below allows you to manage your SiLK storage by automatically aging out (removing) old data:
@ipfix_storage = ("silk_rtr1:10G","silk_rtr2:10G","nsilk_rtr3:15G","silk_rtr4");
Last edit: Joe Loiacono 2024-01-30
Here's the directory structure:
I continue to configure SiLK.
1) If it is possible, I want to show you my results.
As I understand, for comfortable display devices in FlowViewer, I need SiLK to store files from each device in a separate (his own) directory. I achieved this only by launching several instances of rwflowpack pointing different root directories for storing each device dirs and files. In each of these directories I put a sensor.conf file with sensor and sensor settings only for this device. If I make one sensor.conf file for all the sensors and the ports on which they should work, then each rwflowpack instance try to run processes on all the ports specified in the common sensor.conf file, which cause error. Therefore, each device has its own sensor.conf. But for all rwflowpack processes that are launched, you can specify one common silk.conf file. I put it in the root data directory. Sensor.conf files and directory structure are attached.
rwflowpack --no-daemon --root-directory=/var/flows/C6509/ --site-config-file=/var/flows/silk.conf --sensor-configuration=/var/flows/C6509/sensor.conf --log-directory=/var/log/C6509/
rwflowpack --no-daemon --root-directory=/var/flows/C2811/ --site-config-file=/var/flows/silk.conf --sensor-configuration=/var/flows/C2811/sensor.conf --log-directory=/var/log/C2811
2) I can't limit the size of files on disk. FlowViewer_Configuration.pm in attachment. The entire file storage structure in p.1 (above)
@devices = ();
@ipfix_devices = ("Site","C2811","C6509","testFGT","KLM-FGT");
@ipfix_storage = ("testFGT:2M","C6509:1G");
Last edit: kuzma 2024-02-02
It looks like you've set it up well. I remember a little bit of strangeness with the location of the sensor.config files with multiple exporters and I think I did it like you have it. I think the SiLK and my documentation could be better there. How are the results? Can you access by device like you want to?
I have run rwflowpack with the --packinterfaces option. According to SiLK:
"When this switch is present, rwflowpack writes additional information into the packed files: the router's SNMP input and output interfaces and the next-hop IP address. ... Note that this switch only affects newly created files. New records are always appended to an existing file in the file's current output format to maintain file integrity."
To get the SiLK file system trimming you need to run FlowViewer_CleanSiLK. I typically run this from crontab once a day. You can run it from the command line to verify it is working.
Here is my installation steps. SiLK and FlowViewer. Perhaps it will help somebody.
Very nice. Thanks, Nick. I'll add it to SourceForge and http://flowviewer.net