Menu ▾ ▴

Help with SiLK + FlowViewer

General Discussion
Sukrit
2014-11-03
2014-11-04

  • Sukrit

    Sukrit - 2014-11-03

    Hello,

    I have been trying to get FlowViewer + SiLK working for the past week now, have been pretty unsuccessful at it. Have perused the list here and not been able to pinpoint exactly what will get this working. Any help will be GREATLY appreciated. Running a Ubuntu 14.04 Server.

    1. I have gotten SiLK (3.9.0) running and its also getting NetFlow v9 from three Cisco routers. They are being stored as: (%N/%T/%Y/%m/%d/%x)
      /data/Router1/ all,ext2ext,in,int2int,inweb,out,outweb
      /data/Router2/ "
      /data/Router3/ "

    So there is one silk.conf and one sensor.conf in /data

    1. FlowViewer (4.4) is up as well and I can see and traverse the UI. Also, the FlowViewer, FlowTracker and FlowGrapher section all show the three devices from SiLK correctly in the "Netflow Source section"

    2. However, nothing comes up when I try to query. To keep things simple, I only modify the Start and End time+date. Also, as an example, I have tried with selecting the device as "Router1" and the root dir as /data OR /data/Router1. I dont get any reports.

    Here is what my FlowViewer.pm file looks like (I know this is long, sorry, but I am a bit under the gun to get this working)

    ~~~~~~~~~~~~~
    $ENV{PATH} .= ':/usr/local/bin:/usr/sbin';

    Server

    $FlowViewer_server = “removed “for security; # (IP address or hostname)

    Service

    $FlowViewer_service = "http"; # (http, or https)

    Directories and Files:

    $reports_directory = "/var/www/html/FlowViewer";
    $reports_short = "/FlowViewer";
    $graphs_directory = "/var/www/html/FlowGrapher";
    $graphs_short = "/FlowGrapher";
    $tracker_directory = "/var/www/html/FlowTracker";
    $tracker_short = "/FlowTracker";
    $cgi_bin_directory = "/usr/lib/cgi-bin/FlowViewer_4.4";
    $cgi_bin_short = "/cgi-bin/FlowViewer_4.4";
    $work_directory = "/usr/lib/cgi-bin/FlowViewer_4.4/Flow_Working";
    $save_directory = "/var/www/html/FlowViewer_Saves";
    $save_short = "/FlowViewer_Saves";
    $names_directory = "/usr/lib/cgi-bin/FlowViewer_4.4";
    $filter_directory = "/usr/lib/cgi-bin/FlowTracker_Files/FlowTracker_Filters";
    $rrdtool_directory = "/usr/lib/cgi-bin/FlowTracker_Files/FlowTracker_RRDtool";
    $dashboard_directory = "/var/www/html/FlowViewer_Dashboard";
    $dashboard_short = "/FlowViewer_Dashboard";

    @other_dashboards = ();

    @other_dashboards = ("/var/www/html/SOC","/var/www/html/NetOps");

    @dashboard_titles = ();

    @dashboard_titles = ("Performance","SLN","NetOps"); # titles must be in the same order as the directories

    $flow_data_directory = "/data/flows";
    $exporter_directory = "/data/flows/all_routers";
    $flow_bin_directory = "/usr/local/flow-tools/bin";
    $rrdtool_bin_directory = "/usr/bin";

    SiLK parameters

    $silk_data_directory = "/data";
    $silk_bin_directory = "/usr/local/bin";
    $sensor_config_directory = "/data";

    $silk_capture_buffer_pre = (125 * 60); # Start of SiLK file concatenation
    $silk_capture_buffer_post= (5 * 60); # End of SiLK file concatenation

    $silk_init_loadscheme = 1; # For Flows Initiated/Second - see SiLK rwcount documentation
    $silk_active_loadscheme = 5; # For Flows Active/Second - see SiLK rwcount documentation
    $silk_class_default = ""; # General SiLK file structure info. silk.conf, sensor.conf
    $silk_flowtype_default = ""; # General SiLK file structure info. silk.conf, sensor.conf
    $silk_type_default = "all"; # General SiLK file structure info. silk.conf, sensor.conf
    $silk_sensors_default = ""; # General SiLK file structure info. silk.conf, sensor.conf
    $silk_switches_default = ""; # General SiLK file structure info. silk.conf, sensor.conf

    General parameters

    $version = "4.4";
    $no_devices_or_exporters = "N";

    @devices = ("router_1","router_2","router_3","router_4","router_5","router_6"); # for flow-tools

    @ipfix_devices = ("router_ipfix_1","router_ipfix_2"); # for SiLK, if none: @ipfix_devices = ();

    @ipfix_devices = (“Router1”, “Router2", “Router3"); # for SiLK, if none: @ipfix_devices = ();

    @ipfix_storage = ("router_ipfix_1:15G","router_ipfix_2:500M");

    @ipfix_storage = ("Router1:10G", "Router2:10G", "Router3:10G");
    @exporters = ("192.168.200.1:New York Router","192.168.200.2:Prague Router");

    $flow_capture_interval = (35 * 60);
    $flow_file_length = (15 * 60);
    $start_offset = (90 * 60); # e.g., 90 minutes ago
    $end_offset = (30 * 60); # e.g., 30 minutes ago
    $use_even_hours = "Y";
    $N = 3;
    $use_NDBM = "N";
    $pie_chart_default = 0; # 0 = None; 1 = With Others; 2 = Without Others
    $number_slices = 9;
    $pie_colors = ['pie2 color1','pie2 color2','pie2 color3','pie2 color4','pie2 color5','pie2 color6','pie2 color7','pie2 color8','pie2 color9','pie2 color10'];
    $maximum_days = "91";
    $remove_workfiles_time = 286400;
    $remove_graphfiles_time = 7    86400;
    $remove_reportfiles_time = 7*86400;
    $time_zone = ""; # If left empty, will use system time zone
    $time_zone_dst_offset = (60 * 60); # Number of seconds of the Daylight Savings adjustment in your timezone
    $date_format = "MDY"; # MDY=MM/DD/YYYY DMY=DD/MM/YYYY DMY2=DD.MM.YYYY YMD=YYYY-MM-DD
    $labels_in_titles = "1"; # Set to "1" for labels in Tracker graph titles; "0" off
    $sip_prefix_length = "16";
    $dip_prefix_length = "16";

    UI Parameters

    $left_title = "NOC Management System";
    $left_title_link = "$cgi_bin_short/FV.cgi";
    $right_title = "Monitoring SWAN Network Data Flows";
    $right_title_link = "$cgi_bin_short/FV.cgi";

    Debug Parameters

    $debug_viewer = "Y";
    $debug_grapher = "Y";
    $debug_tracker = "Y";
    $debug_group = "Y";
    $debug_files = "N";

    Graphing parameters

    $transparent = "0";
    $x_ticks = "T";
    $long_ticks = "T";
    $skip_undef = "T";
    $graph_height = 310;
    $graph_width = 600;
    $t_margin = 10;
    $b_margin = 60;
    $l_margin = 10;
    $r_margin = 20;
    $bgclr = "white";
    $borderclrs = "black";
    $boxclr = "white";
    $fgclr = "gray90";
    $labelclr = "black";
    $axislabelclr = "black";
    $legendclr = "black";
    $valuesclr = "black";
    $textclr = "black";
    $x_axis_font = "('arial', 16)";
    $title_font = "('arial', 18)";
    $horz_max = ($graph_width / 2) - 44;
    $horz_pct = ($graph_width / 2) - 44;
    $horz_avg = ($graph_width / 2) - 44;
    $horz_min = ($graph_width / 2) - 44;
    $vert_max = ($graph_height - 70) + 2;
    $vert_pct = ($graph_height - 70) + 16;
    $vert_avg = ($graph_height - 70) + 30;
    $vert_min = ($graph_height - 70) + 44;
    $horz_mth = 15;
    $analyze_count = 5; # Any number between 3 and 10 inclusive. Must have at least [ $analyze_count + 1 ] $analyze_colors.
    $analyze_peak_width = 1000; # Number of observations to examine for peaks (per period)

    $analyze_colors = ['gray95','pale green','pale brown','pale red','pale blue','pale yellow'];

    $analyze_colors = ['gray95','pastel orange','pastel rose','pastel blue','pastel green','pastel yellow'];

    $analyze_colors = ['gray95','analysis1','analysis2','analysis3','analysis4','analysis5','analysis6','analysis7','analysis8','analysis9','analysis10'];
    $analyze_extension = 20;

    Tracking parameters

    $actives_webpage = "index.html";
    $log_directory = "/usr/lib/cgi-bin/FlowViewer_4.4/logs";
    $log_collector_short= "Y";
    $log_collector_med = "N";
    $log_collector_long = "N";
    $log_grapher_short = "Y";
    $log_grapher_long = "N";
    $collection_offset = 1800;
    $collection_period = 300;
    $graphing_period = 300;
    $recreate_cat_length= 6(6060); # Time length of concatenated file

    $rrd_dir_perms = 0777; # Scale these back once everything is working
    $filter_dir_perms = 0777;
    $work_dir_perms = 0777;
    $html_dir_perms = 0777;

    $html_file_perms = 0777;
    $graph_file_perms = 0777;
    $rrd_file_perms = 0777;
    $filter_file_perms = 0777;
    $tracker_file_perms = 0777;
    $saved_filters_perms= 0777;
    $actives_file_perms = 0777;

    $rrd_area = "FFE0C0";
    $rrd_line = "000000";
    $rrd_peak = "000000";
    $rrd_width = 600;
    $rrd_height = 150;
    $rrd_font = "000000AA";
    $rrd_back = "FFFFFF";
    $rrd_canvas = "FFFFFF";
    $rrd_grid = "CCCCCC88";
    $rrd_mgrid = "FF000033";
    $rrd_frame = "FFFFFF";
    $rrd_shadea = "FFFFFF";
    $rrd_shadeb = "FFFFFF";
    $rrd_thick = 0.3;
    $rrd_lower_limit = 0;
    $rrd_slope_mode = "--slope-mode"; # $rrd_slope_mode = ""; will square graphs up
    $rrd_vrule_color = "FF0000";
    $rrd_hrule_color = "FF0000";
    $thumbnail_width = 250;
    $thumbnail_height = 80;
    $hr_width = $rrd_width + 130;

    Standard Deviation Alert parameters

    $sigma_type_1 = "6:2.67"; # Num of obs in mean : number of sigmas for threshold : Must restart FlowTracker_Collector
    $sigma_type_2 = "12:4"; # Num of obs in mean : number of sigmas for threshold : Must restart FlowTracker_Collector
    $sigma_type_3 = "12:3"; # Num of obs in mean : number of sigmas for threshold : Must restart FlowTracker_Collector

    Scanning Parameters

    $dscan_parameters = "-w -W"; # flow-tools only, ignores inbound and outbound port 80
    $scan_model = "2"; # SiLK only: 0=TRW&BLR; 1=TRW only; 2=BLR only
    $trw_internal_set = ""; # SiLK only: Full file name, required when using TRW model

    Webpage Parameters

    $filename_color = "#CF7C29";
    $dns_column_width = 60;
    $detail_lines = 200;
    $asn_width = 60;
    $default_report = 10; # See FlowViewer Users Guide for details
    $default_graph = "bps"; # See FlowViewer Users Guide for details
    $default_lines = 100;
    $default_identifier = "DNS"; # Use "IP" for IP addresses; "DNS" to resolve addresses to names
    $default_flows = 1;

    Commands (full directory names)

    $dig = "/usr/bin/dig +time=1 +tries=1 -x ";
    $dig_forward = "/usr/bin/dig +time=1 +tries=1 ";
    ~~~~~~~~~~~

     

    • Joe Loiacono

      Joe Loiacono - 2014-11-03

      "Sukrit" sd2014@users.sf.net wrote on 11/03/2014 09:46:55 AM:

      I have been trying to get FlowViewer + SiLK working for the past
      week now, have been pretty unsuccessful at it. Have perused the list
      here and not been able to pinpoint exactly what will get this
      working. Any help will be GREATLY appreciated. Running a Ubuntu 14.04
      Server.
      1. I have gotten SiLK (3.9.0) running and its also getting NetFlow
      v9 from three Cisco routers. They are being stored as:
      (%N/%T/%Y/%m/%d/%x)
      /data/Router1/ all,ext2ext,in,int2int,inweb,out,outweb
      /data/Router2/ "
      /data/Router3/ "
      So there is one silk.conf and one sensor.conf in /data

      Hello Sukrit,

      Can you please copy silk.conf into each device directory.

      Then, after attempts, can you look into the DEBUG_* files in
      $flow-working?

      Please send what you find.

      We will get this going.

      Joe

       
      alternate

  • Sukrit

    Sukrit - 2014-11-03

    Hi Joe, thanks a lot for looking and your time.

    I did as you suggested: made a copy of the silk.conf file in each directory. Few things I noticed:

    1. Didnt find a $flow_working in the .pm file. I found a $work_directory. This is for me:

    $work_directory = "/usr/lib/cgi-bin/FlowViewer_4.4/Flow_Working";

    1. Dont see any debug files in $work_directory

    root@nf-coll:/usr/lib/cgi-bin# ls -lt FlowViewer_4.4/Flow_Working/
    total 0

    1. After I copied the silk.conf to the device directories, I just visited the FlowViewer site (didnt restart apache or anything)

    Thanks for your time again!

     

  • Joe Loiacono

    Joe Loiacono - 2014-11-03

    Hmm. No DEBUG_VIEWER (or DEBUG_GRAPHER, or ...) file in $work_directory?

    It's beginning to sound like a permissions problem.

    Make sure that Apache can write into your $cgi_bin_directory, $reports_directory, $work_directory, etc.

    We need to be able to see the DEBUG files. Nothing is getting written into $work_directory so I suspect Apache is blocked from writing into it.

    So - give everything 0777 until it is working, and then scale back as you see appropriate given your environment.

    Thanks,

    Joe

     

    Last edit: Joe Loiacono 2014-11-03

  • Sukrit

    Sukrit - 2014-11-03

    Hi Joe,

    Thanks for the tip. Indeed! This was a permissions issue. I gave the apache2 group permissions to the following (for future record):

    chgrp -R www-data /usr/lib/cgi-bin/FlowViewer_4.4/
chgrp -R www-data /usr/lib/cgi-bin/FlowTracker_Files/
chgrp -R www-data /usr/lib/cgi-bin/
chgrp -R www-data /var/www/html/

    Thanks for all your help! This was great! I foresee my team using this tool for quite some time on very large data sets (multi Gig netflow records), so I will probably keep posting some questions for you.

    Thanks!

     

    Last edit: Sukrit 2014-11-03

  • Joe Loiacono

    Joe Loiacono - 2014-11-04

    Excellent! I do look forward to communications. One thing I've come to value is that SiLK does a lot of pre-filtering for you through the sensor.conf file where you segment the data into internal and external ipblocks. This keeps one from using type=all all of the time which is slower.

    For example, I set up one sensor using our internal IPv4 networks, etc. (forgot to include our IPv6 network.) Then found I could very quickly analyze IPv6 via ext2ext! I know that is probably not optimal, but it opened my eyes to the benefits of 'segmenting' by groups in the sensor.conf file.

     

Log in to post a comment.

Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.