Menu

FlowViewer with Silk

2014-04-16
2015-07-15
1 2 > >> (Page 1 of 2)
  • David Taylor

    David Taylor - 2014-04-16

    I have a situation where Silk and FlowViewer seem to be installed correctly, silk is logging data from an ASA for testing and I have the flowFiewer dashboard up, however no data is showing up in reports and the drop-down menus such as "interfaces" are not populating with interface information.

    My configuration is Centod 6.5 with Silk 3.8.1. Silk has all it's config files in /etc and data is being logged to /data.
    The ASA is outputting v9 Netflow and the directories below /data are

    /data/in
    /data/inweb
    /data/out
    /data/outweb

    The section of FlowViewer_Configuration.pm I have changed is

    $flow_data_directory = "/data/flows";
    $exporter_directory = "/data/flows/all_routers";
    $flow_bin_directory = "/usr/local/flow-tools/bin";
    $rrdtool_bin_directory = "/usr/bin";

    $silk_data_directory = "/data";
    $silk_bin_directory = "/usr/local/bin";
    $sensor_config_directory = "/etc";

    General parameters

    $version = "4.3";
    $no_devices_or_exporters = "N";
    @devices = (); # for flow-tools
    @ipfix_devices = ("hf01.syd01"); # for SiLK, if none: @ipfix_devices = ();
    @ipfix_storage = ("hf01.syd01:15G");
    @exporters = ();

    $flow_capture_interval = (35 * 60);
    $flow_file_length = (15 * 60);
    $start_offset = (90 * 60); # e.g., 90 minutes ago
    $end_offset = (30 * 60); # e.g., 30 minutes ago
    $use_even_hours = "Y";
    $N = 3;
    $use_NDBM = "N";
    $pie_chart_default = 1; # 0 = None; 1 = With Others; 2 = Without Others
    $number_slices = 7;
    $maximum_days = "91";
    $remove_workfiles_time = 286400;
    $remove_graphfiles_time = 7
    86400;
    $remove_reportfiles_time = 7*86400;
    $time_zone = ""; # If left empty, will use system time zone
    $time_zone_dst_offset = (60 * 60); # Number of seconds of the Daylight Savings adjustment in your timezone
    $date_format = "YMD"; # MDY=MM/DD/YYYY DMY=DD/MM/YYYY DMY2=DD.MM.YYYY YMD=YYYY-MM-DD
    $labels_in_titles = "1"; # Set to "1" for labels in Tracker graph titles; "0" off
    $sip_prefix_length = "16";
    $dip_prefix_length = "16";
    $silk_all_only = "N";
    $silk_flows_loadscheme = 5;

    Any help would be greatly appreciated.

     

    Last edit: David Taylor 2014-04-16
  • Joe Loiacono

    Joe Loiacono - 2014-04-16

    Hi David,

    The SiLK collection directory structure needs to have devices immediately after the $silk_data_directory. So in your case that would be:

    /data/hf01.syd01

    You'll need to create that directory and give it enough permissions so that silk processes can write into it. Put a copy of silk.conf in there as well. rwflowpack will create the 'in', 'out', etc. subdirectories.

    Kyle Kroskey has offered up diffs for a cleaner way to do this and we'll put that in the next release.

    Regarding the Interface dropdowns. You have to first come up with a 'SNMP index to Interface name' mapping, and then put that in the NamedInterfaces_Devices file provided in the distro.

    Joe

     
  • David Taylor

    David Taylor - 2014-04-16

    Hi Joe,
    Thanks again for you fast reply.

    In terms of the silk path structure, in the silk.conf file I needed to update the "path-format" option as follows

    path-format "%N/%T/%Y/%m/%d/%x"

    I'm still working on the interface mapping.

    D

     

    Last edit: David Taylor 2014-04-17
    • Joe Loiacono

      Joe Loiacono - 2014-04-17

      Looks like the only difference from the default path-format is the %C, which is the "class name". Did you do this in place of manually creating a 'device' subdirectory under $silk_data_directory?

      Can you show the resulting directory structure and your sensor.conf and silk.config files (anonimized) and the location of these?

      Thanks,

      Joe

       
  • Shahzad

    Shahzad - 2014-05-25

    Hi David,

    Any update how did you manage it?
    In my setup, we have only silk with 3 sensors defined having Netflowv9 only and no IPFIX flows.
    After, I changed path-format by adding %N at the start of default string, flowpack process of silk started creating files inside as required.
    After that, I was having error "site configuration file not found" in httpd error log. The resolution was to copy silk.conf to all 3 subdirectories and the main silk data directory; but guess what, now when I run same command extracted from flowViewer DeBug log starting with rwfilter with a pipe to rwstats, I don't get any output on screen.
    Note:- per flowviewer format, rwfilter doesn't pass sensor name but override data directory of each subdirectory. If I change it with senspr name, it works.

    Please update how should i resolve this?

    Rgds,
    Shazad

     
  • Peter

    Peter - 2014-05-28

    I'm running into a similar situation. I've followed the above directions and now have two v9 exporters and what I think is a standard tree:

    /data/F5520N1/(silkdirs)
    /data/R2811N1/(silkdirs)

    they're in the FlowViewer_Configuration.pm file:
    @ipfix_devices = ("F5520N1","R2811N1"); # for SiLK, if none: @ipfix_devices = ();
    @ipfix_storage = ("F5520N1:15G","R2811N1:500M");

    @devices is null
    they're also in exporters:

    @exporters = ("192.168.168.1:Border Firewall","10.20.30.1:IPV6 Tunnel Router");

    but I don't think that should/needs to be there.

    I can get data from the silk tools by hand.

    However, I think I have something misconfigured because the debug files aren't referencing the right directories:

    silk_command: /usr/local/bin/rwfilter --data-rootdir=/data/R2811N1 --type=all (etc)

    Now, the data-rootdir is /data (as referenced in the config files). If I take the output from the debug file and change "--data-rootdir" to "--sensor=R2811N1" I get output, so I know the data is there. I'm not sure what I've got messed up in my config that I can't get Flowviewer to manipulate it....

     
  • Joe Loiacono

    Joe Loiacono - 2014-05-28

    Shazad and Peter,

    I'm anxious to get this cleaned up. Kyle Kroskey has a better way to do this and I'd like to make his changes. I'll work on this now.

     
    • Peter

      Peter - 2014-05-28

      Cool, I'll wait until we hear back.

      Thanks!

       
  • Shahzad

    Shahzad - 2014-05-30

    Hi Peter,

    I have same problem as you do have, in command line rwfilter "-sensor" works but with "data-rootdir" it doesnt work.

    Hello Joe,
    Thank you for the update. How much time will it take?
    Secondly, how can we restrict flowviewer to user based authentication?

    Rgds,
    Shahzad

     
  • Joe Loiacono

    Joe Loiacono - 2014-05-30
    Shazhad,
    
    I am actively working this around my day job :-). Hope to get it soon, and
    will put out a new version together with a better explanation in the
    documentation.
    
    Kyle Kroskey has done a nice job cleaning this up... I've included his
    changes below. Note: he made the change for FlowTracker_Collector; I've
    modified it to match FlowGrapher_Main.cgi which you can use to test more
    quickly. If you don't have graphics set up yet, make the change to
    FlowViewer_Main.cgi.
    
    Note also that he has named his probe, his sensor, and his device all the
    same thing: S0. Thank you Kyle.
    
    Kyle's changes:
    
    Both sensor.conf and silk.conf are in /netflow. The --data_rootdir was there
    from the get go. What I added was the --sensor to the rwfilter command. Just
    breaks it apart from the $data_root_dir variable above it.
    
    root@maverick:/usr/lib/cgi-bin/FlowViewer# diff /usr/src/FlowViewer_4.3/FlowGrapher_Main.cgi /usr/lib/cgi-bin/FlowViewer/FlowGrapher_Main.cgi 
    548c548
    <          $data_root_dir = $silk_data_directory ."/". $device_name;
    ---
    >          $data_root_dir = $silk_data_directory;
    580c580
    <          $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end $window_type=$time_window --pass=$prefiltered_file";
    ---
    >          $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --sensors=$device_name --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end $window_type=$time_window --pass=$prefiltered_file";
    
    ----
    root@maverick:/netflow# cat sensor.conf 
    probe S0 ipfix
       listen-on-port 18001
       protocol tcp
    end probe
    
    sensor S0
       ipfix-probes S0
       internal-ipblocks 192.168.1.0/24 10.0.0.0/8 224.0.0.0/8
       external-ipblocks remainder
    end sensor
    
    ----
    root@maverick:/netflow# cat silk.conf 
    # silk.conf for the "twoway" site
    # RCSIDENT("$SiLK: silk.conf 52d8f4f62ffd 2012-05-25 21:16:30Z mthomas $")
    
    # For a description of the syntax of this file, see silk.conf(5).
    
    # The syntactic format of this file
    #    version 2 supports sensor descriptions, but otherwise identical to 1
    version 2
    
    # NOTE: Once data has been collected for a sensor or a flowtype, the
    # sensor or flowtype should never be removed or renumbered.  SiLK Flow
    # files store the sensor ID and flowtype ID as integers; removing or
    # renumbering a sensor or flowtype breaks this mapping.
    
    sensor 0 S0    "BRAKJ_v9 Sensor"
    
    class all
        sensors S0
    end class
    
    # Editing above this line is sufficient for sensor definition.
    # Be sure you understand the workings of the packing system before
    # editing the class and type definitions below.  In particular, if you
    # change or add-to the following, the C code in packlogic-twoway.c
    # will need to change as well.
    
    class all
        type  0 in      in
        type  1 out     out
        type  2 inweb   iw
        type  3 outweb  ow
        type  4 innull  innull
        type  5 outnull outnull
        type  6 int2int int2int
        type  7 ext2ext ext2ext
        type  8 inicmp  inicmp
        type  9 outicmp outicmp
        type 10 other   other
    
        default-types in inweb inicmp
    end class
    
    default-class all
    
    # The layout of the tree below SILK_DATA_ROOTDIR.
    # Use the default, which assumes a single class.
    path-format "%N/%T/%Y/%m/%d/%x"
    
    # The plug-in to load to get the packing logic to use in rwflowpack.
    # The --packing-logic switch to rwflowpack will override this value.
    # If SiLK was configured with hard-coded packing logic, this value is
    # ignored. 
    packing-logic "/usr/local/lib/silk/packlogic-twoway.so"
    
    ____
    root@maverick:/netflow# ls -l
    total 20
    -rwxrwxrwx 1 root root 6976 Mar 16 19:23 rwflowpack.conf
    drwxrwxrwx 8 root root 4096 Mar 20 19:27 S0
    -rwxrwxrwx 1 root root  196 Mar 22 20:14 sensor.conf
    -rwxrwxrwx 1 root root 1703 Mar 19 20:05 silk.conf
    
     
  • Joe Loiacono

    Joe Loiacono - 2014-05-30

    Shahzad,

    With respect to User Authentication, FlowViewer itself does not provide it. However, you can put basic userid/password by using the normal 'htpasswd' capability.

    Also - Peter had some questions about @exporters ... that should be left blank: @exporters = ();

    Regards,

    Joe

     
  • Shahzad

    Shahzad - 2014-06-01

    Dear Joe,

    Thanks for your help. This is what I was suspecting that we have to append --sensor in the code itself.

    I changed both lines in FlowGrapher_Main.cgi, and it started showing data correctly.

    When I checked FlowViewer_Main.cgi, there $rwfilter_command is in lot of places into the file.
    Thinking your referring to FlowTracker_Collector file, I changed data_root_dir and rwfilter_command accordingly in it but still having problem.
    The problem is that FlowGrapher is populating data from 3 different sensors (networks) properly but FlowViewer is showing data based on data_root_dir, meaning from all sensors at the same time with no filtering to specific sensor based on my appending "--sensor=$device_name".

    Please update in which places should I append "--sensor=device_name" to fix the issue? OR do I append it everywhere in all 3 files i.e. FlowViewer_Main.cgi, FlowGrapher_Main.cgi and FlowTracker_Main.cgi?

    Regards,

    Shahzad

     
  • Shahzad

    Shahzad - 2014-06-01

    Dear Joe,

    In FlowViewer_Main.cgi, I appended --sensor in 4 lines:
    # Prepare rwfilter command including flow window parameters

        if ($flow_select eq 1) {
                $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --sensors=$device_name --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end --active=$time_window $partitioning_switches --pass=stdout";
        } elsif ($flow_select eq 2) {
                $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --sensors=$device_name --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end --etime=$time_window $partitioning_switches --pass=stdout";
        } elsif ($flow_select eq 3) {
                $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --sensors=$device_name --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end --stime=$time_window $partitioning_switches --pass=stdout";
        } elsif ($flow_select eq 4) {
                $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --sensors=$device_name --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end --stime=$time_window --etime=$time_window $partitioning_switches --pass=stdout";
    
     
  • Shahzad

    Shahzad - 2014-06-01

    Joe,

    I changed your suggested configurations in FlowViwer_main.cgi and FlowGrapher_main.cgi i.e. Excluding $device_name from $data_root_dir and adding --sensor=$device_name into $rwfilter_command.
    Both are working fine as required, but FlowTracker_main.cgi doesn't have variables with such names in it to be changed ; and it is main part in flowviewer dashboard :)
    Isn't there any single file where from we can control these all three things?

    Rgds,
    shahzad

     
  • Joe Loiacono

    Joe Loiacono - 2014-06-02
    [ Hmmm I've tried twice now to upload an answer today ]
    
    Shahzad - first thanks for your patience and efforts.
    
    The changes need to go into FlowTracker_Collector (not FlowTracker_Main) and then you need to restart it.
    
    Hare are the diffs for FlowVierwer_Main.cgi, FlowGrapher_Main.cgi, and FlowTracker_Collector:
    
    diff silk_fix/FlowGrapher_Main.cgi /var/www/cgi-bin/FlowViewer_4.3/FlowGrapher_Main.cgi
    548c548
    <       $data_root_dir = $silk_data_directory;
    ---
    >       $data_root_dir = $silk_data_directory ."/". $device_name;
    580c580
    <       $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --sensors=$device_name --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end $window_type=$time_window $partitioning_switches --pass=$prefiltered_file";
    ---
    >       $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end $window_type=$time_window $partitioning_switches --pass=$prefiltered_file";
    
    diff silk_fix/FlowTracker_Collector /var/www/cgi-bin/FlowViewer_4.3/FlowTracker_Collector
    307c307
    <                       $data_root_dir = $silk_data_directory;
    ---
    >                       $data_root_dir = $silk_data_directory ."/". $device_name;
    368c368
    <                               $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --sensors=$device_name --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end $window_type=$time_window --pass=$prefiltered_file";
    ---
    >                               $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end $window_type=$time_window --pass=$prefiltered_file";
    
    diff silk_fix/FlowViewer_Main.cgi /var/www/cgi-bin/FlowViewer_4.3/FlowViewer_Main.cgi
    445c445
    <         $data_root_dir = $silk_data_directory;
    ---
    >         $data_root_dir = $silk_data_directory ."/". $device_name;
    472c472
    <               $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --sensors=$device_name --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end --active=$time_window $partitioning_switches --pass=stdout";
    ---
    >               $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end --active=$time_window $partitioning_switches --pass=stdout";
    474c474
    <               $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --sensors=$device_name --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end --etime=$time_window $partitioning_switches --pass=stdout";
    ---
    >               $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end --etime=$time_window $partitioning_switches --pass=stdout";
    476c476
    <               $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --sensors=$device_name --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end --stime=$time_window $partitioning_switches --pass=stdout";
    ---
    >               $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end --stime=$time_window $partitioning_switches --pass=stdout";
    478c478
    <               $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --sensors=$device_name --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end --stime=$time_window --etime=$time_window $partitioning_switches --pass=stdout";
    ---
    >               $rwfilter_command = "$silk_bin_directory/rwfilter --data-rootdir=$data_root_dir --type=$silk_flow_type --start-date=$silk_cat_start --end-date=$silk_cat_end --stime=$time_window --etime=$time_window $partitioning_switches --pass=stdout";
    
     
  • Peter

    Peter - 2014-06-03

    Hi,

    I've applied these diffs and can now get data from the device. However, for some strange reason the grapher is inconsistent. Sometimes it draws graphs, sometimes it doesn't. Either way I get the raw source data listed in a table blow the (sometimes blank) graph so I know it's picking data to graph. I don't see anything useful in DEBUG_GRAPHER (it just says "done_FLOWS to create_graph to done" in about 3 seconds).

    This only applies to my silk source, I have a V5 exporter that graphs fine down to the sub-minute level. The V9 exporter will graph if I select, say, a whole day but if I pick a one-hour slice I get the table data but no graph. I can't seem to pin down what does and doesn't make a graph come out. Any way I can help narrow it down?

    Thanks,

     
  • Joe Loiacono

    Joe Loiacono - 2014-06-03

    First let's see if you're getting data to plot when it is not plotting.

    Can you add the following debug line to FlowGrapher_Main.cgi (looks like it would be after line 1042):

    1042 $buckets[$i] = $buckets[$i] / $bucket_size;
    1043 print DEBUG "buckets[$i]: $buckets[$i]\n";

    We'll see if it is a data problem, or a graphing problem.

     
  • Peter

    Peter - 2014-06-03

    Hi,

    So, my latest attempt after adding your debug for dates 6/3 14:00 - 15:00, generated 719 buckets, all valued at 0. The text representation of the data did populate under the blank graph. I also tried 01:00-05:00 and got a graph, 707 of 3587 buckets were populated.

    Thanks,

     
  • Joe Loiacono

    Joe Loiacono - 2014-06-03

    Hi Peter,

    Could you check your router time? Sometimes if it is off, the files (based on host system time) don't sync up with the netflow that is exported.

    You can look at the timestamps of the flows by (e.g.,):

    rwcut out-S1_20140603.20 > ~/temp.out

    They should be close to the file time.

     
  • Peter

    Peter - 2014-06-03

    It looks like the firewall is logging in GMT even though the clock is set to EDT. The stamps on the data in the file is +4 from the file date/time (GMT vs EDT). Setting the FlowViewer_Configuration.pm timezone to GMT seems to have fixed it for the ASA, but breaks things for the V5 collector.......

     
    • Joe Loiacono

      Joe Loiacono - 2014-06-04

      Might have to do the same time check with the device you are collecting v5 from.

      I agree with your comment below. In the end, it has turned out for me to be easier overall just to use GMT everywhere.

       
  • Shahzad

    Shahzad - 2014-06-04

    Dear Peter / Joe,

    I am sure we are aware that SiLK uses default TZ UTC for both storing files and taking input/ giving output to users. We can control the second part i.e. input/output by some compilation parameters but still it will store the files per UTC timestamp.

    Dear Joe,

    Thanks a ton for your helping out in our case.

    I am in GST TZ and I haven't changed TZ in FlowViewer_Configuration.pm and for me FlowGrahper is working fine and FlowViewer shows some inconsistency in "Detect Scanning" Report.
    Using FLowViewer, when we extract "Detect Scanning" Report for last one hour i.e. 1300-1400 ZBL with no filters and include if: Entirely in Specified Time Span,
    the start and end time show are in UTC i.e. 0900-1000. In SiLK, it is know that it keeps time in UTC but during compilation, it allows to change TimeZone to your region
    only for Report inputs from users and Report outputs to the users, though it stores the files under same UTC format.

    So, as per my configuration, when I run the rwfilter commands in my setup from CLI, it gives me RIGHT time but not on FlowViwer GUI.

    If I check DEBUG_VIEWER, command is as follows:
    rwfilter_command: /usr/local/bin/rwfilter --data-rootdir=/silk_data_repository --sensors=Zabeel-Secure-Colo --type=all --start-date=2014/06/04:05 --end-date=2014/06/04:07 --stime=2014/06/04:6:00:00-2014/06/04:7:00:00 --etime=2014/06/04:6:00:00-2014/06/04:7:00:00 --pass=/var/www/cgi-bin/FlowViewer_4.3/Flow_Working/FlowViewer_scanner_081416

    Running same command as:
    rwfilter_command: /usr/local/bin/rwfilter --data-rootdir=/silk_data_repository --sensors=Zabeel-Secure-Colo --type=all --start-date=2014/06/04:05 --end-date=2014/06/04:07 --stime=2014/06/04:6:00:00-2014/06/04:7:00:00 --etime=2014/06/04:6:00:00-2014/06/04:7:00:00 --pass=stdout | rwcut
    shows me RIGHT timings

    Secondly Joe, I changed your recommended configs into FlowTracker_Collector file at 2 places but still I can't see any tracker graphs on Flow Trackers.

    here is the Debug (with no commands in it)
    tail: DEBUG_TRACKER: file truncated
    In FlowTracker_Display.cgi
    FlowTracker_Display: filter_hash: TR_c_s filter_filename: c_s.fil
    tail: DEBUG_TRACKER: file truncated
    In FlowTracker_Display.cgi
    FlowTracker_Display: filter_hash: TR_c_s filter_filename: c_s.fil
    tail: DEBUG_TRACKER: file truncated
    In FlowTracker_Display.cgi
    FlowTracker_Display: filter_hash: TR_z_s_c filter_filename: z_s_c.fil
    In FlowTracker_main.cgi action: Revise Tracking
    FORM{start_date}: 01/01/2000 start_date: 01/01/2000 FORM{end_date}: 01/01/2000 end_date: 01/01/2000
    This device is exporting IPFIX

    Please help in both cases.

     
    • Joe Loiacono

      Joe Loiacono - 2014-06-04
      Comments in-line:
      
      > From: "Shahzad" <shahzadenwar@users.sf.net>
      > To: "[flowviewer:discussion] " <general@discussion.flowviewer.p.re.sf.net>
      > Date: 06/04/2014 01:14 AM
      > Subject: [flowviewer:discussion] FlowViewer with Silk
      > 
      > Dear Peter / Joe,
      > I am sure we are aware that SiLK uses default TZ UTC for both 
      > storing files and taking input/ giving output to users. We can 
      > control the second part i.e. input/output by some compilation 
      > parameters but still it will store the files per UTC timestamp.
      
      Good to know.
      
      > I am in GST TZ and I haven't changed TZ in 
      > FlowViewer_Configuration.pm and for me FlowGrahper is working fine
      
      So the flow start and end times are correct? They are listed in GST time?
      
      > and FlowViewer shows some inconsistency in "Detect Scanning" Report.
      > Using FLowViewer, when we extract "Detect Scanning" Report for last 
      > one hour i.e. 1300-1400 ZBL with no filters and include if: Entirely
      > in Specified Time Span,
      > the start and end time show are in UTC i.e. 0900-1000. In SiLK, it 
      > is know that it keeps time in UTC but during compilation, it allows 
      > to change TimeZone to your region
      > only for Report inputs from users and Report outputs to the users, 
      > though it stores the files under same UTC format.
      > So, as per my configuration, when I run the rwfilter commands in my 
      > setup from CLI, it gives me RIGHT time but not on FlowViwer GUI.
      > If I check DEBUG_VIEWER, command is as follows:
      > rwfilter_command: /usr/local/bin/rwfilter --data-rootdir=/
      > silk_data_repository --sensors=Zabeel-Secure-Colo --type=all --
      > start-date=2014/06/04:05 --end-date=2014/06/04:07 --stime=2014/06/
      > 04:6:00:00-2014/06/04:7:00:00 --etime=2014/06/04:6:00:00-2014/06/04:
      > 7:00:00 --pass=/var/www/cgi-bin/FlowViewer_4.3/Flow_Working/
      > FlowViewer_scanner_081416
      > Running same command as:
      > rwfilter_command: /usr/local/bin/rwfilter --data-rootdir=/
      > silk_data_repository --sensors=Zabeel-Secure-Colo --type=all --
      > start-date=2014/06/04:05 --end-date=2014/06/04:07 --stime=2014/06/
      > 04:6:00:00-2014/06/04:7:00:00 --etime=2014/06/04:6:00:00-2014/06/04:
      > 7:00:00 --pass=stdout | rwcut
      > shows me RIGHT timings
      
      OK, so that says the rwfilter_command is OK with respect to time zone.
      
      You can confirm this by looking at the intermediate file:
      
      host> rwcut < /var/www/cgi-bin/FlowViewer_4.3/Flow_Working/FlowViewer_scanner_143833
      
      After this intermediate step, the next steps involve rwsort, and
      then rwscan. You can see these in the DEBUG_VIEWER file.
      
      Can you run each of those and see if and where the times are
      getting changed?
      
      Also - try the Printed Reports: 132 Columns
      
      Also - can we see what happens to FV Scanner, and FG, when you set
      the timezone variable in the config file?
      
      > Secondly Joe, I changed your recommended configs into 
      > FlowTracker_Collector file at 2 places but still I can't see any 
      > tracker graphs on Flow Trackers.
      
      See comment on running flowtracker_restart, plus for debug look into
      DEBUG_TRACKER_C for FlowTracker_Collector, and DEBUG_TRACKER_G for FlowTracker_Grapher.
      
       
  • Shahzad

    Shahzad - 2014-06-04

    Dear Joe,

    Additionally, I am not sure how can we restart the FlowTracker.

    [root@euc010007013102 ~]# /var/www/cgi-bin/FlowViewer_4.3/tools/flowtracker_restart
    Usage: /var/www/cgi-bin/FlowViewer_4.3/tools/flowtracker_restart {start|stop|restart}
    [root@euc010007013102 ~]#
    [root@euc010007013102 ~]# /var/www/cgi-bin/FlowViewer_4.3/tools/flowtracker_restart restart
    Stopping FlowTracker processes: FlowTracker_Collector: no process killed
    Problem stopping FlowTracker_Collector
    FlowTracker_Grapher: no process killed
    Problem stopping FlowTracker_Grapher
    Done.
    Starting FlowTracker processes: Starting FlowTracker_Collector:
    /bin/su: user flowviewer does not exist
    Problem starting FlowTracker_Collector
    Starting FlowTracker_Grapher:
    /bin/su: user flowviewer does not exist
    Problem starting FlowTracker_Grapher
    Done.
    [root@euc010007013102 ~]#

    Rgds,
    Shahzad

     
    • Joe Loiacono

      Joe Loiacono - 2014-06-04

      Shahzad,

      Line 15 in the start-up script sets the userid that you want to use to run FlowTracker_Collector and FlowTracker_Grapher. You can change it however you like.

      Then start the script with 'sudo' which will accomplish the 'su' to that user. Note: root privileges are not required. You can simply start it from a command line:

      host>FlowTracker_Collector &
      host>FlowTracker_Grapher &

       
1 2 > >> (Page 1 of 2)

Log in to post a comment.

Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.