Looking for the latest version? Download FlowViewer_4.6.tar (1.3 MB)
Home
Name Modified Size Downloads / Week Status
Totals: 11 Items   14.3 MB 197
FlowViewer.pdf 2015-04-06 4.3 MB 2626 weekly downloads
FlowViewer_4.6.tar 2015-04-06 1.3 MB 125125 weekly downloads
README 2015-04-06 53.9 kB 3434 weekly downloads
FlowViewer_4.5.tar 2014-11-07 1.4 MB 55 weekly downloads
FlowViewer_4.4.tar 2014-08-15 1.4 MB 11 weekly downloads
FlowViewer_4.3.tar 2014-02-11 1.0 MB 11 weekly downloads
FlowViewer_4.2.2.tar 2013-09-27 1.0 MB 11 weekly downloads
FlowViewer_4.2.1.tar 2013-09-16 1.0 MB 11 weekly downloads
FlowViewer_4.2.tar 2013-09-16 984.6 kB 11 weekly downloads
FlowViewer_4.1.tar 2013-09-16 921.6 kB 11 weekly downloads
FlowViewer_4.0.tar 2013-09-16 890.9 kB 11 weekly downloads
# README (this file) FlowViewer V4.6 Date: 04/06/2015 # # FlowViewer is a set of three tools (FlowViewer, FlowGrapher, # FlowMonitor) that create text reports, graph reports, and # long-term monitor reports from flow-tools and SiLK captured # and stored netflow data. FlowViewer can run with both flow-tools # and SiLK simultaneously. Flow-tools can handle up to v7; SilK # can handle v5, v9, and IPFIX. The User's Guide is very helpful. # # Software Dependencies: # # flow-tools http://code.google.com/p/flow-tools (If collecting v5 only) # SiLK http://tools.netsa.cert.org/silk (If collecting IPFIX) # libfixbuf http://tools.netsa.cert.org/silk (If collecting IPFIX) # gd http://www.libgd.org/Downloads # GD http://search.cpan.org/~lds/GD-2.30 # GD::Graph http://search.cpan.org/~mverb/GDGraph-1.43 # GD::Text http://search.cpan.org/~mverb/GDTextUtil-0.86/Text # RRDtool http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/pub # # Quick Upgrade # # 0. If using SiLK, must upgrade to v3.8.0 or newer # 1. Untar the package into a new cgi-bin subdirectory # 2. Configure FlowViewer_Configuration.pm variables to your environment # and create all necessary directories with proper permissions # 3. Replace old logos with new buttons (will be done automatically) # 4. Copy FlowViewer.css, FlowViewer.pdf to $reports_directory # 5. Configure FlowViewer_Configuration.pm to point to existing # FlowMonitor_Filter and FlowMonitor_RRDtool directories # [For prior v4.5 you can rename these directories to: # FlowMonitor_Filter and FlowMonitor_RRDtool or change the # configuration variables to point to the old directories.] # 6. Configure new FlowViewer_Configuration.pm # 7. Stop old FlowMonitor_Collector and FlowMonitor_Grapher # [Upgrading from v4.4 and prior you will be stopping FlowTracker_Collector # and FlowTracker_Grapher.] # 8. Start new FlowMonitor_Collector and FlowMonitor_Grapher # 9. Copy NamedInterfaces_Devices, names file, user logo to new directory # 10. (If upgrading from pre v4.0) Run convert_pre40_filters against existing filters # (ie FlowViewer_SavedFilters) # 11. Use included 'User Relay' scripts if desired (recommended - see below) # # Quick Install # # 1. Untar into cgi-bin subdirectory # # For netflow v5 and older (option): # # 2. Download, install, configure flow-tools # # For IPFIX (e.g., v9 - also handles v5): # # 3. Download, install, configure SiLK (v3.8.0 or newer) and libfixbuf # # For sflow # # 4. From SiLK FAQ: # # "Support for sFlow v5 is available as of SiLK 3.9.0 when you configure # and build SiLK to use v1.6.0 or later of the libfixbuf library." # # For FlowViewer # # 5. Configure FlowViewer_Configuration.pm variables as necessary # 6. Create all necessary directories with proper permissions # 7. Copy FlowViewer.css, FlowViewer.pdf to $reports_directory # 8. Point browser to FV.cgi # # For FlowGrapher # # 9. Install gd (C), GD (Perl), GD::Graph (Perl) GD::Text (Perl) # 10. Configure FlowViewer_Configuration.pm variables as necessary # 11. Point browser to FV.cgi # # For FlowMonitor # # 12. Install RRDtool (at least version 1.4) # 13. Create FlowMonitor_Filter and FlowMonitor_RRDtool directories # 14. Configure FlowViewer_Configuration.pm variables as necessary # 15. Start FlowMonitor_Collector, FlowMonitor_Grapher in background # 16. Point browser to FV.cgi # # For all FlowViewer tools # # 17. Review all FlowViewer directories and files for proper permissions # # Version 4.6 Release Notes # # Emergency fix to FlowMontitor_Collector to fix $no_devices_or_exporters # not setting times (thanks Vladimir Stepanov) - 04/06/2105 # # Version 4.6 fixes local timezone difficulties that were not fixed as # advertised in version 4.5 for FlowGrapher and FlowViewer. Thanks goes # to Randy Feeney. Also note that version 4.6 removes the "$time_zone" # configurable parameter from FlowViewer_Configuration. Timezone is now # exclusively extracted from the system, using the 'date' function. This # version fixes a problem with FlowGrapher not correctly displaying the # smallest flows when requested (e.g., Detail Lines: -100 for smallest # 100 flows.) Fixes improper listing of very old Saved files. # # Version 4.5 Release Notes # # Version 4.5 resolves an unfortunate name clash in commercial space and # renames FlowTracker to FlowMonitor. The situation where SiLK data is # saved in UTC (GMT) time, but the system is left in local time has been # fixed (thanks to Kees Leune.) A new configuration variable # "$silk_compiled_localtime" has been added for the environment where SiLK # has been comipled with the --enable-localtime switch. FlowGrapher_Analyze # has been fixed to handle hyper-links to IPv6 hosts properly. SiLK IPsets # can now be input through the various tool menus. A problem with # multi-word Dashboards and Group creation has been fixed. Corrected # flows/second initiated calculation. Added the ability to bypass the # printing of pulldowns on the bottom service bar. Fixed an error with # filtering on port equal to '0'. Fixed 'Len' field output for some # FlowGrapher reports. New parameter: $ipfix_default_device allows IPFIX # users to pre-select a primary device (e.g., using one sensor only.) # Extended pie-charts to some Printed reports. A new parameter # $site_config_file is added to make it easier to accomodate various # SiLK stored data file structures. # # New FlowViewer_Configuration.pm parameters in v4.5: # # $silk_compiled_localtime - "Y" if SiLK compiled with local timezone # $ipset_directory - Directory where IPsets can be found # $use_bottom_pulldowns - Will exclude pulldowns on bottom of UI # $ipfix_default_device - Controls the default in device_name pulldown # $sensor_config_file - Changed from $sensor_config_directory # $site_config_file - Left blank (= "";) will look in rootdir # # Note: the rename of FlowTracker to FlowMonitor includes default names # for FlowMonitor related directories. The defaults that will previal if # no changes are made are: # # $monitor_directory = "/var/www/html/FlowMonitor"; # $monitor_short = "/FlowMonitor"; # $filter_directory = " ... /FlowMonitor_Files/FlowMonitor_Filters"; # $rrdtool_directory = " ... /FlowMonitor_Files/FlowMonitor_RRDtool"; # # For users who are upgrading, these can be revised back to 'FlowTracker' # (or whatever) with no problem. The alternative is to simply rename the # existing directories. # # Version 4.4 Release Notes # # Version 4.4 introduces two new significant capabilities; multiple # dashboards and FlowGrapher Analysis. The user can now set up # more than one active dashboard with links on every page to get # to each dashboard. Uses include multiple networks, data centers, # or multiple users. The new version also permits users to quickly # de-aggregate FlowGraphs into the largest (3 to 10) source or # destination IP or Port contributors. This is particularly useful in # analyzing peaks in graphs based on "Flows" for DDOS and the like. # The multiple dashboard changes touched most of the scripts. There # is one new script, FlowGrapher_Analyze, and the script # FlowMonitor_AltDashboard is deprecated. The code was cleansed of # confusing 'prorated (_P)' distinctions. Version 4.4 also greatly # improves the user's ability to interface with various SiLK # configurations other than the 'flow-tools-like' one suggested in # earlier documentation. Note: SiLK users must upgrade SiLK to # version 3.8.0 or newer. # # Version 4.3 Release Notes # # Version 4.3 introduces a new FlowViewer report called "Detect # Scanning" which employs scanning detection software in both # flow-tools (flow-dscan) and SiLK (rwscan.) The output from each # report includes links to the individual scanner sources detected. # The link will create a FlowGrapher report for that host. Modified # field checks to permit 32 bit AS entries. However, this is for the # future when SiLK begins to handle the fields. Thanks Veasna Long. # # Version 4.2.2 Release Notes # # Version 4.2.2 is a quick emergency fix with limited changes. An # array (@temp_ports) in FlowViewer_Utilities was not initialized # and was acumulating contents over many FlowMonitor_Collector runs. # This was causing the whole FlowMonitor_Collector run to slow down # gradually over time as this array would have to get sorted for each # SiLK FlowMonitor (this caused no problems for flow-tools only # users.) # # Version 4.2.1 Release Notes # # Version 4.2.1 extends the use of the recently discovered flow-report # option "linear-interpolated-flows-octets-packets" to FlowMonitor # processing (see User's Guide for further discussion.) This provides # significant speed-ups for FlowMonitor_Recreate (above 75%) and # FlowMonitor_Collector (which will now permit many more FlowMonitors.) # The FlowMonitor_Recreate processing for SiLK was remarkably poorly # implemented and this has been corrected introducing unspeakable # performance gains there as well. The same use of SiLK prefiltering # that was introduced for FlowMonitor in v4.2 has been applied to # FlowGrapher for a speed-up there. A new capability is added to work # with different international date formats. The distribution includes a # new capability, FlowViewer_CleanSiLK, to monitor and adjust diskspace # used by IPFIX devices addressing the SiLK deficiency of not having the # valuable flow-capture feature (-E) of active diskspace usage control. # Made the use of prefiltered files or CONCATs non-optional and fixed # SiLK processing of flows and packets for FlowMonitor_Collector. Modest # changes have been made to FlowViewer_CleanFiles, FV.cgi and # flowmonitor_grapher_nonlazy. An error causing packets to be monitored as # flows in some cases was fixed. A new tool, flowmonitor_grapher_recent # will re-graph (nonlazy) only recently created FlowMonitors. # # Version 4.2 Release Notes # # Version 4.2 incorporates the equivalent of "use existing concatenations" # for SiLK based FlowMonitors. FlowViewer takes advantage of previously # concatenated flow-tools files during FlowMonitor_Collector processing # to speed up the whole run. Now it does this for SiLK files as well by # performing rwfilter "INPUT" filtering only once for all FlowMonitors # that are based on the same sensor/class combination. This is a pretty # significant speed-up and will permit the user to have many more # FlowMonitors. For example, our implementation, a combination of # flow-tools and SiLK based data, now processes 250 FlowMonitors in 35 # seconds (prior to the new version this was taking 50 seconds.) The new # version also corrects processing of exporters [M. Donnelly]. Excluded # fields (e.g., protocols=-17) were being accepted for SiLK FlowMonitors # despite SiLK not being able to handle them; this was fixed. The # analyze_netflow_packets tool has been fixed for IPv6 addresses. Four new # date conversion utilities have been added to the 'tools' subdirectory. # FlowMonitor_Collector was modified in version 4.0 to be able to create # FlowMonitors for flows and packets (as well as bits.) This was # inadvertantly dropped in version 4.1 and is restored in this update # to version 4.2 [7/31]. # # Version 4.1 Release Notes # # Version 4.1 includes a new FlowGrapher capability that creates reports # 3 to 4 times faster than previously. The detail lines are a little bit # different. The previous capability is retained offering the user a choice # from the input form interface. The new FlowGrapher report type is # "Aggregated". It makes use of a heretofore missed flow-tools capability # known as the flow-report "linear-interpolated-flows-octets-packets" option # which aggregates flows, octets, or packets into time buckets. This moves # that processing into the compiled "C" code of flow-tools. New FlowMonitor # capabilities are added to monitor flows or packets as well as the previously # available octets. Version 4.1 introduces the ability to maintain different # dashboards for different users (please see the User's Guide for how to do # this. The new version includes a new FV_Relay.cgi script. The new version # fixes a flaw in FlowMonitor_Collector that erroneously monitored protocols, # tcp_flags, and tos_fields when using SiLK (thanks C. Spitzlay.) It also # includes some small fixes like making directory creation a little easier, # fixes removing (and adding) Trackings from the Dashboard and removes some # minor extraneous formatting. Fixed problem with FlowMonitor_Collector # processing of SiLK interface filtering [07/09/13]. Fixed initialization # of @ipfix_devices in FlowViewer_Configuration.pm [Thanks M. Donnelly.] # # Version 4.0 Release Notes # # Version 4.0 is a major upgrade that enables FlowViewer to handle IPFIX # netflow data (i.e., v9, etc.) The User Interface has been completely redone # and now features a Dashboard. Aside from the new collector interface and user # interface, version 4.0 introduces some new capabilities: # # 1. FlowViewer report sorting by column header # 2. Dashboard of thumbnail versions of selected FlowMonitor graphs # 3. Ability to 'recreate' FlowMonitors, starting at a time specified in the # past # # The distribution manifest has changed significantly. # # Preserved Scripts, Files, and Tools: # # FlowViewer.cgi Modified for new user interface. # FlowViewer_Main.cgi Modified for new user interface. # FlowViewer_Relay.cgi No change. # FlowViewer_Save.cgi Significant modification. # FlowGrapher.cgi Modified for new user interface. # FlowGrapher_Main.cgi Modified for new interface. # FlowGrapher_Colors No change. # FlowGrapher_Relay.cgi No change. # FlowGrapher_Sort.cgi Significant modification. # FlowMonitor.cgi Modified for new user interface. # FlowMonitor_Collector Modified to process stored SiLK data. # FlowMonitor_Grapher Modified to update Thumbnails. # FlowMonitor_Group Modified for new user interface. # FlowMonitor_Dumper Modified for new user interface. # FlowMonitor_Relay.cgi No change. # FlowViewer_CleanASCache No change. # FlowViewer_CleanFiles Minor changes. # FlowViewer_CleanHostCache No change. # FlowViewer_Configuration.pm Modifications for SiLK and user interface. # FlowViewer_Utilities.pm Removed filter output processing. # NamedInterfaces_Devices No change. # NamedInterfaces_Exporters No change. # flowcapture_restart No change. # flow-capture-table.conf No change. # flowmonitor_restart No change. # performance_check Parse FlowMonitor logs and report performance # rsync_flows Rsync all of raw flow data to backup host # rsync_monitors Rsync all of Tracking data to backup host # # New Scripts, Files, and Tools # # FlowViewer_Replay.cgi Presents saved FlowViewer reports # FlowViewer_SaveManage.cgi Manages saved reports # FlowViewer_Sort.cgi Sorts FlowViewer reports # FlowViewer_UI.cgi Utilities for creating user interface # FlowGrapher_Replay.cgi Presents saved FlowGrapher reports # FlowMonitor_Dashboard.cgi Manages the Dashboard contents # FlowMonitor_Display.cgi Presents a FlowMonitor # FlowMonitor_DisplayPublic.cgi Presents a FlowMonitor from Public list # FlowMonitor_Management.cgi Manages FlowMonitors (e.g., remove, etc.) # FlowMonitor_Recreate Background process to recreate FlowMonitors # FlowMonitor_Thumbnail Invoked to create a Thumbnail FlowMonitor # FlowViewer.css FlowViewer cascading style sheet # FV_button.png New button link to FlowViewer from front page # FG_button.png New button link to FlowGrapher from front page # FM_button.png New button link to FlowMonitor from front page # convert_pre40_filters Converts old saved filters (pre version 4.0). # flowmonitor_archive_restore Restores archived FlowMonitors gone astray # flowmonitor_grapher_nonlazy Forces a re-graphing of all FlowMonitor graphs # resize_rrdtools Extends RRDtools created prior to 3-Year graph # rwflowpack_start One-line script starts SiLK collector # analyze_netflow_packets Script analyzes TCPDUMP captured netflow data # # Removed Scripts and Files # # FlowViewer_SavedFilters File kept saved filters # # General Notes: # # This is a major upgrade of FlowViewer. The upgrade preserves this # open-source option for netflow analysis in the age of IPFIX. The user # is urged to read through the User's Guide for a better understanding # of installation and configuration. # # Those who upgrade can preserve all previous filters and reports easily. # Saved reports are automatically available in the new version. The only # manual change requires users to run the 'convert_pre40_filters' script # from the command line to move saved filters into the new format. Example: # # host>convert_pre40_filters .../FlowViewer_3.4/FlowViewer_SavedFilters # # With Respect to SiLK: The SiLK tool suite, developed by the NetSA group # at Carnegie Mellon, is excellent software with equally excellent # documentation. Version 3.0 of SiLK together with libfixbuf v1.1.0 are their # entree into IPFIX/v9 netflow capture and analysis supporting IPv6. Initially # they have chosen to limit the number of IPFIX Information Elements (IE) that # the SiLK software will process. They have chosen a set that matches what # flow-tools has provided with the addition of IPv6 data, but sadly with the # exception of autonomous system (AS) elements. I have requested that they add # the AS Elements, but we'll see. They have mentioned a future overhaul # (beyond v3.0) to handle the entire IE space through user configuration. As # of Spring 2013, SiLK v3.x is not fully through the process required to make # the software open-source to the general public but they are proceeding with # getting the approval. It is currently freely available to US Federal # agencies. # # The FlowViewer_Configuration.pm file has changed: # # New parameters (configurable): # # $dashboard_directory = "/var/www/html/FlowViewer_Dashboard"; # $dashboard_short = "/FlowViewer_Dashboard"; # $silk_data_directory = "/data/flows"; # $silk_bin_directory = "/usr/local/bin"; # $sensor_config_directory = "/data/flows"; # @ipfix_devices = ("Router_v9_1","Router_v9_2","Test_6509_v9"); # $sip_prefix_length = "16"; # $dip_prefix_length = "16"; # $silk_all_only = "N"; # $left_title = "Any Title You Like"; # $left_title_link = "http://abc.com/"; # $right_title = "Any Second Title You Like"; # $right_title_link = "http://abc.com/"; # $recreate_cat_length = 1*(60*60); # Time length of concatenated file # $thumbnail_width = 250; # probably should leave this alone # $thumbnail_height = 80; # probably should leave this alone # $filename_color = "#CF7C29"; # $dig_forward = "/usr/bin/dig +time=1 +tries=1 "; # $default_identifier = "DNS"; # "IP" for addresses; "DNS" for names # # Removed Parameters # # $bg_color = "#FFFFFF"; # $text_color = "#000000"; # $link_color = "#000000"; # $vlink_color = "#BF294D"; # $monitors_title = "Your Company Name"; # $user_logo = "Generic_Logo.jpg"; # $user_hyperlink = "http://www.yourcompany.com/"; # # With respect to the "Relay" scripts, many of you may already have resolved # this issue by setting up a generic 'FlowViewer' directory and simply # re-linking it to the new version's directory. I've been told this is proper # :-). It certainly makes good sense. Otherwise the "Relay" approach is best # explained below in Version 3.4 Release Notes. # # Version 3.4 Release Notes # # Update - 8/17/2011 - Fixed FlowViewer bug when requesting time periods just # shy of midnight. This had already been fixed in FlowGrapher. Modifications # were made to FlowViewer_Main.cgi. # # Update - 5/20/2011 - Modifications have been made to FlowGrapher_Main.cgi to # fix a problem caused by the new speed-up processing. The speed-up was not # accounting for Daylight Savings considerations. # # It's been awhile, so version 3.4 will fix a myriad of little problems which # I mostly can't remember. The primary new capabilities include: # # 1. In most cases, the user may now switch the device without losing entered # filter criteria # 2. The different tool logos now provide a link to the Saved Reports page # 3. Users can now provide a meaningful name for saved FlowViewer and # FlowGrapher reports # 4. Fixes to an end-of-year problem have resulted in a 8% speed up of # FlowGrapher in general # 5. Users can select to limit FlowGrapher stats to no-zero data points, # if desired # 6. Fixed problems with sorting # 7. Corrected the graphing by 'flows' (was graphing 'flags' :-) # 8. Can now provide up to 20 source or destination IP address/address ranges # 9. Can now exclude specified IP addresses from a larger included address # range # # New Scripts and Files: # # FlowGrapherM.png New logo link points to Saved reports web page # FlowGrapherS.png Revised logo link for naming of Saved Reports # FlowViewerM.png New logo link points to Saved reports web page # FlowViewerS.png Revised logo link for naming of Saved Reports # FlowMonitorM.png New logo link points to Saved reports web page # flowcapture_restart Renamed script for restarting flow-captures # flowmonitor_restart New script for re-starting FlowMonitor_Collector # # General Notes: # # Remember to copy into the new directory (e.g., # /usr/lib/cgi-bin/FlowViewer_3.4) user logos, names file, as_names, # NamedInterfaces_Devices, NamedInterface_Exporters, FlowViewer_SavedFilters, # etc., from the old cgi-bin directory. # # The simplest way to transition to the new version is to leave all # FlowViewer_Configuration.pm settings alone except: # # $reports_directory = "/var/www/FlowViewer_3.4"; # $reports_short = "/FlowViewer_3.4"; # $graphs_directory = "/var/www/FlowGrapher_3.4"; # $graphs_short = "/FlowGrapher_3.4"; # $monitor_directory = "/var/www/FlowMonitor_3.4"; # $monitor_short = "/FlowMonitor_3.4"; # $cgi_bin_directory = "/usr/lib/cgi-bin/FlowViewer_3.4"; # $cgi_bin_short = "/cgi-bin/FlowViewer_3.4"; # $work_directory = "/usr/lib/cgi-bin/FlowViewer_3.4/Flow_Working"; # $names_directory = "/usr/lib/cgi-bin/FlowViewer_3.4"; # $log_directory = "/usr/lib/cgi-bin/FlowViewer_3.4" # # The following can remain the same (or copy contents to the new directory): # # $save_directory = "/var/www/FlowViewer_Saves"; # $save_short = "/FlowViewer_Saves"; # $filter_directory = ".../FlowMonitor_Files/FlowMonitor_Filters"; # $rrdtool_directory = ".../FlowMonitor_Files/FlowMonitor_RRDtool"; # # If this is an upgrade for you (e.g., from v3.3.1) I recommend using the # FlowViewer_Relay.cgi, FlowGrapher_Relay.cgi, and the FlowMonitor_Relay.cgi # scripts to alert users to the new version with links and a reminder to # change their bookmarks. In each of the relay scripts tailor the following # line to your environment (point to the new FlowViewer_Configuration.pm file): # # require "/usr/lib/cgi-bin/FlowViewer_3.4/FlowViewer_Configuration.pm"; # # ... then, in your old cgi-bin directory (e.g., FlowViewer_3.3.1), copy the # following: # # cp FlowViewer_Relay.cgi FlowViewer.cgi # cp FlowGrapher_Relay.cgi FlowGrapher.cgi # cp FlowMonitor_Relay.cgi FlowMonitor.cgi # # Now, when users go to their book-marked FlowViewer web page, they will be # directed to the new one. FlowMonitor_Relay.cgi is particularly important # this is an upgrade it copies over archived FlowMonitors which would be a # bit tedious to copy by hand. # # The rsync_flows and rsync_monitors scripts are useful for easily backing # up all raw netflow data and FlowMonitor state information (Filters and # RRDtool databases.) The FlowViewer_CleanFiles script is useful for deleting # aging files that are not necessary anymore. I run it out of 'cron' once a # day. # # The performance_check script can be used from the command line to keep track # of how well your implementation is performing. I run it against my # FlowMonitor_Collector.log file to see how things are going. Here at the # NASA Earth Observing System network I have over 200 FlowMonitors and # they complete in an average of 44 seconds. FlowMonitor_Collector runs every # five minutes and I watch for runs that take longer than five minutes. # Even in those situations, however, FlowMonitor_Collector seems to continue # on with no real visible effects. # # Version 3.3 Release Notes # # ### Version 3.3.1 fixes a FlowMonitor_Collector bug when using exporters # Also fixes problem for users without devices at all. If you are not # using any devices (or exporters) you will now have to set: # $no_devices_or_exporters = "Y"; Fixes FlowGrapher sorting of host # names. This version fixes the problem of links to Trackings embedded # in Group graphs not lining up properly. Fixes problem with # FlowMonitor_Grapher not printing out named interfaces. Fixes # FlowGrapher graph and output to now have exporter name. Fixes # problem with end-of-month graphs (missing days_in_month.) # # New Capabilities # # 1. Some devices will now have 'named interfaces' (thanks C. Kishimoto) # 2. The user can now save filters of interest and recall them later # 3. Data can now be analyzed by Exporter ID (in addition to device name) # 4. Users can now set thresholds on FlowMonitors, and be alerted # 5. Users can now sort FlowGrapher output based on column type # 6. FlowViewer now provides Pie Charts # 7. Capability added to apply a Sampling Multiplier to output # 8. FlowMonitors now have a '3 year' graph # 9. The user can now generate text listings of FlowMonitor output # 10. Filtering on next-hop has been added # 11. Logging has been made more flexible (e.g., less data) # 12. Preserve latest three notations (was keeping first three) # 13. Can now specify and display time-zones # 14. A hook has been provided for a User Logo with link out of FlowViewer # 15. New file cleanup scripts have been added # 16. Unit Conversion capability has been added (thanks C. Kishimoto) # 17. Can now graph Flows, Packets as well as Octets (thanks E.Lautenschlaeger) # 18. Improved AS name resolution (thanks S. Cardus) # 19. New saved_directory for storing saved Reports and Graphs. # # New Scripts and Files: # # FlowGrapher_Sort.cgi Sorts FlowGrapher Detail Lines by column # FlowMonitor_Dumper.cgi Invoked by link in Trackings, prints text values # FlowViewer_Save.png New logo with links for saving filters, reports # Flowgrapher_Save.png New logo with links for saving filters, reports # FlowViewer_CleanASCache Tool used to remove obsolete AS name resolutions # FlowViewer_CleanFiles Tool used to remove old intermediate files # FlowViewer_CleanHostCache Tool used to remove obsolete host name resolutions # FlowViewer_Relay.cgi Optional: points users to new version (see Notes) # FlowGrapher_Relay.cgi Optional: points users to new version (see Notes) # FlowMonitor_Relay.cgi Optional: points users to new version (see Notes) # flowcap Optional start-up script for flow-tools and # FlowMonitor # # NamedInterfaces_Devices Holds interface names for SNMP device indices # NamedInterfaces_Exporters Holds interface names for SNMP index (exporters) # FlowViewer_SavedFilters Created during processing to hold saved filters # # Notes: # # Many thanks to Carles Kishimoto, Eric Lautenschlaeger, and Sean Cardus for # their ideas and code contributions. Thanks to Dario La Guardia for pointing # out a graphing problem that turned out to be a rounding error in FlowGrapher. # Credit to Peter Hoffswell for the idea of linking the tools. # # There are no new software dependencies with FlowViewer version 3.3.1, however # Named Interfaces now requires Javascript in the browser to operate. # # If you are having trouble with creating Tracking Groups, you may have a # problem with the installation of RRDs.pm. This needs to be placed in a # library that Perl includes in it's @INC array. For a fix, see the FlowViewer # FAQ on the web site. # # Using the 'Relay' scripts (these are optional) # # If you have other users and you would like to point them to the new version, # copy the included 'Relay' scripts over the old FlowViewer.cgi, # FlowGrapher.cgi and FlowMonitor.cgi scripts in the last version's directory. # # For example: # # In the old directory /htp/cgi-bin/FlowViewer_3.2: # # mv FlowViewer_Relay.cgi FlowViewer.cgi # mv FlowGrapher_Relay.cgi FlowGrapher.cgi # mv FlowMonitor_Relay.cgi FlowMonitor.cgi # # Then, when the user goes to the old FlowViewer, he will be provided a link # to the new FlowViewer, and asked to change his bookmarks. # # # Setting up crontab file for cleaning FlowViewer files: # # min hr dom moy dow command # # 5 0 * * * .../FlowViewer_3.4/FlowViewer_CleanFiles # > .../FlowViewer_3.4/cleanup.log # 2 >> .../FlowViewer_3.4/cleanup.log # # The file cleanup is controlled by parameters in FlowViewer_Configuration.pm: # # $remove_workfiles_time = 86400; # $remove_graphfiles_time = 7*86400; # $remove_reportfiles_time = 7*86400; # # Remember, whichever crontab account this is started from must have adequate # permissions to remove files created by the web process owner (e.g., apache.) # # Documentation # # The FlowViewer User's Guide is available on the FlowViewer Website: # # http://ensight.eos.nasa.gov/FlowViewer # # Dependencies # # - FlowGrapher requires the Perl GD and GD:Graph packages # gd package. Thomas Boutrell's graphics package written in 'C' # GD package: http://search.cpan.org/~lds/GD-2.30/ # GD::Graph package: http://search.cpan.org/~mverb/GDGraph-1.43/ # - FlowViewer.cgi requires the GDBM or NDBM capability in Perl # - FlowMonitor requires RRDtool (at least version 1.2.12) # RRDtool: http://oss.oetiker.ch/rrdtool # # Contents # # FlowViewer_Configuration.pm # # This file contains parameters that configure and control the # FlowViewer, FlowGrapher, and FlowMonitor environments. This package # should remain in the same directory that the CGI scripts are in. # # FlowViewer_Utilities.pm # # This file contains processing used by multiple programs (e.g., to # create the Report Parameters output for each tool, and other utilities # (e.g., 'epoch_to_date' which converts between typical date formats # and 'seconds since 1972') that are invoked by other scripts. This # package should be placed in the same directory as the CGI scripts. # # FlowViewer.cgi # # This script produces the web page which provides the user the form # for entering analysis selection criteria for FlowViewer. Version 3.0 # reorganized the processing. FlowViewer.cgi is now the old # create_FlowViewer_webpage. This change permits the input date and time # to be updated with each invocation. # # FlowViewer_Main.cgi # # This script responds when the user completes the selection criteria # form and submits the 'Generate Report' command. The script creates a # flow-tools filter file based on the selection criteria. Based on the # input time period, the script concatenates the relevant flow-tools # data files for the selected device. The location of the flow-tools # raw data files is specified via the 'flow_data_directory' parameter. # The script then invokes the selected statistics/print report flow-tools # program and reformats the output into HTML. An option is available in # FlowViewer_Configuration to have this script use the NDBM capability # (for caching resolved host names) instead of the default GDBM # capability for users whose Perl distribution does not have GDBM. # # FlowGrapher.cgi # # This script produces the web page which provides the user the form # for entering analysis selection criteria for FlowGrapher. Version 3.0 # reorganized the processing. FlowGrapher.cgi is now the old # create_FlowGrapher. This change permits the input date and time # to be updated with each invocation. # # FlowGrapher_Main.cgi # # This script responds when the user completes the FlowGrapher selection # criteria form and submits the 'Generate Graph' command. The script # creates intermediate processing files exactly like FlowViewer above. # The script then parses intermediate output, fills time buckets, and # generates a graphic image. Textual output accompanies the graph. An # option is available in FlowViewer_Configuration to have this script use # the NDBM capability (for caching resolved host names) instead of the # default GDBM capability for users whose Perl distribution does not have # GDBM. # # FlowGrapher_Sort.cgi # # This script is invoked when the user clicks on a column header for the # Detail Lines of a FlowGrapher report. The textual data on the page is # sorted and re-presented. # # FlowGrapher_Colors # # This file contains a translation between textual color names and their # RGB value counterparts. # # FlowMonitor.cgi # # This script produces the web page which provides the user the form # for entering analysis selection criteria for FlowMonitor. The script # also provides the user with the ability to review, revise, or remove # existing monitors. FlowMonitor was new in version 3.0. # # FlowMonitor_Main.cgi # # This script responds when the user completes the FlowMonitor selection # criteria form and submits the 'Establish Tracking' command. The script # responds to the users desire to create, remove, or revise a monitor. # # FlowMonitor_Group.cgi # # This script controls the building of groups from existing Individual # FlowMonitors. The user has the ability with FlowMonitor v3.2 to create # 'groups' from pre-defined Individual monitors. A Group Tracking has no # RRD database associated with it, but simply creates a multifaceted graph # from several existing monitors. The Group 'merges' the Individual graphs # onto a single graph. # # FlowMonitor_Dumper.cgi # # This script is invoked when the user clicks on a link within the # FlowMonitor graph labeled '[List values]'. The script dumps the # RRDtool contents onto a web page. # # FlowMonitor_Collector # # The script is started once by the user and placed in the 'background'. # The script will execute and then sleep for the duration of a five minute # period, essentially running every five minutes. For each existing monitor, # the script applies the associated filter to the flow data and extracts the # amount that occured during a 5-minute window approximately 30 miuntes # earlier. This is to permit long-running flows to have been exported and # available to the collector. The script then divides the total bits by # 300 seconds to get an average bits-per-second rate during the period. # The data point is then provided to RRDtool for storage. The script # should be started out of the cgi-bin directory. # # FlowMonitor_Grapher # # The script is started once by the user and placed in the 'background'. # The script will execute and then sleep for the duration of a five minute # period, essentially running every five minutes. The script runs the # RRDtool graph function for each existing monitor. Daily, weekly, # monthly, and yearly graphs are updated with the latest information. The # script creates an html page for each monitor that includes the filter # parameters and the four graphs. The script also creates an overall web # page ($monitor_webpage) that provides links to all active monitor pages. # The script should be started out of the cgi-bin directory. # # FlowViewerM.png # # The FlowViewerM logo with links. Leave this file in the 'cgi-bin_directory', # the FlowViewerM.cgi script will place a copy of the image in # 'html_directory'. This image contains mapped links to FlowGrapher and # FlowMonitor such that those input pages are pre-loaded with the filter # criteria from FlowViewer. # # FlowViewerS.png # # The FlowViewerS logo with links. Leave this file in the # 'cgi-bin_directory', the FlowViewerS.cgi script will place a copy of the # image in 'reports_directory'. This image contains mapped links to the other # tools as well as links for saving the filter used or the report generated. # # FlowGrapherM.png # # The FlowGrapherM logo with links. Leave this file in the 'cgi-bin_directory', # the FlowGrapherM.cgi script will place a copy of the image in # 'graphs_directory'. This image contains mapped links to FlowViewer and # FlowMonitor such that those input pages are pre-loaded with the filter # criteria from FlowGrapher. # # FlowGrapherS.png # # The FlowGrapherS logo with links. Leave this file in the # 'cgi-bin_directory', the FlowGrapherS.cgi script will place a copy of the # image in 'graphs_directory'. This image contains mapped links to the other # tools as well as links for saving the filter used or the report generated. # # FlowMonitorM.png # # The FlowMonitor logo with links. Leave this file in the 'cgi-bin_directory', # the FlowMonitorM.cgi script will place a copy of the image in # 'monitor_directory'. This image contains mapped links to FlowViewer and # FlowGrapher such that those input pages are pre-loaded with the filter # criteria from FlowMonitor. # # FlowViewer_Save.cgi # # This script moves temporary save files into a permanent residence # as defined by either the 'reports_directory' or 'graphs_directory' # environment variables. # # FlowViewer_CleanFiles # # A utility for cleaning out temporary files that have been left # over from debugging (e.g. $debug_files = 'Y'). Files older than # the following configurable parameters are removed: # # $remove_workfiles_time = 86400; # $remove_graphfiles_time = 7*86400; # $remove_reportfiles_time = 7*86400 # # See above for crontab settings for running this automatically. # # FlowViewer_CleanASCache # # A utility for cleaning out from the AS resolving cache ($as_file) a # resolved AS name that is no longer valid. # # FlowViewer_CleanHostCache # # A utility for cleaning out from the DNS resolving cache ($names_file) # a resolved host name that is no longer valid. # # FlowViewer_Relay.cgi, FlowGrapher_Relay.cgi, FlowMonitor_Relay.cgi # # Short scripts that refer users from version 3.3.1 to version 3.4. This # keeps you from having to notify users to go to a different web site. # # flowcapture_restart # # A shell script used for starting up and restarting flow-captures. Tailor # this for your environment. # # flowmonitor_restart # # A shell script used for starting up and restarting FlowMonitor_Collector # and FlowMonitor_Grapher. Tailor this for your environment. # # Generic_Logo.jpg # # This image is to be replaced by your own image that can point back to # anywhere (e.g., your overarching NMS system.) # # NamedInterfaces_Devices # # This file is used for SNMP index to named interface translation. This file # provides translation when you are saving data by individual devices. Examples # are provided. # # NamedInterfaces_Exporters # # This file is used for SNMP index to named interface translation. This file # provides translation when you are saving data into a single directory but # for (possibly) multiple devices differentiated by EXPORTER_ID. Examples are # provided. # # Configuration parameters # # The FlowViewer, FlowGrapher, and FlowMonitor scripts all use parameters # in the FlowViewer_Configuration.pm file to control the environment that # they run in. Here is a brief explanation of some of the relevant # parameters: # # $ENV(PATH) - modify as appropriate for your installation # $FlowViewer_server - IP address of server hosting this software # $FlowViewer_service - Either HTTP (port 80) or HTTPS (port 443) # $reports_directory - Directory to hold saved FlowViewer reports # $reports_short - Reports directory beginning from web server default # $graphs_directory - Directory to hold saved FlowGrapher reports # $graphs_short - Graphs directory beginning from web server default # $monitor_directory - Directory to hold FlowMonitor monitors # $monitor_short - Monitor directory beginning from web server default # $filter_directory - Directory in which to keep FlowMonitor filter files # $rrdtool_directory - Directory in which to keep FlowMonitor RRDtool files # $cgi_bin_directory - Directory which holds cgi scripts # $cgi_bin_short - cgi-bin directory beginning from web server default # $flow_data_directory - Directory that holds all flow-tools data files # $exporter_directory - Directory where netflow stored for multiple exporters # $flow_bin_directory - Directory where all flow-tools reside # $rrdtool_bin_directory - Location of RRDtool programs # $work_directory - Directory to store intermediate files # $names_directory - Directory to save permanent 'names' file # $flow_capture_interval - Interval beyond end point to capture all flows # $flow_file_length - Length (in seconds) of each of your flow files # $devices - List of device names exporting netflow (see #4 below) # $no_devices_or_exporters - Set to "Y" if you have no devices and no exporters # $N - Used to control directory organization (see #5 below) # $dig - Location of DNS utility 'dig' (set to nslookup if required) # $actives_webpage - Name of HTML file which will list your Trackings # $monitors_title - Title for HTML page which lists Trackings # $user_logo - Filename of image used for your logo # $user_hyperlink - Link associated with $user_logo # $use_even_hours - Will start default time periods at the top of the hour # $use_NDBM - Some Perls don't have GDBM (default), but do have NDBM # $start_offset - Offset from current time for beginning pre-loaded time period # $end_offset - Offset from current time for end of pre-loaded time period # $flow_capture_interval - Minutes beyond end period for collecting all flows # $flow_file_length - Size (minutes) of each flow-tools flow file (default = 15) # $labels_in_titles - Whether to print FlowMonitor title in the graph itself # $debug_files - If Yes, will not remove intermediate files # $collection_offset - Seconds into past to begin collection period # $collection_period - Period to examine for FlowMonitor (keep at 5 minutes!) # $use_existing_concats - DEPRECATED. Re-use concatenations (much faster) # $rrd_dir_perms - (And others) UNIX directory or file permissions # # The rest of this file contains basic parameters such as colors, etc. Each # parameter is dicussed in more detail in the User's Guide. # # Additional Considerations # # 1. Directory permissions for the subdirectories created for the # 'htdocs', 'work', 'names', 'cgi-bin' (e.g., FlowMonitor_Filter, # FlowMonitor_RRDtool) directories must permit the owner of the web # server process (e.g., apache) to write into these directories. # The directories may have been created by a different user. Version 3.0 # introduced the use of $dir_perms. There are several of these included # in FlowViewer_Configuration.pm. These are the permissions that the # scripts will set your various FlowViewer files and directories to. # They default to '0777' which permits the open interaction between the # web server process owner and the FlowMonitor background process owner. # You may want to adjust these permissions differently according # to your security policies, and whether you use the same or different # accounts for the web and background processes. # # 2. FlowViewer and FlowGrapher offer the ability to save interesting # reports. To do this, the scripts save a temporary copy of every report # in advance of the user electing to save it permanently. These # intermediate files will accumulate in the 'work' directory specified # in the FlowViewer_Configuration file. These files could be removed # daily via a cron script to prevent unecessary use of disk space. When # the user elects to save a report, it is copied into either the # 'reports_directory', or the 'graphs_directory' depending on which # function he is running. See discussion of FlowViewer_CleanFiles above. # # 3. FlowViewer and FlowGrapher offer the ability to resolve NetFlow IP # addresses into their host names on the fly. This process is speeded # up by caching names into a 'names' file which resides in the directory # specified by the 'names_directory' parameter. This parameter defaults # to /tmp, but this may not be the best directory for you since it will # disappear with a reboot. As you are building up your 'names' file # with early runs, you will notice the speed increase dramatically # as the 'names' file is used more. The process of resolving names is # the primary reason for slower overall FlowViewer performance. You # should preferably use the GDBM array database which is fastest. # However, not all Perl distributions support GDBM but most do support # NDBM. The '$use_NDBM' flag in FlowViewer_Configuration.pm will # cause the FlowViewer_Main and FlowGrapher_Main scripts to use NDBM. # # 4. The FlowViewer and FlowGrapher reporting features use a flow-tools # data directory layout that has a particular device at the top. A # typical flow-tools directory looks like: # # /flows/router_1/2005/2005-07/2005-07-04 # # The device name (router_1) is obtained from an array called 'devices' # in the FlowViewer_Configuration.pm file. Populate this array with your # device names. If your flow-data file structure does not include a # device name, for example you are collecting only from one device, set # the @devices array to empty (i.e., @devices = ("");) On the web page # you can ignore the Devices pulldown selection. # # As of version 3.3, users may now apply FlowViewer to directories that # collect from multiple sources, differentiated by EXPORTER_ID. If you # are using this method (i.e., all flow-captures going into a single # directory), simply set the $exporter_directory parameter to the # directory that is set up to store the flow-data files. # # 5. Different organizations store captured netflow data differently # according to the 'N" setting on the flow-capture statement. However, # there is a bug in the flow-tools documentation such that the default # value is truly '3' and not '0' as indicated. I have set $N = 3 to # reflect the more common setting. The directory structure associated # with $N = 3 is shown below: # # /flows/router_1/2005/2005-07/2005-07-04 # # If you are not seeing output, please check this setting. # # 6. Version 3.2 introduces Groups. Intermediate RRDtool databases are # created on the fly in order to create a temporary FlowMonitor graph that # shows the user how the final graph will look. It uses the Perl RRDs.pm # RRDtool module to speed this up. Make sure your RRDtool distribution # has a compatible RRDs.pm module. # # Change Log # # Version 3.4 - March 17, 2011 # # See Version 3.4 Release Notes above # # Version 3.3 # # See Version 3.3 Release Notes above # # Version 3.2 # # Version 3.2 introduces Group monitors which are simply a monitor graph # made up from the merging of several predefined Individual monitors onto # one graph. There are no permanent RRDtool databases associated with a Group. # In the construction of a group however, temporary RRDtool databases are # created to simulate how the Group will eventually look. The new script # FlowMonitor_Group uses RRDs.pm (comes with RRDtool) to generate these # transient databases quickly. Please ensure that your RRDs.pm module is # compatible with your RRDtool distribution (this should normally be the # case - but if you see "ERR: can't handle RRD file version 0003" in # DEBUG_GRAPHER, you'll need to upgrade your RRDs.pm. # # This version also includes a 'speed-up' for FlowMonitor_Collector which # now concatenates once for each device. This is controlled by # $use_existing_concats, which defaults to "Y". # # Version 3.2 will continue to work happily along with earlier version # exiting Filter and RRDtool files. There are no new FlowViewer_Configuration # parameters of consequence. There are now two types of monitors; Individual # and Group. The FlowMonitor input screen will default to Individual which is # the same as the existing monitors. # # Woj Kozicki has contributed an Autonomous System (AS) resolving capability # and it is included in v 3.2. # # New FlowViewer, FlowGrapher, and FlowMonitor logos have been developed for # version 3.2. These new logos provide embedded links to the other tools so that # the user can switch between them easily and retain input parameters. # # Version 3.1 # # 1. Added MIN, MAX, AVG, 95th PCT to FlowGrapher # 2. Added ability to 'archive' monitors # 3. Added ability to enter port ranges separated by a colon (:) # 4. Can now use any mask length for networks (1 - 32) # 5. Added RRDgraph 'lazy-mode' option to speed up graphing # 6. Upgraded FlowViewer/Grapher ability to go back more than 30 days # 7. Added the ability to configure file permissions # 8. Improved speed of FlowGrapher for larger values of 'detail lines' # 9. Fixed $rrdtool_bin-directory variable name # 10. Added ability to retain intermediate files for debugging # 11. Sorted list of Active Trackings # 12. Fixed bug where non-zero 'cutoof lines' would supress some reports # 13. Fixed FlowViewer rate output to calculate average from all flows # 14. Fixed FlowMonitor_Collector log output (to collect_period_average) # # Version 3.0 # # 1. Major new addition of FlowMonitor # 2. Reorganized scripts so that the date and time fields are updated # with each invocation # 3. Moved common code (e.g., filter creation) to FlowViewer_Utilities.pm # 4. Improved Report Parameters output formatting # 5. Provided host names capability for FlowGrapher (thanks Mark Foster) # 6. Introduced debug and logging capabilities # 7. Merged GDBM/NDBM into a single script (thanks Ed Ravin) # # Version 2.3 # # 1. Modified FlowGrapher record processing to not call 'timelocal' for # epoch times. Other speed improvements. Result: up to 10 times faster. # 2. FlowGrapher error leaving spikes is fixed (thanks Mark Foster) # 3. Bug with concatenation when $N=0 fixed (thanks Dave Faught) # # Version 2.2 # # 1. Added flow_select parameter to control which flows are considered # with respect to the specified time period # 2. Removed Easterm Time (ET) notation. All times are system local # # Version 2.1 # # 1. Fixed concatenation. Needs to start one flow file length before start time # 2. Fixed end-of-year problem in FlowGrapher # 3. Small problem for time requests that end just before midnight # # Version 2.0 # # 1. Used pipe (|) instead of reading intermediate files (thanks Woj Kozicki!) # 2. Introduced configurable variable $N specifies flow-directory nesting levels # 3. Reduced default value of variable $flow_capture_interval to 1800 # 4. Created FlowViewer_NDBM.cgi for users whose Perl does not have GDBM # 5. Created configurable 'work_directory' separate from cgi_bin_directory # 6. Sped up concatenation for requests that cross day boundaries # 7. Added filter fields: Protocol, TOS Field, TCP Flags # 8. Added some more syntax checking # 9. Added FlowGrapher capability (requires GD for Perl) # # Version 1.0 (Original) # # # Vital Assistance # # Special thanks to those FlowViewer users who provided feedback and valuable # suggestions, including Sejin Ahn, Mark Boolootian, Bogdan Ghita, Woj Kozicki, # Ed Arvin, Alex Shepherd, Mike Smith, Scott Wingfield, Vali Magdalinoiu, Ed # ravin, Eric Lautenschlaeger, Sean Cardus, Carles Kishimoto, Shigeki Taniguchi, # Dave Faught, Peter Hofwell, Dario La Guardia, Mike Stowe, Chris Spitzlay and # Mike Donnelly. Big thanks to fellow toiler in the NASA vineyard Mark Foster # for some detailed testing, excellent suggestions, and code to go along with it. # Thanks from all of us to NASA whose unending support of innovation in all fields # has resulted in this toolset. # # Bugs, recommendations # # If you need help installing, have a question, discover a bug, or have a # recommendation, please send an email to: # # Joe Loiacono # jloiacon@csc.com # # FlowViewer is being developed at NASA by a contractor in the employ of the # United States Federal Government in the course of his official duties. # Pursuant to Title 17, Section 105 of the United States Code, this software is # not subject to copyright protection and is in the public domain. FlowViewer is # an experimental system. NASA assumes no responsibility whatsoever for its use # by other parties, and makes no guarantees, expressed or implied, about its # quality, reliability, or any other characteristic.
Source: README, updated 2015-04-06

Thanks for helping keep SourceForge clean.

Screenshot instructions:
Windows
Mac
Red Hat Linux   Ubuntu

Click URL instructions:
Right-click on ad, choose "Copy Link", then paste here →
(This may not be possible with some types of ads)

More information about our ad policies
X

Briefly describe the problem (required):

Upload screenshot of ad (required):
Select a file, or drag & drop file here.

Please provide the ad click URL, if possible:

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

No, thanks