Home

Joe Loiacono

FlowViewer

Powered by flow-tools and SiLK
Advancing Network Traffic Situational Awareness
FAQ

FlowViewer provides a dynamic web front-end to two powerful open-source netflow data collector and analyzers, Mark Fullmer’s flow-tools suite and the Carnegie Mellon NetSA group’s netflow data capture/analyzer, SiLK. The inclusion of the underlying SiLK tool set enables existing FlowViewer users to continue to use the tool with the newer IPFIX netflow data protocol.

FlowViewer has been developed for NASA’s Earth Sciences Data and Information System (ESDIS) networks, and credit goes to NASA for their usual outstanding support of innovation.


FlowViewer Main page


FlowViewer provides users with the ability to quickly extract network management information of interest from voluminous quantities of stored netflow data. The user can configure a [Dashboard] of continuously updating FlowTrackings to maintain a situational awareness of his organization's network traffic. FlowViewer consists of three primary tools: FlowViewer, FlowGrapher and FlowTracker. The user is able to filter data (inclusion or exclusion) by device, IP address range, port, router interface, autonomous system (AS), specified time interval, protocols, TOS field, TCP flags, exporter, and next-hop. All generated reports and filters can be saved for future application. The user can switch between the tools preserving the previously specified filter. FlowViewer makes flow data analysis and tracking quick and easy.

[FlowViewer] enables the user to create text based reports from filtered netflow data. Several different reporting formats are provided. Each of these reports can be sorted by column heading.

[FlowGrapher] enables the user to graph the bandwidth used by a filtered subset of netflow data during a specified time period. Resulting reports include the graph and a textual listing of the largest flows.

[FlowTracker] enables the user to maintain a long-term history of a particular traffic subset. FlowTrackings consist of five graphs of traffic over suceesive longer time periods: Daily, Weekly, Monthly, Yearly, and Last 3 Years.


The user must install and configure either flow-tools, or SiLK. Users already running FlowViewer with flow-tools may opt to install SiLK in parallel to handle IPFIX exporters while leaving older exporters in place. The ability to collect and analyze IPFIX data requires SiLK (download SiLK) now at version 3.7.1. FlowViewer v4.0 continues to work with flow-tools for pre-IPFIX versions of netflow.

The FlowViewer graphing and tracking capabilities make use of such intrepid open source software as Thomas Boutrell’s gd, Lincoln Stein's GD, Martien Verbruggen's GD::Graph, and Tobias Oetiker’s RRDtool packages.

For more information including software requirements and installation instructions, please review the FAQ, User's Guide or the README file. Or, contact me directly at jloiacon@csc.com. For somewhat larger images, please see the Screenshots.


Related

Wiki: Dashboard
Wiki: FlowGrapher
Wiki: FlowTracker
Wiki: FlowViewer

Discussion

  • Junior

    Junior - 2013-06-12

    I see the requirement for Silk V3.0. I recently got Silk v2.5 up and running. Is it a futile effort to have this version of FlowViewer work with Silk v2.5? Reduced functionality, or simply won't work? Can someone please elaborate on this, or give some idea when Silk v3.0 will be available for open release? Thank You!

     
    • Joe Loiacono

      Joe Loiacono - 2013-06-12

      Yes - I tried v2.5 myself, but the limitations are (as I recall) that flow start and end times were not recoverable from netflow v9 and this wreaks havoc on FlowGrapher and FlowTracker. I can't recall whether the FlowViewer tool worked. I didn't try v2.5 on netflow v5 - probably because flow-tools works just fine there - but you might try it as it will be good prep for moving to v3.5 whenever.

      Here's the (unfortunate) latest on SiLK:

      "Due to changes in the oversight of the SEI that are outside of our control, major new releases of NetSA software are required to go through release review by the Office of the Secretary of Defense (OSD) before the software may be given to anyone who is not a federal government employee. Unfortunately, new releases of SiLK have been stuck in this process for a long time, and currently there is no estimate as to when this review will be completed."

       
      • Junior

        Junior - 2013-06-13

        Thanks for your timely reply. That's what I was afraid of.

        I'm just starting to gather some experience with handling netflow data. My ASA firewall seems to be limited to Netflow v9 so I fell into Silk to handle that detail, saw the connection to Flowviewer and was hoping to implement, learn, and have that capability. I suppose there is still lots to learn, and useful data to be had, with Silk 2.5 though.

        Ultimately, I'm pretty agnostic on what tool to use, mostly have been looking for open source (free) tools and chasing various ideas. Have tried nfdump & nfsen a bit. Mostly just collecting data at this point. Always looking for ideas.

        No need to support IPv6.

         
        Last edit: Junior 2013-06-13
    • Joe Loiacono

      Joe Loiacono - 2013-06-21

      But wait! No sooner do we bemoan the delay of the general public release of SiLK and presto: it's available (as of June 20, 2013)!

      Download here: SiLK

       
  • Luke Cleverley

    Luke Cleverley - 2016-09-19

    Joe
    With collector & monitor running I get the following periodic error messages:

    /usr/local/www/apache24/cgi-bin/FlowViewer4.6/FlowMonitorFiles/FlowMonitorRRDtool/I.rrd
    /usr/local/www/apache24/cgi-bin/FlowViewer4.6/FlowMonitorFiles/FlowMonitorRRDtool/INS.rrd

    None of my monitors are called I.rrd or INS.rrd...would you have any pointers
    Regards Luke

     
  • Martyn Noss

    Martyn Noss - 2016-10-25

    Hi Joe,

    Great tool!

    Question: My local Flowviewer box is running in UTC. Among other netflow sources, I've recently started rsyncing a flow-tools data repository from a box in Colorado that happens to be running in MDT (UTC-6). Not a typical configuration, I realize. Does this timezone discrepancy create an insurmountable problem for Flowviewer?

    Flowviewer and Flowgrapher reports pulling data from the Colorado device look reasonable (although I haven't done a detailed analysis on the actual numbers), but Flowmonitor/RRD seem not to be picking up any of the data between hours 00 and 06 UTC, resulting in obvious gaps in the daily and weekly graphs. On the theory that RRD was looking only in the flow-tools daily data directory where it expects to find the data it's looking for, and consequently not finding it for the first 6 hours of each day due to the TZ lag, I wrote a script to copy flow-tools data files for hours 18-24 of the current MDT day into the following day's directory (which I create as part of the script). The script runs immediately following completion of each rsync. This file copy seems to have more-or-less solved the Flowmonitor/RRD output problem (i.e., daily and weekly graphs now look reasonable), but I consider this strategy to be only a diagnostic exercise -- not a good operational solution.

    I see in the User's Guide that FV v4.6 uses the system TZ rather than setting TZ based on configuration files. Seems reasonable, but in my case it appears to mean that Flowmonitor will necessarily mis-interpret my Colorado data (or any non-UTC data, presumably).

    Any advice on how I might tackle this problem (short of asking Colorado to start using UTC)? Thanks!

     
    • Joe Loiacono

      Joe Loiacono - 2016-10-26

      Hey Marty,

      Good to hear from you. This could be a little tricky. First, I believe
      anything that FlowViewer presents for Colorado will be off by 6 hours,
      whether FlowGrapher or FlowMonitor.

      In version 4.6 FlowViewer was updated to be able to work with SiLK's
      timezone handling - but if I recall, this was to be able to better
      accommodate those working completely in local time, and not to accommodate
      multiple timezones in one deployment.

      You might look at the SiLK toolset here and see if you can translate the
      records to your system timezone, namely UTC.

      Best!

      Joe

      From: "Martyn Noss" mjnoss@users.sf.net
      To: "[flowviewer:wiki] " Home@wiki.flowviewer.p.re.sf.net
      Date: 10/25/2016 02:13 PM
      Subject: [flowviewer:wiki] Discussion for Home page

      Hi Joe,
      Great tool!
      Question: My local Flowviewer box is running in UTC. Among other netflow
      sources, I've recently started rsyncing a flow-tools data repository from
      a box in Colorado that happens to be running in MDT (UTC-6). Not a typical
      configuration, I realize. Does this timezone discrepancy create an
      insurmountable problem for Flowviewer?
      Flowviewer and Flowgrapher reports pulling data from the Colorado device
      look reasonable (although I haven't done a detailed analysis on the actual
      numbers), but Flowmonitor/RRD seem not to be picking up any of the data
      between hours 00 and 06 UTC, resulting in obvious gaps in the daily and
      weekly graphs. On the theory that RRD was looking only in the flow-tools
      daily data directory where it expects to find the data it's looking for,
      and consequently not finding it for the first 6 hours of each day due to
      the TZ lag, I wrote a script to copy flow-tools data files for hours 18-24
      of the current MDT day into the following day's directory (which I create
      as part of the script). The script runs immediately following completion
      of each rsync. This file copy seems to have more-or-less solved the
      Flowmonitor/RRD output problem (i.e., daily and weekly graphs now look
      reasonable), but I consider this strategy to be only a diagnostic exercise
      -- not a good operational solution.
      I see in the User's Guide that FV v4.6 uses the system TZ rather than
      setting TZ based on configuration files. Seems reasonable, but in my case
      it appears to mean that Flowmonitor will necessarily mis-interpret my
      Colorado data (or any non-UTC data, presumably).
      Any advice on how I might tackle this problem (short of asking Colorado to
      start using UTC)? Thanks!

      Sent from sourceforge.net because you indicated interest in
      https://sourceforge.net/p/flowviewer/wiki/Home/
      To unsubscribe from further messages, please visit
      https://sourceforge.net/auth/subscriptions/

       
      • Martyn Noss

        Martyn Noss - 2016-10-30

        Hi Joe,

        Thanks for the prompt reply. You've confirmed what I supposed, i.e., FlowViewer isn't designed to handle multiple timezones in a single deployment. Now that I know I'm not dealing with a configuration problem, I can turn to other strategies. Regarding using the SiLK toolset, Colorado is still using flow-tools so I'm restricted to flow-tools commands; not as powerful as SiLK, but your point remains the same. I'll rummage around in the man pages and see what I can find. Thanks again!

        Marty

         

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks