dbFlowc GIT
Status: Beta
Brought to you by:
ruslan_nm
Menu
▾
▴
Tree [53a5dc] master / History
ruslan_nm
Read Me
Flowc-1.4.3rc1
Last updated: 20 May 2003
The flowc package intend for gathering, storing and analyzing traffic
accounting for CISCO routers with NetFlow (version 5) enabled switching.
The author express gratitude to Sergey Korsak (skif@1plus1.net) and
Alexand Sudakov (saa@mail.univ.kiev.ua) for valuable ideas and remarks.
Questions, notes, wishes, beer please send to roman@uninet.kiev.ua.
CONTENT
=======
I System requirements.
II Main features.
III Package structure.
IV Limitations.
I System requirements.
======================
The package was successfuly tested in following configuration:
1. FreeBSD (3.4R, 4.2R, 4.8R, 5.1R) Linux OS (Linux 2.2, RedHat 9)
2. MySQL
3. Net-SNMP or UCD-SNMP
4. Apache web server
5. PHP with graphics support (gd, libpng)
6. Perl5 with DBI module with MySQL support
7. CISCO IOS 12.0 - 12.3 IP plus.
II Main features.
======================
1. Traffic accounting aggregation in a RAM.
2. Using MySQL DB for storing traffic information and fast access to it.
3. Independence stored information of router interfaces SNMP_id.
(If you added or deleteed subinterface or async line, you need to
rebuild internal snmp interface table. You can do it by
"flowd -k rebuild" command. See bellow a description of flowd
options. If router has been rebooted, the flowd performs automatic
rebuild of internal snmp interface table).
4. Report scripts included in distribution produce useful reports in
HTML format.
5. Lightweight (total size is approx. 120Kb).
6. Installation and configuration is easy (via configure script).
7. Abbility to gather data from more than one router.
8. TCP port aggregation option.
9. Internal traffic counters based on IP filter rules.
10. Integrated ISP traffic billing system.
III Package structure.
======================
src/
The src directory contains sources of flowd and loader binaries.
flowd
is a netflow collector. It gathers routers traffic accounting, agre-
gates they in a RAM and periodicaly flush it in to external files.
The default flowd configuration file is /etc/flowd.conf and can be
redefined during building package. The daemon redirects all error
messages to syslog. If debug option in config file is set to ON,
flowd runs in foreground and all debug information and error messages
will be displayed on the stderr too. MySQL DB must be created before
flowd start.
Usage: flowd [-k operation]
where operations are:
shutdown - daemon stutdown
dump - dump gathered traffic in data file
rebuild - rebuild internal snmp interface table
rotate - rotate data file
rotate_db - rotate MySQL flow tables.
WARNING! If snmp id of router interfaces was changed (for example you
have added or deleted subinterfaces or async lines) internal snmp
interface table rebuild is needed.
loader
loader is a MySQL loader. It loads new records from dump files created
by flowd in MySQL database. After the loader added ordinary record in
DB, the LAST_FILE_OFFSET field in 'routers' table will be incremented.
This field contains last added record offset in router dump file. After
start, the loader begins record load process from that offset. If you
created new dump file or rotated old dump file, the
routers.LAST_FILE_OFFSET field must be set to zero for appropriate
router. ("flowd -k rotate" command automatically reset
routers.LAST_FILE_OFFSET field after successlful rotation). MySQL DB
must be created before loader start. The 'routers' table creates
automatically by flowd at startup if it don't exists. The loader must
be periodicaly executed via cron.
Usage: loader [-r router_name [start_date [finish_date [traf_counters_filename]]]
start and finish dates have the following format: yyyymmdd-HHMMSS
where:
yyyy - year in four digits presentation
mm - month (01-12)
dd - day of month (01-31)
HH - hours (00-59)
MM - minutes (00-59)
SS - seconds (00-59)
NOTE: If you are using partial data loading (with start_date option)
the last_file_offset field in ROUTERS table leaves untouched for
specified router. WARNING! Before running partial data loading, check
the timezone settings. Incorrect timezone settings will be cause of
data lost or dublication. If you want to load data for certain traffic
counters (without loading to default 'flows' table and updating
last_file_offset field in ROUTERS table) you can list counters names
in external file and use it as foth argument (traf_counters_filename).
It usefull if you have added new traffic counters (customers) in the
middle of a month and want to load data for corresponding traffic
counters flows tables begins from first month day.
Because loader creates lock file for each loading dump file, more than
one copy of loader process can run in any time. Each process will works
with one routers dump file.
counter
connects to flowd via UNIX Domain socket, reads and displays internal
traffic counters packets and octets values and flowd statistic. See
the flowd.conf example for configuring traffic counters and examples/
directory if you are going to graphical traffic representation.
Usage: ./src/counter [-s] [-t] [-i] [-a] [-d]
options:
-s - display flowd statistic
-t - display traffic counters
-i - display ip address counters
-a - display AS counters
-d - display all traffic counters structure including MySQL counters
without any options all listed above counters will be displayed.
test
binds to specified UDP port, listens and displays on terminal content
of all incoming NetFlow packets without any processing. It may be used
for debuging purposes.
Usage:
test -p port
-------------------------------------------------------------------------------
scripts/
The scripts directory contains MySQL scripts.
create_table
This script automaticaly created by configure. The purpose of this
script is creation all needed databse table structures and
netflow MYSQL user with appropriative MYSQL access permissions.
upgrade-1.4.3.sql
This script used for upgrade database structire from old version to
1.4.3 version.
-------------------------------------------------------------------------------
www/
This directory contains scripts for traffic reports, billing web
interface and flowd grahicas monitoring.
billing/
web interface to ISP traffic billing system. See the
www/billing/README for detail description.
reports/
This directory contains scripts used for generation html
traffic reports
analyser.pl
This script automaticaly creates by configure if the --with-web
option has been used. The purpose of this perl script is a html
traffic report generation. The final report produced by
analyser.pl has two HTML tables (one for incoming traffic,
other for outgoing traffic). That tables contain: destination
address and destination hostname, number of packets and total
bytes transfered. Report pages will be generated for each
interface marked as "External" in netflow MySQL DB (flowd
automaticaly mark as "external" all external interfaces
specified in its config file flowd.conf). This reports have
the first level of traffic detalization.
Usage: analyser.pl h|d|w|m
h - hourly report
d - daily report
w - weekly report
m - monthly report
host_detail.php
HTML files generates by analyser.pl have a links to this file.
It generates more detail traffic reports. Reports produced by
this file will be generated "on the fly", instead static html
pages generated by the analyser.pl. They contain: source
address, source hostname, destination address, destination
hostname, number of packets and total bytes transfered. This
reports have the second level of traffic detalization.
host_very_detail.php
HTML page generated by host_detail.php have a links to this
file. It generates very detail traffic reports. Reports
generated by this file will by generated "on the fly" too.
They contain: timestamp, address, source hostname, source port,
destination address, destination hostname, destination port,
protocol, number of packets and total bytes transfered.
NOTE: timestamp in that report will be arounded to
dump_interval specified in flowd.conf.
custom_report.php
This script allow you to generates reports for individual
interface, host, port, AS, etc...
config.php
This file authomaticaly generates by configure and contains
reports and MYSQL db access settings needed for .php scripts.
IV Limitations.
==============
The main flowc limitation is impossibility correct aggregation of tcp
connections based on client port number. As result, the netflow
database is large especially on heavy traffic routers.
This limitation is a feature of NetFlow technology, because it is not
possible to know, which side was an initiator of tcp connection. In case
if one tcp port is less than 1024, then this port is tcp server port
and another port belongs to tcp client. In this case the client port
number aggregation is possible. This feature significantly reduce the
netflow database size.
Good luck ...
