===============
== HostStats ==
===============
HostStats allows you to look on your NetFlow data from a different point of
view - it calculates statistics about network traffic of each individual host
(IP address) in the network.
The system consists of a daemon, which performs all the calculations, a simple
backend plugin for NfSen announcing availability of new data to the daemon,
and a frontend plugin for visualization of host statistics. Both backend and
frontend plugin communicates with the daemon through a UNIX socket.
The frontend plugin may run as a plugin in NfSen or it may be accessed directly
at http://example.com/nfsen/plugins/HostStats.php.
Every five minutes the daemon reads new NetFlow data and splits them into
profiles based on specified filters. By default there is only one profile which
uses empty filter so it uses all flows. You can define your own profiles
(e.g. to filter SSH traffic only) in configuration file.
Flows in each profile are then used to compute the followin per-IP statistics:
- incoming/outgoing flows/packets/bytes
- incoming/outgoing flows with SYN/ACK/FIN/RST/PSH/URG TCP flag set
- number of addresses this adress communicated with (approximation only)
- flow sources (e.g. routers/probes) which saw a communication of the address
These statistics are then stored into files (default /data/hoststats/data,
one file per profile and timeslot). Data in these files can be accessed later
using HostStats frontend.
The statistics are also searched for suspicious behavior using a simple ruleset.
Currently there are rules for detection of hosts scanning network and for
detection of DoS attackers and victims. More rules can be easily added into
detectionrules.cpp.
All detected events are logged into files (default /data/hoststats/log). It is
also possible to set up a script which is called every time an event is
detected. Such script may send an email to someone, for example.
== Installation ==
Please, see INSTALL file for system requirements and installation instructions.
== Configuration ==
Main configuration file is located in <INSTALLATION_DIR>/hoststats.conf. There
you can specify new profiles, set maximum amount of database size for each
profile, set up scripts run every time a timeslot is processed or an event is
detected and many other parameters. See comments in the cofiguration file for
more information.
Some frontend-specific parameters, such as date format, can be configured in
<FRONTEND_PLUGIN_DIR>/HostStats/app/config/config.neon.
== GUI Query language ==
Query language serves as a way to get a list of hosts, whose statistics meet
the query conditions (on Details page). The laguage syntax is similar to that
of nfdump - you can compare a value of any statistic with another statistic or
a constant via relation operators (=, !=, >, >=, <, <= ). You can also use
logical conjunctions AND, OR, NOT and parentheses. Queries are case insensitive.
Supported statistics:
[in|out] flows|packets|bytes|syn|ack|fin|rst|psh|urg
In/out may be omitted, in that case, value is a sum of both directions,
i.e. syn = in syn + out syn
Example:
in flows > 200
Prints all hosts, whose number of incoming flows in the selected timeslot is
greater than 200.
Another examples:
bytes > 1000
in packets = 1 OR out packets = 1
in packets > 1000 AND out packets < 100
out syn > 100 AND (in ack < 20 OR in fin < 20)
2 * (2 + 2) * 2 > in syn
== IP Map ==
The IP Map page shows activity of all IP addresses in your data on a "map of
IPv4 address space". This map is a 256x256 pixel image where each
pixel corresponds to one /16 prefix and color of the pixel corresponds
to total traffic generated by the prefix. The prefixes are mapped into
the image using Hilbert curve - a method inspired by an XKCD comic
(http://xkcd.com/195/), later adopted by research community. You can
also "zoom in" onto a particular prefix (up to /16 length), then each pixel
correspods to a smaller subnet or a single address.
== Troubleshooting ==
If you have problems with HostStats backend, you can usually find some useful
information in syslog. Alternatively you can run HostStats daemon (hoststatserv)
with -f option - then it will not daemonize and it will print all messages to
standard output. Verbosity level can be set in main configuration file.
A script sendrequest.py may be used to send requests to backend and show
replies. It may be used to tell backend to process data from arbitrary timeslot
or to simulate requests from frontend. See help included in the script for more
information.
Do not heasitate to write us for assistance.
Also, if you think you have found a bug, please report it.
Contact email address: ibartosv@fit.vutbr.cz
== Authors ==
Vaclav Bartos, CESNET
Lukas Hutak, CESNET
Martin Zadnik, CESNET
Tomas Varga, Brno University of Technology
Petr Sladek, Brno University of Technology
Please, report issues to Vaclav Bartos (ibartosv@fit.vutbr.cz)
