Copyright 2008-2009 Lockheed Martin Corporation
The Vortex program is open source software: you can copy it, redistribute it and/or modify
it under the terms of the GNU General Public License version 2.0 as published by
the Free Software Foundation. The Vortex Program and any derivatives of the Vortex program
must be licensed under GPL version 2.0 and may not be licensed under GPL version 3.0.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY OF ANY KIND, including without limitation the implied warranties of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details at http://www.gnu.org/licenses/gpl-2.0.html.
The term "Vortex" should be taken to also include any portions or derived works of Vortex.
You are highly encouraged to send your changes to the Vortex program to charles.smutz@lmco.com
and/or samuel.wenck@lmco.com for possible incorporation into the main distribution.
By sending these changes to Lockheed Martin, you are granting to Lockheed Martin
Corporation the unlimited, perpetual, non-exclusive right to reuse, modify,
and/or relicense the code on a royalty-free basis.
The libraries to which Vortex links are distributed under the terms of their own licenses.
Please see those libraries for their applicable licenses.
Introduction:
Vortex is an application for intrusion detection and network surveillance. It relies heavily on libnids by Rafal Wojtczuk (see http://libnids.sourceforge.net/).
Vortex is designed to facilitate agile creation of advanced network intrusion detection or network surveillance systems. It performs the hard work of packet capture, filtering, and reassembly then provides that data to external programs for analysis. Stream data is stored in files and stream meta data is output via STDOUT (meta data is encoded in file name).
Usage:
Vortex is used by specifying an input (pcap file or interface), a directory in which to store data, amount of data to collect in each direction, filters, and numerous other options. The analyzer program reads the stream meta data (file names), open and analyzes the files, and optionally purges the files after they are analyzed. If large numbers of streams are to be analyzed simultaneously, the connection hash table (-s) may need to be increased. See TCP_LIMIT count in error logs (-E).
Example:
Image that you want to apply the spamsum algorithm to all web requests. You could do the following:
vortex -i eth0 -t /tmp/vortex_demo/ | xargs ssdeep | logger
which would yield logs such as:
Mar 23 18:54:22 hostname logger: 3:ixx1PQcFI5LM38iIjMFykMFPDD7IM/R/CZJMT9yn:ixz7Z3EPLIM/7By,"/tmp/vortex_demo/127.0.0.1:42742s127.0.0.1:80"
Filtering
Vortex supports tcpdump style filter expressions which are used by the BPF filtering mechanism.
Vortex also uses libBSF which is similar to BPF but with stream semantics instead of packet semantics.
Vortex also supports simple sampling or polling.
BPF filtering
It is highly recommended that you use a BPF to limit traffic seen by vortex to the extent possible. In the case of IP fragmented streams (which vortex supports) is possible to loose data with a naive BPF (see libnids documentation).
libBSF
LibBSF is tcpdump filter syntax and the BPF virtual machine with stream semantics. The tcpdump-like expression is very similar to tcpdump filter language. See man tcpdump(8). Syntax is the same. The allowed primitives are close to the subset of primitives relating to streams.
Allowable primitives are: clt, svr, host, net, mask, port, portrange
"clt" indicates the stream client, "svr" indicates the stream server, otherwise the primitives have the same meaning
Don't use primitives like ether, ip, tcp, udp, len, etc. The results will not be coherent even if no errors are thrown. If you are tempted to use them, you are probably trying to use the wrong tool for your needs.
Performance
Vortex has been used to capture large amounts of data with very low packet loss.
RAMdisk
It is recommended that if vortex is used to analyze large amounts of data, especially if the data is deleted after it is analyzed, that temporary stream files be written to a file system in RAM instead of on hard disk. Ex. tmpfs.
Packet Capture:
The following setting changes may help if you are experiencing packet loss between the kernel and vortex (PCAP_DROP in performance stats).
#! /bin/bash
#
#Red Hat Defaults:
#
cat /proc/sys/net/core/rmem_default
cat /proc/sys/net/core/rmem_max
cat /proc/sys/net/core/netdev_max_backlog
#
#New settings
#
echo 67108864 > /proc/sys/net/core/rmem_default
echo 67108864 > /proc/sys/net/core/rmem_max
echo 50000 > /proc/sys/net/core/netdev_max_backlog
#for /etc/sysctl.conf to make the changes persistant
#Set some capture pertinant settings
#defaults
#net.core.rmem_default = 109568
#net.core.rmem_max = 131071
#net.core.netdev_max_backlog = 1000
#
#net.core.rmem_default = 67108864
#net.core.rmem_max = 67108864
#net.core.netdev_max_backlog = 50000
The following may help if you are experiencing packet loss between the NIC and kernel (ifconfig RX dropped):
To check the current setting and determine the maximum, use the ethtool -g option:
/sbin/ethtool -g eth2
# Ring parameters for eth2:
# Pre-set maximums:
# RX: 1020
# RX Mini: 0
# RX Jumbo: 0
# TX: 255
# Current hardware settings:
# RX: 255
# RX Mini: 0
# RX Jumbo: 0
# TX: 255
#To change the rx ring size, use the ethtool -G option:
/sbin/ethtool -G eth2 rx 1020
#To make this change permanent put the command(s) in the /sbin/ifup-local script. It can be created as follows:
touch /sbin/ifup-local
chmod 744 /sbin/ifup-local
echo "/sbin/ethtool -G eth2 rx 1020" >> /sbin/ifup-local
#The max rx ring size is 1020 bnx2 and 4096 for e1000e.
Improvements
There are many improvements to be made to vortex. If you make useful improvements, please provide them to the authors so we can consider integrating them. Even improvements to documentation, packaging, etc are both needed and desired.
Support and Feature requests
The authors are not currently soliciting feature requests and can not be asked to provide technical support.