NFDUMP - Netflow processing tools / News: Recent posts

The project nfdump moved to GitHub:
https://github.com/phaag/nfdump

Fix a few issues, mainly for NSEL/NEL/NAT
- Merge NSEL/NEL/NAT elements. Fix differences
- Fix v1 extension size bug
- Add htonll check for autoconf
- Fix AddExtensionMap compare bug
- Fix ipfix templare withdraw problems - free all maps correctly
- Add minilzo 2.08 - fixes CVE-2014-4607
- Cleanup some stat code. more needs to be done ..
- Cleanup man pages for -O -n
- Remove SunPro test in configure - no longer supported anyway
- Cleanup NAT/NSEL filter differences
- Fix 64bit alignment bug.

Bugfix and feature update release:
- Add IP defragmentation to nfpcapd
- Add NAT pool port allocation
- Modify/fix NAT vrf tags. Add egress vrf ID
- Extend exporters to 65535
- Modify common record due to exporter exhaustion. new common record
type 10 adds 4 extra bytes. Reads v1 common record transparently
- Fix sflow potential crash

Please note: nfdump-1.6.12 updates the data records. Older versions < 1.6.12 can not correctly read 1.6.12 data. However, upgrade from old versions is transparent

Maintenance Release and Updates:
- Add compatibility with ASA 9.1
- llvm compiler compatible (MACOSX and FreeBSD).
- minor bug fixes and clean ups.

Minor bugfix Release:
- Fix -t +/-n time window option.
- Fix bug in nfanon - stat record update in anon file.
- Fix bug in netflow v5 mudule: extension map size wrong.
- Fix bug nfexport: In some cases could result in wrong flow counter.
- Fix nftrack - could coredump in some cases.

Maintainance/bugfix release.
You should update, if you use IPFIX, or ASA/NSEL

- Fix SPARC compile/optimise bug
- Add output packet/bytes counter to global stat for NSEL flows ASA > 8.5
- Add NSEL filter options xnet
- Modify extension descriptor code for nfdump1.7.
Still use 1.6 extension map layout for compatibility
- Add prototype for nfpcapd - pcap -> nfdump collector.
Converts traffoc directly to nfdump files. - experimental - not enabled
- Fix bug in ipfix module: uninitialised variable
- Cleanup syslog/LogError calls
- Fix minor non critical bugs and compile issues

nfdump-1.6.9 integrates two new decoding modules:
- ASA/NSEL decoder - new implementation written from scratch
- NEL NAT event logging
- Few bugs fixed.
nfdump-1.5.8-x-NSEL is deprecated now. Use nfdump-1.6.9

nfdump-1.5.8-4-NSEL is the last release in the 1.5.x-NSEL tree.
It fixes a few minor bugs and adds support for NSEL ASA 8.4.
This release is intended for all users already using nfdump-1.5.8-NSEL. For new projects it's no longer recommended, please use nfdump-1.6.9 instead, which includes full NSEL ASA support

nfdump-1.6.8p1 fixes an exporter bug ( exporter > 256 ) as well as an IPFIX collector bug

nfdump-1.6.8 add some more features and fixes some bugs.
Features:
- Add ip list option for 'next ip' in filter syntax
- Accept v9 sampler_id in 2bytes
- Add IPFIX packet/octet TotalCount fields 85/86
- Add received timestamp to sflow collector
- Add IPFIX options as rfc5101 section-6.2
- Add exporter records for sflow collector
- Add received time stamp extension
- Add recursive format parser. Allows to extend predefined formats.
- Change flow record sorting to heapsort. remove limit 1000
- Merge -m option to -O tstart. -m now depricated.
- Add -O tend. Print order according to tend of flows ascending
- Apply -O print order for printing flow cache. Applies to -A... read more

nfdump-1.6.4 is mainly a maintainance release and fixes a few bugs in various modules. Due to the increasing demand netflow v1 was added for simple and high volume netflow traffic. A few new output format tags were added and the code was cleaned up for integrating new modules e.g. IPFIX, ASA flows.1.6.4 also includes code for extended statistics for port, bpp histograms - still experimental and not enabled by default.

nfdump-1.6.3 is a bug fix release:
- Fix SysUptime 32bit overflow in v5 header
- Add fix for strange first/last swap reported by some JUNOS users.
- Fix extension size bug
- Move IP anonymisation to separate binary nfanon
- Fix initialise bug of -o fmt: and not available fields

Updated version of nfsen-1.5.7-nsel.
Limitation: Due to a major code cleanup and in respect to future upwards compatibility with nfdump-1.6.x, the binary data format changed from nfdump-1.5.7-nsel to nfdump-1.5.8-NSEL. Therefore flows collected with nfdump-1.5.7-nsel can no longer be processed be nfdump-1.5.8-NSEL.

nfdump-1.5.8-NSEL is fully nfdump-1.5.8 up and downwards compatible. Both versions can read either data
likewise, with the limitation of course, that nfdump-1.5.8 skips NSEL specifics but displays other data correctly. This
also allows, that upcoming nfdump-1.6.x with NSEL support will be able to read and upgrade data from nfdump-1.5.8-NSEL
transparently. It's fully 64bit compatible and should compile and run on any standard *NIX.... read more

nfdump-1.6.2 is a bugfix and maintainance release. It contains bugfixes for the sflow collector, adds more flow details to the flow-tools converter and fixes some minor issues.

nfdump-1.6.1 has been released. It contains sampling support for Juniper routers (JunOS) and fixes some coredump bugs. nfdump-1.6 users should upgrade to 1.6.1.

For a full description of nfdump-1.6, see the README File.

In brief, new in 1.6 since 1.5.8
----------------------------------------
o Add router IP extension.
o Add router ID extension (engine type/ID)
o Add srcmask and dstmask aggregation
o Add possibility to save aggregated flows into file ( -w )
Note: This results in a behaviour change for -w in combination
with aggregation )
o Extend -N ( do not scale numbers ) to all text output not just summary
o Remove header lines of -s stat, when using -q ( quiet )
Note: This results in a behaviour change for -N
o Remove legacy v1.4 file compatibility
o Remove -S option from nfdump ( legacy 1.4 compatibility )
o Make use of log (syslog) functions for nfprofile.
o Move log functions to util.c
o Update sflow collector.
o Add parse_csv.pl script as an example to parse csv output
o Add csv output format ( -o cvs ) as replacement for -o pipe - keep
-o pipe for now.
o Flow-tools converter updated - supports all common elements.
o Sflow collector updated. Supports more common elements.
o Add sampling to nfdump. Sampling is automatically recognised
in undocumented v5 header fields and in v9 option templates.
see nfcapd.1(1)
o Add @include option for filter to include more filter files.
o Add flexible aggregation comparable to Flexible Netflow (FNF)
over all available v9 tags
o All new tags can be selected in -o fmt:... see nfdump(1)
o topN stat for all new tags is implemented
o Integrate developer code to read from pcap files into stable branch
o Update filter syntax for new tags
o Add flexible storage option for nfcapd. To save disk space, the
data extensions to be stored in the data file are user selectable.
o Added more v9 tags for netflow v9.
The detailed tags are listed in nfcapd(1) Beside of MAC addresses
and VLAN labels, also MPLS labels and many more v9 tags are now
supported. AS numbers and interface numbers are now 32bit clean.
Adding new tags also extended the binary file format with
data block type 2, which is extension based. File format
for version <= 1.5.* ( Data block format type 1 ) is read
transparently. ( --enable-compat15 ) Data block type 2 are skipped
by nfdump 1.5.7.
o Added option for multiple netflow stream to same port.
-n <Ident,IP,base_directory>
Example: -n router1,192.168.100.1,/var/nfdump/router1
So multiple -n options may be given at the command line
Old style syntax still works for compatibility, ( -I .. -l ... )
but then only one source is supported.
o Move to automake for building nfdump
o Make nfdump fully 64bit compliant. ( 32/64bit data alignments
and access )
Compiles and runs cleanly on 32/64bit systems
o Switch scaling factor ( k, M, G ) from 1024 to 1000.

A next beta snapshot for nfdump 1.6 is available.
The feature list is stable now for 1.6. Testers are welcome and required.
For a production grade environment, use stable 1.5.8.

Changelog:
- Add srcmask and dstmask aggregation
- Add cvs output mode. -o cvs
- Fix some bugs of previous beta
- Add bidirectional aggregation of flows ( -b, -B )
- Add possibility to save aggregated flows into file ( -w )
Note: This results in a behaviour change for -w in combination
with aggragation )
- Extend -N ( do not scale numbers ) to all text output not just summary
- Make extension handling more robust for some moody IOSes.
- Remove header lines of -s stat, when using -q ( quiet )
Note: This results in a behaviour change for -N
- Remove -S option from nfdump ( legacy 1.4 compatibility )
- Make use of log (syslog) functions for nfprofile.
- Move log functions to util.c

The 1.6b release of nfdump is a step towards stable 1.6 for a more complete support for netflow v9. Beside of MAC addresses and VLAN labels, also MPLS labels and many more v9 tags are now supported. AS numbers are 32bit compatible. Sampling is now supported.
nfdump-1.6b compiles and runs cleanly on 32/64 bit systems.

nfdump 1.5.8 is a bugfix release of 1.5.7. It compiles and runs clean on 64bit systems, and is a drop in replacement.

The current snapshot adds more v9 tags and has other new features. See the README file. Please note: This is a developer snapshot. Not intended in productive environments

Latest nfdump-1.5.6 now supports fast file compression, reducing file size down to 50% avg.

The developer snapshot nfdump-snapshot-20060413 contains the first release of an sflow collector.

After a longer beta periode, nfdump v1.5 is released. It adds a number of new features and supports netflow v9 and is IPv6 enabled

For beta testers: nfdump 1.5 implements netflow v9 and IPv6. See the ChangeLog for more information.
The stable release is still nfdump 1.4.1

This update fixes a bug, when collecting statistics from large time windows. The resulting packets/bytes may overflow, when printed.
Minor Solaris issue fixed.