[Nfdump-discuss] Case of the missing data!

[Brian C. <b.c...@po...>]

Here's a really odd one.

Both my cacti graphs and nfsen graphs are showing an unexpected increase 
in data usage; but digging down in nfsen I can't find the source.  Then 
I find even nfdump itself doesn't seem to agree internally on what the 
total data volume is.

The data source is a Cisco ASA5520 and I'm running nfdump with 
--enable-nfprofile --enable-nftrack --enable-nsel.

$ nfdump -V
nfdump: Version: NSEL-NEL1.6.11 $Date: 2013-11-16 09:04:43 +0100 (Sat, 
16 Nov 2013) $

Here is an example single 5-minute timeslot. nfsen shows the total 
traffic for this slot is 12GB. But if I ask nfsen/nfdump to group it by 
protocol, I see much less:

$ nfdump -M /var/nfsen/profiles-data/live/wrn-asa1  -T  -r 
2014/04/03/nfcapd.201404031445 -n 100 -s proto/bytes
Top 100 Protocol ordered by bytes:
Date first seen          Duration Proto Protocol    Flows(%)     
Packets(%)       Bytes(%) pps      bps   bpp
2014-04-03 14:44:59.963   299.827 TCP 6    15491(35.0)        0( 0.0)  
106.1 M( 0.9)        0    2.8 M     0
2014-04-03 14:45:16.740   278.051 ESP 50       37( 0.1)        0( 
0.0)    9.6 M( 0.1)        0 274810     0
2014-04-03 14:45:00.203   299.597 UDP 17    28500(64.5)        0( 
0.0)    2.5 M( 0.0)        0 67030     0
2014-04-03 14:45:08.151   244.776 SCPS 105       15( 0.0)        0( 
0.0)    16104( 0.0)        0 526     0
2014-04-03 14:45:00.353   299.197 ICMP 1      172( 0.4)        0( 
0.0)    11290( 0.0)        0 301     0

Summary: total flows: 44215, total bytes: 12090101416, total packets: 0, 
avg bps: 322577971, avg pps: 0, avg bpp: 0
Time window: 2014-04-03 14:44:59 - 2014-04-03 14:49:59
Total flows processed: 44215, Blocks skipped: 0, Bytes read: 5723064
Sys: 0.040s flows/second: 1105319.7  Wall: 0.038s flows/second: 1157097.2

Notice that under "summary" it shows total bytes 12GB. But TCP is only 
106MB, representing 0.9% of the total, and the other protocols much 
less. Where is everything else? What is nfdump adding to get the 'total 
bytes' figure?

The total number of flows is correct! And if I exclude the protocols 
listed above, I get nothing:

$ nfdump -M /var/nfsen/profiles-data/live/wrn-asa1  -T  -r 
2014/04/03/nfcapd.201404031445 -c 100 'not proto tcp and not proto esp 
and not proto udp and not proto scps and not proto icmp'
Date first seen          Event  XEvent Proto      Src IP 
Addr:Port          Dst IP Addr:Port     X-Src IP Addr:Port X-Dst IP 
Addr:Port   In Byte Out Byte
Summary: total flows: 0, total bytes: 0, total packets: 0, avg bps: 0, 
avg pps: 0, avg bpp: 0
Time window: 2014-04-03 14:44:59 - 2014-04-03 14:49:59
Total flows processed: 44215, Blocks skipped: 0, Bytes read: 5723064

Using some Perl to add up the raw data I get around 1.2GB in total:

$ nfdump -M /var/nfsen/profiles-data/live/wrn-asa1  -T  -r 
2014/04/03/nfcapd.201404031445 | perl -ne 'if (/\s+(\d+)\s+(\d+)$/) { 
$in+=$1,$out+=$2; } END { print "in=$in,out=$out\n"; }'
in=35021687,out=1196947214

Or about 0.5GB if I look only at flow DELETE records:

$ nfdump -M /var/nfsen/profiles-data/live/wrn-asa1  -T  -r 
2014/04/03/nfcapd.201404031445 | grep DELETE | perl -ne 'if 
(/\s+(\d+)\s+(\d+)$/) { $in+=$1,$out+=$2; } END { print 
"in=$in,out=$out\n"; }'
in=18751754,out=475720863

Neither matches with either the total shown by nfdump, or the total of 
the -s proto/bytes figures.

Any suggestions? I can send the file to the author privately if requested.

Thanks,

Brian.