Re: [Snort-inline-users] snort_inline constantly bad latency

[Dave R. <da...@re...>]

You've run into the "other" "stuck packet issue" with netlink_queues and 
heavy traffic. When you're seeing the 2 sec round trip times, do a "cat 
/proc/net/netfilter/nfnetlink_queue". You'll probably see something like 
this:

   50   5679     1 2     65535         0     34    1155049     90214 

The 34 in the 7th column means that there wasn't room on the netlink 
socket's queue for the number of packets you had queued up, so some 34 
were dropped; when that happens, the last packet you received is always in 
the nf queue and doesn't get delivered to snort until the next packet 
arrives, effectively pushing it out. (That's the permanent 1 in the 3rd 
field; under normal circumstances, this will go to 0 a lot of the time). 
In the real world, this is usually in less than a second, but your pings 
are a second apart, and probably go through snort twice, hence the 2 
second timing. Try "watch cat /proc/net/netfilter/nfnetlink_queue".

Couple of things to do:

1. Make sure you're using the latest/greatest libnfnetlink and 
libnetfilter_queue:

http://www.netfilter.org/projects/libnfnetlink/files/libnfnetlink-0.0.25.tar.bz2

and 

http://www.netfilter.org/projects/libnetfilter_queue/files/libnetfilter_queue-0.0.13.tar.bz2

They have fixes that help with this; the 2.6.18 kernel has the kernel side 
fixes (added at 2.6.15).

Also, you should increase the amount of memory available for packets on 
your sockets with:

sysctl -w net.core.rmem_default = 8388608
sysctl -w net.core.rmem_max = 16777216

and last, but not least, set the 

#define NFQNL_QMAX_DEFAULT 1024

value in nfnetlink_queue.c (in your kernel tree) to something like 8192 
and rebuild the nfnetlink_queue module. (Harald's defaults are fine if 
you're using a ton of queues and the programs reading/writing them are 
extremely fast, snort_inline doesn't fit that category 8-).

Cheers,

Dave


+
+Message: 2
+Date: Mon, 09 Apr 2007 19:04:25 +0200
+From: Roman Glebov <sl...@sl...>
+Subject: [Snort-inline-users] snort_inline constantly bad latency
+	after	heavy load until you restart
+To: sno...@li...
+Message-ID: <461...@sl...>
+Content-Type: text/plain; charset="iso-8859-1"
+
+-----BEGIN PGP SIGNED MESSAGE-----
+Hash: SHA1
+
+My problem is following :
+
+When i start snort_inline and then do simple ping i get 0.5ms responses.
+Then i generate 100mbit load for 2 minutes or so.
+
+Then when i try to ping again i get response time of 2000ms! with 10%
+packet los all the time until i restart snort_inline.
+
+During heavy load i see :
+[4788] packet recv contents failure
+messasges. they disappear then.
+
+The slow down is definetly snort_inline or the netlink QUEUE  problem
+because then i remove a rule to put packets to the QUEUE i get
+imediatly 0.5ms ping.
+
+My system is standart debian etch box with 2.6.18 smp kernel
+i compiled libnet pcre and snort_inline from sources with gcc 4.1
+compiler with default compile flags.
+
+My machine is dual athlon box in 32 bit mode.
+
+software :
+libnet 1.0.2a dfault configure
+libdnet-1.11 default config
+pcre-7.0 default configure
+
+snort_inline snort_inline-2.6.1.2-BETA1
+./configure --enable-nfnetlink --enable-queue --enable-linux-smp-stats
+- --enable-perfprofiling --with-libipq-includes=/usr/local/include/
+- --enable-clamav

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.