Re: [Snort-inline-users] snort_inline constantly bad latency

[Dave R. <da...@re...>]

On Mon, 9 Apr 2007, Roman Glebov wrote:

+-----BEGIN PGP SIGNED MESSAGE-----
+Hash: SHA1
+
+Hallo!
+Thank you for your response!

Sure thing!

+Dave Remien wrote:
+> You've run into the "other" "stuck packet issue" with netlink_queues and
+> heavy traffic. When you're seeing the 2 sec round trip times, do a "cat
+> /proc/net/netfilter/nfnetlink_queue". You'll probably see something like
+> this:
+>
+>    50   5679     1 2     65535         0     34    1155049     90214
+>
+> The 34 in the 7th column means that there wasn't room on the netlink
+> socket's queue for the number of packets you had queued up, so some 34
+> were dropped; when that happens, the last packet you received is always in
+> the nf queue and doesn't get delivered to snort until the next packet
+> arrives, effectively pushing it out. (That's the permanent 1 in the 3rd
+> field; under normal circumstances, this will go to 0 a lot of the time).
+> In the real world, this is usually in less than a second, but your pings
+> are a second apart, and probably go through snort twice, hence the 2
+> second timing. Try "watch cat /proc/net/netfilter/nfnetlink_queue".
+>
+But normaly when load disappears the queue should become empty and
+this problem should disappear. right?
+And it is not the case here. It stays forever. Even when there is no
+load at all.

This is why this is what I call the non-fatal stuck packet problem. Yes, 
it should go to zero. No, it doesn't go to zero. Harald Welte mentioned 
this at one point in time, while working on another "stuck packet" bug, 
but I don't think it's currently listed in bugzilla.netfilter.org.

Feel free to raise it with Harald and Patrick 8-).

+
+Here is the output
+
+zaga4:~# cat /proc/net/netfilter/nfnetlink_queue
+    1   4787     1 2 65535     0 889803  4062359  1
+    2   4788     1 2 65535     0 672482  3527632  1
+
+Ideas ?

Yeah, you've definitely blown the memory buffers. I know of no way to 
clear the stuck packet, other than either stopping/starting snort, or 
adding code in snort to disconnect/reconnect to the netlink queue. 

Try setting the mem buffer up. After that, you could see if it still 
happens with Harald's test program (nfqnl_test.c). It did for me...
As far as I can see, it's an outstanding bug, but as I say - in a "normal" 
network, the amount of latency introduced is the time for the next packet 
to show up, which usually is "not very long".

Cheers!

Dave


+
+> Couple of things to do:
+>
+> 1. Make sure you're using the latest/greatest libnfnetlink and
+> libnetfilter_queue:
+>
+> http://www.netfilter.org/projects/libnfnetlink/files/libnfnetlink-0.0.25.tar.bz2
+>
+> and
+>
+> http://www.netfilter.org/projects/libnetfilter_queue/files/libnetfilter_queue-0.0.13.tar.bz2
+>
+> They have fixes that help with this; the 2.6.18 kernel has the kernel side
+> fixes (added at 2.6.15).
+>
+> Also, you should increase the amount of memory available for packets on
+> your sockets with:
+>
+> sysctl -w net.core.rmem_default = 8388608
+> sysctl -w net.core.rmem_max = 16777216
+>
+> and last, but not least, set the
+>
+> #define NFQNL_QMAX_DEFAULT 1024
+>
+> value in nfnetlink_queue.c (in your kernel tree) to something like 8192
+> and rebuild the nfnetlink_queue module. (Harald's defaults are fine if
+> you're using a ton of queues and the programs reading/writing them are
+> extremely fast, snort_inline doesn't fit that category 8-).
+>
+> Cheers,
+>
+> Dave
+>
+>
+I try all the things now, you have suggested!
+
+With best wishes and lots of thanks
+
+Roman
+-----BEGIN PGP SIGNATURE-----
+Version: GnuPG v1.4.2.2 (GNU/Linux)
+Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
+
+iQIVAwUBRhqWYrhQu20hGMIkAQKeZQ/9EQ/6Kc50bKrLfP4xbqAatjTIknX+m639
+w/Pffe4ZZtwpyJnwt6KpWyJSmhDBm1FOtduJ9jyqjVj0JcobhhU/APwHRO/F8Qhg
+zHqoLBNegQoRFWbMSL9FsafWrAV5pbWPj1jRzD66hLY8GarerQ812emzWEDNEfch
+CoM0eG9tONmV3A5MuUVEwjNVrbxT2ggsbPBaqgHIC7YpfskrY0Ew9glFsmjbkZFS
+ShrU603+fVr7XwB74J1+3S4qp8QXQxabt9sLyxkhzbejogrTgfJSPRBMS4BFV+4x
+jUTDgkbsye70Uuq0qUK4jJ7xxvEuU+IyrkepgUQ+5WDWfbSHEOndCMkUtb9WKkxG
+ppjkLfXHskn3ImacxQncl6MwXlPJNZI3NU898o3n3nnsTsIE+/b4kAtKm3VHAGCw
+fhbCaSTmsfQs1Ia2/46f3L1RJ5oOttBQFCWSFTMY2XWEBCkx/Nfpl0c7or/8AMI0
+bHo7Mo935yn1LgjRQ+glj8TDMMnSWrS58bAW1Zg0j27CqPt9xLGfdV0j90iOQddz
+HF6yujbOcnQ/1OD7OX9Zp3VloRhN5wZDAoVqZ30Ybq5ge17WdELaQzUe8jkoK2eb
+1mSD0/0kPfE6yfc3q/heCDR+z6HAYFOzMrIy8b+KnnuSRT1ziklhjL7siK1LU+BT
+NJao1C7ri7I=
+=Kn1+
+-----END PGP SIGNATURE-----
+
+
+-- 
+This message has been scanned for viruses and
+dangerous content by MailScanner, and is
+believed to be clean.
+


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.