Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Christopher E. <ce...@lc...>]

On 30.08.20 23:31, Bu...@Bu... wrote:
> Two thoughts... 25% is an interesting number, it implies either you
> are actually getting 2Core of 2Threads each in your VM (saturating a
> single processor)

Yeah, that seems to be the case. Qemu consistently reports ~50% CPU
usage whenever I saturate any of the backends, and the host also shows 1
core fully utilised.

> Second thought ... rejigger your test and run it to native ipset add 
> commands. That will give you a best-case performance value and
> eliminate the firewall-cmd and dbus overhead. That should be your
> yard-stick.

I did some more testing over the weekend with the nftables & iptables
backends as well. I didn't extensively test ipset, but it can add 6000
IPs in 5s at maximum throughput. The guest runs Fedora, where firewalld
uses nftables as the backend, so the nft-sets backend is the relevant
comparison.

All this was done using a fixed list of 6000 block commands using
randomly generated IPs (roughly 50/50 IPv4/IPv6).

Test 1, maximum throughput:
Just dumping the list into the backend via 'cat iplist | <backend>':

- 2.4.1 backends:
firewalld: ~1400s to complete, 50% CPU (i.e. 1 core maxed out)
nft-sets: ~50s to complete, 50% CPU
iptables: ~22s to complete, 50% CPU
ipset: ~ 5s to complete, 50% CPU
- 1-second-batch: (i.e. one command in this case)
firewalld: ~35s to complete
nft-sets: 1-2s to complete
iptables: 1-2s to complete

- that's 4.3 / 120 / 273 / 1200 blocks per second for the 2.4.1
firewalld/nftables/iptables/ipset backends, and those are indeed roughly
the rates at which the backends starts maxing out the CPU.
- adding all IPs in one command completes more or less instantly with
iptables/nft-sets (ipset can't do that), but takes a whopping 35 seconds
with firewalld. That's .... not very fast.

Test 2, CPU load at 10 blocks/second:
Dumping the list via 'while read line; do echo $line; sleep 0.1; done <
iplist' | <backend>:
- 2.4.1 backends:
firewalld: ~1400s to complete, 50% CPU (i.e. 1 core)
nft-sets: ~640s to complete, ~22% CPU
iptables: ~640s to complete, ~12% CPU
- 1-second-batch:
firewalld: ~640s to complete, ~27% CPU
nft-sets: ~640s to complete ~4% CPU
iptables: ~640s to complete ~4% CPU

The '1s-batch' firewalld backend can deal with ~200 commands per second
before maxing out the CPU.

Given that the nftables commands seem to be slower than iptables/ipset
commands, I also checked what happens when you switch firewalld to use
iptables as the backend, answer, nothing much.

There are two other interaction modes for firewalld that I haven't
tested, direct (allows direct access to the backends' tables/chains) &
passthrough (passes commands unparsed to backend). I don't know if
they're much faster, but I'd be against using them if it can be avoided,
because that would mean that sshguard would have to start detecting
things about firewalld's configuration.


My suggestion would be to
a) modify the backends where the underlying command supports it to
collect block/release commands before sending them to the firewall
(although the non-firewalld backends can keep up with very high
blockrates, the grouping still reduces CPU load quite a bit).
b) suggest that people switch from firewalld to nft-sets/iptables
backends if performance is still an issue (they can even keep using
firewalld for non-sshguard stuff)
c) talk to firewalld upstream about maybe making firewalld more
efficient ...

I can create a PR for (a), though someone else will have to test the pf
and ipfw backends, I don't have any system that uses them. The rest can
remain unmodified, as the respective commands can't add multiple IPs in
one call.

Christopher