Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Kevin Z. <kev...@gm...>]

On 8/31/20 5:12 AM, Christopher Engelhard wrote:
> I did some more testing over the weekend with the nftables & iptables
> backends as well. I didn't extensively test ipset, but it can add 6000
> IPs in 5s at maximum throughput. The guest runs Fedora, where firewalld
> uses nftables as the backend, so the nft-sets backend is the relevant
> comparison.

This comparison is great work and gives us pretty useful information.
Thank you for doing this research.

> Test 1, maximum throughput:
> Just dumping the list into the backend via 'cat iplist | <backend>':
> 
> - 2.4.1 backends:
> firewalld: ~1400s to complete, 50% CPU (i.e. 1 core maxed out)
> nft-sets: ~50s to complete, 50% CPU
> iptables: ~22s to complete, 50% CPU
> ipset: ~ 5s to complete, 50% CPU
> - 1-second-batch: (i.e. one command in this case)
> firewalld: ~35s to complete
> nft-sets: 1-2s to complete
> iptables: 1-2s to complete

Just to be clear, does "one-second batch" collect inputs from stdin over
one second, then issues one large command to the backend?

> Test 2, CPU load at 10 blocks/second:
> Dumping the list via 'while read line; do echo $line; sleep 0.1; done <
> iplist' | <backend>:
> - 2.4.1 backends:
> firewalld: ~1400s to complete, 50% CPU (i.e. 1 core)
> nft-sets: ~640s to complete, ~22% CPU
> iptables: ~640s to complete, ~12% CPU
> - 1-second-batch:
> firewalld: ~640s to complete, ~27% CPU
> nft-sets: ~640s to complete ~4% CPU
> iptables: ~640s to complete ~4% CPU

If your list is 6000 addresses, and attacks come in at 10 attacks/sec,
and the backends are not CPU-limited (which they might be), shouldn't
this total 600 seconds? It seems like even the fastest backend, when run
in this mode, can only complete in 640 seconds.

> My suggestion would be to
> a) modify the backends where the underlying command supports it to
> collect block/release commands before sending them to the firewall
> (although the non-firewalld backends can keep up with very high
> blockrates, the grouping still reduces CPU load quite a bit).
This would be great.

> b) suggest that people switch from firewalld to nft-sets/iptables
> backends if performance is still an issue (they can even keep using
> firewalld for non-sshguard stuff)
> c) talk to firewalld upstream about maybe making firewalld more
> efficient ...

For now, I think it would also be useful to clearly document that
firewalld is slow. I wonder what upstream will say what you're using
firewalld for, heh.

> I can create a PR for (a), though someone else will have to test the pf
> and ipfw backends, I don't have any system that uses them. The rest can
> remain unmodified, as the respective commands can't add multiple IPs in
> one call.

I can test pf and ipfw. If need be, they can also be separate backends
(currently they share sshg-fw.in, which does not need to be the case).

Regards,
Kevin