sshguard-users

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once (Christopher Engelhard)

[<Bu...@Bu...>]

Two thoughts... 25% is an interesting number, it implies either you are
actually getting 2Core of 2Threads each in your VM (saturating a single
processor) or something is seriously restricting your ability to beat a core
to death. VMware statistics would be interesting too.

Second thought ... rejigger your test and run it to native ipset add
commands. That will give you a best-case performance value and eliminate the
firewall-cmd and dbus overhead. That should be your yard-stick.

-----Burton



Date: Fri, 28 Aug 2020 17:18:38 +0200
From: Christopher Engelhard <ce...@lc...>
To: ssh...@li...
Subject: Re: [SSHGuard-users] performance when using firewalld:
	adding/removing many entries at once
Message-ID: <86a...@lc...>
Content-Type: text/plain; charset=utf-8

On 28.08.20 13:26, Christopher Engelhard wrote:
> & I haven't done any benchmarking yet.

OK, did some playing around. All testing with 6000 random IP block requests
at 100/s (i.e. over 1 min) on a 2 core/8GB RAM virtual machine)

Using the firewalld backend of 2.4.1, it takes 24min to add all IPs to the
blocklist, CPU load during that time is fairly consistently 25% for
firewalld & 5-10% for firewalld-cmd.

Using the "collecting" version & collecting requests for 1s doesn't
significantly change the overall load, but the process now completes in just
over 3 min.

Collecting for 5s or 10s reduces this a bit further to ~2:30 min, again with
no significant load reduction in firewalld. Firewall-cmd only causes
significant CPU load whenever it is triggered by the backend.

Given that there's no difference between 5s or 10s grouping in overall
runtime, I'd say that pretty much reflects the speed at which firewalld is
able to add ips to the ipset.

I think the bottleneck is firewalld processing the commands it receives on
DBus, not firewall-cmd sending them off, otherwise I'd expect to see much
less load on firewalld in the first test compared to the later ones.

Christopher

P.S.: The total number of IPs that ended up in the ipsets dropped to
~5200 in the last two tests, so it is possible that I'm running into some
issues with maximum string lengths/command lengths or so. Probably not a
good idea to set it this high outside of testing.

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Christopher E. <ce...@lc...>]

On 30.08.20 23:31, Bu...@Bu... wrote:
> Two thoughts... 25% is an interesting number, it implies either you
> are actually getting 2Core of 2Threads each in your VM (saturating a
> single processor)

Yeah, that seems to be the case. Qemu consistently reports ~50% CPU
usage whenever I saturate any of the backends, and the host also shows 1
core fully utilised.

> Second thought ... rejigger your test and run it to native ipset add 
> commands. That will give you a best-case performance value and
> eliminate the firewall-cmd and dbus overhead. That should be your
> yard-stick.

I did some more testing over the weekend with the nftables & iptables
backends as well. I didn't extensively test ipset, but it can add 6000
IPs in 5s at maximum throughput. The guest runs Fedora, where firewalld
uses nftables as the backend, so the nft-sets backend is the relevant
comparison.

All this was done using a fixed list of 6000 block commands using
randomly generated IPs (roughly 50/50 IPv4/IPv6).

Test 1, maximum throughput:
Just dumping the list into the backend via 'cat iplist | <backend>':

- 2.4.1 backends:
firewalld: ~1400s to complete, 50% CPU (i.e. 1 core maxed out)
nft-sets: ~50s to complete, 50% CPU
iptables: ~22s to complete, 50% CPU
ipset: ~ 5s to complete, 50% CPU
- 1-second-batch: (i.e. one command in this case)
firewalld: ~35s to complete
nft-sets: 1-2s to complete
iptables: 1-2s to complete

- that's 4.3 / 120 / 273 / 1200 blocks per second for the 2.4.1
firewalld/nftables/iptables/ipset backends, and those are indeed roughly
the rates at which the backends starts maxing out the CPU.
- adding all IPs in one command completes more or less instantly with
iptables/nft-sets (ipset can't do that), but takes a whopping 35 seconds
with firewalld. That's .... not very fast.

Test 2, CPU load at 10 blocks/second:
Dumping the list via 'while read line; do echo $line; sleep 0.1; done <
iplist' | <backend>:
- 2.4.1 backends:
firewalld: ~1400s to complete, 50% CPU (i.e. 1 core)
nft-sets: ~640s to complete, ~22% CPU
iptables: ~640s to complete, ~12% CPU
- 1-second-batch:
firewalld: ~640s to complete, ~27% CPU
nft-sets: ~640s to complete ~4% CPU
iptables: ~640s to complete ~4% CPU

The '1s-batch' firewalld backend can deal with ~200 commands per second
before maxing out the CPU.

Given that the nftables commands seem to be slower than iptables/ipset
commands, I also checked what happens when you switch firewalld to use
iptables as the backend, answer, nothing much.

There are two other interaction modes for firewalld that I haven't
tested, direct (allows direct access to the backends' tables/chains) &
passthrough (passes commands unparsed to backend). I don't know if
they're much faster, but I'd be against using them if it can be avoided,
because that would mean that sshguard would have to start detecting
things about firewalld's configuration.


My suggestion would be to
a) modify the backends where the underlying command supports it to
collect block/release commands before sending them to the firewall
(although the non-firewalld backends can keep up with very high
blockrates, the grouping still reduces CPU load quite a bit).
b) suggest that people switch from firewalld to nft-sets/iptables
backends if performance is still an issue (they can even keep using
firewalld for non-sshguard stuff)
c) talk to firewalld upstream about maybe making firewalld more
efficient ...

I can create a PR for (a), though someone else will have to test the pf
and ipfw backends, I don't have any system that uses them. The rest can
remain unmodified, as the respective commands can't add multiple IPs in
one call.

Christopher

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Kevin Z. <kev...@gm...>]

On 8/31/20 5:12 AM, Christopher Engelhard wrote:
> I did some more testing over the weekend with the nftables & iptables
> backends as well. I didn't extensively test ipset, but it can add 6000
> IPs in 5s at maximum throughput. The guest runs Fedora, where firewalld
> uses nftables as the backend, so the nft-sets backend is the relevant
> comparison.

This comparison is great work and gives us pretty useful information.
Thank you for doing this research.

> Test 1, maximum throughput:
> Just dumping the list into the backend via 'cat iplist | <backend>':
> 
> - 2.4.1 backends:
> firewalld: ~1400s to complete, 50% CPU (i.e. 1 core maxed out)
> nft-sets: ~50s to complete, 50% CPU
> iptables: ~22s to complete, 50% CPU
> ipset: ~ 5s to complete, 50% CPU
> - 1-second-batch: (i.e. one command in this case)
> firewalld: ~35s to complete
> nft-sets: 1-2s to complete
> iptables: 1-2s to complete

Just to be clear, does "one-second batch" collect inputs from stdin over
one second, then issues one large command to the backend?

> Test 2, CPU load at 10 blocks/second:
> Dumping the list via 'while read line; do echo $line; sleep 0.1; done <
> iplist' | <backend>:
> - 2.4.1 backends:
> firewalld: ~1400s to complete, 50% CPU (i.e. 1 core)
> nft-sets: ~640s to complete, ~22% CPU
> iptables: ~640s to complete, ~12% CPU
> - 1-second-batch:
> firewalld: ~640s to complete, ~27% CPU
> nft-sets: ~640s to complete ~4% CPU
> iptables: ~640s to complete ~4% CPU

If your list is 6000 addresses, and attacks come in at 10 attacks/sec,
and the backends are not CPU-limited (which they might be), shouldn't
this total 600 seconds? It seems like even the fastest backend, when run
in this mode, can only complete in 640 seconds.

> My suggestion would be to
> a) modify the backends where the underlying command supports it to
> collect block/release commands before sending them to the firewall
> (although the non-firewalld backends can keep up with very high
> blockrates, the grouping still reduces CPU load quite a bit).
This would be great.

> b) suggest that people switch from firewalld to nft-sets/iptables
> backends if performance is still an issue (they can even keep using
> firewalld for non-sshguard stuff)
> c) talk to firewalld upstream about maybe making firewalld more
> efficient ...

For now, I think it would also be useful to clearly document that
firewalld is slow. I wonder what upstream will say what you're using
firewalld for, heh.

> I can create a PR for (a), though someone else will have to test the pf
> and ipfw backends, I don't have any system that uses them. The rest can
> remain unmodified, as the respective commands can't add multiple IPs in
> one call.

I can test pf and ipfw. If need be, they can also be separate backends
(currently they share sshg-fw.in, which does not need to be the case).

Regards,
Kevin

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Christos C. <ch...@cr...>]

> I can test pf and ipfw. If need be, they can also be separate backends
> (currently they share sshg-fw.in, which does not need to be the case).

ipfw adds 2326 IPs / second (one by one) in a ipfw table using Intel i7-6700 CPU.

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Christopher E. <ce...@lc...>]

On 2020-08-31 20:36, Kevin Zheng wrote:
> Just to be clear, does "one-second batch" collect inputs from stdin 
> over
> one second, then issues one large command to the backend?

Yes. The fw_block/release() functions collect the inputs they receive 
and issue one big command to the backend if 1s has elapsed since the 
last time they did that. Technically, in this test there were 2 
commands, because all tested backends have to deal with IPv4 and IPv6 
separately.

> If your list is 6000 addresses, and attacks come in at 10 attacks/sec,
> and the backends are not CPU-limited (which they might be), shouldn't
> this total 600 seconds?

Yes, but that is not due to the backend, but to a badly designed 
experiment: The while loop issuing the commands doesn't actually 
complete in 600s, it take longer. I think the issue is that the sleep 
command is not that precise, as the total accumulated "surplus" depends 
on the sleep value.
Regardless, the execution times are extremely reproducible (both for the 
issuing loop on its own & for loop plus backend), so I don't think it 
matters here (other than 10/s being actually "nominally 10/s").

> For now, I think it would also be useful to clearly document that
> firewalld is slow. I wonder what upstream will say what you're using
> firewalld for, heh.

Execution speed is probably not really a focus for them, but I'm sort of 
hoping that there are some straightforward bottlenecks that simply 
nobody has bothered to identify yet. And to be fair, only being able to 
add ~5 IPs per second to a set is REALLY slow.

> Regards,
> Kevin

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Kevin B. <kev...@gm...>]

On 2020/09/01 16:36, Christopher Engelhard wrote:
> 
> Execution speed is probably not really a focus for them, but I'm sort of
> hoping that there are some straightforward bottlenecks that simply
> nobody has bothered to identify yet. And to be fair, only being able to
> add ~5 IPs per second to a set is REALLY slow.
> 

Although I saw that the thread dried up in September last
year, I thought I'd pitch a bit more info into it.

I have just started up an SSHGuard 2.4.2 instance, on a
SLES 15-based Cray/HPE box, and have looked at the time
it appears to be taking to ingest IP addresses from the
blacklist that was placed there ahead of process starting.

I have assumed that the ingest is done a line at a time,
hence the grep -n of the blacklist for the IP address
that a systemctl status shows was being processed, so
as to give a rough timeline:


    Active: active (running) since Mon 2022-01-10 11:03:41 AWST; 1h 8min ago
            `-93206 /usr/bin/python3 -Es /usr/bin/firewall-cmd --quiet \
            --ipset=sshguard4 --add-entry=119.63.84.130/32

# grep -n 119.63.84.130  /var/lib/sshguard/blacklist
7936:1616965857|100|4|119.63.84.130

    Active: active (running) since Mon 2022-01-10 11:03:41 AWST; 1h 12min ago
            `-95043 /usr/bin/python3 -Es /usr/bin/firewall-cmd --quiet \
            --ipset=sshguard4 --add-entry=140.249.197.153/32

# grep -n 140.249.197.153  /var/lib/sshguard/blacklist
8405:1617116224|100|4|140.249.197.153

    Active: active (running) since Mon 2022-01-10 11:03:41 AWST; 1h 45min ago
            `-108270 /usr/bin/python3 -Es /usr/bin/firewall-cmd --quiet \
            --ipset=sshguard4 --add-entry=188.166.237.18/32

# grep -n 188.166.237.18  /var/lib/sshguard/blacklist
12116:1620694731|100|4|188.166.237.18

    Active: active (running) since Mon 2022-01-10 11:03:41 AWST; 2h 0min ago
            |-61184 /usr/bin/python3 -Es /usr/bin/firewall-cmd --quiet \
            --ipset=sshguard4 --add-entry=150.138.114.102/32

# grep -n 150.138.114.102  /var/lib/sshguard/blacklist
13916:1622407810|100|4|150.138.114.102


To summarise that data:


2h 00min 13916
1h 45min 12116  15mins 1800    120.00/m   2.00/s
1h 12min  8405  33mins 3711    112.45/m   1.87/s
1h  8min  7936   4mins  469    117.25/m   1.95/s


It does seem a bit slow, especially when compared to some
SLES 12-based instances here that use the IPTables backend.
(Apologies, I don't have any comparative figures to hand,
  at time of writing, but will look to get some)


Probably doesn't tell you anything you didn't already know,
Kevin

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Kevin Z. <kev...@gm...>]

On 1/9/22 9:22 PM, Kevin Buckley wrote:
> 2h 00min 13916
> 1h 45min 12116  15mins 1800    120.00/m   2.00/s
> 1h 12min  8405  33mins 3711    112.45/m   1.87/s
> 1h  8min  7936   4mins  469    117.25/m   1.95/s

These numbers don't look good.

I don't have a system handy where I can test firewall-cmd, but is there 
an interface (or another command) that lets you bulk-add address to an 
ipset without invoking firewall-cmd once per address?

I took a cursory look at what our "competitor" fail2ban does:

https://github.com/fail2ban/fail2ban/blob/80805cabfcf57dd0454d47d7f86d43c6738ce629/config/action.d/firewallcmd-ipset.conf

Which, to summarize, seems to be:

actionban = firewall-cmd --ipset=<ipmset> --add-entry=<ip>

actionunban = firewall-cmd --ipset=<ipmset> --remove-entry=<ip>

Which seems to be exactly what SSHGuard is doing.

Anyone with more Linux firewall experience who could tell us if there's 
a faster way to add to a firewalld ipset? Is it time to teach SSHGuard 
how to use the dbus interface?

Regards,
Kevin

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Kevin B. <kev...@gm...>]

On 2022/01/11 12:36, Kevin Zheng wrote:
>> I don't have a system handy where I can test firewall-cmd, but is there
> an interface (or another command) that lets you bulk-add address to an
> ipset without invoking firewall-cmd once per address?

Sadly, can't claim to have huge experience (yet?) with firewalld,
and, as you may have seen in another recent post to the list,
into this thread

   Firewalld backend: do I need to create the two ipsets ?

I'm having issues with the ipset that SSHGuard creates anyway, but
am yet to get around to looking at why I'm seeing what I am, vis:

Jan 10 11:03:44 ln01 firewalld[48409]: WARNING: sshguard4: INVALID_TYPE: 'hash:net' is not supported by ipset., ignoring for run-time.
Jan 10 11:03:44 ln01 firewalld[48409]: WARNING: sshguard6: INVALID_TYPE: 'hash:net' is not supported by ipset., ignoring for run-time.
Jan 10 11:03:45 ln01 firewalld[48409]: WARNING: INVALID_IPSET: sshguard6
Jan 10 11:03:45 ln01 firewalld[48409]: WARNING: INVALID_IPSET: sshguard4
Jan 10 11:03:45 ln01 firewalld[48409]: ERROR: INVALID_IPSET: sshguard4
Jan 10 11:03:46 ln01 firewalld[48409]: ERROR: INVALID_IPSET: sshguard4
Jan 10 11:03:46 ln01 firewalld[48409]: ERROR: INVALID_IPSET: sshguard4

so, that might be part of the slow rate of ingest I've seen so far,
and/or an indication that nothing is actually being ingested?

To expand on that last point. SSHGuard is clearly working, even to the
extent of blocking (again) two IP addresses that were in the blacklist,
but which  presumably hadn't been ingested from it by the time the latest
"attack" passed the threshold, or, had been read from the file but just
not added to the ipset.

Kevin

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Felix S. <fel...@os...>]

Am 11.01.22 um 05:36 schrieb Kevin Zheng:
> I don't have a system handy where I can test firewall-cmd, but is there an 
> interface (or another command) that lets you bulk-add address to an ipset 
> without invoking firewall-cmd once per address?

I just checked the man page but "--add-entries-from-file=filename" looks 
interesting (there is also "--remove-entries-from-file=filename").

Of course firewalld also has a DBUS interface.

Felix

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Ryan M. R. <rr...@ro...>]

On 1/10/2022 8:36 PM, Kevin Zheng wrote:
> On 1/9/22 9:22 PM, Kevin Buckley wrote:
>> 2h 00min 13916
>> 1h 45min 12116  15mins 1800    120.00/m   2.00/s
>> 1h 12min  8405  33mins 3711    112.45/m   1.87/s
>> 1h  8min  7936   4mins  469    117.25/m   1.95/s
>
> These numbers don't look good.
>
> I don't have a system handy where I can test firewall-cmd, but is 
> there an interface (or another command) that lets you bulk-add address 
> to an ipset without invoking firewall-cmd once per address?

firewall-cmd --ipset=ipset --add-entries-from-file=filename seems to be 
possibly what you are looking for but that may in fact just be 
programmed to invoke firewall-cmd one per address in the file as opposed 
to a bulk load.  I would read carefully that documentation and run some 
tests to see if it's faster for you and works for you.  A script to read 
a line from a file you prepare and invoke the firewall-cmd command like 
SSHGuard currently does as you are using it timed and then another 
script to invoke the firewall-cmd command with the 
--add-entries-from-file options times should give you benchmarks.

You also could investigate other BACKENDS as SSHGuard calls them and see 
if others happen to be faster.

>
> I took a cursory look at what our "competitor" fail2ban does:
I don't think many would agree that fail2ban is a competitor of 
SSHGuard.  At least to me they seem to have very different strategies 
both of which could be integrated into the same system or used on a 
network for different systems in a way where they are not competitors 
per se but both used in different situations depending on their 
strengths in solving particular problems.
>
> https://github.com/fail2ban/fail2ban/blob/80805cabfcf57dd0454d47d7f86d43c6738ce629/config/action.d/firewallcmd-ipset.conf 
>
>
> Which, to summarize, seems to be:
>
> actionban = firewall-cmd --ipset=<ipmset> --add-entry=<ip>
>
> actionunban = firewall-cmd --ipset=<ipmset> --remove-entry=<ip>
>
> Which seems to be exactly what SSHGuard is doing.
>
> Anyone with more Linux firewall experience who could tell us if 
> there's a faster way to add to a firewalld ipset? Is it time to teach 
> SSHGuard how to use the dbus interface?
I think if you want to stick to the firewalld ipset option with SSHGuard 
carefully reading their documentation and in particular for your 
questions information about firewall-cmd options such as the one listing 
the ability to add ip addresses from a file on their web site if you 
have access to that would be a good idea.  Documentation about 
firewall-cmd is available at the web site 
https://firewalld.org/documentation/man-pages/firewall-cmd.html if you 
don't have access to a machine to run something like 'man 
firewall-cmd'.  As I said above though, I'm not sure it would be 
faster.  It would seem it would but I haven't done any testing.
>
> Regards,
> Kevin
>
>
> _______________________________________________
> sshguard-users mailing list
> ssh...@li...
> https://lists.sourceforge.net/lists/listinfo/sshguard-users

-- 
This email has been checked for viruses by AVG.
https://www.avg.com

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[lists <li...@la...>]

I dropped using sshguard specifically for the load cause by adding IPs to firewalld. It was less of a load to allow failed ssh attempts than to block them. I use PKI so I think the odds of a breach are small.  More likely than not some software vulnerability will lead to a breach than someone hacking ssh. 

I realize sshguard also can protect your email. What I did is block all out of country use on all email ports other than 25 in addition to the servers list that I block. Totally impractical except if you have a personal web server. I see very little hacking on my email and I use anvil to throttle was little does occur. 

My situation is I am using a one CPU VPS.

Now to complete the story, I have a very large list of IP space that  I block. I assume firewalld processes the cidrs into some hash table that it efficiently searches. That is the overhead of a static IP blocking list is nil. It does use a fair amount of memory but that isn't an issue for the VPS.

At times my VPS would stall due to the CPU load. Probably at moments when sshguard was altering the block list and the VPS wasn't getting much CPU juice.

Sshguard is a fine program but firewalld doesn't handle dynamic blocking. 

Given the state of centos, I probably will have to change to a new disty in a year or so. I stay on the list since the traffic is light and I may learn something for the next time I install it. 

Blathering on a bit more, something like a two step firewall is what you need. Use the big list first then follow up with the small list. I have over 30k cidrs in the big list. I never had more than a few hundred at a time in the sshguard block list. 



  Original Message  


From: ssh...@li...
Sent: January 19, 2022 3:10 PM
To: ssh...@li...
Reply-to: rr...@ro...
Subject: Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once


On 1/10/2022 8:36 PM, Kevin Zheng wrote:
> On 1/9/22 9:22 PM, Kevin Buckley wrote:
>> 2h 00min 13916
>> 1h 45min 12116  15mins 1800    120.00/m   2.00/s
>> 1h 12min  8405  33mins 3711    112.45/m   1.87/s
>> 1h  8min  7936   4mins  469    117.25/m   1.95/s
>
> These numbers don't look good.
>
> I don't have a system handy where I can test firewall-cmd, but is
> there an interface (or another command) that lets you bulk-add address
> to an ipset without invoking firewall-cmd once per address?

firewall-cmd --ipset=ipset --add-entries-from-file=filename seems to be
possibly what you are looking for but that may in fact just be
programmed to invoke firewall-cmd one per address in the file as opposed
to a bulk load.  I would read carefully that documentation and run some
tests to see if it's faster for you and works for you.  A script to read
a line from a file you prepare and invoke the firewall-cmd command like
SSHGuard currently does as you are using it timed and then another
script to invoke the firewall-cmd command with the
--add-entries-from-file options times should give you benchmarks.

You also could investigate other BACKENDS as SSHGuard calls them and see
if others happen to be faster.

>
> I took a cursory look at what our "competitor" fail2ban does:
I don't think many would agree that fail2ban is a competitor of
SSHGuard.  At least to me they seem to have very different strategies
both of which could be integrated into the same system or used on a
network for different systems in a way where they are not competitors
per se but both used in different situations depending on their
strengths in solving particular problems.
>
> https://github.com/fail2ban/fail2ban/blob/80805cabfcf57dd0454d47d7f86d43c6738ce629/config/action.d/firewallcmd-ipset.conf
>
>
> Which, to summarize, seems to be:
>
> actionban = firewall-cmd --ipset=<ipmset> --add-entry=<ip>
>
> actionunban = firewall-cmd --ipset=<ipmset> --remove-entry=<ip>
>
> Which seems to be exactly what SSHGuard is doing.
>
> Anyone with more Linux firewall experience who could tell us if
> there's a faster way to add to a firewalld ipset? Is it time to teach
> SSHGuard how to use the dbus interface?
I think if you want to stick to the firewalld ipset option with SSHGuard
carefully reading their documentation and in particular for your
questions information about firewall-cmd options such as the one listing
the ability to add ip addresses from a file on their web site if you
have access to that would be a good idea.  Documentation about
firewall-cmd is available at the web site
https://firewalld.org/documentation/man-pages/firewall-cmd.html if you
don't have access to a machine to run something like 'man
firewall-cmd'.  As I said above though, I'm not sure it would be
faster.  It would seem it would but I haven't done any testing.
>
> Regards,
> Kevin
>
>
> _______________________________________________
> sshguard-users mailing list
> ssh...@li...
> https://lists.sourceforge.net/lists/listinfo/sshguard-users

--
This email has been checked for viruses by AVG.
https://www.avg.com



_______________________________________________
sshguard-users mailing list
ssh...@li...
https://lists.sourceforge.net/lists/listinfo/sshguard-users

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Chris J. <joh...@as...>]

I set up a cron job to zero out the blacklist file every week. When the 
system reboots (usually for kernel update), it can rebuild the smaller 
file in a timely manner. Since the IPs are ever-changing anyway, the 
current "bad" ones get re-added to the list quickly. To keep the file 
size even smaller, I also have the block subnet size set to 24 (which 
I'm sure is not a preferable option for everyone).

It's not elegant, but it works for me.

Chris

On 1/19/22 19:51, lists wrote:
> I dropped using sshguard specifically for the load cause by adding IPs to firewalld. It was less of a load to allow failed ssh attempts than to block them. I use PKI so I think the odds of a breach are small.  More likely than not some software vulnerability will lead to a breach than someone hacking ssh.

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[lists <li...@la...>]

It won't help because I still want to block hosting services, servers, VPS, etc. I need the ability to have a large static block list. 

Incidentally if the developers want something else to block, tor exit nodes are documented. I had a script to block them too but dropped it due to the same firewalld issue. There is an API for the exit nodes. 

Nothing I serve needs to have the viewers IP hidden. Every time I read about some hacking I look at the IP and I usually have it blocked. 





	  Original Message  	


From: joh...@as...
Sent: January 19, 2022 9:31 PM
To: ssh...@li...
Subject: Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once


I set up a cron job to zero out the blacklist file every week. When the
system reboots (usually for kernel update), it can rebuild the smaller
file in a timely manner. Since the IPs are ever-changing anyway, the
current "bad" ones get re-added to the list quickly. To keep the file
size even smaller, I also have the block subnet size set to 24 (which
I'm sure is not a preferable option for everyone).

It's not elegant, but it works for me.

Chris

On 1/19/22 19:51, lists wrote:
> I dropped using sshguard specifically for the load cause by adding IPs to firewalld. It was less of a load to allow failed ssh attempts than to block them. I use PKI so I think the odds of a breach are small.  More likely than not some software vulnerability will lead to a breach than someone hacking ssh.


_______________________________________________
sshguard-users mailing list
ssh...@li...
https://lists.sourceforge.net/lists/listinfo/sshguard-users

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Kevin B. <kev...@gm...>]

On 2022/01/11 12:36, Kevin Zheng wrote:
> On 1/9/22 9:22 PM, Kevin Buckley wrote:
>> 2h 00min 13916
>> 1h 45min 12116  15mins 1800    120.00/m   2.00/s
>> 1h 12min  8405  33mins 3711    112.45/m   1.87/s
>> 1h  8min  7936   4mins  469    117.25/m   1.95/s
> 
> These numbers don't look good.
> 
> I don't have a system handy where I can test firewall-cmd, but is there
> an interface (or another command) that lets you bulk-add address to an
> ipset without invoking firewall-cmd once per address?
> 
> I took a cursory look at what our "competitor" fail2ban does:
> 
> https://github.com/fail2ban/fail2ban/blob/80805cabfcf57dd0454d47d7f86d43c6738ce629/config/action.d/firewallcmd-ipset.conf
> 
> Which, to summarize, seems to be:
> 
> actionban = firewall-cmd --ipset=<ipmset> --add-entry=<ip>
> 
> actionunban = firewall-cmd --ipset=<ipmset> --remove-entry=<ip>
> 
> Which seems to be exactly what SSHGuard is doing.
> 
> Anyone with more Linux firewall experience who could tell us if there's
> a faster way to add to a firewalld ipset? Is it time to teach SSHGuard
> how to use the dbus interface?
> 

I am not going to claim vast firewalld experience, but

one thing worth noting is that adding IP addresses, or ranges, read
from an existing blakclist file at SSHGuard startup, results in every
line in

var/lib/sshguard/blacklist

being added to

/etc/firewalld/ipsets/sshguard4.xml

as entries between these lines


<?xml version="1.0" encoding="utf-8"?>
<ipset type="hash:net">
   <option name="family" value="inet"/>
   ... entries here ...
</ipset>

so, for example,

   <entry>192.102.251.102/32</entry>

That suggests that one could take the blacklist lines
and simply pre-populate the

sshguard4.xml

IPSet file, before starting SSHGuard, however I am not sure
what SSHGuard would do with existing entries if, on starting,
it finds that the sshguard4 IPSet exists, and already has
entries.

I can't currently speak to the speed with which a pre-populated
IPSet file is read into FirewallD, because my instance is still
slowly churning through the ingest of my blacklist, after the
most recent restart but once, I have the IPSet file, I intend
to take a look at that.

Kevin

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Christopher E. <ce...@lc...>]

I had originally looked into this when the topic first came up. The main 
overhead seems to be not so much the adding but rather the call to 
firewall-cmd. It is possible to collect addresses and add them all in 
one call, which dramatically speeds up the process.

The dbus-interface might be another option to speed things up, as might 
be bypassing firewall-cmd and interacting with the underlying library 
directly. I wanted to look into that but haven't so far.

On 19.01.22 10:10, Kevin Buckley wrote:
> That suggests that one could take the blacklist lines
> and simply pre-populate the
> 
> sshguard4.xml
> 
> IPSet file, before starting SSHGuard, however I am not sure
> what SSHGuard would do with existing entries if, on starting,
> it finds that the sshguard4 IPSet exists, and already has
> entries.

This would be another option, but seems rather brittle to me. a) it 
messes with firewalld-internal config files that might or might not stay 
in that location/format and b) actually getting firewalld to read those 
files means triggering a global reload which might or might not 
influence other things the user has been doing in the meantime.


There is one fundamental tradeoff to consider when collecting addresses 
for a single call: You're either inefficient (collect very few 
addresses) or slow (addresses are collected for a long time before send 
to the firewall).
Example: IP x gets put in the UNBAN collection at the end of a 10 minute 
timeout. That collection will be send to firewalld every y minutes or z 
entries. But since the backend script is only called when sshguard 
notices something, this happens the next time that happens AT THE 
EARLIEST, no matter what we choose for y,z. If that time is a day (low 
traffic server which only gets attacked occasionally) then x gets 
unbanned after a day.

The only way around that is letting sshguard trigger the backend 
periodically as well as on events.


Christopher

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Kevin B. <kev...@gm...>]

On 2022/01/19 19:34, Christopher Engelhard wrote:
> 
> On 19.01.22 10:10, Kevin Buckley wrote:
>> That suggests that one could take the blacklist lines
>> and simply pre-populate the
>> 
>> sshguard4.xml
>> 
>> IPSet file, before starting SSHGuard, however I am not sure
>> what SSHGuard would do with existing entries if, on starting,
>> it finds that the sshguard4 IPSet exists, and already has
>> entries.
> 
> This would be another option, but seems rather brittle to me. a) it
> messes with firewalld-internal config files that might or might not stay
> in that location/format and b) actually getting firewalld to read those
> files means triggering a global reload which might or might not
> influence other things the user has been doing in the meantime.

Yes.

It didn't "feel quite right" to me either, but it does separate
all ongoing blocks, initiated by SSHGuard, from already known to
FirewallD drops.

Christpher also writes

> IMO there is no reason not to use the --permanent flag when setting the 
> entries, but that should be checked to be sure. Main difference is that 
> the banned IPs will survive a restart, while sshguards banlist doesn't?, 
> which might lead to IPs not being unbanned. But that is easily remedied 
> by letting SSHGuard wipe the sets on startup. It might already, I can check.


So here's my experience with that:

I start SSHGuard, against a FirewallD that has never seen SSHGUard,
with a blaklist containing 28793, and with the "--permanent" option
added to the SSHGuard's ads/remove block functions, as detailed
previously.

It took around SEVEN hours for SSHGuard to read the entries from the
blacklist and add them into FirewallD.

Jan 19 13:02:11 host systemd[1]: Started SSHGuard - blocks brute-force login at>
Jan 19 13:02:11 host sshguard[25082]: blacklist: blocking 28793 addresses
Jan 19 20:07:32 host sshguard[25082]: Now monitoring attacks.

As may have been already documented, each addition sees the exiting IPSet
file backed up on disk, before the new entry is added.

I then stop SSHGuard.

The sshguard4 IPSet, as known to FirewallD is still there.

I remove the SSHGuard blacklist.

I restart SSHGuard.

10:51:40 host systemd[1]: Started SSHGuard - blocks brute-force login attempts.
10:51:40 host sshguard[15354]: blacklist: blocking 0 addresses
10:51:40 host sshguard[15354]: Now monitoring attacks.

The sshguard4 IPSet, as known to FirewallD is still there.

So now we have a situation where FirewallD "knows about" all 28793 original
blacklisted entries, plus any new blocks before SSHGuard was stopped, but
SSHGuard has no record of anything in the sshguard4 IPSet.

Then again, if FirewallD is dropping connections from all 28793 original
blacklisted entries, plus any new blocks, does SSHGuard need to know that
it was the original source of a given block/drop ?

I think this comes down to whether SSHGuard, or more correctly its blacklist,
has to be the "sole arbiter" of what gets blocked, or whether it is seen as
something that merely adds to, or removes from, whatever system firewalling
mechanisms are in place.

The issue for me now becomes one of removing blocks known to SSHGuard
that are also known to FirewallD.

If this can only occur via a re-reading of a blacklist file from which
"to be unblocked" entries have been removed, then it's going to take
too long to be operationally viable.

If, on other hand, unblocking is handled via direct interaction with
FirewallD, and so without needing to re-read its blacklist file, it
becomes workable.

Not sure what that means for the SSHGuard blacklist though.

Maybe there's a case for two blacklists, one that only gets read at
startup and which gets compared with existing rules in the OS firewall
and an "operational" blacklist that collects any new blocks after the
SSHGuard starts.

Kevin