sshguard-users — User questions and answers, bugs, and general support

[SSHGuard-users] Messages logged to root xterm

[Doug D. <do...@sa...>]

The system in question is a FreeBSD jail using sshguard-2.1.0_1 and inet 
as the attacks on all jails are not the same. Nor do I have the same error 
weights on all the jails within a single host.

On the first root console I'm getting the following types of messages, nay 
several hundred per 24 hour period:

Could not resolve 'netcupde.tor-exit.de' to address
Could not resolve 'netcupde.tor-exit.de' to address
Could not resolve 'netcupde.tor-exit.de' to address
Could not resolve '207.188.84.69.tor.pathcom.com' to address
Could not resolve '207.188.84.69.tor.pathcom.com' to address
Could not resolve 'vps-917b9a34.vps.ovh.ca' to address
Could not resolve 'netcupde.tor-exit.de' to address
Could not resolve '207.188.84.69.tor.pathcom.com' to address

In auth.log I can find corresponding entries:

Mar  1 22:08:40 host1 sshd[40511]: error: PAM: authentication error for root from netcupde.tor-exit.de
Mar  1 22:08:51 host1 sshd[40523]: refused connect from vps-917b9a34.vps.ovh.ca (51.222.15.164)
Mar  1 23:27:47 host1 sshd[88474]: refused connect from 207.188.84.69.tor.pathcom.com (207.188.84.69)
Mar  1 23:27:48 host1 sshd[88458]: error: PAM: authentication error for root from 207.188.84.69.tor.pathcom.com

sshguard blocks and refuses about 50,000 ssh attacks/per day so do not 
think all attacks with DNS issues get written out to the terminal. The 
'Could not resolve' errors are not logged, they are just written to the 
xterm. There is too much volume to be sure if any of these messages 
come from the refused group. Counts for a typical day:

    1       => Accepted publickey
    15      => Failed keyboard-interactive/pam
    15      => Postponed keyboard-interactive
    19      => Bad protocol
    45      => error: maximum
    140     => error: PAM:
    175     => Disconnected from
    175     => Received disconnect
    301     => Invalid user
    430     => Did not
    56773   => refused connect
    58089 total attempts

This particular server has attracted the attention of China and other bad 
actors so about 1M attacks per day are blocked by ipfw from the host.

All of this is working as it should and is effectly countering a 24/7 
denial of service to this system. My only question is are the messages to 
the xterm coming from DNS errors within sshguard and, can I configure 
around that?

Thanks for any assistance/and or thoughts

Doug

_____
Douglas Denault
http://www.safeport.com
do...@sa...
Voice: 301-217-9220
   Fax: 301-217-9277

[SSHGuard-users] SSHGuard 2.4.2 around the corner

[Kevin Z. <kev...@gm...>]

Hi everyone,

Regular releases for SSHGuard have slowed down given less recent 
changes. However, I'm thinking about cutting 2.4.2 soon to reduce some 
SSH false positives and update a signature for Postfix.

If you run from Git, your problem reports running the latest version are 
certainly appreciated.

And, if there are new signatures that you have patches and tests ready 
for, they can probably make 2.4.2.

I will probably go cut 2.4.2 next weekend, 3/6.

Cheers,
Kevin

Re: [SSHGuard-users] Feature request and suggested patch to merge attacks from subnets

[hvjunk <hv...@gm...>]

I do agree that with IPv6, you ctually want to score and block based on /64 (perhaps /80 minimum) to decide to block as that is the size allocated to a user/client/host typically. For memory/etc. reasons, I’d say to use a (variable?) minimum subnet size and when there had been $threshold attacks from that subnet, block that subnet, you could log it IPv6 addresses, but block on the subnet.

going bigger than /64 (and similar for IPv4), a sort of AI/learning mechanism where you should be looking at the attacks, and start to agregate from the bottom up, ie. if half the address space generated attacks originate in a single /24 (ie. 128 of 256) then it’s a good idea to block the /24. 

My concern with the above: it’ll either be crude (and blocking too many false positives) or too fine grained overburdening the CPU.
In essence, something I’ve been thinking about, is sshguard pulling (and submitting) from (configurable) servers IP list of attacks, then that server can correlate and aggregate/etc. 

> On 10 Jan 2021, at 12:51 , Testudo Aquatilis <tes...@po...> wrote:
> 
> Hello James,
> 
> I agree with you regarding IPv4. Here typically a machine gets a single address and subnet blocking is done manually after seeing a larger amount of attacks from a given subnet. But with IPv6 even single machines often get a /64 prefix assigned and thus can choose one of so many addresses to send from, that blocking could be ineffective or even circumvented.
> 
> My main goal was to allow this for IPv6, IPv4 was just added for consistency. But to not break existing functionality I would then suggest a second set of config variables and command line switches for the merge-prefix size (which then should be >= the existing subnet to block value).
> 
> So here an updated patch providing the attack-merge feature as separate option without breaking existing functionality (added -m/-M switches for the merge-prefix size in blocker). So if left at default sshguard behaves as before and if IPV6_MERGE_PREFIX and IPV6_SUBNET are set to e.g. 64 the described merging can be activated based on the assumption that an individual attacker has a /64 prefix assigned.
> 
> Regards,
> 
> Andreas
> 
> Am 10.01.21 um 01:22 schrieb James Harris:
>> Andreas,
>> 
>> I have been thinking about this type of change for a while. I don't know that threats come from clean subnets of similar sizes.  My guess is that threats are more strongly correlated to autonomous systems than just subnets.  I would guess fixed subnet sizes will just limit the number of rules proportional to the size of the subnetting.  I wonder if one approach might be to score based on AS then block all IPs associated with that AS. Similarly instead of a fixed subnet size pick some weights that allow a bigger subnet if there are enough attacks compared to the number of IPs represented in that group.  As these weights and subnet sizes vs number of firewall rules might need a significant amount of tuning I was thinking this might be an offline operation where the admin needs to approve the proposed ruleset.   
>> 
>> It might be better to gather real log data, possibly filtered to just remote IP for privacy reasons. Then simulate the different approaches on those data sets and determine what number of rules we get.  Finally run those rules through a few of the popular backend firewalls to determine performance impact.  
>> 
>> On Sat, Jan 9, 2021 at 2:40 PM Testudo Aquatilis <tes...@po...> wrote:
>> Hello,
>> 
>> as sshguard already has the feature to block subnets after an attack, I
>> would suggest to also merge attacks of the configured subnets.
>> Especially for IPv6 this would be quite useful because attackers might
>> have larger subnets available and could otherwise flood with attacks
>> from individual IPv6 addresses without getting blocked, as attacks are
>> counted individually.
>> 
>> The attached patch implements this to the best of my knowledge, so a
>> review would not harm. It basically uses arpa/inet.h functions, which
>> are also used in sshguard_whitelist.c. It parses the IP address into
>> integer format, applies the mask and writes the resulting address back
>> before further handling the attack.
>> 
>> The patch does what I would like to have as behavior when setting the
>> subnet config-variables, so using the same subnet-size for blocking and
>> merging is a feature from my point of view. But if this conflicts with
>> other use-cases, it might be considered to have 2 separate subnet-size
>> command-line flags and config variables for merging and for blocking.
>> 
>> Best regards,
>> Andreas
>> _______________________________________________
>> sshguard-users mailing list
>> ssh...@li...
>> https://lists.sourceforge.net/lists/listinfo/sshguard-users
>> 
>> 
>> -- 
>> James Harris
>> Software Engineer
>> jam...@gm...
> <0001-Blocker-allow-merging-of-attacks-from-subnets.patch>_______________________________________________
> sshguard-users mailing list
> ssh...@li...
> https://lists.sourceforge.net/lists/listinfo/sshguard-users

Re: [SSHGuard-users] Parser support for nextcloud login failure

[Testudo A. <tes...@po...>]

fix: forgot to add service name.

Am 10.01.21 um 16:43 schrieb Testudo Aquatilis:
> Hello,
> 
> one more submission request: support for nextcloud login failures based 
> on trial+error logfile + nextcloud documentation for fail2ban regexps 
> (https://docs.nextcloud.com/server/20/admin_manual/installation/harden_server.html#setup-a-filter-and-a-jail-for-nextcloud). 
> 
> 
> Regards,
> 
> Andreas
> 

[SSHGuard-users] Parser support for nextcloud login failure

[Testudo A. <tes...@po...>]

Hello,

one more submission request: support for nextcloud login failures based 
on trial+error logfile + nextcloud documentation for fail2ban regexps 
(https://docs.nextcloud.com/server/20/admin_manual/installation/harden_server.html#setup-a-filter-and-a-jail-for-nextcloud).

Regards,

Andreas

Re: [SSHGuard-users] Feature request and suggested patch to merge attacks from subnets

[Testudo A. <tes...@po...>]

Hello James,

I agree with you regarding IPv4. Here typically a machine gets a single 
address and subnet blocking is done manually after seeing a larger 
amount of attacks from a given subnet. But with IPv6 even single 
machines often get a /64 prefix assigned and thus can choose one of so 
many addresses to send from, that blocking could be ineffective or even 
circumvented.

My main goal was to allow this for IPv6, IPv4 was just added for 
consistency. But to not break existing functionality I would then 
suggest a second set of config variables and command line switches for 
the merge-prefix size (which then should be >= the existing subnet to 
block value).

So here an updated patch providing the attack-merge feature as separate 
option without breaking existing functionality (added -m/-M switches for 
the merge-prefix size in blocker). So if left at default sshguard 
behaves as before and if IPV6_MERGE_PREFIX and IPV6_SUBNET are set to 
e.g. 64 the described merging can be activated based on the assumption 
that an individual attacker has a /64 prefix assigned.

Regards,

Andreas

Am 10.01.21 um 01:22 schrieb James Harris:
> Andreas,
>
> I have been thinking about this type of change for a while. I don't 
> know that threats come from clean subnets of similar sizes.  My guess 
> is that threats are more strongly correlated to autonomous systems 
> than just subnets.  I would guess fixed subnet sizes will just limit 
> the number of rules proportional to the size of the subnetting.  I 
> wonder if one approach might be to score based on AS then block all 
> IPs associated with that AS. Similarly instead of a fixed subnet size 
> pick some weights that allow a bigger subnet if there are enough 
> attacks compared to the number of IPs represented in that group.  As 
> these weights and subnet sizes vs number of firewall rules might need 
> a significant amount of tuning I was thinking this might be an offline 
> operation where the admin needs to approve the proposed ruleset.
>
> It might be better to gather real log data, possibly filtered to just 
> remote IP for privacy reasons. Then simulate the different approaches 
> on those data sets and determine what number of rules we get.  Finally 
> run those rules through a few of the popular backend firewalls to 
> determine performance impact.
>
> On Sat, Jan 9, 2021 at 2:40 PM Testudo Aquatilis 
> <tes...@po... <mailto:tes...@po...>> wrote:
>
>     Hello,
>
>     as sshguard already has the feature to block subnets after an
>     attack, I
>     would suggest to also merge attacks of the configured subnets.
>     Especially for IPv6 this would be quite useful because attackers might
>     have larger subnets available and could otherwise flood with attacks
>     from individual IPv6 addresses without getting blocked, as attacks are
>     counted individually.
>
>     The attached patch implements this to the best of my knowledge, so a
>     review would not harm. It basically uses arpa/inet.h functions, which
>     are also used in sshguard_whitelist.c. It parses the IP address into
>     integer format, applies the mask and writes the resulting address back
>     before further handling the attack.
>
>     The patch does what I would like to have as behavior when setting the
>     subnet config-variables, so using the same subnet-size for
>     blocking and
>     merging is a feature from my point of view. But if this conflicts with
>     other use-cases, it might be considered to have 2 separate subnet-size
>     command-line flags and config variables for merging and for blocking.
>
>     Best regards,
>     Andreas
>     _______________________________________________
>     sshguard-users mailing list
>     ssh...@li...
>     <mailto:ssh...@li...>
>     https://lists.sourceforge.net/lists/listinfo/sshguard-users
>     <https://lists.sourceforge.net/lists/listinfo/sshguard-users>
>
>
>
> -- 
> James Harris
> Software Engineer
> jam...@gm... <mailto:jam...@gm...>

Re: [SSHGuard-users] Feature request and suggested patch to merge attacks from subnets

[lists <li...@la...>]

<html><head><style id="outgoing-font-settings">#response_container_BBPPID{font-family: initial; font-size:initial; color: initial;}</style></head><body style="background-color: rgb(255, 255, 255); background-image: initial; line-height: initial;"><div id="response_container_BBPPID" style="outline:none;" dir="auto" contenteditable="false"> <div name="BB10" id="BB10_response_div_BBPPID" dir="auto" style="width:100%;"> I would really like a logging feature and just create a static block list after examination the source. If you use PKI nobody is going to get through. So it becomes a matter of how much CPU effort you spend on blocking versus just let the OS reject the fool. On Centos updating the firewall is a CPU drain.&nbsp;</div><div name="BB10" id="BB10_response_div_BBPPID" dir="auto" style="width:100%;"><br></div><div name="BB10" id="BB10_response_div_BBPPID" dir="auto" style="width:100%;">I use static blocking lists now for my web and email server. Firewalld uses a fair amount of RAM but very little CPU once the blocking list is processed. With foreign IP space blocked virtually&nbsp; no one messes with my mail server. I have another list of hosting/VPS space that I block from the web server and mail except for port 25. It takes a week these days to gather enough IPs to bother investigating. I find maybe half a dozen companies to block. I block the hackers just to be safe since you never know what zero&nbsp; day is out there.&nbsp;</div>                                                                                                                                      <div name="BB10" id="response_div_spacer_BBPPID" dir="auto" style="width:100%;"> <br style="display:initial"></div><div name="BB10" id="response_div_spacer_BBPPID" dir="auto" style="width:100%;">I find it odd how many no name cloud companies there are out there. There can't possibly be the need for so many players.&nbsp;</div> <div id="blackberry_signature_BBPPID" name="BB10" dir="auto">     <div id="_signaturePlaceholder_BBPPID" name="BB10" dir="auto"></div> </div></div><div id="_original_msg_header_BBPPID" dir="auto">                                                                                                                                             <table width="100%" style="border-spacing: 0px; display: table; outline: none;" contenteditable="false"><tbody><tr><td colspan="2" style="padding: initial; font-size: initial; text-align: initial;">                           <div style="border-right: none; border-bottom: none; border-left: none; border-image: initial; border-top: 1pt solid rgb(181, 196, 223); padding: 3pt 0in 0in; font-family: Tahoma, &quot;BB Alpha Sans&quot;, &quot;Slate Pro&quot;; font-size: 10pt;">  <div id="from"><b>From:</b> jam...@gm...</div><div id="sent"><b>Sent:</b> January 9, 2021 4:23 PM</div><div id="to"><b>To:</b> tes...@po...</div><div id="reply_to"><b>Reply-to:</b> Jam...@gm...</div><div id="cc"><b>Cc:</b> ssh...@li...</div><div id="subject"><b>Subject:</b> Re: [SSHGuard-users] Feature request and suggested patch to merge attacks from subnets</div></div></td></tr></tbody></table> <br> </div><!--start of _originalContent --><div name="BB10" dir="auto" style="background-image: initial; line-height: initial; outline: none;" contenteditable="false"><div dir="ltr">Andreas,<div><br></div><div>I have been thinking about this type of change for a while. I don't know that threats come from clean subnets of similar sizes.&nbsp; My guess is that threats are more strongly&nbsp;correlated to autonomous systems than just subnets.&nbsp; I would guess fixed subnet sizes will just limit the number of rules proportional&nbsp;to the size of the subnetting.&nbsp; I wonder if one approach might be to score based on AS then block all IPs associated&nbsp;with that AS. Similarly instead of a fixed subnet size pick some weights that allow a bigger subnet if there are enough attacks compared to the number of IPs represented in that group.&nbsp; As these weights and subnet sizes vs number of firewall rules might&nbsp;need a significant amount of tuning I was thinking this might be an offline operation where the admin needs to approve the proposed ruleset.&nbsp; &nbsp;</div><div><br></div><div>It might be better to gather real log data, possibly filtered to just remote IP for privacy reasons. Then simulate&nbsp;the different approaches on those data sets and determine what number of rules we get.&nbsp; Finally run those rules through a few of the popular backend firewalls to determine performance impact.&nbsp;&nbsp;</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jan 9, 2021 at 2:40 PM Testudo Aquatilis &lt;<a href="mailto:tes...@po...">tes...@po...</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb( 204 , 204 , 204 );padding-left:1ex">Hello,<br>
<br>
as sshguard already has the feature to block subnets after an attack, I<br>
would suggest to also merge attacks of the configured subnets.<br>
Especially for IPv6 this would be quite useful because attackers might<br>
have larger subnets available and could otherwise flood with attacks<br>
from individual IPv6 addresses without getting blocked, as attacks are<br>
counted individually.<br>
<br>
The attached patch implements this to the best of my knowledge, so a<br>
review would not harm. It basically uses arpa/inet.h functions, which<br>
are also used in sshguard_whitelist.c. It parses the IP address into<br>
integer format, applies the mask and writes the resulting address back<br>
before further handling the attack.<br>
<br>
The patch does what I would like to have as behavior when setting the<br>
subnet config-variables, so using the same subnet-size for blocking and<br>
merging is a feature from my point of view. But if this conflicts with<br>
other use-cases, it might be considered to have 2 separate subnet-size<br>
command-line flags and config variables for merging and for blocking.<br>
<br>
Best regards,<br>
Andreas<br>
_______________________________________________<br>
sshguard-users mailing list<br>
<a href="mailto:ssh...@li...">ssh...@li...</a><br>
<a href="https://lists.sourceforge.net/lists/listinfo/sshguard-users">https://lists.sourceforge.net/lists/listinfo/sshguard-users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">James Harris<br>Software Engineer<br><a href="mailto:jam...@gm...">jam...@gm...</a><br></div>
<!--end of _originalContent --></div></body></html>

Re: [SSHGuard-users] Feature request and suggested patch to merge attacks from subnets

[James H. <jam...@gm...>]

Andreas,

I have been thinking about this type of change for a while. I don't know
that threats come from clean subnets of similar sizes.  My guess is that
threats are more strongly correlated to autonomous systems than just
subnets.  I would guess fixed subnet sizes will just limit the number of
rules proportional to the size of the subnetting.  I wonder if one approach
might be to score based on AS then block all IPs associated with that AS.
Similarly instead of a fixed subnet size pick some weights that allow a
bigger subnet if there are enough attacks compared to the number of IPs
represented in that group.  As these weights and subnet sizes vs number of
firewall rules might need a significant amount of tuning I was thinking
this might be an offline operation where the admin needs to approve the
proposed ruleset.

It might be better to gather real log data, possibly filtered to just
remote IP for privacy reasons. Then simulate the different approaches on
those data sets and determine what number of rules we get.  Finally run
those rules through a few of the popular backend firewalls to determine
performance impact.

On Sat, Jan 9, 2021 at 2:40 PM Testudo Aquatilis <
tes...@po...> wrote:

> Hello,
>
> as sshguard already has the feature to block subnets after an attack, I
> would suggest to also merge attacks of the configured subnets.
> Especially for IPv6 this would be quite useful because attackers might
> have larger subnets available and could otherwise flood with attacks
> from individual IPv6 addresses without getting blocked, as attacks are
> counted individually.
>
> The attached patch implements this to the best of my knowledge, so a
> review would not harm. It basically uses arpa/inet.h functions, which
> are also used in sshguard_whitelist.c. It parses the IP address into
> integer format, applies the mask and writes the resulting address back
> before further handling the attack.
>
> The patch does what I would like to have as behavior when setting the
> subnet config-variables, so using the same subnet-size for blocking and
> merging is a feature from my point of view. But if this conflicts with
> other use-cases, it might be considered to have 2 separate subnet-size
> command-line flags and config variables for merging and for blocking.
>
> Best regards,
> Andreas
> _______________________________________________
> sshguard-users mailing list
> ssh...@li...
> https://lists.sourceforge.net/lists/listinfo/sshguard-users
>


-- 
James Harris
Software Engineer
jam...@gm...

[SSHGuard-users] Feature request and suggested patch to merge attacks from subnets

[Testudo A. <tes...@po...>]

Hello,

as sshguard already has the feature to block subnets after an attack, I
would suggest to also merge attacks of the configured subnets.
Especially for IPv6 this would be quite useful because attackers might
have larger subnets available and could otherwise flood with attacks
from individual IPv6 addresses without getting blocked, as attacks are
counted individually.

The attached patch implements this to the best of my knowledge, so a
review would not harm. It basically uses arpa/inet.h functions, which
are also used in sshguard_whitelist.c. It parses the IP address into
integer format, applies the mask and writes the resulting address back
before further handling the attack.

The patch does what I would like to have as behavior when setting the
subnet config-variables, so using the same subnet-size for blocking and
merging is a feature from my point of view. But if this conflicts with
other use-cases, it might be considered to have 2 separate subnet-size
command-line flags and config variables for merging and for blocking.

Best regards,
Andreas

Re: [SSHGuard-users] use of IPV4_SUBNET

[Chris J. <joh...@as...>]

FYI, I'm using CentOS 8.2 with the same /24 setting, but I get the expected
results. Small sample from "ipset list":

106.12.127.0/24
106.12.129.0/24
106.12.130.0/24
106.12.13.0/24
106.12.131.0/24
106.12.132.0/24
106.12.133.0/24
106.12.134.0/24
106.12.14.0/24
106.12.144.0/24

It could be something Debian related?

Cheers!

Chris


On Mon, Nov 23, 2020 at 9:56 AM jack keradec <cm...@li...> wrote:

> Hello
>
> As a long term user of sshgard, I upgraded my Debian box to version 2.4.1
> which I compiled,
> and use it with Nftables.
> However, I am quite often attacked by Chinese sites who tries different
> addresses on a same
> subnet, say 212.85.42.Z, Z ranging from 120 to 250 (just an example)
> So I put IPV4_SUBNET=24 in my ssguard.conf but to no avail: addresses are
> baned one by one
> instead of a whole subnet 112.85.42.0/24 as I expected.
>
> Am I doing something wrong or ...?
>
> And thanks for your good job !
> Regards
>

[SSHGuard-users] use of IPV4_SUBNET

[jack k. <cm...@li...>]

Hello

As a long term user of sshgard, I upgraded my Debian box to version 2.4.1 which I compiled, 
and use it with Nftables. 
However, I am quite often attacked by Chinese sites who tries different addresses on a same 
subnet, say 212.85.42.Z, Z ranging from 120 to 250 (just an example)
So I put IPV4_SUBNET=24 in my ssguard.conf but to no avail: addresses are baned one by one
instead of a whole subnet 112.85.42.0/24 as I expected.

Am I doing something wrong or ...?

And thanks for your good job !
Regards 

Re: [SSHGuard-users] Injecting blacklist content into a runing SSHGuard

[Laurence P. <lpe...@op...>]

>> 2. For someone who runs multiple servers, it sounds like addresses that
>> are blacklisted in one place should be blacklisted elsewhere.
>
>This one, well, yeah, we had ideas there too.
>
>We actually started talking about propagating the machine-local
>blacklist info from SSHGuard out towards "the edge", but the
>talking dried up.
>
>Kevin

Propagating toward the edge is pretty trivial.  On my systems I just created a little expect script that could log into my router and block or unblock addresses and hooked it into sshguard's firewall script.  Something similar would likely work for propagating to other servers as well as long as you don't care about the sshguard instances on the other machines managing the blacklist.  Which, considering that anything that makes it clear to the blacklist probably never gets unblocked, there's not much reason to have the local sshguard worry about it.

For a way to propagate temporary blocks without much for code changes, just add a special match pattern for a line that's simply a block request and then set sshguard to watch an extra logfile somewhere.  When the local sshguard blocks something, use a script to snag its block message from the log output, generate a block request, and push it to the special log files on the other machines.  Their local sshguard will then block the offending address temporarily until the number of attacks exceeds whatever those particular nodes have for blacklist threshold.

LMP

Re: [SSHGuard-users] Injecting blacklist content into a runing SSHGuard

[Kevin B. <kev...@gm...>]

On 2020/09/11 02:11, Kevin Zheng wrote:
> On 9/3/20 12:37 AM, Kevin Buckley wrote:
>> What I am thinking about is, rather than combining the two files,
>> I weed out the duplicates from server B and, say, send a SIGnal
>> to SSHGuard that causes it to read new IPs from a known location,
>> poke them into the firewall, and add them to the live blacklist file.
> 
> It sounds like we're trying to accomplish two things here:
> 
> 1. Teach SSHGuard how to re-load a blacklist file while running. It
> doesn't currently know how to do this. This will probably involve a
> not-too-difficult change to sshg-blocker.

Not so sure that is what is required, though I'll happily bow to
your inside knowledge of the application here: for me though:

simply re-reading THE blacklist file opens up all kinds of issues,
not least that SSHGuard could be (should be?) trying to update the
on-disk file with new blocked addresses, just as an external user
tries to add content to it as well, ahead of instigating a re-load.

That's why I feel a separate "injection thread", that allows the
SSHGuard process sole access to its on-disk blaclist file, whilst
then able to accept source info from a second channel, to be a good
thing.

Then again, maybe the external user could add blocks directly into
the "your firewall here" and SSHGuard could be "taught" how to
occasionally sync itself against that?

Just floating ideas though.


> 2. For someone who runs multiple servers, it sounds like addresses that
> are blacklisted in one place should be blacklisted elsewhere.

This one, well, yeah, we had ideas there too.

We actually started talking about propagating the machine-local
blacklist info from SSHGuard out towards "the edge", but the
talking dried up.

Kevin

Re: [SSHGuard-users] Injecting blacklist content into a runing SSHGuard

[Kevin Z. <kev...@gm...>]

On 9/3/20 12:37 AM, Kevin Buckley wrote:
> What I am thinking about is, rather than combining the two files,
> I weed out the duplicates from server B and, say, send a SIGnal
> to SSHGuard that causes it to read new IPs from a known location,
> poke them into the firewall, and add them to the live blacklist file.

It sounds like we're trying to accomplish two things here:

1. Teach SSHGuard how to re-load a blacklist file while running. It
doesn't currently know how to do this. This will probably involve a
not-too-difficult change to sshg-blocker.

2. For someone who runs multiple servers, it sounds like addresses that
are blacklisted in one place should be blacklisted elsewhere.

Perhaps you could have a central syslogd that all your severs log to
where you run a "master" sshg-blocker instance. It could then issue
sshg-fw-style commands to the individual servers via some authenticated
IPC mechanism, be it secure socket, message queue/broker, etc.

It seems that the SSHGuard pieces are there. It would take some glue to
put together a system like this.

Re: [SSHGuard-users] Repeaters

[Jos C. <ssh...@cl...>]

On 6-9-20 18:51, Kevin Zheng wrote:
> On 9/6/20 2:51 AM, Jos Chrispijn wrote:
> Can you show some examples of legitimate SMTP sessions, so that we can
> try to see what the differences are?

Of course, here it is:

- - - start - - -

Sep  6 13:30:48 poseidon postfix/smtpd[25472]: connect from 
mxout1-ec2-va.apache.org[3.227.148.255]
Sep  6 13:30:48 poseidon postfix/smtpd[25472]: Trusted TLS connection 
established from mxout1-ec2-va.apache.org[3.227.148.255]: TLSv1.3 with 
cipher TLS_AES_256_GCM_S>
Sep  6 13:30:51 poseidon postfix/policy-spf[25478]: Policy 
action=PREPEND Received-SPF: pass (httpd.apache.org: Sender is 
authorized to use 'users-return-119859-apac>
Sep  6 13:30:51 poseidon postgrey[1002]: action=pass, reason=triplet 
found, delay=486, client_name=mxout1-ec2-va.apache.org, 
client_address=3.227.148.255, sender=use>
Sep  6 13:30:51 poseidon postfix/smtpd[25472]: E0CBFBFC5: 
client=mxout1-ec2-va.apache.org[3.227.148.255]
Sep  6 13:30:52 poseidon postfix/cleanup[25480]: E0CBFBFC5: 
message-id=<109...@ma...>
Sep  6 13:30:52 poseidon opendkim[999]: E0CBFBFC5: 
mxout1-ec2-va.apache.org [3.227.148.255] not internal
Sep  6 13:30:52 poseidon postfix/qmgr[50777]: E0CBFBFC5: 
from=<users-return-119859-apache=clo...@ht...>, 
size=11517, nrcpt=1 (queue active)
Sep  6 13:30:52 poseidon postfix/smtpd[25472]: disconnect from 
mxout1-ec2-va.apache.org[3.227.148.255] ehlo=2 starttls=1 mail=1 rcpt=1 
data=1 quit=1 commands=7
Sep  6 13:30:52 poseidon clamsmtpd[2406]: 10013A: accepted connection 
from: x.x.x.x
Sep  6 13:30:52 poseidon postfix/smtpd[25482]: connect from 
cloudzeeland.nl[x.x.x.x]
Sep  6 13:30:52 poseidon postfix/smtpd[25482]: 5024DBFC6: 
client=cloudzeeland.nl[x.x.x.x], orig_queue_id=E0CBFBFC5, 
orig_client=mxout1-ec2-va.apache.org[3.227.14>
Sep  6 13:30:52 poseidon postfix/cleanup[25480]: 5024DBFC6: 
message-id=<109...@ma...>
Sep  6 13:30:52 poseidon clamsmtpd[2406]: 10013A: 
from=users-return-119859-apache=clo...@ht..., 
to=ap...@cl..., status=CLEAN
Sep  6 13:30:52 poseidon postfix/qmgr[50777]: 5024DBFC6: 
from=<users-return-119859-apache=clo...@ht...>, 
size=11765, nrcpt=1 (queue active)
Sep  6 13:30:52 poseidon postfix/smtp[25481]: E0CBFBFC5: 
to=<xxxxxx@xxxx.xx>, relay=x.x.x.x[x.x.x.x]:10025, delay=3.3, 
delays=3/0.02/0.14/0.14, dsn=2>
Sep  6 13:30:52 poseidon postfix/qmgr[50777]: E0CBFBFC5: removed
Sep  6 13:30:52 poseidon postfix/smtpd[25482]: disconnect from 
cloudzeeland.nl[x.x.x.x] ehlo=1 xforward=2 mail=1 rcpt=1 data=1 quit=1 
commands=7
Sep  6 13:30:59 poseidon postfix/local[25483]: 5024DBFC6: 
to=<jo...@cl...>, orig_to=<xxxxxx@xxxx.xx>, relay=local, 
delay=7.4, delays=0.14/0.01/0/7.2,>
Sep  6 13:30:59 poseidon postfix/qmgr[50777]: 5024DBFC6: removed

- - - end - - -

Best, Jos

-- With both feed on the ground you will never make a step forward

Re: [SSHGuard-users] Repeaters

[Kevin Z. <kev...@gm...>]

On 9/6/20 2:51 AM, Jos Chrispijn wrote:
> Can you tell me how I can trigger SSHGuard blocking this ip address' action?
> 
> Sep  6 11:47:41 poseidon postfix/postscreen[14766]: HANGUP after 0.07
> from [158.174.61.67]:50969 in tests after SMTP handshake
> Sep  6 11:47:41 poseidon postfix/postscreen[14766]: DISCONNECT
> [158.174.61.67]:50969
> Sep  6 11:47:42 poseidon postfix/postscreen[14766]: CONNECT from
> [158.174.61.67]:54563 to [10.10.10.36]:25
> Sep  6 11:47:42 poseidon postfix/postscreen[14766]: PREGREET 14 after
> 0.04 from [158.174.61.67]:54563: EHLO ylmf-pc\r\n
We can determine if one of these lines only appear in "attacks" and
trigger based on that line.

Can you show some examples of legitimate SMTP sessions, so that we can
try to see what the differences are?

Thanks,
Kevin

[SSHGuard-users] Repeaters

[Jos C. <ssh...@cl...>]

Hi All,

Can you tell me how I can trigger SSHGuard blocking this ip address' action?

Sep  6 11:47:41 poseidon postfix/postscreen[14766]: HANGUP after 0.07 
from [158.174.61.67]:50969 in tests after SMTP handshake
Sep  6 11:47:41 poseidon postfix/postscreen[14766]: DISCONNECT 
[158.174.61.67]:50969
Sep  6 11:47:42 poseidon postfix/postscreen[14766]: CONNECT from 
[158.174.61.67]:54563 to [10.10.10.36]:25
Sep  6 11:47:42 poseidon postfix/postscreen[14766]: PREGREET 14 after 
0.04 from [158.174.61.67]:54563: EHLO ylmf-pc\r\n
Sep  6 11:47:42 poseidon postfix/postscreen[14766]: HANGUP after 0.07 
from [158.174.61.67]:54563 in tests after SMTP handshake
Sep  6 11:47:42 poseidon postfix/postscreen[14766]: DISCONNECT 
[158.174.61.67]:54563
Sep  6 11:47:42 poseidon postfix/postscreen[14766]: CONNECT from 
[158.174.61.67]:60033 to [10.10.10.36]:25
Sep  6 11:47:42 poseidon postfix/postscreen[14766]: PREGREET 14 after 
0.04 from [158.174.61.67]:60033: EHLO ylmf-pc\r\n
Sep  6 11:47:42 poseidon postfix/postscreen[14766]: HANGUP after 0.07 
from [158.174.61.67]:60033 in tests after SMTP handshake
Sep  6 11:47:42 poseidon postfix/postscreen[14766]: DISCONNECT 
[158.174.61.67]:60033
Sep  6 11:47:42 poseidon postfix/postscreen[14766]: CONNECT from 
[158.174.61.67]:50116 to [10.10.10.36]:25
Sep  6 11:47:42 poseidon postfix/postscreen[14766]: PREGREET 14 after 
0.04 from [158.174.61.67]:50116: EHLO ylmf-pc\r\n
Sep  6 11:47:42 poseidon postfix/postscreen[14766]: HANGUP after 0.08 
from [158.174.61.67]:50116 in tests after SMTP handshake
Sep  6 11:47:42 poseidon postfix/postscreen[14766]: DISCONNECT 
[158.174.61.67]:50116
Sep  6 11:47:42 poseidon postfix/postscreen[14766]: CONNECT from 
[158.174.61.67]:55868 to [10.10.10.36]:25
Sep  6 11:47:43 poseidon postfix/postscreen[14766]: PREGREET 14 after 
0.04 from [158.174.61.67]:55868: EHLO ylmf-pc\r\n
Sep  6 11:47:43 poseidon postfix/postscreen[14766]: HANGUP after 0.07 
from [158.174.61.67]:55868 in tests after SMTP handshake
Sep  6 11:47:43 poseidon postfix/postscreen[14766]: DISCONNECT 
[158.174.61.67]:55868
Sep  6 11:47:43 poseidon postfix/postscreen[14766]: CONNECT from 
[158.174.61.67]:63038 to [10.10.10.36]:25
Sep  6 11:47:43 poseidon postfix/postscreen[14766]: PREGREET 14 after 
0.03 from [158.174.61.67]:63038: EHLO ylmf-pc\r\n
Sep  6 11:47:43 poseidon postfix/postscreen[14766]: HANGUP after 0.07 
from [158.174.61.67]:63038 in tests after SMTP handshake
Sep  6 11:47:43 poseidon postfix/postscreen[14766]: DISCONNECT 
[158.174.61.67]:63038
Sep  6 11:47:44 poseidon postfix/postscreen[14766]: CONNECT from 
[158.174.61.67]:53589 to [10.10.10.36]:25
Sep  6 11:47:44 poseidon postfix/postscreen[14766]: PREGREET 14 after 
0.04 from [158.174.61.67]:53589: EHLO ylmf-pc\r\n
Sep  6 11:47:44 poseidon postfix/postscreen[14766]: HANGUP after 0.08 
from [158.174.61.67]:53589 in tests after SMTP handshake
Sep  6 11:47:44 poseidon postfix/postscreen[14766]: DISCONNECT 
[158.174.61.67]:53589
Sep  6 11:47:45 poseidon postfix/postscreen[14766]: CONNECT from 
[158.174.61.67]:56945 to [10.10.10.36]:25
Sep  6 11:47:45 poseidon postfix/postscreen[14766]: PREGREET 14 after 
0.04 from [158.174.61.67]:56945: EHLO ylmf-pc\r\n
Sep  6 11:47:45 poseidon postfix/postscreen[14766]: HANGUP after 0.08 
from [158.174.61.67]:56945 in tests after SMTP handshake
Sep  6 11:47:45 poseidon postfix/postscreen[14766]: DISCONNECT 
[158.174.61.67]:56945
Sep  6 11:47:45 poseidon postfix/postscreen[14766]: CONNECT from 
[158.174.61.67]:61645 to [10.10.10.36]:25
Sep  6 11:47:45 poseidon postfix/postscreen[14766]: PREGREET 14 after 
0.04 from [158.174.61.67]:61645: EHLO ylmf-pc\r\n
Sep  6 11:47:45 poseidon postfix/postscreen[14766]: HANGUP after 0.07 
from [158.174.61.67]:61645 in tests after SMTP handshake
Sep  6 11:47:45 poseidon postfix/postscreen[14766]: DISCONNECT 
[158.174.61.67]:61645
Sep  6 11:47:45 poseidon postfix/postscreen[14766]: CONNECT from 
[158.174.61.67]:51261 to [10.10.10.36]:25
Sep  6 11:47:45 poseidon postfix/postscreen[14766]: PREGREET 14 after 
0.04 from [158.174.61.67]:51261: EHLO ylmf-pc\r\n
Sep  6 11:47:46 poseidon postfix/postscreen[14766]: HANGUP after 0.08 
from [158.174.61.67]:51261 in tests after SMTP handshake

-- cut ---

Thanks for your reply!
Best, Jos

-- With both feed on the ground you will never make a step forward

[SSHGuard-users] Injecting blacklist content into a runing SSHGuard

[Kevin B. <kev...@gm...>]

Apologies if I have missed details about this functionality in the docs.
I think the "log polling" code may already be where one would want to
start from but my reading suiggests that you would have to start SSHGuard
in that mode,  and there are caveats about doing both polling file and
consuming "system logs".

The scenario:

I start up SSH Guard on server A.

At some time later, a colleague brings me a blacklist from another
server, say server B.

Clearly, I can take  server A's live blacklist, combine* it with
server B's, stop server A's SSHGuard, put the combined file in
place and restart SSHGuard.

* combine, as in: weed out duplicates; go for the earliest seen
    for any IP addresses in both files, etc, whatever

What I am thinking about is, rather than combining the two files,
I weed out the duplicates from server B and, say, send a SIGnal
to SSHGuard that causes it to read new IPs from a known location,
poke them into the firewall, and add them to the live blacklist file.

I thought about an SSHGuard "utility" that takes a blacklist file
and creates a set of IPTables (if that's your poison) commands
that one then could poke into the firewall, however that sees
your on-disk blacklist file no-longer consistent with the rules
on the SSHGuard IPT Chain.

Any thoughts or anything similar out there?
Kevin

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Christopher E. <ce...@lc...>]

On 2020-08-31 20:36, Kevin Zheng wrote:
> Just to be clear, does "one-second batch" collect inputs from stdin 
> over
> one second, then issues one large command to the backend?

Yes. The fw_block/release() functions collect the inputs they receive 
and issue one big command to the backend if 1s has elapsed since the 
last time they did that. Technically, in this test there were 2 
commands, because all tested backends have to deal with IPv4 and IPv6 
separately.

> If your list is 6000 addresses, and attacks come in at 10 attacks/sec,
> and the backends are not CPU-limited (which they might be), shouldn't
> this total 600 seconds?

Yes, but that is not due to the backend, but to a badly designed 
experiment: The while loop issuing the commands doesn't actually 
complete in 600s, it take longer. I think the issue is that the sleep 
command is not that precise, as the total accumulated "surplus" depends 
on the sleep value.
Regardless, the execution times are extremely reproducible (both for the 
issuing loop on its own & for loop plus backend), so I don't think it 
matters here (other than 10/s being actually "nominally 10/s").

> For now, I think it would also be useful to clearly document that
> firewalld is slow. I wonder what upstream will say what you're using
> firewalld for, heh.

Execution speed is probably not really a focus for them, but I'm sort of 
hoping that there are some straightforward bottlenecks that simply 
nobody has bothered to identify yet. And to be fair, only being able to 
add ~5 IPs per second to a set is REALLY slow.

> Regards,
> Kevin

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Christos C. <ch...@cr...>]

> I can test pf and ipfw. If need be, they can also be separate backends
> (currently they share sshg-fw.in, which does not need to be the case).

ipfw adds 2326 IPs / second (one by one) in a ipfw table using Intel i7-6700 CPU.

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Kevin Z. <kev...@gm...>]

On 8/31/20 5:12 AM, Christopher Engelhard wrote:
> I did some more testing over the weekend with the nftables & iptables
> backends as well. I didn't extensively test ipset, but it can add 6000
> IPs in 5s at maximum throughput. The guest runs Fedora, where firewalld
> uses nftables as the backend, so the nft-sets backend is the relevant
> comparison.

This comparison is great work and gives us pretty useful information.
Thank you for doing this research.

> Test 1, maximum throughput:
> Just dumping the list into the backend via 'cat iplist | <backend>':
> 
> - 2.4.1 backends:
> firewalld: ~1400s to complete, 50% CPU (i.e. 1 core maxed out)
> nft-sets: ~50s to complete, 50% CPU
> iptables: ~22s to complete, 50% CPU
> ipset: ~ 5s to complete, 50% CPU
> - 1-second-batch: (i.e. one command in this case)
> firewalld: ~35s to complete
> nft-sets: 1-2s to complete
> iptables: 1-2s to complete

Just to be clear, does "one-second batch" collect inputs from stdin over
one second, then issues one large command to the backend?

> Test 2, CPU load at 10 blocks/second:
> Dumping the list via 'while read line; do echo $line; sleep 0.1; done <
> iplist' | <backend>:
> - 2.4.1 backends:
> firewalld: ~1400s to complete, 50% CPU (i.e. 1 core)
> nft-sets: ~640s to complete, ~22% CPU
> iptables: ~640s to complete, ~12% CPU
> - 1-second-batch:
> firewalld: ~640s to complete, ~27% CPU
> nft-sets: ~640s to complete ~4% CPU
> iptables: ~640s to complete ~4% CPU

If your list is 6000 addresses, and attacks come in at 10 attacks/sec,
and the backends are not CPU-limited (which they might be), shouldn't
this total 600 seconds? It seems like even the fastest backend, when run
in this mode, can only complete in 640 seconds.

> My suggestion would be to
> a) modify the backends where the underlying command supports it to
> collect block/release commands before sending them to the firewall
> (although the non-firewalld backends can keep up with very high
> blockrates, the grouping still reduces CPU load quite a bit).
This would be great.

> b) suggest that people switch from firewalld to nft-sets/iptables
> backends if performance is still an issue (they can even keep using
> firewalld for non-sshguard stuff)
> c) talk to firewalld upstream about maybe making firewalld more
> efficient ...

For now, I think it would also be useful to clearly document that
firewalld is slow. I wonder what upstream will say what you're using
firewalld for, heh.

> I can create a PR for (a), though someone else will have to test the pf
> and ipfw backends, I don't have any system that uses them. The rest can
> remain unmodified, as the respective commands can't add multiple IPs in
> one call.

I can test pf and ipfw. If need be, they can also be separate backends
(currently they share sshg-fw.in, which does not need to be the case).

Regards,
Kevin

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Christopher E. <ce...@lc...>]

On 30.08.20 23:31, Bu...@Bu... wrote:
> Two thoughts... 25% is an interesting number, it implies either you
> are actually getting 2Core of 2Threads each in your VM (saturating a
> single processor)

Yeah, that seems to be the case. Qemu consistently reports ~50% CPU
usage whenever I saturate any of the backends, and the host also shows 1
core fully utilised.

> Second thought ... rejigger your test and run it to native ipset add 
> commands. That will give you a best-case performance value and
> eliminate the firewall-cmd and dbus overhead. That should be your
> yard-stick.

I did some more testing over the weekend with the nftables & iptables
backends as well. I didn't extensively test ipset, but it can add 6000
IPs in 5s at maximum throughput. The guest runs Fedora, where firewalld
uses nftables as the backend, so the nft-sets backend is the relevant
comparison.

All this was done using a fixed list of 6000 block commands using
randomly generated IPs (roughly 50/50 IPv4/IPv6).

Test 1, maximum throughput:
Just dumping the list into the backend via 'cat iplist | <backend>':

- 2.4.1 backends:
firewalld: ~1400s to complete, 50% CPU (i.e. 1 core maxed out)
nft-sets: ~50s to complete, 50% CPU
iptables: ~22s to complete, 50% CPU
ipset: ~ 5s to complete, 50% CPU
- 1-second-batch: (i.e. one command in this case)
firewalld: ~35s to complete
nft-sets: 1-2s to complete
iptables: 1-2s to complete

- that's 4.3 / 120 / 273 / 1200 blocks per second for the 2.4.1
firewalld/nftables/iptables/ipset backends, and those are indeed roughly
the rates at which the backends starts maxing out the CPU.
- adding all IPs in one command completes more or less instantly with
iptables/nft-sets (ipset can't do that), but takes a whopping 35 seconds
with firewalld. That's .... not very fast.

Test 2, CPU load at 10 blocks/second:
Dumping the list via 'while read line; do echo $line; sleep 0.1; done <
iplist' | <backend>:
- 2.4.1 backends:
firewalld: ~1400s to complete, 50% CPU (i.e. 1 core)
nft-sets: ~640s to complete, ~22% CPU
iptables: ~640s to complete, ~12% CPU
- 1-second-batch:
firewalld: ~640s to complete, ~27% CPU
nft-sets: ~640s to complete ~4% CPU
iptables: ~640s to complete ~4% CPU

The '1s-batch' firewalld backend can deal with ~200 commands per second
before maxing out the CPU.

Given that the nftables commands seem to be slower than iptables/ipset
commands, I also checked what happens when you switch firewalld to use
iptables as the backend, answer, nothing much.

There are two other interaction modes for firewalld that I haven't
tested, direct (allows direct access to the backends' tables/chains) &
passthrough (passes commands unparsed to backend). I don't know if
they're much faster, but I'd be against using them if it can be avoided,
because that would mean that sshguard would have to start detecting
things about firewalld's configuration.


My suggestion would be to
a) modify the backends where the underlying command supports it to
collect block/release commands before sending them to the firewall
(although the non-firewalld backends can keep up with very high
blockrates, the grouping still reduces CPU load quite a bit).
b) suggest that people switch from firewalld to nft-sets/iptables
backends if performance is still an issue (they can even keep using
firewalld for non-sshguard stuff)
c) talk to firewalld upstream about maybe making firewalld more
efficient ...

I can create a PR for (a), though someone else will have to test the pf
and ipfw backends, I don't have any system that uses them. The rest can
remain unmodified, as the respective commands can't add multiple IPs in
one call.

Christopher

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once (Christopher Engelhard)

[<Bu...@Bu...>]

Two thoughts... 25% is an interesting number, it implies either you are
actually getting 2Core of 2Threads each in your VM (saturating a single
processor) or something is seriously restricting your ability to beat a core
to death. VMware statistics would be interesting too.

Second thought ... rejigger your test and run it to native ipset add
commands. That will give you a best-case performance value and eliminate the
firewall-cmd and dbus overhead. That should be your yard-stick.

-----Burton



Date: Fri, 28 Aug 2020 17:18:38 +0200
From: Christopher Engelhard <ce...@lc...>
To: ssh...@li...
Subject: Re: [SSHGuard-users] performance when using firewalld:
	adding/removing many entries at once
Message-ID: <86a...@lc...>
Content-Type: text/plain; charset=utf-8

On 28.08.20 13:26, Christopher Engelhard wrote:
> & I haven't done any benchmarking yet.

OK, did some playing around. All testing with 6000 random IP block requests
at 100/s (i.e. over 1 min) on a 2 core/8GB RAM virtual machine)

Using the firewalld backend of 2.4.1, it takes 24min to add all IPs to the
blocklist, CPU load during that time is fairly consistently 25% for
firewalld & 5-10% for firewalld-cmd.

Using the "collecting" version & collecting requests for 1s doesn't
significantly change the overall load, but the process now completes in just
over 3 min.

Collecting for 5s or 10s reduces this a bit further to ~2:30 min, again with
no significant load reduction in firewalld. Firewall-cmd only causes
significant CPU load whenever it is triggered by the backend.

Given that there's no difference between 5s or 10s grouping in overall
runtime, I'd say that pretty much reflects the speed at which firewalld is
able to add ips to the ipset.

I think the bottleneck is firewalld processing the commands it receives on
DBus, not firewall-cmd sending them off, otherwise I'd expect to see much
less load on firewalld in the first test compared to the later ones.

Christopher

P.S.: The total number of IPs that ended up in the ipsets dropped to
~5200 in the last two tests, so it is possible that I'm running into some
issues with maximum string lengths/command lengths or so. Probably not a
good idea to set it this high outside of testing.

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Christopher E. <ce...@lc...>]

On 28.08.20 13:26, Christopher Engelhard wrote:
> & I haven't done any benchmarking yet.

OK, did some playing around. All testing with 6000 random IP block
requests at 100/s (i.e. over 1 min) on a 2 core/8GB RAM virtual machine)

Using the firewalld backend of 2.4.1, it takes 24min to add all IPs to
the blocklist, CPU load during that time is fairly consistently 25% for
firewalld & 5-10% for firewalld-cmd.

Using the "collecting" version & collecting requests for 1s doesn't
significantly change the overall load, but the process now completes in
just over 3 min.

Collecting for 5s or 10s reduces this a bit further to ~2:30 min, again
with no significant load reduction in firewalld. Firewall-cmd only
causes significant CPU load whenever it is triggered by the backend.

Given that there's no difference between 5s or 10s grouping in overall
runtime, I'd say that pretty much reflects the speed at which firewalld
is able to add ips to the ipset.

I think the bottleneck is firewalld processing the commands it receives
on DBus, not firewall-cmd sending them off, otherwise I'd expect to see
much less load on firewalld in the first test compared to the later ones.

Christopher

P.S.: The total number of IPs that ended up in the ipsets dropped to
~5200 in the last two tests, so it is possible that I'm running into
some issues with maximum string lengths/command lengths or so. Probably
not a good idea to set it this high outside of testing.

Re: [SSHGuard-users] performance when using firewalld: adding/removing many entries at once

[Christopher E. <ce...@lc...>]

I think it's a good idea to try to see if we can make the calls to
firewalld faster, but in the meantime, I've taken a stab at letting the
backend function group requests here [1] (branch: batch-process).

If you're on Fedora, you can install that version from Copr [2] as well.

The backend now collects all requests that come within 1 second and
sends them as one to the fw_block()/fw_release() functions. Those then
can do something smart with that. So far, only the firewalld backend
tries to be smart, all the others just disaggregate the combined request
and then do what they did before.

It seems to work, but I have only tested the firewalld and null backends
& I haven't done any benchmarking yet.

On 27.08.20 21:15, Felix Schwarz wrote:
> I guess I should try to create a test scenario where I call add random IP
> addresses via firewall-cmd and check if I also see high CPU load. Ideally I'd
> see much a lower CPU load - though I'm a bit swamped currently so it'll take a
> few days.

You can use the fakeip.sh [3] script from my fork (or from the doc dir
of the forked sshguard package) to send random block/release requests to
the sshguard backends. There shouldn't be significant overhead in
interacting with firewall-cmd in that manner. Just pipe the output into
/usr/libexec/sshguard/sshg-fw-firewalld.

Christopher

[1]
https://bitbucket.org/lcts/sshguard/branches/compare/lcts/sshguard:batch-process%0Dsshguard/sshguard:master#diff
[2]
https://copr.fedorainfracloud.org/coprs/lcts/fedora-rpm-forks/build/1637820/
[3] https://bitbucket.org/lcts/sshguard/src/batch-process/fakeip.sh