sshguard-users

[Sshguard-users] xtables lock

[James H. <jam...@gm...>]

Recently I have seen many of these messages during a boot on my FedoraCore
21 box.

"sshguard: Another app is currently holding the xtables lock. Perhaps you
want to use the -w option?"

It seems libvirtd is execing "/usr/sbin/iptables" after it starts up. I
have tweeked my sshguard.service file for systemd to have
After=sshd.service
After=syslog.target
After=iptables.target
After=ip6tables.target
After=libvirtd.service
After=firewalld.service

But it doesn't look like that is enough.  I suspect libvirtd has 'finished'
starting up but has not completed the iptable changes it requires.  What
about adding "-w" to the iptables firwall commands?  The downside is the
command could block for a while. Should run_command() fork/exec and wait
for a specific timeout before giving up on the command? Is blocking
indefinitely acceptable, or is this all on deck for the libev rewrite?

-- 
James Harris
Software Engineer
jam...@gm...

Re: [Sshguard-users] xtables lock

[Kevin Z. <kev...@gm...>]

Hi James,

Thanks for analyzing and reporting the issue.

On 03/24/2015 16:19, James Harris wrote:
> But it doesn't look like that is enough.  I suspect libvirtd has
> 'finished' starting up but has not completed the iptable changes it
> requires.  What about adding "-w" to the iptables firwall commands?  The
> downside is the command could block for a while. Should run_command()
> fork/exec and wait for a specific timeout before giving up on the
> command? Is blocking indefinitely acceptable, or is this all on deck for
> the libev rewrite?

It doesn't sound like the xtables lock is held for a long time; if
you're feeling adventurous you can try adding "-w" (I can give you a
patch to test if you wish) and see what happens.

If the firewall command blocks, SSHGuard will stop processing new log
entries until system() returns. On a practical note, if xtables is
really locked then returning immediately doesn't gain anything; the next
attack cannot be blocked, so we might as well wait.

This would not be fixed by the libev rewrite, although sometime in the
distant future firewall commands may be executed in a separate process.
However, even this does not solve the underlying issue; although
SSHGuard will continue to run, nothing will be blocked.

I'm not familiar with, nor do I run iptables, but it sounds like adding
"-w" doesn't have huge issues and makes sense.

Thanks,
Kevin Zheng

-- 
Kevin Zheng
kev...@gm... | ke...@kd... | PGP: 0xC22E1090

Re: [Sshguard-users] xtables lock

[James H. <jam...@gm...>]

I created a pull request with the suggested changes. In addition to waiting
for the lock it includes improves start up time as it avoids reverse dns
lookup on all blocked ips. I have been running with these changes for a
while without issue.

I agree if the table is locked most likely any other operation sshguard
attempts will also fail. The question is should sshguard wait indefinitely
to acquire the lock?  I would suggest after a reasonably long timeout it
should log the failure and enter a failed state by exiting. If sshguard was
blocked for a significant period of time without warning one might assume
they are still being actively protected when they are not.

On Tue, Mar 24, 2015 at 7:39 PM, Kevin Zheng <kev...@gm...> wrote:

> Hi James,
>
> Thanks for analyzing and reporting the issue.
>
> On 03/24/2015 16:19, James Harris wrote:
> > But it doesn't look like that is enough.  I suspect libvirtd has
> > 'finished' starting up but has not completed the iptable changes it
> > requires.  What about adding "-w" to the iptables firwall commands?  The
> > downside is the command could block for a while. Should run_command()
> > fork/exec and wait for a specific timeout before giving up on the
> > command? Is blocking indefinitely acceptable, or is this all on deck for
> > the libev rewrite?
>
> It doesn't sound like the xtables lock is held for a long time; if
> you're feeling adventurous you can try adding "-w" (I can give you a
> patch to test if you wish) and see what happens.
>
> If the firewall command blocks, SSHGuard will stop processing new log
> entries until system() returns. On a practical note, if xtables is
> really locked then returning immediately doesn't gain anything; the next
> attack cannot be blocked, so we might as well wait.
>
> This would not be fixed by the libev rewrite, although sometime in the
> distant future firewall commands may be executed in a separate process.
> However, even this does not solve the underlying issue; although
> SSHGuard will continue to run, nothing will be blocked.
>
> I'm not familiar with, nor do I run iptables, but it sounds like adding
> "-w" doesn't have huge issues and makes sense.
>
> Thanks,
> Kevin Zheng
>
> --
> Kevin Zheng
> kev...@gm... | ke...@kd... | PGP: 0xC22E1090
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for
> all
> things parallel software development, from weekly thought leadership blogs
> to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Sshguard-users mailing list
> Ssh...@li...
> https://lists.sourceforge.net/lists/listinfo/sshguard-users
>



-- 
James Harris
Software Engineer
jam...@gm...

Re: [Sshguard-users] xtables lock

[Kevin Z. <kev...@gm...>]

Hi James,

On 03/25/2015 19:08, James Harris wrote:
> I created a pull request with the suggested changes. In addition to
> waiting for the lock it includes improves start up time as it avoids
> reverse dns lookup on all blocked ips. I have been running with these
> changes for a while without issue.  

Thanks for the patch; it's been merged in b30e6a8.

> I agree if the table is locked most likely any other operation sshguard
> attempts will also fail. The question is should sshguard wait
> indefinitely to acquire the lock?  I would suggest after a reasonably
> long timeout it should log the failure and enter a failed state by
> exiting. If sshguard was blocked for a significant period of time
> without warning one might assume they are still being actively protected
> when they are not. 

I'm not sure about exiting; this requires manual intervention to restart
the daemon. I agree that it should indicate failure, perhaps by logging
this to syslog and moving on. The flip side is that if the firewall is
blocked indefinitely, you'll get a lot of messages.

In any case, your fix is better than the status quo :)

Thanks,
Kevin Zheng

-- 
Kevin Zheng
kev...@gm... | ke...@kd... | PGP: 0xC22E1090

Re: [Sshguard-users] xtables lock

[James H. <jam...@gm...>]

Kevin,

Since moving to a systemd based init exiting no longer feels like a huge
problem. It can handle restarting if required but it also can handle exit
codes which allow failures to escalate until resolved.  Spamming the log
with a message every few minutes that something is holding the firewall
lock doesn't feel like such a bad thing especially as new brute force
attempts aren't being dealt with.

On a separate thought. Has anyone thought about being proactive with
blocking? I have noticed attacks can come from IPs in the same class C
address blocks in a small period of time. I was thinking of something like
if X attacks come from an address block (or autonomous system?)  in a
configured window add a temporary rule against that ip block.

One down side is blocked attacks wont generate a blacklist entry. If
temporary rules were used I would want the option to log the number of hits
to the rule at the point it was expired.

My guess is the attackers are finding open proxies in a given range and
then running across them to avoid blocking. Of course they could counter
with creating a large proxy list across a diverse range then choosing a
proxy at random but that might be another battle.


On Wed, Mar 25, 2015 at 5:21 PM, Kevin Zheng <kev...@gm...> wrote:

> Hi James,
>
> On 03/25/2015 19:08, James Harris wrote:
> > I created a pull request with the suggested changes. In addition to
> > waiting for the lock it includes improves start up time as it avoids
> > reverse dns lookup on all blocked ips. I have been running with these
> > changes for a while without issue.
>
> Thanks for the patch; it's been merged in b30e6a8.
>
> > I agree if the table is locked most likely any other operation sshguard
> > attempts will also fail. The question is should sshguard wait
> > indefinitely to acquire the lock?  I would suggest after a reasonably
> > long timeout it should log the failure and enter a failed state by
> > exiting. If sshguard was blocked for a significant period of time
> > without warning one might assume they are still being actively protected
> > when they are not.
>
> I'm not sure about exiting; this requires manual intervention to restart
> the daemon. I agree that it should indicate failure, perhaps by logging
> this to syslog and moving on. The flip side is that if the firewall is
> blocked indefinitely, you'll get a lot of messages.
>
> In any case, your fix is better than the status quo :)
>
> Thanks,
> Kevin Zheng
>
> --
> Kevin Zheng
> kev...@gm... | ke...@kd... | PGP: 0xC22E1090
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for
> all
> things parallel software development, from weekly thought leadership blogs
> to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Sshguard-users mailing list
> Ssh...@li...
> https://lists.sourceforge.net/lists/listinfo/sshguard-users
>



-- 
James Harris
Software Engineer
jam...@gm...

Re: [Sshguard-users] xtables lock

[Kevin Z. <kev...@gm...>]

On 03/25/2015 19:55, James Harris wrote:
> Since moving to a systemd based init exiting no longer feels like a huge
> problem. It can handle restarting if required but it also can handle
> exit codes which allow failures to escalate until resolved.  Spamming
> the log with a message every few minutes that something is holding the
> firewall lock doesn't feel like such a bad thing especially as new brute
> force attempts aren't being dealt with.

For now at least, I think spamming syslog is the better option.

> On a separate thought. Has anyone thought about being proactive with
> blocking? I have noticed attacks can come from IPs in the same class C
> address blocks in a small period of time. I was thinking of something
> like if X attacks come from an address block (or autonomous system?)  in
> a configured window add a temporary rule against that ip block.

It might be a little more complex than that. These days addresses are
allocated from arbitrary size subnets rather than A, B, and C blocks.

> One down side is blocked attacks wont generate a blacklist entry. If
> temporary rules were used I would want the option to log the number of
> hits to the rule at the point it was expired. 

If it's a small enough subnet, perhaps it's better to extend the amount
of time required to trigger a block. In a class C, it's probably true
that some addresses will repeat.

Thanks,
Kevin Zheng

-- 
Kevin Zheng
kev...@gm... | ke...@kd... | PGP: 0xC22E1090