PeerGuardian / Support Requests / #63 pglcmd start or restart random fail in log - works ok after trying many times quickly

jre-phoenix - 2017-02-26

Hi,
no, absolutely no idea. Generally, for Ubuntu I'd recommend to use the pre-built packages from my PPA - but since it was working before I doubt that you have a broken installation.
Did you change any configuration (in pgl or anything networking related)?

~~~~~
..Setting up iptables for INPUT:
....Creating chainiptables: Chain already exists.
~~~~~~
--> The iptables chain pgl_in can't be created because it still exists from a previous run. But ususally on every start pgl creates a file .pglcmd.iptables.remove to remember to delete this chain. Maybe you have installed anything that also takes care of iptables rules and always readds this chain, although it shouldn't.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

I keep seeing a lot of these messages as well:

2017-02-26 02:11:03 MST End: pglcmd restart
Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
pglcmd.wd: iptables setup broken!
In most cases the reason for this problem is that another firewall application
messed up the iptables rules. To avoid this remember to do a "pglcmd restart"
whenever you start, stop or change the configuration of your other firewall
applications.
To disable the watchdog set WATCHDOG="0" in /etc/pgl/pglcmd.conf.
Now doing a restart:

 * Restarting only PeerGuardian Linux pgld

2017-02-26 04:01:04 MST Begin: pglcmd restart_not_wd
Emailing stats   ...done.
Deleting iptables ...
..Executing iptables remove script /var/lib/pgl/.pglcmd.iptables.remove.shAnother app is currently holding the xtables lock. Perhaps you want to use the -w option?
Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
   ...fail!
..Removing iptables remove script /var/lib/pgl/.pglcmd.iptables.remove.sh   ...done.
Iptables deleted.
Stopping pgld   ...done.
Inserting iptables ...
..Setting up iptables for INPUT:
....Creating chainAnother app is currently holding the xtables lock. Perhaps you want to use the -w option?
   ...fail!
Deleting iptables ...
..Executing iptables remove script /var/lib/pgl/.pglcmd.iptables.remove.shAnother app is currently holding the xtables lock. Perhaps you want to use the -w option?
Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
   ...fail!
..Removing iptables remove script /var/lib/pgl/.pglcmd.iptables.remove.sh   ...done.
Iptables deleted.

 * "/usr/local/bin/pglcmd restart_not_wd" failed (Error 8).
Stopping pglcmd.wd now!

Does this look right in terms of the chain pgl has built (sensitive IPs removed):

Chain pgl_fwd (1 references)
target     prot opt source               destination
RETURN     all  --  {MYSERVER's RANGE}    {MYSERVER's RANGE}
RETURN     all  --  0.0.0.0/0            208.67.220.220
RETURN     all  --  0.0.0.0/0            208.67.222.222
DROP       all  --  0.0.0.0/0            0.0.0.0/0            mark match 0xa
NFQUEUE    all  --  0.0.0.0/0            0.0.0.0/0            NFQUEUE num 60091

Chain pgl_in (1 references)
target     prot opt source               destination
RETURN     all  --  {MYSERVER's RANGE}    0.0.0.0/0
RETURN     all  --  0.0.0.0/0            0.0.0.0/0
DROP       all  --  0.0.0.0/0            0.0.0.0/0            mark match 0xa
RETURN     all  --  {MYSERVER's RANGE}    0.0.0.0/0
NFQUEUE    all  --  0.0.0.0/0            0.0.0.0/0            NFQUEUE num 60091

Chain pgl_out (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            {MYSERVER's RANGE}
RETURN     all  --  0.0.0.0/0            208.67.220.220
RETURN     all  --  0.0.0.0/0            208.67.222.222
RETURN     all  --  0.0.0.0/0            0.0.0.0/0
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            mark match 0xa reject-with icmp-port-unreachable
RETURN     all  --  0.0.0.0/0            {MYSERVER's RANGE}
RETURN     udp  --  0.0.0.0/0            0.0.0.0/0            udp
RETURN     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp
NFQUEUE    all  --  0.0.0.0/0            0.0.0.0/0            NFQUEUE num 60091

It looks like my servers keep going down randomly until I flush IPtables (they don't accept any incoming traffic at all). Not sure what's going on, but I am using Fail2Ban, PGL, and am banning IP addresses using a script I've written. Evidently, one of my servers has 11,000 iptables rules after about 8 hours. I'm wondering if maybe I have too many? In any event, right after flushing iptables, pglcmd restart still doesn't want to work.

What's going on when it outputs "fail"? Why would the xtables be locked?

jre-phoenix - 2017-02-27

Another app is currently holding the xtables lock. Perhaps you want to use the -w option? pglcmd.wd: iptables setup broken! In most cases the reason for this problem is that another firewall application messed up the iptables rules.

So that's the issue, something holds a xtables lock (never heard of xtables locks before).

11,000 iptables rules indeed sounds problematic - that's why PG was written years ago to handle large amounts of IPs to be blocked. In the meantime much changed (I'm unfortunately not up to date), but generally I think you might have to decide for one system. So either move your script thing to make PG lists, or look into using some modern implication, e.g. ipset.

Between you issuing the flush-command and the iptables really being flushed might be quite some time. I think the problem is really the 11,000 rules.

About your setup: pgl_fwd will not block anything because in your network traffic will always be sent to or from your hosts, so your rule always matches.

Besides what you posted looks good on a very quick glance. However e.g. in OUTPUT you need a rule that sends traffic to pgl_out, for pgl to work.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Athanasius - 2017-02-27

These days it would be worth looking into nftables instead. It's ramping up to totally replace {ip,ip6,arp,eb}tables, and one thing touted about it is much increased performance of rule-checking, it's not strictly linear like *tables.

Edit: Oops, forgot the URL https://netfilter.org/projects/nftables/

Last edit: Athanasius 2017-02-27

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Eric - 2017-02-28

I wouldn't know what to do without PeerGuardian for my Linux servers, so if this could be updated to work with the latest stuff, that would be awesome! As for who's responsible for breaking iptables... well, it appears to be PeerGuardian or the blocklist I linked to. I disabled PeerGuardian for now until I can figure out what's going on and stopped pulling the list from blocklist.de. I am just using Fail2Ban and a list of manually banned IP addresses I've created.

Maybe PGL should use the -w option if xtables is enabled?

I'd be willing to donate some funds for development... wouldn't be much, but it mgiht help. We very much still need applications like PeerGuardian and PeerBlock (for Windows) these days.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

jre-phoenix - 2017-02-28

No promises at all, since I didn't work on pgl for more than a year and don't know at all if/when I will resume work. On the other side this issue may be a lowhanging fruit to motivate me working on this again. If I set up something for donations I'll tell you ;)

TODO pgl development:
short-run: try "iptables -w"
medium: replace "iptables" with "nft"
long run (probably): replace pgld with ipset

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Eric - 2017-03-11

I converted everything over to PGL's list format and updated Fail2Ban to use the -w flag with iptables. That should remove any conflicts from happening, but I'm still having problems on one of my servers. The other seems to be happy after converting it over the same way.

..Setting up iptables for OUTPUT: ....Creating chainiptables: Chain already exists. [fail] Deleting iptables ...

I'm still running into the above issue where it outputs fail without any information when trying to start or restart pglcmd on the problematic server. It eventually works if I call restart and stop enough times. Here's how I installed and compiled it on Ubuntu 14.04 on the server where it's having problems (I did it a little differently on the server that is working, but I can't remember the differences):

http://blog.eamster.tk/?p=471 (didn't use instructions for 12.04 and older or 16.04)

Last edit: Eric 2017-03-11
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

pglcmd start or restart random fail in log - works ok after trying many...

PeerGuardian - a privacy oriented firewall application

Group

Searches

Help

#63 pglcmd start or restart random fail in log - works ok after trying many times quickly - started recently Ubuntu 14.04 x64

Discussion