From: <gi1...@gm...> - 2019-10-30 16:00:23
|
Hi All, Thanks for sshguard; I've been using it happily for some time now. I'm writing to report a security issue I noticed recently: In short, blocked attackers are still able to access docker containers. Steps to reproduce: 1. Run any docker container with an exposed port. Personally I'm running https://hub.docker.com/r/jgiannuzzi/gitolite docker run -p 2022:22 jgiannuzzi/gitolite 2. Attack the host machine from some remote until it gets blocked. remote> repeat 10 ssh spam@host ... host> journalctl --unit=sshguard.service ... Oct 30 11:19:20 gi sshguard[962]: Blocking "HOSTIP/32" for 960 secs (4 attacks in 345 secs, after 4 abuses over 2721 secs.) 3. Check that the remote attacker is blocked as expected: remote> ping host (no response) remote> ssh host (no response) 4. The exposed ports from docker containers are still visible to attackers: remote> ssh -p 2022 git@host SUCCEEDED Any container with an exposed port will do. Doesn't have to be the one I used above. I used to use iptables back in the day, but looks like modern systems use nft. I checked out the rules added. Looks like the relevant parts (from nft list ruleset) are: table ip sshguard { set attackers { type ipv4_addr flags interval elements = { REMOTE_IP, ... } } chain blacklist { type filter hook input priority -10; policy accept; ip saddr @attackers drop } } table ip nat { chain PREROUTING { type nat hook prerouting priority -100; policy accept; fib daddr type local counter packets 64 bytes 4890 jump DOCKER } chain INPUT { type nat hook input priority 100; policy accept; } chain POSTROUTING { type nat hook postrouting priority 100; policy accept; oifname != "docker0" ip saddr 172.17.0.0/16 counter packets 0 bytes 0 masquerade meta l4proto tcp ip saddr 172.17.0.3 ip daddr 172.17.0.3 tcp dport 22 counter packets 0 bytes 0 masquerade } chain OUTPUT { type nat hook output priority -100; policy accept; ip daddr != 127.0.0.0/8 fib daddr type local counter packets 0 bytes 0 jump DOCKER } chain DOCKER { iifname "docker0" counter packets 0 bytes 0 return iifname != "docker0" meta l4proto tcp ip daddr HOST_IP tcp dport 2022 counter packets 0 bytes 0 dnat to 172.17.0.3:22 } } I replaced the IP addresses with HOST_IP / REMOTE_IP above. I don't fully understand the sequence above. Perhaps because sshguard adds rules with priority -10 and docker with prority -100? Regardless, the upshot is that docker containers are not protected at all by sshguard. Moreover, if your container runs a ssh service, you can't currently use sshguard on the host to test/block ssh attacks on the container. (This is what I was trying to setup when I found the above problem. My container passes logs to sshguard fine, and sshguard claims to have blocked the IP of the attacker. But it only blocks the attacker from accessing other ports on the host. The attacker still has full access to all container exposed ports, and so can continue with his ssh brute force attack.) If you know how to fix this, or if I should report this somewhere else, please let me know. GI -- 'Confidence' -- The feeling a person has before he fully understands the situation. |
From: Kevin Z. <kev...@gm...> - 2019-10-30 18:14:12
|
Thanks for the report. I think you're right in pointing out that the priority for SSHGuard is -10, but the priority for Docker is -100. Is someone on the list familiar with firewalls on Linux? Is the right fix here just to decrease the priority for SSHGuard? Thanks, Kevin -- Kevin Zheng kev...@gm... | ke...@be... XMPP: ke...@ee... |
From: <gi1...@gm...> - 2019-10-30 19:12:34
|
On Wed, Oct 30, 2019 at 11:14:04AM -0700, Kevin Zheng wrote: > Thanks for the report. I think you're right in pointing out that the > priority for SSHGuard is -10, but the priority for Docker is -100. > > Is someone on the list familiar with firewalls on Linux? Is the right > fix here just to decrease the priority for SSHGuard? >From the nft man page: The priority parameter accepts a signed integer value which specifies the order in which chains with same hook value are traversed. The ordering is ascending, i.e. lower priority values have precedence over higher ones. I just opened /usr/lib/x86_64-linux-gnu/sshg-fw-nft-sets. In fw_init() sshguard does run_nft "add chain" "${NFT_CHAIN}"' { type filter hook input priority -10 ; }' 4 run_nft "add chain" "${NFT_CHAIN}"' { type filter hook input priority -10 ; }' 6 I changed priority to -200 (Docker had -100), and restarted sshguard. There was no change in the behavior. I can still access containers from blocked IPs. Best, GI PS: If you're trying to reproduce it, it might be simpler to run a vanilla nginx container instead of the gitolite container I suggested in my last email. mkdir ./html # put static web content in ./html. Plain text files are fine docker run --rm --name=nginx-test \ -v ./html:/usr/share/nginx/html:ro -p 8080:80 -d nginx This exposes port 8080 on the host. The exposed port stays accessible from all sshguard blocked attackers. -- 'Civilization' -- Going from shoeless toes to toeless shoes. |
From: <gi1...@gm...> - 2019-10-30 19:39:03
|
OK, going through my nft rules again, I see a chain called "DOCKER-USER". I found the (possibly outdated) documentation here: https://docs.docker.com/network/iptables/ and it says: By default, all external source IPs are allowed to connect to the Docker daemon. To allow only a specific IP or network to access the containers, insert a negated rule at the top of the DOCKER filter chain. For example, the following rule restricts external access to all IP addresses except 192.168.1.1: iptables -I DOCKER-USER -i ext_if ! -s 192.168.1.1 -j DROP So I'm guessing the documentation is outdated (iptables instead of nftables), and also slightly incorrect (it says DOCKER instead of DOCKER-USER). If we could also add sshguards blacklist rule to the DOCKER-USER chain it might solve the problem. I don't know how to do this reliably though. GI -- A plateau is a high form of flattery. |
From: Christopher E. <ce...@lc...> - 2019-10-31 16:46:53
|
Hi, my guess would be that the problem lies with dockers bridge network setup (caveat, I am in no way an expert on either containers or networking). If I read those nftables rules correctly, then: - chain PREROUTING checks if the package is destined for an ip assigned to a local interface (yes), and if so, counts it and sends it to the DOCKER chain - chain DOCKER, a) if that local interface was docker0 (probably no?), resets the counter and returns or, b) if it was destined for HOSTIP:2022 (that's us), resets the counter and NATs the package to 172.17.0.3:22 So apart from the added complexity which I assume has something to do with how Docker does networking, this is just basic NAT - anything that is send to HOSTIP:2022 gets forwarded to 172.17.0.3:22, which is the IP of the docker0 bridge device, correct? If so, then chains assigned to the input hook, i.e. INPUT & blacklist are not called [1][2] on the host, the packet goes prerouting -> forward -> postrouting, not prerouting -> input. You could try: 1) changing the blacklist chain to 'hook prerouting' instead of 'hook input', with a higher priority than that of chain PREROURTING, i.e. block the traffic before it even reaches the NAT chain. This should make sshguard block both container- and host-destined traffic. 2) changing it to 'hook forward', i.e. catch the traffic on its new path after the docker chains (this will effectively disable sshguard for the host machine) 3) running sshguard inside the container, kinda not the point of containers .... (1) is probably the best bet, but it depends on what else your host is doing, network-wise. Hope this helps, somewhat Christopher [1] https://people.netfilter.org/pablo/docs/login.pdf [2] https://netdevconf.info/1.1/proceedings/papers/Bridge-filter-with-nftables.pdf |
From: <gi1...@gm...> - 2019-11-01 02:40:18
|
On Thu, Oct 31, 2019 at 05:46:40PM +0100, Christopher Engelhard wrote: > You could try: > 1) changing the blacklist chain to 'hook prerouting' instead of 'hook > input', with a higher priority than that of chain PREROURTING, i.e. > block the traffic before it even reaches the NAT chain. This should > make sshguard block both container- and host-destined traffic. Thank you so much! This worked. When I blindly increased the priority last time I didn't realize it was on a different chain. I hope it doesn't cause trouble with other things to switch that hook. Do you know why sshguard doesn't hook prerouting by default? GI -- The path to inner peace starts with three words: "Not My Business." |
From: Christopher E. <ce...@lc...> - 2019-11-01 12:24:28
|
On 01.11.19 03:40, gi1...@gm... wrote: > I hope it doesn't cause trouble with other things to switch that hook. The main difference is that you now block before deciding how to route traffic instead of after, so it's a very global block. Unless your machine does a lot of intricate forwarding all over the place, it shouldn't matter. > Do you know why sshguard doesn't hook prerouting by default? No. But input is where most simple firewall setups have ALL their rules for incoming traffic, so putting sshguards there as well makes it easier for the non-expert to understand what their firewall is doing. It is also the chain in which sshguard is least likely to interfere with more complex network setups. It also potentially reduces load by not running local filters on non-local traffic. Sshguard detects attacks on local services through local logs, so it makes sense to block access locally and not mess with traffic forwarded elsewhere. In the containerized world, where suddenly the host is an entire network with a shared harddrive and the host interface essentially does nothing but forwarding, it might be worth revisiting that decision. Christopher |
From: <gi1...@gm...> - 2019-11-01 14:00:08
|
This makes sense, thanks a lot. sshguard blocks about 15 attacks an hour on my machine, and I haven't seen any noticeable CPU increase since I switched to hooking pre-routing. Thanks again, GI -- The poor guy fell into a glass grinding machine and made a spectacle of himself. |
From: Christopher E. <ce...@lc...> - 2019-11-01 14:40:35
|
On 01.11.19 14:59, gi1...@gm... wrote: > This makes sense, thanks a lot. sshguard blocks about 15 attacks an hour > on my machine, and I haven't seen any noticeable CPU increase since I > switched to hooking pre-routing. Unless there's enough traffic that the firewall itself is causing significant load, I doubt one would ever notice the difference. Re using prerouting by default: Hooking into prerouting might also not be quite as easy with backends that don't act directly on the firewall, like e.g. the firewalld backend. But that's a question the devs can answer much better than me. Anyway, I'm glad things work for you now. Christopher |
From: Jos C. <ssh...@cl...> - 2020-02-08 12:10:46
|
On 1-11-19 3:40, gi1...@gm... wrote: > You could try: >> 1) changing the blacklist chain to 'hook prerouting' instead of 'hook >> input', with a higher priority than that of chain PREROURTING, i.e. >> block the traffic before it even reaches the NAT chain. This should >> make sshguard block both container- and host-destined traffic. Kev, could you implement/default that in the next update/grade of SSHGuard? Have a good weekend y'all! /jos -- With both feed on the ground you will never make a step forward |
From: hvjunk <hv...@gm...> - 2020-02-13 06:57:10
Attachments:
signature.asc
|
On 08 Feb 2020, at 13:50 , Jos Chrispijn <ssh...@cl...> wrote: > > On 1-11-19 3:40, gi1...@gm... wrote: >> You could try: >>> 1) changing the blacklist chain to 'hook prerouting' instead of 'hook >>> input', with a higher priority than that of chain PREROURTING, i.e. >>> block the traffic before it even reaches the NAT chain. This should >>> make sshguard block both container- and host-destined traffic. >>> > Kev, could you implement/default that in the next update/grade of SSHGuard? > Have a good weekend y’all! Hmmm… a docker host, is basically a router, not a host, ie. it is a “non-standard” case. I have a case where I’d not want that to happen, ie. the bastion/router host to get protected, but the honeypots behind it to be allowed through for capturing purposes/etc. Rather have that a toggle for routers/hypervisors(ie. docker/kvm/lxc hosts), than a blanket setting. |
From: Kevin Z. <kev...@gm...> - 2020-02-08 16:25:34
|
On 2/8/20 3:50 AM, Jos Chrispijn wrote: > On 1-11-19 3:40, gi1...@gm... wrote: >> You could try: >>> 1) changing the blacklist chain to 'hook prerouting' instead of 'hook >>> input', with a higher priority than that of chain PREROURTING, i.e. >>> block the traffic before it even reaches the NAT chain. This should >>> make sshguard block both container- and host-destined traffic. > Kev, could you implement/default that in the next update/grade of SSHGuard? > Have a good weekend y'all! I don't have a machine that I can test that on. Is somebody impacted by this willing to submit a patch that a few on the list can test? -- Kevin Zheng kev...@gm... | ke...@be... XMPP: ke...@ee... |
From: <gi1...@gm...> - 2020-02-12 21:37:25
|
On Sat, Feb 08, 2020 at 08:25:24AM -0800, Kevin Zheng wrote: >>>> 1) changing the blacklist chain to 'hook prerouting' instead of 'hook >>>> input', with a higher priority than that of chain PREROURTING, i.e. >>>> block the traffic before it even reaches the NAT chain. This should >>>> make sshguard block both container- and host-destined traffic. >> Kev, could you implement/default that in the next update/grade of >> SSHGuard? Have a good weekend y'all! > > I don't have a machine that I can test that on. Is somebody impacted > by this willing to submit a patch that a few on the list can test? It worked perfectly for me, and I can test it if you include it in the official code. The changes I made to get it working are at the bottom of this message. You can use it as a patch by changing the diff filenames of course. Best, Gautam --- /usr/lib/x86_64-linux-gnu/sshg-fw-nft-sets 2019-02-11 22:11:23.000000000 -0500 +++ /etc/sshguard/sshg-fw-nft-sets-local 2019-10-31 22:10:03.475621324 -0400 @@ -24,8 +24,8 @@ run_nft "add table" "" 4 run_nft "add table" "" 6 - run_nft "add chain" "${NFT_CHAIN}"' { type filter hook input priority -10 ; }' 4 - run_nft "add chain" "${NFT_CHAIN}"' { type filter hook input priority -10 ; }' 6 + run_nft "add chain" "${NFT_CHAIN}"' { type filter hook prerouting priority -200 ; }' 4 + run_nft "add chain" "${NFT_CHAIN}"' { type filter hook prerouting priority -200 ; }' 6 # Create sets run_nft "add set" "${NFT_SET} { type ipv4_addr; flags interval; }" 4 -- TWELVE REASONS WHY GOD NEVER RECEIVED TENURE 12. His office hours were infrequent and usually held on a mountaintop. |