1. Summary
  2. Files
  3. Support
  4. Report Spam
  5. Create account
  6. Log in

SSL Denial of Service

Moderators: pdreissen, fribo

SSL Denial of Service

Postby tillo » Thu Dec 20, 2012 1:25 pm

Hello,

We just upgraded ASSP from the latest v1 to 2.2.2(12343).
We also added listenPortSSL=465.

On ASSPv1 the implicit SSL listener was tunneled through an instance of "stunnel" as a workaround, because of some obscure library problems.
Of course this means that ASSP did not know the connecting IP address, and we had to fix it somehow.
Tests showed that those problems would go away with ASSPv2, and they did.

Unfortunately, we now have another situation.

When a connection is made on port 465, but the connecting peer does not begin or complete the SSL handshake, the following kind of warning is logged :

[Worker_2] Error: Worker_2 accept to client failed IO::Socket::SSL=GLOB(0x7f5456500108) (timeout: 5 s) : Resource temporarily unavailable


The reason may vary. For example we can replicate the same error with "nc -z host 465", but the message is instantly written to the logs (without the 5 seconds timeout) and the reason is missing:

[Worker_2] Error: Worker_2 accept to client failed IO::Socket::SSL=GLOB(0x10152210) (timeout: 5 s) :


This does not happen a lot by itself, but it happens sometimes and we have plenty of those in the logs.
Bad SSL implementations, port scanning and exploitation of SSL vulnerabilities are just some of the reasons we could think of.

Anyway, it seems to be just a warning (a warning about a socket error, logged as an error, but still a warning) and nothing bad happens: the worker is freed and the service continue to work correctly.

The problem begins when the resetFH subroutine is called, apparently after 10 of those errors :

[Main_Thread] Info: try to renewed listening on port 0.0.0.0:465 - after too many errors
[Main_Thread] Info: Main_Thread is waiting until Workers finished current SMTP-connections or 630 seconds - to renew Socket-Listener


To begin with, the 600 seconds timeout is a long timeout for a reason: should be used for a soft restart, not in case of emergency.
Also, 30 seconds are added in the code, so lowering the setting does not help.
In the meantime, no SSL service is available.

When the timeout is reached or all the workers finished their jobs, the following happens :

[Main_Thread] Info: Main_Thread detected - all Workers are finished current SMTP-connections
[Main_Thread] Couldn't create server SSL-socket on port '465' -- maybe another service is running or I'm not root (uid=65534)? - or a wrong IP address is specified? -- Permission denied - IO::Socket::INET configuration failederror:00000000:lib(0):func(0):reason(0)


The service lowered its rights after being started, therefore it can not renew the listener.
There is no secure workaround to this problem: we want the 465 port to be protected from unprivileged users.
At this moment assp restarts, and more time is lost.

We obviously do not want this behavior, but there is no configuration variables to deactivate it.
We had to comment the last "if" block within the ConToThread subroutine, in order to avoid the call of resetFH "after too many errors".

We think that this is not just a blocking problem, but also a security issue.
With a few connections to the SSL port, anyone can deny access to ASSP for a while (via SSL at first, any access during the restart).
By doing this continually, the DoS is permanent.

Thank you for correcting this issue in the next release, or please help us find what we did wrong.
Martino Dell'Ambrogio tillo@tillo.ch http://www.tillo.ch/ Security Auditor
tillo
 
Posts: 3
Joined: Fri Jul 15, 2011 8:06 am

Re: SSL Denial of Service

Postby thockar » Fri Dec 21, 2012 8:56 am

>There is no secure workaround to this problem: we want the 465 port to be protected from unprivileged users.

If assp uses a port - no other can use it. Instead of ports <1024 use higher ports in assp and configure a port map from your firewall (using the lower ports) to assp.
Never use ASSP without a firewall in front - assp is a perl script and can never be solid rock against DoS.

>the worker is freed and the service continue to work correctly

No - the failed listener (465 in your case) is not freed and could never be reused until an accept->FH or a listener reset (resetFH).
ASSP counts only concurrent mistakes on the accept->FH ... if 10 is reached, it tries to renew the listener.

>(a warning about a socket error, logged as an error, but still a warning)
In case the listener is blocked by someone (DoS) this would be an error - we log the higher possible reason.

>We had to comment the last "if" block within the ConToThread subroutine, in order to avoid the call of resetFH "after too many errors".

So assp will not renew the listener - hmm... - some OS will wipe out the blocked connection, some not - in the later case the permanent DoS is perfect.
The mainthread will go crasy, because POLL (or SELECT) detecting the socket as available and it will try to interrupt the workers permanently to transfer the connection without success (this is the reason for the '10' counter).

I'll try to make the implementation of the SSL listener a bit less weak - how ever, because the underlying 'Net::SSLeay' C-Lib uses the blocking mode for the SSL-handshake, it will be not possible to prevent a DoS on that listener.

A possible solution is to disable the SSL-listener and to 'hope' that the client will use 'STARTTLS' at the default port 25.
In this case, the connection is already transfered to the worker and if the SSL->start failes the connection is falling back in to the unsecured mode (if supported by the client) or it is simply closed without any penalty to assp's performance.

Thomas

Thomas
thockar
Site Admin
 
Posts: 460
Joined: Mon Mar 09, 2009 7:05 pm

Re: SSL Denial of Service

Postby tillo » Fri Dec 21, 2012 4:04 pm

thockar wrote:If assp uses a port - no other can use it. Instead of ports <1024 use higher ports in assp and configure a port map from your firewall (using the lower ports) to assp.
Never use ASSP without a firewall in front - assp is a perl script and can never be solid rock against DoS.

There is a firewall, for external and internal uses, but we actually should and want to use a protected port on a server that has multiple services, in order to protect the other service from local, unprivileged attacks. If you need a longer explanation I can give you one.

thockar wrote:
tillo wrote:the worker is freed and the service continue to work correctly

No - the failed listener (465 in your case) is not freed and could never be reused until an accept->FH or a listener reset (resetFH).

I don't know if it is due to our platform (an up-to-date Debian GNU/Linux stable distribution), but actually yes. I can launch batches of 5 connections, 5 seconds apart, and all of them are always served. That is, if we remove the resetFH call "after too many errors".

thockar wrote:In case the listener is blocked by someone (DoS) this would be an error - we log the higher possible reason.

Well, it seems to be triggered by a blocking (and disconnecting) error or by a timeout. Aren't all those unblocking situation, by definition?

thockar wrote:So assp will not renew the listener - hmm... - some OS will wipe out the blocked connection, some not - in the later case the permanent DoS is perfect.
The mainthread will go crasy, because POLL (or SELECT) detecting the socket as available and it will try to interrupt the workers permanently to transfer the connection without success (this is the reason for the '10' counter).

I guess there may be a platform that does not free the socket which encountered an SSL blocking error, but that seems odd to me. Of course if this is the case, and the only way to free the socket is to renew the listener, then your implementation is correct. In that case, however, I guess we should report the problem higher up.

thockar wrote:I'll try to make the implementation of the SSL listener a bit less weak - how ever, because the underlying 'Net::SSLeay' C-Lib uses the blocking mode for the SSL-handshake, it will be not possible to prevent a DoS on that listener.

Thank you. If you need testing or upstream sponsoring let us know, we will more than happy to help you help us.

thockar wrote:A possible solution is to disable the SSL-listener and to 'hope' that the client will use 'STARTTLS' at the default port 25.

Unfortunately this is not secure enough for us, thank you for the hint anyway.
We already block authentication without STARTTLS, but this does not stop mail clients from trying, thus revealing the credentials.
The only way is to enforce an SSL-implicit port.
Martino Dell'Ambrogio tillo@tillo.ch http://www.tillo.ch/ Security Auditor
tillo
 
Posts: 3
Joined: Fri Jul 15, 2011 8:06 am

Re: SSL Denial of Service

Postby thockar » Sat Dec 22, 2012 12:38 pm

> I can launch batches of 5 connections, 5 seconds apart,

You should be able to use 10

$parms{Listen} = 10;
$parms{Reuse} = 1;

How ever - if all 10 initialized listeners are blocked by the DoS ....

The next release will do the following:

- if high ports are used (or on windows) - the listener will be renewed
- if low ports are used and assp runs as root - the listener will be renewed
- if low ports are used and the user/group was switched away from root - assp will remove the failed listener from POLL/SELECT for 10 seconds

Thomas
thockar
Site Admin
 
Posts: 460
Joined: Mon Mar 09, 2009 7:05 pm

Re: SSL Denial of Service

Postby tillo » Sun Dec 23, 2012 9:26 am

Thank you, I will let you know it that's working for us as soon as we can test it.
Martino Dell'Ambrogio tillo@tillo.ch http://www.tillo.ch/ Security Auditor
tillo
 
Posts: 3
Joined: Fri Jul 15, 2011 8:06 am


Return to Bug reports

Who is online

Users browsing this forum: No registered users and 1 guest

cron