#90 Memory Leak with TLS SNI

[marcel]

Noticed a memory leak when using config value SSLSMTPConfigure.

For each SMTP connection, the certificates defined in the callback function seem to be loaded to memory.
This memory is never released again, even if the smtp connections are long closed.

Depending on the amount of SSL certificates defined, this can result in tens of megabytes allocated per SMTP connection, which will quickly overload any server.

I had to disable this feature for now.

[Thomas Eckardt]

Having not a SINGLE hint about Perl, IO::Socket::SSL, OpenSSL-library, Net::SSLeay ..... does not help at all.

Thomas

[marcel]

Hi Thomas

Sorry, I am using perl v5.20.3 x86_64-linux-thread-multi

IO::Socket::SSL 2.052
Net::SSLeay 1.82
OpenSSL-lib 1.0.2k-fips 26 Jan 2017

ASSP version 2.5.6 Fortress build 17281

The function I was trying to use with letsencrypt certificates:

sub configWebSMTP {
    my $parms = shift;
    my $listenerName = &main::getSMTPListenerConfigName($parms->{LocalAddr},$parms->{LocalPort}); # returns listenPort , listenPort2 , listenPortSSL , relayPort or undef - may be used to implement different parameter settings for some or each SMTP listener
    if ($listenerName eq 'listenPortSSL') { # enable SNI at the listenPortSSL
        # used when nothing matches or the SMTP peer does not support SNI
        $parms->{SSL_cert_file} = {
            "" => "/etc/exim.crt"
        };
        $parms->{SSL_key_file} = {
            "" => "/etc/exim.key"
        };
        my @certs = glob("/var/cpanel/ssl/domain_tls/*/combined");
        for my $cert (@certs) {
            #my $domain = `basename \$(dirname $cert)`;
            my ($domain) = $cert =~ m|^.*[/]([^/]+?)/combined$|;
            chomp($domain);
            $parms->{SSL_cert_file}{$domain} = $cert;
            $parms->{SSL_key_file}{$domain} = $cert;
        }
    }
}

Let me know if you need anything else.

Marcel

Edit: fixed IO-socket-ssl version number and code snippet

[Thomas Eckardt]

please download assp.pl.gz and version txt from
https://sourceforge.net/p/assp/svn/HEAD/tree/assp2/trunk/test/

tell me if this fixes the problem

Thomas

[marcel]

Unfortunately I still see the same behaviour after enabling multiple certificates

[Thomas Eckardt]

It is normal that assp consumes more and more memory over the time. This depends on the setup and workload. Is there really a big difference if you use SNI or not ?

Please show two examples - with SNI and without:

memory after startup:
memory after X SSL/TLS connections or Y hours:

Thomas

[marcel]

# ls -la /var/cpanel/ssl/domain_tls/*/combined | wc -l
186
# du -hc /var/cpanel/ssl/domain_tls/*/combined
1.5M    total

testing/faking connections with

# echo | openssl s_client -connect ${hostname}:25 -servername ${hostname} -starttls smtp | openssl x509 -noout -dates

Without SNI, following lines commented out in configWebSMTP:

            #$parms->{SSL_cert_file}{$domain} = $cert;
            #$parms->{SSL_key_file}{$domain} = $cert;

memory after startup: 484 MB
memory after 5 TLS connections: 500 M

With SNI (all 186 certificates)::

memory after startup: 484 MB
memory after 5 TLS connections: 1266 M

I have redone both tests 2-3 times with similar results.

Marcel

[Thomas Eckardt]

1.5M total
186 certs

There is not much I can do. If Net::SSLeay requires this amount of memory to calculate and hold the SSL-context data, you need to provide it.
Depending on the workload, it may take some time for the perl garbage collector to free up memory. It may also possible, that the Net::SSLeay C-lib leaks some memory or maloc is unable to defrag and freeup fragmented memory.
Notice, that the minimum RAM required for assp in ISP mode is 16GB. Such a setup is common!

What we can try is to prevent to create a new SSL-context for each new connection. There is an option reuse a still existing context. How ever, it will take some time to implement this behavior in a switchable way. I'll give you a hint, if I have a new try version available.

Thomas

[Thomas Eckardt]

please change these two lines
$parms->{SSL_cert_file}{$domain} = $cert;
$parms->{SSL_key_file}{$domain} = $cert;
to
$parms->{SSL_cert_file}->{$domain} = $cert;
$parms->{SSL_key_file}->{$domain} = $cert;

At the same download location an update is available.

ASSP uses a permanent SSL-Context, but watches for changes and renews the context, if this is required.

tell me if this works or not

Thomas

[marcel]

Hi Thomas

I modified those 2 lines and installed your patched version.

1. smtp connections while SNI is enabled are way faster than before (<1s vs 2-3s).

2. Memory usage is now a bit higher than without SNI (expected) but not as excessive as before:
   ~650 MB after multiple connection-tests and not visibly increasing with each request like before

So far it looks like your modifications solved the leak.
I will leave SNI enabled, monitor the situation over the next few days and report back again.

Thank you very much for the effort you put into this.

Marcel

[Thomas Eckardt]

sounds good - waiting for your responds.

btw: 'SSLDEBUG' now writes to maillog.txt and SYSLOG - so you can have an realtime debug output for the SSL/TLS actions

Thomas

[marcel]

After enabling assp on all listenports, some of the users had issues with maxSSLRenegotiations, while they never had any issues communicating directly with exim.
Setting the parameter to 10 or 20 was not enough.
I disabled that feature and so far it is running very stable with acceptable memory consumption.

Marcel

[Thomas Eckardt]

The next version will ignore maxSSLRenegotiations for local and outgoing mails.
How ever, it is not normal for a SSL-Client to force a SSL renegotiation so many tmes.

Thomas

[Thomas Eckardt]

Please try assp 2.5.6 Fortress build 17339 and tell me how it works.

https://sourceforge.net/p/assp/svn/HEAD/tree/assp2/trunk/

Thomas

[marcel]

I installed the new build and set maxSSLRenegotiations back to 10.

Will monitor.the behaviour and report back again.

Thank you very much

Marcel

[marcel]

Hi Thomas

There was no issue regarding maxSSLRenegotiations since the update and no user reported any problems.

In my opinion this ticket is resolved.

Thank you very much for your effort.

Marcel