wrapper-user

[Wrapper-user] wrapper 3.5.25 memory leak on CentOS 7

[Aleksey C. <ale...@gm...>]

Hi,

We are using wrapper 3.5.25 community edition on a bunch of high loaded 
servers without any issues.

Currently I'm testing the same wrapper and java application on a new 
CentOS 7 server and facing a huge memory leak by the wrapper process.

KiB Mem:  16423540 total, 16210652 used,   212888 free,      728 buffers
KiB Swap:  4194300 total,        0 used,  4194300 free.   262020 cached Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
20381 root      20   0 7531520 6.979g    808 S   0.0 44.6   3:00.90 wrapper
20383 root      20   0 19.037g 6.959g   8860 S   1.3 44.4   1175:24 java


At the same time there where a lot of wrapper messages dropped by 
systemd-journal because of rate limiting

Nov 15 13:28:09 srv94 journal: Suppressed 27674 messages from 
/user.slice/user-1013.slice
Nov 15 13:28:39 srv94 journal: Suppressed 29040 messages from 
/user.slice/user-1013.slice
Nov 15 13:29:09 srv94 journal: Suppressed 28509 messages from 
/user.slice/user-1013.slice
Nov 15 13:29:39 srv94 journal: Suppressed 19124 messages from 
/user.slice/user-1013.slice
Nov 15 13:30:09 srv94 journal: Suppressed 18889 messages from 
/user.slice/user-1013.slice
Nov 15 13:30:40 srv94 journal: Suppressed 10076 messages from 
/user.slice/user-1013.slice

Do not ask why so many messages :) It's normal behaviour for our 
application.

Seem like wrapper leak memory on dropped syslog messages. May be some 
internal buffers are not cleaned?

We are using the following log related settings

wrapper.console.format=PM
wrapper.console.loglevel=INFO

wrapper.logfile=/var/log/myapp.log
wrapper.logfile.format=LPTM
wrapper.logfile.loglevel=INFO
wrapper.logfile.maxsize=100m
wrapper.logfile.maxfiles=10

wrapper.syslog.facility=LOCAL4
wrapper.syslog.loglevel=INFO
wrapper.syslog.ident=myapp


I can reproduce the problem.


Regards,
Aleksey

Re: [Wrapper-user] wrapper 3.5.25 memory leak on CentOS 7

[Aleksey C. <ale...@gm...>]

Finally I found that memory leak relates to wrapper.logfile options. As 
soon as I comment the following options in configuration file the 
problem goes away.

wrapper.logfile=/var/log/myapp.log
wrapper.logfile.format=LPTM
wrapper.logfile.loglevel=INFO
wrapper.logfile.maxsize=100m
wrapper.logfile.maxfiles=10

After enabling options one by one I found two problem options. As soon 
as I uncomment the following options wrapper starts to eat memory.

wrapper.logfile.maxsize=100m
wrapper.logfile.maxfiles=10

I can reproduce the problem.


Regards,
Aleksey

Re: [Wrapper-user] wrapper 3.5.25 memory leak on CentOS 7

[Tim L. <tim...@gm...>]

There are two bugs at play here. One is in glibc, the other in the wrapper
itself.

- glibc leak:
https://sourceware.org/bugzilla/show_bug.cgi?id=17370
It is fixed in newer releases of glibc, so if you can upgrade your glibc
libraries please do so.

If you can not upgrade your glibc libraries, see patch
02.workaround-memory-leak-in-ftell.

- wrapper leak:
I've also attached a fix for the memory leak in logging in the wrapper
itself in patch 01.fixes-memory-leak-in-error-logging.
It uses the same solution as the windows version to prevent using up all
the memory.

Regards,
Tim


On Sun, Nov 16, 2014 at 11:51 AM, Aleksey Chudov <ale...@gm...>
wrote:

> Finally I found that memory leak relates to wrapper.logfile options. As
> soon as I comment the following options in configuration file the
> problem goes away.
>
> wrapper.logfile=/var/log/myapp.log
> wrapper.logfile.format=LPTM
> wrapper.logfile.loglevel=INFO
> wrapper.logfile.maxsize=100m
> wrapper.logfile.maxfiles=10
>
> After enabling options one by one I found two problem options. As soon
> as I uncomment the following options wrapper starts to eat memory.
>
> wrapper.logfile.maxsize=100m
> wrapper.logfile.maxfiles=10
>
> I can reproduce the problem.
>
>
> Regards,
> Aleksey
>
>
>
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>

Re: [Wrapper-user] wrapper 3.5.25 memory leak on CentOS 7

[Aleksey C. <ale...@gm...>]

Thanks for the reply!

Unfortunately I can't update glibc unless it will be updated by upstream 
(CentOS maintainers).

So I patched wrapper sources using previously attached patches and quick 
test shows memory leak is fixed!

Do you have plans to release a new wrapper version includes the proposed 
fixes? Don't want to maintain custom wrapper version :)

Regards,
Aleksey


On 11/17/2014 11:39 AM, Tim Lammens wrote:
> There are two bugs at play here. One is in glibc, the other in the 
> wrapper itself.
>
> - glibc leak:
> https://sourceware.org/bugzilla/show_bug.cgi?id=17370
> It is fixed in newer releases of glibc, so if you can upgrade your 
> glibc libraries please do so.
>
> If you can not upgrade your glibc libraries, see patch 
> 02.workaround-memory-leak-in-ftell.
>
> - wrapper leak:
> I've also attached a fix for the memory leak in logging in the wrapper 
> itself in patch 01.fixes-memory-leak-in-error-logging.
> It uses the same solution as the windows version to prevent using up 
> all the memory.
>
> Regards,
> Tim
>