#8 Memory Leak problem

[AJ1]

For Project:
mod_aspdotnet
Category:
Apache.Web.NET Assembly
Summary:
Memory Leak problem
Detailed Description:
We use Apache as a primary web server to call ASP.NET WebServices.
Our environment is the following:
• Windows Server 2003 Standard Edition SP 2 or Windows XP SP 2
• .NET 1.1 (ASP.NET Web Services)
• Apache 2.0.59 / Apache 2.2.4
• Module mod_aspdotnet 2.0.0.2006 (two builds for diff version of Apache server)

There is the following problem with this configuration:
Under heavy loading there is a continual memory growth of Apache child process. In our case heavy loading means that we just call to simple web service method from multithread application (25 – 30 threads) with small time interval (2 seconds between requests) during long period of time (from half of day to one or two days). As a result we have low virtual memory size message from Windows OS, sometimes Apache stops to serve requests due to real slowdown of system.
The workaround is to restart Apache process every night but after some investigation we think that this is a bug in mod_aspdotnet module.
We use the following approach to see the memory leak:
Memory leak can occur either in unmanaged heap or managed heap. Perfmon (Windows tool) is used to examine the counters such as Process/Private bytes and .Net CLR Memory/# bytes in All heaps. If only Process/Private bytes are increasing but the .NET CLR Memory is not increasing then unmanaged memory is leaking else if both are increasing then the managed memory is leaking.
As a result unmanaged memory leak has been detected and reproduced on the configuration described above. In our case unmanaged memory leak is continual memory (Private bytes for child process of Apache) growth under the loading.
Here you can find log files of UMDH tool:
apache1.log – before loading
apache2.log – starting loading (stabilization period – CLR memory allocation)
apache3.log – just after start loading
apache4.log – during the work (stable small memory growth)
apache5.log - during the work (stable small memory growth)
apache6.log - during the work (stable small memory growth)
apache7.log – after stop loading

And compare reports:
results12.txt
results23.txt
results34.txt
results45.txt
results56.txt
results67.txt

As you can see the most memory allocations are started from Apache.Web.dll during the work and after stop loading there are no memory freeing. Memory never comes back to the operating system but the CLR memory is always on the same level more or less. That’s why it looks like a bug inside unmanaged part of mod_aspdotnet module.
Could you please confirm this or give us explanation for behaviour like described above.
To reproduce it is necessary to create loading with calling simple web service method which will return string for example and put it on Apache server during the day.
Also it is important to notice that Apache has the following configuration:
• KeepAlive On
• MaxKeepAliveRequests 100
• KeepAliveTimeout 15
It means that in our case Apache should create 25-30 connections and handle requests within this limited number of connections and memory consumptions should be stable during the loading.
Looking forward for your answer.

[William A. Rowe, Jr.]

For reference, this may be an allocation bug under the winnt mpm of httpd and not actually
related to .net, if you can confirm this was memory growth of the .NET vm and not the
process, that information would be useful to pursue this as an asp.net module flaw.

[Anonymous]

I'm having the same problem. Is there a fix?

Ciao,
Max

[S. Brandt]

Hi!
I've got the same problem .... 100MB/day or so; yet, we need to run the server for weeks ...
Could anyone give me a hint how to find out how to locate the memory leak, esp. whether it is in the apache itself, the module code, the .Net VM, etc?
Just checking the memory allocated and not freed by the various systems should be enough ... the managed memory doesn't get above 25MB.

Yours,
Sebastian

[S. Brandt]

I checked the mem usage with process explorer; the managed memory is not the problem.
The .Net CLR memory page shows "Total reserved" 50MB and "Total committed" ~25MB unchanged for hours, while the httpd.exe "private bytes" are rising all the time.
Of course, this doesn't say anything about memory leaks in the CLR itself ...

Is there any way to find out which module allocates how much memory?

Just running apache with mod_aspdotnet loaded, but not accessed, and heavy file download (50 parallel requests, via localhost, ~250MB/s altogether) doesn't show any change in private bytes. (I believe "private bytes" is the best indicator for actual memory usage).
Accessing an .aspx first time raises the private bytes from 27M to 58M.
Accessing more times raises it to ~70M, but falls down to 58M after a few secs.
(50x100MB download still running in the background)
With 10x aspx parallel private bytes rise to peak 287M, and regularly fall down to 110M.

Now this is strange.
After some time and a lot of downloads (~8k), the private bytes cycle between 180M and 310M, so, about 70M more than in the beginning.
Is is possible that memory is leaked when an apache thread is terminated?
MPM Name: WinNT
MPM Information: Max Daemons: 64 Threaded: yes Forked: no
Server Architecture: 32-bit
No MPM configuration in httpd.conf, so MaxRequestsPerChild should be 0 for mpm_winnt ...
(it's a 2.2.21)

When private bytes of the process reach ~550M (about 2GB virtual), the process crashes. Another fork takes over.
That's reached every few minutes with 20 parallel requests, which is not something we need, luckily.
Number of requests per connection is never more than 0 in extended server status (no keep-alive used).

In any case, we will set MaxRequestsPerChild to 5000 or so, probably this will solve the memory leak problem when running over weeks. This relates to the process; is there any configuration value that influences the number of requests a thread handles?
If not, the idea with the thread termination memory leak does not fit ..

Yours,
Sebastian