From: Sergei G. <sgo...@gm...> - 2012-04-28 08:18:43
|
Hi! I've recently found a strange memory behavior of Erlang running YAWS with CGI. When I request the following script in an infinite cycle (while true ; do wget http://localhost:8080/env.cgi ; done) the beam process starts consume memory ------------------------------------------------------------------------------ #include <stdio.h> int main(int argc, char **argv, char **env) { char **p = env; printf("Content-Type: text/plain\n\n"); while (*p != NULL) { printf("%s\n", *p++); } } ------------------------------------------------------------------------------ (not actually a script but a program in C, which just prints the environment). As I can see in the top output, RES remains constant, but VIRT grows indefinitely and after some time beam crashes with Crash dump was written to: erl_crash.dump eheap_alloc: Cannot allocate 10336 bytes of memory (of type "heap"). Aborted (Actually, to get this crash I did set limit for the YAWS address space, but I think that it'd grow to the physical limits and crash.) memory/0 output doesn't show any leak in system and processes. total_heap_size of long-live processes don't grow either. The problem can be reproduces on the latest R15B01 and R14B04 (I don't have an older Erlang at the moment). Inserting sleep(1); somewhere in the 'script' makes the leak go away. It appears only on high load (but one wget in a cycle is high enough on my Intel core i7). Also, using a very slow escript implementation "fixes" the leak ------------------------------------------------------------------------------ #! /usr/bin/escript main(_) -> io:format("Content-Type: text/plain~n~n", []), lists:foreach(fun(V) -> io:format("~s~n", [V]) end, os:getenv()). ------------------------------------------------------------------------------ Can someone help me in understanding where is the bug and may be in fixing it? Cheers! -- Sergei Golovan |
From: CGS <cgs...@gm...> - 2012-04-28 11:31:52
|
Hi, I did get the same behavior from fastcgi and I didn't know what to do, so, I moved that part to Apache HTTPD. I am also interested in how to solve this problem, so, I am adding here my voice as well. I am willing to test some ideas if anyone has any because I do not know CGI (I had just to install some scripts). Cheers, CGS On Sat, Apr 28, 2012 at 10:18 AM, Sergei Golovan <sgo...@gm...> wrote: > Hi! > > I've recently found a strange memory behavior of Erlang running YAWS > with CGI. When I request the following script in an infinite cycle > (while true ; do wget http://localhost:8080/env.cgi ; done) the beam > process starts consume memory > > ------------------------------------------------------------------------------ > #include <stdio.h> > > int main(int argc, char **argv, char **env) { > char **p = env; > > printf("Content-Type: text/plain\n\n"); > > while (*p != NULL) { > printf("%s\n", *p++); > } > } > > ------------------------------------------------------------------------------ > (not actually a script but a program in C, which just prints the > environment). > > As I can see in the top output, RES remains constant, but VIRT grows > indefinitely and after some time beam crashes with > > Crash dump was written to: erl_crash.dump > eheap_alloc: Cannot allocate 10336 bytes of memory (of type "heap"). > Aborted > > (Actually, to get this crash I did set limit for the YAWS address > space, but I think that it'd grow to the physical limits and crash.) > > memory/0 output doesn't show any leak in system and processes. > total_heap_size of long-live processes don't grow either. > > The problem can be reproduces on the latest R15B01 and R14B04 (I don't > have an older Erlang at the moment). > > Inserting sleep(1); somewhere in the 'script' makes the leak go away. > It appears only on high load (but one wget in a cycle is high enough > on my Intel core i7). Also, using a very slow escript implementation > "fixes" the leak > > ------------------------------------------------------------------------------ > #! /usr/bin/escript > > main(_) -> > io:format("Content-Type: text/plain~n~n", []), > lists:foreach(fun(V) -> io:format("~s~n", [V]) end, os:getenv()). > > ------------------------------------------------------------------------------ > > Can someone help me in understanding where is the bug and may be in fixing > it? > > Cheers! > -- > Sergei Golovan > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Erlyaws-list mailing list > Erl...@li... > https://lists.sourceforge.net/lists/listinfo/erlyaws-list > |
From: Steve V. <vi...@ie...> - 2012-04-28 15:05:51
|
On Sat, Apr 28, 2012 at 4:18 AM, Sergei Golovan <sgo...@gm...> wrote: > Hi! > > I've recently found a strange memory behavior of Erlang running YAWS > with CGI. When I request the following script in an infinite cycle > (while true ; do wget http://localhost:8080/env.cgi ; done) the beam > process starts consume memory > ------------------------------------------------------------------------------ > #include <stdio.h> > > int main(int argc, char **argv, char **env) { > char **p = env; > > printf("Content-Type: text/plain\n\n"); > > while (*p != NULL) { > printf("%s\n", *p++); > } > } > ------------------------------------------------------------------------------ > (not actually a script but a program in C, which just prints the environment). > > As I can see in the top output, RES remains constant, but VIRT grows > indefinitely and after some time beam crashes with > > Crash dump was written to: erl_crash.dump > eheap_alloc: Cannot allocate 10336 bytes of memory (of type "heap"). > Aborted > > (Actually, to get this crash I did set limit for the YAWS address > space, but I think that it'd grow to the physical limits and crash.) > > memory/0 output doesn't show any leak in system and processes. > total_heap_size of long-live processes don't grow either. > > The problem can be reproduces on the latest R15B01 and R14B04 (I don't > have an older Erlang at the moment). What operating system and what version of yaws? > Inserting sleep(1); somewhere in the 'script' makes the leak go away. > It appears only on high load (but one wget in a cycle is high enough > on my Intel core i7). Also, using a very slow escript implementation > "fixes" the leak > ------------------------------------------------------------------------------ > #! /usr/bin/escript > > main(_) -> > io:format("Content-Type: text/plain~n~n", []), > lists:foreach(fun(V) -> io:format("~s~n", [V]) end, os:getenv()). > ------------------------------------------------------------------------------ > > Can someone help me in understanding where is the bug and may be in fixing it? I've been running your C program under yaws from github HEAD on OS X Lion for about 15-20 minutes and I see no memory growth at all. If you can give me more information to allow me to reproduce the problem, I can then try to fix it. --steve |
From: Sergei G. <sgo...@gm...> - 2012-04-28 15:13:43
|
On Sat, Apr 28, 2012 at 7:05 PM, Steve Vinoski <vi...@ie...> wrote: > > I've been running your C program under yaws from github HEAD on OS X > Lion for about 15-20 minutes and I see no memory growth at all. If you > can give me more information to allow me to reproduce the problem, I > can then try to fix it. I'm using Debian Linux 32 bit. YAWS 1.92. It grows both for Erlang R15B01 and R14B04 for me. -- Sergei Golovan |
From: Sergei G. <sgo...@gm...> - 2012-04-29 18:52:09
|
Hi! On Sat, Apr 28, 2012 at 7:05 PM, Steve Vinoski <vi...@ie...> wrote: > > What operating system and what version of yaws? After some testing I've found that virtual memory grows in Debian Gnu/Linux with kernel version 2.6.32, but doesn't grow on newer kernels (tested with 3.2.15 in Debian 32-bit and 64-bit x86 and MIPS, 3.3.1 in Fedora 64-bit x86). So, I guess the problem isn't in YAWS or Erlang but in Linux kernel itself. Cheers! -- Sergei Golovan |