From: Damien O'R. <da...@en...> - 2014-06-05 05:56:10
|
Hi, I'm using Yaws 1.98 as an API server in front of a PHP app sitting behind an nginx server. One of the functions is to provide for file downloads through the API from the nginx/PHP app after appropriate vetting. I'm using the following snippet to handle the file downloads as the file sizes can vary from < 1Mb to > 100MB. ... case httpc:request(get,{Uri,[{"SID",Sid},{"apikey",Apikey}]},[],[]) of {ok, {Status,Headers,Data}} -> .... Self = self(), spawn(fun() -> yaws_api:stream_chunk_deliver(Self, Data), yaws_api:stream_chunk_end(Self), exit(normal) end), {streamcontent, "application/octet-stream", <<>>} ... This works fine ... up to a point, and that point varies depending on the resources available on the server. e.g. on one server a 9MB file downloads fine, but a 55MB produces an erl_crash.dump file with the first couple of lines ... Slogan: eheap_alloc: Cannot allocate 926079912 bytes of memory (of type "heap_frag"). System version: Erlang R14A (erts-5.8) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:true] So it appears to be trying to allocate just under 1GB of memory for a 55MB file ... and the amount of free memory available at the time of the crash is 1.9GB. Is there a better, more efficient method of achieving the same result? This seems to rather inefficient ... at least memory-wise. Or maybe I'm not using this method correctly. Thanks Damien |
From: Damien O'R. <da...@en...> - 2014-06-05 06:03:14
|
Just to complete the picture ... I'm running an mnesia instance which just stores a set of url string to apikey string mappings ... there wouldn't be more than a hundred of them. Thanks Damien On 5/06/2014 3:55 pm, Damien O'Rourke wrote: > Hi, > > I'm using Yaws 1.98 as an API server in front of a PHP app sitting > behind an nginx server. One of the functions is to provide for file > downloads through the API from the nginx/PHP app after appropriate > vetting. I'm using the following snippet to handle the file downloads as > the file sizes can vary from < 1Mb to > 100MB. > > ... > case httpc:request(get,{Uri,[{"SID",Sid},{"apikey",Apikey}]},[],[]) of > {ok, {Status,Headers,Data}} -> > .... > Self = self(), > spawn(fun() -> > yaws_api:stream_chunk_deliver(Self, Data), > yaws_api:stream_chunk_end(Self), > exit(normal) > end), > {streamcontent, "application/octet-stream", <<>>} > ... > > This works fine ... up to a point, and that point varies depending on > the resources available on the server. e.g. on one server a 9MB file > downloads fine, but a 55MB produces an erl_crash.dump file with the > first couple of lines ... > > Slogan: eheap_alloc: Cannot allocate 926079912 bytes of memory (of type > "heap_frag"). > System version: Erlang R14A (erts-5.8) [source] [64-bit] [smp:2:2] > [rq:2] [async-threads:0] [kernel-poll:true] > > So it appears to be trying to allocate just under 1GB of memory for a > 55MB file ... and the amount of free memory available at the time of the > crash is 1.9GB. Is there a better, more efficient method of achieving > the same result? This seems to rather inefficient ... at least > memory-wise. Or maybe I'm not using this method correctly. > > Thanks > > Damien > > ------------------------------------------------------------------------------ > Learn Graph Databases - Download FREE O'Reilly Book > "Graph Databases" is the definitive new guide to graph databases and their > applications. Written by three acclaimed leaders in the field, > this first edition is now available. Download your free book today! > http://p.sf.net/sfu/NeoTech > _______________________________________________ > Erlyaws-list mailing list > Erl...@li... > https://lists.sourceforge.net/lists/listinfo/erlyaws-list |
From: Christopher F. <chr...@ca...> - 2014-06-05 09:07:56
Attachments:
signature.asc
|
On 05/06/2014 07:55, Damien O'Rourke wrote: > Hi, > > I'm using Yaws 1.98 as an API server in front of a PHP app sitting > behind an nginx server. One of the functions is to provide for file > downloads through the API from the nginx/PHP app after appropriate > vetting. I'm using the following snippet to handle the file downloads as > the file sizes can vary from < 1Mb to > 100MB. > > ... > case httpc:request(get,{Uri,[{"SID",Sid},{"apikey",Apikey}]},[],[]) of > {ok, {Status,Headers,Data}} -> > .... > Self = self(), > spawn(fun() -> > yaws_api:stream_chunk_deliver(Self, Data), > yaws_api:stream_chunk_end(Self), > exit(normal) > end), > {streamcontent, "application/octet-stream", <<>>} > ... > > This works fine ... up to a point, and that point varies depending on > the resources available on the server. e.g. on one server a 9MB file > downloads fine, but a 55MB produces an erl_crash.dump file with the > first couple of lines ... > > Slogan: eheap_alloc: Cannot allocate 926079912 bytes of memory (of type > "heap_frag"). > System version: Erlang R14A (erts-5.8) [source] [64-bit] [smp:2:2] > [rq:2] [async-threads:0] [kernel-poll:true] > > So it appears to be trying to allocate just under 1GB of memory for a > 55MB file ... and the amount of free memory available at the time of the > crash is 1.9GB. Is there a better, more efficient method of achieving > the same result? This seems to rather inefficient ... at least > memory-wise. Or maybe I'm not using this method correctly. > Hi Damien, By default, httpc delivers the body response as a string. So for a 55MB file, on a 64-bit machine, it takes ~880MB (1 word + 2 words per character, 1 word = 8 bytes). And lists in erlang are not shared between processes, so your huge list will be copied. This is far from ideal. Instead, you can set the 'body_format' option in httpc:request to get your file as binary. This will do the job. A better solution would be to set the 'stream' option to avoid any memory problem at this step. -- Christopher |
From: Damien O'R. <da...@en...> - 2014-06-05 09:41:00
|
Hi Christopher, Yikes! ... thanks for pointing that out. I'll try the binary first to see the diff and then with the stream. Cheers Damien On 5/06/2014 7:07 pm, Christopher Faulet wrote: > On 05/06/2014 07:55, Damien O'Rourke wrote: >> Hi, >> >> I'm using Yaws 1.98 as an API server in front of a PHP app sitting >> behind an nginx server. One of the functions is to provide for file >> downloads through the API from the nginx/PHP app after appropriate >> vetting. I'm using the following snippet to handle the file downloads as >> the file sizes can vary from < 1Mb to > 100MB. >> >> ... >> case httpc:request(get,{Uri,[{"SID",Sid},{"apikey",Apikey}]},[],[]) of >> {ok, {Status,Headers,Data}} -> >> .... >> Self = self(), >> spawn(fun() -> >> yaws_api:stream_chunk_deliver(Self, Data), >> yaws_api:stream_chunk_end(Self), >> exit(normal) >> end), >> {streamcontent, "application/octet-stream", <<>>} >> ... >> >> This works fine ... up to a point, and that point varies depending on >> the resources available on the server. e.g. on one server a 9MB file >> downloads fine, but a 55MB produces an erl_crash.dump file with the >> first couple of lines ... >> >> Slogan: eheap_alloc: Cannot allocate 926079912 bytes of memory (of type >> "heap_frag"). >> System version: Erlang R14A (erts-5.8) [source] [64-bit] [smp:2:2] >> [rq:2] [async-threads:0] [kernel-poll:true] >> >> So it appears to be trying to allocate just under 1GB of memory for a >> 55MB file ... and the amount of free memory available at the time of the >> crash is 1.9GB. Is there a better, more efficient method of achieving >> the same result? This seems to rather inefficient ... at least >> memory-wise. Or maybe I'm not using this method correctly. >> > Hi Damien, > > By default, httpc delivers the body response as a string. So for a 55MB > file, on a 64-bit machine, it takes ~880MB (1 word + 2 words per > character, 1 word = 8 bytes). And lists in erlang are not shared between > processes, so your huge list will be copied. This is far from ideal. > > Instead, you can set the 'body_format' option in httpc:request to get > your file as binary. This will do the job. A better solution would be to > set the 'stream' option to avoid any memory problem at this step. > > > > ------------------------------------------------------------------------------ > Learn Graph Databases - Download FREE O'Reilly Book > "Graph Databases" is the definitive new guide to graph databases and their > applications. Written by three acclaimed leaders in the field, > this first edition is now available. Download your free book today! > http://p.sf.net/sfu/NeoTech > > > _______________________________________________ > Erlyaws-list mailing list > Erl...@li... > https://lists.sourceforge.net/lists/listinfo/erlyaws-list |