Hi Christopher,

Yikes! ... thanks for pointing that out. I'll try the binary first to see the diff and then with the stream.

Cheers

Damien
 
On 5/06/2014 7:07 pm, Christopher Faulet wrote:
On 05/06/2014 07:55, Damien O'Rourke wrote:
Hi,

I'm using Yaws 1.98 as an API server in front of a PHP app sitting 
behind an nginx server. One of the functions is to provide for file 
downloads through the API from the nginx/PHP app after appropriate 
vetting. I'm using the following snippet to handle the file downloads as 
the file sizes can vary from < 1Mb to > 100MB.

...
case httpc:request(get,{Uri,[{"SID",Sid},{"apikey",Apikey}]},[],[]) of
     {ok, {Status,Headers,Data}} ->
      ....
         Self = self(),
             spawn(fun() ->
                yaws_api:stream_chunk_deliver(Self, Data),
                yaws_api:stream_chunk_end(Self),
                exit(normal)
             end),
             {streamcontent, "application/octet-stream", <<>>}
...

This works fine ... up to a point, and that point varies depending on 
the resources available on the server. e.g. on one server a 9MB file 
downloads fine, but a 55MB produces an erl_crash.dump file with the 
first couple of lines ...

Slogan: eheap_alloc: Cannot allocate 926079912 bytes of memory (of type 
"heap_frag").
System version: Erlang R14A (erts-5.8) [source] [64-bit] [smp:2:2] 
[rq:2] [async-threads:0] [kernel-poll:true]

So it appears to be trying to allocate just under 1GB of memory for a 
55MB file ... and the amount of free memory available at the time of the 
crash is 1.9GB. Is there a better, more efficient method of achieving 
the same result? This seems to rather inefficient ... at least 
memory-wise. Or maybe I'm not using this method correctly.

Hi Damien,

By default, httpc delivers the body response as a string. So for a 55MB
file, on a 64-bit machine, it takes ~880MB (1 word + 2 words per
character, 1 word = 8 bytes). And lists in erlang are not shared between
processes, so your huge list will be copied. This is far from ideal.

Instead, you can set the 'body_format' option in httpc:request to get
your file as binary. This will do the job. A better solution would be to
set the 'stream' option to avoid any memory problem at this step.



------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their 
applications. Written by three acclaimed leaders in the field, 
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech


_______________________________________________
Erlyaws-list mailing list
Erlyaws-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/erlyaws-list