From: Damien O'R. <da...@en...> - 2014-06-05 09:41:00
|
Hi Christopher, Yikes! ... thanks for pointing that out. I'll try the binary first to see the diff and then with the stream. Cheers Damien On 5/06/2014 7:07 pm, Christopher Faulet wrote: > On 05/06/2014 07:55, Damien O'Rourke wrote: >> Hi, >> >> I'm using Yaws 1.98 as an API server in front of a PHP app sitting >> behind an nginx server. One of the functions is to provide for file >> downloads through the API from the nginx/PHP app after appropriate >> vetting. I'm using the following snippet to handle the file downloads as >> the file sizes can vary from < 1Mb to > 100MB. >> >> ... >> case httpc:request(get,{Uri,[{"SID",Sid},{"apikey",Apikey}]},[],[]) of >> {ok, {Status,Headers,Data}} -> >> .... >> Self = self(), >> spawn(fun() -> >> yaws_api:stream_chunk_deliver(Self, Data), >> yaws_api:stream_chunk_end(Self), >> exit(normal) >> end), >> {streamcontent, "application/octet-stream", <<>>} >> ... >> >> This works fine ... up to a point, and that point varies depending on >> the resources available on the server. e.g. on one server a 9MB file >> downloads fine, but a 55MB produces an erl_crash.dump file with the >> first couple of lines ... >> >> Slogan: eheap_alloc: Cannot allocate 926079912 bytes of memory (of type >> "heap_frag"). >> System version: Erlang R14A (erts-5.8) [source] [64-bit] [smp:2:2] >> [rq:2] [async-threads:0] [kernel-poll:true] >> >> So it appears to be trying to allocate just under 1GB of memory for a >> 55MB file ... and the amount of free memory available at the time of the >> crash is 1.9GB. Is there a better, more efficient method of achieving >> the same result? This seems to rather inefficient ... at least >> memory-wise. Or maybe I'm not using this method correctly. >> > Hi Damien, > > By default, httpc delivers the body response as a string. So for a 55MB > file, on a 64-bit machine, it takes ~880MB (1 word + 2 words per > character, 1 word = 8 bytes). And lists in erlang are not shared between > processes, so your huge list will be copied. This is far from ideal. > > Instead, you can set the 'body_format' option in httpc:request to get > your file as binary. This will do the job. A better solution would be to > set the 'stream' option to avoid any memory problem at this step. > > > > ------------------------------------------------------------------------------ > Learn Graph Databases - Download FREE O'Reilly Book > "Graph Databases" is the definitive new guide to graph databases and their > applications. Written by three acclaimed leaders in the field, > this first edition is now available. Download your free book today! > http://p.sf.net/sfu/NeoTech > > > _______________________________________________ > Erlyaws-list mailing list > Erl...@li... > https://lists.sourceforge.net/lists/listinfo/erlyaws-list |