Re: [Erlyaws-list] File downloading best practice

[Damien O'R. <da...@en...>]

Just to complete the picture ... I'm running an mnesia instance which 
just stores a set of url string to apikey string mappings ... there 
wouldn't be more than a hundred of them.

Thanks

Damien

On 5/06/2014 3:55 pm, Damien O'Rourke wrote:
> Hi,
>
> I'm using Yaws 1.98 as an API server in front of a PHP app sitting
> behind an nginx server. One of the functions is to provide for file
> downloads through the API from the nginx/PHP app after appropriate
> vetting. I'm using the following snippet to handle the file downloads as
> the file sizes can vary from < 1Mb to > 100MB.
>
> ...
> case httpc:request(get,{Uri,[{"SID",Sid},{"apikey",Apikey}]},[],[]) of
>       {ok, {Status,Headers,Data}} ->
>        ....
>           Self = self(),
>               spawn(fun() ->
>                  yaws_api:stream_chunk_deliver(Self, Data),
>                  yaws_api:stream_chunk_end(Self),
>                  exit(normal)
>               end),
>               {streamcontent, "application/octet-stream", <<>>}
> ...
>
> This works fine ... up to a point, and that point varies depending on
> the resources available on the server. e.g. on one server a 9MB file
> downloads fine, but a 55MB produces an erl_crash.dump file with the
> first couple of lines ...
>
> Slogan: eheap_alloc: Cannot allocate 926079912 bytes of memory (of type
> "heap_frag").
> System version: Erlang R14A (erts-5.8) [source] [64-bit] [smp:2:2]
> [rq:2] [async-threads:0] [kernel-poll:true]
>
> So it appears to be trying to allocate just under 1GB of memory for a
> 55MB file ... and the amount of free memory available at the time of the
> crash is 1.9GB. Is there a better, more efficient method of achieving
> the same result? This seems to rather inefficient ... at least
> memory-wise. Or maybe I'm not using this method correctly.
>
> Thanks
>
> Damien
>
> ------------------------------------------------------------------------------
> Learn Graph Databases - Download FREE O'Reilly Book
> "Graph Databases" is the definitive new guide to graph databases and their
> applications. Written by three acclaimed leaders in the field,
> this first edition is now available. Download your free book today!
> http://p.sf.net/sfu/NeoTech
> _______________________________________________
> Erlyaws-list mailing list
> Erl...@li...
> https://lists.sourceforge.net/lists/listinfo/erlyaws-list