Re: [Cppcms-users] Concurrent download question

[Shiv S. D. <shi...@gm...>]

I think that I will start with a synchronous application with thread
per connection and also have another version of asynchronous. I will
let both run and decide after serious testing and real world usage.

The answer to the questions varies. We will have to try combinations
in real world before arriving at the solution. I think I will use FUSE
so that I can do X-Send-File which seems to be optimal for me. But
without real testing it is hard to say what approach will be final.

Thanks a lot for all the help you have provided. I really appreciate it.

On Sun, Mar 27, 2016 at 3:25 PM, Artyom Beilis <art...@gm...> wrote:
>> The problem is that individual file will have rare downloads(usually
>> one file will be downloaded once a month or two)
>> and no. of downloads are large. So caching will not help.
>> They have to be read asynchronously and concurrently. I am not sure
>> that should I create booster::io::service or cppcms::service. Also,
>> I find it hard to create booster::io::service as its example is not
>> exhaustive. Even if I create booster::io::service the filesystem
>> calls may block. I will be using GlusterFS for storage. But that is
>> something not within CppCMS's domain so I will take care of
>> that.
>
> Got it, sorry I was mislead by previous question regarding the uploads.
>
> Regarding downloads.
>
> 1st if you run behind lighttpd or nginx web server (which you probably
> should do) I suggest
> to use X-Send-File stuff
> http://cppcms.com/wikipp/en/page/cppcms_1x_serving_static_files
>
> Also in such case you may need to mount GlusterFS via FUSE or NFS or
> use a special plugin
> (I've seen one for lighttpd)
>
> They do the best they can in serving static files - but you probably
> check the solution explicitly
> because maybe lighttpd or nginx are not optimized for GlusterFS as
> normal file system.
>
> ---------------------------------------------------
>
> If you use CppCMS's internal file server it can work in both sync and async mode
> starting from CppCMS 1.2 but you'll need to use stuff like FUSE or NFS
> to mount file system.
>
> But in async mode if GlusterFS hangs in opening/reading file you may block
> entire event loop.
>
> You can write your own application that does the job.
>
> Also note - if you don't have lots of concurrent downloads it may be
> simpler just to use thread per connection - in standard synchronous
> application as it would deal with both concurrency and "io optimization"
>
> if you do it asynchronously it can be tricky as if GlusterFS API may block
> and it would stall entire event loop. So you need to handle it someway,
> i.e. use booster::aio::io_service to handle such and events (I assyme
> glusterfs API has some event driven handling)
>
> Also you should note that web server may read the request
> faster than it sends it to the client - lighttpd in such a case would
> put it into memory and nginx create its own temporary file
> so it can be tricky stuff - neither of them would block you from
> writing to client (AFAIR) - which can be bad for huge files.
>
> Bottom line several questions you need to answer:
>
> 1. Do you work with GlusterFS via FUSE/NFS or its own API?
> 2. What concurrency level do you expect?
> 3. What web server does when client does not read in same speed as you send?
>
>
> Artyom
>
> ------------------------------------------------------------------------------
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140
> _______________________________________________
> Cppcms-users mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users



-- 
Respect,
Shiv Shankar Dayal