Re: [Cppcms-users] Slow http:file copy_stream for big files

[Artyom B. <art...@ya...>]

Thanks...

Can you take a look on v1.0_fixes branch of the cppcms

This is the fix I had written: https://sourceforge.net/p/cppcms/code/2271/

It improved 600MB file upload from 30-40s to 4-5s. And it is twice faster than PHP
(that has multipart parsing embedded into the C code of PHP)

So I assume it is almost as efficient as it can get.
 
Take a look and if it is ok I'll close the ticket. It would get
released in 1.0.5

Artyom Beilis 
-------------- 
CppCMS - C++ Web Framework:   http://cppcms.com/ 
CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/


----- Original Message -----
> From: Cristian Oneț <one...@gm...>
> To: cppcms-users <cpp...@li...>
> Cc: 
> Sent: Monday, October 20, 2014 6:41 PM
> Subject: Re: [Cppcms-users] Slow http:file copy_stream for big files
> 
> 2014-10-20 17:04 GMT+03:00 Artyom Beilis <art...@ya...>:
>>  1st of all please open a ticket...
> 
> Done.
> 
> https://sourceforge.net/p/cppcms/feature-requests/27/
> 
> Regards,
> Cristian
> 
>> 
>>  I got the problem... it is indeed in multipart_parser or more accuratly
>>  in the way I write to the buffered iostream...
>> 
>>  My assumption that the iostream optimization is very good was actually 
> wrong
>>  and there is some terrible bottle neck.
>> 
>>  I'll get this fixed for 1.0.5.
>> 
>>  Artyom Beilis
>>  --------------
>>  CppCMS - C++ Web Framework:  http://cppcms.com/
>>  CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
>> 
>> 
>>  ----- Original Message -----
>>>  From: Cristian Oneț <one...@gm...>
>>>  To: cppcms-users <cpp...@li...>
>>>  Cc:
>>>  Sent: Monday, October 20, 2014 3:54 PM
>>>  Subject: Re: [Cppcms-users] Slow http:file copy_stream for big files
>>> 
>>>  2014-10-20 16:10 GMT+03:00 Cristian Oneț 
> <one...@gm...>:
>>>>   2014-10-20 14:51 GMT+03:00 Artyom Beilis 
> <art...@ya...>:
>>>>>   1st of all for the iostreams there is a huge difference 
> between debug
>>>  and
>>>>>   release mode additionally all the callback
>>>>>   code of http code depends on optimization.
>>>> 
>>>>   Well the difference is 20 something seconds in release vs 1 minute 
> and
>>>>   some seconds in debug (~270 MB file), the CPU is still hugged 
> during
>>>>   those 20 seconds, I'll try the 'out << 
> in.rdbuf()'
>>>  version, just need
>>>>   to rebuild cppcms.
>>> 
>>>  It didn't made any difference (still 20-30 secs) so the cause must 
> be
>>>  somewhere else like in multipart_parser (it's just a hunch), 
> I'll let
>>>  you look at this issue :) and stop making assumptions.
>>> 
>>>  Thanks,
>>>  Cristian
>>> 
>>> 
>>>> 
>>>>> 
>>>>>   Also you are right out << in.rdbuf() should be better... 
> probably
>>>  one of
>>>>>   these parts of code I written too fast without thinking...
>>>>> 
>>>>>   However I doubt that it is the problem.
>>>>> 
>>>>>   How   http::file works when we receive an input data we write 
> it to the
>>>>>   memory at first place but after certain threshold it
>>>>>   is written directly to the file instead of memory.  So this 
> part of
>>>  code has
>>>>>   only small use.
>>>> 
>>>>   I'm aware of that.
>>>> 
>>>>> 
>>>>>   Also I suggest when you actually "save" the file use
>>>  http::file::save_to as
>>>>>   it would rather move the file on the disk and not copy it.
>>>>> 
>>>> 
>>>>   I use it.
>>>> 
>>>>>   If you still have problems with uploading in release more it 
> would be a
>>>  good
>>>>>   idea for me to profile the code. If so I need to know
>>>>>   what version of cppcms you use cppcms 1.0.x or trunk (there 
> were some
>>>>>   changes in that part)
>>>> 
>>>>   I use cppcms-1.0.2. This is the test code, using the following 
> settings:
>>>> 
>>>>   "security.multipart_form_data_limit" -> 300000; // 
> ~300MB
>>>  multipart limit
>>>>   "security.uploads_path" ->
>>>  "d:\\tmp\\uploads"
>>>> 
>>>>   Code (note that it's a test, I know about the security 
> issues):
>>>> 
>>>>   class uploader: public cppcms::application {
>>>>   public:
>>>>     uploader(cppcms::service &s) :
>>>>         cppcms::application(s)
>>>>         {
>>>>         }
>>>> 
>>>>         void main(std::string /*unused*/)
>>>>         {
>>>>           if (request().request_method() == "POST")
>>>>           {
>>>>             const cppcms::http::request::files_type &files =
>>>  request().files();
>>>>             for (int i = 0; i < files.size(); ++i)
>>>>             {
>>>> 
>>> 
> files[i]->save_to("D:\\tmp\\uploads\\" +
>>>  files[i]->filename());
>>>>             }
>>>>           }
>>>>           else
>>>>           {
>>>>             response().out() << "<form 
> action='/upload'
>>>>   enctype='multipart/form-data' 
> method='post'><input
>>>  type='text'
>>>>   name='title'><br><input type='file'
>>>  name='upload'
>>>>   multiple='multiple'><br><input 
> type='submit'
>>>  value='Upload'></form>";
>>>>           }
>>>>         }
>>>>   };
>>>> 
>>>>   Regards,
>>>>   Cristian
>>>> 
>>>>> 
>>>>> 
>>>>>   Artyom Beilis
>>>>>   --------------
>>>>>   CppCMS - C++ Web Framework:  http://cppcms.com/
>>>>>   CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
>>>>> 
>>>>>   ________________________________
>>>>>   From: Cristian Oneț <one...@gm...>
>>>>>   To: cppcms-users <cpp...@li...>
>>>>>   Sent: Monday, October 20, 2014 1:06 PM
>>>>>   Subject: [Cppcms-users] Slow http:file copy_stream for big 
> files
>>>>> 
>>>>>   Hi,
>>>>> 
>>>>>   I'm using a simplified version (no widgets::file only
>>>  request::files)
>>>>>   of the uploader [1] to implement an upload service. What I 
> observed is
>>>>>   that uploading large files takes a lot of CPU and takes way 
> more than
>>>>>   a similar nodejs service [2] (1 min vs 1 s for the same file - 
> 300
>>>>>   MB). I have to admit that this was cppcms compiled in debug 
> against a
>>>>>   release nodejs but the difference is too great.
>>>>> 
>>>>>   Intrigued by this finding I've took a glimpse at how the 
> file is
>>>  saved
>>>>>   to the temporary location using the input stream and found 
> [3].
>>>>>   Reading from the input stream in 1024 byte chunks then writing 
> that to
>>>>>   the output does not seem such a great idea and could explain 
> the high
>>>>>   CPU usage and the overall slowness.
>>>>> 
>>>>>   Shouldn't this perform better?
>>>>> 
>>>>>   out << in.rdbuf();
>>>>> 
>>>>>   I promise I'll get back with the results of running a 
> release
>>>  build.
>>>>> 
>>>>>   Regards,
>>>>>   Cristian
>>>>> 
>>>>>   [1]
>>>  http://cppcms.com/cppcms_ref/latest/ex_uploads____uploader__cpp.html
>>>>>   [2] https://github.com/felixge/node-formidable
>>>>>   [3]
>>>>> 
>>> 
> http://sourceforge.net/p/cppcms/code/HEAD/tree/framework/trunk/src/http_file.cpp#l78
>>>>> 
>>>>> 
>>> 
> ------------------------------------------------------------------------------
>>>>>   Comprehensive Server Monitoring with Site24x7.
>>>>>   Monitor 10 servers for $9/Month.
>>>>>   Get alerted through email, SMS, voice calls or mobile push
>>>  notifications.
>>>>>   Take corrective actions from your mobile device.
>>>>>   http://p.sf.net/sfu/Zoho
>>>>>   _______________________________________________
>>>>>   Cppcms-users mailing list
>>>>>   Cpp...@li...
>>>>>   https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>> 
> ------------------------------------------------------------------------------
>>>>>   Comprehensive Server Monitoring with Site24x7.
>>>>>   Monitor 10 servers for $9/Month.
>>>>>   Get alerted through email, SMS, voice calls or mobile push
>>>  notifications.
>>>>>   Take corrective actions from your mobile device.
>>>>>   http://p.sf.net/sfu/Zoho
>>>>>   _______________________________________________
>>>>>   Cppcms-users mailing list
>>>>>   Cpp...@li...
>>>>>   https://lists.sourceforge.net/lists/listinfo/cppcms-users
> 
>>>>> 
>>> 
>>> 
> ------------------------------------------------------------------------------
>>>  Comprehensive Server Monitoring with Site24x7.
>>>  Monitor 10 servers for $9/Month.
>>>  Get alerted through email, SMS, voice calls or mobile push 
> notifications.
>>>  Take corrective actions from your mobile device.
>>>  http://p.sf.net/sfu/Zoho
>>>  _______________________________________________
>>>  Cppcms-users mailing list
>>>  Cpp...@li...
>>>  https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>> 
>> 
>> 
> ------------------------------------------------------------------------------
>>  Comprehensive Server Monitoring with Site24x7.
>>  Monitor 10 servers for $9/Month.
>>  Get alerted through email, SMS, voice calls or mobile push notifications.
>>  Take corrective actions from your mobile device.
>>  http://p.sf.net/sfu/Zoho
>>  _______________________________________________
>>  Cppcms-users mailing list
>>  Cpp...@li...
>>  https://lists.sourceforge.net/lists/listinfo/cppcms-users
> 
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Cppcms-users mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users
>