Re: [Cppcms-users] Slow http:file copy_stream for big files

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

2014-10-20 17:04 GMT+03:00 Artyom Beilis <art...@ya...>:
> 1st of all please open a ticket...

Done.

https://sourceforge.net/p/cppcms/feature-requests/27/

Regards,
Cristian

>
> I got the problem... it is indeed in multipart_parser or more accuratly
> in the way I write to the buffered iostream...
>
> My assumption that the iostream optimization is very good was actually wrong
> and there is some terrible bottle neck.
>
> I'll get this fixed for 1.0.5.
>
> Artyom Beilis
> --------------
> CppCMS - C++ Web Framework:   http://cppcms.com/
> CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
>
>
> ----- Original Message -----
>> From: Cristian Oneț <one...@gm...>
>> To: cppcms-users <cpp...@li...>
>> Cc:
>> Sent: Monday, October 20, 2014 3:54 PM
>> Subject: Re: [Cppcms-users] Slow http:file copy_stream for big files
>>
>> 2014-10-20 16:10 GMT+03:00 Cristian Oneț <one...@gm...>:
>>>  2014-10-20 14:51 GMT+03:00 Artyom Beilis <art...@ya...>:
>>>>  1st of all for the iostreams there is a huge difference between debug
>> and
>>>>  release mode additionally all the callback
>>>>  code of http code depends on optimization.
>>>
>>>  Well the difference is 20 something seconds in release vs 1 minute and
>>>  some seconds in debug (~270 MB file), the CPU is still hugged during
>>>  those 20 seconds, I'll try the 'out << in.rdbuf()'
>> version, just need
>>>  to rebuild cppcms.
>>
>> It didn't made any difference (still 20-30 secs) so the cause must be
>> somewhere else like in multipart_parser (it's just a hunch), I'll let
>> you look at this issue :) and stop making assumptions.
>>
>> Thanks,
>> Cristian
>>
>>
>>>
>>>>
>>>>  Also you are right out << in.rdbuf() should be better... probably
>> one of
>>>>  these parts of code I written too fast without thinking...
>>>>
>>>>  However I doubt that it is the problem.
>>>>
>>>>  How   http::file works when we receive an input data we write it to the
>>>>  memory at first place but after certain threshold it
>>>>  is written directly to the file instead of memory.  So this part of
>> code has
>>>>  only small use.
>>>
>>>  I'm aware of that.
>>>
>>>>
>>>>  Also I suggest when you actually "save" the file use
>> http::file::save_to as
>>>>  it would rather move the file on the disk and not copy it.
>>>>
>>>
>>>  I use it.
>>>
>>>>  If you still have problems with uploading in release more it would be a
>> good
>>>>  idea for me to profile the code. If so I need to know
>>>>  what version of cppcms you use cppcms 1.0.x or trunk (there were some
>>>>  changes in that part)
>>>
>>>  I use cppcms-1.0.2. This is the test code, using the following settings:
>>>
>>>  "security.multipart_form_data_limit" -> 300000; // ~300MB
>> multipart limit
>>>  "security.uploads_path" ->
>> "d:\\tmp\\uploads"
>>>
>>>  Code (note that it's a test, I know about the security issues):
>>>
>>>  class uploader: public cppcms::application {
>>>  public:
>>>    uploader(cppcms::service &s) :
>>>        cppcms::application(s)
>>>        {
>>>        }
>>>
>>>        void main(std::string /*unused*/)
>>>        {
>>>          if (request().request_method() == "POST")
>>>          {
>>>            const cppcms::http::request::files_type &files =
>> request().files();
>>>            for (int i = 0; i < files.size(); ++i)
>>>            {
>>>
>> files[i]->save_to("D:\\tmp\\uploads\\" +
>> files[i]->filename());
>>>            }
>>>          }
>>>          else
>>>          {
>>>            response().out() << "<form action='/upload'
>>>  enctype='multipart/form-data' method='post'><input
>> type='text'
>>>  name='title'><br><input type='file'
>> name='upload'
>>>  multiple='multiple'><br><input type='submit'
>> value='Upload'></form>";
>>>          }
>>>        }
>>>  };
>>>
>>>  Regards,
>>>  Cristian
>>>
>>>>
>>>>
>>>>  Artyom Beilis
>>>>  --------------
>>>>  CppCMS - C++ Web Framework:  http://cppcms.com/
>>>>  CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
>>>>
>>>>  ________________________________
>>>>  From: Cristian Oneț <one...@gm...>
>>>>  To: cppcms-users <cpp...@li...>
>>>>  Sent: Monday, October 20, 2014 1:06 PM
>>>>  Subject: [Cppcms-users] Slow http:file copy_stream for big files
>>>>
>>>>  Hi,
>>>>
>>>>  I'm using a simplified version (no widgets::file only
>> request::files)
>>>>  of the uploader [1] to implement an upload service. What I observed is
>>>>  that uploading large files takes a lot of CPU and takes way more than
>>>>  a similar nodejs service [2] (1 min vs 1 s for the same file - 300
>>>>  MB). I have to admit that this was cppcms compiled in debug against a
>>>>  release nodejs but the difference is too great.
>>>>
>>>>  Intrigued by this finding I've took a glimpse at how the file is
>> saved
>>>>  to the temporary location using the input stream and found [3].
>>>>  Reading from the input stream in 1024 byte chunks then writing that to
>>>>  the output does not seem such a great idea and could explain the high
>>>>  CPU usage and the overall slowness.
>>>>
>>>>  Shouldn't this perform better?
>>>>
>>>>  out << in.rdbuf();
>>>>
>>>>  I promise I'll get back with the results of running a release
>> build.
>>>>
>>>>  Regards,
>>>>  Cristian
>>>>
>>>>  [1]
>> http://cppcms.com/cppcms_ref/latest/ex_uploads____uploader__cpp.html
>>>>  [2] https://github.com/felixge/node-formidable
>>>>  [3]
>>>>
>> http://sourceforge.net/p/cppcms/code/HEAD/tree/framework/trunk/src/http_file.cpp#l78
>>>>
>>>>
>> ------------------------------------------------------------------------------
>>>>  Comprehensive Server Monitoring with Site24x7.
>>>>  Monitor 10 servers for $9/Month.
>>>>  Get alerted through email, SMS, voice calls or mobile push
>> notifications.
>>>>  Take corrective actions from your mobile device.
>>>>  http://p.sf.net/sfu/Zoho
>>>>  _______________________________________________
>>>>  Cppcms-users mailing list
>>>>  Cpp...@li...
>>>>  https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>>>
>>>>
>>>>
>>>>
>> ------------------------------------------------------------------------------
>>>>  Comprehensive Server Monitoring with Site24x7.
>>>>  Monitor 10 servers for $9/Month.
>>>>  Get alerted through email, SMS, voice calls or mobile push
>> notifications.
>>>>  Take corrective actions from your mobile device.
>>>>  http://p.sf.net/sfu/Zoho
>>>>  _______________________________________________
>>>>  Cppcms-users mailing list
>>>>  Cpp...@li...
>>>>  https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>>>
>>
>> ------------------------------------------------------------------------------
>> Comprehensive Server Monitoring with Site24x7.
>> Monitor 10 servers for $9/Month.
>> Get alerted through email, SMS, voice calls or mobile push notifications.
>> Take corrective actions from your mobile device.
>> http://p.sf.net/sfu/Zoho
>> _______________________________________________
>> Cppcms-users mailing list
>> Cpp...@li...
>> https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>
>
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Cppcms-users mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users