Re: [Cppcms-users] Slow http:file copy_stream for big files

[Cristian O. <one...@gm...>]

2014-10-20 16:10 GMT+03:00 Cristian Oneț <one...@gm...>:
> 2014-10-20 14:51 GMT+03:00 Artyom Beilis <art...@ya...>:
>> 1st of all for the iostreams there is a huge difference between debug and
>> release mode additionally all the callback
>> code of http code depends on optimization.
>
> Well the difference is 20 something seconds in release vs 1 minute and
> some seconds in debug (~270 MB file), the CPU is still hugged during
> those 20 seconds, I'll try the 'out << in.rdbuf()' version, just need
> to rebuild cppcms.

It didn't made any difference (still 20-30 secs) so the cause must be
somewhere else like in multipart_parser (it's just a hunch), I'll let
you look at this issue :) and stop making assumptions.

Thanks,
Cristian

>
>>
>> Also you are right out << in.rdbuf() should be better... probably one of
>> these parts of code I written too fast without thinking...
>>
>> However I doubt that it is the problem.
>>
>> How   http::file works when we receive an input data we write it to the
>> memory at first place but after certain threshold it
>> is written directly to the file instead of memory.  So this part of code has
>> only small use.
>
> I'm aware of that.
>
>>
>> Also I suggest when you actually "save" the file use http::file::save_to as
>> it would rather move the file on the disk and not copy it.
>>
>
> I use it.
>
>> If you still have problems with uploading in release more it would be a good
>> idea for me to profile the code. If so I need to know
>> what version of cppcms you use cppcms 1.0.x or trunk (there were some
>> changes in that part)
>
> I use cppcms-1.0.2. This is the test code, using the following settings:
>
> "security.multipart_form_data_limit" -> 300000; // ~300MB multipart limit
> "security.uploads_path" -> "d:\\tmp\\uploads"
>
> Code (note that it's a test, I know about the security issues):
>
> class uploader: public cppcms::application {
> public:
>   uploader(cppcms::service &s) :
>       cppcms::application(s)
>       {
>       }
>
>       void main(std::string /*unused*/)
>       {
>         if (request().request_method() == "POST")
>         {
>           const cppcms::http::request::files_type &files = request().files();
>           for (int i = 0; i < files.size(); ++i)
>           {
>             files[i]->save_to("D:\\tmp\\uploads\\" + files[i]->filename());
>           }
>         }
>         else
>         {
>           response().out() << "<form action='/upload'
> enctype='multipart/form-data' method='post'><input type='text'
> name='title'><br><input type='file' name='upload'
> multiple='multiple'><br><input type='submit' value='Upload'></form>";
>         }
>       }
> };
>
> Regards,
> Cristian
>
>>
>>
>> Artyom Beilis
>> --------------
>> CppCMS - C++ Web Framework:   http://cppcms.com/
>> CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
>>
>> ________________________________
>> From: Cristian Oneț <one...@gm...>
>> To: cppcms-users <cpp...@li...>
>> Sent: Monday, October 20, 2014 1:06 PM
>> Subject: [Cppcms-users] Slow http:file copy_stream for big files
>>
>> Hi,
>>
>> I'm using a simplified version (no widgets::file only request::files)
>> of the uploader [1] to implement an upload service. What I observed is
>> that uploading large files takes a lot of CPU and takes way more than
>> a similar nodejs service [2] (1 min vs 1 s for the same file - 300
>> MB). I have to admit that this was cppcms compiled in debug against a
>> release nodejs but the difference is too great.
>>
>> Intrigued by this finding I've took a glimpse at how the file is saved
>> to the temporary location using the input stream and found [3].
>> Reading from the input stream in 1024 byte chunks then writing that to
>> the output does not seem such a great idea and could explain the high
>> CPU usage and the overall slowness.
>>
>> Shouldn't this perform better?
>>
>> out << in.rdbuf();
>>
>> I promise I'll get back with the results of running a release build.
>>
>> Regards,
>> Cristian
>>
>> [1] http://cppcms.com/cppcms_ref/latest/ex_uploads____uploader__cpp.html
>> [2] https://github.com/felixge/node-formidable
>> [3]
>> http://sourceforge.net/p/cppcms/code/HEAD/tree/framework/trunk/src/http_file.cpp#l78
>>
>> ------------------------------------------------------------------------------
>> Comprehensive Server Monitoring with Site24x7.
>> Monitor 10 servers for $9/Month.
>> Get alerted through email, SMS, voice calls or mobile push notifications.
>> Take corrective actions from your mobile device.
>> http://p.sf.net/sfu/Zoho
>> _______________________________________________
>> Cppcms-users mailing list
>> Cpp...@li...
>> https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Comprehensive Server Monitoring with Site24x7.
>> Monitor 10 servers for $9/Month.
>> Get alerted through email, SMS, voice calls or mobile push notifications.
>> Take corrective actions from your mobile device.
>> http://p.sf.net/sfu/Zoho
>> _______________________________________________
>> Cppcms-users mailing list
>> Cpp...@li...
>> https://lists.sourceforge.net/lists/listinfo/cppcms-users
>>