Re: [Cppcms-users] Slow http:file copy_stream for big files

[Cristian O. <one...@gm...>]

2014-10-20 14:51 GMT+03:00 Artyom Beilis <art...@ya...>:
> 1st of all for the iostreams there is a huge difference between debug and
> release mode additionally all the callback
> code of http code depends on optimization.

Well the difference is 20 something seconds in release vs 1 minute and
some seconds in debug (~270 MB file), the CPU is still hugged during
those 20 seconds, I'll try the 'out << in.rdbuf()' version, just need
to rebuild cppcms.

>
> Also you are right out << in.rdbuf() should be better... probably one of
> these parts of code I written too fast without thinking...
>
> However I doubt that it is the problem.
>
> How   http::file works when we receive an input data we write it to the
> memory at first place but after certain threshold it
> is written directly to the file instead of memory.  So this part of code has
> only small use.

I'm aware of that.

>
> Also I suggest when you actually "save" the file use http::file::save_to as
> it would rather move the file on the disk and not copy it.
>

I use it.

> If you still have problems with uploading in release more it would be a good
> idea for me to profile the code. If so I need to know
> what version of cppcms you use cppcms 1.0.x or trunk (there were some
> changes in that part)

I use cppcms-1.0.2. This is the test code, using the following settings:

"security.multipart_form_data_limit" -> 300000; // ~300MB multipart limit
"security.uploads_path" -> "d:\\tmp\\uploads"

Code (note that it's a test, I know about the security issues):

class uploader: public cppcms::application {
public:
  uploader(cppcms::service &s) :
      cppcms::application(s)
      {
      }

      void main(std::string /*unused*/)
      {
        if (request().request_method() == "POST")
        {
          const cppcms::http::request::files_type &files = request().files();
          for (int i = 0; i < files.size(); ++i)
          {
            files[i]->save_to("D:\\tmp\\uploads\\" + files[i]->filename());
          }
        }
        else
        {
          response().out() << "<form action='/upload'
enctype='multipart/form-data' method='post'><input type='text'
name='title'><br><input type='file' name='upload'
multiple='multiple'><br><input type='submit' value='Upload'></form>";
        }
      }
};

Regards,
Cristian

>
>
> Artyom Beilis
> --------------
> CppCMS - C++ Web Framework:   http://cppcms.com/
> CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/
>
> ________________________________
> From: Cristian Oneț <one...@gm...>
> To: cppcms-users <cpp...@li...>
> Sent: Monday, October 20, 2014 1:06 PM
> Subject: [Cppcms-users] Slow http:file copy_stream for big files
>
> Hi,
>
> I'm using a simplified version (no widgets::file only request::files)
> of the uploader [1] to implement an upload service. What I observed is
> that uploading large files takes a lot of CPU and takes way more than
> a similar nodejs service [2] (1 min vs 1 s for the same file - 300
> MB). I have to admit that this was cppcms compiled in debug against a
> release nodejs but the difference is too great.
>
> Intrigued by this finding I've took a glimpse at how the file is saved
> to the temporary location using the input stream and found [3].
> Reading from the input stream in 1024 byte chunks then writing that to
> the output does not seem such a great idea and could explain the high
> CPU usage and the overall slowness.
>
> Shouldn't this perform better?
>
> out << in.rdbuf();
>
> I promise I'll get back with the results of running a release build.
>
> Regards,
> Cristian
>
> [1] http://cppcms.com/cppcms_ref/latest/ex_uploads____uploader__cpp.html
> [2] https://github.com/felixge/node-formidable
> [3]
> http://sourceforge.net/p/cppcms/code/HEAD/tree/framework/trunk/src/http_file.cpp#l78
>
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Cppcms-users mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users
>
>
>
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Cppcms-users mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users
>