Re: [Cppcms-users] Slow http:file copy_stream for big files
Brought to you by:
artyom-beilis
|
From: Artyom B. <art...@ya...> - 2014-10-21 05:40:50
|
Thanks... Can you take a look on v1.0_fixes branch of the cppcms This is the fix I had written: https://sourceforge.net/p/cppcms/code/2271/ It improved 600MB file upload from 30-40s to 4-5s. And it is twice faster than PHP (that has multipart parsing embedded into the C code of PHP) So I assume it is almost as efficient as it can get. Take a look and if it is ok I'll close the ticket. It would get released in 1.0.5 Artyom Beilis -------------- CppCMS - C++ Web Framework: http://cppcms.com/ CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/ ----- Original Message ----- > From: Cristian Oneț <one...@gm...> > To: cppcms-users <cpp...@li...> > Cc: > Sent: Monday, October 20, 2014 6:41 PM > Subject: Re: [Cppcms-users] Slow http:file copy_stream for big files > > 2014-10-20 17:04 GMT+03:00 Artyom Beilis <art...@ya...>: >> 1st of all please open a ticket... > > Done. > > https://sourceforge.net/p/cppcms/feature-requests/27/ > > Regards, > Cristian > >> >> I got the problem... it is indeed in multipart_parser or more accuratly >> in the way I write to the buffered iostream... >> >> My assumption that the iostream optimization is very good was actually > wrong >> and there is some terrible bottle neck. >> >> I'll get this fixed for 1.0.5. >> >> Artyom Beilis >> -------------- >> CppCMS - C++ Web Framework: http://cppcms.com/ >> CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/ >> >> >> ----- Original Message ----- >>> From: Cristian Oneț <one...@gm...> >>> To: cppcms-users <cpp...@li...> >>> Cc: >>> Sent: Monday, October 20, 2014 3:54 PM >>> Subject: Re: [Cppcms-users] Slow http:file copy_stream for big files >>> >>> 2014-10-20 16:10 GMT+03:00 Cristian Oneț > <one...@gm...>: >>>> 2014-10-20 14:51 GMT+03:00 Artyom Beilis > <art...@ya...>: >>>>> 1st of all for the iostreams there is a huge difference > between debug >>> and >>>>> release mode additionally all the callback >>>>> code of http code depends on optimization. >>>> >>>> Well the difference is 20 something seconds in release vs 1 minute > and >>>> some seconds in debug (~270 MB file), the CPU is still hugged > during >>>> those 20 seconds, I'll try the 'out << > in.rdbuf()' >>> version, just need >>>> to rebuild cppcms. >>> >>> It didn't made any difference (still 20-30 secs) so the cause must > be >>> somewhere else like in multipart_parser (it's just a hunch), > I'll let >>> you look at this issue :) and stop making assumptions. >>> >>> Thanks, >>> Cristian >>> >>> >>>> >>>>> >>>>> Also you are right out << in.rdbuf() should be better... > probably >>> one of >>>>> these parts of code I written too fast without thinking... >>>>> >>>>> However I doubt that it is the problem. >>>>> >>>>> How http::file works when we receive an input data we write > it to the >>>>> memory at first place but after certain threshold it >>>>> is written directly to the file instead of memory. So this > part of >>> code has >>>>> only small use. >>>> >>>> I'm aware of that. >>>> >>>>> >>>>> Also I suggest when you actually "save" the file use >>> http::file::save_to as >>>>> it would rather move the file on the disk and not copy it. >>>>> >>>> >>>> I use it. >>>> >>>>> If you still have problems with uploading in release more it > would be a >>> good >>>>> idea for me to profile the code. If so I need to know >>>>> what version of cppcms you use cppcms 1.0.x or trunk (there > were some >>>>> changes in that part) >>>> >>>> I use cppcms-1.0.2. This is the test code, using the following > settings: >>>> >>>> "security.multipart_form_data_limit" -> 300000; // > ~300MB >>> multipart limit >>>> "security.uploads_path" -> >>> "d:\\tmp\\uploads" >>>> >>>> Code (note that it's a test, I know about the security > issues): >>>> >>>> class uploader: public cppcms::application { >>>> public: >>>> uploader(cppcms::service &s) : >>>> cppcms::application(s) >>>> { >>>> } >>>> >>>> void main(std::string /*unused*/) >>>> { >>>> if (request().request_method() == "POST") >>>> { >>>> const cppcms::http::request::files_type &files = >>> request().files(); >>>> for (int i = 0; i < files.size(); ++i) >>>> { >>>> >>> > files[i]->save_to("D:\\tmp\\uploads\\" + >>> files[i]->filename()); >>>> } >>>> } >>>> else >>>> { >>>> response().out() << "<form > action='/upload' >>>> enctype='multipart/form-data' > method='post'><input >>> type='text' >>>> name='title'><br><input type='file' >>> name='upload' >>>> multiple='multiple'><br><input > type='submit' >>> value='Upload'></form>"; >>>> } >>>> } >>>> }; >>>> >>>> Regards, >>>> Cristian >>>> >>>>> >>>>> >>>>> Artyom Beilis >>>>> -------------- >>>>> CppCMS - C++ Web Framework: http://cppcms.com/ >>>>> CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/ >>>>> >>>>> ________________________________ >>>>> From: Cristian Oneț <one...@gm...> >>>>> To: cppcms-users <cpp...@li...> >>>>> Sent: Monday, October 20, 2014 1:06 PM >>>>> Subject: [Cppcms-users] Slow http:file copy_stream for big > files >>>>> >>>>> Hi, >>>>> >>>>> I'm using a simplified version (no widgets::file only >>> request::files) >>>>> of the uploader [1] to implement an upload service. What I > observed is >>>>> that uploading large files takes a lot of CPU and takes way > more than >>>>> a similar nodejs service [2] (1 min vs 1 s for the same file - > 300 >>>>> MB). I have to admit that this was cppcms compiled in debug > against a >>>>> release nodejs but the difference is too great. >>>>> >>>>> Intrigued by this finding I've took a glimpse at how the > file is >>> saved >>>>> to the temporary location using the input stream and found > [3]. >>>>> Reading from the input stream in 1024 byte chunks then writing > that to >>>>> the output does not seem such a great idea and could explain > the high >>>>> CPU usage and the overall slowness. >>>>> >>>>> Shouldn't this perform better? >>>>> >>>>> out << in.rdbuf(); >>>>> >>>>> I promise I'll get back with the results of running a > release >>> build. >>>>> >>>>> Regards, >>>>> Cristian >>>>> >>>>> [1] >>> http://cppcms.com/cppcms_ref/latest/ex_uploads____uploader__cpp.html >>>>> [2] https://github.com/felixge/node-formidable >>>>> [3] >>>>> >>> > http://sourceforge.net/p/cppcms/code/HEAD/tree/framework/trunk/src/http_file.cpp#l78 >>>>> >>>>> >>> > ------------------------------------------------------------------------------ >>>>> Comprehensive Server Monitoring with Site24x7. >>>>> Monitor 10 servers for $9/Month. >>>>> Get alerted through email, SMS, voice calls or mobile push >>> notifications. >>>>> Take corrective actions from your mobile device. >>>>> http://p.sf.net/sfu/Zoho >>>>> _______________________________________________ >>>>> Cppcms-users mailing list >>>>> Cpp...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/cppcms-users >>>>> >>>>> >>>>> >>>>> >>> > ------------------------------------------------------------------------------ >>>>> Comprehensive Server Monitoring with Site24x7. >>>>> Monitor 10 servers for $9/Month. >>>>> Get alerted through email, SMS, voice calls or mobile push >>> notifications. >>>>> Take corrective actions from your mobile device. >>>>> http://p.sf.net/sfu/Zoho >>>>> _______________________________________________ >>>>> Cppcms-users mailing list >>>>> Cpp...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/cppcms-users > >>>>> >>> >>> > ------------------------------------------------------------------------------ >>> Comprehensive Server Monitoring with Site24x7. >>> Monitor 10 servers for $9/Month. >>> Get alerted through email, SMS, voice calls or mobile push > notifications. >>> Take corrective actions from your mobile device. >>> http://p.sf.net/sfu/Zoho >>> _______________________________________________ >>> Cppcms-users mailing list >>> Cpp...@li... >>> https://lists.sourceforge.net/lists/listinfo/cppcms-users >>> >> >> > ------------------------------------------------------------------------------ >> Comprehensive Server Monitoring with Site24x7. >> Monitor 10 servers for $9/Month. >> Get alerted through email, SMS, voice calls or mobile push notifications. >> Take corrective actions from your mobile device. >> http://p.sf.net/sfu/Zoho >> _______________________________________________ >> Cppcms-users mailing list >> Cpp...@li... >> https://lists.sourceforge.net/lists/listinfo/cppcms-users > > ------------------------------------------------------------------------------ > Comprehensive Server Monitoring with Site24x7. > Monitor 10 servers for $9/Month. > Get alerted through email, SMS, voice calls or mobile push notifications. > Take corrective actions from your mobile device. > http://p.sf.net/sfu/Zoho > _______________________________________________ > Cppcms-users mailing list > Cpp...@li... > https://lists.sourceforge.net/lists/listinfo/cppcms-users > |