Re: [Cppcms-users] Slow http:file copy_stream for big files
Brought to you by:
artyom-beilis
Menu
▾
▴
Re: [Cppcms-users] Slow http:file copy_stream for big files
|
From: Cristian O. <one...@gm...> - 2014-10-21 07:46:03
|
2014-10-21 8:40 GMT+03:00 Artyom Beilis <art...@ya...>: > Thanks... > > Can you take a look on v1.0_fixes branch of the cppcms > > This is the fix I had written: https://sourceforge.net/p/cppcms/code/2271/ I've tried only this patch applied on 1.0.2 (I could setup my tests faster this way). > > It improved 600MB file upload from 30-40s to 4-5s. And it is twice faster than PHP > (that has multipart parsing embedded into the C code of PHP) It improved my previous ~20s upload to ~5s. The nodejs app still beats this with ~3s but I guess that's acceptable since it could depend on build options. I'm still building cppcms with MSVC 2005. Thanks for the fix. Regards, Cristian > > So I assume it is almost as efficient as it can get. > > Take a look and if it is ok I'll close the ticket. It would get > released in 1.0.5 > > Artyom Beilis > -------------- > CppCMS - C++ Web Framework: http://cppcms.com/ > CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/ > > > ----- Original Message ----- >> From: Cristian Oneț <one...@gm...> >> To: cppcms-users <cpp...@li...> >> Cc: >> Sent: Monday, October 20, 2014 6:41 PM >> Subject: Re: [Cppcms-users] Slow http:file copy_stream for big files >> >> 2014-10-20 17:04 GMT+03:00 Artyom Beilis <art...@ya...>: >>> 1st of all please open a ticket... >> >> Done. >> >> https://sourceforge.net/p/cppcms/feature-requests/27/ >> >> Regards, >> Cristian >> >>> >>> I got the problem... it is indeed in multipart_parser or more accuratly >>> in the way I write to the buffered iostream... >>> >>> My assumption that the iostream optimization is very good was actually >> wrong >>> and there is some terrible bottle neck. >>> >>> I'll get this fixed for 1.0.5. >>> >>> Artyom Beilis >>> -------------- >>> CppCMS - C++ Web Framework: http://cppcms.com/ >>> CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/ >>> >>> >>> ----- Original Message ----- >>>> From: Cristian Oneț <one...@gm...> >>>> To: cppcms-users <cpp...@li...> >>>> Cc: >>>> Sent: Monday, October 20, 2014 3:54 PM >>>> Subject: Re: [Cppcms-users] Slow http:file copy_stream for big files >>>> >>>> 2014-10-20 16:10 GMT+03:00 Cristian Oneț >> <one...@gm...>: >>>>> 2014-10-20 14:51 GMT+03:00 Artyom Beilis >> <art...@ya...>: >>>>>> 1st of all for the iostreams there is a huge difference >> between debug >>>> and >>>>>> release mode additionally all the callback >>>>>> code of http code depends on optimization. >>>>> >>>>> Well the difference is 20 something seconds in release vs 1 minute >> and >>>>> some seconds in debug (~270 MB file), the CPU is still hugged >> during >>>>> those 20 seconds, I'll try the 'out << >> in.rdbuf()' >>>> version, just need >>>>> to rebuild cppcms. >>>> >>>> It didn't made any difference (still 20-30 secs) so the cause must >> be >>>> somewhere else like in multipart_parser (it's just a hunch), >> I'll let >>>> you look at this issue :) and stop making assumptions. >>>> >>>> Thanks, >>>> Cristian >>>> >>>> >>>>> >>>>>> >>>>>> Also you are right out << in.rdbuf() should be better... >> probably >>>> one of >>>>>> these parts of code I written too fast without thinking... >>>>>> >>>>>> However I doubt that it is the problem. >>>>>> >>>>>> How http::file works when we receive an input data we write >> it to the >>>>>> memory at first place but after certain threshold it >>>>>> is written directly to the file instead of memory. So this >> part of >>>> code has >>>>>> only small use. >>>>> >>>>> I'm aware of that. >>>>> >>>>>> >>>>>> Also I suggest when you actually "save" the file use >>>> http::file::save_to as >>>>>> it would rather move the file on the disk and not copy it. >>>>>> >>>>> >>>>> I use it. >>>>> >>>>>> If you still have problems with uploading in release more it >> would be a >>>> good >>>>>> idea for me to profile the code. If so I need to know >>>>>> what version of cppcms you use cppcms 1.0.x or trunk (there >> were some >>>>>> changes in that part) >>>>> >>>>> I use cppcms-1.0.2. This is the test code, using the following >> settings: >>>>> >>>>> "security.multipart_form_data_limit" -> 300000; // >> ~300MB >>>> multipart limit >>>>> "security.uploads_path" -> >>>> "d:\\tmp\\uploads" >>>>> >>>>> Code (note that it's a test, I know about the security >> issues): >>>>> >>>>> class uploader: public cppcms::application { >>>>> public: >>>>> uploader(cppcms::service &s) : >>>>> cppcms::application(s) >>>>> { >>>>> } >>>>> >>>>> void main(std::string /*unused*/) >>>>> { >>>>> if (request().request_method() == "POST") >>>>> { >>>>> const cppcms::http::request::files_type &files = >>>> request().files(); >>>>> for (int i = 0; i < files.size(); ++i) >>>>> { >>>>> >>>> >> files[i]->save_to("D:\\tmp\\uploads\\" + >>>> files[i]->filename()); >>>>> } >>>>> } >>>>> else >>>>> { >>>>> response().out() << "<form >> action='/upload' >>>>> enctype='multipart/form-data' >> method='post'><input >>>> type='text' >>>>> name='title'><br><input type='file' >>>> name='upload' >>>>> multiple='multiple'><br><input >> type='submit' >>>> value='Upload'></form>"; >>>>> } >>>>> } >>>>> }; >>>>> >>>>> Regards, >>>>> Cristian >>>>> >>>>>> >>>>>> >>>>>> Artyom Beilis >>>>>> -------------- >>>>>> CppCMS - C++ Web Framework: http://cppcms.com/ >>>>>> CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/ >>>>>> >>>>>> ________________________________ >>>>>> From: Cristian Oneț <one...@gm...> >>>>>> To: cppcms-users <cpp...@li...> >>>>>> Sent: Monday, October 20, 2014 1:06 PM >>>>>> Subject: [Cppcms-users] Slow http:file copy_stream for big >> files >>>>>> >>>>>> Hi, >>>>>> >>>>>> I'm using a simplified version (no widgets::file only >>>> request::files) >>>>>> of the uploader [1] to implement an upload service. What I >> observed is >>>>>> that uploading large files takes a lot of CPU and takes way >> more than >>>>>> a similar nodejs service [2] (1 min vs 1 s for the same file - >> 300 >>>>>> MB). I have to admit that this was cppcms compiled in debug >> against a >>>>>> release nodejs but the difference is too great. >>>>>> >>>>>> Intrigued by this finding I've took a glimpse at how the >> file is >>>> saved >>>>>> to the temporary location using the input stream and found >> [3]. >>>>>> Reading from the input stream in 1024 byte chunks then writing >> that to >>>>>> the output does not seem such a great idea and could explain >> the high >>>>>> CPU usage and the overall slowness. >>>>>> >>>>>> Shouldn't this perform better? >>>>>> >>>>>> out << in.rdbuf(); >>>>>> >>>>>> I promise I'll get back with the results of running a >> release >>>> build. >>>>>> >>>>>> Regards, >>>>>> Cristian >>>>>> >>>>>> [1] >>>> http://cppcms.com/cppcms_ref/latest/ex_uploads____uploader__cpp.html >>>>>> [2] https://github.com/felixge/node-formidable >>>>>> [3] >>>>>> >>>> >> http://sourceforge.net/p/cppcms/code/HEAD/tree/framework/trunk/src/http_file.cpp#l78 >>>>>> >>>>>> >>>> >> ------------------------------------------------------------------------------ >>>>>> Comprehensive Server Monitoring with Site24x7. >>>>>> Monitor 10 servers for $9/Month. >>>>>> Get alerted through email, SMS, voice calls or mobile push >>>> notifications. >>>>>> Take corrective actions from your mobile device. >>>>>> http://p.sf.net/sfu/Zoho >>>>>> _______________________________________________ >>>>>> Cppcms-users mailing list >>>>>> Cpp...@li... >>>>>> https://lists.sourceforge.net/lists/listinfo/cppcms-users >>>>>> >>>>>> >>>>>> >>>>>> >>>> >> ------------------------------------------------------------------------------ >>>>>> Comprehensive Server Monitoring with Site24x7. >>>>>> Monitor 10 servers for $9/Month. >>>>>> Get alerted through email, SMS, voice calls or mobile push >>>> notifications. >>>>>> Take corrective actions from your mobile device. >>>>>> http://p.sf.net/sfu/Zoho >>>>>> _______________________________________________ >>>>>> Cppcms-users mailing list >>>>>> Cpp...@li... >>>>>> https://lists.sourceforge.net/lists/listinfo/cppcms-users >> >>>>>> >>>> >>>> >> ------------------------------------------------------------------------------ >>>> Comprehensive Server Monitoring with Site24x7. >>>> Monitor 10 servers for $9/Month. >>>> Get alerted through email, SMS, voice calls or mobile push >> notifications. >>>> Take corrective actions from your mobile device. >>>> http://p.sf.net/sfu/Zoho >>>> _______________________________________________ >>>> Cppcms-users mailing list >>>> Cpp...@li... >>>> https://lists.sourceforge.net/lists/listinfo/cppcms-users >>>> >>> >>> >> ------------------------------------------------------------------------------ >>> Comprehensive Server Monitoring with Site24x7. >>> Monitor 10 servers for $9/Month. >>> Get alerted through email, SMS, voice calls or mobile push notifications. >>> Take corrective actions from your mobile device. >>> http://p.sf.net/sfu/Zoho >>> _______________________________________________ >>> Cppcms-users mailing list >>> Cpp...@li... >>> https://lists.sourceforge.net/lists/listinfo/cppcms-users >> >> ------------------------------------------------------------------------------ >> Comprehensive Server Monitoring with Site24x7. >> Monitor 10 servers for $9/Month. >> Get alerted through email, SMS, voice calls or mobile push notifications. >> Take corrective actions from your mobile device. >> http://p.sf.net/sfu/Zoho >> _______________________________________________ >> Cppcms-users mailing list >> Cpp...@li... >> https://lists.sourceforge.net/lists/listinfo/cppcms-users >> > > ------------------------------------------------------------------------------ > Comprehensive Server Monitoring with Site24x7. > Monitor 10 servers for $9/Month. > Get alerted through email, SMS, voice calls or mobile push notifications. > Take corrective actions from your mobile device. > http://p.sf.net/sfu/Zoho > _______________________________________________ > Cppcms-users mailing list > Cpp...@li... > https://lists.sourceforge.net/lists/listinfo/cppcms-users |
