Re: [Cppcms-users] Slow http:file copy_stream for big files
Brought to you by:
artyom-beilis
Menu
▾
▴
Re: [Cppcms-users] Slow http:file copy_stream for big files
|
From: Artyom B. <art...@ya...> - 2014-10-20 14:04:11
|
1st of all please open a ticket... I got the problem... it is indeed in multipart_parser or more accuratly in the way I write to the buffered iostream... My assumption that the iostream optimization is very good was actually wrong and there is some terrible bottle neck. I'll get this fixed for 1.0.5. Artyom Beilis -------------- CppCMS - C++ Web Framework: http://cppcms.com/ CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/ ----- Original Message ----- > From: Cristian Oneț <one...@gm...> > To: cppcms-users <cpp...@li...> > Cc: > Sent: Monday, October 20, 2014 3:54 PM > Subject: Re: [Cppcms-users] Slow http:file copy_stream for big files > > 2014-10-20 16:10 GMT+03:00 Cristian Oneț <one...@gm...>: >> 2014-10-20 14:51 GMT+03:00 Artyom Beilis <art...@ya...>: >>> 1st of all for the iostreams there is a huge difference between debug > and >>> release mode additionally all the callback >>> code of http code depends on optimization. >> >> Well the difference is 20 something seconds in release vs 1 minute and >> some seconds in debug (~270 MB file), the CPU is still hugged during >> those 20 seconds, I'll try the 'out << in.rdbuf()' > version, just need >> to rebuild cppcms. > > It didn't made any difference (still 20-30 secs) so the cause must be > somewhere else like in multipart_parser (it's just a hunch), I'll let > you look at this issue :) and stop making assumptions. > > Thanks, > Cristian > > >> >>> >>> Also you are right out << in.rdbuf() should be better... probably > one of >>> these parts of code I written too fast without thinking... >>> >>> However I doubt that it is the problem. >>> >>> How http::file works when we receive an input data we write it to the >>> memory at first place but after certain threshold it >>> is written directly to the file instead of memory. So this part of > code has >>> only small use. >> >> I'm aware of that. >> >>> >>> Also I suggest when you actually "save" the file use > http::file::save_to as >>> it would rather move the file on the disk and not copy it. >>> >> >> I use it. >> >>> If you still have problems with uploading in release more it would be a > good >>> idea for me to profile the code. If so I need to know >>> what version of cppcms you use cppcms 1.0.x or trunk (there were some >>> changes in that part) >> >> I use cppcms-1.0.2. This is the test code, using the following settings: >> >> "security.multipart_form_data_limit" -> 300000; // ~300MB > multipart limit >> "security.uploads_path" -> > "d:\\tmp\\uploads" >> >> Code (note that it's a test, I know about the security issues): >> >> class uploader: public cppcms::application { >> public: >> uploader(cppcms::service &s) : >> cppcms::application(s) >> { >> } >> >> void main(std::string /*unused*/) >> { >> if (request().request_method() == "POST") >> { >> const cppcms::http::request::files_type &files = > request().files(); >> for (int i = 0; i < files.size(); ++i) >> { >> > files[i]->save_to("D:\\tmp\\uploads\\" + > files[i]->filename()); >> } >> } >> else >> { >> response().out() << "<form action='/upload' >> enctype='multipart/form-data' method='post'><input > type='text' >> name='title'><br><input type='file' > name='upload' >> multiple='multiple'><br><input type='submit' > value='Upload'></form>"; >> } >> } >> }; >> >> Regards, >> Cristian >> >>> >>> >>> Artyom Beilis >>> -------------- >>> CppCMS - C++ Web Framework: http://cppcms.com/ >>> CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/ >>> >>> ________________________________ >>> From: Cristian Oneț <one...@gm...> >>> To: cppcms-users <cpp...@li...> >>> Sent: Monday, October 20, 2014 1:06 PM >>> Subject: [Cppcms-users] Slow http:file copy_stream for big files >>> >>> Hi, >>> >>> I'm using a simplified version (no widgets::file only > request::files) >>> of the uploader [1] to implement an upload service. What I observed is >>> that uploading large files takes a lot of CPU and takes way more than >>> a similar nodejs service [2] (1 min vs 1 s for the same file - 300 >>> MB). I have to admit that this was cppcms compiled in debug against a >>> release nodejs but the difference is too great. >>> >>> Intrigued by this finding I've took a glimpse at how the file is > saved >>> to the temporary location using the input stream and found [3]. >>> Reading from the input stream in 1024 byte chunks then writing that to >>> the output does not seem such a great idea and could explain the high >>> CPU usage and the overall slowness. >>> >>> Shouldn't this perform better? >>> >>> out << in.rdbuf(); >>> >>> I promise I'll get back with the results of running a release > build. >>> >>> Regards, >>> Cristian >>> >>> [1] > http://cppcms.com/cppcms_ref/latest/ex_uploads____uploader__cpp.html >>> [2] https://github.com/felixge/node-formidable >>> [3] >>> > http://sourceforge.net/p/cppcms/code/HEAD/tree/framework/trunk/src/http_file.cpp#l78 >>> >>> > ------------------------------------------------------------------------------ >>> Comprehensive Server Monitoring with Site24x7. >>> Monitor 10 servers for $9/Month. >>> Get alerted through email, SMS, voice calls or mobile push > notifications. >>> Take corrective actions from your mobile device. >>> http://p.sf.net/sfu/Zoho >>> _______________________________________________ >>> Cppcms-users mailing list >>> Cpp...@li... >>> https://lists.sourceforge.net/lists/listinfo/cppcms-users >>> >>> >>> >>> > ------------------------------------------------------------------------------ >>> Comprehensive Server Monitoring with Site24x7. >>> Monitor 10 servers for $9/Month. >>> Get alerted through email, SMS, voice calls or mobile push > notifications. >>> Take corrective actions from your mobile device. >>> http://p.sf.net/sfu/Zoho >>> _______________________________________________ >>> Cppcms-users mailing list >>> Cpp...@li... >>> https://lists.sourceforge.net/lists/listinfo/cppcms-users >>> > > ------------------------------------------------------------------------------ > Comprehensive Server Monitoring with Site24x7. > Monitor 10 servers for $9/Month. > Get alerted through email, SMS, voice calls or mobile push notifications. > Take corrective actions from your mobile device. > http://p.sf.net/sfu/Zoho > _______________________________________________ > Cppcms-users mailing list > Cpp...@li... > https://lists.sourceforge.net/lists/listinfo/cppcms-users > |
