Thread: [Cppcms-users] upload progress
Brought to you by:
artyom-beilis
Menu
▾
▴
cppcms-users
|
From: Frank E. <fra...@an...> - 2010-07-18 07:50:48
|
hi, i need (must have) the possibility to know how much data of an ongoing post to the cppcms application has been transmitted. i need to be able to give somekind of upload progress feedback. when i'm not mistaken this is not possible using cpcms - is there any workaround for that? or is it possible to implement something like that? maybe somekind of instance which can be queried for in progress post transfers using some kind of i which the js application also uses to transmit the file in the post and returns the number of bytes transmitted and if available the full size to be transmitted? any ideas? frank. -- Dipl.-Ing. (FH) Frank Enderle anamica UG (haftungsbeschränkt) Beinsteinerstr. 6 71334 Waiblingen Telefon: +49 151 14981091 Telefax: +49 7151 1335770 E-Mail: fra...@an... Internet: www.anamica.de Handelsregister: AG Stuttgart HRB 732357 Geschäftsführer: Yvonne Holzwarth, Frank Enderle |
|
From: Artyom <art...@ya...> - 2010-07-18 09:33:51
|
Hello, Currently, all upload is done without involvement of the user application, the user application is called only after processing all POST data. But better upload handling should be implemented in RC1, as part of better upload validation. It is in task list, see: http://art-blog.no-ip.info/wikipp/en/page/cppcms_1x_tasks#Implement+Pre-Upload+file+validation+-+Mrc. Generally as part of API I think it would be possible to add some upload hooks so the application can have better control over upload process and so it would be able to do anything, it wants. Additional option may be give to give an application full control over processing POST data. So in some cases, application could be "called" before actual POST data is transferred. In such case would be able to handle all notification as in regular comet application. But this requires work and API design. So if you can propose some suggestions and requirements for such API, they may be actually implemented in CppCMS :-) Best, Artyom P.S.: Isn't java-script has an ability to do such things on client side? > > hi, > > i need (must have) the possibility to know how much data of an ongoing > post to the cppcms application has been transmitted. i need to be able > to give somekind of upload progress feedback. when i'm not mistaken this > is not possible using cpcms - is there any workaround for that? or is it > possible to implement something like that? > > maybe somekind of instance which can be queried for in progress post > transfers using some kind of i which the js application also uses to > transmit the file in the post and returns the number of bytes > transmitted and if available the full size to be transmitted? > > any ideas? > > frank. > > -- > Dipl.-Ing. (FH) Frank Enderle > > anamica UG (haftungsbeschränkt) > Beinsteinerstr. 6 > 71334 Waiblingen > > Telefon: +49 151 14981091 > Telefax: +49 7151 1335770 > E-Mail: fra...@an... > Internet: www.anamica.de > > Handelsregister: AG Stuttgart HRB 732357 > Geschäftsführer: Yvonne Holzwarth, Frank Enderle > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by Sprint > What will you do first with EVO, the first 4G phone? > Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first > _______________________________________________ > Cppcms-users mailing list > Cpp...@li... > https://lists.sourceforge.net/lists/listinfo/cppcms-users > |
|
From: Frank E. <fra...@an...> - 2010-07-18 11:27:46
|
i suppose the callbacks would be a good thing. it should include - a possibility to cancel a transfer before it has even started (i guess this is what you mean with preupload validation) - an option to let the file be received by cppcms (cppcms handles storing the file just like it does now) optionally with a callback whcih tells the applicationt he amount of received bytes. - an option to pass the data while receiving directly to an application callback so the application can save the data to the target itself (no duplicate data holding since no temp files or temp memory blocks are involved in receiving the file) it should be simple and simple to use. since i now need this functionality i'll try to hack something in the the current library version, but it won't be a beautiful solution and i'd like to replace it as soosn as available with a version from upstream. frank Am 18.07.2010 11:33, schrieb Artyom: > Hello, > > Currently, all upload is done without involvement of the user application, > the user application is called only after processing all POST data. > But better upload handling should be implemented in RC1, as part of better > upload validation. > > It is in task list, see: > > > http://art-blog.no-ip.info/wikipp/en/page/cppcms_1x_tasks#Implement+Pre-Upload+file+validation+-+Mrc. > > > Generally as part of API I think it would be possible to add some upload hooks > so the > application can have better control over upload process and so it would be able > to do anything, > it wants. > > Additional option may be give to give an application full control over > processing POST data. > So in some cases, application could be "called" before actual POST data is > transferred. > > In such case would be able to handle all notification as in regular comet > application. > > But this requires work and API design. So if you can propose some suggestions > and requirements for > such API, they may be actually implemented in CppCMS :-) > > Best, > Artyom > > P.S.: Isn't java-script has an ability to do such things on client side? > > >> >> hi, >> >> i need (must have) the possibility to know how much data of an ongoing >> post to the cppcms application has been transmitted. i need to be able >> to give somekind of upload progress feedback. when i'm not mistaken this >> is not possible using cpcms - is there any workaround for that? or is it >> possible to implement something like that? >> >> maybe somekind of instance which can be queried for in progress post >> transfers using some kind of i which the js application also uses to >> transmit the file in the post and returns the number of bytes >> transmitted and if available the full size to be transmitted? >> >> any ideas? >> >> frank. >> >> -- >> Dipl.-Ing. (FH) Frank Enderle >> >> anamica UG (haftungsbeschränkt) >> Beinsteinerstr. 6 >> 71334 Waiblingen >> >> Telefon: +49 151 14981091 >> Telefax: +49 7151 1335770 >> E-Mail: fra...@an... >> Internet: www.anamica.de >> >> Handelsregister: AG Stuttgart HRB 732357 >> Geschäftsführer: Yvonne Holzwarth, Frank Enderle >> >> ------------------------------------------------------------------------------ >> This SF.net email is sponsored by Sprint >> What will you do first with EVO, the first 4G phone? >> Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first >> _______________________________________________ >> Cppcms-users mailing list >> Cpp...@li... >> https://lists.sourceforge.net/lists/listinfo/cppcms-users >> > > > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by Sprint > What will you do first with EVO, the first 4G phone? > Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first > _______________________________________________ > Cppcms-users mailing list > Cpp...@li... > https://lists.sourceforge.net/lists/listinfo/cppcms-users -- Dipl.-Ing. (FH) Frank Enderle anamica UG (haftungsbeschränkt) Beinsteinerstr. 6 71334 Waiblingen Telefon: +49 151 14981091 Telefax: +49 7151 1335770 E-Mail: fra...@an... Internet: www.anamica.de Handelsregister: AG Stuttgart HRB 732357 Geschäftsführer: Yvonne Holzwarth, Frank Enderle |
|
From: Artyom <art...@ya...> - 2010-07-18 11:34:22
|
>
> it should be simple and simple to use. since i now need this
> functionality i'll try to hack something in the the current library
> version, but it won't be a beautiful solution and i'd like to replace it
> as soosn as available with a version from upstream.
>
> frank
>
For this you may take a look on:
- http_file.cpp, which handles incoming chunks.
- cgi_api.cpp functions: load_content and on_some_multipart_read
Note: at this point there is no associations with an application yet,
and there is no session information loaded, so you should be very careful
with the assumptions of what you can actually do.
Artyom
|
|
From: augustin <aug...@ov...> - 2010-07-18 11:00:51
|
On Sunday 18 July 2010 05:33:44 pm Artyom wrote: > But this requires work and API design. So if you can propose some > suggestions and requirements for > such API, they may be actually implemented in CppCMS I piggy-back on this thread to ask a related question: It is one thing to upload a simple picture, a pdf, a text file or any kind of small file. It is another thing entirely to upload a 100MB or 1GB file... Must such big files be uploaded via ftp? Isn't there any practical limit with the browser or the HTTProtocol about such large file transfers? Frank: if you need an upload progress API, I presume it's precisely because your application needs to process some large file transfers. Can you shed some light on this topic? From CppCMS's point of view, would the file size matter for the possibly upcoming file transfer API? Thanks, Augustin. -- Friends: http://www.reuniting.info/ http://activistsolutions.org/ My projects: http://astralcity.org/ http://3enjeux.overshoot.tv/ http://linux.overshoot.tv/ http://overshoot.tv/ http://charityware.info/ http://masquilier.org/ http://openteacher.info/ http://minguo.info/ http://www.wechange.org/ http://searching911.info/ . |
|
From: Artyom <art...@ya...> - 2010-07-18 11:27:43
|
> It is one thing to upload a simple picture, a pdf, a text file or any kind of
> small file.
> It is another thing entirely to upload a 100MB or 1GB file... Must such big
> files be uploaded via ftp? Isn't there any practical limit with the browser or
>
> the HTTProtocol about such large file transfers?
Why wouldn't you use HTTP? It does not have any overhead when uploading big
files.
There are many cases when HTTP is quite convenient, and uploading 50, 100MB or
even 10GB
is fine from technological point of view (for example YouSendIt)
>
> From CppCMS's point of view, would the file size matter for the possibly
> upcoming file transfer API?
Currently there is no technical limits on file size upload, but:
- It is limited to 64MB by default, this can be changed by setting:
security.multipart_form_data_limit to the required size in KB.
- For large files it is better to use save_to() member function of http::file
which would cause rather file rename then copy.
|
|
From: Frank E. <fra...@an...> - 2010-07-18 11:12:23
|
> Frank: if you need an upload progress API, I presume it's precisely > because your application needs to process some large file transfers. > Can you shed some light on this topic? the app needs to transfer bigger files in a multipart post if possible. the filesize will be between 1 and 10mb. so the full post might easiliy reach a few tens of megs. Am 18.07.2010 12:59, schrieb augustin: > On Sunday 18 July 2010 05:33:44 pm Artyom wrote: >> But this requires work and API design. So if you can propose some >> suggestions and requirements for >> such API, they may be actually implemented in CppCMS > > I piggy-back on this thread to ask a related question: > > It is one thing to upload a simple picture, a pdf, a text file or any kind of > small file. > It is another thing entirely to upload a 100MB or 1GB file... Must such big > files be uploaded via ftp? Isn't there any practical limit with the browser or > the HTTProtocol about such large file transfers? > > Frank: if you need an upload progress API, I presume it's precisely because > your application needs to process some large file transfers. Can you shed some > light on this topic? > >>From CppCMS's point of view, would the file size matter for the possibly > upcoming file transfer API? > > Thanks, > > Augustin. > > > -- Dipl.-Ing. (FH) Frank Enderle anamica UG (haftungsbeschränkt) Beinsteinerstr. 6 71334 Waiblingen Telefon: +49 151 14981091 Telefax: +49 7151 1335770 E-Mail: fra...@an... Internet: www.anamica.de Handelsregister: AG Stuttgart HRB 732357 Geschäftsführer: Yvonne Holzwarth, Frank Enderle |
|
From: augustin <aug...@ov...> - 2010-07-18 11:19:19
|
On Sunday 18 July 2010 07:12:14 pm Frank Enderle wrote: > the app needs to transfer bigger files in a multipart post if possible. > the filesize will be between 1 and 10mb. so the full post might easiliy > reach a few tens of megs. Thanks. There is indeed a limit. Yet, there must be a way that is user friendly and does not require to break the file into bits: how does youtube manage this? We can upload very large files, there. The interface is friendly enough and all is seemingly done via http with some help from javascript. Maybe we should try to figure out how they do it. Augustin. -- Friends: http://www.reuniting.info/ http://activistsolutions.org/ My projects: http://astralcity.org/ http://3enjeux.overshoot.tv/ http://linux.overshoot.tv/ http://overshoot.tv/ http://charityware.info/ http://masquilier.org/ http://openteacher.info/ http://minguo.info/ http://www.wechange.org/ http://searching911.info/ . |
|
From: Frank E. <fra...@an...> - 2010-07-21 19:59:33
|
hi, after some research i found the following facts to be helpful for upload progress using nginx with cppcms backend in fastcgi mode: i implemented some global object to track running uploads in cppcms. after i got it running i learned that nginx buffers the _complete_ post before sending it to the fastcgi backend. this is by design and not alterable. i also stumbled upon a nginx module which can be used to implement upload progress indicators using java script and XHR - solved completely within nginx. so i found what i searched for (upload progress indicator), but this also means that - cppcms with nginx can not be used to 'stream' post data directly to cppcms backend in fastcgi mode - nginx always cashes post requests and sends them then completely to the fastcgi backend (cppcms) which does not block cppcms while receiving data from a slow connection (which could be a good thing) - i don't need to patch cppcms to support progress indicatorrs for post requests (which i'm glad about) though this is not really new it might help a little bit. it's fo sure something to be mentioned in the documentation when the upload handling is changed in RC1. as i understood this will offer the possibility to abort a running upload; well if you have nginx in front it might not work as expected. http://wiki.nginx.org/NginxHttpUploadProgressModule - lighthttpd also has a module like that. regards, frank Am 18.07.2010 13:34, schrieb Artyom: >> >> it should be simple and simple to use. since i now need this >> functionality i'll try to hack something in the the current library >> version, but it won't be a beautiful solution and i'd like to replace it >> as soosn as available with a version from upstream. >> >> frank >> > > > For this you may take a look on: > > - http_file.cpp, which handles incoming chunks. > - cgi_api.cpp functions: load_content and on_some_multipart_read > > Note: at this point there is no associations with an application yet, > and there is no session information loaded, so you should be very careful > with the assumptions of what you can actually do. > > Artyom > > > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by Sprint > What will you do first with EVO, the first 4G phone? > Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first > _______________________________________________ > Cppcms-users mailing list > Cpp...@li... > https://lists.sourceforge.net/lists/listinfo/cppcms-users -- Dipl.-Ing. (FH) Frank Enderle anamica UG (haftungsbeschränkt) Beinsteinerstr. 6 71334 Waiblingen Telefon: +49 151 14981091 Telefax: +49 7151 1335770 E-Mail: fra...@an... Internet: www.anamica.de Handelsregister: AG Stuttgart HRB 732357 Geschäftsführer: Yvonne Holzwarth, Frank Enderle |
|
From: Artyom <art...@ya...> - 2010-07-23 09:49:15
|
> hi,
>
>
> i implemented some global object to track running uploads in cppcms.
> after i got it running i learned that nginx buffers the _complete_ post
> before sending it to the fastcgi backend. this is by design and not
> alterable.
Ok, its quite sad, -1 more point for Nginx for me.
> - cppcms with nginx can not be used to 'stream' post data directly to
> cppcms backend in fastcgi mode
> - nginx always cashes post requests and sends them then completely to
> the fastcgi backend (cppcms) which does not block cppcms while receiving
> data from a slow connection (which could be a good thing)
Ok, it is good point to remember this in future, probably even
it is good to add it to the wiki at some point
> though this is not really new it might help a little bit. it's fo sure
> something to be mentioned in the documentation when the upload handling
> is changed in RC1.
It would not be "changed" but rather enhanced so users would be able
to have better control over uploads that today if they want.
In any case as preparation to future implementation of web-sockets
API I think I'll add an asynchronous API for fetching POST data.
But this is in TODO list.
> as i understood this will offer the possibility to
> abort a running upload; well if you have nginx in front it might not
> work as expected.
This is mostly web server issue, to be honest, almost every web server has
its own quirks, for example only nginx reports on closed-by-peer connections
lighttpd and apache don't do this at all.
Regards,
Artyom
|
|
From: Zheng P. <ky...@gm...> - 2010-07-24 01:33:57
|
I think swfupload is a good solution. > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by Sprint > What will you do first with EVO, the first 4G phone? > Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first > _______________________________________________ > Cppcms-users mailing list > Cpp...@li... > https://lists.sourceforge.net/lists/listinfo/cppcms-users > -- with kind regards |
|
From: pavel k. <un...@fu...> - 2010-11-07 00:10:21
|
Zheng Ping <kytexzy@...> writes: > > > I think swfupload is a good solution. maybe plupload is better long term solution. did anybody tried to use it with cppcms? or anything else to handle multiple files upload (with chunking)? |
|
From: Aris S. <ari...@gm...> - 2010-11-07 02:45:14
|
another solution in the client side is using YUI 3 uploader, an swfuploader, which have BDS license. The YUI Uploader leverages Flash to provide file upload functionality beyond the native browser-based methods. Specifically, the Uploader allows for: 1. Multiple file selection in a single "Open File" dialog. 2. File extension filters to facilitate the user's selection. 3. Progress tracking for file uploads. 4. A range of available file metadata: filename, size, date created, date modified, and author. 5. A set of events dispatched on various aspects of the file upload process: file selection, upload progress, upload completion, data return, and upload errors. 6. Inclusion of additional data in the file upload POST request. 7. Faster file upload on broadband connections (due to the modified SEND buffer size). for a complete description, read on http://developer.yahoo.com/yui/3/uploader/ for single file upload example, read on http://developer.yahoo.com/yui/3/examples/uploader/uploader-simple.html# for multiple file, read on http://developer.yahoo.com/yui/3/examples/uploader/uploader-multiple.html another feature of YUI oploader is that "the file upload can be aborted" with calling "void cancel(fileID)" method, read on http://www.yuiblog.com/blog/2009/02/26/flickr-uploadr/ for the complete api, read on http://developer.yahoo.com/yui/3/api/Uploader.html |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X