cppcms-users

[Cppcms-users] CPPCMS and huge file uploads

[Julian P. <ju...@wh...>]

Hallo,

I'm currently implementing an application which requires that files of 
approx. 130 MB can be uploaded via multipart/form-data-POST.
While files of smaller sizes (tested with up to 10MB) work, the file of 
130 MB does not. As it seems, the file is simply missing (my script 
gives error output to user if the file has not been uploaded, and when 
the big file is uploaded it indicates that the file is missing).
On the page with the upload form, two file fields are specified: a 
checksum file (32 Bytes) and the image file.
The checksum file is uploaded correctly everytime, but the image file 
only if its not too big.

This is my config.js which should contain the required settings for 
uploads of this size:

{
         "service" : {
                 "api" : "http",
                 "ip" : "0.0.0.0",
                 "port": 4444
         },
         "security" : {
                 "multipart_form_data_limit" : 3000000,
                 "uploads_path": "/mnt/root.parent"
         },
         "logging" : {
                 "level": "debug"
         }
}

Because of memory constraints, the uploads_path is set to another than 
the default location. Does this work in the current beta release 
(0.99.2.1) or do I have to use SVN?

As you see, there is no other webserver which could put constraints on 
upload file size, currently the HTTP implementation of CPPCMS is used.

There are no error outputs on stderr concerning a too big POST size, 
only logs of the HTTP requests.

Help is greatly appreciated,
Julian

Re: [Cppcms-users] CPPCMS and huge file uploads

[Artyom <art...@ya...>]

> 
> Because of memory constraints, the uploads_path is set to  another than 
> the default location. Does this work in the current beta  release 
> (0.99.2.1) or do I have to use SVN?
> 

Just to make clear what version of CppCMS are you using? If you are using 0.99.1
then upgrade to 0.99.2.1 where lots of file handing fixes where done, it it does
not work on 0.99.2.1 then I'll try to figure out what happens.

Artyom


      

Re: [Cppcms-users] CPPCMS and huge file uploads

[Julian P. <ju...@wh...>]

Am 10.08.2010 12:58, schrieb Artyom:
>>
>> Because of memory constraints, the uploads_path is set to  another than
>> the default location. Does this work in the current beta  release
>> (0.99.2.1) or do I have to use SVN?
>>
>
> Just to make clear what version of CppCMS are you using? If you are using 0.99.1
> then upgrade to 0.99.2.1 where lots of file handing fixes where done, it it does
> not work on 0.99.2.1 then I'll try to figure out what happens.
>

Currently, I'm using 0.99.2.1, now compiling latest SVN revision just to 
check whether it works there.
In my code, I did not invoke the file widget's limits() method to set 
limits, now I did this and set them to 0 for min and -1 for max, and 
validate now returns true, but the file is zero bytes long (if I set min 
to 1, validate returns false again).
Later in the code, I'm using save_to to store the uploaded file to 
another file on the same filesystem, this file is only zero bytes in 
size (checked with ls -l).
I don't know whether cppcms respects my upload_files setting, because in 
/mnt/root.parent I don't see any temporary file, and under /tmp is not 
enough space to store the entire file.

Re: [Cppcms-users] CPPCMS and huge file uploads

[Artyom <art...@ya...>]

> 
> Currently, I'm using 0.99.2.1, now compiling latest SVN  revision just to 
> check whether it works there.

There is no difference in file handling between svn and beta 0.99.2
so don't need to do.

> In my code, I did not  invoke the file widget's limits() method to set 
> limits, now I did this and  set them to 0 for min and -1 for max, and 
> validate now returns true, but the  file is zero bytes long (if I set min 
> to 1, validate returns false  again).

Before you use widget validation, just for debugging:
Print what you are actually uploaded without relation to specific widget:

  cppcms::http::request::files_type files=request().files();

  std::cout << "Got files" << std::endl;
  for(unsigned i=0;i<files.size();i++) {
     std::cout << files[i]->name() << " " << files[i]->filename() << " " << 
files[i]->size() << std::endl;
  }
  std::cout << "Done" << std::endl;

And see if you get anything uploaded.

Once you got continue to widget itself to check validation procedure.


      

Re: [Cppcms-users] CPPCMS and huge file uploads

[Artyom <art...@ya...>]

Hello,

I've tryed uploading 150Mb files with this setting, no problems.

If you are using 0.99.2.1

Then:

1. Make sure /mnt/root.parent is accessible directory.
2. Make sure that uploaded files has both file name and mime types
   provided, otherwise they would be treated as ordinary form fields.
3. Take a look in this directory if temporary files like

   /mnt/root.parent/cppcms_uploads_60ddf620746d2b8832c5028bb03a2671.tmp

   Are created during upload process.


> 
> This is my config.js which should contain the required settings for 
> uploads of this size:
> 
> {
>          "service" :  {
>                  "api" :  "http",
>                  "ip" :  "0.0.0.0",
>                  "port":  4444
>          },
>           "security" : {
>                   "multipart_form_data_limit" : 3000000,
>                   "uploads_path": "/mnt/root.parent"
>           },
>          "logging" : {
>                   "level": "debug"
>           }
> }
> 
> Because of memory constraints, the uploads_path is set to  another than 
> the default location. Does this work in the current beta  release 
> (0.99.2.1) or do I have to use SVN?
> 
> As you see, there is no  other webserver which could put constraints on 
> upload file size, currently  the HTTP implementation of CPPCMS is used.
> 
> There are no error outputs on  stderr concerning a too big POST size, 
> only logs of the HTTP  requests.
> 
> Help is greatly  appreciated,
> Julian
> 
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by 
> 
> Make an app they can't live without
> Enter the BlackBerry Developer  Challenge
> http://p.sf.net/sfu/RIM-dev2dev 
> _______________________________________________
> Cppcms-users mailing  list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cppcms-users
> 


      

Re: [Cppcms-users] CPPCMS and huge file uploads

[Julian P. <ju...@wh...>]

Ok, excuse the noise, it was a layer 8 problem here.
Because I created the image file as root (permissions are 700), I 
couldn't read it as the user which run the browser I tried to upload with.
Should have stumbled upon the fact, that the image had been transferred 
unbelievable fast via the 100Mbits network connection ^^.

Just another question: Are there any possibilities to gain information 
about the current upload progress to display it in browser via ajax or 
so in current SVN?

Thanks,
Julian

Re: [Cppcms-users] CPPCMS and huge file uploads

[Artyom <art...@ya...>]

> 
> Just another question: Are there any possibilities  to gain information 
> about the current upload progress to display it in  browser via ajax or 
> so in current  SVN?
> 

Not yet, for such information take a look on following discussion

http://comments.gmane.org/gmane.comp.lib.cppcms.user/143

Artyom