cppcms-users

[Cppcms-users] Uploads to slow storage devices

[Julian P. <ju...@wh...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hallo,

I recently experienced some problems when uploading (large) files (>=30
MB) with cherokee to a cppcms based backend application: After a part of
the upload has completed, a 500 Internal Server error is issued to the
browser and the connection (and upload) is terminated. These problems
only occur on an embedded board with a Cortex A8-ARM processor and a
very slow NAND memory (where the file shall be stored to). For file
corruption prevention, the file system (ext3) on this NAND is mounted in
sync mode so that no data is buffered in RAM but anything is written
directly to NAND. Even if async mode is used instead, the error occurs
after some more time (probably if the kernel's I/O buffer is full).
cherokee delivers the upload to the backend application via FastCGI.
My question is: Is there anything within the FastCGI/upload handling in
cppcms that may cause such a behaviour, if e.g. the backend application
needs to long to handle the request (because the file has to be stored
to NAND before the request is handled), or is it all the fault of the
webserver?
As a sidenote I may add that we successfully use the very same backend
application on several other systems with different architectures, but
faster storage devices and even on the very same board using SD cards
instead of onboard NAND.
As far as I know, the cherokee webserver passes uploaded files directly
to FastCGI applications (while other webservers like lighttpd or nginx
buffer the whole or at least part of the upload file before they deliver
it to the FastCGI application).

Thanks for your help,
Julian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJPhY8dAAoJENidYKvYQHlQRuUP/20Ydttxaqk2rwCydAQKZZIz
DiqAzr0qL9FaJCHkUswOCYXhnp5upbB6qpBcjfYFE0Rtr81skGhLvP8b2EhyyU7X
bQ9wtvm/gYaigWhULtLs7XNDxQGVLWv3CEUD83h2gGqNFt1zGc/Er+R0ogjlFIFk
jbe94IXAzOpYRepIwKOifxdU+QOZCrcrvRZhRMiFnA+XxGBfQsZTHrIYCxCYAAIY
BBmcxrOMzyHhoVoZQNQDPPwqqbtpnUkxmWT2lHB5BBMvauokHSkRnvu73zxPonH3
WUj+6X4Jtc3CT9N2cbvzAnXqXUnFx5VWnIJ+aQuKUxqjStPE94DKdjxiJ4PHivQP
5f30ebdOxZa9SUHS+5liD+yIiojCYslyLpjciCCio4TFWxDoJdbPDns59Q/v5zQO
yyEh8+jKZq214HuiVaFW/jHOLPcOilsmS9ywsvz6FCXOhj4wHQH/Io7exoaa26i7
9Cr3SMsITyT3a0DnPfZZq9YsKy2K7Y0w6xp3MHP82gahOdaDMaiDAEXOaY/xT07v
8a17GQFRVsHFGLeILD16zE35YTH4tK5gJCZbL2lL9/8mbITcv8zNDzguP8vxXnuq
dQ/u33gU/Wqpc/pKgCkkTugMHCBRIhwshl+KWILoTuCuhBOyX9gHEOPWvAghNwkn
AoksBVwbHj5QWMwSEJsa
=yEBf
-----END PGP SIGNATURE-----

Re: [Cppcms-users] Uploads to slow storage devices

[Artyom B. <art...@ya...>]

> 

> I recently experienced some problems when uploading (large) files (>=30
> MB) with cherokee to a cppcms based backend application: After a part of
> the upload has completed, a 500 Internal Server error is issued to the
> browser and the connection (and upload) is terminated. These problems
> only occur on an embedded board with a Cortex A8-ARM processor and a
> very slow NAND memory (where the file shall be stored to). For file
> corruption prevention, the file system (ext3) on this NAND is mounted in
> sync mode so that no data is buffered in RAM but anything is written
> directly to NAND. Even if async mode is used instead, the error occurs
> after some more time (probably if the kernel's I/O buffer is full).
> cherokee delivers the upload to the backend application via FastCGI.
> My question is: Is there anything within the FastCGI/upload handling in
> cppcms that may cause such a behaviour, if e.g. the backend application
> needs to long to handle the request (because the file has to be stored
> to NAND before the request is handled), or is it all the fault of the
> webserver?

What version of CppCMS do you use? I assume latest.

I'd recommend you to run cppcms process with strace and
see what exactly happens when you write to file, i.e. what
system call fails: you get the error from the file
system (i.e. write(2) failes) or the connection is closed
by the cherokee.

Also I'd recommend to see if there is anything in the log.

Generally there is no timeouts on FastCGI so I don't see
were the problem can come from.

Also make sure the the files are uploaded to correct location
so you have enough space.

  http://cppcms.com/wikipp/en/page/cppcms_1x_config#security.uploads_path


> As a sidenote I may add that we successfully use the very same backend
> application on several other systems with different architectures, but
> faster storage devices and even on the very same board using SD cards
> instead of onboard NAND.
> As far as I know, the cherokee webserver passes uploaded files directly
> to FastCGI applications (while other webservers like lighttpd or nginx
> buffer the whole or at least part of the upload file before they deliver
> it to the FastCGI application).
> 
> Thanks for your help,
> Julian


Best Regards,

Artyom Beilis

Re: [Cppcms-users] Uploads to slow storage devices

[Julian P. <ju...@wh...>]

Am 11.04.2012 19:54, schrieb Artyom Beilis:
>> I recently experienced some problems when uploading (large) files (>=30
>> MB) with cherokee to a cppcms based backend application: After a part of
>> the upload has completed, a 500 Internal Server error is issued to the
>> browser and the connection (and upload) is terminated. These problems
>> only occur on an embedded board with a Cortex A8-ARM processor and a
>> very slow NAND memory (where the file shall be stored to). For file
>> corruption prevention, the file system (ext3) on this NAND is mounted in
>> sync mode so that no data is buffered in RAM but anything is written
>> directly to NAND. Even if async mode is used instead, the error occurs
>> after some more time (probably if the kernel's I/O buffer is full).
>> cherokee delivers the upload to the backend application via FastCGI.
>> My question is: Is there anything within the FastCGI/upload handling in
>> cppcms that may cause such a behaviour, if e.g. the backend application
>> needs to long to handle the request (because the file has to be stored
>> to NAND before the request is handled), or is it all the fault of the
>> webserver?
> What version of CppCMS do you use? I assume latest.

To be honest, I think that this problem occurred in 0.99.6. I didn't
test any later versions yet. Are there changes that impact file I/O in
later releases?

> Also make sure the the files are uploaded to correct location
> so you have enough space.
>
>
The location is correct and there is plenty of space available (>2-3x of
the amount needed).

My idea just was, that sometime during upload the write to the NAND
stalls and the corresponding write call blocks so long, that a timeout
or such is hit inside the libcppcms. Another probability would be, that
something in the communication between cherokee and libcppcms fails.
How does transferring requests from cherokee to libcppcms work exactly?
Think of an incoming HTTP POST request with a huge file appended.
Cherokee sends this request to libcppcms, and it would start to store
the file to the designated temporary location. What does happen, if
writing to NAND is very slow? Would libcppcms still accept bytes of the
request from cherokee or would it fetch new bytes only if the already
received bytes have been written to NAND (the consequence could be, that
the network buffer queue inside of the kernel grows until cherokee has
to wait with writing because no new packages can be sent, and this might
hit a timeout there or lead to a buffer exhaustion that makes cherokee
close the connection)? If libcppcms instead buffers internally if more
bytes are incoming than can be written to NAND, are there any size
limits to these buffers?

Best regards,
Julian

Re: [Cppcms-users] Uploads to slow storage devices

[Artyom B. <art...@ya...>]

----- Original Message -----

> From: Julian Pietron <ju...@wh...>

>>  What version of CppCMS do you use? I assume latest.
> 
> To be honest, I think that this problem occurred in 0.99.6. I didn't
> test any later versions yet. Are there changes that impact file I/O in
> later releases?

I'm not sure if there were but many things had changed since.

> 
> My idea just was, that sometime during upload the write to the NAND
> stalls and the corresponding write call blocks so long, that a timeout
> or such is hit inside the libcppcms.

The file upload process happens withing the event loop, libcppcms
does not have any timeouts on FastCGI connections as it assumes
that the web server handles all needed (there are timeouts
on HTTP but this is not the case).

> Another probability would be, that
> something in the communication between cherokee and libcppcms fails.
> How does transferring requests from cherokee to libcppcms work exactly?

When a chunk of data received by libcppcms from FastCGI connection
it is parsed and written to the file.

> Think of an incoming HTTP POST request with a huge file appended.
> Cherokee sends this request to libcppcms, and it would start to store
> the file to the designated temporary location. What does happen, if
> writing to NAND is very slow?

It should just slow down the process. There are no particular timers
on this.

> Would libcppcms still accept bytes of the
> request from cherokee or would it fetch new bytes only if the already
> received bytes have been written to NAND 

Yes, the files are written directly (no AIO or stuff like that is used.

>(the consequence could be, that
> the network buffer queue inside of the kernel grows until cherokee has
> to wait with writing because no new packages can be sent, and this might
> hit a timeout there or lead to a buffer exhaustion that makes cherokee
> close the connection)? 

It may be that the cherokee does it.


>If libcppcms instead buffers internally if more
> bytes are incoming than can be written to NAND, are there any size
> limits to these buffers?
> 

The biggest FastCGI chunk of data is 64KB, and it is the maximal buffer
size that would be used (of course at the beginning libcppcms tries to
put as much as it can in memory and afterwards it writes to file)

The limits are defined there: 

http://cppcms.com/wikipp/en/page/cppcms_1x_config#security.file_in_memory_limit

> Best regards,
> Julian

I'd recommend you to use strace tool to see what exactly fails and check the logs. It allows you
to see system calls. If not, try to look into the line:


   file_->write_data().write(boundary_.c_str(),position_);

in 

   private/multipart_parser.h

File to see if anything goes wrong here.
 
Artyom Beilis
--------------
CppCMS - C++ Web Framework:   http://cppcms.com/
CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/