Re: [Cppcms-users] Uploads to slow storage devices

[Julian P. <ju...@wh...>]

Am 11.04.2012 19:54, schrieb Artyom Beilis:
>> I recently experienced some problems when uploading (large) files (>=30
>> MB) with cherokee to a cppcms based backend application: After a part of
>> the upload has completed, a 500 Internal Server error is issued to the
>> browser and the connection (and upload) is terminated. These problems
>> only occur on an embedded board with a Cortex A8-ARM processor and a
>> very slow NAND memory (where the file shall be stored to). For file
>> corruption prevention, the file system (ext3) on this NAND is mounted in
>> sync mode so that no data is buffered in RAM but anything is written
>> directly to NAND. Even if async mode is used instead, the error occurs
>> after some more time (probably if the kernel's I/O buffer is full).
>> cherokee delivers the upload to the backend application via FastCGI.
>> My question is: Is there anything within the FastCGI/upload handling in
>> cppcms that may cause such a behaviour, if e.g. the backend application
>> needs to long to handle the request (because the file has to be stored
>> to NAND before the request is handled), or is it all the fault of the
>> webserver?
> What version of CppCMS do you use? I assume latest.

To be honest, I think that this problem occurred in 0.99.6. I didn't
test any later versions yet. Are there changes that impact file I/O in
later releases?

> Also make sure the the files are uploaded to correct location
> so you have enough space.
>
>
The location is correct and there is plenty of space available (>2-3x of
the amount needed).

My idea just was, that sometime during upload the write to the NAND
stalls and the corresponding write call blocks so long, that a timeout
or such is hit inside the libcppcms. Another probability would be, that
something in the communication between cherokee and libcppcms fails.
How does transferring requests from cherokee to libcppcms work exactly?
Think of an incoming HTTP POST request with a huge file appended.
Cherokee sends this request to libcppcms, and it would start to store
the file to the designated temporary location. What does happen, if
writing to NAND is very slow? Would libcppcms still accept bytes of the
request from cherokee or would it fetch new bytes only if the already
received bytes have been written to NAND (the consequence could be, that
the network buffer queue inside of the kernel grows until cherokee has
to wait with writing because no new packages can be sent, and this might
hit a timeout there or lead to a buffer exhaustion that makes cherokee
close the connection)? If libcppcms instead buffers internally if more
bytes are incoming than can be written to NAND, are there any size
limits to these buffers?

Best regards,
Julian