Re: [Cppcms-users] Uploads to slow storage devices

[Artyom B. <art...@ya...>]

----- Original Message -----

> From: Julian Pietron <ju...@wh...>

>>  What version of CppCMS do you use? I assume latest.
> 
> To be honest, I think that this problem occurred in 0.99.6. I didn't
> test any later versions yet. Are there changes that impact file I/O in
> later releases?

I'm not sure if there were but many things had changed since.

> 
> My idea just was, that sometime during upload the write to the NAND
> stalls and the corresponding write call blocks so long, that a timeout
> or such is hit inside the libcppcms.

The file upload process happens withing the event loop, libcppcms
does not have any timeouts on FastCGI connections as it assumes
that the web server handles all needed (there are timeouts
on HTTP but this is not the case).

> Another probability would be, that
> something in the communication between cherokee and libcppcms fails.
> How does transferring requests from cherokee to libcppcms work exactly?

When a chunk of data received by libcppcms from FastCGI connection
it is parsed and written to the file.

> Think of an incoming HTTP POST request with a huge file appended.
> Cherokee sends this request to libcppcms, and it would start to store
> the file to the designated temporary location. What does happen, if
> writing to NAND is very slow?

It should just slow down the process. There are no particular timers
on this.

> Would libcppcms still accept bytes of the
> request from cherokee or would it fetch new bytes only if the already
> received bytes have been written to NAND 

Yes, the files are written directly (no AIO or stuff like that is used.

>(the consequence could be, that
> the network buffer queue inside of the kernel grows until cherokee has
> to wait with writing because no new packages can be sent, and this might
> hit a timeout there or lead to a buffer exhaustion that makes cherokee
> close the connection)? 

It may be that the cherokee does it.


>If libcppcms instead buffers internally if more
> bytes are incoming than can be written to NAND, are there any size
> limits to these buffers?
> 

The biggest FastCGI chunk of data is 64KB, and it is the maximal buffer
size that would be used (of course at the beginning libcppcms tries to
put as much as it can in memory and afterwards it writes to file)

The limits are defined there: 

http://cppcms.com/wikipp/en/page/cppcms_1x_config#security.file_in_memory_limit

> Best regards,
> Julian

I'd recommend you to use strace tool to see what exactly fails and check the logs. It allows you
to see system calls. If not, try to look into the line:


   file_->write_data().write(boundary_.c_str(),position_);

in 

   private/multipart_parser.h

File to see if anything goes wrong here.
 
Artyom Beilis
--------------
CppCMS - C++ Web Framework:   http://cppcms.com/
CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/