Menu
▾
▴
#207 Save the state of downloads in the MongoDB.
Dear aria2 developers.
I have some issues with my (big, really big) download server and area state retrieval. It is slow :) I have a lot of downloads in my aria2 instance and i've made a web-interface for it, so i can control my aria remotely and check downloads progresses.
So, the problem is that aria2.tellStatus method is slow and even worse, it encourages polling. With about couple of dozens of downloads (especially torrents) it could take up to five minutes just to get status. And that's not all - aria2 becomes unresponsive, when tellSTatus is processed. I guess that's because of single threaded XML RPC serving in aria2.
This ends in a trade-off between usability and manageability. I can either poll area every 5 to 10 minutes (and wait for 5 minutes for answer) or control aria2, but without any status updates.
My question (well, to be honest, my feature request) is to make some automatic dumping of aria2 tellStatus. Could it just write it to file or may be socket? It could be nice, if aria2 would be able to write status directly to mongodb :) cause i use it internally, but it's barefaced impudence to ask for that :)
Some aria2 initiated status dump will be enough, even dumping to file will be enough, cause inotify/fanotify could be used for event generation and i kindly ask you to add this feature.
Thanks in advance!
Paul, proud aria2 user.
It is always interesting to hear about performance.
How many downloads in one aria2 instance? aria2 is not designed to handle tens or hundreds of downloads unfortunately.
What does CPU usage of aria2c look like in your environment?
It seems 5min processing time of aria2.tellStatus sounds strange.
While processing it, does CPU usage of aria2 become full?
Could you profile the bottleneck of this slowdown?
Maybe file system is so slow so that it blocks aria2 here and there?
If gathering status information itself is slow, then dumping such information to file or socket is still slow.
Thanks for the quick reply !
We use the servers with Intel(R) Xeon(R) CPU E5603 @ 1.60GHz.
Each server - 10 processes aria2c. If each aria2c of 10 downloads - processor load is only 1-5% on each aria2c.
File system is slow (periodically) =( It clarifies a lot. What are your recommendations for solving the problem?
I think we should first figure out that where the bottleneck is.
You wrote that RPC request/response was slow. To check it is really slow, we can check the log file of aria2.
You can enable logging of aria2 using -l option.
When RPC request is received by aria2, it logs the following:
2012-07-05 21:22:21.183058 INFO - [HttpListenCommand.cc:80]RPC: Accepted the connection from ::1:12345.
Then after aria2 received request body:
2012-07-05 21:22:21.183290 INFO - [HttpServer.cc:138]HTTP Server received request
POST /rpc HTTP/1.1
User-Agent: XMLRPC::Client (Ruby 1.9.3)
Content-Type: text/xml; charset=utf-8
Content-Length: 98
...
Then before executing RPC processing:
2012-07-05 21:22:21.183445 INFO - [HttpServerBodyCommand.cc:207]Executing RPC method aria2.getVersion
After processing is done, aria2 starts sending response:
2012-07-05 21:22:21.183490 DEBUG - [HttpServer.cc:256]HTTP Server sends response:
HTTP/1.1 200 OK
Date: Thu, 05 Jul 2012 12:22:21 GMT
Content-Length: 609
...
After all response body was transmitted:
2012-07-05 21:22:21.183523 INFO - [AbstractHttpServerResponseCommand.cc:85]CUID#8 - HttpServer: all response transmitted.
You can check the timestamp to see where the most time-consuming part is.
"After processing is done, aria2 starts sending response:"
The processing (aria2.tellStatus on torrent downloads) is slow if file system is slow.....=(
So, in the current architecture of aria2, to improve the performance we have no choice but to use faster file system at the moment.