Re: [torrus-devel] multithreading in collector

[Stanislav S. <ssi...@ya...>]

--- Christian Schnidrig <chr...@gm...> wrote:

> Does anyone know if tuning the network buffers does improve performance of
> Net::SNMP. i. e. is the amount of parallelism going to increase if the
> buffers are larger? I'm asking this, since I noticed once that Net::SNMP
> will not go beyond a certain number of outstanding requests, even if there
> are a lot more in it's queue.

Currently the UDP receive buffer is set to 128k, which is the default limit 
for Linux. Have a look at "netstat -s -w" and "netstat -s -u" output, 
those counters would show if packets are dropped. Then you probably need 
to set a new limit on the kernel level, and increase the parameter in torrus 
config.

The scheduler works as follows:

1. For each host in a collecting job, we create a new non-blocking snmp 
session, and then add one or more PDUs.

2. The PDUs are in the dispatcher queue, ordered by host.

3. We launch the dispatcher. It creates one socket for all operation.

4. It enters the following loop:

while( unanswered requests in the queue )
{
  non-blocking select() on the open socket. 
  If( anything pops out )
  {
     process the incoming packet;
  }
  else
  {
     send the PDU out and delete it from the head of the queue
   }
}

Thus, if there are too many responces in the socket receive buffer, 
it will not send any new PDUs until the buffer is empty. I think it's quite 
good regulating mechanism, it prevents it from generating traffic bursts 
that it's not able to handle. 

well, maybe it's worth thinking it over once again, and distribute the jobs
to the threads, one per host...